qixXdZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZddlmZddlmZddlmZmZdd lmZmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'GddeZ(Gdde(Z)Gdde(Z*GddeZ+GddeZ,GddeZ-y)aVoicemail detection module for Pipecat. This module provides voicemail detection capabilities using parallel pipeline processing to classify incoming calls as either voicemail messages or live conversations. It's specifically designed for outbound calling scenarios where a bot needs to determine if a human answered or if the call went to voicemail. Note: The voicemail module is optimized for text LLMs only. N)ListOptional)logger) EndFrameFrameLLMFullResponseEndFrameLLMFullResponseStartFrame LLMTextFrame StopFrame SystemFrameTTSAudioRawFrameTTSStartedFrameTTSStoppedFrame TTSTextFrameUserStartedSpeakingFrameUserStoppedSpeakingFrame)ParallelPipeline) LLMContext)LLMContextAggregatorPairLLMUserAggregatorParams)FrameDirectionFrameProcessorFrameProcessorSetup) LLMService)ExternalUserTurnStrategies) BaseNotifier) EventNotifiercbeZdZdZd dedeffd Zdeffd ZfdZ de d e ffd Z d Z xZS) NotifierGateaBase gate processor that controls frame flow based on notifier signals. This base class provides common gate functionality for processors that need to start open and close permanently when a notifier signals. Subclasses define which frames are allowed through when the gate is closed. The gate starts open to allow initial processing and closes permanently once the notifier signals. This ensures controlled frame flow based on external decisions or events. notifier task_namecZt|||_||_d|_d|_y)zInitialize the notifier gate. Args: notifier: Notifier that signals when the gate should close. task_name: Name for the notification waiting task (for debugging). TN)super__init__ _notifier _task_name _gate_opened _gate_task)selfr r! __class__s a/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/extensions/voicemail/voicemail_detector.pyr$zNotifierGate.__init__?s- !# 26setupcKt||d{|j|j|_y7)wzSet up the processor with required components. Args: setup: Configuration object containing setup parameters. N)r#r- create_task_wait_for_notificationr(r)r-r*s r+r-zNotifierGate.setupLs; gmE"""**4+F+F+HI #AA*AcKt|d{|jr+|j|jd{d|_yy7<7wz!Clean up the processor resources.N)r#cleanupr( cancel_taskr)r*s r+r6zNotifierGate.cleanupUsKgo ??""4??3 3 3"DO  3!AA.AAAAframe directioncKt|||d{|jr|j||d{yt |t t tfr|j||d{yy7b7>7 wzProcess frames and control gate state based on notifier signals. Args: frame: The frame to process. direction: The direction of frame flow in the pipeline. N)r# process_framer' push_frame isinstancer rr r)r:r;r*s r+r>zNotifierGate.process_frame\s~g#E9555   //%3 3 3   (I . //%3 3 3  6 4 4s1BA:%BA<5B3A>4B<B>Bc|K|jjd{|jrd|_yy7w)zWait for notifier signal and close the gate. This method blocks until the notifier signals, then closes the gate permanently to change frame filtering behavior. NF)r%waitr'r)s r+r1z#NotifierGate._wait_for_notificationns9 nn!!###    %D   $s <:<)gate)__name__ __module__ __qualname____doc__rstrr$rr-r6rrr>r1 __classcell__r*s@r+rr3sG  7 7# 7J!4J#44>4$ &r,rc\eZdZdZdedeffd Zdeffd ZfdZde d e fd Z d Z xZ S) ClassifierGateaMGate processor that controls frame flow based on classification decisions. Inherits from NotifierGate and starts open to allow initial classification processing. Closes permanently once a classification decision is made (CONVERSATION or VOICEMAIL). This ensures the classifier only runs until a definitive decision is reached, preventing unnecessary LLM calls and maintaining system efficiency. When closed, only allows system frames and user speaking frames to continue. Speaking frames are needed for voicemail timing control, but not for conversation. gate_notifierconversation_notifiercRt||d||_d|_d|_y)aInitialize the classifier gate. Args: gate_notifier: Notifier that signals when a classification decision has been made and the gate should close. conversation_notifier: Notifier that signals when conversation is detected. classifier_gater!FN)r#r$_conversation_notifier_conversation_detected_conversation_task)r)rOrPr*s r+r$zClassifierGate.__init__s0 2CD&;#&+#:>r,r-cKt||d{|j|j|_y7)wr/)r#r-r0_wait_for_conversationrVr2s r+r-zClassifierGate.setups< gmE""""&"2"243N3N3P"Q #r3cKt|d{|jr+|j|jd{d|_yy7<7wr5)r#r6rVr7r8s r+r6zClassifierGate.cleanupsPgo  " """4#:#:; ; ;&*D # # ;r9r:r;cKtj|||d{|jr|j||d{yt |t t fr(|js|j||d{yyt |tttfr|j||d{yy77|7A7 wr=) rr>r'r?r@rrrUr rr r)r:r;s r+r>zClassifierGate.process_frames**4 BBB   //%3 3 3  8:RS T..ooeY777/  XyA B//%3 3 3 C C 4 8 4sECB>%CCrXrKrLs@r+rNrNzsG  ?l ?< ?R!4R+44>40+r,rNc(eZdZdZdeffd ZxZS)ConversationGateaGate processor that blocks conversation flow when voicemail is detected. Inherits from NotifierGate and starts open to allow normal conversation processing. Closes permanently when voicemail is detected to prevent the main conversation LLM from processing additional input after voicemail classification. When closed, only allows system frames and user speaking frames to continue. voicemail_notifierc(t||dy)zInitialize the conversation gate. Args: voicemail_notifier: Notifier that signals when voicemail has been detected and the conversation should be blocked. conversation_gaterSN)r#r$)r)r_r*s r+r$zConversationGate.__init__s +7JKr,)rFrGrHrIrr$rKrLs@r+r^r^sL<LLr,r^cteZdZdZdedededeffd Zdeffd Zfd Z d e d e ffd Z d e fdZdZxZS)ClassificationProcessoraRProcessor that handles LLM classification responses and triggers events. This processor aggregates LLM text tokens into complete responses and analyzes them to determine if the call reached a voicemail system or a live person. It uses the LLM response frame delimiters (LLMFullResponseStartFrame and LLMFullResponseEndFrame) to ensure complete token aggregation regardless of how the LLM tokenizes the response words. The processor expects responses containing either "CONVERSATION" (indicating a human answered) or "VOICEMAIL" (indicating an automated system). Once a decision is made, it triggers the appropriate notifications and event handlers. For voicemail detection, the event handler timer starts immediately and is cancelled and restarted based on user speech patterns to ensure proper timing. rOrPr_voicemail_response_delaycJt|||_||_||_||_|j d|j dd|_d|_d|_ d|_ d|_ tj|_|jjy)aInitialize the voicemail processor. Args: gate_notifier: Notifier to signal the ClassifierGate about classification decisions so it can close and stop processing. conversation_notifier: Notifier to signal the TTSGate to release all gated TTS frames for normal conversation flow. voicemail_notifier: Notifier to signal the TTSGate to clear gated TTS frames since voicemail was detected. voicemail_response_delay: Delay in seconds after user stops speaking before triggering the voicemail event handler. This ensures the voicemail greeting or user message is complete before responding. on_conversation_detectedon_voicemail_detectedFN)r#r$_gate_notifierrT_voicemail_notifier_voicemail_response_delay_register_event_handler_processing_response_response_buffer_decision_made_voicemail_detected_voicemail_taskasyncioEvent_voicemail_eventset)r)rOrPr_rdr*s r+r$z ClassificationProcessor.__init__s* +&;##5 )A& $$%?@ $$%<=%*! "#$) 7; '  !!#r,r-cKt||d{|j|j|_y7)wr/)r#r-r0_delayed_voicemail_handlerrqr2s r+r-zClassificationProcessor.setups< gmE"""#//0O0O0QR #r3cKt|d{|jr+|j|jd{d|_yy7<7wr5)r#r6rqr7r8s r+r6zClassificationProcessor.cleanupsPgo   ""4#7#78 8 8#'D   8r9r:r;cTKt|||d{t|trd|_d|_yt|t rX|jr=|js1|j|j jd{d|_d|_yt|tr,|jr |xj |jz c_yt|trA|jr|jj|j!||d{yt|t"rA|jr|jj%|j!||d{y|j!||d{y777w7(7w)a+Process frames and handle LLM classification responses. This method implements a state machine for aggregating LLM responses: 1. LLMFullResponseStartFrame: Begin collecting tokens 2. LLMTextFrame: Accumulate text tokens into buffer 3. LLMFullResponseEndFrame: Process complete response and make decision 4. UserStartedSpeakingFrame/UserStoppedSpeakingFrame: Manage voicemail timing Args: frame: The frame to process. direction: The direction of frame flow in the pipeline. NTrhF)r#r>r@r rmrnrro_process_classificationstripr textrrprtrur?rclearrAs r+r>z%ClassificationProcessor.process_frame"sXg#E9555 e6 7(,D %$&D ! 6 7((1D1D2243H3H3N3N3PQQQ(-D %$&D ! | ,1J1J  ! !UZZ / ! 7 8''%%))+//%3 3 3 7 8''%%++-//%3 3 3 //%3 3 3C 6R 4 4 4sZF(FA7F(FBF(*F"+AF(;F$<F(F&F(F("F($F(&F( full_responsecK|jry|j}tj|d|dd|vr}d|_tj|d|j j d{|jj d{|jdd{yd|vrd|_d|_ tj|d |j j d{|jj d{|jd{|jjytj|d |dy7777{7[7Ew) aProcess the complete LLM classification response and trigger actions. Analyzes the aggregated response text to determine if it contains "CONVERSATION" or "VOICEMAIL" and triggers the appropriate notifications and callbacks based on the classification result. Args: full_response: The complete aggregated response text from the LLM. Nz: Classifying response: '' CONVERSATIONTz: CONVERSATION detectedrf VOICEMAILz: VOICEMAIL detectedz: No classification found: ')roupperrdebuginforinotifyrT_call_event_handlerrprjbroadcast_interruptionrtr})r)r~responses r+rzz/ClassificationProcessor._process_classificationRsb     &&( v6}oQGH X %"&D  KK4& 78 9%%,,. . .--446 6 6**+EF F F H $"&D '+D $ KK4& 45 6%%,,. . .**113 3 3--/ / /  ! ! ' ' ) LLD6!=m_AN O) / 6 F / 3 0smA9F;E9rJrzrwrKrLs@r+rcrcsr ($$($ , ($ ) ($ #( ($TS!4S(.4.4>.4`(P3(PTr,rccfeZdZdZdedeffd Zdeffd ZfdZde d e ffd Z d Z d Z xZS) TTSGateaGates TTS frames until voicemail classification decision is made. This processor holds TTS output frames in a gate while the voicemail classification is in progress. This prevents audio from being played to the caller before determining if they're human or a voicemail system. The gate operates in two modes based on the classification result: - CONVERSATION: Opens the gate to release all held frames for normal dialogue - VOICEMAIL: Clears held frames since they're not needed for voicemail The gating only applies to TTS-related frames (TTSTextFrame, TTSAudioRawFrame). All other frames pass through immediately to maintain proper pipeline flow. rPr_cvt|||_||_g|_d|_d|_d|_y)a]Initialize the TTS gate. Args: conversation_notifier: Notifier that signals when a conversation is detected and gated frames should be released for playback. voicemail_notifier: Notifier that signals when voicemail is detected and gated frames should be cleared (not played). TN)r#r$rTrj _frame_buffer_gating_activerVrq)r)rPr_r*s r+r$zTTSGate.__init__s@ &;##5 AC":>7;r,r-cKt||d{|j|j|_|j|j |_y7Mwr/)r#r-r0rXrV_wait_for_voicemailrqr2s r+r-z TTSGate.setupsX gmE""""&"2"243N3N3P"Q#//0H0H0JK #sA&A$AA&c Kt|d{|jr*|j|jd{d|_|jr+|j|jd{d|_yy7r7E7wr5)r#r6rVr7rqr8s r+r6zTTSGate.cleanupsgo  " """4#:#:; ; ;&*D #   ""4#7#78 8 8#'D  ; 9s3BB.BB 5B:B ;B B Br:r;cKt|||d{|jr>t|tt t tfr|jj||fy|j||d{y7i7w)a}Process frames and handle gating logic based on frame type. TTS frames are gated while classification is active. All other frames pass through immediately. The gating state is controlled by the classification notifications. Args: frame: The frame to process. direction: The direction of frame flow in the pipeline. N) r#r>rr@rrrr rappendr?rAs r+r>zTTSGate.process_frames{g#E9555   : O_lDTU$     % %ui&8 9//%3 3 3 6 4s"BBA#B;B<BBcK|jjd{d|_|jD]\}}|j ||d{!|jj y7T7#w)aWait for conversation detection notification and release gated frames. When a conversation is detected, all gated TTS frames are released in order to continue normal dialogue flow. This allows the bot to respond naturally to the human caller. NF)rTrCrrr?r}r[s r+rXzTTSGate._wait_for_conversationsu))..000$ $ 2 2 4 E9//%3 3 3 4   " 1 4s!A9A52A9A7"A97A9cK|jjd{d|_|jj y7&w)aTWait for voicemail detection notification and clear gated frames. When voicemail is detected, all gated TTS frames are discarded since they were intended for human conversation and are not appropriate for voicemail systems. The developer event handlers will handle voicemail- specific audio output. NF)rjrCrrr}rDs r+rzTTSGate._wait_for_voicemailsA&&++---$   " .sA A'A )rFrGrHrIrr$rr-r6rrr>rXrrKrLs@r+rrsM 4. # #r,rceZdZdZdZdezZddddeded ee ffd Z d e d dfd Z ddZ d e fdZde ffd ZxZS)VoicemailDetectora] Parallel pipeline for detecting voicemail vs. live conversation in outbound calls. This detector uses a parallel pipeline architecture to perform real-time classification of outbound phone calls without interrupting the conversation flow. It determines whether a human answered the phone or if the call went to a voicemail system. Architecture: - Conversation branch: Empty pipeline that allows normal frame flow - Classification branch: Contains the LLM classifier and decision logic The system uses a gate mechanism to control when classification runs and a gating system to prevent TTS output until classification is complete. Once a decision is made, the appropriate action is taken: - CONVERSATION: Continue normal bot dialogue - VOICEMAIL: Trigger developer event handler for custom voicemail handling Example:: classification_llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) detector = VoicemailDetector(llm=classification_llm) @detector.event_handler("on_voicemail_detected") async def handle_voicemail(processor): await processor.push_frame(TTSSpeakFrame("Please leave a message.")) pipeline = Pipeline([ transport.input(), stt, detector.detector(), # Classification context_aggregator.user(), llm, tts, detector.gate(), # TTS gating transport.output(), context_aggregator.assistant(), ]) # For custom prompts, append the required response instruction: custom_prompt = "Your custom classification logic here. " + VoicemailDetector.CLASSIFIER_RESPONSE_INSTRUCTION Events: on_conversation_detected: Triggered when a human conversation is detected. The event handler receives one argument: the ClassificationProcessor instance which can be used to push frames. on_voicemail_detected: Triggered when voicemail is detected after the configured delay. The event handler receives one argument: the ClassificationProcessor instance which can be used to push frames. Constants: CLASSIFIER_RESPONSE_INSTRUCTION: The exact text that must be included in custom system prompts to ensure proper classification functionality. zbRespond with ONLY "CONVERSATION" if a person answered, or "VOICEMAIL" if it's voicemail/recording.aYou are a voicemail detection classifier for an OUTBOUND calling system. A bot has called a phone number and you need to determine if a human answered or if the call went to voicemail based on the provided text. HUMAN ANSWERED - LIVE CONVERSATION (respond "CONVERSATION"): - Personal greetings: "Hello?", "Hi", "Yeah?", "John speaking" - Interactive responses: "Who is this?", "What do you want?", "Can I help you?" - Conversational tone expecting back-and-forth dialogue - Questions directed at the caller: "Hello? Anyone there?" - Informal responses: "Yep", "What's up?", "Speaking" - Natural, spontaneous speech patterns - Immediate acknowledgment of the call VOICEMAIL SYSTEM (respond "VOICEMAIL"): - Automated voicemail greetings: "Hi, you've reached [name], please leave a message" - Phone carrier messages: "The number you have dialed is not in service", "Please leave a message", "All circuits are busy" - Professional voicemail: "This is [name], I'm not available right now" - Instructions about leaving messages: "leave a message", "leave your name and number" - References to callback or messaging: "call me back", "I'll get back to you" - Carrier system messages: "mailbox is full", "has not been set up" - Business hours messages: "our office is currently closed" g@N)rdcustom_system_promptllmrdrc ||_||n |j|_||_||j |d|jdg|_t |j |_t|jtt|_ t|_ t|_t|_t!|j|j|_t%|j|_t)|j|j|j||_t-|j|j|_t0|e|j&g|j"|jj5|j|j*|jj7g|j9dy)aInitialize the voicemail detector with classification and buffering components. Args: llm: LLM service used for voicemail vs conversation classification. Should be fast and reliable for real-time classification. voicemail_response_delay: Delay in seconds after user stops speaking before triggering the voicemail event handler. This allows voicemail responses to be played back after a short delay to ensure the response occurs during the voicemail recording. Default is 2.0 seconds. custom_system_prompt: Optional custom system prompt for classification. If None, uses the default prompt optimized for outbound calling scenarios. Custom prompts should instruct the LLM to respond with exactly "CONVERSATION" or "VOICEMAIL" for proper detection functionality. Nsystem)rolecontent)user_turn_strategies) user_params)rOrPr_rdrg)_classifier_llmDEFAULT_SYSTEM_PROMPT_promptrk_validate_prompt _messagesr_contextrrr_context_aggregatorrrirTrjrN_classifier_gater^_conversation_gaterc_classification_processorr_voicemail_gater#r$user assistantrl)r)rrdrr*s r+r$zVoicemailDetector.__init__Ms* #$8$D $JdJd *B& +  ! !"6 7 !<<  #4>>2 #; MM/E_Eab$  ,o&3o##0? !/t/B/BDD_D_ `"243K3K"L)@--"&"="=#77%= * & 't'B'BDD\D\]   $ $ %%%((--/$$..((224    $$%<=r,promptreturncdd|v}d|v}|r|s$tjd|jdyy)aValidate custom prompt contains required response format instructions. Custom prompts must instruct the LLM to respond with exactly "CONVERSATION" or "VOICEMAIL" for the detection logic to work properly. This method checks for the presence of these keywords and warns if they're missing. Args: prompt: The custom system prompt to validate. rrzCustom system prompt should instruct the LLM to respond with exactly "CONVERSATION" or "VOICEMAIL" for proper detection functionality. Consider appending VoicemailDetector.CLASSIFIER_RESPONSE_INSTRUCTION to your prompt: ""N)rwarningCLASSIFIER_RESPONSE_INSTRUCTION)r)rhas_conversation has_voicemails r+rz"VoicemailDetector._validate_promptsI*V3#v- } NN889< (5r,c|S)aLGet the detector pipeline for placement after STT in the main pipeline. This should be placed after the STT service and before the context aggregator in your main pipeline to enable voicemail classification. Returns: The VoicemailDetector instance itself (which is a ParallelPipeline). rDs r+detectorzVoicemailDetector.detectors  r,c|jS)aGet the gate processor for placement after TTS in the main pipeline. This should be placed after the TTS service and before the transport output to enable TTS frame gating during classification. Returns: The TTSGate processor instance. )rrDs r+rEzVoicemailDetector.gates###r, event_namech|dvr|jj||yt|||y)zAdd an event handler for voicemail detection events. Args: event_name: The name of the event to handle. handler: The function to call when the event occurs. )rfrgN)radd_event_handlerr#)r)rhandlerr*s r+rz#VoicemailDetector.add_event_handlers3 N N  * * < M>#( M> 'sm M>^st*  $g $ ;C ; ;r,r).rIrrtypingrrlogururpipecat.frames.framesrrrr r r r r rrrrr"pipecat.pipeline.parallel_pipeliner*pipecat.processors.aggregators.llm_contextr5pipecat.processors.aggregators.llm_response_universalrr"pipecat.processors.frame_processorrrrpipecat.services.llm_servicer"pipecat.turns.user_turn_strategiesr pipecat.utils.sync.base_notifierr!pipecat.utils.sync.event_notifierrrrNr^rcrrrr,r+rs !@Acb3I9;D&>D&NE+\E+PL|L*tntnh#nh#VX;(X;r,