qit4dZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZddlmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)dZ*Gdde Z+Gdde+Z,Gdde+e'Z-y)zRBase classes for Speech-to-Text services with continuous and segmented processing.N)abstractmethod)AnyAsyncGeneratorOptional)logger)State) AudioRawFrame ErrorFrameFrameInterruptionFrame#ServiceSwitcherRequestMetadataFrame StartFrameSTTMetadataFrame STTMuteFrameSTTUpdateSettingsFrameTranscriptionFrameVADUserStartedSpeakingFrameVADUserStoppedSpeakingFrame)FrameDirection) AIService) STTSettingsis_given)DEFAULT_TTFS_P99)WebsocketServiceLanguageg?ceZdZUdZeed<dddddddddeed ed eed eed ed eef fdZ e de fdZ dZ dZe defdZdefdZdefdZdedeefdZededeedffdZdeffd ZfdZdedeeefffd Zde d e!fd!Z"ded e!ffd" Z#e!jHfded e!ffd# Z%d$Z&d%Z'd&Z(de)fd'Z*de+fd(Z,d)Z-d*Z.d+Z/d,Z0de fd-Z1d.efd/Z2xZ3S)0 STTServiceaMBase class for speech-to-text services. Provides common functionality for STT services including audio passthrough, muting, settings management, and audio processing. Subclasses must implement the run_stt method to provide actual speech recognition. Includes an optional keepalive mechanism that sends silent audio when no real audio has been sent for a configurable timeout, preventing servers from closing idle connections (e.g. when behind a ServiceSwitcher). Subclasses that enable keepalive must override ``_send_keepalive()`` to deliver the silence in the appropriate service-specific protocol. Event handlers: on_connected: Called when connected to the STT service. on_disconnected: Called when disconnected from the STT service. on_connection_error: Called when a connection to the STT service error occurs. Example:: @stt.event_handler("on_connected") async def on_connected(stt: STTService): logger.debug(f"STT connected") @stt.event_handler("on_disconnected") async def on_disconnected(stt: STTService): logger.debug(f"STT disconnected") @stt.event_handler("on_connection_error") async def on_connection_error(stt: STTService, error: str): logger.error(f"STT connection error: {error}") _settingsTNg@g@)audio_passthrough sample_ratestt_ttfb_timeoutttfs_p99_latencykeepalive_timeoutkeepalive_intervalsettingsr!r"r#r$r%r&c t |d d|xs ti|||_||_d|_d|_d|_||_||_ d|_ d|_ d|_ d|_ d|_||_||_d|_d|_|j'd|j'd|j'dy) uInitialize the STT service. Args: audio_passthrough: Whether to pass audio frames downstream after processing. Defaults to True. sample_rate: The sample rate for audio input. If None, will be determined from the start frame. stt_ttfb_timeout: Time in seconds to wait after VAD stop before reporting TTFB. This delay allows the final transcription to arrive. Defaults to 2.0. Note: STT "TTFB" differs from traditional TTFB (which measures from a discrete request to first response byte). Since STT receives continuous audio, we measure from when the user stops speaking to when the final transcript arrives—capturing the latency that matters for voice AI applications. ttfs_p99_latency: P99 latency from speech end to final transcript in seconds. This is broadcast via STTMetadataFrame at pipeline start for downstream processors (e.g., turn strategies) to optimize timing. Subclasses provide measured defaults; pass a value here to override for your deployment. keepalive_timeout: Seconds of no audio before sending silence to keep the connection alive. None disables keepalive. Useful for services that close idle connections (e.g. behind a ServiceSwitcher). keepalive_interval: Seconds between idle checks when keepalive is enabled. settings: The runtime-updatable settings for the STT service. **kwargs: Additional arguments passed to the parent AIService. r&rFN on_connectedon_disconnectedon_connection_error)super__init__r_audio_passthrough_init_sample_rate _sample_rate_muted_user_id_ttfs_p99_latency_stt_ttfb_timeout_ttfb_timeout_task_user_speaking_finalize_pending_finalize_requested_last_transcript_time_keepalive_timeout_keepalive_interval_keepalive_task_last_audio_time_register_event_handler) selfr r!r"r#r$r%r&kwargs __class__s N/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/services/stt_service.pyr.zSTTService.__init__PsH  }    #4!,!  !1"2:>$)',). ,-"#4#5 7;'( $$^4 $$%67 $$%:;returnc|jS)zCheck if the STT service is currently muted. Returns: True if the service is muted and will not process audio. )r2r@s rCis_mutedzSTTService.is_muteds{{rDcd|_y)aMark that a finalize request has been sent, awaiting server confirmation. For providers that have explicit server confirmation of finalization (e.g., Deepgram's from_finalize field), call this when sending the finalize request. Then call confirm_finalize() when the server confirms. For providers without server confirmation, don't call this method - just send the finalize/flush/commit command and rely on the TTFB timeout. TN)r9rGs rCrequest_finalizezSTTService.request_finalizes $( rDc:|jrd|_d|_yy)aFConfirm that the server has acknowledged the finalize request. Call this when the server response confirms finalization (e.g., Deepgram's from_finalize=True). The next TranscriptionFrame pushed will be marked as finalized. Only has effect if request_finalize() was previously called. TFN)r9r8rGs rCconfirm_finalizezSTTService.confirm_finalizes"  # #%)D "',D $ $rDc|jS)zoGet the current sample rate for audio processing. Returns: The sample rate in Hz. )r1rGs rCr!zSTTService.sample_rates   rDmodelc`Ktj5tjdtjdtddddt j d|dt|j}|j||d{y#1swYXxYw7w) zSet the speech recognition model. .. deprecated:: 0.0.104 Use ``STTUpdateSettingsFrame(model=...)`` instead. Args: model: The name of the model to use for speech recognition. alwayszK'set_model' is deprecated, use 'STTUpdateSettingsFrame(model=...)' instead. stacklevelNzSwitching STT model to: [])rN warningscatch_warnings simplefilterwarnDeprecationWarningrinfotyper_update_settings)r@rN settings_clss rC set_modelzSTTService.set_models $ $ &   ! !( + MM]"    /wa89DNN+ ##Lu$=>>>   ?)B.2B  AB.B,B. B)%B.languagec`Ktj5tjdtjdtddddt j d|dt|j}|j||d{y#1swYXxYw7w) zSet the language for speech recognition. .. deprecated:: 0.0.104 Use ``STTUpdateSettingsFrame(language=...)`` instead. Args: language: The language to use for speech recognition. rPzQ'set_language' is deprecated, use 'STTUpdateSettingsFrame(language=...)' instead.rQrRNzSwitching STT language to: [rT)rarU)r@rar^s rC set_languagezSTTService.set_languages $ $ &   ! !( + MMc"    28*A>?DNN+ ##L($CDDD   Er`ct|S)zConvert a language to the service-specific language format. Args: language: The language to convert. Returns: The service-specific language identifier, or None if not supported. r)r@ras rClanguage_to_service_languagez'STTService.language_to_service_languages!!rDaudioc Kyw)aNRun speech-to-text on the provided audio data. This method must be implemented by subclasses to provide actual speech recognition functionality. Args: audio: Raw audio bytes to transcribe. Yields: Frame: Frames containing transcription results (typically TextFrame). Nr,)r@rfs rCrun_sttzSTTService.run_stts  sframec~Kt||d{|jxs |j|_y7$w)zwStart the STT service. Args: frame: The start frame containing initialization parameters. N)r-startr0audio_in_sample_rater1r@rirBs rCrkzSTTService.starts9 gmE""" 22Pe6P6P #s =;%=cKt|d{|jd{|jd{y7577 w)zClean up STT service resources.N)r-cleanup_cancel_ttfb_timeout_cancel_keepalive_taskr@rBs rCrozSTTService.cleanup sHgo'')))))+++ )+s1AA AA AAA AAdeltacKt|jr>t|jtr$|j |j}|||_t ||d{}|S7w)aApply an STT settings delta. Handles ``model`` (via parent). Translates ``Language`` enum values before applying so the stored value is a service-specific string. Concrete services should override this method and handle language changes (including any reconnect logic) based on the returned changed-field dict. Args: delta: An STT settings delta. Returns: Dict mapping changed field names to their previous values. N)rra isinstancerrer-r])r@rs convertedchangedrBs rCr]zSTTService._update_settingss_ ENN # 5>>8(L99%..II$!*0778sA&A2)A0*A2 directioncK|jrytj|_t |dr|j |_nd|_|js0tjd|jd|jy|j|j|jd{y7w)aProcess an audio frame for speech recognition. If the service is muted, this method does nothing. Otherwise, it processes the audio frame and runs speech-to-text on it, yielding transcription results. If the frame has a user_id, it is stored for later use in transcription. Args: frame: The audio frame to process. direction: The direction of frame processing. Nuser_idr(z,Empty audio frame received for STT service:  )r2time monotonicr>hasattrrzr3rfrwarningname num_framesprocess_generatorrh)r@rirxs rCprocess_audio_framezSTTService.process_audio_frame)s ;;  $ 0 5) $!MMDMDM{{ NN>tyyk5K[K[J\]  $$T\\%++%>???sB5B?7B=8B?cKt|||d{t|tr3|j ||d{|j d{yt|t r3|j d{|j ||d{yt|trB|j||d{|jr|j ||d{yyt|tr4|j|d{|j ||d{yt|tr4|j|d{|j ||d{yt|tr|j$|j!|jd{y|j"rt%j&5t%j(dt%j*dt,ddddt/|j0j3|j"}|j!|d{yyt|t4r8|j6|_t;j<d|j6rdndyt|t>r3|jAd{|j ||d{y|j ||d{y77777u7M7*77777p#1swYxYw77m7U7<w) zProcess frames, handling VAD events and audio segmentation. Args: frame: The frame to process. direction: The direction of frame processing. NrPzPassing a dict via STTUpdateSettingsFrame(settings={...}) is deprecated since 0.0.104, use STTUpdateSettingsFrame(delta=STTSettings(...)) instead.rQrRz STT service mutedunmuted)!r- process_framerur push_frame_push_stt_metadatar r rr/r!_handle_vad_user_started_speakingr!_handle_vad_user_stopped_speakingrrsr]r&rVrWrXrYrZr\r from_mappingrmuter2rdebugr _reset_stt_ttfb_state)r@rirxrsrBs rCrzSTTService.process_frameJsg#E9555 eZ (//%3 3 3))+ + + B C))+ + +//%3 3 3 } -**5)< < <&&ooeY777' : ;88? ? ?//%3 3 3 : ;88? ? ?//%3 3 3 5 6{{&++EKK888,,.))(3MMe*#$ T^^,99%..I++E222 | ,**DK LL<5::9'MN O 0 1,,. . .//%3 3 3//%3 3 3Y 6 4 + + 3 =7 ? 3 ? 393 / 3 3s1L>L)L>LL>L (L>LL>L*L>L%L>-L.*L>LL>2L3)L>L L>6L#7?L>6L&7%L>2L)A L>L6A1L> L8 L>$L:%L>?L<L>L> L>L>L>L>L>L>L> L>#L>&L>)L3. L>8L>:L><L>cHKt|trotj|_|jrd|_d|_|j r0|j d{|jd{t|%||d{y7577 w)aPush a frame downstream, tracking TranscriptionFrame timestamps for TTFB. Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout. Args: frame: The frame to push. direction: The direction to push the frame. TFN) rurr|r:r8 finalizedstop_ttfb_metricsrpr-rr@rirxrBs rCrzSTTService.push_frames e/ 0)-D &%%"&).&,,...//111g  222 /12s6A#B"&B'B">B?B"B B"B" B"cK|j}|+t}tj|jd|d|j t |j|d{y7w)zJPush STT metadata frame for downstream processors (e.g., turn strategies).Nz*: ttfs_p99_latency not set, using default s) service_namer#)r4rrrrbroadcast_framer)r@ttfss rCrzSTTService._push_stt_metadatas]%% <#D NNdii[(RSWRXXYZ [""#3$))^b"cccsAA)!A'"A)c~K|jr+|j|jd{d|_yy7 w)z%Cancel any pending TTFB timeout task.N)r6 cancel_taskrGs rCrpzSTTService._cancel_ttfb_timeouts9  " """4#:#:; ; ;&*D # # ; +=;=c@K|jd{y7w)aReset STT TTFB measurement state. Called when starting a new utterance or on interruption to ensure we don't use stale state for TTFB calculations. This specifically guards against the case where a TranscriptionFrame is received without corresponding VADUserStartedSpeakingFrame and VADUserStoppedSpeakingFrame frames. Note: Does not reset _user_speaking since InterruptionFrame can arrive while user is still speaking. N)rprGs rCrz STTService._reset_stt_ttfb_states'')))s cxK|jd{d|_d|_d|_d|_y7!w)a[Handle VAD user started speaking frame to start tracking transcriptions. Cancels any pending TTFB timeout, resets TTFB tracking state, and marks user as speaking. Also resets finalization state to prevent stale finalization from a previous utterance. Args: frame: The VAD user started speaking frame. NTFr)rr7r9r8r:r@ris rCrz,STTService._handle_vad_user_started_speakings?((***"#( !&%&" +s :8":cKd|_|jdk(ry|j|jz }|j|d{|j |j d|_y7+w)aHandle VAD user stopped speaking frame. Calculates the actual speech end time and starts a timeout task to wait for the final transcription before reporting TTFB. Args: frame: The VAD user stopped speaking frame. FgN) start_timer"r)r7 stop_secs timestampstart_ttfb_metrics create_task_ttfb_timeout_handlerr6)r@rispeech_end_times rCrz,STTService._handle_vad_user_stopped_speakings{$ ??c !   //EOO;%%%AAA#'"2"2  & & (/A#3#  BsAA6A4 ,A6c"K tj|jd{|jdkDr$|j |jd{d|_y7?7#tj $rY!wxYw#d|_wxYww)a{Wait for timeout then report TTFB using the last transcript timestamp. This timeout allows the final transcription to arrive before we calculate and report TTFB. Uses _last_transcript_time as the end time so we measure to when the transcript actually arrived, not when the timeout fired. If no transcription arrived, no TTFB is reported. Nr)end_time)asynciosleepr5r:rCancelledErrorr6rGs rCrz STTService._ttfb_timeout_handlers +-- 6 67 7 7))A-,,d6P6P,QQQ '+D # 8Q%%   '+D #sVB"A*A&2A*A(A*B&A*(A**B=B?BB B  Bc|j@tj|_|j |j d|_yy)z1Start the keepalive task if keepalive is enabled.N keepaliver)r;r|r}r>r_keepalive_task_handlerr=rGs rC_create_keepalive_taskz!STTService._create_keepalive_tasksH  " " .$(NN$4D !#'#3#3,,.[$4$D  /rDc~K|jr+|j|jd{d|_yy7 w)z#Stop the keepalive task if running.N)r=rrGs rCrqz!STTService._cancel_keepalive_tasks9   ""4#7#78 8 8#'D  8rc&K tj|jd{ |js:t j |j z }||jkrkt|jtz}d|dzz}|j|d{t j |_tj|d778#t$r$}tj|d|Yd}~yd}~wwxYww)aSend periodic silent audio to prevent the server from closing the connection. When keepalive is enabled, this task checks periodically if the connection has been idle (no audio sent) for longer than keepalive_timeout seconds. If so, it generates silent 16-bit mono PCM audio and passes it to _send_keepalive() for service-specific formatting and sending. NrQz sent keepalive silencez keepalive error: )rrr<_is_keepalive_readyr|r}r>r;intr!_KEEPALIVE_SILENCE_DURATION_send_keepalivertrace Exceptionr)r@elapsed num_samplessilencees rCrz"STTService._keepalive_task_handlers-- 8 89 9 9 //1..*T-B-BBT444!$"2"25P"PQ ![1_5**7333(,(8% v%<=> 94 $'9!=> s`$DCDC!D0C!-D.8C!&C'5C!DC!! D*D D DDcy)zCheck if the service is ready to send keepalive. Subclasses should override this to check their connection state. Returns: True if keepalive can be sent. Tr,rGs rCrzSTTService._is_keepalive_readysrDrc Ktdw)zSend silent audio to keep the connection alive. Subclasses that enable keepalive must override this to deliver silence in their service-specific protocol. Args: silence: Silent 16-bit mono PCM audio bytes. z(Subclasses must override _send_keepalive)NotImplementedErrorr@rs rCrzSTTService._send_keepalive$s""LMMs )4__name__ __module__ __qualname____doc__r__annotations__rrfloatr.propertyboolrHrJrLr!strr_rrcrerbytesrr rhrrkrodictrr]r rrr DOWNSTREAMrrrprrrrrrrrqrrr __classcell__rBs@rCrr-s@ %)"%,0-1$'*.C<c] C<  C< #5/ C<$E?C<"C<;'C34jJXIbIb3e33:d+ * '=X ' =X 6+$( 2T NU NrDrceZdZdZdddeeffdZdeffd Ze jfde de ffd Z de de ffd Z defd Zdefd Zdede fd ZxZS)SegmentedSTTServiceaSTT service that processes speech in segments using VAD events. Uses Voice Activity Detection (VAD) events to detect speech segments and runs speech-to-text only on those segments, rather than continuously. Requires VAD to be enabled in the pipeline to function properly. Maintains a small audio buffer to account for the delay between actual speech start and VAD detection. N)r!r!c ~t|dd|i|d|_d|_t |_d|_d|_y)aInitialize the segmented STT service. Args: sample_rate: The sample rate for audio input. If None, will be determined from the start frame. **kwargs: Additional arguments passed to the parent STTService. r!NrFr,)r-r._content_wave bytearray _audio_buffer_audio_buffer_size_1sr7)r@r!rArBs rCr.zSegmentedSTTService.__init__;sB ;[;F;  &[%&"#rDrichKt||d{|jdz|_y7w)zStart the segmented STT service and initialize audio buffer. Args: frame: The start frame containing initialization parameters. NrQ)r-rkr!rrms rCrkzSegmentedSTTService.startJs3 gmE"""%)%5%5%9" #s 202rxcpKt|trd|_t|||d{y7w)afPush a frame, marking TranscriptionFrames as finalized. Segmented STT services process complete speech segments and return a single TranscriptionFrame per segment, so every transcription is inherently finalized. Args: frame: The frame to push. direction: The direction of frame flow in the pipeline. TN)rurrr-rrs rCrzSegmentedSTTService.push_frameSs0 e/ 0"EOg  222s +646cKt|||d{t|tr|j |d{yt|t r|j |d{yy7Y727 w)z;Process frames, handling VAD events and audio segmentation.N)r-rrur_handle_user_started_speakingr_handle_user_stopped_speakingrs rCrz!SegmentedSTTService.process_frameasng#E9555 e8 944U; ; ; : ;44U; ; ;< 6 < ;s3A7A1(A7A3)A7*A5+A73A75A7cKd|_yw)NT)r7rs rCrz1SegmentedSTTService._handle_user_started_speakingjs"s cKd|_tj}tj|d}|j d|j d|j|j|j|j|j|jd|jj|j|j|j!d{y7w)NFwbrQr)r7ioBytesIOwaveopen setsampwidth setnchannels setframerater! writeframesrcloseseekclearrrhread)r@ricontentwavs rCrz1SegmentedSTTService._handle_user_stopped_speakingms#**,ii&   ))* **+  Q   "$$T\\',,.%ABBBsC6D8C>9Dc`Kt|dr|j|_nd|_|xj|jz c_|j sZt |j|jkDr7t |j|jz }|j|d|_yyyw)aProcess audio frames by buffering them for segmented transcription. Continuously buffers audio, growing the buffer while user is speaking and maintaining a small buffer when not speaking to account for VAD delay. If the frame has a user_id, it is stored for later use in transcription. Args: frame: The audio frame to process. direction: The direction of frame processing. rzr(N)r~rzr3rrfr7lenr)r@rirx discardeds rCrz'SegmentedSTTService.process_audio_frame~s 5) $!MMDMDM ekk)""s4+=+='>A[A['[D../$2L2LLI!%!3!3IJ!?D (\"sB,B.)rrrrrrr.rrkrrr rrrrrrr rrrs@rCrr0s8< $x} $::JXIbIb 3e 3 3<<><#9T#C9TC"@}@@rDrcveZdZdZdddefdZfdZfdZded effd Z d efd Z d e fd Z de fdZxZS)WebsocketSTTServicea#Base class for websocket-based STT services. Combines STT functionality with websocket connectivity, providing automatic error handling, reconnection capabilities, and optional silence-based keepalive. The keepalive feature (inherited from STTService) sends silent audio when no real audio has been sent for a configurable timeout, preventing servers from closing idle connections (e.g. when behind a ServiceSwitcher). Subclasses can override ``_send_keepalive()`` to wrap the silence in a service-specific protocol. T)reconnect_on_errorrc `tj|fi|tj|fd|i|y)a.Initialize the Websocket STT service. Args: reconnect_on_error: Whether to automatically reconnect on websocket errors. **kwargs: Additional arguments passed to parent classes (including keepalive_timeout and keepalive_interval for STTService). rN)rr.r)r@rrAs rCr.zWebsocketSTTService.__init__s1 D+F+!!$X;MXQWXrDc^Kt|d{|jy7w)z,Connect and start keepalive task if enabled.N)r-_connectrrrs rCrzWebsocketSTTService._connects)g    ##% !s -+-crKt|d{|jd{y77w)z%Disconnect and cancel keepalive task.N)r- _disconnectrqrrs rCrzWebsocketSTTService._disconnects3g!###))+++ $+s737577attempt_numberrEcKt||d{}|r(|jd{|j|S707w)zReconnect and restart keepalive task. The keepalive task breaks out of its loop on send errors, so it may be dead after the websocket failure that triggered this reconnect. N)r-_reconnect_websocketrqr)r@rresultrBs rCrz(WebsocketSTTService._reconnect_websocketsK w3NCC --/ / /  ' ' ) D /sA AA A A  A cn|jduxr&|jjtjuS)z7Check if the websocket is open and ready for keepalive.N) _websocketstaterOPENrGs rCrz'WebsocketSTTService._is_keepalive_readys)d*Rt/D/D /RRrDrcVK|jj|d{y7w)a.Send silent audio over the websocket to keep the connection alive. The default implementation sends raw PCM bytes directly. Subclasses can override this to wrap the silence in a service-specific protocol. Args: silence: Silent 16-bit mono PCM audio bytes. N)rsendrs rCrz#WebsocketSTTService._send_keepalives oo""7+++s )')errorcK|jd|jd{|j|d{y77w)Nr+)_call_event_handlerrpush_error_frame)r@rs rC _report_errorz!WebsocketSTTService._report_errors?&&'rsY  00%    >1;9?4"@N@NFg@*g@TC+*&6C+rD