qiRdZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z m Z ddlZddlmZddlmZddlmZmZmZmZmZmZmZmZmZdd lmZdd lm Z m!Z!m"Z"m#Z#dd l$m%Z%m&Z&dd l'm(Z(m)Z)dd l*m+Z+m,Z,ddl-m.Z.ddl/m0Z0 ddl1m2Z3ddl4m5Z5de+de e:fdZ;Gdde:e ZGdde(Z?d e@de:fd!ZAGd"d#e)ZBy#e6$r7Z7ejpde7ejpde9de7dZ7[7wwxYw)$zElevenLabs speech-to-text service implementation. This module provides integration with ElevenLabs' Speech-to-Text API for transcription using segmented audio processing. The service uploads audio files and receives transcription results directly. N) dataclassfield)Enum)AnyAsyncGeneratorOptional)logger) BaseModel) CancelFrameEndFrame ErrorFrameFrameInterimTranscriptionFrame StartFrameTranscriptionFrameVADUserStartedSpeakingFrameVADUserStoppedSpeakingFrame)FrameDirection) NOT_GIVEN STTSettings _NotGiven_warn_deprecated_param)ELEVENLABS_REALTIME_TTFS_P99ELEVENLABS_TTFS_P99)SegmentedSTTServiceWebsocketSTTService)Languageresolve_language)time_now_iso8601) traced_stt)connect)Statez Exception: zZIn order to use ElevenLabs Realtime STT, you need to `pip install pipecat-ai[elevenlabs]`.zMissing module: languagereturnc@ itjdtjdtjdtjdtj dtj dtjdtjdtjd tjd tjd tjd tjd tjdtjdtj dtj"ditj$dtj&dtj(dtj*dtj,dtj.dtj0dtj2dtj4dtj6dtj8dtj:dtj<dtj>dtj@d tjBd!tjDd"itjFd#tjHd$tjJd%tjLd&tjNd'tjPd(tjRd)tjTd*tjVd+tjXd,tjZd-tj\d.tj^d/tj`d0tjbd1tjdd2tjfd3itjhd4tjjd5tjld6tjnd7tjpd8tjrd9tjtd:tjvd;tjxd<tjzd=tj|d>tj~d?tjd@tjdAtjdBtjdCtjdDitjdEtjdFtjdGtjdHtjdItjdJtjdKtjdLtjdMtjdNtjdOtjdPtjdQtjdRtjdStjdTtjdUtjdVtjdWtjdXtjdYtjdZtjd[tjd\tjd]tjd^tjd_tjd`tjdatjdbtjdci}t||ddeS)fa"Convert a Language enum to ElevenLabs language code. Source: https://elevenlabs.io/docs/capabilities/speech-to-text Args: language: The Language enum value to convert. Returns: The corresponding ElevenLabs language code, or None if not supported. aframharahyeasmastazebelbenbosbulmyayuecatcebnyahrvcesdannldengestfilfinfrafulglglugkatdeuellgujhauhebhinhunisliboindgleitajpnjavkeakankazkhmkorkurkirlaolavlinlitluoltzmkdmsamalmltzhomrimarmonnepnsonorocioripusfaspolporpanronrussrpsnasndslkslvsomspaswaswetamtgktelthaturukrumburduzbviecymwolxhozulF) use_base_code)erAFAMARHYASASTAZBEBNBSBGMYYUECACEBNYHRCSDANLENETFILFIFRFFGLLGKADEELGUHAHEHIHUISIGIDGAITJAJVKEAKNKKKMKOKUKYLOLVLNLTLUOLBMKMSMLMTZHMIMRMNNENSONOOCORPSFAPLPTPARORUSRSNSDSKSLSOESSWSVTATGTETHTRUKUMBURUZVICYWOXHZUr)r# LANGUAGE_MAPs Q/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/services/elevenlabs/stt.pylanguage_to_elevenlabs_languager8sd Ud Ud  Ud  U d  U d  e d  Ud  Ud  Ud  Ud  Ud  Ud  ed  Ud  ed  U!d"  U#d$  U%d&  U'd(  U)d*  U+d,  U-d.  e/d0  U1d2  U3d4  U5d6  U7d8  U9d:  U;d<  U=d>  U?d@  UAdB  UCdD  UEdF  UGdH  UIdJ  UKdL  UMdN  UOdP  UQdR  USdT  UUdV  UWdX  eYdZ  U[d\  U]d^  U_d`  Uadb  Ucdd  Uedf  Ugdh  Uidj  Ukdl  Umdn  eodp  Uqdr  Usdt  Uudv  Uwdx  Uydz  U{d|  U}d~  Ud@  UAdB  UCdD  eEdF  UGdH  UIdJ  UKdL  UMdN  UOdP  UQdR  USdT  UUdV  UWdX  UYdZ  U[d\  U]d^  U_d`  Uadb  Ucdd  Uedf  Ugdh  Uidj  Ukdl  U U U U U U e U U U U U U UGdLL Hl% HHceZdZdZdZdZy)CommitStrategyz.Commit strategies for transcript segmentation.manualvadN)__name__ __module__ __qualname____doc__MANUALVADrrrrs8 F Crrc<eZdZUdZedZedzezed<y)ElevenLabsSTTSettingszSettings for ElevenLabsSTTService. Parameters: tag_audio_events: Whether to include audio events like (laughter), (coughing) in the transcription. ctSNrrrrzElevenLabsSTTSettings.sirdefault_factoryNtag_audio_events) rrrrrrboolr__annotations__rrrrrs#16FW0XdTkI-XrrceZdZUdZedZedzezed<edZ edzezed<edZ e dzezed <ed Z e dzezed <y) ElevenLabsRealtimeSTTSettingsaSettings for ElevenLabsRealtimeSTTService. See ``ElevenLabsRealtimeSTTService.InputParams`` for detailed descriptions. Parameters: vad_silence_threshold_secs: Seconds of silence before VAD commits (0.3-3.0). vad_threshold: VAD sensitivity (0.1-0.9, lower is more sensitive). min_speech_duration_ms: Minimum speech duration for VAD (50-2000ms). min_silence_duration_ms: Minimum silence duration for VAD (50-2000ms). ctSrrrrrrz&ElevenLabsRealtimeSTTSettings.sYbrrNvad_silence_threshold_secsctSrrrrrrz&ElevenLabsRealtimeSTTSettings.sIr vad_thresholdctSrrrrrrz&ElevenLabsRealtimeSTTSettings.sS\rmin_speech_duration_msctSrrrrrrz&ElevenLabsRealtimeSTTSettings.sT]rmin_silence_duration_ms) rrrrrr floatrrr rintrrrrrrsq >#7#78 (?(?@ >> * * 6 NN-s4>>3R3R/S/Y/Y/[ \==%%cg%F  (#%#+==?2  ? |LM"CHOOCTTVWaVb cdd#==?*F   2+     smDGF'G!#F/F)A F/F+F/ G!F-"G)F/+F/-G/G5F8 6G=G transcriptis_finalc@K|jd{y7wz+Handle a transcription result with tracing.N)stop_processing_metricsr)rPrQr#s r_handle_transcriptionz*ElevenLabsSTTService._handle_transcriptionms **,,, audiocK |jd{|j|d{}|jddj}|rl|jdd}|j |d|d{t j d|dt||jt|| yy777H#t$r}td | Yd}~yd}~wwxYww) aTranscribe an audio segment using ElevenLabs' STT API. Args: audio: Raw audio bytes in WAV format (already converted by base class). Yields: Frame: TranscriptionFrame containing the transcribed text, or ErrorFrame on failure. Note: The audio is already in WAV format from the SegmentedSTTService. Only non-empty transcriptions are yielded. NrGr:r:TzTranscription: []rNzUnknown error occurred: )rH) start_processing_metricsrOgetstriprVr debugr_user_idrrIr )r)rXrNrGdetected_languagees rrun_sttzElevenLabsSTTService.run_sttts C//1 1 1 11%88F::fb)//1D$*JJ$F!00t=NOOO /vQ78(MM$&%!  29P C%=aS#AB B B CsbC2C CC CAC ?CAC C2C C C C/C*%C2*C//C2r)rrrrrSettingsrr rrrCr> ClientSessionrrrr%rr/rr2bytesdictrOr rVrrrd __classcell__r,s@rrrs:%H$$ &i &&4#%)(,48,?E(E(!.. E(  E( } E(c]E(%E(01E(#5/E(Nd 9X 9(3- 9%%%D%NIM--)--9A#-- %C5%C^E4K-H%Crrrc|xdk(ryxdk(ryxdk(ryxdk(ryxd k(ry d k(ry tjd |dy)zGet the appropriate audio format string for a given sample rate. Args: sample_rate: The audio sample rate in Hz. Returns: The ElevenLabs audio format string. i@pcm_8000i> pcm_16000i"V pcm_22050i] pcm_24000iD pcm_44100i pcm_48000zD>_>_ ;171E1E .:@:W:W 7;A;Y;Y 8%+%>%>"!'!6!6-3-N-N*    ) )( 3  #-  %      !!!0#5 -+E( '  !!#rr$cy)zCheck if the service can generate processing metrics. Returns: True, as ElevenLabs Realtime STT service supports metrics generation. Trr.s rr/z1ElevenLabsRealtimeSTTService.can_generate_metricsLr0rdeltacKt||d{}|s|S|jr0|jd{|j d{|S7F7 7 w)aApply a settings delta and reconnect if anything changed. Args: delta: A :class:`STTSettings` (or ``ElevenLabsRealtimeSTTSettings``) delta. Returns: Dict mapping changed field names to their previous values. N)r$_update_settings _websocket _disconnect_connect)r)rchangedr,s rrz-ElevenLabsRealtimeSTTService._update_settingsTs_077N ??""$ $ $--/ ! !8 % !s1A#A'A#AA#A!A#A#!A#framecKt||d{t|j|_|j d{y777w)zStart the STT service and establish WebSocket connection. Args: frame: Frame indicating service should start. N)r$startrsrrrr)rr,s rrz"ElevenLabsRealtimeSTTService.starthsF gmE""":4;K;KLmmo #s!AA1AA AActKt||d{|jd{y77w)zStop the STT service and close WebSocket connection. Args: frame: Frame indicating service should stop. N)r$stoprrs rrz!ElevenLabsRealtimeSTTService.stoprs6 gl5!!!    " 848688ctKt||d{|jd{y77w)zCancel the STT service and close WebSocket connection. Args: frame: Frame indicating service should be cancelled. N)r$cancelrrs rrz#ElevenLabsRealtimeSTTService.cancel{s6 gnU###    $ rc@K|jd{y7w)zBStart performance metrics collection for transcription processing.N)r]r.s r_start_metricsz+ElevenLabsRealtimeSTTService._start_metricss++---rW directioncxKt|||d{t|tr|j d{yt|t r|j tjk(r|jr|jjtjur^ ddd|jd}|jjtj |d{t#j$dyyyyy777"#t&$r"}t#j(d|Yd}~yd}~wwxYww)zProcess incoming frames and handle speech events. Args: frame: The frame to process. direction: Direction of frame flow in the pipeline. Ninput_audio_chunkrZT message_type audio_base_64commitrz Sent manual commit to ElevenLabszFailed to send commit: )r$ process_frame isinstancerrrrrrrstater"OPENrsendrJdumpsr tracerIrr)r)rrcommit_messagercr,s rrz*ElevenLabsRealtimeSTTService.process_framesg#E9555 e8 9%%' ' ' : ;$$(=(==??t'<'< 'J F,?-/&*+/+;+; * #oo224::n3MNNN %GH(K?>< 6 (O$F)@'DEEFs^D:D'D:DA$D:%AD 'D (D D:D: D D7D2-D:2D77D:rXcK|jjd{|jr&|jjtj ur|j d{|jr|jjtjurl tj|jd}d|d|jd}|jjtj|d{dy777#t$r$}t!dt#|Yd}~4d}~wwxYww)zProcess audio data for speech-to-text transcription. Args: audio: Raw audio bytes to transcribe. Yields: None - transcription results are handled via WebSocket responses. Nutf-8rFrzElevenLabs Realtime STT error: )rwaitrrr"CLOSEDrrbase64 b64encodedecoderrrJrrIr rC)r)rX audio_base64messagercs rrdz$ElevenLabsRealtimeSTTService.run_stts ##((***$//"7"75<<"G--/ ! ! ??t44 B M%//6==gF %8%1##'#3#3  oo**4::g+>??? - + "@ M #B3q6(!KLLL Ms_E DA E *D+6E "A&DD D E E D E"E<E EE cK|jj |jd{t|d{|j r;|j s/|j|j|j|_|jjy7|7h#|jjwxYww)z:Establish WebSocket connection to ElevenLabs Realtime STT.N) rclear_connect_websocketr$rrr create_task_receive_task_handler _report_errorrr)r,s rrz%ElevenLabsRealtimeSTTService._connects ##% ())+ + +'"$ $ $t'9'9%)%5%5..t/A/AB&"  ! ! % % ' , $  ! ! % % 's?CB3B/B3B1 A B3C/B31B33CCcKt|d{|jr*|j|jd{d|_|j d{y7S7&7 w)z-Close WebSocket connection and cleanup tasks.N)r$rr cancel_task_disconnect_websocketrs rrz(ElevenLabsRealtimeSTTService._disconnectsfg!###   ""4#5#56 6 6!%D ((*** $ 7 +s3A/A).A/A+A/#A-$A/+A/-A/silencecKtj|jd}d|d|jd}|jj t j|d{y7w)zSend silent audio wrapped in ElevenLabs' JSON protocol. Args: silence: Silent 16-bit mono PCM audio bytes. rrFrN)rrrrrrrJr)r)rrrs r_send_keepalivez,ElevenLabsRealtimeSTTService._send_keepalives_ ''077@ /)++   oo""4::g#6777sA'A1)A/*A1cK |jr'|jjtjuryt j dd|j jg}|j jr(|jd|j j|jd|j|jd|jj|jr5|jdt|jj|j r5|jdt|j j|j"r5|jd t|j"j|jt$j&k(r|j j((|jd |j j(|j j*(|jd |j j*|j j,(|jd |j j,|j j.(|jd |j j.d|j0ddj3|}d|j4i}t7||d{|_|j9dd{t j dy787#t:$r)}|j=d||d{7Yd}~yd}~wwxYww)z6Connect to ElevenLabs Realtime STT WebSocket endpoint.Nz%Connecting to ElevenLabs Realtime STTz model_id=zlanguage_code=z audio_format=zcommit_strategy=zinclude_timestamps=zenable_logging=zinclude_language_detection=zvad_silence_threshold_secs=zvad_threshold=zmin_speech_duration_ms=zmin_silence_duration_ms=zwss://z/v1/speech-to-text/realtime?&r5)additional_headers on_connectedz$Connected to ElevenLabs Realtime STTz.Unable to connect to ElevenLabs Realtime STT:  error_msg exception)rrr"rr r`rrr#appendrrvaluerrCrDrrrrr r rrr'joinr&websocket_connect_call_event_handlerrI push_error)r)rws_urlr<rcs rrz/ElevenLabsRealtimeSTTService._connect_websockets4 4??#8#8EJJ#F LL@ A"$.."6"6!789F~~&& t~~/F/F.GHI MMM$*<*<)=> ? MM,T-B-B-H-H,IJ K'' 3C8P8P4Q4W4W4Y3Z[\## D4H4H0I0O0O0Q/RST// 1#d6V6V2W2]2]2_1`a $$(:(::>><<HMM5dnn6_6_5`a>>//;MMN4>>3O3O2P"QR>>88DMM$;DNN>99EMM24>>3Y3Y2Z[dnn--I#((SYJZI[\F#T]]3G$5fQX$YYDO**>: : : LL? @Z : //J1#NZ["   siM32L>M3K L>L:L>L< L>9M3:L><L>> M0M+ M#!M+&M3+M00M3cK |jr]|jjtjur7t j d|jj d{d|_|jdd{y7%#t$r)}|jd||d{7Yd}~Pd}~wwxYw7<#d|_|jdd{7wxYww)z2Disconnect from ElevenLabs Realtime STT WebSocket.z*Disconnecting from ElevenLabs Realtime STTNzError closing websocket: ron_disconnected) rrr"rr r`closerIrr)r)rcs rrz2ElevenLabsRealtimeSTTService._disconnect_websocket&s >4??#8#8EJJ#F IJoo++---#DO**+<= = = . Z//.Gs,KWX/Y Y Y Z >#DO**+<= = =swC-A$B(B)B-C-C C-B CB=2B53B=8C=CCC-C*#C&$C**C-cH|jr |jStd)zGet the current WebSocket connection. Returns: The WebSocket connection. Raises: Exception: If WebSocket is not connected. zWebsocket not connected)rrIr.s r_get_websocketz+ElevenLabsRealtimeSTTService._get_websocket2s! ???? "122rc\K|j23d{} tj|}|j|d{9747#tj$rt j d|Yjt$r"}t jd|Yd}~d}~wwxYw6yw)z4Continuously receive and process WebSocket messages.NzReceived non-JSON message: zError processing message: ) rrJloads_process_responseJSONDecodeErrorr rrrIrH)r)rr;rcs r_receive_messagesz.ElevenLabsRealtimeSTTService._receive_messages?s!002 ? ?' ?zz'*,,T222 ?3'' H!> ? 3sfB,B*A B*B,)AAA B, B*A+B';B,=B'B"B,"B''B,r;cK|jd}|dk(rtjd|y|dk(r|j|d{y|dk(r|j |d{y|dk(r|j |d{y|dvr3|jd d |}|j d | d{ytjd |y77v7Y7$w)zpProcess a response message from ElevenLabs. Args: data: Parsed JSON response data. rsession_startedzElevenLabs session started: partial_transcriptNcommitted_transcript$committed_transcript_with_timestamps)rH auth_errorquota_exceeded_errortranscriber_error input_errorcommit_throttledrunaccepted_terms_error rate_limitedqueue_overflowresource_exhaustedsession_time_limit_exceededchunk_size_exceededinsufficient_audio_activityrHzUnknown error - zError: )rzUnknown message type: )r^r r`_on_partial_transcript_on_committed_transcript(_on_committed_transcript_with_timestampsr)r)r;rrs rrz.ElevenLabsRealtimeSTTService._process_responseKs xx/ , , LL7v> ? 1 1--d3 3 3 3 3//5 5 5 C C??E E E   ,<\N*KLI//gi[,A/B B B LL1,@ A7 4 6 F& CsHA C' C C'*C!+C' C# 6C'C%C'!C'#C'%C'c K|jddj}|sy|jd}tjd|d|j t ||j t||d{y7w)zoHandle partial transcript (interim results). Args: data: Partial transcript data. rGrZNr:zPartial transcript: [r[r\)r^r_r r push_framerrar)r)r;rGr#s rrz3ElevenLabsRealtimeSTTService._on_partial_transcriptts xx#))+ 88O, ,TF!45oo %  "     sBB B B rPrQr#c KywrSrrUs rrVz2ElevenLabsRealtimeSTTService._handle_transcriptions sc K|jry|jddj}|sy|jd{|jd}t j d|d|j |d|d{|jtjk(}|jt||jt|||d{y77^7 w) zqHandle committed transcript (final results). Args: data: Committed transcript data. NrGrZr:zCommitted transcript: [r[TrN finalized)rr^r_rTr r`rVrrrrrrarr)r;rGr#rs rrz5ElevenLabsRealtimeSTTService._on_committed_transcripts  # # xx#))+ **,,,88O, .tfA67((tX>>>))^-B-BB oo   "#     - ? s8AC.C(AC. C* AC."C,#C.*C.,C.c K|jddj}|sy|jd{|jd}tjd|d|j |d|d{|j tjk(}|jt||jt|||d{y77^7 w) aHandle committed transcript with word-level timestamps. This message is sent when include_timestamps=true. The result data includes: - text: The transcribed text - language_code: Detected language (if available) - words: Array of word objects with timing information: - text: The word text - start: Start time in seconds - end: End time in seconds - type: "word" or "spacing" - speaker_id: Speaker identifier (if available) - logprob: Log probability score (if available) - characters: Array of character strings (if available) Args: data: Committed transcript data with timestamps. rGrZNr:z'Committed transcript with timestamps: [r[Tr) r^r_rTr r`rVrrrrrrarrs rrzEElevenLabsRealtimeSTTService._on_committed_transcript_with_timestampss$xx#))+ **,,,88O, >tfAFG((tX>>>))^-B-BB oo   "#     - ? s57C!CAC!>C?AC!CC!C!C!r)2rrrrrrerr rrrrrCrrrrr%r/rrhrrrrr rr rrrrrrgrrdrrrrrrrrrr rVrrrirjs@rrurus -H,,1i1H,*8*?*?#%)#($+0(,<@,Hc$c$ c$ ( c$ } c$c]c$!c$c$%)c$%c$89c$#5/c$JdKDcN(!!!+!.FF>F8 5 ^E4K-H D (+ 8U 86p > 3 ?'BD'BR  2IM  )- 9A#  # 4# J, 4, rru)CrrrrArJ dataclassesrrenumrtypingrrrr>logurur pydanticr pipecat.frames.framesr r r rrrrrr"pipecat.processors.frame_processorrpipecat.services.settingsrrrrpipecat.services.stt_latencyrrpipecat.services.stt_servicerrpipecat.transcriptions.languagerrpipecat.utils.timer(pipecat.utils.tracing.service_decoratorsr websockets.asyncio.clientr!rwebsockets.protocolr"ModuleNotFoundErrorrcrHrIrCrrrrrrrsrurrrrsK (00   >__ZQF/?,F)rIhrI8C=rIjS$ YKY Y _K_ _$FC.FCRss8m #6m Q ,FLL;qc"#FLLd &qc* ++ ,s C**D&/2D!!D&