qimdZddlZddlZddlmZmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZmZmZmZmZmZmZmZmZddlmZdd lmZmZm Z m!Z!dd l"m#Z#m$Z$dd l%m&Z&dd l'm(Z(m)Z)m*Z*dd l+m,Z,ddl-m.Z.ddl/m0Z0 ddl1m2Z3ddl4m5Z5eGdde)Z7Gdde(Z8dZ9eGddeZ:Gdde&Z;y#e6$rdZ3dZ5YDwxYw)a$OpenAI Speech-to-Text service implementations. Provides two STT services: - ``OpenAISTTService``: REST-based transcription using the Audio API (Whisper / GPT-4o). - ``OpenAIRealtimeSTTService``: WebSocket-based streaming transcription using the Realtime API in transcription-only mode. N) dataclassfield)AnyAsyncGeneratorLiteralOptionalUnion)logger)create_stream_resampler) CancelFrameEndFrameFrameInterimTranscriptionFrame StartFrameTranscriptionFrameUserStartedSpeakingFrameUserStoppedSpeakingFrameVADUserStartedSpeakingFrameVADUserStoppedSpeakingFrame)FrameDirection) NOT_GIVEN STTSettings _NotGiven_warn_deprecated_param)OPENAI_REALTIME_TTFS_P99OPENAI_TTFS_P99)WebsocketSTTService)BaseWhisperSTTServiceBaseWhisperSTTSettings Transcription)Language)time_now_iso8601) traced_stt)connect)StateceZdZdZy)OpenAISTTSettingsz$Settings for the OpenAI STT service.N)__name__ __module__ __qualname____doc__M/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/services/openai/stt.pyr'r':s.r-r'ceZdZUdZeZeed<dddejddde dde e de e de e de ed e e d e e d e ed e e ffd Z dedefdZxZS)OpenAISTTServicezOpenAI Speech-to-Text service that generates text from audio. Uses OpenAI's transcription API to convert audio to text. Requires an OpenAI API key set via the api_key parameter or OPENAI_API_KEY environment variable. _settingsN)modelapi_keybase_urllanguageprompt temperaturesettingsttfs_p99_latencyr2r3r4r5r6r7r8r9c T|xstj} td|j| dd} |t dtd|| _|t dtd|| _|t dtd|| _|| j|t |(d||| |d| y) uWInitialize OpenAI STT service. Args: model: Model to use — either gpt-4o or Whisper. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTSettings(model=...)`` instead. api_key: OpenAI API key. Defaults to None. base_url: API base URL. Defaults to None. language: Language of the audio input. Defaults to English. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTSettings(language=...)`` instead. prompt: Optional text to guide the model's style or continue a previous segment. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTSettings(prompt=...)`` instead. temperature: Optional sampling temperature between 0 and 1. Defaults to 0.0. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTSettings(temperature=...)`` instead. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. ttfs_p99_latency: P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark **kwargs: Additional arguments passed to BaseWhisperSTTService. gpt-4o-transcribeN)r2r5r6r7r2r6r7)r3r4r8r9r,) r!ENr'language_to_service_languagerr2r6r7 apply_updatesuper__init__) selfr2r3r4r5r6r7r8r9kwargs _languagedefault_settings __class__s r.r@zOpenAISTTService.__init__KsZ+ ,%66yA     "7,=w G%*  "   "8-> I&,  #  " "=2C] S+6  (    ) )( 3  %-    r-audioreturnc6K|jjJd|df|jj|jjd}|jr)|jjdvr d|d<dg|d<nd |d<|jj|jj|d <|jj |jj |d <|j jjjd i|d{S7w) Nz audio.wavz audio/wav)filer2r5)r;zgpt-4o-mini-transcribejsonresponse_formatlogprobsinclude verbose_jsonr6r7r,) r1r5r2_include_prob_metricsr6r7_clientrFtranscriptionscreate)rArFrBs r. _transcribezOpenAISTTService._transcribes~~&&222!%5^^))//   % %~~##'VV,2()%/Ly!-;() >> ,#~~44F8  >> % % 1$(NN$>$>F= !=T\\''66==GGGGGsDDDD)r(r)r*r+r'Settings__annotations__r!r<rrstrfloatr@bytesr rS __classcell__rEs@r.r0r0As !H  $!%"&'/{{ $'+04,;L }L # L 3- L 8$ L  L e_L ,-L #5/L \HuHHr-r0i]c<eZdZUdZedZedzezed<y)OpenAIRealtimeSTTSettingszSettings for OpenAIRealtimeSTTService. Parameters: prompt: Optional prompt text to guide transcription style. ctSN)rr,r-r.z"OpenAIRealtimeSTTSettings.s9r-)default_factoryNr6) r(r)r*r+rr6rVrrUr,r-r.r\r\s" &+;L%MFC$J "Mr-r\c@eZdZUdZeZeed<ddejddddde d de d e e d e d e ed e e d e e e edfde eddede ede effdZed ede fdZdefdZdede e efffd Zdeffd Zdeffd Zdeffd Zdedee dffdZ!de de"ffd Z#fd!Z$fd"Z%d#Z&d$Z'd%e fd&Z(d'Z)defd(Z*d)Z+d*Z,d+Z-d,e fd-Z.d,e fd.Z/d,e fd/Z0d,e fd0Z1e2 d8d1e d2ed e efd3Z3d,e fd4Z4d,e fd5Z5d,e fd6Z6d,e fd7Z7xZ8S)9OpenAIRealtimeSTTServiceuhOpenAI Realtime Speech-to-Text service using WebSocket transcription sessions. Uses OpenAI's Realtime API in transcription-only mode for real-time streaming speech recognition with optional server-side VAD and noise reduction. The model does not generate conversational responses — only transcription output. This service supports two VAD modes: **Local VAD** (default): Disable server-side VAD and use a local VAD processor in the pipeline instead. When a ``VADUserStoppedSpeakingFrame`` is received, the service commits the audio buffer so that the server begins transcription for the completed speech segment. **Server-side VAD** (``turn_detection=None``): The OpenAI server performs voice-activity detection. The service broadcasts ``UserStartedSpeakingFrame`` and ``UserStoppedSpeakingFrame`` when the server detects speech boundaries. Do **not** use a separate VAD processor in the pipeline in this mode. Audio is sent as 24 kHz 16-bit mono PCM as required by the OpenAI Realtime API. If the pipeline runs at a different sample rate (e.g. 16 kHz for Silero VAD compatibility), audio is automatically upsampled before sending. Example:: stt = OpenAIRealtimeSTTService( api_key="sk-...", model="gpt-4o-transcribe", noise_reduction="near_field", ) r1Nz wss://api.openai.com/v1/realtimeFT) r2r4r5r6turn_detectionnoise_reductionshould_interruptr8r9r3r2r4r5r6rcrd) near_field far_fieldrer8r9c  t tdtdtjd} |t dtd|| _|+|tjk7rt dtd|| _|t dtd|| _| | j| t |,d | | d| ||_ ||_ ||_||_||_d|_d |_t'|_|d u|_y) aKInitialize the OpenAI Realtime STT service. Args: api_key: OpenAI API key for authentication. model: Transcription model. Supported values are ``"gpt-4o-transcribe"`` and ``"gpt-4o-mini-transcribe"``. .. deprecated:: 0.0.105 Use ``settings=OpenAIRealtimeSTTSettings(model=...)`` instead. base_url: WebSocket base URL for the Realtime API. Defaults to ``"wss://api.openai.com/v1/realtime"``. language: Language of the audio input. Defaults to English. .. deprecated:: 0.0.105 Use ``settings=OpenAIRealtimeSTTSettings(language=...)`` instead. prompt: Optional prompt text to guide transcription style or provide keyword hints. .. deprecated:: 0.0.105 Use ``settings=OpenAIRealtimeSTTSettings(prompt=...)`` instead. turn_detection: Server-side VAD configuration. Defaults to ``False`` (disabled), which relies on a local VAD processor in the pipeline. Pass ``None`` to use server defaults (``server_vad``), or a dict with custom settings (e.g. ``{"type": "server_vad", "threshold": 0.5}``). noise_reduction: Noise reduction mode. ``"near_field"`` for close microphones, ``"far_field"`` for distant microphones, or ``None`` to disable. should_interrupt: Whether to interrupt bot output when speech is detected by server-side VAD. Only applies when turn detection is enabled. Defaults to True. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. ttfs_p99_latency: P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark **kwargs: Additional arguments passed to parent WebsocketSTTService. Nzdwebsockets is required for OpenAIRealtimeSTTService. Install it with: pip install pipecat-ai[openai]r;)r2r5r6r2r5r6)r9r8Fr,)websocket_connect ImportErrorr\r!r<rr2r5r6r>r?r@_api_key _base_url_turn_detection_noise_reduction_should_interrupt _receive_task_session_readyr _resampler_server_vad_enabled)rAr3r2r4r5r6rcrdrer8r9rBrDrEs r.r@z!OpenAIRealtimeSTTService.__init__s#p  $B  5%[[    "7,Ew O%*  "  H $; ":/H* U(0  %   "8-F Q&,  #    ) )( 3  -%    !- /!1!#13$2#> r-rGcZ|jjddjS)zConvert a Language enum value to an ISO-639-1 code. Args: language: The Language enum value. Returns: Two-letter ISO-639-1 language code. -r)valuesplitlower)r5s r._language_to_codez*OpenAIRealtimeSTTService._language_to_codeRs'~~##C(+1133r-cy)zCheck if the service can generate processing metrics. Returns: True, as this service supports metrics generation. Tr,rAs r.can_generate_metricsz-OpenAIRealtimeSTTService.can_generate_metrics_s r-deltacKt||d{}|r$|jr|jd{|S7,7w)aLApply a settings delta and send session update if needed. Sends a ``session.update`` to the server when the session is active. Args: delta: A :class:`STTSettings` (or ``OpenAIRealtimeSTTSettings``) delta. Returns: Dict mapping changed field names to their previous values. N)r?_update_settingsrq_send_session_update)rAr}changedrEs r.rz)OpenAIRealtimeSTTService._update_settingsgsH077 t**++- - - 8 .sAA%AAAAframectKt||d{|jd{y77w)zStart the service and establish WebSocket connection. Args: frame: The start frame triggering service initialization. N)r?start_connectrArrEs r.rzOpenAIRealtimeSTTService.startys3 gmE"""mmo #848688ctKt||d{|jd{y77w)zStop the service and close WebSocket connection. Args: frame: The end frame triggering service shutdown. N)r?stop _disconnectrs r.rzOpenAIRealtimeSTTService.stops6 gl5!!!    " rctKt||d{|jd{y77w)zCancel the service and close WebSocket connection. Args: frame: The cancel frame triggering service cancellation. N)r?cancelrrs r.rzOpenAIRealtimeSTTService.cancels6 gnU###    $ rrFcLK|j|d{dy7 w)uSend audio data to the transcription session. Audio is streamed over the WebSocket. Transcription results arrive asynchronously via the receive task and are pushed as ``InterimTranscriptionFrame`` or ``TranscriptionFrame``. Args: audio: Raw audio bytes (16-bit mono PCM at the pipeline sample rate). Automatically resampled to 24 kHz. Yields: None — results are delivered via the WebSocket receive task. N) _send_audio)rArFs r.run_sttz OpenAIRealtimeSTTService.run_stts&u%%%  &s $" $ directioncKt|||d{|jsSt|tr|j d{yt|t r|jd{yyy7d727 w)aProcess frames from the pipeline. Extends the base STT service to handle local VAD events when server-side VAD is disabled. On ``VADUserStoppedSpeakingFrame``, commits the audio buffer so the server begins transcription for the completed speech segment. Args: frame: The frame to process. direction: The direction of frame flow in the pipeline. N)r? process_framers isinstancerstart_processing_metricsr_commit_audio_buffer)rArrrEs r.rz&OpenAIRealtimeSTTService.process_frames}g#E9555''%!<=33555E#>?//111@( 6 61s3BA<3B A> (B4B5B>BBcKt|d{|jd{|jr=|js0|j |j |j|_yyy7f7Pw)z:Connect to the transcription endpoint and start receiving.N)r?r_connect_websocket _websocketrp create_task_receive_task_handler _report_errorrArEs r.rz!OpenAIRealtimeSTTService._connectsmg   %%''' ??4#5#5!%!1!1$2L2LTM_M_2`!aD $6? !'s BA<BA>AB>BcKt|d{|jr,|j|jdd{d|_|j d{y7U7&7 w)z)Disconnect and clean up background tasks.Ng?)timeout)r?rrp cancel_task_disconnect_websocketrs r.rz$OpenAIRealtimeSTTService._disconnectsgg!###   ""4#5#5s"C C C!%D ((*** $ C*s3A1A+0A1A-A1%A/&A1-A1/A1cK |jr'|jjtjuryd|_|j d}t |dd|jid{|_|jdd{y7#7#t$r0}|jd|| d{7d|_Yd}~yd}~wwxYww) zAEstablish the WebSocket connection to the transcription endpoint.NFz?intent=transcription AuthorizationzBearer )uriadditional_headers on_connectedz)Error connecting to OpenAI Realtime STT:  error_msg exception) rstater%OPENrqrlrirk_call_event_handler Exception push_error)rAurles r.rz+OpenAIRealtimeSTTService._connect_websockets #4??#8#8EJJ#F"'D ^^$$9:C$5#wt}}o%>$%DO **>: : :  ; #//EaSI"  #DOO  #shC2BC5B,B-B B BCBB C C6B97 CCC  CcvK d|_|jr"|jjd{d|_|j dd{y7%#t$r)}|j d||d{7Yd}~Pd}~wwxYw7<#d|_|j dd{7wxYww)zClose the WebSocket connection.FNzError disconnecting: ron_disconnected)rqrcloserrr)rArs r.rz.OpenAIRealtimeSTTService._disconnect_websockets >"'D oo++---#DO**+<= = =. //1!5"    >#DO**+<= = =ssB90AAAB9BB9A B%B >B?B B BBB9B6/B20B66B9messagecVK |jsD|jr7|jjtj|d{yyy7#t $rF}|js |jsYd}~y|j d||d{7Yd}~yd}~wwxYww)z|Send a JSON message over the WebSocket. Args: message: The message dict to serialize and send. NzError sending message: r)_disconnectingrsendrJdumpsrr)rArrs r._ws_sendz!OpenAIRealtimeSTTService._ws_sends &&4??oo**4::g+>???,;&? ""$////3A37"   sYB)A A AAB)A B& B!8B)=B!BB!B)!B&&B)cKd|jji}|jjr%|j|jjnd}|r||d<|jjr|jj|d<dt d|d}|j durd|d <n|j |j |d <|jrd |ji|d <|jd d d|iddd{y7w)z?Send ``session.update`` to configure the transcription session.r2Nr5r6z audio/pcm)typerate)format transcriptionFrcrrdzsession.updaterinputrrF)rsession) r1r2r5ryr6_OPENAI_SAMPLE_RATErmrnr)rAr language_code input_audios r.rz-OpenAIRealtimeSTTService._send_session_updates &(<(<= @D~~?V?VD " "4>>#:#: ;\`  (5M* % >> &*nn&;&;M( #$++     5 (,0K( )  ! ! -,0,@,@K( )  --.K) *mm(+   sDD DD cK|jj||jtd{}|syt j |j d}|jd|dd{y7H7w)zSend audio data via ``input_audio_buffer.append``. Resamples from the pipeline sample rate to 24 kHz if needed. Args: audio: Raw audio bytes at the pipeline sample rate. Nzutf-8zinput_audio_buffer.appendr)rrresample sample_raterbase64 b64encodedecoder)rArFpayloads r.rz$OpenAIRealtimeSTTService._send_audio4sxoo..ud6F6FH[\\ ""5)009mm3     ] s"/A>A:AA>4A<5A><A>cFK|jddid{y7w)z2Commit the current audio buffer for transcription.rzinput_audio_buffer.commitNrr{s r.rz-OpenAIRealtimeSTTService._commit_audio_bufferGsmmV%@ABBB !!cFK|jddid{y7w)zClear the current audio buffer.rzinput_audio_buffer.clearNrr{s r._clear_audio_bufferz,OpenAIRealtimeSTTService._clear_audio_bufferKsmmV%?@AAArcK|j23d{} tj|}|j dd}|dk(r|j|d{P|dk(r|j|d{o|dk(r|j|d{|dk(r|j|d{|d k(r|j|d{|d k(r|j|d{|d k(r|j|d{ |d k(r*t jd |j dd:|dk(r|j|d{Zt jd|t7p#tj$rt j dYwxYw7\7@7$77777d6yw)zReceive and dispatch server events from the transcription session. Called by ``WebsocketService._receive_task_handler`` which wraps this method with automatic reconnection on connection errors. Nz!Failed to parse WebSocket messagerzsession.createdzsession.updatedz1conversation.item.input_audio_transcription.deltaz5conversation.item.input_audio_transcription.completedz2conversation.item.input_audio_transcription.failedz!input_audio_buffer.speech_startedz!input_audio_buffer.speech_stoppedzinput_audio_buffer.committedz Audio buffer committed: item_id=item_iderrorzUnhandled event: )rrJloadsJSONDecodeErrorr warningget_handle_session_created_handle_session_updated_handle_transcription_delta_handle_transcription_completed_handle_transcription_failed_handle_speech_started_handle_speech_stoppedtrace _handle_error)rArevtevt_types r._receive_messagesz*OpenAIRealtimeSTTService._receive_messagesSs "__ = =' jj) wwvr*H,,223777..223777PP66s;;;TT::3???QQ77<<<@@11#666@@11#666;; ? SU@V?WXYW$((--- 0 ;<9 ='' BC 87;?<66.5-s G G FG G F+G F5G 8F89G F;G 6F>7G GG 4G5G GAG "G# G G (F2.G 1F22G 8G ;G >G G G G G  G rcjKtjd|jd{y7w)zHandle ``session.created``. Sent immediately after connecting. We respond by configuring the session with our desired settings. Args: evt: The session created event from the server. z4Transcription session created, sending configurationN)r debugrrArs r.rz0OpenAIRealtimeSTTService._handle_session_createdws'  KL'')))s )313cDKtjdd|_yw)zHandle ``session.updated``. The session is now fully configured and ready to transcribe. Args: evt: The session updated event from the server. z*Transcription session configured and readyTN)r rrqrs r.rz0OpenAIRealtimeSTTService._handle_session_updateds  AB"s cK|jdd}|r9|jt||jt |d{yy7w)a+Handle incremental transcription text. For ``gpt-4o-transcribe`` and ``gpt-4o-mini-transcribe``, deltas contain streaming partial text. For ``whisper-1``, each delta contains the full turn transcript. Args: evt: The delta event from the server. r}rresultN)r push_framer_user_idr")rArr}s r.rz4OpenAIRealtimeSTTService._handle_transcription_deltasV$ //)MM$&     sAA A AcK|jdd}|rk|jt||jt |d{|j |dd{|j d{yy787 7 w)zHandle a completed transcription for a speech segment. Pushes a ``TranscriptionFrame`` and records the result for tracing. Args: evt: The completed event containing the full transcript. transcriptrrNT)rrrrr"_handle_transcription_tracestop_processing_metrics)rArrs r.rz8OpenAIRealtimeSTTService._handle_transcription_completedsWW\2. //"MM$&   22:tD D D..0 0 0   E 0s6AB  B B $B%B <B=B B B ris_finalc Kyw)zRecord transcription result for tracing. Args: transcript: The transcribed text. is_final: Whether this is a final transcription result. language: Optional language of the transcription. Nr,)rArrr5s r.rz4OpenAIRealtimeSTTService._handle_transcription_traces  scKtjd|jtd{|jr|j d{|j d{y7A77 w)zHandle server-side VAD speech start. Broadcasts ``UserStartedSpeakingFrame`` and optionally triggers interruption of current bot output. Args: evt: The ``input_audio_buffer.speech_started`` event. zServer VAD: speech startedN)r rbroadcast_framerrobroadcast_interruptionrrs r.rz/OpenAIRealtimeSTTService._handle_speech_startedsd  12""#;<<<  ! !--/ / /++--- = /-s3.A8A2#A8A4A8,A6-A84A86A8ctKtjd|jtd{y7w)aHandle server-side VAD speech stop. Broadcasts ``UserStoppedSpeakingFrame``. The audio buffer is automatically committed by the server when VAD is enabled. Args: evt: The ``input_audio_buffer.speech_stopped`` event. zServer VAD: speech stoppedN)r rrrrs r.rz/OpenAIRealtimeSTTService._handle_speech_stoppeds*  12""#;<<22b+#, >d*+ Z u &CB"=H * * # #T*11, (,      8$     . . = = Sd Str-rb)rs (@@7   >__R< 5/?F)   .   qH,qHh N N N|2|] Es C CC