qiGdZddlZddlmZmZddlmZddlmZm Z ddl Z ddl m Z ddlmZmZddlmZmZmZdd lmZmZmZmZdd lmZdd lmZmZdd lm Z dd l!m"Z"er ddl#m$Z$ ddl)Z)GddeZ*GddeZ+dede e,fdZ-eGddeZ.eGddeZ/GddeZ0Gd d!e0Z1y#e%$r7Z&e jNde&e jNde(de&dZ&[&wwxYw#e%$r7Z&e jNde&e jNde(de&dZ&[&wwxYw)"zWhisper speech-to-text services with locally-downloaded models. This module implements Whisper transcription using locally-downloaded models, supporting both Faster Whisper and MLX Whisper backends for efficient inference. N) dataclassfield)Enum)AsyncGeneratorOptional)logger) TYPE_CHECKINGoverride) ErrorFrameFrameTranscriptionFrame) NOT_GIVEN STTSettings _NotGiven_warn_deprecated_param)SegmentedSTTService)Languageresolve_language)time_now_iso8601) traced_stt WhisperModel Exception: GIn order to use Whisper, you need to `pip install pipecat-ai[whisper]`.zMissing module: zKIn order to use Whisper, you need to `pip install pipecat-ai[mlx-whisper]`.c0eZdZdZdZdZdZdZdZdZ dZ d Z y ) ModelaWhisper model selection options for Faster Whisper. Provides various model sizes and specializations for speech recognition, balancing quality and performance based on use case requirements. Parameters: TINY: Smallest multilingual model, fastest inference. BASE: Basic multilingual model, good speed/quality balance. SMALL: Small multilingual model, better speed/quality balance than BASE. MEDIUM: Medium-sized multilingual model, better quality. LARGE: Best quality multilingual model, slower inference. LARGE_V3_TURBO: Fast multilingual model, slightly lower quality than LARGE. DISTIL_LARGE_V2: Fast multilingual distilled model. DISTIL_MEDIUM_EN: Fast English-only distilled model. tinybasesmallmediumzlarge-v3z)deepdml/faster-whisper-large-v3-turbo-ct2z&Systran/faster-distil-whisper-large-v2z'Systran/faster-distil-whisper-medium.enN) __name__ __module__ __qualname____doc__TINYBASESMALLMEDIUMLARGELARGE_V3_TURBODISTIL_LARGE_V2DISTIL_MEDIUM_ENN/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/services/whisper/stt.pyrr-s5" D D E F E@N>OAr.rc(eZdZdZdZdZdZdZdZdZ y) MLXModelasMLX Whisper model selection options for Apple Silicon. Provides various model sizes optimized for Apple Silicon hardware, including quantized variants for improved performance. Parameters: TINY: Smallest multilingual model for MLX. MEDIUM: Medium-sized multilingual model for MLX. LARGE_V3: Best quality multilingual model for MLX. LARGE_V3_TURBO: Finetuned, pruned Whisper large-v3, much faster with slightly lower quality. DISTIL_LARGE_V3: Fast multilingual distilled model for MLX. LARGE_V3_TURBO_Q4: LARGE_V3_TURBO quantized to Q4 for reduced memory usage. zmlx-community/whisper-tinyz mlx-community/whisper-medium-mlxz"mlx-community/whisper-large-v3-mlxz$mlx-community/whisper-large-v3-turboz%mlx-community/distil-whisper-large-v3z'mlx-community/whisper-large-v3-turbo-q4N) r!r"r#r$r%r(LARGE_V3r*DISTIL_LARGE_V3LARGE_V3_TURBO_Q4r-r.r/r1r1Ks(  (D /F3H;N=OAr.r1languagereturncitjdtjdtjdtjdtj dtj dtjdtjdtjd tjd tjd tjd tjd tjdtjdtj dtj"dtj$dtj&dtj(dtj*dtj,dtj.dtj0dtj2dtj4dtj6dtj8dtj:dtj<di }t?||d S)!a@Maps pipecat Language enum to Whisper language codes. Args: language: A Language enum value representing the input language. Returns: str or None: The corresponding Whisper language code, or None if not supported. Note: Only includes languages officially supported by Whisper. arbncsdadeelenesfafifrhihuiditjakonlplptrorusksvthtrukurvizhT) use_base_code) rARBNCSDADEELENESFAFIFRHIHUIDITJAKONLPLPTRORUSKSVTHTRUKURVIZHr)r5 LANGUAGE_MAPs r/language_to_whisper_languagervcs= T=  T =  T =  T=  T=  T=  T=  T!=$  T%=(  T)=,  T-=0  T1=4  T5=8  T9=<  T==@  TA=D  TE=H  T T T T T T T T T T T T Ty=L~ Hl$ GGr.c6eZdZUdZedZeezed<y)WhisperSTTSettingszSettings for WhisperSTTService. Parameters: no_speech_prob: Probability threshold for filtering non-speech segments. ctSNrr-r.r/zWhisperSTTSettings.ir.default_factoryno_speech_probN) r!r"r#r$rrfloatr__annotations__r-r.r/rxrxs ).>O(PNEI%Pr.rxc~eZdZUdZedZeezed<edZ eezed<edZ e ezed<y ) WhisperMLXSTTSettingszSettings for WhisperMLXSTTService. Parameters: no_speech_prob: Probability threshold for filtering non-speech segments. temperature: Sampling temperature (0.0-1.0). engine: Whisper engine identifier. ctSrzr{r-r.r/r|zWhisperMLXSTTSettings.r}r.r~rctSrzr{r-r.r/r|zWhisperMLXSTTSettings.s9r. temperaturectSrzr{r-r.r/r|zWhisperMLXSTTSettings.sIr.engineN) r!r"r#r$rrrrrrrstrr-r.r/rrsG).>O(PNEI%P%*;L%MK"M#4EFFC)OFr.rceZdZUdZeZeed<ddddddddeee zded ed ee d ee d eef fd Z de fdZd e deefdZdZe ddede d ee fdZdedeedffdZxZS)WhisperSTTServicezClass to transcribe audio with a locally-downloaded Whisper model. This service uses Faster Whisper to perform speech-to-text transcription on audio segments. It supports multiple languages and various model sizes. _settingsNautodefault)modeldevice compute_typerr5settingsrrrrr5rc ttjjtj d}|4t dtdt|tr|n |j|_ |t dtd||_ |t dtd||_ ||j|t |8dd|i|||_||_d|_|j%y) aInitialize the Whisper STT service. Args: model: The Whisper model to use for transcription. Can be a Model enum or string. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTSettings(model=...)`` instead. device: The device to run inference on ('cpu', 'cuda', or 'auto'). Defaults to ``"auto"``. compute_type: The compute type for inference ('default', 'int8', 'int8_float16', etc.). Defaults to ``"default"``. no_speech_prob: Probability threshold for filtering out non-speech segments. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTSettings(no_speech_prob=...)`` instead. language: The default language for transcription. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTSettings(language=...)`` instead. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. **kwargs: Additional arguments passed to SegmentedSTTService. g?)rr5rNrrr5rr-)rxrr,valuerr]r isinstancerrrr5 apply_updatesuper__init___device _compute_type_model_load) selfrrrrr5rkwargsdefault_settings __class__s r/rzWhisperSTTService.__init__sL.((..[[    "7,> H.8.DU%++  "  % "#35GIY Z.<  +   ":/A: N(0  %    ) )( 3  %   ).2  r.r6cy)zIndicates whether this service can generate metrics. Returns: bool: True, as this service supports metric generation. Tr-rs r/can_generate_metricsz&WhisperSTTService.can_generate_metricss r.ct|S)zConvert from pipecat Language to Whisper language code. Args: language: The Language enum value to convert. Returns: str or None: The corresponding Whisper language code, or None if not supported. )rv)rr5s r/language_to_service_languagez.WhisperSTTService.language_to_service_language's,H55r.cj ddlm}tjd||jj |j |j|_tjdy#t$r>}tjd|tjdd|_Yd}~yd}~wwxYw) zLoads the Whisper model. Note: If this is the first time this model is being run, it will take time to download from the Hugging Face model hub. rrzLoading Whisper model...)rrzLoaded Whisper modelrrN) faster_whisperrrdebugrrrrrModuleNotFoundErrorerror)rres r/rzWhisperSTTService._load2s  3 LL3 4&$$T\\HZHZDK LL/ 0"  LL;qc* + LLb cDKK sA(A++ B244B--B2 transcriptis_finalc Kywz+Handle a transcription result with tracing.Nr-rrrr5s r/_handle_transcriptionz'WhisperSTTService._handle_transcriptionF audiocVK|jstdy|jd{tj|tj j tjdz }tj|jj||jjd{\}}d}|D]8}|j|jjks'||jdz }:|jd{|r|j!|d|jjd{t#j$d |d t'||j(t+|jjyy7o777]w) aTranscribe audio data using Whisper. Args: audio: Raw audio bytes in 16-bit PCM format. Yields: Frame: Either a TranscriptionFrame containing the transcribed text or an ErrorFrame if transcription fails. Note: The audio is expected to be 16-bit signed PCM data. The service will normalize it to float32 in the range [-1, 1]. zWhisper model not availableNdtype@)r5 TTranscription: [])rr start_processing_metricsnp frombufferint16astypefloat32asyncio to_thread transcriberr5rtextstop_processing_metricsrrrr _user_idr)rr audio_floatsegments_rsegments r/run_sttzWhisperSTTService.run_sttMsh{{:; ; ++---mmE:AA"**MPWW #-- KK " "K$..:Q:Q  ! +G%%(E(EE7<<.** +**,,, ,,T49P9PQ Q Q LL+D63 4$  "''    .   - RsM/F)F B F)>F#?1F)1'F)F%0F) F' AF)#F)%F)'F)rz)r!r"r#r$rxSettingsrrrrrrrboolrrrrrbytesrr r __classcell__)rs@r/rrs "H!! (,%*.'+15He $H H  H ! H8$H-.HTd 6X 6(3- 6(NR  )- 9A(9K  )5)^E4K-H)r.rc eZdZUdZeZeed<dddddddeee zdee dee dee d eef d Z e d Ze dd ed edee fdZe dedeedffdZy)WhisperSTTServiceMLXzSubclass of `WhisperSTTService` with MLX Whisper model support. This service uses MLX Whisper to perform speech-to-text transcription on audio segments. It's optimized for Apple Silicon and supports multiple languages and quantizations. rN)rrr5rrrrr5rrc ttjjtj ddd}|4t dtdt|tr|n |j|_ |t dtd||_ |t dtd||_ |t d td ||_ ||j|tj|fd |i|y) ahInitialize the MLX Whisper STT service. Args: model: The MLX Whisper model to use for transcription. Can be an MLXModel enum or string. .. deprecated:: 0.0.105 Use ``settings=WhisperMLXSTTSettings(model=...)`` instead. no_speech_prob: Probability threshold for filtering out non-speech segments. .. deprecated:: 0.0.105 Use ``settings=WhisperMLXSTTSettings(no_speech_prob=...)`` instead. language: The default language for transcription. .. deprecated:: 0.0.105 Use ``settings=WhisperMLXSTTSettings(language=...)`` instead. temperature: Temperature for sampling. Can be a float or tuple of floats. .. deprecated:: 0.0.105 Use ``settings=WhisperMLXSTTSettings(temperature=...)`` instead. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. **kwargs: Additional arguments passed to SegmentedSTTService. g333333?mlx)rr5rrrNrrr5rr)rr1r%rrr]rrrrrr5rrrr)rrrr5rrrrs r/rzWhisperSTTServiceMLX.__init__sL1--%%[[     "7,A7 K.8.DU%++  "  % "#35JL\ ].<  +   ":/Dj Q(0  %  " "=2G W+6  (    ) )( 3 $$  %  r.cy)z7MLX Whisper loads models on demand, so this is a no-op.Nr-rs r/rzWhisperSTTServiceMLX._loads r.rrc Kywrr-rs r/rz*WhisperSTTServiceMLX._handle_transcriptionrrrr6cVK ddl}|jd{tj|tjj tj dz }tj|j||jj|jj|jjd{}d}|jdgD]Z}|jddd k(r|jd d |jjksC||jd dd z }\t!|j#dk(rd}|j%d{|r|j'|d|jjd{t)j*d|dt-||j.t1|jjyy77277^#t2$r}t5d|Yd}~yd}~wwxYww)aTranscribe audio data using MLX Whisper. The audio is expected to be 16-bit signed PCM data. MLX Whisper will handle the conversion internally. Args: audio: Raw audio bytes in 16-bit PCM format. Yields: Frame: Either a TranscriptionFrame containing the transcribed text or an ErrorFrame if transcription fails. rNrr)path_or_hf_reporr5rrcompression_ratiogrq?rrrrTrrzUnknown error occurred: )r) mlx_whisperrrrrrrrrrrrrr5getrlenstriprrrrr rr Exceptionr )rrrrchunkrrrs r/rzWhisperSTTServiceMLX.run_stts( C //1 1 1--RXX>EEbjjQT[[K!++&& $ 4 4 NN6600 ED 99Z4 :;;2D9=OO;;/58U8UUw{{6267q99D  :4::< A%..0 0 000tT^^=T=TUUU /vQ78(MM$&NN++ 5 2 & 1V C%=aS#AB B B Cs~H)HG7B,HG: AH$A H/G=00H G?!AH5H)7H:H=H?H H& H!H)!H&&H)rz)r!r"r#r$rrrrrr1rrrr rrrrrrr rr-r.r/rrys %H$$ +/*.'+'+48G h'G ! G 8$ G e_ G 01G V  NR  )- 9A(9K  5C55C^E4K-H5C5Cr.r)2r$r dataclassesrrenumrtypingrrnumpyrlogururtyping_extensionsr r pipecat.frames.framesr r r pipecat.services.settingsrrrrpipecat.services.stt_servicerpipecat.transcriptions.languagerrpipecat.utils.timer(pipecat.utils.tracing.service_decoratorsrrrrrrrrrr1rrvrxrrrr-r.r/rsu (+5GG__<F/?0/ 0ADA<BtB0KH8KH KH\ QQ Q  GK G  Gk+k\WC,WCs 0 {1#&' ^_*1#.//0 0 {1#&' bc*1#.//0s0"C)DD 2C<<DE 2D;;E