qi4ddlZddlZddlmZmZddlmZddlm Z m Z m Z ddl m Z ddlmZmZerddlZej$eZGdd e d ZGd d e Zd gZy)N) AudioInputmake_list_of_audio) BatchFeature)ProcessingKwargsProcessorMixinUnpack) TextInput)is_torch_availableloggingc,eZdZddidddddddd d Zy ) AudioFlamingo3ProcessorKwargspaddingTi>g>@ max_length) sampling_rate chunk_lengthreturn_attention_maskrptleft)return_tensors padding_side) text_kwargs audio_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsn/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/audioflamingo3/processing_audioflamingo3.pyrr"s5 t # %)#  #" Ir rF)totalc eZdZdZ dfd ZddZ ddeeezdedzd e dzd e e de f d Z edeefd Z ddeeezezd eeezdzd e e de fdZdddZdedefdZxZS)AudioFlamingo3Processora: Constructs an AudioFlamingo3 processor which wraps an AudioFlamingo3 feature extractor and an AudioFlamingo3 tokenizer into a single processor. [`AudioFlamingo3Processor`] offers all the functionalities of [`WhisperFeatureExtractor`] and [`Qwen2TokenizerFast`]. See the [`~AudioFlamingo3Processor.__call__`] for more information. Args: feature_extractor ([`WhisperFeatureExtractor`]): The feature extractor is a required input. tokenizer ([`Qwen2TokenizerFast`]): The tokenizer is a required input. chat_template (`Optional[str]`, *optional*): The Jinja template to use for formatting the conversation. If not provided, the tokenizer's default chat template will be used. audio_token (`Optional[str]`, *optional*, defaults to `""`): Special token used to represent audio inputs in the chat template. default_transcription_prompt (`str`, *optional*, defaults to `"Transcribe the input speech."`): Default prompt to use for transcription tasks when applying transcription requests. max_audio_len (`int`, *optional*, defaults to 600): Maximum length of audio sequences in seconds. Audio longer than this will be truncated. Nc||_|j||_||_||_t ||||y)N) chat_template) audio_tokenconvert_tokens_to_idsaudio_token_iddefault_transcription_prompt max_audio_lensuper__init__)selffeature_extractor tokenizerr&r'r*r+ __class__s r!r-z AudioFlamingo3Processor.__init__LsE''==kJ,H)* *I]Sr returnc2|dz dzdz}|dz dzdz}|S)Nr)r. audio_lengthsconv_output_lengthsaudio_tokens_lengthss r!_get_audio_token_lengthz/AudioFlamingo3Processor._get_audio_token_length[s2,q0Q6: 3a 7A=A##r Ftextaudio output_labelskwargsc |jtfd|jji|}|d}|d}|j d}|dk7r"t |j jdt|tr|g}n3t|ttfrtd|Ds t di} |3t|}t|t|k7r$t d t|d t|d t|d |d z} t|j |d z} g} g} |D]}t|j"d}t%d|| zdz | z}|| kDr` placeholders in the text based on the post-pool frame counts of the audio windows, then tokenizes the provided strings as-is, and extracts log-mel features with [`WhisperFeatureExtractor`]. If `audio` is `None`, no audio processing is performed and the text is tokenized as-is (LM-only behavior). Args: text (`str` or `list[str]`): Input sequence or batch of sequences. audio (`np.ndarray` or `list[np.ndarray]`): Input audio or batch of audios as NumPy arrays. If provided, there must be as many `text` inputs as `audio` inputs. output_labels (bool, *optional*, default=False): Whether to return labels for training. Returns: [`BatchFeature`]: A dictionary with tokenized text (`input_ids`, `attention_mask`) and audio features (`input_features`, `input_features_mask`). tokenizer_init_kwargsrrrrz% only supports `return_tensors='pt'`.c3<K|]}t|tywN isinstancestr).0ts r! z3AudioFlamingo3Processor.__call__..s9[QR*Q:L9[zAInvalid input text. Please provide a string, or a list of stringszGot z text but z audios; they must match 1:1.rrrr4zAudio duration (z.1fz s) exceeds zs; truncating to first zs.attention_maskinput_features_mask input_idsilabels)data tensor_type)( _merge_kwargsrr0 init_kwargsget ValueErrorr1rrCrDlisttupleallrlenintr+shapemaxloggerwarningappendminranger/poptorchstacksplitsumr9 enumerateresubescaper'cloner) pad_token_idr)r.r:r;r<r= call_kwargsrrr audio_inputs window_size max_windowsper_sample_windows flat_chunksaudio_el n_samplesn_wintime_capistartend padding_masksr6r8 audio_lengthexpanded text_inputsrNrMs r!__call__z AudioFlamingo3Processor.__call__`s:)d(( ) "&.."<"<  "-0 ">2 $)9: T ! 7 788]^_ _ dC 6DTD%=1c9[VZ9[6[`a a  &u-E4yCJ& 4D {*SZLHe!fggl?;l>>ZZ[Kd00L4PPQK,. ,.K! <q 12 A K 7! ; KL;&NN*9|O7T+TUX*YYdeiewewdxyPQUQcQcPddfg(E"))%0y%+*=>uC&&xc':;< <"2411+NNL'++,<=L2>L. /"KK%++lFVFVWYFZ\n:o(pQ(pqM#'#?#? #N $--A#B #<66"))D,<,<"=t?O?OR^?^`def`gh"Q # %dnnT9[9 .+.. +&,,.F48F6T000 1<@F6T^^888 9#DN>BB%)qs$Mc|jj}|jj}ttj ||zdgzS)NrJ)r0model_input_namesr/rTdictfromkeys)r. tok_names fea_namess r!rz)AudioFlamingo3Processor.model_input_namessDNN44 **<< DMM)i"7;P:Q"QRSSr promptc t|tr|g}nt|ttfr |rt d|Dr t|}nstt |}t rU|Dcgc]J}t|tjr,|jjjn|L}}t|}|dk(r td||jg|z}nt|tr|g|z}nt|ttfr}t||k7rtdt|d|dg}|D]L}||j|j!t|tr|j|Ct!dn t!dt#||D cgc](\} } d d | d t| trd | d nd | dgdg*} } } |j$| fdddd|Scc}wcc} } w)a Prepare inputs for automatic speech recognition without manually writing the default transcription prompt. Args: audio (`str`, `list[str]`, `np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): Audio to transcribe. Strings are interpreted as local paths or URLs and will be loaded automatically by the chat template loader; NumPy arrays and PyTorch tensors are forwarded directly. prompt (`str` or `list[str]`, *optional*): Custom prompt(s) to include in the user turn. A list must be the same length as the batch. When `None`, each sample uses `"Transcribe the input speech."`. **kwargs: Additional keyword arguments forwarded to [`~AudioFlamingo3Processor.apply_chat_template`] (for example `text_kwargs`, `audio_kwargs`, ...). Returns: [`BatchFeature`]: Processor outputs ready to be passed to [`AudioFlamingo3ForConditionalGeneration.generate`]. c3<K|]}t|tywrArB)rEels r!rGzFAudioFlamingo3Processor.apply_transcription_request..s?dXZ 2s@S?drHrz)`audio` must contain at least one sample.z Received z prompt(s) for z$ audio sample(s); counts must match.z'Each prompt must be a string or `None`.z<`prompt` must be a string, a sequence of strings, or `None`.userr:)typer:r;)rpath)rr;)rolecontentT)tokenizeadd_generation_prompt return_dict)rCrDrTrUrVrr raTensordetachcpunumpyrWrSr*r] TypeErrorzipapply_chat_template) r.r;rr= audio_itemsr batch_sizepromptsitem prompt_text audio_item conversationss r!apply_transcription_requestz3AudioFlamingo3Processor.apply_transcription_requests2 eS !38'K e} -%C?d^c?d889JFG  $h+G u .6{j( F }OJv> QXYt>>tDYGYZs?c@|j}dD]1}|j|s|t|dj}n|jdr|ddj}t|dk\r%|d|dk(r|ddvr|ddj}|S) zi Remove the assistant prefix and surrounding quotes from a decoded transcription string. )z"The spoken content of the audio isz!The transcription of the audio isN.rKr5r>"'r4)strip startswithrWendswith)r.r:strippedprefixs r!rz:AudioFlamingo3Processor._strip_assistant_prefix_and_quotes)s ::<  F""6*#CKM288:     S !}**,H x=A (1+""=(1+Q[B["~++-Hr )NzzTranscribe the input speech.iX)r6 torch.Tensorr2r)NFrA)rrr__doc__r-r9r rTrboolr rrr}propertyrDrrrr __classcell__)r1s@r!r$r$4s6%C T$$(%* cC$y/)cCD cCd{ cC 67 cC  cCJT49TT*.O T#Y+O d3i$&O 67 O  O b05 ssr r$)rfrnp audio_utilsrrfeature_extraction_utilsrprocessing_utilsrrr tokenization_utils_baser utilsr r ra get_loggerrr[rr$__all__rr r!rsf  94HH00   H %$4E$JnJZ % %r