qin8ddlZddlZddlmZmZddlmZddlm Z m Z m Z ddl m Z ddlmZmZerddlZej$eZGdd e d ZGd d e Zd gZy)N) AudioInputmake_list_of_audio) BatchFeature)ProcessingKwargsProcessorMixinUnpack) TextInput)is_torch_availableloggingc,eZdZddidddddddd d Zy ) GlmAsrProcessorKwargspaddingTi>g>@ max_length) sampling_rate chunk_lengthreturn_attention_maskrptleft)return_tensors padding_side) text_kwargs audio_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaults^/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/glmasr/processing_glmasr.pyrr(s5 t # %)#  #" Ir rF)totalc eZdZdZ dfd ZddZ ddeeezdedzd e dzd e e de f d Z edeefd Z ddeeezezd eeezdzd e e de fdZdddZdedefdZxZS)GlmAsrProcessora Constructs an GlmAsr processor which wraps an GlmAsr feature extractor and an GlmAsr tokenizer into a single processor. [`GlmAsrProcessor`] offers all the functionalities of [`WhisperFeatureExtractor`] and [`Qwen2TokenizerFast`]. See the [`~GlmAsrProcessor.__call__`] for more information. Args: feature_extractor ([`WhisperFeatureExtractor`]): The feature extractor is a required input. tokenizer ([`Qwen2TokenizerFast`]): The tokenizer is a required input. chat_template (`Optional[str]`, *optional*): The Jinja template to use for formatting the conversation. If not provided, the tokenizer's default chat template will be used. audio_token (`Optional[str]`, *optional*, defaults to `"<|pad|>`"): Special token used to represent audio inputs in the chat template. default_transcription_prompt (`str`, *optional*, defaults to `"Please transcribe this audio into text"`): Default prompt to use for transcription tasks when applying transcription requests. max_audio_len (`int`, *optional*, defaults to 655): Maximum length of audio sequences in seconds. Audio longer than this will be truncated. 655 gives approximately 8192 tokens, corresponding to the maximum sequence length of the text model. Nc||_|j||_||_||_t ||||y)N) chat_template) audio_tokenconvert_tokens_to_idsaudio_token_iddefault_transcription_prompt max_audio_lensuper__init__)selffeature_extractor tokenizerr&r'r*r+ __class__s r!r-zGlmAsrProcessor.__init__SsE''==kJ,H)* *I]Sr returncdd}dD]\}}}|d|zz|dz z dz |zdz}||z |zdz}|S)N))rr5)r5rr6r5r)r. audio_lengths merge_factorr kernel_sizestride num_tokenss r!_get_audio_token_lengthz'GlmAsrProcessor._get_audio_token_lengthbsc ,B ` (G[&*Q[8K!OLqPU[[^__M `$l2|CaG r Ftextaudio output_labelskwargsc |jtfd|jji|}|d}|d}|j d}|dk7r"t |j jdt|tr|g}n3t|ttfrtd|Ds t di} |3t|}t|t|k7r$t d t|d t|d t|d |d z} t|j |d z} g} g} |D]}t|j"d}t%d|| zdz | z}|| kDr` placeholders in the text based on the post-pool frame counts of the audio windows, then tokenizes the provided strings as-is, and extracts log-mel features with [`WhisperFeatureExtractor`]. If `audio` is `None`, no audio processing is performed and the text is tokenized as-is (LM-only behavior). Args: text (`str` or `list[str]`): Input sequence or batch of sequences. audio (`np.ndarray` or `list[np.ndarray]`): Input audio or batch of audios as NumPy arrays. If provided, there must be as many `text` inputs as `audio` inputs. output_labels (bool, *optional*, default=False): Whether to return labels for training. Returns: [`BatchFeature`]: A dictionary with tokenized text (`input_ids`, `attention_mask`) and audio features (`input_features`, `input_features_mask`). tokenizer_init_kwargsrrrrz% only supports `return_tensors='pt'`.c3<K|]}t|tywN isinstancestr).0ts r! z+GlmAsrProcessor.__call__..s9[QR*Q:L9[zAInvalid input text. Please provide a string, or a list of stringszGot z text but z audios; they must match 1:1.rrrr5zAudio duration (z.1fz s) exceeds zs; truncating to first zs.attention_maskinput_features_mask input_idsilabels)data tensor_type)( _merge_kwargsrr0 init_kwargsget ValueErrorr1rrFrGlisttupleallrlenintr+shapemaxloggerwarningappendminranger/poptorchstacksplitsumr< enumerateresubescaper'cloner) pad_token_idr)r.r=r>r?r@ call_kwargsrrr audio_inputs window_size max_windowsper_sample_windows flat_chunksaudio_el n_samplesn_wintime_capistartend padding_masksr7audio_tokens_lengths audio_lengthexpanded text_inputsrQrPs r!__call__zGlmAsrProcessor.__call__js:)d(( ! "&.."<"<  "-0 ">2 $)9: T ! 7 788]^_ _ dC 6DTD%=1c9[VZ9[6[`a a  &u-E4yCJ& 4D {*SZLHe!fggl?;l>>ZZ[Kd00L4PPQK,. ,.K! <q 12 A K 7! ; KL;&NN*9|O7T+TUX*YYdeiewewdxyPQUQcQcPddfg(E"))%0y%+*=>uC&&xc':;< <"2411+NNL'++,<=L2>L. /"KK%++lFVFVWYFZ\n:o(pQ(pqM#'#?#? #N $--A#B #<66"))D,<,<"=t?O?OR^?^`def`gh"Q # %dnnT9[9 .+.. +&,,.F48F6T000 1<@F6T^^888 9#DN>BB%)qs$Mc|jj}|jj}ttj ||zdgzS)NrM)r0model_input_namesr/rWdictfromkeys)r. tok_names fea_namess r!rz!GlmAsrProcessor.model_input_namessDNN44 **<< DMM)i"7;P:Q"QRSSr promptc t|tr|g}nt|ttfr |rt d|Dr t|}nstt |}t rU|Dcgc]J}t|tjr,|jjjn|L}}t|}|dk(r td||jg|z}nt|tr|g|z}nt|ttfr}t||k7rtdt|d|dg}|D]L}||j|j!t|tr|j|Ct!dn t!dt#||D cgc](\} } d t| trd | d nd | d d | dgdg*} } } |j$| fdddd|Scc}wcc} } w)a  Prepare inputs for automatic speech recognition without manually writing the default transcription prompt. Args: audio (`str`, `list[str]`, `np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): Audio to transcribe. Strings are interpreted as local paths or URLs and will be loaded automatically by the chat template loader; NumPy arrays and PyTorch tensors are forwarded directly. prompt (`str` or `list[str]`, *optional*): Custom prompt(s) to include in the user turn. A list must be the same length as the batch. When `None`, each sample uses `"Transcribe the input speech."`. **kwargs: Additional keyword arguments forwarded to [`~GlmAsrProcessor.apply_chat_template`] (for example `text_kwargs`, `audio_kwargs`, ...). Returns: [`BatchFeature`]: Processor outputs ready to be passed to [`GlmAsrForConditionalGeneration.generate`]. c3<K|]}t|tywrDrE)rHels r!rJz>GlmAsrProcessor.apply_transcription_request..s?dXZ 2s@S?drKrz)`audio` must contain at least one sample.z Received z prompt(s) for z$ audio sample(s); counts must match.z'Each prompt must be a string or `None`.z<`prompt` must be a string, a sequence of strings, or `None`.userr>)typepath)rr>r=)rr=)rolecontentT)tokenizeadd_generation_prompt return_dict)rFrGrWrXrYrr rdTensordetachcpunumpyrZrVr*r` TypeErrorzipapply_chat_template) r.r>rr@ audio_itemsr batch_sizepromptsitem prompt_text audio_item conversationss r!apply_transcription_requestz+GlmAsrProcessor.apply_transcription_requests2 eS !38'K e} -%C?d^c?d889JFG  $h+G u .6{j( F }OJv> QXYt>>tDYGYZs?c@|j}dD]1}|j|s|t|dj}n|jdr|ddj}t|dk\r%|d|dk(r|ddvr|ddj}|S) zi Remove the assistant prefix and surrounding quotes from a decoded transcription string. )z"The spoken content of the audio isz!The transcription of the audio isN.rNr6r>"'r5)strip startswithrZendswith)r.r=strippedprefixs r!rz2GlmAsrProcessor._strip_assistant_prefix_and_quotes3s ::<  F""6*#CKM288:     S !}**,H x=A (1+""=(1+Q[B["~++-Hr )Nz<|pad|>z&Please transcribe this audio into texti)r7 torch.Tensorr2r)NFrD)rrr__doc__r-r<r rWrboolr rrrpropertyrGrrrr __classcell__)r1s@r!r$r$:s8%M T$(%* cC$y/)cCD cCd{ cC ./ cC  cCJT49TT*.O T#Y+O d3i$&O ./ O  O b05 ssr r$)rirnp audio_utilsrrfeature_extraction_utilsrprocessing_utilsrrr tokenization_utils_baser utilsr r rd get_loggerrr^rr$__all__rr r!rsf, 94HH00   H %,E$NnNb  r