qi(3ddlZddlmZddlmZddlmZddlmZddlm Z m Z ddl m Z dd l mZmZmZmZdd lmZdd lmZd d lmZmZd dlmZmZd dlmZmZddlm Z ejBe"Z#GddeZ$GddeZ%GddeZ&edGddeZ'GddeZ(edGdd eZ)gd!Z*y)"N)nn)ACT2FN)Cache)create_bidirectional_mask)BaseModelOutputWithPoolingCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs)Qwen2AudioEncoderQwen2AudioPreTrainedModel)VoxtralForConditionalGenerationVoxtralMultiModalProjector)WhisperAttentionWhisperEncoderLayer)AudioFlamingo3Configc eZdZy)AudioFlamingo3AttentionN__name__ __module__ __qualname__k/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/audioflamingo3/modular_audioflamingo3.pyrr'r!rc eZdZy)AudioFlamingo3EncoderLayerNrr r!r"r%r%+r#r!r%c eZdZy)AudioFlamingo3PreTrainedModelNrr r!r"r'r'/r#r!r'zT The audio model from AudioFlamingo3 without any head or projection on top.  custom_introc zeZdZdZeedZee dde jde jdzde e zfdZ y) AudioFlamingo3EncoderzY AudioFlamingo3 encoder: Whisper encoder, average pool (time/2), then LayerNorm. ) hidden_states attentionsNinput_featuresinput_features_maskreturnc |jddz dzdz}|jd}|dz dzdz}tj||j|dddfk}t j j|j|}t j j|j|}|jddd}||jjz}t j j||j|j}t|j ||}|j"D]A} |jxr"tj$g|j&k} | r6| ||d}C|jddd}|j)|jddd}|j+|}t-| S) ap Args: input_features (`torch.FloatTensor` of shape `(batch_size, feature_size, sequence_length)`): Log-Mel features extracted from raw audio. Use the processor/feature extractor to compute and pad these features from waveform input. input_features_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. rrdeviceNr)ptraining)config inputs_embedsattention_mask)last_hidden_state)shapesumtorcharanger4r functionalgeluconv1conv2permuteembed_positionsweightdropoutr6rr7layersrand layerdrop avg_pooler layer_normr) selfr.r/kwargsseq_leninput_features_lengthsr8r,r9layerdrops r"forwardzAudioFlamingo3Encoder.forwardBs("''+a/A59!4!8!8!<"81"<!BQ!F#ll7>;P;PQTjklnrkrTss **4::n+EF  **4::m+DE %--aA6 &(<(<(C(CC  --mt||VZVcVc-d 2;;'. [[ HE==DUZZ^dnn%DD %m^ DQ G  H &--aA6  6>>q!QG  6 )+  r!N)rrr__doc__r%r_can_record_outputsrrr=TensortuplerrRr r!r"r+r+3sg  4-  483  3 #\\D03 + + 3  3 r!r+c(eZdZdZdeffd ZxZS)!AudioFlamingo3MultiModalProjectorz Audio adaptor (small MLP) that projects AudioFlamingo3Encoder features to the LLM embedding space so they can replace `` tokens. r7ct|tj|jj |j j |j|_t|j|_ tj|j j |j j |j|_ y)N)bias) super__init__rLinear audio_config hidden_size text_configprojector_biaslinear_1rprojector_hidden_actactlinear_2rLr7 __class__s r"r]z*AudioFlamingo3MultiModalProjector.__init__s     + +V-?-?-K-KRXRgRg  &556    * *F,>,>,J,JQWQfQf  r!)rrrrTrr] __classcell__rhs@r"rYrYzs  3  r!rYz The AudioFlamingo3 model which consists of a fine-tuned Whisper encoder, a multi-modal projector and a Qwen2 language model. ceZdZdZdZdZfdZeedde jde jde e deezfd Zee dd e j"dzde jdzde jdzd e jdzd e j"dzd edzde jdzde j"dzdedzde j"dzdee jzde e defdZfdZxZS)&AudioFlamingo3ForConditionalGenerationNc$t||yrS)r\r]rgs r"r]z/AudioFlamingo3ForConditionalGeneration.__init__s  r!zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.r(r.r/rMr0c d|j|f|dd|}|j|j}|jddz dzdz}t j |j d|jdddf|dddfk}||j|j}||_ |S)a  input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padded feature indices. T)r/ return_dictr2rrr3N) audio_towermulti_modal_projectorr:r<r=r>r;r4to pooler_output)rLr.r/rM audio_output audio_embeds post_lengths valid_masks r"get_audio_featuresz9AudioFlamingo3ForConditionalGeneration.get_audio_featuress,(t''  0CQU Y_ 11,2P2PQ ,//3a7A=A \\,"4"4Q"7 @S@STUY[\U\]`lmnptmt`uu #JMM,2E2E$FG %1 "r! input_idsr9 position_idspast_key_valuesr8labels use_cachecache_positionlogits_to_keepc ||j|}|||j||dj} ||jjk(j d}|j |j|j| j|j}|jd|||||| | | d| }|S)a+ input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor >>> model_id = "nvidia/audio-flamingo-3-hf" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") >>> conversations = [ >>> [ >>> { >>> "role": "user", >>> "content": [ >>> {"type": "text", "text": "Transcribe the input speech."}, >>> { >>> "type": "audio", >>> "path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/t_837b89f2-26aa-4ee2-bdf6-f73f0dd59b26.wav", >>> }, >>> ], >>> } >>> ], >>> [ >>> { >>> "role": "user", >>> "content": [ >>> { >>> "type": "text", >>> "text": "This track feels really peaceful and introspective. What elements make it feel so calming and meditative?", >>> }, >>> {"type": "audio", "path": "https://huggingface.co/datasets/nvidia/AudioSkills/resolve/main/assets/FPSbCAANfbJLVSwD.mp3"}, >>> ], >>> } >>> ], >>> ] >>> inputs = processor.apply_chat_template( >>> conversations, >>> tokenize=True, >>> add_generation_prompt=True, >>> return_dict=True, >>> ).to(model.device) >>> outputs = model.generate(**inputs, max_new_tokens=500) >>> decoded_outputs = processor.batch_decode( >>> outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True >>> ) >>> print(decoded_outputs) ["The spoken content of the audio is...", "The track's calming and meditative feel can be attributed to..."] ```T)ror2)r8r9rzr{r|r}r~rr ) get_input_embeddingsrxrsr7audio_token_id unsqueezemasked_scatterrrr4language_model)rLryr.r/r9rzr{r8r|r}r~rrMruaudio_token_maskoutputss r"rRz.AudioFlamingo3ForConditionalGeneration.forwards`  7D557 BM  %)*?22>CVdh2iwwL!*T[[-G-G GRRSUV )88 ##M$8$89$*=*= + ')%+)) +  + r!c|jdd}|jdd}|jd}t| |i|}||dddk(r|||d<|||d<|S)Nr.r/r~r)popgetr\prepare_inputs_for_generation)rLargsrMr.r/r~ model_inputsrhs r"rzDAudioFlamingo3ForConditionalGeneration.prepare_inputs_for_generation$s $4d;$jj)>E$45w.23737.204(,26*.!%26-.e##d*e))D0e#\\D0 e  t+ e &&- ee((4/e  4'e$;e((4/eell*e+,e eeNr!rl)rlr'r+)+r=r activationsr cache_utilsr masking_utilsrmodeling_outputsrr processing_utilsr utilsr r r r utils.genericrutils.output_capturingr qwen2_audio.modeling_qwen2_audiorrvoxtral.modeling_voxtralrrwhisper.modeling_whisperrrconfiguration_audioflamingo3r get_loggerrloggerrr%r'r+rYrl__all__r r!r"rs  ! 6R&RR75cL>   H % .  !4  $=  ? -?  ? D (B " d-Ld  dN or!