qiddlZddlZddlmZddlZddlZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*GddeZ+GddejXZ-GddejXZ.GddejXZ/GddejXZ0GddejXZ1 dPdejXd ejdd!ejdd"ejdd#ejddzd$e3dzd%e3d&e$e&fd'Z4Gd(d)ejXZ5Gd*d+ejXZ6Gd,d-eZ7Gd.d/ejXZ8Gd0d1ejXZ9Gd2d3ejXZ:e'Gd4d5e"Z; dQd6eZCe'd?@GdAdBe;ZDe'dC@GdDdEe;ZEe'GdFdGe;ZFGdHdIejXZGGdJdKejXZHe'dL@GdMdNe;ZIgdOZJy)RN)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_peft_available)Data2VecAudioConfigc&eZdZdfd ZdZxZS)Data2VecAudioConvLayerct||dkDr|j|dz nd|_|j||_t j |j|j|j||j||j|_ t j|jd|_ t|j|_y)Nrr) kernel_sizestridebiasTelementwise_affine)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconv LayerNorm layer_normrfeat_extract_activation activationselfconfiglayer_id __class__s f/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/data2vec/modeling_data2vec_audio.pyr%zData2VecAudioConvLayer.__init__4s !#Car8cV|jdkDr|ddddd|j f}|SNr)rKr=s r7r?zData2VecAudioPadLayer.forwardSs6    ")!Q0F43F3F2F0F*FGMr8rArFs@r7rHrHNsKr8rHc$eZdZfdZdZxZS) Data2VecAudioPositionalConvLayerczt|tj|j|j|j |j dz|j |_t|j |_ t|j|_ tj|jd|_y)NrJ)rpaddinggroupsFr")r$r%rr) hidden_sizeconv_pos_kernel_sizenum_conv_pos_embedding_groupsr-rHrRrr0r1r.r/r3r4r6s r7r%z)Data2VecAudioPositionalConvLayer.__init__Zs II      33//1477   -V-H-HI  !?!?@,,v'9'9eTr8c|j|}|j|}|jdd}|j|}|jdd}|j |}|SNrrJ)r-rRr<r/r1r=s r7r?z(Data2VecAudioPositionalConvLayer.forwardisd -0  ]3 %//15  6 %//15  6 r8rArFs@r7rPrPYs Ur8rPc$eZdZfdZdZxZS)$Data2VecAudioPositionalConvEmbeddingct|tjt |j Dcgc] }t |c}|_ycc}wN)r$r%r ModuleListrangerLrPlayersr3r4_r6s r7r%z-Data2VecAudioPositionalConvEmbedding.__init__us@ mm?DVEcEc?d e! -f 5 e  esAc|jdd}|jD] }||} |jdd}|SrY)r<r`)r3r>layers r7r?z,Data2VecAudioPositionalConvEmbedding.forward{sI%//15 [[ 1E!-0M 1%//15 r8rArFs@r7r[r[ts  r8r[c.eZdZdZfdZdZdZxZS)Data2VecAudioFeatureEncoderz.Construct the features from raw audio waveformc t|tjt |j Dcgc]}t ||c}|_d|_d|_ ycc}w)N)r5FT) r$r%rr^r_num_feat_extract_layersr conv_layersgradient_checkpointing_requires_grad)r3r4ir6s r7r%z$Data2VecAudioFeatureEncoder.__init__sX ==AFvGeGeAf gA #FQ 7 g ',#" hsA%cJ|jD] }d|_ d|_yNF) parameters requires_gradrkr3params r7_freeze_parametersz.Data2VecAudioFeatureEncoder._freeze_parameterss(__& (E"'E  (#r8c|dddf}|jr|jrd|_|jD] }||} |S)NT)rktrainingrpri)r3 input_valuesr> conv_layers r7r?z#Data2VecAudioFeatureEncoder.forwardsP$QW-    4==*.M '** 6J&}5M 6r8)rBrCrD__doc__r%rsr?rErFs@r7rfrfs8#$ r8rfc$eZdZfdZdZxZS)Data2VecAudioFeatureProjectionc4t|tj|jd|j |_tj|jd|j|_ tj|j|_ y)Nr;eps) r$r%rr.r&layer_norm_epsr/LinearrT projectionDropoutfeat_proj_dropoutdropoutrWs r7r%z'Data2VecAudioFeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r8cp|j|}|j|}|j|}||fSr])r/rr)r3r>norm_hidden_statess r7r?z&Data2VecAudioFeatureProjection.forwards:!__];(:;  ]3 000r8rArFs@r7rzrzs < 1r8rzmodulequerykeyvalueattention_maskscalingrkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr;rJrdim)prur) sizetorchmatmulr<r functionalsoftmaxrru contiguous) rrrrrrrr attn_weights attn_outputs r7eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r8c(eZdZdZ ddedededededed edzffd Z dd e jd e jdzd e jdzdedzde e de e je jdze e jdzff dZxZS)Data2VecAudioAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr is_decoderr! is_causalr4c t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r!)r$r%rrrhead_dimr4 ValueErrorrrrrrk_projv_projq_projout_proj) r3rrrrr!rr4r6s r7r%zData2VecAudioAttention.__init__s "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r8r>key_value_statesroutput_attentionsrreturnc |du}|jdd\}}|r|jdn|} ||d|jf} || d|jf} |j|j| j dd} |r|n|} |j | j| j dd}|j | j| j dd}tj|jjt}||| |||f|jsdn |j|j|d|\}}|j||dj!}|j#|}||dfS)z#Input shape: Batch x Time x ChannelNr;rrJ)rrr)shaperrviewr<rrr get_interfacer4_attn_implementationrrurrreshaperr)r3r>rrrris_cross_attentionbsztgt_lensrc_len q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrs r7r?zData2VecAudioAttention.forwards.T9%**3B/ W/A"((+wgr4==9 wDMM:7t{{=166 FPPQRTUV -?)]5T[[055~FPPQRTUV 7t{{>277HRRSTVWX (?(M(M KK , ,.E) %8      % $}}C$,,LL/ %  % ! \"))#w;FFH mmK0 L$..r8)rFTFN)NNF)rBrCrDrxintfloatboolrr%rTensorrr tupler?rErFs@r7rrsG  -1CCC C  C  CC$d*CD15.2). 1/||1/ ,,-1/ t+ 1/  $; 1/-.1/ u||U\\D0% 2E2LL M1/r8rc$eZdZfdZdZxZS)Data2VecAudioFeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yr])r$r%rractivation_dropoutintermediate_dropoutrrTintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrWs r7r%z!Data2VecAudioFeedForward.__init__"s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r8c|j|}|j|}|j|}|j|}|j |}|Sr])rrrrrr=s r7r?z Data2VecAudioFeedForward.forward/sX// > 00? 11-@ ))-8 ++M: r8rArFs@r7rr!s @r8rc&eZdZfdZddZxZS)Data2VecAudioEncoderLayerct|t|j|j|j d||_tj|j|_ tj|j|j|_ t||_tj|j|j|_y)NF)rrrrr4r|)r$r%rrTnum_attention_headsattention_dropout attentionrrrrr.r~r/r feed_forwardfinal_layer_normrWs r7r%z"Data2VecAudioEncoderLayer.__init__:s /((00,,  zz&"7"78 ,,v'9'9v?T?TU4V< " V-?-?VEZEZ [r8c|}|j|||\}}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|S)Nrr)rrr/rr)r3r>rr attn_residualrrboutputss r7r?z!Data2VecAudioEncoderLayer.forwardIs% )- .L]*8* & |Q ]3 % 5  6 %(9(9-(HH --m<  "    &Gr8rnrArFs@r7rr9s \r8rc reZdZfdZ d dej dej dzdededef dZxZ S) Data2VecAudioEncoderct|||_t||_t j |j|j|_ t j|j|_ t jt|jDcgc] }t!|c}|_d|_ycc}w)Nr|F)r$r%r4r[pos_conv_embedrr.rTr~r/rrrr^r_num_hidden_layersrr`rjras r7r%zData2VecAudioEncoder.__init__^s  B6J,,v'9'9v?T?TUzz&"7"78 mmPUV\VnVnPo$p1%>v%F$pq &+#%qs!CNr>rroutput_hidden_states return_dictc|rdnd}|rdnd}|5|jdjdd|jd}d||<t|j||}|j |} || j |jz}|j|}|j|}txs t|} |jD]j} |r||fz}tjg} |jxr| |jj k} | r| r| |||}|d}| rd}|sb|dfz}l|r||fz}|st#d |||fDSt%||| S) Nr;rrJr)r4 inputs_embedsrrNNc3&K|] }|| ywr]r).0vs r7 z/Data2VecAudioEncoder.forward..smq_`_lms)last_hidden_stater> attentions) unsqueezerepeatrr r4rtodevicer/rr r r`rrandru layerdroprr)r3r>rrrrall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrddropout_probabilityskip_the_layer layer_outputss r7r?zData2VecAudioEncoder.forwardgs#7BD$5b4  %$2$<$(>}?S?S(TT  6  ]3 02R6LT6R [[ PE#$58H$H!#(**R. !]]Z/BT[[EZEZ/ZN![ %!.Te! !.a 0  ,  &9]1=M.s#p!$=f$E#ps)r$r%rrTrrprojr.proj_layer_normr^r_num_adapter_layersr`rrWs `r7r%zData2VecAudioAdapter.__init__s   $ $(:(: : &"4"4f6O6OPDI#%<<0I0I#JD /3 3DI,mm#puU[UnUnOo#pp ))r8ch|j.|j"|j|}|j|}|jdd}|jD]D}tj j }|j r||jkDs=||}F|jdd}|SrY)r r r<r`nprandomrur)r3r>rdlayerdrop_probs r7r?zData2VecAudioAdapter.forwards 99 T%9%9%E IIm4M 00?M%//15 [[ 5EYY--/N==^dnn%D %m 4  5 &//15 r8rArFs@r7rrs  *r8rceZdZUeed<dZdZdZdZdZ dZ dZ e jdZdde jezd edzfd Z dd ed e jfd Zy)Data2VecAudioPreTrainedModelr4data2vec_audiorvaudioTct|trtjd|jj z }t j|jj| |t j|jj| |yt|tr+t j|jjdyt|tjrct j|jd|j j"|j t j$|jyyt|tj&tj(frX|jt j$|j|j t j*|jyyt|tj,rt j.|j|j`tj|j0|j2|j4dzz }t j|j| |yyy)zInitialize the weightsr)abrr)meanstdN)rrzmathsqrtr in_featuresinituniform_weightr!rP constant_r-rrnormal_r4initializer_rangezeros_r. GroupNormones_r)kaiming_normal_rS in_channelsr)r3rks r7 _init_weightsz*Data2VecAudioPreTrainedModel._init_weightss f< = !f//;;; MM&++00QB! <  @ A NN6;;++Q /  * LLSdkk6S6S T{{& FKK(' r|| < ={{& FKK(}}( 6==))  *   /{{&IIfmmv/A/AFDVDVWXDY/YZ[ fkkaR15'+r8N input_lengths add_adaptercT||jjn|}d}t|jj|jjD]\}}||||}|rBt |jj D] }||d|jj}"|S)zH Computes the output length of the convolutional layers c>tj||z |ddzS)Nfloor) rounding_moder)rdiv input_lengthrr s r7_conv_out_lengthzWData2VecAudioPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths"99\K7wWZ[[ [r8r)r4r*zipr*r+r_r r)r3r)r*r2rr rbs r7 _get_feat_extract_output_lengthsz=Data2VecAudioPreTrainedModel._get_feat_extract_output_lengthss 2=1Ddkk--+  \ $'t{{'>'> @W@W#X Q K,]KPM Q 4;;99: _ 04;;C]C] ^  _r8feature_vector_lengthrc|jddddf}|j||}|jtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)Nr;rr*r)dtyperr)r) cumsumr4rrlongrzerosr8rarangeflipr)r3r5rr*non_padded_lengthsoutput_lengths batch_sizes r7"_get_feature_vector_attention_maskz?Data2VecAudioPreTrainedModel._get_feature_vector_attention_masks ,22r2:1b5A>>?Q_j>k'**5::6#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr8r])rBrCrDr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrno_gradr( LongTensorrrr4rArr8r7rrs ($O&*#NU]]_662e>N>NQT>Tcgjncn,Y]%(:?:J:Jr8rr mask_prob mask_length min_masksrc|\}dkr tdkDrtdddtjjdj fd}|-|j j djnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)rmax)r1num_masked_spanepsilonrMrLrNsequence_lengths r7compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spanKsoi,6DwNOoy9 [ (? :-mask_time_indicesrct|jdds|S|j\}}}|)|jj |j ||<n|jj dkDr|jrt||f|jj |jj||jj}tj||jtj}|jj |j ||<|jjdkDr|jrt||f|jj|jj |jj"}tj||jtj}|dddfj%d|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)rLrMrrN)rr8)rLrMrNr;)getattrr4rrvrr8rsrurlmask_time_lengthmask_time_min_masksrrrrrtmask_feature_lengthmask_feature_min_masksexpand)r3r>r~rr@rUrTmask_feature_indicess r7_mask_hidden_statesz&Data2VecAudioModel._mask_hidden_statesst{{$8$? 4A3E3E3G0 O[  (/3/E/E/H/HI\I\/]M+ , [[ ' '! + 5_-++44 KK88-++99 ! !& ->}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r8rvrrrrc H||n|jj}||n|jj}||n|jj}|j |}|j dd}|!|j |jd|d}|j|\} }|j| ||} |j| ||||} | d} |j|j| } |s | |f| ddzSt| || j| j S) a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. NrrJFr7)r~rrrrrr)rextract_featuresr>r)r4rruse_return_dictrqr<rArrrrrwrxData2VecAudioBaseModelOutputr>r) r3rvrr~rrrrrr>encoder_outputss r7r?zData2VecAudioModel.forwardsb 2C1N-TXT_T_TqTq$8$D $++JjJj &1%uEN+/*A*ABR*S' '00 ->~1 ,, )/!5# ' (* << # LL7M!#34qr7JJ J++-)77&11   r8rNNNNN)rBrCrDrr%r}r FloatTensorrKrrrrrrr?rErFs@r7rnrns2"47;26 ,((,!,,t3,((4/ ,\/36:)-,0#'8 llT)8  t+8 !,,t3 8  $; 8 #Tk 8 D[8  - -8 8 r8rnrJzu Data2VecAudio Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ) custom_introceZdZfdZdZe d dejdzdejdzdedzdedzdedzd ejdzd e e zfd Z xZ S) Data2VecAudioForCTCct||t||_t j |j |_|jtd|jdt|dr|jr |jn |j}t j||j|_|j#y)aZ config ([`Data2VecAudioForCTC`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Data2VecAudioForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r*)r$r%rnrrr final_dropoutr vocab_sizerr6hasattrr*rrTrlm_headry)r3r4rr6s r7r%zData2VecAudioForCTC.__init__,s  08zz&"6"67    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  r8cL|jjjyr{rrqrsr|s r7r}z*Data2VecAudioForCTC.freeze_feature_encoderG --@@Br8Nrvrrrrlabelsrc ||n|jj}|I|j|jjk\r"t d|jj|j |||||}|d} |j | } |j| } d} |b||n$tj|tj}|j|jdjtj} |dk\} | jd}|j| }tj j#| dtj$j'dd}tj(j*j-d 5tj j/||| ||jj0|jj2|jj4 } ddd|s| f|t6dz}| | f|zS|St9| | |j:|j< S#1swYExYw) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrrWr;)rr8rF)enabled)blank reduction zero_infinitylosslogitsr>r)r4rrRrrrrrr ones_liker:r4r[r masked_selectrr log_softmaxfloat32r<backendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr>r)r3rvrrrrrrrr>rrr) labels_masktarget_lengthsflattened_targets log_probsoutputs r7r?zData2VecAudioForCTC.forwardNs)$&1% 1` a classification loss is computed (Cross-Entropy). NTrrrr;rrJrr)r4rrrrrstackrrrrrr[rrrArrrrrrrr>r)r3rvrrrrrrrr> norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrs r7r?z.Data2VecAudioForSequenceClassification.forwards!0&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M.. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r8r)rBrCrDr%r}rrrrrrrr?rErFs@r7rrs"C(/3)-,0#'&*C llT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C r8rceZdZfdZdZdZe d dejdzdejdzdejdzde dzd e dzd e dzd e e zfd Z xZ S)(Data2VecAudioForAudioFrameClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_|j |_|j%y)Nr*zgAudio frame classification does not support the use of Data2VecAudio adapters (config.add_adapter=True)r)r$r%rr*rrnrrrrrurr`rrrTrrryrs r7r%z1Data2VecAudioForAudioFrameClassification.__init__s  6= )f.@.@y 18--1  ( (!#ejj.Dz.Q!RD ))F$6$68I8IJ ++ r8cL|jjjyr{rr|s r7r}z?Data2VecAudioForAudioFrameClassification.freeze_feature_encoderrr8cP|jjD] }d|_ yrrrqs r7rz:Data2VecAudioForAudioFrameClassification.freeze_base_modelrr8Nrvrrrrrrc ||n|jj}|jjrdn|}|j|||||}|jjrr|t} t j | d} tjj|jd} | | jdddzjd} n|d} |j| } d} |\t} | | jd|jt j |jd|jd} |s| f|tdz}|St#| | |j$|j& S) rNTrrrr;r)axisr)r4rrrrrrrrrrrr[rrrargmaxrr>r)r3rvrrrrrrrr>rrrrrs r7r?z0Data2VecAudioForAudioFrameClassification.forward&sj0&1%->r1a-HHMMRSMTM#AJM/  ')HFKKDOOtuDY)F)G!HHFM$!//))   r8r)rBrCrDr%r}rrrrrrrr?rErFs@r7rrs C(/3&*)-,0#': llT):  t+:  t# :  $; : #Tk : D[:  & &: : r8rc&eZdZdfd ZdZxZS) AMSoftmaxLossct|||_||_||_t j tj||d|_ t j|_ y)NT)rp) r$r%scalemarginrrrurrandnrrr)r3 input_dimrrrr6s r7r%zAMSoftmaxLoss.__init__esQ   $ll5;;y*#EUYZ '') r8c|j}tjj|jd}tjj|d}t j ||}||jz }tjj||j}|jt j|j||z}|j||}|S)Nrrr)flattenrr normalizerrmmrone_hotrrwhererr) r3r>rr cos_thetapsionehotrrs r7r?zAMSoftmaxLoss.forwardms!((!(< // 1/E HH]F3 $++%&&vt?ekk&++-iHHyy( r8)g>@g?rArFs@r7rrds * r8rcXeZdZdfd Zdej dej fdZxZS) TDNNLayerct||dkDr|j|dz n|j||_|j||_|j ||_|j||_tj|j|j z|j|_ tj|_ y)Nrr)r$r%tdnn_dimr'r( tdnn_kernelr tdnn_dilationdilationrrkernelReLUr1r2s r7r%zTDNNLayer.__init__|s rc&trddlm}tr+t|jrt j d|jdd}|jjj|j|j|jjdd}tjj|||jj |j"}|jdd}|j%|}|S)Nr) LoraLayerzDetected LoRA on TDNNLayer. LoRA weights won't be applied due to optimization. You should exclude TDNNLayer from LoRA's target modules.rrJ)r)rpeft.tuners.lorarrrwarningswarnr<rrr(rr'rrconv1dr!rr1)r3r>rrs r7r?zTDNNLayer.forwards   2  $++y1 O &//15 ##(():):D+BFD]D]!^))F$=$=v?X?XY&v'@'@&BSBST Rs>FcL|jjjyr{rr|s r7r}z.Data2VecAudioForXVector.freeze_feature_encoderrr8cP|jjD] }d|_ yrrrqs r7rz)Data2VecAudioForXVector.freeze_base_modelrr8r)cVd}|jjD] }|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nrrr0s r7r2zJData2VecAudioForXVector._get_tdnn_output_lengths.._conv_out_lengths!;.69A= =r8r)r4r)r3r)r2rs r7_get_tdnn_output_lengthsz0Data2VecAudioForXVector._get_tdnn_output_lengthss:  >  ;;22 LK,]KKM Lr8Nrvrrrrrrc ||n|jj}|jjrdn|}|j|||||}|jjrr|t} t j | d} tjj|jd} | | jdddzjd} n|d} |j| } |jD] } | | } |%| jd} | j!d} n|j#|jd}|j%|}g} g} t'|D]U\}}| j)| |d|fjd| j)| |d|fj!dWt j | } t j | } t j*| | gd}|j-|}|j/|}d}||j1||}|s||f|tdz}||f|zS|St3||||j4|j6S) rNTrrrr;r)rr embeddingsr>r)r4rrrrrrrrrrrr[rrrrr4r enumeraterbcatrqrrrr>r)r3rvrrrrrrrr>r tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrllengthstatistic_poolingoutput_embeddingsrrrs r7r?zData2VecAudioForXVector.forwards0&1%->r1a-HHMMRSMTM#AJM}5 )) 6J&}5M 6  !)..1.5M(,,,3L*.*O*OP^PbPbghPbPi*j '"&"?"?@["\ ML&':; J 6$$]1gvg:%>%C%C%C%JK##M!WfW*$=$A$Aa$A$HI J"KK 6M ;;|4L!II}l&CL 223DE!23  >>&&1D/07;X;Y3ZZF)-)9TGf$ Ev E(!//))   r8r)rBrCrDr%r}rrrKrrrrrrrr?rErFs@r7rrs &C( e6F6F6L /3)-,0#'&*P llT)P  t+P  $; P #Tk P D[ P  t#P   P P r8r)rrrrrnrrprN)Krrcollections.abcrnumpyr rrtorch.nnrrr activationsrintegrations.deepspeedr integrations.fsdpr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrconfiguration_data2vec_audiorrModulerHrPr[rfrzrrrrrrrrrrrrrKndarrayrlrrnrrrrrrr__all__rr8r7r s* $ %&!@76B9G&JJ=76BIIryy6 299 ")): 1RYY 1*!% II% <<% % << % LL4' % T\ %%'(%8S/RYYS/lryy0! :!HE 299E P $299>K?KKd/3 t c?ttt$$t+ t  t ZZ tn 7@ 5@ @ F!" i 6i  i X e -Ie  e P[ /K[ [ |BII. @ C :C  C L r8