qiDdZddlZddlmZddlZddlmZddlmZ ddl m Z m Z ddl mZddlmZmZd d lmZmZmZmZmZmZmZmZmZd d lmZej<eZ eed Gdde Z!GddeZ"GddeZ#GddeZ$GddeZ%GddeZ&GddeZ'eGddeZ(e Z)Gdd e(eZ*ed!Gd"d#e(Z+Gd$d%eZ,Gd&d'eZ-gd(Z.y))zPyTorch UniSpeech model.N) dataclass)initialization) ModelOutputWav2Vec2BaseModelOutput)PreTrainedModel)auto_docstringlogging) Wav2Vec2EncoderWav2Vec2EncoderStableLayerNormWav2Vec2FeatureEncoderWav2Vec2FeatureProjectionWav2Vec2ForCTC!Wav2Vec2ForSequenceClassificationWav2Vec2GumbelVectorQuantizer Wav2Vec2ModelWav2Vec2PositionalConvEmbedding)UniSpeechConfigzh Output type of [`UniSpeechForPreTrainingOutput`], with potential hidden states and attentions. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<y) UniSpeechForPreTrainingOutputa loss (*optional*, returned when model is in train mode, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentions)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrrtuplera/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/unispeech/modular_unispeech.pyrr+s &*D%  d ")15e''$.5;? 1 1D 8?6:5,,t3:59M5**+d2926Je''(4/6r)rc eZdZy) UniSpeechPositionalConvEmbeddingNr r!r"r(r)r*r,r,Hr)r,c eZdZy)UniSpeechFeatureEncoderNr-r(r)r*r0r0Lr.r)r0c eZdZy)UniSpeechFeatureProjectionNr-r(r)r*r2r2Pr.r)r2c eZdZy)UniSpeechEncoderNr-r(r)r*r4r4Tr.r)r4c eZdZy)UniSpeechEncoderStableLayerNormNr-r(r)r*r6r6Xr.r)r6c"eZdZedZdZy)UniSpeechGumbelVectorQuantizerc|jd}tjtjtj||d j}|S)Nrdim)meanr$expsumxlogy)probsmarginal_probs perplexitys r*_compute_perplexityz2UniSpeechGumbelVectorQuantizer._compute_perplexity]sI*YY %++nn*U[] ^^_cce r)c|j\}}}|j|}|j||z|jzd}|jrt j j|j|jdj|}tj|j||z|jdjd}|j|}n}|jd}|j|jj!d|jddd}|j||z|jd}|j|}|j||zd}|j#d|j$z} | j||z|j|j&d} | j)dj||d} | |fS)Nr<T)tauhardr:rg?)shape weight_projview num_groupstrainingnn functionalgumbel_softmaxfloat temperaturetype_asr$softmaxrDargmax new_zerosscatter_ unsqueeze codevectorsnum_varsr?) selfr batch_sizesequence_length hidden_sizecodevector_probscodevector_soft_distrCcodevector_idxcodevectors_per_grouprYs r*forwardz&UniSpeechGumbelVectorQuantizer.forwardcs3@3F3F0 O[((7 %**:+G$//+Y[]^ ==!}};;##%4+;+;$ < gm$  $)=="":#?RTU[[]ce$ 112FGJ+11b19N6}66 8K8KLUUN''A.   044Z/5QSWSbSbdfg 112BCJ+00o1MrR 0 : :2 >AQAQ Q+00o1Mt`d`m`moqr !oob)..z?BO J&&r)N)r r!r" staticmethodrDrcr(r)r*r8r8\s #'r)r8ceZdZUeed<dZdZdZdZdZ dZ dZ e jdZde jezfdZd ed e jfd Zy ) UniSpeechPreTrainedModelconfig unispeech input_valuesaudioTc t|trutj|jj ddtj |jjtj|jyt|trtj|jj ddtjd|jjd|jjzz ztj |jjdyt|t"rtjd|j$j&z }tj|j$j | |tj|j$j| |yt|t(j*rctj|j d|j,j.|j tj |jyyt|t(j0t(j2fr?tj |jtj4|j yt|t(j6rtj8|j |j`tj|j:|j|jdzz }tj|j| |yyy)zInitialize the weightsr)r=stdrr )abN) isinstancer8initnormal_rJweightzeros_biasuniform_rYr,convmathsqrt kernel_size in_channels constant_r2 projection in_featuresrNLinearrginitializer_range LayerNorm GroupNormones_Conv1dkaiming_normal_groups)r[moduleks r* _init_weightsz&UniSpeechPreTrainedModel._init_weightss* f< = LL++22! D KK**// 0 MM&,, -  @ A LL "" !v{{'>'>q'AFKKD[D['["\]]  NN6;;++Q /  : ; !f//;;; MM&++00QB! <  * LLSdkk6S6S T{{& FKK(' r|| < = KK $ JJv}} %  *   /{{&IIfmmv/A/AFDVDVWXDY/YZ[ fkkaR15'+r) input_lengthscd}t|jj|jjD]\}}||||}|S)zH Computes the output length of the convolutional layers c>tj||z |ddzS)Nfloor) rounding_moder)r$div) input_lengthrzstrides r*_conv_out_lengthzSUniSpeechPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths"99\K7wWZ[[ [r))ziprg conv_kernel conv_stride)r[rrrzrs r* _get_feat_extract_output_lengthsz9UniSpeechPreTrainedModel._get_feat_extract_output_lengthssQ  \ $'t{{'>'> @W@W#X Q K,]KPM Qr)feature_vector_lengthattention_maskc|jddddf}|j|jtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)Nr<r:r)dtypedevicer)r) cumsumrtor$longrIzerosrrarangeflipbool)r[rrnon_padded_lengthsoutput_lengthsr\s r*"_get_feature_vector_attention_maskz;UniSpeechPreTrainedModel._get_feature_vector_attention_masks,22r2:1b5A>>?QRUUV[V`V`a#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr)N)r r!r"rr&base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr$no_gradr LongTensorintrrr(r)r*rfrfs} #$O&*#NU]]_66B e>N>NQT>T   ]b]m]m r)rfceZdZdefdZdZ d dejdzdejdzdejdzde dzd e dzd e dzd e e zfd Z y)UniSpeechModelrgctj||||_t||_t ||_|jdkDs|jdkDrEtjtj|jj|_|j rt#||_nt'||_|j)y)Nrl)rf__init__rgr0feature_extractorr2feature_projectionmask_time_probmask_feature_probrN Parameterr$Tensorr^rvmasked_spec_embeddo_stable_layer_normr6encoderr4 post_init)r[rgs r*rzUniSpeechModel.__init__s ))$7 !8!@"~1 ,, )/!5# ' (* !#34qr7JJ J'+-)77&11   r))NNNNN) r r!r"rrrr$rr%rr'rrcr(r)r*rrs"9 /36:)-,0#'3 llT)3  t+3 !,,t3 3  $; 3 #Tk 3 D[3  ) )3 r)rzZ UniSpeech Model with a vector-quantization module and ctc loss for pre-training. ceZdZdeffd ZdefdZdZe dde jde jde jdefd Z e dd e jd zd e jd zd ed zded zded zdeezf dZxZS)UniSpeechForPreTrainingrgc.t||t||_t j |j |_t||_ t j|j|j|_ t j|j|j|_t j|j|j |_t j |j$|_|j)y)N)superrrrhrNDropoutfeat_quantizer_dropoutdropout_featuresr8 quantizerrcodevector_dimproj_codevector_dim project_qr^ project_hidnum_ctc_classesctc_proj final_dropoutdropoutr)r[rg __class__s r*rz UniSpeechForPreTraining.__init__)s  '/ " 6+H+H I7?6#8#8&:T:TU99V%?%?ASAST &"4"4f6L6LM zz&"6"67  r)rRc&||j_y)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rrR)r[rRs r*set_gumbel_temperaturez.UniSpeechForPreTraining.set_gumbel_temperature8s&1"r)cL|jjjy)z Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rhr_freeze_parametersrs r*rz.UniSpeechForPreTraining.freeze_feature_encoder>s ((;;=r)target_featuresnegative_featurespredicted_featuresctj||gd}tj|j|jd}|j |}||z }|S)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rr:r<)r$catcosine_similarityrQrS)rrrrRlogitss r*compute_contrastive_logitsz2UniSpeechForPreTraining.compute_contrastive_logitsEsa ))_6G$HaP(();)A)A)C_EZEZE\bde0+% r)Nrirrrrrc ||n|jj}|j|||||}|d}|j|d} |j | \} } |j | j |j jj} |j| } tj|jd|jdj|jj} | jdd} tj | j#j |j$} | jdd} | j'd} |j)| d| j)| dz}|j+|}|j-|}d}|s| ||| | f|ddzS|| | f|ddzSt/||| | |j0|j2S) a Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, UniSpeechForPreTraining >>> feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> model = UniSpeechForPreTraining.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> # TODO: Add full pretraining example ```Nrrrr<rlr )rrrrrr)rgrrhrrrrrsrrr$emptysizefill_ replace_probr bernoullirrrX masked_fillrrrrr)r[rirrrrroutputstransformer_featuresrquantized_featuresrprob_replace_matrixsampled_replace_matrixrrs r*rczUniSpeechForPreTraining.forwardYs,&1%,A,A$..BWBWB]B],^_!--.@A#kk*>*C*CA*FH\HaHabcHdekk KK $ $ 2;;AqA!&1D!E!J!J!L!O!OPdPkPk!l!7!A!A!Q!G!7!A!A"!E%112H#N  * *,B+BC H  f%v&24FH]^ahijikalll(*<>STW^_`_aWbb b,1'9"7!//))   r))r)NNNN)r r!r"rrrrrrdr$r%rr rrr'rrc __classcell__)rs@r*rr#s  1#1 >  ** ,,"-- &/3)-,0#' E llT)E  t+E  $; E #Tk E D[ E  . .E E r)rc eZdZy)UniSpeechForCTCNr-r(r)r*rrr.r)rc eZdZy)"UniSpeechForSequenceClassificationNr-r(r)r*rrr.r)r)rrrrrf)/r#rx dataclassesrr$torch.nnrNrrqmodeling_outputsrrmodeling_utilsrutilsr r wav2vec2.modeling_wav2vec2r r rrrrrrrconfiguration_unispeechr get_loggerr loggerrr,r0r2r4r6r8rfrrrrr__all__r(r)r*r sH ! &D-,   5   H %  7K7  7. 'F  4  !:    &D *'%B*'ZHHHV3H -}H V w 6w  w t n  )J  r)