qi4ddlZddlmZddlZddlZddlmZddlmZddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'e#jPe)Z*GddeZ+GddeZ,GddeZ-Gddej\Z/Gddej\Z0Gddej\Z1Gdd ej\Z2 dDd!ej\d"ejfd#ejfd$ejfd%ejfdzd&e4dzd'e4d(ee!fd)Z5Gd*d+ej\Z6Gd,d-ej\Z7Gd.d/eZ8Gd0d1ej\Z9e"Gd2d3eZ: dEd4e;ed?e:ZBe"d@=GdAdBe:ZCgdCZDy)FN)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack)TransformersKwargsauto_docstringlogging)is_flash_attention_requested) SEWConfigc&eZdZdfd ZdZxZS)SEWNoLayerNormConvLayercdt||dkDr|j|dz nd|_|j||_t j |j|j|j||j||j|_ t|j|_ y)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__s V/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/sew/modeling_sew.pyr"z SEWNoLayerNormConvLayer.__init__/s !#Car3cV|jdkDr|ddddd|j f}|SNr)rqr6s r2r8zSEWSamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr3r:r?s@r2rmrmsKr3rmc$eZdZfdZdZxZS) SEWUpsamplingct|tj|j|j|j z|_t|j|_ |j |_yr5) r!r"rLinearr_rb projectionrr+r,r.r/r1s r2r"zSEWUpsampling.__init__sW ))F$6$68J8JVMbMb8bc !?!?@$33r3c.|j|}|j|}|jdkDrc|j\}}}||jz}||jz}|j |||j|}|j |||}|S)Nr)rxr,rbsizereshape)r.r7bszsrc_len src_embed_dimtgt_len tgt_embed_dims r2r8zSEWUpsampling.forwards 6  6    "*7*<*<*> 'C- 3 33G)T-@-@@M)11#w@S@SUbcM)11#w NMr3r:r?s@r2rurus 4 r3ruc.eZdZdZfdZdZdZxZS)SEWFeatureEncoderz.Construct the features from raw audio waveformc t||jdk(rDt|dgt |j dz Dcgc]}t ||dzc}z}nV|jdk(r.t |j Dcgc]}t||}}ntd|jdtj||_ d|_ d |_ ycc}wcc}w) Ngroupr)r0rlayerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)r!r"feat_extract_normrKrangenum_feat_extract_layersrrA ValueErrorr ModuleList conv_layersgradient_checkpointing_requires_grad)r.r/irr1s r2r"zSEWFeatureEncoder.__init__s   # #w .0!DEINvOmOmpqOqIrIDE'Q?IK % % 0NSTZTrTrNst0!DtKt01I1I0JJst ==5&+#"Ius C" C'cJ|jD] }d|_ d|_yNF) parameters requires_gradrr.params r2_freeze_parametersz$SEWFeatureEncoder._freeze_parameterss(__& (E"'E  (#r3c|dddf}|jr|jrd|_|jD] }||} |S)NT)rtrainingrr)r. input_valuesr7 conv_layers r2r8zSEWFeatureEncoder.forwardsP$QW-    4==*.M '** 6J&}5M 6r3)r;r<r=__doc__r"rr8r>r?s@r2rrs8#"$ r3rmodulequerykeyvalueattention_maskscalingdropoutkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)NrHrUrr])prr) r{torchmatmulrIr functionalsoftmaxrr contiguous) rrrrrrrr attn_weights attn_outputs r2eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r3c(eZdZdZ ddedededededed edzffd Z dd e jd e jdzd e jdzdedzde e de e je jdze e jdzff dZxZS) SEWAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr is_decoderr is_causalr/c t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r )r!r"rrrhead_dimr/rrrrrrwk_projv_projq_projout_proj) r.rrrrr rr/r1s r2r"zSEWAttention.__init__s "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r3r7key_value_statesroutput_attentionsrreturnc |du}|jdd\}}|r|jdn|} ||d|jf} || d|jf} |j|j| j dd} |r|n|} |j | j| j dd}|j | j| j dd}tj|jjt}||| |||f|jsdn |j|j|d|\}}|j||dj!}|j#|}||dfS)z#Input shape: Batch x Time x ChannelNrHrrU)rrr)shaperrviewrIrrr get_interfacer/_attn_implementationrrrrr|rr)r.r7rrrris_cross_attentionr}rr~ q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrs r2r8zSEWAttention.forward&s.T9%**3B/ W/A"((+wgr4==9 wDMM:7t{{=166 FPPQRTUV -?)]5T[[055~FPPQRTUV 7t{{>277HRRSTVWX (?(M(M KK , ,.E) %8      % $}}C$,,LL/ %  % ! \"))#w;FFH mmK0 L$..r3)rFTFN)NNF)r;r<r=rintfloatboolrr"rTensorrr tupler8r>r?s@r2rrsG  #'CCC C  C  CCD CD15.2). 1/||1/ ,,-1/ t+ 1/  $; 1/-.1/ u||U\\D0% 2E2LL M1/r3rc$eZdZfdZdZxZS)SEWFeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yr5)r!r"rDropoutactivation_dropoutintermediate_dropoutrwr_intermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrys r2r"zSEWFeedForward.__init__[s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r3c|j|}|j|}|j|}|j|}|j |}|Sr5)rrrrrr6s r2r8zSEWFeedForward.forwardhsX// > 00? 11-@ ))-8 ++M: r3r:r?s@r2rrZs @r3rc&eZdZfdZddZxZS)SEWEncoderLayerct|t|j|j|j d||_tj|j|_ tj|j|j|_ t||_tj|j|j|_y)NF)rrrrr/eps)r!r"rr_num_attention_headsattention_dropout attentionrrrrrDlayer_norm_epsrEr feed_forwardfinal_layer_normrys r2r"zSEWEncoderLayer.__init__ss %((00,,  zz&"7"78 ,,v'9'9v?T?TU*62 " V-?-?VEZEZ [r3c|}|j|||\}}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|S)Nrr)rrrErr)r.r7rr attn_residualr_outputss r2r8zSEWEncoderLayer.forwards% )- .L]*8* & |Q ]3 % 5  6 %(9(9-(HH --m<  "    &Gr3rr:r?s@r2rrrs \r3rc.eZdZfdZ ddZxZS) SEWEncoderct|||_t||_t j |j|j|_t j|j|j|_ t j|j|_t j t#|j$Dcgc] }t'|c}|_t+||_d|_ycc}w)NrF)r!r"r/rSpos_conv_embedr AvgPool1drbpoolrDr_rrErrrrrnum_hidden_layersrlayersruupsampler)r.r/rr1s r2r"zSEWEncoder.__init__s  8@LL!6!68M8MN ,,v'9'9v?T?TUzz&"7"78 mmeFLdLdFe$f_V%<$fg %f- &+#%gsDc *|rdnd}|rdnd}||jdjdd|jd}t|jrd||<|d|vr|nd}ngd||<|j j d} | |jjz} |jd|jjz} tjd| | jjddj| jdd} | | jddkj }d|ddddddfj|j z }|tj|jj z}|j|jdd|jd|jd}|jd} |j#dd}|j%|}|j'|}t!|j)d|j)d}|d d|f|d d|fz}|j#dd}|j+|}|j-|}t/xs t1|}|j2D]j}|r||fz}tj4g}|j6xr||jj8k}|r|r|||| }|d}|rd }|sb|dfz}l|r||fz}|j;|}|jd| kr4t<j>jA|ddd| |jdz f}|stCd |||fDStE|||S)NrHrrUrrdeviceg?dtype.rNNc3&K|] }|| ywr5r).0vs r2 z%SEWEncoder.forward..smq_`_lmslast_hidden_stater7 attentions)# unsqueezerepeatrrr/longsumrbrarangerrexpandtorfinfominrIrrr{rErr r rrandr layerdroprrrpadrr )r.r7rroutput_hidden_states return_dictall_hidden_statesall_self_attentionsexpand_attention_mask input_lengthsoutput_lengthsmax_encoder_length attention_idsn_input_timestepsposition_embeddingspooled_hidden_states min_length synced_gpusrdropout_probabilityskip_the_layer layer_outputss r2r8zSEWEncoder.forwards#7BD$5b4  %$2$<$Q>Q2R2V2V!V!/!6!6"((+Q0D0DR0H.J^J^_aJb"*//2%//15 "11-@#yy7,11"57K7P7PQS7TU ,S+:+-=>ATUXZe[eZeUeAff %//15  6  ]3 02R6LT6R [[ PE#$58H$H!#(**R. !]]Z/BT[[EZEZ/ZN![ %!.Te! !.a 0  ,  &9]1=M'>q'AFKKD[D['["\]]  NN6;;++Q /  * LLSdkk6S6S T r|| < = KK $ JJv}} %  *)+ 6:.76:3N"::FOOV__;]mn:o<,,V]];<<#::6==XY:Z<,,V]];<<$$V]]3 fryy"))4 5&++:Q KK $;R 5<<<tj||z |ddzS)Nfloor) rounding_moder)rdiv) input_lengthrrs r2_conv_out_lengthzMSEWPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length+s"99\K7wWZ[[ [r3)zipr/r'r()r.rr>rrs r2 _get_feat_extract_output_lengthsz3SEWPreTrainedModel._get_feat_extract_output_lengths&sQ  \ $'t{{'>'> @W@W#X Q K,]KPM Qr3feature_vector_lengthrc|j|jdjtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)NrHr)rrrr) r@r rrr rzerosrrr flipcumsumr)r.rArr batch_sizes r2"_get_feature_vector_attention_maskz5SEWPreTrainedModel._get_feature_vector_attention_mask5s>>~?Q?QRT?UVYYZ_ZdZde#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr3N)r;r<r=r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrno_gradr7 LongTensorrr@rGrr3r2r'r's| $O&*#NU]]_%%< e>N>NQT>T   ]b]m]m r3r'r mask_prob mask_length min_masksrc|\}dkr tdkDrtdddtjjdj fd}|-|j j djnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)rmax)r=num_masked_spanepsilonrSrRrTsequence_lengths r2compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spanhsoi,6DwNOoy9 [ (? :-}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r3rrrrrc Z||n|jj}||n|jj}||n|jj}|j |}|j dd}|j |}|jr|j|}|j|} ||j| jd|}|j| |} |j| ||||} | d} |s | f| ddzSt| | j| j S)a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. NrrU)rrrrrrr)r/rruse_return_dictrvrIrErwrxrzrGrrrr r7r) r.rrrrrrrextract_featuresr7encoder_outputss r2r8zSEWModel.forwardsU 2C1N-TXT_T_TqTq$8$D $++JjJj &1%  %!DD]EXEXYZE[]klN00Rc0d ,, )/!5# ' (* !#oab&99 9+)77&11  r3rNNNNN)r;r<r=rr"r FloatTensorrQrrrrrr r8r>r?s@r2rtrtsy.7;26 ,((,!,,t3,((4/ ,\/36:)-,0#'4 llT)4  t+4 !,,t3 4  $; 4 #Tk 4 D[4   4 4 r3rtzk SEW Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ) custom_introceZdZddedzffd ZdZdZdZe dde jdzde jdzd e dzd e dzd e dzd e jdzd e e zfdZxZS) SEWForCTCN target_langct||t||_t j |j |_||_|jtd|jdt|dr|jr |jn |j}t j ||j|_|j%y)a- target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`SEWForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SEWForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r!r"rtr(rr final_dropoutrr vocab_sizerr1rdroutput_hidden_sizer_rwlm_headr)r.r/rrr1s r2r"zSEWForCTC.__init__>s  F#zz&"6"67 &    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  r3c 8ttjdk(ry|j}|&t |j ddt d|d|-t |j ddtjdy||j|dyy) a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load) rrrrrr/rloggerinfo load_adapter)r.rrs r2 tie_weightszSEWForCTC.tie_weights[s 6 75<<;O O  &&  "wt{{r?s@r2rr8s C$J:<08(/3)-,0#'&*E llT)E  t+E  $; E #Tk E D[ E  t#E   E E r3rz SEW Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZe d dejdzdejdzde dzde dzd e dzd ejdzd e e zfd Z xZ S)SEWForSequenceClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_tj|j |j$|_|j)y)NrzZSequence classification does not support the use of SEW adapters (config.add_adapter=True)r)r!r"rdrrrtr(ruse_weighted_layer_sumrr}rrf layer_weightsrwr_classifier_proj_size projector num_labels classifierr)r.r/ num_layersr1s r2r"z%SEWForSequenceClassification.__init__s  6= )f.@.@l F#--1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r3cL|jjjyrrrs r2rz3SEWForSequenceClassification.freeze_feature_encoderrr3cP|jjD] }d|_ yrrrs r2rz.SEWForSequenceClassification.freeze_base_modelrr3Nrrrrrrrc <||n|jj}|jjrdn|}|j|||||}|jjrr|t} t j | d} tjj|jd} | | jdddzjd} n|d} |j| } || jd} n|j| j d|} | j#dj%dd| j d} d | | <| jd| jdjddz } |j'| }d}|Ft)}||jd|jj*|jd}|s|f|tdz}||f|zS|St-|||j.|j0 S) a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SEWProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrrHrrUrr)r/rrr(rrstackrrrrrr rr+rGrr r rrrrr7r)r.rrrrrrrrr7 norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrs r2r8z$SEWForSequenceClassification.forwards0&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M.. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r3r)r;r<r=r"rrrrrrrrr8r>r?s@r2rrs"8(/3)-,0#'&*C llT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C r3r)rrrtr')Nrrs)Er/collections.abcrnumpyr^rrtorch.nnrrr- activationsrintegrations.deepspeedr integrations.fsdpr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr rrmodeling_utilsrrrprocessing_utilsrrcrrr utils.genericrconfiguration_sewr get_loggerr;rrrArKModulerSrmrurrrrrrrrr'rrrQndarrayrrrtrrr__all__rr3r2rs* $ %&!@7B9YYrr&@@9(   H %8*6660((VbiiBII,# #X!% II% <<% % << % LL4' % T\ %%'(%8S/299S/lRYY0!0!Hc c LBBBR/3 t c?ttt$$t+ t  t ZZ tnx !x x v!" K "K  K \ e #5e  e P Zr3