qisdZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z ddl mZdd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZej<eZ GddejBZ"GddejFZ$GddejFZ%GddeZ&eGddeZ'eGdde'Z(edGdde'eZ)gd Z*y)!zPyTorch XGLM model.N)nn)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging) XGLMConfigc `eZdZdZd dededededzffd Zdejffd Z xZ S) XGLMScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. num_embeddings embedding_dim padding_idx embed_scaleNc6t||||||_yN)super__init__r)selfrrrr __class__s X/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/xglm/modeling_xglm.pyrz XGLMScaledWordEmbedding.__init__*s  D& input_idsc<t|||jzSr)rforwardr)rr#r s r!r%zXGLMScaledWordEmbedding.forward.swy)D,<,<<z/XGLMSinusoidalPositionalEmbedding.get_embeddingEs!A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#C Q vve--/00r" position_idspast_key_values_lengthc|j\}}||jz}d|z|z}||jjdkDr'|j||j|j |jj d|jdj|||jjdjS)Nr5rrF) sizer6r9r7rr index_selectrPshapedetach)rrUrVbszseq_lenmax_poss r!r%z)XGLMSinusoidalPositionalEmbedding.forwardZs#((* W#dkk1 g+ 66 T\\&&q) )   gt'9'94;K;K L||((L,=,=b,ABGGWVZVbVbVhVhikVlmttvvr"r)Nr)r'r(r)r*r+rr7 staticmethodr>r-no_gradr.r%r/r0s@r!r2r22sNScS#SCRVJSG3GsGQTW[Q[G1c1#1CRVJ11(U]]_wELL4$7wX[wwr"r2cPeZdZdZ ddedededzdedzdedzdedzf fd Z dd ejd ejdzd e dzd ejdzdedejdzde ejejdze ejdzffdZ xZ S) XGLMAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsdropout is_decoderbias layer_idxct|||_||_||_||z|_|j |z|jk7rt d|jd|d|j dz|_||_||_ tj||||_ tj||||_ tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rg)rrrcrdrehead_dim ValueErrorscalingrfrhrLineark_projv_projq_projout_proj)rrcrdrerfrgrhr s r!rzXGLMAttention.__init__is "" !Y. MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r" hidden_stateskey_value_statespast_key_valuesattention_maskoutput_attentionscache_positionreturnc, |du}|j\}} } |r|jdn| } |j||jz} d} |St |t rA|j j|j} |r |j}n|j}n|}|r|n|}|rK|I| rGj|jj}|j|jj}n|j|}|j|}|j!|| d|j"j%dd}|j!|| d|j"j%dd}|T|s|nd}j'|||jd|i\}}|r)t |t rd|j |j<||j(zd|j"f}| j!|| |j(|j"j%dd} | j*|} |j*|}|j*|}|jd} t-j.| |j%dd}|j||j(z| | fk7r/t1d||j(z| | fd |j||j|d| | fk7r#t1d |d| | fd |j|j!||j(| | |z}t-j2|t-j4t-j6|j8j:|j< }|j!||j(z| | }|j8t,j>k(rNt@jBjE|dt,jF jIt,j>}n!t@jBjE|d }|r?|j!||j(| | }|j!||j(z| | }nd}t@jBjK||jJ|jL}t-j.||}|j||j(z| |j"fk7r7t1d||j(| |j"fd |j|j!||j(| |j"}|j%dd}|j+|| |jN}|jQ|}||fS)z#Input shape: Batch x Time x ChannelNrFrFr5rxTz$Attention weights should be of size z , but is z!Attention mask should be of size r<)rEr;rDptrainingz `attn_output` should be of size ))rXrZrqrm isinstancer is_updatedgetrhcross_attention_cacheself_attention_cachelayerskeysvaluesrorprPrk transposeupdaterdreshaper-bmmrlmaxtensorfinfor;minr<float16r functionalsoftmaxfloat32r@rer~rcrr)rrsrtrurvrwrxis_cross_attentionr\tgt_len_src_len query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shape attn_weightsattn_weights_reshaped attn_probs attn_outputs r!r%zXGLMAttention.forwards0.T9',,.Wa/A"((+w{{=1DLL@   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6L#gr4==ISSTUWXYJ',,S'2t}}MWWXY[\]L*7It+?+F+F dnn?OQ_>`,( L&*_FY*ZAEO..t~~>DNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL 99ell5;;|7I7I+J+N+NWcWjWjkL(,,S4>>-A7GTL    .==002U]]0[^^_d_l_lmL==0020FL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<$>?"(";";  % % -.. 0000# !D ,.<<+GD ($&LL$@!99T^^V^^<99V^^T^^< " T^^ <r"rsrvencoder_hidden_statesencoder_attention_maskrurw use_cacherxryc .|} |j|}|j|||||\}} tjj ||j|j }| |z}d} |h|} |j |}|j||||||\}} tjj ||j|j }| |z}|} |j|}|j|j|}tjj ||j|j }|j|}tjj ||j|j }| |z}|f} |r| | | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rsrurvrwrxr|N)rsrtrvrurwrx) rrrrrer~rrrrrrr) rrsrvrrrurwrrxresidualself_attn_weightscross_attn_weightsoutputss r!r%zXGLMDecoderLayer.forwards2!11-@ ,0>>'+)/) ,:, ( ( --mt||VZVcVc-d  =0 " ,$H 88GM040A0A+!65 /"3- 1B1 -M-MM11-4< >Gr"r)NNNNFTN) r'r(r)rrr-r.rrr%r/r0s@r!rrs=z=D/3596:(,).!%.2F||F t+F %||d2 F !& t 3 F  F $;F$;F t+F Fr"rc8eZdZUeed<dZdZdgZfdZxZ S)XGLMPreTrainedModelrmodelTrct||t|tr_|j |j |j z|j|j}tj|j|yyr) r _init_weightsrr2r>r3r6rrinitcopy_r9)rmodulerBr s r!rz!XGLMPreTrainedModel._init_weightslsg f% f? @ ..$$v}}4f6J6JFL^L^K JJv~~{ 3 Ar") r'r(r)r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr/r0s@r!rres( &*#+,44r"rcxeZdZdeffd Ze ddejdzdejdzdejdzdejdzdejdzd edzd ejdzd e dzd e dzd e dzde dzdejdzde eje zfdZ xZ S) XGLMModelrc t|||j|_|j|_|j|_|j |_|jrtj|jnd}t|j|j|j ||_t|j |j|j|_t#j$t'|j(Dcgc]}t+||c}|_t#j.|j|_d|_|j5ycc}w)Nr&)r)rhF)rrre layerdrop pad_token_idrmax_position_embeddingsmax_target_positionsscale_embeddingrGsqrtrr vocab_size embed_tokensr2embed_positionsr ModuleListrange num_layersrrr layer_normgradient_checkpointing post_init)rrrir s r!rzXGLMModel.__init__ws   ~~ ))!..$*$B$B!393I3Idii/s 3   v~~t/?/?[  A  * * NN     mmTYZ`ZkZkTl$mq%5f%J$mn ,,v~~6&+#  %ns E&Nr#rvrUrrru inputs_embedsrrwoutput_hidden_states return_dictrxryc 4| | n|jj} | | n|jj} ||n|jj}| | n|jj} | | t d|8|j |||j}|jd|d}n!||jdd}n t d||j|}|jr%|jr|rtjdd}|rd|b||jjr4tt!|jt!|jnt!|j}||j#nd}| 2t%j&|||j(d z|j* } t-|j||| | }|Vt%j&||d|zt$j.| |j*n |j* }|j1d}||t3|j||| }||j5||j7|j*z}t8j:j=|t?|j<|j}| rdnd}| rdnd}| r|dnd}tA|jBD]k\}}| r||fz }|jr%t%jDg}||jFkr?||||||| || }|d}| sW||d fz }|c||dfz }m|jI|}| r||fz }| stKd|||||fDStM|||||S)a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) NzDYou cannot specify both input_ids and inputs_embeds at the same timerFz5You have to specify either input_ids or inputs_embedsz_`use_cache = True` is incompatible with gradient checkpointing`. Setting `use_cache = False`...F)rrrr{)rrrvrxrur:)rrrvrr|)rrurwrrxr5c3$K|]}|| ywrr).0vs r! z$XGLMModel.forward..s=s)last_hidden_staterurs attentionscross_attentions)'rrwrruse_return_dictrl%warn_if_padding_and_no_attention_maskrXrPrrr~logger warning_onceis_encoder_decoderr rget_seq_lengthr-rJrZr<r longrLr rr@rrrer, enumeraterrandrrrr)rr#rvrUrrrurrrwrrrxkwargs input_shaperVrsall_hidden_statesall_self_attnsall_cross_attentionsidx decoder_layerdropout_probability layer_outputss r!r%zXGLMModel.forwards<2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I  &',,.s3KTU U   --i8M  & &4==##u"  0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg  !"\\&(>ATATUVAW(W`m`t`tN,;;'))+     <<&B"88jj+4+@y''mFZFZ L (11!4L ! ,1G1S%>{{+5&; & "&(<(<\Ka(b(e(e  )   --muT\\?R]a]j]j-k #7BD0d&7sJ2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %4%% ;;11![[55   DY,F'+'7D7V# CV C0#33!//))$55   r")NNNNNNNNNNNNNr)r'r(r)r_tied_weights_keysrrr-r.rrr+rrr%r/r0s@r!rr,su *,GH*..2,0596:(,-1&*!%)-,0#'.2-.T <<$&T  t+T llT) T %||d2 T !& t 3 T T ||d*T  t#T $;T  $;T #TkT D[T  t+T ell*T " u|| @ @#T T r"r)rrr)+r*rGr-rrr activationsr cache_utilsrrr generationr masking_utilsr r modeling_layersr modeling_outputsrrmodeling_utilsrutilsrrconfiguration_xglmr get_loggerr'r EmbeddingrModuler2rbrrrr__all__rr"r!rs &!CC)J9l-,*   H % =bll =1w 1whS2BIIS2lf1fR 4/ 4 4s #s s l a )?a  a H Br"