qiRdZddlZddlmZddlmZddlZddlmZddlm Z m Z m Z ddl m Zdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e(jZe.Z/Gddej`Z1dejdde3dejdfdZ4GddZ5 d>dej`d ejdd!ejdd"ejdd#ejddzd$e6d%e6fd&Z7Gd'd(ej`Z8Gd)d*ej`Z9Gd+d,ej`Z:Gd-d.ej`Z;Gd/d0eZ<Gd1d2eZ=e'Gd3d4e#Z>e'Gd5d6e>Z?Gd7d8e>eZ@e'd9:Gd;dZAgd=ZBy)?zPyTorch Zamba model.N)Callable)Any)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache)GenerationMixin)lazy_load_kernel)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)resolve_internal_import) ZambaConfigcheZdZddeddffd Zdej dej fdZdZxZ S) ZambaRMSNormepsreturnNct|tjt j ||_||_y)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer __class__s Z/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/zamba/modeling_zamba.pyr#zZambaRMSNorm.__init__2s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor%float32powmeanrsqrtr(r')r)r. input_dtypevariances r,forwardzZambaRMSNorm.forward:sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r-c^t|jjd|jS)Nz, eps=)tupler'shaper(r)s r, extra_reprzZambaRMSNorm.extra_reprAs* ))*+6$2G2G1HIIr-)gư>) __name__ __module__ __qualname__floatr#r%Tensorr;r@ __classcell__r+s@r,rr1s7$$$$;U\\;ell;Jr-rr.n_repr c|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r>expandreshape)r.rHbatchnum_key_value_headsslenhead_dims r, repeat_kvrPFso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr-c .eZdZdZdZej dfdZdZ ddejdejde d e e e fdzd eejejff d Zd ej fd Zdde dzd e fdZdejde d ee e ffdZy)ZambaHybridDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNc ,||_d|_|j|_d|_|j|j z|_|j|_|j|_ |j|_ g|_ g|_ g|_i|_i|_i|_t%|j&D]}|xjt)j*||j |j||gz c_ ||j|j |jz|jf}|xjt)j*|||gz c_ |j|dk(s|jj-|t%|j&Dcgc]}t)j.gg|z|c}|_t%|j&Dcgc]}t)j.gg|z|c}|_ycc}wcc}w)NFdevicer3hybridrU)r3is_compileablelayers_block_typehas_previous_state mamba_expandr*intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr%zerosappendtensor key_cache value_cache)r)config batch_sizer3rUi cache_shape_s r,r#z ZambaHybridDynamicCache.__init__bs #!'!9!9"'!'!4!4v7I7I!I$22 & 3 3#11"$  v//0 2A    J(>(>@U@U^dlqr!  ""&&$*<*<<## K OO Ke TU UO%%a(H4''..q1 2SXX^XpXpRqrQ%,,tj'8HrTYZ`ZrZrTstqELL" ):6Jtsts "H !"Hc,t|jSN)lenrmr?s r,__len__zZambaHybridDynamicCache.__len__s4>>""r- key_states value_states layer_idx cache_kwargsr c|j|jddk(r||j|<||j|<nftj|j||gd|j|<tj|j||gd|j|<|j||j|fS)Nr1rr0dim)rmr>rnr%cat)r)rxryrzr{s r,updatezZambaHybridDynamicCache.updates >>) $ * *2 .! 3(2DNN9 %*6D  Y '(- 4>>)3Lj2Y_`(aDNN9 %*/))T5E5Ei5PR^4_ef*gD  Y '~~i($*:*:9*EEEr-beam_idxc|jdkDrvtt|jD]S}|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<Vyy)zDReorders the cache for beam search, given the selected beam indices.rN) get_seq_lengthrhrvrmrU index_selectr4rnrbrc)r)rrzrUs r, reorder_cachez%ZambaHybridDynamicCache.reorder_caches[    1 $"3t~~#67 m  299,0NN9,E,R,RSTV^VaVabhVi,jy))))4;;.2.>.>y.I.V.VWXZbZeZeflZm.n  +)))4;;.2.>.>y.I.V.VWXZbZeZeflZm.n  +3::-1__Y-G-T-TUVX`XcXcdjXk-l * m %r-c||jvr|jdn|}t|j|ks|j|jddk(ry|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rr1)rdrvrmr>)r)rzs r,rz&ZambaHybridDynamicCache.get_seq_lengthsn3<4CZCZ2ZD++A.`i t~~ ) +t~~i/H/N/Nr/RVW/W~~i(..r22r-cache_positioncTd}|jd}|j||z}||fS)zDReturn the length and offset of the cache, used to generate the maskr)r>r)r)rrz kv_offset query_length kv_lengths r,get_mask_sizesz&ZambaHybridDynamicCache.get_mask_sizess7 %++A. '' 2\A )##r-ru)r)rArBrC__doc__rXr%float16r#rwrEintdictstrrr=r LongTensorrrrr-r,rRrRRs N16tu@#/3 FLLFllF F 38nt+ F u||U\\) * F$ me&6&6 m3d 333$U\\$c$eTWY\T\o$r-rRmodulequerykeyvalueattention_maskscalingdropoutc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr0r r1)r~r3)ptrainingr)rPnum_key_value_groupsr%matmul transposer functionalsoftmaxr5r4r3rr contiguous) rrrrrrrkwargsrxry attn_weights attn_outputs r,eager_attention_forwardrs3 ; ;$>&B\B\$\!'-'E'E$ )d2 !'!9!9ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii : :T]] JFL^L^ejk r-Nr.rpast_key_valuesrr c |jdd}g|d|j}|j|j|j dd}|j |j|j dd} |j |j|j dd} ||j| | |\} } tj|jjt} | ||| | |f|jsdn |j|jd|\} } | j g|dj#} |j%| } | | fS)Nr1rr0)rr)r>rOrviewrrrrr get_interfacero_attn_implementationrrrrrKrr)r)r.rzrrr input_shape hidden_shape query_statesrxryattention_interfacerrs r,r;zZambaAttention.forwardsk$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  &'6'='=j,Xa'b $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r-ru)rArBrCrrrr#r%rErRrrr=r;rFrGs@r,rrs l{lsl.;? #)||#)#) t+ #) 147 #) -. #) u||U\\D0% 2E2LL M#)r-rcleZdZdZdeffd Z d dejdefdZ d defdZ d defdZ xZ S) ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. roc  t|||_||_|j|_|j |_|j|_|j|jz|_ |j|_ |j|_ |j|jz|_|j|_|j"|_t'j(|j|j|j |j|j|jdz |_|j,|_t0|j,|_|j4|_t'j8|j|jdz|j$|_t'j<t?j@|j|j|j dzz|j|_!t'j<t?j@|j|j|jdz dz|jdzz |_"t'j<t?j@|j|j|_#t?jHd|j dzt>jJdddf}|jM|jdjO}t'j<t?jP|jS|j|jd|_*t'j<t?jV|j|j|_,t'j8|j|j|j$|_-t]da/tat^d da1tat^d da2t]d a3titfd a5tatfdda6tatfdda7tqtjtltdtbtnfa9trsttjwdyy)Nr) in_channels out_channelsr kernel_sizegroupspaddingr0rg?r3r1z causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmz8ops.triton.selective_state_update.selective_state_update) chained_pathselective_scan_fnmamba_inner_fnaqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)|j>gd\} }}|j@dddf| jddz}t'jB|jDjG }|jH|jHjGnd}t'jJ|d|f|jL|jN}|rtQ|jD]}tS|jT|jdd|f||d df||d df||||dddf||dddf|jV|| |d df||d j%d}t'jX||fd}nAt'jJ|d|j6|j>f|jL|jN}tQ|jD]}t[||||||||jdd||jdd|jV|jG| |||d d \}}t'jX||fdj}t'jX||j%dfd}|*|(|jT|jj3||j]|jdd}|S) Nrr0r1r}r)rrrT.T) dt_softplus)delta_softplusreturn_last_state)/r>rZrrrchunksqueezerrKrarr'sizerrbrzrr unsqueezer%rrrpadr`copy_rrrsplitrr^rexprrDremptyrUr3rhrrcrrrr)r)r.rrrpseq_lenrsuse_precomputed_statesprojected_statesgate conv_weightsrbssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r,cuda_kernels_forwardz$ZambaMambaMixer.cuda_kernels_forwardss"/!4!4 GQ!-T!9!nl>]>]!nbimnbn << 6@@AF.33JAwOUUVW]^U_ t%--a0;;= ||A||J(:(:BHRRSTVWX{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)((8    M*33B7M)%))Na>{K,]L$++JZJZgkgvgvwM)%))Nar3rrrrrrrKra isinstancerRrcrzrcloner4rUrZrbr%rollsumrr'rrrrrrrr`rjrr^rrrrrsoftplusrrrDrhrrkstackrr)r) input_statesrrrprrsr3rr.r use_cacher conv_staterrrr r r discrete_A discrete_BdeltaB_ur rq scan_outputrs r, slow_forwardzZambaMambaMixer.slow_forwards!-!3!3 GQ""<< 5??1E.33JAwOUUVW]^U_ t%--a0;;= ||A||J(:(:BHRRSTVWX|-DE 00@FFqIZW}}(33DNNCIIK (33DNNC ! ]%9%9:I//qL ,,T^^<BB1ES)55dnnE "ZZ 2BG '4Q1W'= 1a8$;E ((8 % *t{{7I7I!QPQ'7R*RXZ [ %%!T[[%5%55M $ 7 : :5 A K KB O !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM]]..}t?T?TWdWjWjkmWn?npq>rs ;E ((8 $])CC'M)R S !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM T//1D1DdFYFYZ$++I ) -0H0H0K K  HHT[[%?XgX %NOM) -0H0H0K K &--b$2D2DdFYFY[bcmmnoqrs ,,Qa];mKVVWY[]^++ T00$2E2EtGZGZ[ac  1a#11!T':Y=P=PQSUW=XX\`\m\m tQ ]   ]]334FGYYtzz'') * *YYqD!T1!458J1aQRTUW[K[8\\] '1aD(89AaD!Q>N ? E E GG w 9A"1aAq=1;;AqAIMPXYZ\]_`bcefYfPgPqPqrsuvPwwI,,y':':1a'@'C'CE'JAaQRTUWXjMLcLcdfLghK    Aq!QJ 7 8 9kk,B7 !]TVVAtQ=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r) rrrrrrrrrUtype ValueErrorrr#)r)r.rrrs r,r;zZambaMambaMixer.forward1s!$ #%68HJ^`n o"   )V4;M;M;T;T;Y;Y-Y i ,,]LYg,h h   ^ \\r-)NN) rArBrCrrr#r%rErRrr#r;rFrGs@r,rrsZ M{M`im_%"\\_%9P_%B[%7N[%z ]3J ]r-rc$eZdZfdZdZxZS)ZambaMLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFr) r"r#ror*r\rr gate_projup_proj down_projr hidden_actact_fnr)ror+s r,r#zZambaMLP.__init__Cs  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r-c|j|j|j||j|z}|Sru)r.r0r,r-)r)xr.s r,r;zZambaMLP.forwardMs6NN4;;t~~a/@#ADLLQRO#ST r-)rArBrCr#r;rFrGs@r,r)r)Bs 0r-r)ceZdZddededzffd Z ddejdejdedejdzdedzd e dzd e dzd e e d e eje ejejfdzffd ZxZS)ZambaAttentionDecoderLayerNrorzct|t|||_t ||_t |j|j|_ t |j|j|_ y)Nr) r"r#r self_attnr) feed_forwardrr rms_norm_epsinput_layernormr*pre_ff_layernormrs r,r#z#ZambaAttentionDecoderLayer.__init__Ss_ ' :$V,+F,H,HfNaNab ,V-?-?VEXEX Yr-r.original_hidden_statesrroutput_attentionsrrr c tj||gd}|j|}|jd||||||d|\}} |j |}|j |}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r1r})r.rzrrr>rr)r% concatenater;r8r<r9) r)r.r=rzrrr>rrself_attn_weightsoutputss r,r;z"ZambaAttentionDecoderLayer.forward[s>))=:P*QWYZ ,,]; +94>>, ')+/ , , ( (--m< ))-8  "  )+ +Gr-ru)NNFF)rArBrCrrr#r%rErRboolrrr= FloatTensorr;rFrGs@r,r5r5RsZ{ZsTzZ/3:>).!&3||3!& 3 3  t+ 3 147 3 $;3$;3-.3 u  %(9(95;L;L(L"MPT"TT U3r-r5ceZdZdedeffd Z ddejdejdzdedzdejdzdejdzd edzd e dzd e dzd ejdzd ejdzdejdzde eje ejejfdzffdZ xZS)ZambaMambaDecoderLayerrorzct|t|||_t |j |j |_||_y)N)rorzr7) r"r#rmambarr*r:r;rzrs r,r#zZambaMambaDecoderLayer.__init__s> $FiH +F,>,>FDWDWX"r-Nr.r=r causal_maskrr>rr position_idstransformer_hidden_statesr c  |} | || zn|}|j|}|j|||}d}| |z}|f}|r||fz }|r||fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r.rr)r;rH)r)r.r=rzrrIrr>rrrJrKrresidualrArBs r,r;zZambaMambaDecoderLayer.forwards>! :S9^M5 5dq ,,];  '()# !!=0  "  )+ +G  ) )Gr-) NNNNNFFNNN)rArBrCrrr#r%rErRrCrr=rDr;rFrGs@r,rFrFs,#{#s#7; $.2+/:>).!&26049=;||;!& t 3;: ;  t+ ; \\D( ;147; $;;$;;((4/;&&-;$)<<$#6; u  %(9(95;L;L(L"MPT"TT U;r-rFcleZdZdedej deffd Z ddejdejdzde dzd ejdzd ejdzd e dzd e dzd e dzdejdzdeejeejejfdzffdZxZS)ZambaHybridLayer shared_transflinearrHcLt|||_||_||_yru)r"r#rPrQ mamba_decoder)r)rPrQrHr+s r,r#zZambaHybridLayer.__init__s% * "r-Nr.r=rzrrIrr>rrr c |j|||||||| } | d} |r| d} |j| } |j|| ||||| } |r | d f| ddz} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )r=rzrrr>rrrr)rKrrr>rrr0N)rPrQrS) r)r.r=rzrrIrr>rr layer_outputsrKrAs r,r;zZambaHybridLayer.forwards>** #9&+/)+  %2!$4!  -a 0 $(KK0I$J!** &?)+/)+  *1-/@AMRSRTDUUMr-)NNNNNFFN)rArBrCr5rrrFr#r%rErrRrCrr=rDr;rFrGs@r,rOrOs #&@#"))#\r#7; $.2+/:>).!&26>||>!& t 3>: >  t+ > \\D( >147> $;>$;>((4/> u  %(9(95;L;L(L"MPT"TT U>r-rOcreZdZUeed<dZdZddgZdZdZ dZ dZ e jfdZxZS) ZambaPreTrainedModelromodelTr5rFrFc|jj}t| |t |t rt j|jd||jjdz}t j|j| ||jj|jjz|jjz}tj tj"|jj|t%j&|jj(t%j&|jj*z zt%j&|jj*zj-|jj.}|tj&tj0|  z}t j2|j4|tj6d|j8dztj:dddf}|j=|j>djA}t j2|jBtj&|jE|j|jFdt jH|jJyy)Nr)r7stdr)minrrr1)&roinitializer_ranger" _init_weightsrrinitnormal_rruniform_rr[r*rar%rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrr^r5rJr\rrrKrones_r) r)rrZ dt_init_stdrdtinv_dtrr+s r,r]z"ZambaPreTrainedModel._init_weights*skk++ f% fo . LL--CS A++33T9K MM&//+{ K![[55 8O8OOSWS^S^SlSllN 4;;44nE88DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F JJv**F 3 Q 5 5 9OPTVWPWXA1126AACA JJv||UYYq\%9%9&:N:NPVPePegi%j k JJvxx % /r-)rArBrCr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulr%no_gradr]rFrGs@r,rWrWsQ &*#57OP"3 NLU]]_!!r-rWc"eZdZdZdeffd Ze ddejdzdejdzdejdzde dzd ejdzd e dzd e dzd e dzd e dzdejdzde ezfdZxZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rocht|||j|_|j|_t j |j|j|j|_|j|_ g}d|_ t|jD]\}}t||}|dk(rt j|jj|jjd}|jt!t#||||jd|dd|di|_ |j|t j$||_|j(|_t+|j|j, |_d|_|j3y) N)rzrVFrz layers.(?!z\.)\d+.shared_transfzlayers.z.shared_transfr7)r"r# pad_token_id padding_idx vocab_sizer Embeddingr* embed_tokensrY_tied_weights_keys enumeraterFrrorkrOr5 ModuleListlayersrrr:final_layernormgradient_checkpointing post_init)r)rorlayer_id layer_typerHrQr+s r,r#zZambaModel.__init__Lsq  !.. ++LL):):F,>FDWDWX&+# r-N input_idsrrJr inputs_embedsrr>output_hidden_states return_dictrr c ||n|jj}||n|jj}||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||j|}|} tj|} |r|tjd| .tj| jd| j} || j!d}t#|j||| ||}|rd nd}|rd nd}t%|j&D]5\}}|r|| fz }|| | ||||||| }|d} |s'|d-||dfz }7|j)| } |r|| fz }|r|j*sd |_t-| |r|nd|| }| r|S|j/S) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rrWr)rorrrrrJr)rr>rrT)last_hidden_staterr. attentions)ror>rruse_return_dictr'rrrrr|r%rrr>rUrrr~rrrZrto_tuple)r)rrrJrrrr>rrrrr.r=rIall_hidden_statesall_self_attnsrzlayerrUoutputs r,r;zZambaModel.forwardisW2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK26!%)-,0#'26f<##d*f< t+f<&&- f< 147 f< ((4/ f<$;f< $;f<#Tkf<D[f<((4/f< ( (f<frrrlogits_to_keepr c ||n|jj}| | n|jj} | | n|jj} |j |||||||| | |  }|d}t | t r t| dn| }|j|dd|ddf}d}||j|||jfi| }| s|f|ddz}||f|zS|St|||j|j|jS)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrJrrrr>rrrrrlosslogitsrr.r)ror>rrrXrrslicer loss_functionrzrrr.r)r)rrrJrrrrr>rrrrrrBr. slice_indicesrrrs r,r;zZambaForCausalLM.forwardsLP2C1N-TXT_T_TqTq%9$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %4%%ffdooPPDY,F'+'7D7V# CV C%#33!//))   r-c |:t|j|jd|j|j}|jj | d<t ||f|||||||d| } | S)Nr)r3rUr)rrrrrJris_first_iteration)rRror>r3rUnum_logits_to_keepr"prepare_inputs_for_generation) r)rrrrrrJrrr model_inputsr+s r,rz.ZambaForCausalLM.prepare_inputs_for_generation2s  "5 Y__Q/tzz$++O$(;;#A#A w<   +)')%1    r-) NNNNNNNNNNNr)NNNNNTF)rArBrCr}rr#rr%rrErRrDrCrr=rr;rrFrGs@r,rrsj*,GH{.2.204:>26*.!%)-,0#'26-.O ##d*O  t+O &&- O 147 O ((4/ O   4'O $;O  $;O #TkO D[O ((4/O ell*O  ' 'O O h   r-ra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZfdZe ddej dzdejdzdej dzdedzdejdzdej dzd e dzd e dzd e dzd e dzd e e zfdZ xZS)ZambaForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyr+) r"r# num_labelsrvrXrrr*scorerr1s r,r#z'ZambaForSequenceClassification.__init__dsS   ++' YYv114??O  r-NrrrJrrrrr>rrr c | | n|jj} |j|||||||| |  } | d} |j| }||jd}n|jd}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d|tj||j |f}d}||j|j}|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:| j< S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrJrrrr>rrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r1rTz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rW regressionsingle_label_classificationmulti_label_classificationr)rorrXrr>rxr'r4rUr%int32rargmaxrrr+rA problem_typerr3longrrrrrrrrr.r)r)rrrJrrrrr>rrrtransformer_outputsr.rrplast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r,r;z&ZambaForSequenceClassification.forwardms*&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r-r)rArBrCr#rr%rrEr rDrCr=rr;rFrGs@r,rrUs.2.204(,26*.!%)-,0#'\ ##d*\  t+\ &&- \  \ ((4/ \   4'\ $;\  $;\ #Tk\ D[\  1 1\ \ r-r)rrrvrW)r)Crrbcollections.abcrtypingrr%rtorch.nnrrrr r^ activationsr cache_utilsr generationr integrations.hub_kernelsr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrutils.import_utilsrconfiguration_zambar get_loggerrArModulerrErrPrRrDrrrr)r5rFrOrWrvrr__all__rr-r,rs& $ AA&! )8/B9qqF&,9,   H %J299J* UU\\ U# U%,, Uc$c$Z% II% <<% % << % LL4' %  %%2C)RYYC)Lg]biig]V ryy  <<~B7BJE1EP!!?!!!!HL<%L<L<`~+_~B  g %9g  g T gr-