qiyddlmZddlmZmZmZddlZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZddlmZddlmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8e/jre:Z;dZ<eddTdZ=dej|de?dej|fdZ@ dUd ejd!ej|d"ej|d#ej|d$ej|dzd%eBd&eBd'e)e+fd(ZCee=Gd)d*ejZDGd+d,ZEd-ej|d.e?fd/ZFd0ZGd1ZHd2ZIGd3d4ejZJGd5d6ejjZKGd7d8ejZLGd9d:ejZMGd;dejZOGd?d@ejZPGdAdBedCDZQedEGdFdGejZRGdHdIeZSe,GdJdKe'ZTe,GdLdMeTZU dVdNej|eVej|zdzdOe?dzd$ej|dzdej|e?zfdPZWe,GdQdReTeZXgdSZYy)W)Callable)AnyOptional TypedDictN)nn) functional)ACT2FN)initialization)Cache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)lazy_load_kernel)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging)maybe_autocastmerge_with_config_defaults)resolve_internal_import)capture_outputs)GraniteMoeHybridConfigc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s p/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py rotate_halfr33sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''rotary_pos_embc|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer3)qkcossin unsqueeze_dimq_embedk_embeds r2apply_rotary_pos_embr?:sY& -- &C -- &C3w;q>C/0G3w;q>C/0G G r4 hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r%N)r,expandreshape)r@rAbatchnum_key_value_headsslenhead_dims r2 repeat_kvrJTso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr4modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr)r r()r+dtype)ptrainingr%)rJnum_key_value_groupsr-matmul transposerrsoftmaxfloat32torTrQrV contiguous) rKrLrMrNrOrPrQrR key_states value_states attn_weights attn_outputs r2eager_attention_forwardrb`s3 ; ;$>&B\B\$\!22 !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r4Nr@rOpast_key_valuescache_positionposition_embeddingsrRrBc |jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} ||\} } t| | | | \} } |%d|i}|j| | |j|\} } tj|jjt}||| | | |f|jsdn |j |j"d|\}}|j$g|dj'}|j)|}||fS)Nr(r%r)r|)rQrP)r,rIrtviewrYrurvr?updaterfr get_interfacere_attn_implementationrbrVrprPrEr]rw)ryr@rOr{r|r}rR input_shape hidden_shape query_statesr^r_r:r; cache_kwargsattention_interfacerar`s r2forwardz!GraniteMoeHybridAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  **HC';L*VY[^'_ $L*  &,n=L'6'='=j,X\XfXfht'u $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r4NNN)__name__ __module__ __qualname____doc__r&intrkr-Tensorr LongTensortuplerrr __classcell__rzs@r2rdrdysG 5 # 6)-26HL ))||)) t+)) )) ((4/ )) #5<<#=>E ))+,)) u||U\\) *))r4rdc :eZdZdZdZej dfdefdZdZ dZ ddejd ejd e d e eefdzd eejejff d Zdej$fdZdejd e d ee e ffdZdd e dzd e fdZy) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNrec ,|j|_d|_|j}|j}g|_g|_g|_t|jD]*}|j|dk(r|xjtj||j|jzd|jz|zz|||gz c_|xj tj||j|j|||gz c_|xjtj gg|z|gz c_|xj tj gg|z|gz c_|j j#|-t|jDcgc]}tj gg|z|c}|_t|jDcgc]}tj gg|z|c}|_ycc}wcc}w)NFmambar)devicerTr)layers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statestransformer_layersrangenum_hidden_layersr-zeros mamba_expandrmmamba_n_groups mamba_n_heads mamba_d_headtensorappend key_cache value_cache) ryre batch_sizerTrconv_kernel_sizessm_state_sizei_s r2rkz)HybridMambaAttentionDynamicCache.__init__s!'!9!9"'!..--"$v//0 2A%%a(G3  KK",,v/A/AAAH]H]D]`nDnn(%# % KK",,++&%#  $   U\\2$2CF%S$TT ELL" 1B6$R#SS''..q11 24SXX^XpXpRqrQ%,,tj'8HrTYZ`ZrZrTstqELL" ):6Jtsts "H !"Hc,t|jSN)lenrrys r2__len__z(HybridMambaAttentionDynamicCache.__len__s4>>""r4c>|j||j|fSr)rrryrfs r2 __getitem__z,HybridMambaAttentionDynamicCache.__getitem__s!~~i($*:*:9*EEEr4r^r_rfrrBc|j|jddk(r||j|<||j|<nftj|j||gd|j|<tj|j||gd|j|<|j||j|fS)Nr(rr)r*)rr,rr-r.)ryr^r_rfrs r2rz'HybridMambaAttentionDynamicCache.updates >>) $ * *2 .! 3(2DNN9 %*6D  Y '(- 4>>)3Lj2Y_`(aDNN9 %*/))T5E5Ei5PR^4_ef*gD  Y '~~i($*:*:9*EEEr4beam_idxc|jdkDrvtt|jD]S}|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<Vyy)zDReorders the cache for beam search, given the selected beam indices.rN) get_seq_lengthrrrr index_selectr\rrr)ryrrfrs r2 reorder_cachez.HybridMambaAttentionDynamicCache.reorder_cache s[    1 $"3t~~#67 m  299,0NN9,E,R,RSTV^VaVabhVi,jy))))4;;.2.>.>y.I.V.VWXZbZeZeflZm.n  +)))4;;.2.>.>y.I.V.VWXZbZeZeflZm.n  +3::-1__Y-G-T-TUVX`XcXcdjXk-l * m %r4r|cTd}|jd}|j||z}||fS)zDReturn the length and offset of the cache, used to generate the maskr)r,r)ryr|rf kv_offset query_length kv_lengths r2get_mask_sizesz/HybridMambaAttentionDynamicCache.get_mask_sizess7 %++A. '' 2\A )##r4c||jvr|jdn|}t|j|ks|j|jddk(ry|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rr()rrrr,rs r2rz/HybridMambaAttentionDynamicCache.get_seq_length"sn3<4CZCZ2ZD++A.`i t~~ ) +t~~i/H/N/Nr/RVW/W~~i(..r22r4r)r)rrrris_compileabler-float16r&rkrrrrdictstrrrrrrrrr4r2rrs NIN_c$u5$uL#F/3 FLLFllF F 38nt+ F u||U\\) * F" me&6&6 m$U\\$c$eTWY\T\o$3d 333r4r input_tensorpad_sizect|jdk(r ddddd|ddfnddd|ddf}tjjj ||ddS)z Padding x tensor with `pad_size` on the seq_len dim (dim=1) Assumes that we only have tensors of either size 4 or 3 rconstant)moderN)rr,r-rrpad)rr pad_shapes r2pad_tensor_by_sizer.sf 47|7I7I3Ja3OAq!Q!Q/VWYZ\]_gijlmUnI 88   " "<ST " UUr4ct||}t|jdk(r.|j|jdd||jdS|j|jdd||jd|jdS)z Padding input_tensor with `pad_size` on the seq_len dim (dim=1) and simultaneously splitting it into chunk sequences. Assumes that we only have tensors of either size 4 or 3 r rr(r))rrr,rE)rr chunk_sizes r2reshape_into_chunksr9s&lH=L <  !###L$6$6q$92z