qi&ddlmZddlZddlmZddlmZddlmZm Z m Z m Z m Z ddl mZdd lmZej"eZd ZGd d eZGd de ZdZddZGddeZGdde ZGdde ZGdde ZgdZy))OptionalN)logging)LlamaAttentionLlamaForCausalLMLlamaForSequenceClassificationLlamaForTokenClassificationLlamaRotaryEmbedding)Phi3MLP) GlmConfigzTHUDM/glm-4-9bc eZdZy)GlmMLPN__name__ __module__ __qualname__U/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/glm/modular_glm.pyrr%rrc ReZdZe d dedzdeddedzdedeffdZ y) GlmRotaryEmbeddingNconfigdevicez torch.deviceseq_lenreturnz torch.Tensorcn|jd}|jjdd}t|ddxs|j|jz}t ||z}d}d|t jd|dt jj|t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrr)dtype)rr#) rope_parametersgetgetattr hidden_sizenum_attention_headsinttorcharangeint64tofloat) rrrbaser!r"dimattention_factorinv_freqs rcompute_default_rope_parametersz2GlmRotaryEmbedding.compute_default_rope_parameters*s&%%l3 & 6 6 : :;RTW X6:t4h8J8JfNhNh8h(223 U\\!S!5;;?BB&X]XcXcBdgjj k )))r)NNN) rrr staticmethodrrr)tupler.r3rrrrr)sY#'+/"*D *(*t* ~u$ % **rrc||ddddf}|ddddf}tj| |fdjdS) z*Rotates half the hidden dims of the input..rNrr r0)r*stackflatten)xx1x2s r rotate_halfr?KsJ 319B 319B ;;Ryb ) 1 1" 55rc|j|}|j|}|dd|jddzfjdd}|dd|jddzfjdd}|jd}|dd|f|d|df}}|dd|f|d|df} }||zt||zz} ||zt||zz} t j | |gd} t j | | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr7rr8) unsqueezeshaperepeat_interleaver?r*cat) qkcossin unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds rapply_rotary_pos_embrQRsD$ -- &C -- &C c'SYYr]a''' ( : :1" : EC c'SYYr]a''' ( : :1" : EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{{51C78Gs{{51C78Gii&)r2Gii&)r2G G rc0eZdZddededzffd ZxZS) GlmAttentionNr layer_idxct|||tj|j|j z|j d|_y)NF)bias)super__init__nnLinearr(r"r'o_proj)selfrrT __class__s rrXzGlmAttention.__init__{s< +ii : :T]] JFL^L^ejk r)N)rrrrr)rX __classcell__)r]s@rrSrSzs#lylS4ZllrrSc eZdZy)GlmForCausalLMNrrrrr`r`rrr`c eZdZy)GlmForSequenceClassificationNrrrrrbrbrrrbc eZdZy)GlmForTokenClassificationNrrrrrdrdrrrd)GlmPreTrainedModelGlmModelr`rbrd)r )typingrr*torch.nnrYutilsrllama.modeling_llamarrr r r phi3.modeling_phi3r configuration_glmr get_loggerrlogger_CHECKPOINT_FOR_DOCrrr?rQrSr`rbrd__all__rrrrqs )(   H %& W *-*D6%Pl>l  %  #A  ;  r