qiy`ddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*GddejVZ,GddejVZ-dej\de/dej\fdZ0 d4dejVdej\dej\dej\d ej\dzd!e1d"e1d#ee!fd$Z2d%Z3d5d&Z4ee4Gd'd(ejVZ5Gd)d*ejVZ6Gd+d,eZ7e"Gd-d.eZ8e"Gd/d0e8Z9e"Gd1d2e8e Z:gd3Z;y)6)Callable)OptionalN)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernelized_func)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)capture_outputs) Cohere2Configc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)Cohere2RotaryEmbeddinginv_freqNconfigct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jdy)N rope_typedefaultrF) persistentoriginal_inv_freq) super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr rope_parametersr"compute_default_rope_parametersrattention_scalingregister_bufferclone)selfr device rope_init_fnr __class__s ^/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/cohere2/modeling_cohere2.pyr'zCohere2RotaryEmbedding.__init__.s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr1z torch.deviceseq_lenreturnz torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rdtype)r1r=) r+getattr hidden_sizenum_attention_headstorcharangeint64tofloat)r r1r6basedimattention_factorrs r4r,z6Cohere2RotaryEmbedding.compute_default_rope_parameters>s&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r5c|jddddfjj|jddd}|dddddfj}t |j j tr/|j j dk7r|j j nd}t|d5|j|jzjdd}tj|dd }|j|jz}|j|jz} dddj|j  j|j fS#1swYAxYw) NrrmpscpuF) device_typeenabledr;rGr<)rrEexpandshape isinstancer1typestrr transposerArepeat_interleavecosr-sinrDr=) r0x position_idsinv_freq_expandedposition_ids_expandedrMfreqsembrWrXs r4forwardzCohere2RotaryEmbedding.forward\s@!MM$4-8>>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E))%;C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s =BFF N)NNN)__name__ __module__ __qualname__rATensor__annotations__rr' staticmethodrinttuplerEr,no_gradrr_ __classcell__r3s@r4rr+sllV}V '++/"*$*(*t* ~u$ % **:U]]_ <