qidZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z dd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&e$jNe(Z)GddejTZ+ d0dejXdejZdejZdejZdejZdzde.de.fdZ/GddejXZ0Gdd eZ1e"Gd!d"eZ2Gd#d$e2Z3e"Gd%d&e2Z4Gd'd(e2eZ5e"d)*Gd+d,e2Z6e"Gd-d.e2Z7gd/Z8y)1zPyTorch OPT model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging) OPTConfigcxeZdZdZdedeffd Z d dejdedejdzffd ZxZ S) OPTLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcNd|_t| ||jz|yN)offsetsuper__init__)selfrr __class__s V/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/opt/modeling_opt.pyr&z&OPTLearnedPositionalEmbedding.__init__0s$  $++5}ENattention_maskpast_key_values_length position_idsc|8tj|d}||zdz j}|dd|df}t|||j zS)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)torchcumsumlongr%forwardr$)r'r+r,r-r(s r)r4z%OPTLearnedPositionalEmbedding.forward6s^   <<A>L(>9A=CCEL'+A+B(BCLw|dkk9::r*)rN) __name__ __module__ __qualname____doc__intr&r1 LongTensorr4 __classcell__r(s@r)rr+s]FsF3F'(04 ;((;!$;&&- ;;r*rmodulequerykeyvaluer+scalingdropoutc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)N)r0dtypeptrainingrr#) r1matmul transposer functionalsoftmaxfloat32torFrBrI contiguous) r=r>r?r@r+rArBkwargs attn_weights attn_outputs r)eager_attention_forwardrTHs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r*ceZdZdZ d dededzffd Z ddejde dzdejdzd e d ejdzd e ejejdze dzff d Z xZ S) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfig layer_idxc t|||_|j|_|j |_|j|_|j|_ ||_ |-tjd|jjd|j|j z|_d|_|j|j z|jk7r&t#d|jd|j d|jdz|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_y) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r%r&rW hidden_size embed_dimnum_attention_heads num_headsattention_dropoutrB enable_biasrXlogger warning_oncer(r5head_dim is_causal ValueErrorrArLineark_projv_projq_projout_proj)r'rWrXrQr(s r)r&zOPTAttention.__init__bs  ++33// !--"      !8!8 9:,,  $..8  MMDNN *t~~ =MdnnM]$T^^$4B8 }}d* iiTEUEUV iiTEUEUV iiTEUEUV  $..$..tGWGWX r* hidden_statespast_key_valuesr+output_attentionscache_positionreturnc j|j\}}} |j||jz} | j|d|j|j j dd} |j|} |j|} | j|d|j|j j dd} | j|d|j|j j dd} |#|j| | |jd|i\} } tj|jjt} | || | | |f|j sdn |j"dd|\}}|j%||dj'}|j)|}|sd}||fS) z#Input shape: Batch x Time x ChannelrDrr#Nrog?)rBrA)sizerjrAviewr_rdrKrhriupdaterXr get_interfacerW_attn_implementationrTrIrBreshaperPrk)r'rlrmr+rnrorQbsztgt_len_ query_states key_states value_statesattention_interfacerSrRs r)r4zOPTAttention.forwards(,,.Wa{{=1DLL@ #((b$..$--PZZ[\^_` [[/ {{=1 __S"dnndmmLVVWXZ[\ #((b$..$--PZZ[\^_`  &'6'='=L$..;K^:\( $J )@(M(M KK , ,.E) %8      % $}}C$,, %  % ! \"))#w;FFH mmK0  LL((r*N)NNFN)r5r6r7r8rr9r&r1Tensorr booltupler4r;r<s@r)rVrV_sG !%!Y!Y:!YL)-.2"'.2 4)||4)4) t+ 4)  4)  t+ 4) u||U\\D0%$,> ?4)r*rVc:eZdZddededzffd Z ddejdejdzdedzde dzd e dzd ejdzd ejdzd e e d e eje ejejfdzffdZxZS)OPTDecoderLayerNrWrXct||j|_t |||_|j |_|j|_t|j|_ tj|j|j|_tj|j|j |j"|_tj|j |j|j"|_tj|j|j|_y)N)rWrXelementwise_affinerZ)r%r&r\r]rV self_attndo_layer_norm_beforerBr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normrgffn_dimrafc1fc2final_layer_norm)r'rWrXr(s r)r&zOPTDecoderLayer.__init__s ++%VyI$*$?$?!~~ #F$>$>?$&LL NNv/S/S% !99T^^V^^&BTBTU99V^^T^^&BTBTU " T^^PVPtPt ur*rlr+rmrn use_cacher-rorQrpc |} |jr|j|}|jd||||||d|\}} tjj ||j |j }| |z}|js|j|}|j} |jd|jd}|} |jr|j|}|j|}|j|}|j|}tjj ||j |j }| |zj| }|js|j|}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence.. )rlrmr-r+rnrorGrD)rrrrrLrBrIshaperxrsrrrrrt) r'rlr+rmrnrr-rorQresidualself_attn_weightshidden_states_shapeoutputss r)r4zOPTDecoderLayer.forwards6!  $ $ 55mDM,:4>>, '+%)/) , , ( ( --mt||VZVcVc-d  =0 (( 55mDM,11%--b-2D2DR2HI    $ $ 11-@M/ **=9 /  --mt||VZVcVc-d !M1778KL (( 11-@M "  )+ +Gr*r)NNFFNN)r5r6r7rr9r&r1rr rr:rrr FloatTensorr4r;r<s@r)rrsvyvS4Zv(/3(,).!&04.2L||L t+L L  $; L $; L&&-L t+L-.L u  %(9(95;L;L(L"MPT"TT ULr*rc:eZdZUeed<dZdZdgZdZdZ dZ dZ dZ y)OPTPreTrainedModelrWmodelTrN) r5r6r7r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrr*r)rrs: &*#*+"&N!r*rc,eZdZdZdeffd Ze ddejdzdejdzde dzdejdzd e dzd e dzd e dzd e dzd ejdzdejdzde edeezfdZxZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rWc 8t|||j|_|j|_|j|_|j |_|j|_tj|j|j|j |_ t|j |j|_|j|jk7r2tj |j|jd|_nd|_|j|jk7r2tj |j|jd|_nd|_|j&r=|j(s1tj*|j|j,|_nd|_tj0t3|j4Dcgc]}t7||c}|_d|_|j=ycc}w)NFrZr)rX)r%r&rB layerdrop pad_token_id padding_idxmax_position_embeddingsmax_target_positions vocab_sizer Embeddingword_embed_proj_dim embed_tokensrr\embed_positionsrg project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing post_init)r'rWir(s r)r&zOPTDecoder.__init__2s  ~~ ))!..$*$B$B! ++LL):):F