qi<dZddlZddlZddlmZddlmZmZmZmZddlm Z ddl m Z m Z ddlmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZmZddlm Z ejBe"Z#d'dZ$GddejJZ&GddejJZ'GddeZ(eGddeZ)eGdde)Z*edGdde)eZ+edGd d!e)Z,eGd"d#e)Z-eGd$d%e)Z.gd&Z/y)(zPyTorch MPT model.N)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) MptConfigcRtjd|z dtj|jddd|}dt j t j |z}tjd|dztj|j}|||z z}dtjd|z }|jd|dd}||k7r9tj|ddddddf|ddddddfgdddd|df}||z}|jdS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedeviceg?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopess V/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensorr3*s LL_,au{{6 R W WXY[\^_ap qE $))I*> ?? <</!35;;v V \ \ ^D >$88 9D 599Q% %F [[0!Q 7Fy(vaAsl3VAssCK5HIqQRSU_V_U_adRde FNE == c eZdZdZd dededzffd Z d dejdejde dzd ejdzd ejdzf d Z xZ S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. Nconfig layer_idxct||j|_|j|_|j|_|j|jz|_|jj|_|j4dtj|j|jz z |_|jj|_ |jj|_ tj|jd|jzd|_tj|j|jd|_||_y)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler#sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr8)selfr7r8 __class__s r2r=zMptAttention.__init__Fs !--~~ $00((DLL8 #//==    %!"TYYt/?/?$,,/N%O!OD $00;;**33 IId..D4D4D0D5Q  $"2"2D4D4D5Q "r4 hidden_states position_biaspast_key_valuesattention_maskcache_positioncp|jdd\}}|j|}|jr(|j|j |j}|j dd\} } } | j |||j |jjdd} | j |||j |jjdd} | j |||j |jjdd} |%d|i} |j| | |j| \} } tj| | jdd|jz} ||n||jz}|t|jdk7r!t!d t|j| jd}t#d |j%d|z }t#d |j%d|z }|dd|d|df}| |z} |9| j'|tj(| j*j,} t.j0j3| j5dj7| j*}t.j0j9||j:|j< }tj|| }|j?d dddjAjC||d}|jE|}||fS) Nr)minmaxr rrrRz6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperJrHclampchunkreshaper?rB transposeupdater8rmatmulrDget_seq_lengthlen ValueErrorrUsize masked_fillfinforrTrrsoftmaxr'todropoutrGrZpermute contiguousr"rK)rLrNrOrPrQrR batch_size seq_length mixed_qkv query_states key_states value_states cache_kwargsattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputs r2forwardzMptAttention.forwardVs"/!4!4Ra!8 JIIm, ==!T]]N NI1:1J. j,#++J DLLRVR_R_`jjklnop '' J dmm\ffghjkl #++J DLLRVR_R_`jjklnop  &,n=L'6'='=j,X\XfXfht'u $J  << j6J6J2r6RSVZVhVhh%4%rrIup_projGELUact down_projrCrFhidden_dropout)rLr7r>rMs r2r=zMptMLP.__init__sm (( yya+oEJ 77v.1{?KeL$00;;r4rNresidualreturnc|j|j|}|j|}tj||j |j }||z}|S)NrX)rrrFrjrrZ)rLrNrintermediate_outputoutputs r2r|zMptMLP.forwardsWm!<= "nn];.$2E2EPTP]P]^(" r4) r~rrrr=rrr|rrs@r2rrs5rlayer_norm_epsilonnorm_1r;r?r+r6attnnorm_2rffnrCrF dropout_raterDropoutresid_attn_dropout)rLr7r8r>rMs r2r=zMptBlock.__init__s ((  1J1JK   3  1J1JK  &>"..99"$**T->->"?r4rNrOrQ layer_past use_cacheoutput_attentionsrRc|j|}|} |j|||||\} } |j| | z}|j|}|} |j || } | | fS)N)rOrQrPrR)rrrrr) rLrNrOrQrrrrRlayernorm_outputr attn_outputsryrs r2r|zMptBlock.forwards ;;}5 &*YY ')&) &/& " l// =H ;;}5!*H5|##r4r})NFFN) r~rrrrr=rrr boolr|rrs@r2rrs@y@S4Z@2$("'.2"$||"$||"$ "$ DL "$  "$ "$ t+"$r4rc&eZdZUeed<dZdZdgZy)MptPreTrainedModelr7 transformerTrN)r~rrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr4r2rrs %&*## r4rcJeZdZdeffd ZdZddZdejfdZ e ddejdzd e dzd ejdzd ejdzd e dzd e dzde dzde dzdejdzdeejdfezfdZxZS)MptModelr7c t|||j|_|j|_t j |j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_d|j _d|_|j'ycc}w)N)r8rF)r<r=r>r?r+r Embedding vocab_sizewte ModuleListrangen_layersrblocksrrnorm_fr;gradient_checkpointing post_init)rLr7irMs r2r=zMptModel.__init__s  !--<< 1 143C3CDmmERXRaRaLb$cqXf%B$cd   0 0f6O6OP  &+# %dsC7c|jSr}r)rLs r2get_input_embeddingszMptModel.get_input_embeddingss xxr4Nct||||Sr})r3)rLr+r,r-rs r2r3zMptModel.build_mpt_alibi_tensors%i.RXYYr4new_embeddingsc||_yr}rrLrs r2set_input_embeddingszMptModel.set_input_embeddingss !r4 input_idsrPrQ inputs_embedsrroutput_hidden_states return_dictrRr.c v||n|jj}||n|jj}||n|jj}||n|jj}| | t d||j \} } n||j \} } } n t d|jr%|jr|rtjdd}||j|}|r|t|j}|}|rdnd}|rdnd}|j|j|jj|j }||j#nd }| %t%j&||| z|j } t)|j||| | j+t$j,}|j.D]*}|r||fz}|||||||| }|d }|s"||d fz},|j1|}|r||fz}|st3d ||||fDSt5||||S) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r7rrr)r7rrQrRrP)rrQrrrOrRrc3&K|] }|| ywr}r).0vs r2 z#MptModel.forward..isghgts)last_hidden_staterPrN attentions)r7rrruse_return_dictrdr[rrZlogger warning_oncerr r3r+r@rrbrr r rirrrtupler)rLrrPrQrrrrrrRkwargsrmrn_rNall_self_attentionsall_hidden_statesr.past_key_values_length causal_maskblockoutputss r2r|zMptModel.forwards62C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "%.__ "J  &(5(;(; %J ATU U  & &4==##p"   HHY/M 0*$++>O% $5b4"6BD++DNNDKK(KTaThThN);;'))+  "UZZ. [[ JE#$58H$H!**#"3#-G$AJM &9WQZM&I#! J& M2  1]4D D )?rlm_headrrLr7rMs r2r=zMptForCausalLM.__init__~sI  #F+yy!3!3V5F5FUS  r4rc||_yr})rrs r2set_output_embeddingsz$MptForCausalLM.set_output_embeddingss % r4NrrPrQrlabelsrrrrrRlogits_to_keeprc | | n|jj} |j|||||||| |  } | d}t| tr t | dn| }|j |dd|ddf}d}|*|jd|||jjd| }| s|f| ddz}||f|zS|St||| j| j| jS)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)rPrQrrrrrrRr)logitsrrrlossrrPrNrr) r7rr isinstancerslicer loss_functionrrrPrNr)rLrrPrQrrrrrrrRrrtransformer_outputsrN slice_indicesrrrs r2r|zMptForCausalLM.forwardsB&1%SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopDY!4QR!88F)-)9TGf$ Ev E0/??-;;*55   r4) NNNNNNNNNNr)r~rr_tied_weights_keysrr=rrrrrr rrrrr|rrs@r2rrusC+,DEy&ELL&.2(,.2-1&*!%)-,0#'.2-.A ##d*A A  t+ A ||d* A  t# A $;A  $;A #TkA D[A  t+A ell*A  u|| @ @A A r4ra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c8eZdZdeffd Zdej fdZe ddejdzde dzdej dzd ej dzd ej dzd e dzd e dzd e dzde dzde ej e zfdZxZS)MptForSequenceClassificationr7ct|||j|_t||_t j |j|jd|_|jyr) r<r= num_labelsrrrrIr>scorerrs r2r=z%MptForSequenceClassification.__init__sV   ++#F+YYv1163D3D5Q  r4rc||_yr})rrs r2rz2MptForSequenceClassification.set_output_embeddingss # r4NrrPrQrrrrrrrc r| | n|jj} |j|||||||| } | d} |j| } ||jd}n|jd}|jj |dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj|| j |f}d}|^|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}nc|||}nY|jj"d k(rt1}|||}n,|jj"d k(rt3}|||}| s|f| ddz}||f|zS|St5||| j6| j8| j: S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrPrQrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.rV)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r7rrrr[ pad_token_idrdrirrr!r argmaxrrrMr~ problem_typerrlongrrr*rrrrPrNr)rLrrPrQrrrrrrrrrNrrmlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r2r|z$MptForSequenceClassification.forwards>&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- v6))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r4r)r~rrrr=rrrrrr rrrr|rrs@r2rrsy$ELL$.2(,.2-1&*!%)-,0#'e ##d*e e  t+ e ||d* e  t# e $;e  $;e #Tke D[e  u|| ? ?e e r4rceZdZdeffd Ze ddejdzdedzdejdzdejdzdejdzd e dzd e dzd e dzd e dzd e eje zfdZ xZS)MptForTokenClassificationr7ct|||j|_t||_t |dr|j |j }n't |dr|j |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropoutrg?)r<r=rrrhasattrrrrrrjrIr> classifierr)rLr7rrMs r2r=z"MptForTokenClassification.__init__Ts   ++#F+ 6/ 0V5N5N5Z!'!:!:  V- .63H3H3T!'!6!6 !$ zz"45 ))F$6$68I8IJ r4NrrPrQrrrrrrrc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|l|j | j }|j\}}t}|| j||z|j|j||z}| s| f| ddz}||f|zS|St|| | j| jS)rNrrr)rrrNr)r7rrrjrrirr[rr"rrrNr)rLrrPrQrrrrrrdeprecated_argumentsrrNrrrmrnrrs r2r|z!MptForTokenClassification.forwardes+>&1% qa_outputsrrs r2r=z MptForQuestionAnswering.__init__sA  #F+))F$6$6: r4NrrQrstart_positions end_positionsrrrrc  "||n|jj}|j||||||} | d} |j| } | j dd\} }| j dj } |j dj }d}||t|jdkDr|j d}t|jdkDr|j d}| jd}|jd|}|jd|}t|}|| |}|||}||zdz }|s| |f| ddz}||f|zS|St|| || j| j S) rN)rQrrrrrrrVr) ignore_indexr)r start_logits end_logitsrNr)r7rrrsplitr*rlrcrer\rrrNr)rLrrQrrrrrrrrsequence_outputrr r! total_loss ignored_indexr start_lossend_lossrs r2r|zMptForQuestionAnswering.forwards4&1%r5s_ LL$.)/9.,(   H %.F)299F)RRYY*7$)7$t%%%P !P P f P 'P  P f  s #5s  s lU 2U U pP 0P P f r4