qiadZddlmZddlZddlZddlmZddlmZm Z m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ddl1m2Z2e)jfe4Z5de6de6dejnfdZ8de6de6dejnfdZ9GddejtZ; dAdejtdejnd ejnd!ejnd"ejndzd#eGd)d*ejtZ?Gd+d,eZ@Gd-d.ejtZAe(Gd/d0e!ZBe(Gd1d2eBZCe(d34Gd5d6eBZDe(d74Gd8d9eBZEe(Gd:d;eBZFe(Gd<d=eBZGe(Gd>d?eBZHgd@ZIy)Bz PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) )CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)get_activation)PreTrainedConfig)is_deepspeed_zero3_enabled)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)deprecate_kwarg)can_return_tuplemerge_with_config_defaults)capture_outputs)DistilBertConfign_posdimoutctrdddl}|jj|d5tj j dk(rt|||cdddS dddyt|||S#1swYyxYw)Nr) modifier_rankr"r#r$)r deepspeedzeroGatheredParameterstorch distributedget_rank_create_sinusoidal_embeddings)r"r#r$r(s d/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/distilbert/modeling_distilbert.pycreate_sinusoidal_embeddingsr0?s!# ^^ . .s! . D T  ))+q045csS T T0 T T-5csKK  T Ts /A==Bctjt|Dcgc];}t|Dcgc]$}|tjdd|dzz|z z &c}=c}}}d|_t j tj|dddddf|dddddf<t j tj|dddddf|dddddf<|j|Scc}wcc}}w)Ni'Frr ) nparrayrangepower requires_gradr+ FloatTensorsincosdetach_)r"r#r$posj position_encs r/r.r.Js88hmnshtuadQVWZQ[\AcBHHUAaL34F$GG\uvLC$$RVVLADqD,A%BCC14a4L$$RVVLADqD,A%BCC14a4LKKM J ]usC< )C7C< 7C< c eZdZdeffd Zeddd d dejdejdzd ejdzd ejfd Z xZ S) Embeddingsconfigc t|tj|j|j |j |_tj|j|j |_ tj|j d|_ tj|j|_ |jdtj|jj!ddy)N) padding_idx-q=eps position_idsr F) persistent)super__init__r Embedding vocab_sizer# pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutregister_bufferr+arangeexpandselfrA __class__s r/rLzEmbeddings.__init__Ts !||F,=,=vzzW]WjWjk#%<<0N0NPVPZPZ#[ fjje<zz&..1   ELL)G)GHOOPWXej  input_embedsz5.6.0 inputs_embeds)versionnew_nameN input_idsrGreturnc||j|}|jd}|rt|dr|jddd|f}nPt j |tj |j}|jdj|}|j|}||z}|j|}|j|}|S)Nr rG)dtypedevicer) rPsizehasattrrGr+rWlongre unsqueeze expand_asrRrSrU)rZrar^rG seq_lengthrR embeddingss r/forwardzEmbeddings.forward_s   00;M"''*  t^,#00KZK@ $||JejjQZQaQab +55a8BB9M "66|D"%88 ^^J/ \\*- r\)NN) __name__ __module__ __qualname__r rLrr+Tensor LongTensorrm __classcell__r[s@r/r@r@Ssx  /  ^WO.204 <<||d*&&-   Pr\r@modulequerykeyvalueattention_maskscalingrUkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)NrIr2rr#)ptrainingr ) rfr+matmul transposer functionalsoftmaxrUr contiguous) rurvrwrxryrzrUr{ attn_weights attn_outputs r/eager_attention_forwardr~s**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r\c eZdZdeffd Z d dej dejdzdee de ej fdZ xZ S) DistilBertSelfAttentionrAc4t|||_|j|_|j|_|j|jz|_|j dz|_|j|jzdk7r&td|jd|jdtj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_d|_y) Nr}rzself.n_heads: z must divide self.dim:  evenly in_features out_featuresrF)rKrLrAn_headsr#attention_head_sizerz ValueErrorrLinearq_link_linv_linout_linrTattention_dropoutrU is_causalrYs r/rLz DistilBertSelfAttention.__init__s  ~~ ::#'88t||#; //5  88dll "a '~dll^;RSWS[S[R\\cde eYY6::FJJO YY6::FJJO YY6::FJJO yyVZZfjjQ zzF$<$<= r\N hidden_statesryr{rbc |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd}tj|jjt} | |||||f|jsdn|jj|jd|\} } | j g|dj#} |j%| } | | fS)NrIr r2)rUrz)shaperrviewrrrr get_interfacerA_attn_implementationrrrUrrzreshaperr) rZrryr{ input_shape hidden_shape query_layer key_layer value_layerattention_interfacerrs r/rmzDistilBertSelfAttention.forwardsV $))#2. CCbC$*B*BC 5djj/44lCMMaQRS 2DJJ}-22LAKKAqQ 4djj/44lCMMaQRS (?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH ll;/ L((r\N) rnrorpr rLr+rqr8rrtuplermrsrts@r/rrs^/248)||)))D0)+, ) u||  )r\rceZdZdeffd Zdej dej fdZdej dej fdZxZ S)FFNrAct|tj|j|_|j |_d|_tj|j|j|_ tj|j|j|_ t|j|_ y)Nrr r)rKrLrrTrUchunk_size_feed_forward seq_len_dimrr# hidden_dimlin1lin2r activationrYs r/rLz FFN.__init__s zzFNN3 '-'E'E$II&**6CTCTU II&*;*;&**U ():):;r\inputrbcZt|j|j|j|Sr)rff_chunkrr)rZrs r/rmz FFN.forwards%(8T8TVZVfVfhmnnr\c|j|}|j|}|j|}|j|}|Sr)rrrrU)rZrxs r/rz FFN.ff_chunks= IIe  OOA  IIaL LLOr\) rnrorpr rLr+rqrmrrsrts@r/rrsI</<oU\\oelloellu||r\rc eZdZdeffd Z d dej dej dzdeede ej dffd Z xZ S) TransformerBlockrAc~t||j|jzdk7r&t d|jd|jdt ||_tj|jd|_ t||_ tj|jd|_ y)Nrzconfig.n_heads z must divide config.dim rrD)normalized_shaperF) rKrLr#rrr attentionrrS sa_layer_normrffnoutput_layer_normrYs r/rLzTransformerBlock.__init__s  :: &! +v~~.>>VW]WaWaVbbijk k08\\6::5Qv;!#vzzu!Ur\Nrryr{rb.c |j|fd|i|\}}|j||z}|j|}|j||z}|S)Nry)rrrr)rZrryr{attention_output_ ffn_outputs r/rmzTransformerBlock.forwardsu-dnn  )  !  --.>.NOXX./ ++J9I,IJ r\r) rnrorpr rLr+rqrrrrmrsrts@r/rrsc V/ V /3|| t++,  u||S ! r\rc teZdZdeffd Z d dej dej dzdeede fdZ xZ S) TransformerrAct||j|_tjt |jDcgc] }t |c}|_d|_ycc}w)NF) rKrLn_layersr ModuleListr5rlayergradient_checkpointing)rZrArr[s r/rLzTransformer.__init__ sQ  ]]eFOOF\#]$4V$<#]^ &+#$^sA-Nrryr{rbc P|jD] }|||fi|}t|S)N)last_hidden_state)rr)rZrryr{ layer_modules r/rmzTransformer.forwards> !JJ L(M ??r\r) rnrorpr rLr+rqrrrrmrsrts@r/rr sX,/,/3 @|| @ t+ @+, @  @r\rceZdZUeed<dZdZdZdZdZ dZ e e dZ ejdej"ffd ZxZS)DistilBertPreTrainedModelrA distilbertT)r attentionsruc ,t||t|tr|jj rt j|jjt|jj|jjtj|jjt j|jtj |jj"dj%dyy)zInitialize the weights.rIrHN)rK _init_weights isinstancer@rAsinusoidal_pos_embdsinitcopy_rRweightr0rQr#r+ empty_likerGrWrrX)rZrur[s r/rz'DistilBertPreTrainedModel._init_weights0s f% fj ){{// ..550 ;; (()C)C)J)JK JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h *r\)rnrorpr!__annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsr+no_gradrModulerrsrts@r/rr"sd $&*#N"&)- U]]_ iBII i ir\rcjeZdZdeffd Zdej fdZdefdZ dej fdZ dej fd Z e e e dd ej d zd ej d zd ej d zdej d zdeedeeej dfzf dZxZS)DistilBertModelrAct||t||_t ||_|j yr)rKrLr@rlr transformer post_initrYs r/rLzDistilBertModel.__init__Cs5  $V,&v. r\rbc.|jjSz1 Returns the position embeddings )rlrRrZs r/get_position_embeddingsz'DistilBertModel.get_position_embeddingsLs222r\new_num_position_embeddingsc||jjz }|dk(rytjd|d||j_|jj j j}tj|jj|jj|j_|jjrKt|jj|jj|j j ntj5|dkDr8tj||jj j d| n1tj|d||jj _ddd|jj j!|j"y#1swY9xYw) Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. rNz(Setting `config.max_position_embeddings=z`...r')rArQloggerinforlrRrclonerrMr#rr0r+r Parametertore)rZrnum_position_embeds_diffold_position_embeddings_weights r/resize_position_embeddingsz*DistilBertModel.resize_position_embeddingsRsu$?AdAd#d  $q (  >?Z>[[_`a.I +)-)L)L)S)S)Y)Y)[&.0ll4;;;^;^`d`k`k`o`o.p+ ;; + + (kk99t{{TXTlTlTsTs  +a/]_]i]i6^DOO77>>?YAY@YZBD67P8PQBDOO77>   ++..t{{;  s :A/G!!G*c.|jjSrrlrPrs r/get_input_embeddingsz$DistilBertModel.get_input_embeddings|s...r\new_embeddingsc&||j_yrrrZrs r/set_input_embeddingsz$DistilBertModel.set_input_embeddingss*8'r\Nraryr^rGr{.c |du|duz r td|j|||}t|j||}|jd||d|S) input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds)rAr^ry)rry)rrlr rAr)rZraryr^rGr{rls r/rmzDistilBertModel.forwardsw0  -t"; <YZ Z__Y |L 2;;$)   t $)   r\)NNNN)rnrorpr rLrrMrintrrrrrrr+rqrrrrrmrsrts@r/rrAs/33 ( 112CD 001BC  (():)?)?DUDZDZ[]D^)_agalalmoapqH$&44#..   r\NNNNN)rnrorp_tied_weights_keysr rLrrMrrrrrrrrr+rqrrrrrrrmrsrts@r/rrs 34bc 2/ 299 Pc P$ryy$.BII.*..2-1*.,0 1 <<$&1  t+1 ||d* 1   4' 1 llT) 1 +,1  % c 12 21 1 r\rz DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c@eZdZdeffd Zdej fdZdefdZ e e dde jdzd e jdzd e jdzd e jdzd e jdzd eedeee jdfzfdZxZS)#DistilBertForSequenceClassificationrAct|||j|_||_t ||_t j|j|j|_ t j|j|j|_ t j|j|_ |jyr)rKrL num_labelsrArrrrr#pre_classifier classifierrTseq_classif_dropoutrUrrYs r/rLz,DistilBertForSequenceClassification.__init__s   ++ )&1 ii FJJ?))FJJ0A0ABzz&"<"<=  r\rbc6|jjSrr rs r/rz;DistilBertForSequenceClassification.get_position_embeddings'rr\rc:|jj|yrrrs r/rz>DistilBertForSequenceClassification.resize_position_embeddings-rr\Nraryr^rrGr{.c F|jd ||||dd|}|d}|dddf} |j| } tj| } |j | } |j | } d} ||j j|jdk(rd|j _nl|jdkDrL|jtjk(s|jtjk(rd|j _nd|j _|j jdk(rIt} |jdk(r&| | j|j} n| | |} n|j jdk(r=t} | | j!d |j|j!d } n,|j jdk(rt#} | | |} t%| | |j&|j( S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). TrrNr regressionsingle_label_classificationmulti_label_classificationrIrr)rr)rReLUrUr*rA problem_typer(rdr+rhrrsqueezerrrrrr) rZraryr^rrGr{distilbert_output hidden_state pooled_outputrrloss_fcts r/rmz+DistilBertForSequenceClassification.forward;s",DOO )'%    )+ $QT* ++M: ! -0  ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'+99(33   r\r#)rnrorpr rLrrMrrrrrr+rqrrrrrrrmrsrts@r/r&r&s / 99 Pc P*..2-1*.,0 : <<$&:  t+: ||d* :   4' : llT) : +,:  "E%,,*;$< <: : r\r&c`eZdZdeffd Zdej fdZdefdZ e e dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzdeedeee jdfzfdZxZS)DistilBertForQuestionAnsweringrAcXt||t||_t j |j |j|_|jdk7rtd|jt j|j|_ |jy)Nr2z)config.num_labels should be 2, but it is )rKrLrrrrr#r( qa_outputsrrT qa_dropoutrUrrYs r/rLz'DistilBertForQuestionAnswering.__init__|s  )&1))FJJ0A0AB    !HIZIZH[\] ]zz&"3"34  r\rbc6|jjSrr rs r/rz6DistilBertForQuestionAnswering.get_position_embeddingsrr\rc:|jj|yrrrs r/rz9DistilBertForQuestionAnswering.resize_position_embeddingsrr\Nraryr^start_positions end_positionsrGr{.c |jd ||||dd|}|d} |j| } |j| } | jdd\} } | j dj } | j dj } d} ||t |jdkDr|j d}t |jdkDr|j d}| jd}|jd|}|jd|}tj|}|| |}|| |}||zd z } t| | | |j|j S) rTrrr rIr~N) ignore_indexr2)r start_logits end_logitsrrr)rrUr<splitr4rlenrfclamprrrrr)rZraryr^r@rArGr{r5rrrDrE total_loss ignored_indexr8 start_lossend_losss r/rmz&DistilBertForQuestionAnswering.forwards2,DOO )'%    *!,  ]3 /#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM** FH!,@J M:H$x/14J+%!+99(33   r\)NNNNNN)rnrorpr rLrrMrrrrrr+rqrrrrrmrsrts@r/r:r:zs / 99 Pc P*..2-1/3-1,0> <<$&>  t+> ||d* > , > ||d* > llT)> +,>  &ellC.?(@ @> > r\r:c@eZdZdeffd Zdej fdZdefdZ e e dde jdzd e jdzd e jdzd e jdzd e jdzd eedeee jdfzfdZxZS) DistilBertForTokenClassificationrAc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr) rKrLr(rrrrTrUr hidden_sizer*rrYs r/rLz)DistilBertForTokenClassification.__init__sg   ++)&1zz&..1 ))F$6$68I8IJ r\rbc6|jjSrr rs r/rz8DistilBertForTokenClassification.get_position_embeddingsrr\rc:|jj|yrrrs r/rz;DistilBertForTokenClassification.resize_position_embeddingsrr\Nraryr^rrGr{.c F|j|f|||dd|}|d}|j|}|j|} d} |rr\Nraryr^rrGr{.c ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f|||dd|}|d} | dddf} |j | } t j | } |j| } |j| } | jd|} d} |t}|| |} t| | |j|jS) a^ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Examples: ```python >>> from transformers import AutoTokenizer, DistilBertForMultipleChoice >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased") >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> choice0 = "It is eaten with a fork and a knife." >>> choice1 = "It is eaten while held in the hand." >>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True) >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels) # batch size is 1 >>> # the linear classifier still needs to be trained >>> loss = outputs.loss >>> logits = outputs.logits ```Nr rITrTrr) rrrfrr)rr2rUr*rrrr)rZraryr^rrGr{ num_choicesrUr6r7rreshaped_logitsrr8s r/rmz#DistilBertForMultipleChoice.forwardLsb-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImq(   r=#5#5b#9=;M;Mb;Q R  "$//  )'%    qz $QT* ++M: ! -0  ]3 / ++b+6  ')HOV4D("!//))   r\r#)rnrorpr rLrrMrrrrrr+rqrrrrrrrmrsrts@r/rXrX+s / 99 Pc P*..2-1*.,0 U <<$&U  t+U ||d* U   4' U llT) U +,U  #U5<<+<%= =U U r\rX)rrXr:r&rNrr)Nr)J__doc__collections.abcrnumpyr3r+rtorch.nnrrrr r activationsr configuration_utilsr integrations.deepspeedr masking_utilsr modeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrutils.deprecationr utils.genericrrutils.output_capturingrconfiguration_distilbertr! get_loggerrnrrrqr0r.rr@floatrrrrrrrrr&r:rNrX__all__rr\r/rvs{ % AA&)3@69G& 1I56   H % LL#LELLL3U\\''b!% II% <<% % << % LL4' % T\ %%'(%85)bii5)p"))* 1 F@"))@0iii<g /g g T _ 5_  _ D ^ *C^  ^ Bb %>b b JG '@G G Tw ";w w t r\