qiddlmZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e%jZe.Z/Gddej`Z1ejde1dZ3Gddej`Z4 d_dej`dejjdejjdejjdejjdzde6dzde6de e#fd Z7Gd!d"ej`Z8Gd#d$ej`Z9Gd%d&ej`Z:Gd'd(ej`Z;Gd)d*ej`Z<Gd+d,ej`Z=Gd-d.ej`Z>Gd/d0ej`Z?Gd1d2ej`Z@Gd3d4ej`ZAGd5d6eZBGd7d8ej`ZCGd9d:ej`ZDGd;dej`ZFGd?d@ej`ZGGdAdBej`ZHe$GdCdDeZIee$dEFGdGdHe"ZJe$GdIdJeIZKe$dKFGdLdMeIZLe$GdNdOeIZMGdPdQej`ZNe$dRFGdSdTeIZOe$dUFGdVdWeIZPe$GdXdYeIZQe$GdZd[eIZRe$Gd\d]eIZSgd^ZTy)`)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)MobileBertConfigcXeZdZdfd Zdej dej fdZxZS)NoNormct|tjt j ||_tjt j||_yN) super__init__r Parametertorchzerosbiasonesweight)self feat_sizeeps __class__s d/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/mobilebert/modeling_mobilebert.pyr'zNoNorm.__init__8s@ LLY!78 ll5::i#89 input_tensorreturnc:||jz|jzSr%)r-r+)r.r4s r2forwardzNoNorm.forward=sdkk)DII55r3r%__name__ __module__ __qualname__r'r)Tensorr7 __classcell__r1s@r2r#r#7s#: 6ELL6U\\6r3r#) layer_normno_normc eZdZdZfdZ d dej dzdej dzdej dzdejdzdejf d Z xZ S) MobileBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cXt||j|_|j|_|j|_t j |j|j|j|_ t j |j|j|_ t j |j|j|_ |jrdnd}|j|z}t j||j|_t!|j"|j|_t j&|j(|_|j-dt/j0|jj3ddy)N) padding_idxr r position_idsr F) persistent)r&r' trigram_inputembedding_size hidden_sizer Embedding vocab_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsLinearembedding_transformationNORM2FNnormalization_type LayerNormDropouthidden_dropout_probdropoutregister_bufferr)arangeexpand)r.configembed_dim_multiplierembedded_input_sizer1s r2r'zMobileBertEmbeddings.__init__GsF #11$33!--!||F,=,=v?T?Tbhbubuv#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"$($6$6qA"114HH(* 2EvGYGY(Z% !:!:;F J"%88;PP ^^J/ \\*- r3)NNNN) r9r:r;__doc__r'r) LongTensor FloatTensorr<r7r=r>s@r2rBrBDs~Q 0.2260426 0##d*0((4/0&&- 0 ((4/ 0  0r3rBmodulequerykeyriattention_maskscalingr[kwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)NrGrjr rk)ptrainingr ) rmr)matmul transposerrpsoftmaxr[r contiguous) rxryrzrir{r|r[r} attn_weights attn_outputs r2eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r3ceZdZfdZ d dej dej dej dej dzdeede ej f d Z xZ S) MobileBertSelfAttentionct|||_|j|_t |j |jz |_|j|j z|_|j dz|_tj|j |j|_ tj|j |j|_ tj|jr |j n |j|j|_tj |j"|_d|_y)NrF)r&r'r_num_attention_headsinttrue_hidden_sizeattention_head_size all_head_sizer|rrTryrzuse_bottleneck_attentionrKrirYattention_probs_dropout_probr[ is_causalr.r_r1s r2r'z MobileBertSelfAttention.__init__s  #)#=#= #&v'>'>A[A['[#\ !558P8PP//5 YYv668J8JK 99V44d6H6HIYY'-'F'FF # #FL^L^`d`r`r  zz&"E"EF r3N query_tensor key_tensor value_tensorr{r}r5c |jdd}g|d|j}|j|j|j dd}|j |j|j dd} |j |j|j dd} tj|jjt} | ||| | |f|jsdn|jj|jd|\} } | j g|dj#} | | fS)NrGr rjrh)r[r|)shaperryviewrrzrir get_interfacer__attn_implementationrrr[rr|reshaper)r.rrrr{r}rr hidden_shape query_layer key_layer value_layerattention_interfacerrs r2r7zMobileBertSelfAttention.forwardsH#(("- CCbC$*B*BC 4djj.33\BLLQPQR -DHHZ(--|<FFq!L 3djj.33\BLLQPQR (?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((r3r% r9r:r;r'r)r<rwrrtupler7r=r>s@r2rrsr,48 )ll)LL)ll ) ))D0 ) +, ) u|| )r3rcneZdZfdZdej dej dej fdZxZS)MobileBertSelfOutputcjt||j|_tj|j |j |_t|j|j |j|_ |js%tj|j|_ yyNr0)r&r'use_bottleneckrrTrdenserVrWlayer_norm_epsrXrYrZr[rs r2r'zMobileBertSelfOutput.__init__s $33YYv668O8OP  !:!:;Fs@r2rrs2BU\\ELLUZUaUar3rceZdZfdZ d dej dej dej dej dej dzdeed e ej fd Z xZ S) MobileBertAttentioncbt|t||_t ||_yr%)r&r'rr.routputrs r2r'zMobileBertAttention.__init__s& +F3 *62 r3Nrrr layer_inputr{r}r5c `|j||||fi|\}}|j||}||fSr%)r.r) r.rrrrr{r}attention_outputrs r2r7zMobileBertAttention.forwardsO*3     *  * &, ;;'7E--r3r%rr>s@r2rrs348 .ll.LL.ll . \\ . ))D0 .+,. u|| .r3rcVeZdZfdZdej dej fdZxZS)MobileBertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr%) r&r'rrTrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrs r2r'zMobileBertIntermediate.__init__s] YYv668P8PQ f'' -'-f.?.?'@D $'-'8'8D $r3rr5cJ|j|}|j|}|Sr%)rrr.rs r2r7zMobileBertIntermediate.forwards& =1 00? r3r8r>s@r2rrs#9U\\ellr3rcneZdZfdZdej dej dej fdZxZS)OutputBottleneckc.t|tj|j|j |_t|j|j |j|_ tj|j|_ yr)r&r'rrTrrKrrVrWrrXrYrZr[rs r2r'zOutputBottleneck.__init__sh YYv668J8JK  !:!:;Fs@r2rrs1> U\\ELLUZUaUar3rceZdZfdZdej dej dej dej fdZxZS)MobileBertOutputcrt||j|_tj|j |j |_t|j|j |_ |js%tj|j|_ yt||_yr%)r&r'rrrTrrrrVrWrXrYrZr[r bottleneckrs r2r'zMobileBertOutput.__init__,s $33YYv779P9PQ  !:!:;F>,9J*JKL >>,9J*JKL??<9JKLr3r8r>s@r2rr+s?7 #(<< DILL ejeqeq   r3rcVeZdZfdZdej dej fdZxZS)BottleneckLayerct|tj|j|j |_t|j|j |j|_ yr) r&r'rrTrKintra_bottleneck_sizerrVrWrrXrs r2r'zBottleneckLayer.__init__DsR YYv1163O3OP  !:!:;Fs@r2rrCs$u U\\ellr3rc\eZdZfdZdej deej fdZxZS) Bottleneckct||j|_|j|_t ||_|jrt ||_yyr%)r&r'key_query_shared_bottleneckrrinput attentionrs r2r'zBottleneck.__init__PsP +1+M+M((.(G(G%$V,  + +,V4DN ,r3rr5c|j|}|jr|fdzS|jr|j|}||||fS||||fS)N)rrrr)r.rbottlenecked_hidden_statesshared_attention_inputs r2r7zBottleneck.forwardXsc"&*ZZ %>"  ( (.014 4  - -%)^^M%B "*,BMSmn n!=-A[\ \r3 r9r:r;r'r)r<rr7r=r>s@r2rrOs+5]U\\]eELL6I]r3rcneZdZfdZdej dej dej fdZxZS) FFNOutputct|tj|j|j |_t|j|j |j|_ yr) r&r'rrTrrrrVrWrrXrs r2r'zFFNOutput.__init__tsR YYv779P9PQ  !:!:;Fs@r2rrss2p U\\ELLUZUaUar3rcVeZdZfdZdej dej fdZxZS)FFNLayercbt|t||_t ||_yr%)r&r'r intermediaterrrs r2r'zFFNLayer.__init__s' 26:' r3rr5cL|j|}|j||}|Sr%)rr)r.rintermediate_outputrs r2r7zFFNLayer.forwards*"// > $7G r3r8r>s@r2rrs#( U\\ellr3rc eZdZfdZ ddej dej dzdeede ej fdZ xZ S) MobileBertLayerct||j|_|j|_t ||_t ||_t||_ |jrt||_ |jdkDrHtjt|jdz Dcgc] }t|c}|_yycc}wNr )r&r'rnum_feedforward_networksrrrrrrrrr ModuleListrangerffnr.r__r1s r2r'zMobileBertLayer.__init__s $33(.(G(G%,V426:&v.   (0DO  * *Q .}}fFeFehiFi@j%k1hv&6%klDH /%ks6CNrr{r}r5c 2|jr|j|\}}}}n |gdz\}}}}|j|||||fi|\}} |} |jdk7r|jD] } | | } |j | } |j | | |} | S)Nrr )rrrrrrr)r.rr{r}rrrrself_attention_outputrr ffn_modulerrs r2r7zMobileBertLayer.forwards   BF//R_B` ?L*lKCP/TUBU ?L*lK#14>>      $  $  q1  ( (A -"hh @ #-.>#?  @#//0@A{{#68H-X r3r%rr>s@r2rrsW m 48||))D0+,  u||  r3rc teZdZfdZ ddej dej dzdeede e zfdZ xZ S) MobileBertEncoderct|tjt |j Dcgc] }t |c}|_ycc}wr%)r&r'rrrnum_hidden_layersrlayerrs r2r'zMobileBertEncoder.__init__s< ]]U6KcKcEd#eOF$;#ef #esANrr{r}r5c ht|jD]\}}|||fi|}t|S)N)last_hidden_state) enumeraterr)r.rr{r}i layer_modules r2r7zMobileBertEncoder.forwardsG )4 OA|(M  ??r3r%) r9r:r;r'r)r<rwrrrrr7r=r>s@r2rrsXg48 @|| @))D0 @+, @  @r3rcVeZdZfdZdej dej fdZxZS)MobileBertPoolerct||j|_|jr0t j |j |j |_yyr%)r&r'classifier_activation do_activaterrTrKrrs r2r'zMobileBertPooler.__init__sH !77   6#5#5v7I7IJDJ r3rr5c|dddf}|js|S|j|}tj|}|S)Nr)rrr)tanh)r.rfirst_token_tensor pooled_outputs r2r7zMobileBertPooler.forwardsE+1a40% % JJ'9:M!JJ}5M r3r8r>s@r2rrs$K !U\\ !ell !r3rcVeZdZfdZdej dej fdZxZS)!MobileBertPredictionHeadTransformcZt|tj|j|j|_t |jtrt|j|_ n|j|_ td|j|j|_ y)Nr?r)r&r'rrTrKrrrrr transform_act_fnrVrrXrs r2r'z*MobileBertPredictionHeadTransform.__init__s| YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D ! .v/A/AvG\G\]r3rr5cl|j|}|j|}|j|}|Sr%)rrrXrs r2r7z)MobileBertPredictionHeadTransform.forwards4 =1 --m< }5 r3r8r>s@r2r r s$^U\\ellr3r cVeZdZfdZdej dej fdZxZS)MobileBertLMPredictionHeadct|t||_t j |j |j|jz d|_ t j |j|j d|_ t jtj|j |_y)NF)r+T)r&r'r  transformrrTrMrKrJrdecoderr(r)r*r+rs r2r'z#MobileBertLMPredictionHead.__init__s :6BYYv00&2D2DvG\G\2\chi yy!6!68I8IPTU LLV->->!?@ r3rr5c|j|}|jtj|jj j |jj gd}||jjz }|S)Nrrk) rrr)rorr-trr+rs r2r7z"MobileBertLMPredictionHead.forwardsk}5 %,,UYY 8K8K8M8M8OQUQ[Q[QbQb7cij-kl *** r3r8r>s@r2rrs$AU\\ellr3rcVeZdZfdZdej dej fdZxZS)MobileBertOnlyMLMHeadcBt|t||_yr%)r&r'r predictionsrs r2r'zMobileBertOnlyMLMHead.__init__s 5f=r3sequence_outputr5c(|j|}|Sr%)r)r.rprediction_scoress r2r7zMobileBertOnlyMLMHead.forwards ,,_=  r3r8r>s@r2rrs#>!u||! !r3rcteZdZfdZdej dej deej fdZxZS)MobileBertPreTrainingHeadsct|t||_t j |j d|_yNrj)r&r'rrrrTrKseq_relationshiprs r2r'z#MobileBertPreTrainingHeads.__init__ s4 5f= " &*<*s@r2rr s8A 9u||9ELL9UZ[`[g[gUh9r3rcteZdZUeed<dZdZdZdZdZ dZ e e dZ ejfdZxZS)MobileBertPreTrainedModelr_ mobilebertT)r attentionsct||t|tr?t j |j t j|jyt|tr t j |j yt|trZt j|jtj|jjdj!dyy)zInitialize the weightsrGrFN)r& _init_weightsrr#initzeros_r+ones_r-rrBcopy_rEr)r]rr^)r.rxr1s r2r*z'MobileBertPreTrainedModel._init_weights$s f% ff % KK $ JJv}} %  : ; KK $  4 5 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h6r3)r9r:r;r!__annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsr)no_gradr*r=r>s@r2r&r&sX $&*#N"&(- U]]_ i ir3r&z6 Output type of [`MobileBertForPreTraining`]. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<y)MobileBertForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr() r9r:r;rur;r)rwr/r<r=rrr(r3r2r:r:1s~ &*D%  d ")26u((4/68<U..5<59M5**+d2926Je''(4/6r3r:c eZdZdZdfd ZdZdZeee dde jdzde jdzde jdzd e jdzd e jdzd e ed eezfd ZxZS)MobileBertModelz2 https://huggingface.co/papers/2004.02984 ct||||_d|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r&r'r_gradient_checkpointingrBrtrencoderrpooler post_init)r.r_add_pooling_layerr1s r2r'zMobileBertModel.__init__PsV   &+#.v6(0 2C&v.  r3c.|jjSr%rtrOr.s r2get_input_embeddingsz$MobileBertModel.get_input_embeddingsas...r3c&||j_yr%rH)r.ris r2set_input_embeddingsz$MobileBertModel.set_input_embeddingsds*/'r3Nrbr{rcrErdr}r5c |du|duz r td|j||||}t|j||}|j|fd|i|}|d} |j |j | nd} t | | S)Nz:You must specify exactly one of input_ids or inputs_embeds)rbrErcrd)r_rdr{r{r)r pooler_output) ValueErrorrtr r_rCrDr) r.rbr{rcrErdr}embedding_outputencoder_outputsrr s r2r7zMobileBertModel.forwardgs  -t"; <YZ Z??%)' + 3;;*)  '$,,  )   *!,8< 8O O4UY )-'  r3)T)NNNNN)r9r:r;rur'rJrLrrrr)rvrwrrrrr7r=r>s@r2r@r@Js"/0 .237260426 $ ##d*$ ))D0$ ((4/ $ &&- $ ((4/ $ +,$  + +$  $ r3r@z MobileBert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. cveZdZdddZfdZdZdZddedzd ejffd Z e e dd e jdzd e jdzd e jdzde jdzde jdzde jdzde jdzdeed eezfdZxZS)MobileBertForPreTrainingcls.predictions.bias,mobilebert.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightct||t||_t ||_|j yr%)r&r'r@r'rclsrErs r2r'z!MobileBertForPreTraining.__init__s4  )&1-f5 r3cB|jjjSr%rXrrrIs r2get_output_embeddingsz.MobileBertForPreTraining.get_output_embeddingsxx##+++r3c||jj_|j|jj_yr%rXrrr+r.new_embeddingss r2set_output_embeddingsz.MobileBertForPreTraining.set_output_embeddings,'5$$2$7$7!r3Nnew_num_tokensr5c|j|jjj|d|jj_t||SNT)rc transposed)rc_get_resized_lm_headrXrrr&resize_token_embeddingsr.rcr1s r2riz0MobileBertForPreTraining.resize_token_embeddingssR%)%>%> HH & &~RV&?& "w.n.MMr3rbr{rcrErdlabelsnext_sentence_labelr}c |j|f||||dd|} | dd\} } |j| | \} } d}|u|st}|| jd|jj |jd}|| jdd|jd}||z}t || | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForPreTraining.from_pretrained("google/mobilebert-uncased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Tr{rcrErd return_dictNrjrG)r;r<r=rr() r'rXrrr_rMr:rr()r.rbr{rcrErdrkrlr}outputsrr rr$ total_lossloss_fctmasked_lm_lossnext_sentence_losss r2r7z MobileBertForPreTraining.forwardsR"$//  ))%'   *1!&48HH_m4\11  "5"A')H%&7&<&s@r2rSrSs/)?*X ,8NcDjNBLLN.237260426*.7;@ ##d*@ ))D0@ ((4/ @ &&- @ ((4/ @   4'@ #--4@ +,@  / /@ @ r3rScVeZdZdddZfdZdZdZddedzd ejffd Z e e dd e jdzd e jdzd e jdzde jdzde jdzde jdzdeed eezfdZxZS)MobileBertForMaskedLMrTrUrVct||t|d|_t ||_||_|jyNF)rF)r&r'r@r'rrXr_rErs r2r'zMobileBertForMaskedLM.__init__s=  )&EJ(0  r3cB|jjjSr%rZrIs r2r[z+MobileBertForMaskedLM.get_output_embeddings r\r3c||jj_|j|jj_yr%r^r_s r2raz+MobileBertForMaskedLM.set_output_embeddings rbr3Nrcr5c|j|jjj|d|jj_t||Srergrjs r2riz-MobileBertForMaskedLM.resize_token_embeddingssR%)%>%> HH & &~RV&?& "w.n.MMr3rbr{rcrErdrkr}c :|j|f||||dd|}|d} |j| } d} |Ft} | | jd|jj |jd} t | | |j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` TrnrNrGr;logitsrr() r'rXrrr_rMrrr() r.rbr{rcrErdrkr}rprrrsrrs r2r7zMobileBertForMaskedLM.forwards$"$//  ))%'   "!* HH_5  ')H%&7&<&s@r2rxrxs )?*X ,8NcDjNBLLN.237260426*.' ##d*' ))D0' ((4/ ' &&- ' ((4/ '   4'' +,'   ' ' r3rxcVeZdZfdZdej dej fdZxZS)MobileBertOnlyNSPHeadclt|tj|jd|_yr!)r&r'rrTrKr"rs r2r'zMobileBertOnlyNSPHead.__init__Ds'  " &*<*s@r2rrCs$A&U\\&ell&r3rzZ MobileBert Model with a `next sentence prediction (classification)` head on top. ceZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e e zfd Z xZS) #MobileBertForNextSentencePredictionct||t||_t ||_|j yr%)r&r'r@r'rrXrErs r2r'z,MobileBertForNextSentencePrediction.__init__Ss4  )&1(0 r3Nrbr{rcrErdrkr}r5c |j|f||||dd|}|d} |j| } d} |2t} | | jdd|jd} t | | |j |j S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```Trnr NrGrjr)r'rXrrrrr() r.rbr{rcrErdrkr}rpr r$rtrrs r2r7z+MobileBertForNextSentencePrediction.forward\sL"$//  ))%'     !%-!8!  ')H!)*@*E*Eb!*LfkkZ\o!^ *#)!//))   r3r)r9r:r;r'rrr)rvrwrrrrr7r=r>s@r2rrMs .237260426*.; ##d*; ))D0; ((4/ ; &&- ; ((4/ ;   4'; +,;  , ,; ; r3rz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) #MobileBertForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr%)r&r' num_labelsr_r@r'classifier_dropoutrZrrYr[rTrK classifierrEr.r_rr1s r2r'z,MobileBertForSequenceClassification.__init__s   ++ )&1)/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r3Nrbr{rcrErdrkr}r5c |j|f||||dd|}|d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | |j |j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trnr N regressionsingle_label_classificationmulti_label_classificationrGr)r'r[rr_ problem_typerrfr)rnrrsqueezerrrrrr() r.rbr{rcrErdrkr}rpr rr;rrs r2r7z+MobileBertForSequenceClassification.forwards$"$//  ))%'      ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r3r)r9r:r;r'rrr)r<rrrrr7r=r>s@r2rrs *..2.2,0-1&*; <<$&;  t+;  t+ ; llT) ; ||d* ;  t#; +,;  u|| 7 7; ; r3rcJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)MobileBertForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyrz) r&r'rr@r'rrTrK qa_outputsrErs r2r'z'MobileBertForQuestionAnswering.__init__sU   ++)&EJ))F$6$68I8IJ r3Nrbr{rcrErdstart_positions end_positionsr}r5c |j|f||||dd|} | d} |j| } | jdd\} } | jdj } | jdj } d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|| |}||zdz }t|| | | j| j S) NTrnrr rGrk) ignore_indexrj)r; start_logits end_logitsrr() r'rsplitrrlenrmclamprrrr()r.rbr{rcrErdrrr}rprrrrrq ignored_indexrr start_lossend_losss r2r7z&MobileBertForQuestionAnswering.forwards"$//  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   r3ru)r9r:r;r'rrr)r<rrrrr7r=r>s@r2rrs*..2.2,0-1/3-13 <<$&3  t+3  t+ 3 llT) 3 ||d* 3 ,3 ||d*3 +,3  u|| ; ;3 3 r3rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) MobileBertForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jyr) r&r'r@r'rrZrrYr[rTrKrrErs r2r'z$MobileBertForMultipleChoice.__init__;su  )&1)/)B)B)NF % %TZTnTn zz"45 ))F$6$6: r3Nrbr{rcrErdrkr}r5c ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f||||dd|} | d} |j | } |j | } | jd|} d} |t }|| |} t| | | j| jS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr rGTrnr) rrrmr'r[rrrrr()r.rbr{rcrErdrkr} num_choicesrpr rreshaped_logitsr;rrs r2r7z#MobileBertForMultipleChoice.forwardHsT-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  "$//  ))%'      ]3 / ++b+6  ')HOV4D("!//))   r3r)r9r:r;r'rrr)r<rrrrr7r=r>s@r2rr8s *..2.2,0-1&*N <<$&N  t+N  t+ N llT) N ||d* N  t#N +,N  u|| 8 8N N r3rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) MobileBertForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyrz)r&r'rr@r'rrZrrYr[rTrKrrErs r2r'z)MobileBertForTokenClassification.__init__s   ++)&EJ)/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r3Nrbr{rcrErdrkr}r5c H|j|f||||dd|}|d} |j| } |j| } d} |s@r2rrs *..2.2,0-1&*' <<$&'  t+'  t+ ' llT) ' ||d* '  t#' +,'  u|| 4 4' ' r3r) rxrrrSrrrrr@r&)Nrh)Ucollections.abcr dataclassesrr)rtorch.nnrrrr r+ activationsr masking_utilsr modeling_layersr modeling_outputsrrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_mobilebertr! get_loggerr9loggerModuler#rXrVrBr<floatrrrrrrrrrrrrrrr rrrr&r:r@rSrxrrrrrr__all__r>r3r2rs.%! AA&!69   G&MMI56   H %6RYY6& 9I299If!% II% <<% % << % LL4' % T\ %%'(%81)bii1)h299".")).8 RYY  ryy ryy0 bii !]!]H   ryy )0)X@ @(!ryy!& ""!BII! 9 9iii4  7[7  7&C /C C L _ 8_  _ DF 5F F R&BII& G *CG  G TM *CM M `@ %>@ @ F^ ";^ ^ B8 '@8 8 v r3