qi_ddlmZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m6Z6e/jne8Z9GddejtZ; dZdejtdejxdejxdejxdejxdzde=dzde=de(e-fdZ>Gd d!ejtZ?Gd"d#ejtZ@Gd$d%ejtZAGd&d'ejtZBGd(d)ejtZCGd*d+ejtZDGd,d-eZEGd.d/ejtZFGd0d1ejtZGGd2d3ejtZHGd4d5ejtZIe.Gd6d7e&ZJe.d89Gd:d;eJZKee.d<9Gd=d>e,ZLGd?d@ejtZMe.dA9GdBdCeJZNGdDdEejtZOe.dF9GdGdHeJeZPe.GdIdJeJZQGdKdLejtZRe.dM9GdNdOeJZSe.dP9GdQdReJZTe.GdSdTeJZUe.GdUdVeJZVe.GdWdXeJZWgdYZXy)[)Callable) dataclassN)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) ErnieConfigceZdZdZfdZ d dej dzdej dzdej dzdej dzdejdzd ed ejfd Z xZ S) ErnieEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0d|j2|_|j2r0tj|j4|j |_yy) N) padding_idxeps position_idsr&F) persistenttoken_type_ids)dtype)super__init__nn Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr.sizelong use_task_idtask_type_vocab_sizetask_type_embeddingsselfconfig __class__s Z/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/ernie/modeling_ernie.pyr5zErnieEmbeddings.__init__<sX !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "--   (* V5P5PRXRdRd(eD % N input_idsr2 task_type_idsr. inputs_embedspast_key_values_lengthreturnc|||j}n|jdd}|\}} ||jdd|| |zf}|t|drT|jj |j dd} t j| d|} | j || }n:t j|t j|jj}||j|}|j|} |j| j}|| z} |j|} | | z} |jrR|:t j|t j|jj}|j!|}| |z } |j#| } |j%| } | S)Nr0r2rr&)dimindex)r3device)rJr.hasattrr2rHshaperFgatherrIrKr]r;r?tor=rLrNr@rD)rPrUr2rVr.rWrX input_shape batch_size seq_lengthbuffered_token_type_idsr? embeddingsr=rNs rSforwardzErnieEmbeddings.forwardPs  #..*K',,.s3K!, J  ,,Q0FVlIl0l-lmL  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J&(()>)E)EF "%:: "66|D"55    $ % KuzzRVRcRcRjRj k #'#<#<]#K . .J^^J/ \\*- rT)NNNNNr) __name__ __module__ __qualname____doc__r5rF LongTensor FloatTensorintTensorrg __classcell__rRs@rSr)r)9sQf,.226150426&'3##d*3((4/3''$. 3 &&- 3 ((4/ 3!$3 3rTr)modulequerykeyvalueattention_maskscalingrDkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr0rr[)ptrainingr&) rJrFmatmul transposer6 functionalsoftmaxrDr~ contiguous) rrrsrtrurvrwrDrx attn_weights attn_outputs rSeager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rTceZdZd fd Z d dej dej dzdedzdej dzdee de ej f d Z xZ S) ErnieSelfAttentionNc@t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_|j&|_||_||_yNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rz)r4r5r9num_attention_headsr^ ValueErrorrQrnattention_head_size all_head_sizerwr6LinearrsrtrurBattention_probs_dropout_probrD is_decoder is_causal layer_idxrPrQrrrRs rSr5zErnieSelfAttention.__init__sP     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F?*9*N*N'&=%C%C!>2 & "I{)@(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rTFNNNN rhrirjr5rFrormr rr tuplergrprqs@rSrrs}#648(,.2 -)||-)))D0-) -)  t+ -) +, -) u|| -)rTrceZdZd fd Z d dej dej dzdej dzdedzdee de ej f d Z xZ S) ErnieCrossAttentionNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_||_||_yr)r4r5r9rr^rrQrnrrrwr6rrsrtrurBrrDrrrs rSr5zErnieCrossAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \"))#w;FFH L((rTrr) rhrirjr5rFrormr rr rrgrprqs@rSrrs#4;?376: 2)||2) %00472)))D0 2) -t3 2) +, 2) u|| 2)rTrcneZdZfdZdej dej dej fdZxZS)ErnieSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr,) r4r5r6rr9denser@rArBrCrDrOs rSr5zErnieSelfOutput.__init__9s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rTr input_tensorrYcr|j|}|j|}|j||z}|SNrrDr@rPrrs rSrgzErnieSelfOutput.forward?7 =1  ]3 }|'CD rTrhrirjr5rFrorgrprqs@rSrr81> U\\RWR^R^rTrceZdZd fd Z d dej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z xZ S)ErnieAttentionNct|||_|rtnt}|||||_t ||_y)Nrr)r4r5is_cross_attentionrrrProutput)rPrQrrrattention_classrRs rSr5zErnieAttention.__init__Gs= "41C-I[#Fi9U %f- rTrrvrencoder_attention_maskrrrxrYc |js|n|}|j|f||||d|\}} |j||}|| fS)N)rrvrr)rrPr) rPrrvrrrrrxattention_outputrs rSrgzErnieAttention.forwardNsg04/F/FLb)2 * "7)+) *  * &, ;;'7G--rT)FNFNNNNNrrqs@rSrrFs.48:>;?(,.2.||.))D0. %0047 . !& 1 1D 8 .  . t+.+,. u|| .rTrcVeZdZfdZdej dej fdZxZS)ErnieIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r4r5r6rr9intermediate_sizerr hidden_actstrr intermediate_act_fnrOs rSr5zErnieIntermediate.__init__fs] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rTrrYcJ|j|}|j|}|Sr)rrrPrs rSrgzErnieIntermediate.forwardns& =1 00? rTrrqs@rSrres#9U\\ellrTrcneZdZfdZdej dej dej fdZxZS) ErnieOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r4r5r6rrr9rr@rArBrCrDrOs rSr5zErnieOutput.__init__us` YYv779K9KL f&8&8f>S>STzz&"<"<= rTrrrYcr|j|}|j|}|j||z}|Srrrs rSrgzErnieOutput.forward{rrTrrqs@rSrrtrrTrceZdZd fd Z ddej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z d Z xZ S) ErnieLayerNct||j|_d|_t ||j ||_|j |_|j|_|jr.|j st|dt |d|d|_ t||_ t||_ y)Nr&rz> should be used as a decoder model if cross attention is addedFT)rrr)r4r5chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rPrQrrRs rSr5zErnieLayer.__init__s '-'E'E$'&:K:KW`a ++#)#=#=  # #?? D6)g!hii"0##' #D  .f5!&) rTrrvrrrrrxrYc "|j||f||d|\}} |} |jr:|8t|dstd|d|j|d||fd|i|\} } | } t |j |j|j| } | S)N)rrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrr^rrrfeed_forward_chunkrr) rPrrvrrrrrxself_attention_output_rcross_attention_output layer_outputs rSrgzErnieLayer.forwards$24>>  $ ,) $  $  q1 ??4@4!12 =dVDDD )<(;(;%%& ) !0 )  ) % "A 6 0  # #T%A%A4CSCSUe rTcL|j|}|j||}|Sr)rr)rPrintermediate_outputrs rSrzErnieLayer.feed_forward_chunks,"//0@A{{#68HI rTrr)rhrirjr5rFrormr rr rrgrrprqs@rSrrs*,48:>;?(,.2'||'))D0' %0047 ' !& 1 1D 8 '  ' t+'+,' u|| 'RrTrcVeZdZfdZdej dej fdZxZS) ErniePoolerct|tj|j|j|_tj |_yr)r4r5r6rr9rTanh activationrOs rSr5zErniePooler.__init__s9 YYv1163E3EF '')rTrrYc\|dddf}|j|}|j|}|S)Nr)rr)rPrfirst_token_tensor pooled_outputs rSrgzErniePooler.forwards6+1a40 #56  6 rTrrqs@rSrrs#$ U\\ellrTrcVeZdZfdZdej dej fdZxZS)ErniePredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r4r5r6rr9rrrrr transform_act_fnr@rArOs rSr5z%ErniePredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrTrrYcl|j|}|j|}|j|}|Sr)rrr@rs rSrgz$ErniePredictionHeadTransform.forwards4 =1 --m< }5 rTrrqs@rSrrs$UU\\ellrTrc$eZdZfdZdZxZS)ErnieLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r4r5r transformr6rr9r8decoder ParameterrFrIrrOs rSr5zErnieLMPredictionHead.__init__s[ 5f=yy!3!3V5F5FTR LLV->->!?@ rTcJ|j|}|j|}|Sr)rrrs rSrgzErnieLMPredictionHead.forwards$}5  ]3 rTrhrirjr5rgrprqs@rSrrsArTrceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd ej dzd e e d e ej e zfd Z xZS) ErnieEncoderc t|||_tjt |j Dcgc]}t||c}|_ycc}w)N)r) r4r5rQr6 ModuleListrangenum_hidden_layersrlayer)rPrQirRs rSr5zErnieEncoder.__init__sF  ]]USYSkSkMl#mJv$C#mn #msANrrvrrr use_cacherrxrYc t|jD]\} } | |||f|||d|}t||r|SdS)N)rrr)last_hidden_stater) enumeraterr) rPrrvrrrrrrxr layer_modules rSrgzErnieEncoder.forwardsq )4 OA|(%(> /- M 9+/8O  >B  rT)NNNNNN)rhrirjr5rFrormr boolrr rrrgrprqs@rSr r so48:>;?(,!%.2 || ))D0  %0047  !& 1 1D 8    $;  t+ +,  u|| H H rTr cneZdZeZdZdZdZdZdZ dZ e e e dZej fdZxZS)ErniePreTrainedModelernieT)r attentionscross_attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weightsr0r/N)r4 _init_weightsrrinitzeros_rr)copy_r.rFrGr_rHr2)rPrrrRs rSrz"ErniePreTrainedModel._init_weights&s f% f3 4 KK $  0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .1rT)rhrirjr' config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrFno_gradrrprqs@rSrrsXL&*#N"&#(/ U]]_//rTra  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introceZdZdgZdfd ZdZdZeee dde jdzde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde dzde dzde jdzdeedee jezfdZdZxZS) ErnieModelrct||||_d|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r4r5rQgradient_checkpointingr)rfr encoderrpooler post_init)rPrQadd_pooling_layerrRs rSr5zErnieModel.__init__@sU   &+#)&1#F+ ->k&)D  rTc.|jjSrrfr;rPs rSget_input_embeddingszErnieModel.get_input_embeddingsQs...rTc&||j_yrr5)rPrus rSset_input_embeddingszErnieModel.set_input_embeddingsTs*/'rTNrUrvr2rVr.rWrrrrrrxrYc |jjr| | n|jj} nd} |jr%|jr| rt j dd} | rd| b||jjr4tt|jt|jnt|j} | | td|#|j|||j} n!||jdd} n td| \}}| |jn |j}| | jnd}| tj |||z| } |j#|||||| }|j%||||| | \}}|j&|f|||| | | |d | }|d}|j(|j)|nd}t+|||j, S) task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] NFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...)rQzDYou cannot specify both input_ids and inputs_embeds at the same timer0z5You have to specify either input_ids or inputs_embedsr)r])rUr.r2rVrWrX)rvrembedding_outputrrr)rvrrrrrr.)r pooler_outputr)rQrrr/r~logger warning_onceis_encoder_decoderr r r%warn_if_padding_and_no_attention_maskrJr]get_seq_lengthrFrGrf_create_attention_masksr0r1rr)rPrUrvr2rVr.rWrrrrrrxrbrcrdr]rXr<encoder_outputssequence_outputrs rSrgzErnieModel.forwardWs12 ;; ! !%.%:  @U@UII  & &4==##p" 0)4 8V8V$L $DlZ^ZeZeFfg!5   ]%>cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TETE`!?!?!Afg  !"\\*@BX[eBentuN??%)''#9+ 261M1M)#9-"7)+ 2N2 ..'$,,   )"7#9+)%    *!,8< 8O O4UY ;-'+;;  rTc|jjrt|j||||}nt|j||}|t|j|||}||fS)N)rQrWrvrr)rQrWrv)rQrWrvr)rQrrr)rPrvrr<rrrs rSrCz"ErnieModel._create_attention_maskssx ;; ! !/{{.-- / N7{{.-N " -%>{{.5&; & "555rT)T) NNNNNNNNNNN)rhrirj_no_split_modulesr5r7r9r$r%r!rFror rrr rrrgrCrprqs@rSr-r-1sf&"/0 *..2.2-1,0-1596:(,!%.2_ <<$&_  t+_  t+ _ ||d* _ llT) _ ||d*_  %||d2_ !& t 3_ _ $;_  t+_ +,_  u|| K K_  _ B 6rTr-z1 Output type of [`ErnieForPreTraining`]. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<y)ErnieForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr) rhrirjrkrJrFrm__annotations__rKrLrrrrTrSrIrIs~ &*D%  d ")26u((4/68<U..5<59M5**+d2926Je''(4/6rTrIc$eZdZfdZdZxZS)ErniePreTrainingHeadsct|t||_t j |j d|_yNr{)r4r5r predictionsr6rr9seq_relationshiprOs rSr5zErniePreTrainingHeads.__init__s4 08 " &*<*>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trvr2rVr.rW return_dictNr{r0)rJrKrLrr) rr`rrrQr8rIrr)rPrUrvr2rVr.rWrkrlrxoutputsrErrVrW total_lossloss_fctmasked_lm_lossnext_sentence_losss rSrgzErnieForPreTraining.forwards^$**   ))'%'    *1!&48HH_m4\11  "5"A')H%&7&<&warningr-rrxr`r2rOs rSr5zErnieForCausalLM.__init__sL    NNj k%@ #F+ rTcB|jjjSrrbr6s rSrcz&ErnieForCausalLM.get_output_embeddingsrdrTc||jj_|j|jj_yrrfrgs rSriz&ErnieForCausalLM.set_output_embeddingsrjrTNrUrvr2rVr.rWrrrkrrrlogits_to_keeprxrYc | d} |j|f|||||||| | | dd |}|j}t| tr t | dn| }|j |dd|ddf}d}| *|j d|| |jjd|}t|||j|j|j|jS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rvr2rVr.rWrrrrrro)logitsrkr8)rJrrrrrrN)rrrrnslicer` loss_functionrQr8rrrrr)rPrUrvr2rVr.rWrrrkrrrrrxrpr slice_indicesrrJs rSrgzErnieForCausalLM.forwards<  I@J A ))'%'"7#9+)A A   11 8B>SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopD0#33!//))$55   rT) NNNNNNNNNNNNr)rhrirjrvr5rcrir#r!rFrolistrrnrr rrrgrprqs@rSr|r|vs+T(> ,8*..2.2-1,0-1596:&*59!%.2-.? <<$&?  t+?  t+ ? ||d* ? llT) ? ||d*?  %||d2? !& t 3?  t#? ell+d2? $;?  t+? ell*? +,? u|| @ @!? ? rTr|ceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde jdzde e de e jezfdZxZS)ErnieForMaskedLMrZr[r\ct|||jrtj dt |d|_t||_|jy)NzlIf you want to use `ErnieForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr~rrOs rSr5zErnieForMaskedLM.__init__sR     NN1   %@ #F+ rTcB|jjjSrrbr6s rSrcz&ErnieForMaskedLM.get_output_embeddingsrdrTc||jj_|j|jj_yrrfrgs rSriz&ErnieForMaskedLM.set_output_embeddingsrjrTNrUrvr2rVr.rWrrrkrxrYc @|j|f|||||||dd| } | d} |j| } d}| Ft}|| jd|jj | jd}t || | j| jS)as task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rvr2rVr.rWrrrorNr0rJrrr) rr`rrrQr8rrr)rPrUrvr2rVr.rWrrrkrxrprErVrsrrs rSrgzErnieForMaskedLM.forwards4$**   ))'%'"7#9    "!* HH_5  ')H%&7&<&>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trnr&Nr0r{r)rr`rrrrr)rPrUrvr2rVr.rWrkrxrprseq_relationship_scoresrtrrs rSrgz&ErnieForNextSentencePrediction.forwardEsZ$**   ))'%'      "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   rTNNNNNNN)rhrirjr5r#r!rFrorr rrrgrprqs@rSrr6s *..2.2-1,0-1&*D <<$&D  t+D  t+ D ||d* D llT) D ||d*D  t#D +,D  u|| : :D D rTrz Ernie Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr)r4r5 num_labelsrQr-rclassifier_dropoutrCr6rBrDrr9 classifierr2rPrQrrRs rSr5z'ErnieForSequenceClassification.__init__s   ++ ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rTNrUrvr2rVr.rWrkrxrYc |j|f|||||dd|} | d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | | j | j" S) a^ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trnr&N regressionsingle_label_classificationmulti_label_classificationr0r)rrDrrQ problem_typerr3rFrKrnrsqueezerrrrrr)rPrUrvr2rVr.rWrkrxrprrrJrrs rSrgz&ErnieForSequenceClassification.forwards0$**   ))'%'       ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   rTr)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrs *..2.2-1,0-1&*B <<$&B  t+B  t+ B ||d* B llT) B ||d*B  t#B +,B  u|| 7 7B B rTrcJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jy)Nr&) r4r5r-rrrCr6rBrDrr9rr2rs rSr5zErnieForMultipleChoice.__init__su  ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rTNrUrvr2rVr.rWrkrxrYc ||jdn|jd} |!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f|||||dd|} | d} |j | } |j | } | jd| } d}|t }|| |}t|| | j| jS)a9 input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr&r0Trnr) r_rrJrrDrrrrr)rPrUrvr2rVr.rWrkrx num_choicesrprrreshaped_logitsrJrrs rSrgzErnieForMultipleChoice.forwards`-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $**   ))'%'       ]3 / ++b+6  ')HOV4D("!//))   rTr)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrs *..2.2-1,0-1&*U <<$&U  t+U  t+ U ||d* U llT) U ||d*U  t#U +,U  u|| 8 8U U rTrcJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyNFr~)r4r5rr-rrrCr6rBrDrr9rr2rs rSr5z$ErnieForTokenClassification.__init__Vs   ++%@ )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rTNrUrvr2rVr.rWrkrxrYc J|j|f|||||dd|} | d} |j| } |j| } d} |Moduler)rofloatrrrrrrrrrrrr rr-rIrPrYrxr|rrrrrrr__all__rNrTrSrs*%! AA&!CC)J9   G&6MMI5,   H %JbiiJf!% II% <<% % << % LL4' % T\ %%'(%8F)F)RJ)"))J)Z bii .RYY.>   ")) @+@F ")) 299" BII   299 D/?//2  ^6%^6 ^6B  7 7  7& 9BII 9 ` .`  ` F!ryy! Z +_Z  Z zP +P P f&ryy& P %9P  P f T %9T  T ne 1e e P? "6? ? DI 4I I X rT