qidZddlZddlmZddlmZmZmZddlmZ ddl m Z m Z m Z ddlmZmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZdd lmZm Z m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:e!jve<Z=Gdde)Z>Gdde8Z?Gdde(Z@Gdde3ZAGdde7ZBGdde5ZCGdde*ZDe Gd d!eZEGd"d#e6ZFGd$d%e/ZGGd&d'e.ZHGd(d)e4ZIGd*d+e+ZJGd,d-e-ZKGd.d/e1ZLGd0d1e,ZMGd2d3e2ZNGd4d5e0ZOgd6ZPy)7zPyTorch ERNIE model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCacheEncoderDecoderCache)create_bidirectional_maskcreate_causal_mask),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)BertCrossAttentionBertEmbeddings BertEncoderBertForMaskedLMBertForMultipleChoiceBertForNextSentencePredictionBertForPreTrainingBertForPreTrainingOutputBertForQuestionAnsweringBertForSequenceClassificationBertForTokenClassification BertLayerBertLMHeadModelBertLMPredictionHead BertModel BertPoolerBertSelfAttention) ErnieConfigceZdZdZfdZ d dej dzdej dzdej dzdej dzdejdzd ed ejfd Z xZ S) ErnieEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|||j|_|jr0tj|j |j |_yy)N)super__init__ use_task_idnn Embeddingtask_type_vocab_size hidden_sizetask_type_embeddings)selfconfig __class__s Y/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/ernie/modular_ernie.pyr5zErnieEmbeddings.__init__BsL  !--   (* V5P5PRXRdRd(eD % N input_idstoken_type_ids task_type_ids position_ids inputs_embedspast_key_values_lengthreturnc|||j}n|jdd}|\}} ||jdd|| |zf}|t|drT|jj |j dd} t j| d|} | j || }n:t j|t j|jj}||j|}|j|} |j| j}|| z} |j|} | | z} |jrR|:t j|t j|jj}|j!|}| |z } |j#| } |j%| } | S)NrBrr/)dimindex)dtypedevice)sizerDhasattrrBexpandshapetorchgatherzeroslongrMword_embeddingstoken_type_embeddingstoposition_embeddingsr6r; LayerNormdropout)r<rArBrCrDrErF input_shape batch_size seq_lengthbuffered_token_type_idsrW embeddingsrYr;s r?forwardzErnieEmbeddings.forwardIs  #..*K',,.s3K!, J  ,,Q0FVlIl0l-lmL  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J&(()>)E)EF "%:: "66|D"55    $ % KuzzRVRcRcRjRj k #'#<#<]#K . .J^^J/ \\*- r@)NNNNNr) __name__ __module__ __qualname____doc__r5rR LongTensor FloatTensorintTensorra __classcell__r>s@r?r2r2?sQf.226150426&'3##d*3((4/3''$. 3 &&- 3 ((4/ 3!$3 3r@r2c eZdZy)ErnieSelfAttentionNrbrcrdr@r?rmrmr@rmc eZdZy)ErnieCrossAttentionNrnror@r?rrrrrpr@rrc eZdZy) ErnieLayerNrnror@r?rtrtrpr@rtc eZdZy) ErniePoolerNrnror@r?rvrvrpr@rvc eZdZy)ErnieLMPredictionHeadNrnror@r?rxrxrpr@rxc eZdZy) ErnieEncoderNrnror@r?rzrzrpr@rzcneZdZeZdZdZdZdZdZ dZ e e e dZej fdZxZS)ErniePreTrainedModelernieT) hidden_states attentionscross_attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weightsrI)r/rIN)r4 _init_weights isinstancerxinitzeros_biasr2copy_rDrRarangerQrPrB)r<moduler>s r?rz"ErniePreTrainedModel._init_weightss f% f3 4 KK $  0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .1r@)rbrcrdr0 config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrtrmrr_can_record_outputsrRno_gradrrjrks@r?r|r|sXL&*#N"&#(/ U]]_//r@r|ceZdZdgZdfd Zeee ddejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzdejdzde e deejezfdZdZxZS) ErnieModelrtct|||||_d|_t ||_t ||_|r t|nd|_ |jy)NF) r4r5r=gradient_checkpointingr2r`rzencoderrvpooler post_init)r<r=add_pooling_layerr>s r?r5zErnieModel.__init__sU v& &+#)&1#F+ ->k&)D  r@NrAattention_maskrBrCrDrEencoder_hidden_statesencoder_attention_maskpast_key_values use_cachecache_positionkwargsrGc |jjr| | n|jj} nd} |jr%|jr| rt j dd} | rd| b||jjr4tt|jt|jnt|j} | | td|#|j|||j} n!||jdd} n td| \}}| |jn |j}| | jnd}| tj |||z| } |j#|||||| }|j%||||| | \}}|j&|f|||| | | |d | }|d}|j(|j)|nd}t+|||j, S) task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] NFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...)r=zDYou cannot specify both input_ids and inputs_embeds at the same timerIz5You have to specify either input_ids or inputs_embedsr)rM)rArDrBrCrErF)rrembedding_outputrrr)rrrrrrrD)last_hidden_state pooler_outputr)r= is_decoderrrtraininglogger warning_onceis_encoder_decoderr r ValueError%warn_if_padding_and_no_attention_maskrNrMget_seq_lengthrRrr`_create_attention_masksrrr r)r<rArrBrCrDrErrrrrrr\r]r^rMrFrencoder_outputssequence_output pooled_outputs r?razErnieModel.forwards12 ;; ! !%.%:  @U@UII  & &4==##p" 0)4 8V8V$L $DlZ^ZeZeFfg!5   ]%>cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TETE`!?!?!Afg  !"\\*@BX[eBentuN??%)''#9+ 261M1M)#9-"7)+ 2N2 ..'$,,   )"7#9+)%    *!,8< 8O O4UY ;-'+;;  r@c|jjrt|j||||}nt|j||}|t|j|||}||fS)N)r=rErrr)r=rEr)r=rErr)r=rr r )r<rrrrrrs r?rz"ErnieModel._create_attention_masks&sx ;; ! !/{{.-- / N7{{.-N " -%>{{.5&; & "555r@)T) NNNNNNNNNNN)rbrcrd_no_split_modulesr5rrrrRrirboolrrtupler rarrjrks@r?rrsZ%  *..2.2-1,0-1596:(,!%.2_ <<$&_  t+_  t+ _ ||d* _ llT) _ ||d*_  %||d2_ !& t 3_ _ $;_  t+_ +,_  u|| K K_  _ D 6r@rc eZdZy)ErnieForPreTrainingOutputNrnror@r?rrIrpr@rcbeZdZdddZee ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ee de eje zfdZ y)ErnieForPreTrainingcls.predictions.bias'ernie.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightNrArrBrCrDrElabelsnext_sentence_labelrrGc |j|f|||||dd| } | dd\} } |j| | \} }d}|u|st}|| jd|jj |jd}||jdd|jd}||z}t || || j| jS)a: task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` TrrBrCrDrE return_dictNrrI)lossprediction_logitsseq_relationship_logitsr~r) r}clsrviewr= vocab_sizerr~r)r<rArrBrCrDrErrroutputsrrprediction_scoresseq_relationship_score total_lossloss_fctmasked_lm_lossnext_sentence_losss r?razErnieForPreTraining.forwardSs^$**   ))'%'    *1!&48HH_m4\11  "5"A')H%&7&<&*S *..2.2-1,0-1&*37H <<$&H  t+H  t+ H ||d* H llT) H ||d*H  t#H #\\D0H +,H  u|| 8 8H H r@rc"eZdZee ddej dzdej dzdej dzdej dzdej dzdej dzdej dzd ej dzd ej dzd eej dzd edzd ej dzde ej zde e de ej e zfdZy)ErnieForCausalLMNrArrBrCrDrErrrrrrlogits_to_keeprrGc | d} |j|f|||||||| | | dd |}|j}t| tr t | dn| }|j |dd|ddf}d}| *|j d|| |jjd|}t|||j|j|j|jS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rrBrCrDrErrrrrr)logitsrr)rrrr~rrro)r}rrrhslicer loss_functionr=rrrr~rr)r<rArrBrCrDrErrrrrrrrrr~ slice_indicesrrs r?razErnieForCausalLM.forwards<  I@J A ))'%'"7#9+)A A   11 8B>SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopD0#33!//))$55   r@) NNNNNNNNNNNNr)rbrcrdrrrRrilistrrhrrrrraror@r?rrsc*..2.2-1,0-1596:&*59!%.2-.? <<$&?  t+?  t+ ? ||d* ? llT) ? ||d*?  %||d2? !& t 3?  t#? ell+d2? $;?  t+? ell*? +,? u|| @ @!? ? r@rceZdZdddZee ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzdee de eje zfdZ y)ErnieForMaskedLMrrrNrArrBrCrDrErrrrrGc @|j|f|||||||dd| } | d} |j| } d}| Ft}|| jd|jj | jd}t || | j| jS)as task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rrBrCrDrErrrrNrIrrr~r) r}rrrr=rrr~r)r<rArrBrCrDrErrrrrrrrrs r?razErnieForMaskedLM.forwards4$**   ))'%'"7#9    "!* HH_5  ')H%&7&<&*S *..2.2-1,0-1596:&*2 <<$&2  t+2  t+ 2 ||d* 2 llT) 2 ||d*2  %||d22 !& t 32  t#2 +,2  u|| ~ -2 2 r@rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForNextSentencePredictionNrArrBrCrDrErrrGc |j|f|||||dd|} | d} |j| } d} |2t} | | jdd|jd} t | | | j | j S)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trr/NrIrr)r}rrrrr~r)r<rArrBrCrDrErrrrseq_relationship_scoresrrs r?raz&ErnieForNextSentencePrediction.forward#sZ$**   ))'%'      "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   r@NNNNNNN) rbrcrdrrrRrirrrrraror@r?rr"s*..2.2-1,0-1&*D <<$&D  t+D  t+ D ||d* D llT) D ||d*D  t#D +,D  u|| : :D D r@rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForSequenceClassificationNrArrBrCrDrErrrGc |j|f|||||dd|} | d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | | j | j" S) a^ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr/N regressionsingle_label_classificationmulti_label_classificationrIr)r}r[ classifierr= problem_type num_labelsrLrRrUrhrsqueezerrrrr~r)r<rArrBrCrDrErrrrrrrs r?raz&ErnieForSequenceClassification.forwardms0$**   ))'%'       ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r@r) rbrcrdrrrRrirrrrraror@r?rrls*..2.2-1,0-1&*B <<$&B  t+B  t+ B ||d* B llT) B ||d*B  t#B +,B  u|| 7 7B B r@rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForMultipleChoiceNrArrBrCrDrErrrGc ||jdn|jd} |!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f|||||dd|} | d} |j | } |j | } | jd| } d}|t }|| |}t|| | j| jS)a9 input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr/rITrr) rQrrNr}r[rrrr~r)r<rArrBrCrDrErr num_choicesrrrreshaped_logitsrrs r?razErnieForMultipleChoice.forwards`-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $**   ))'%'       ]3 / ++b+6  ')HOV4D("!//))   r@r) rbrcrdrrrRrirrrrraror@r?rrs*..2.2-1,0-1&*U <<$&U  t+U  t+ U ||d* U llT) U ||d*U  t#U +,U  u|| 8 8U U r@rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForTokenClassificationNrArrBrCrDrErrrGc J|j|f|||||dd|} | d} |j| } |j| } d} |rs AA&CCJ   .&@@I5&-   H %=n=@ *  ,    *  0  ; /?//2U6U6p 8 P ,P fB B J: : zG %BG TE %BE PX 2X v1 "<1 h? 8? D r@