qiQIdZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZmZmZdd lmZdd lmZd d lmZmZmZmZmZd dlmZd dlmZddlm Z GddeZ!eGddeZ"GddeZ#GddeZ$eGdde"Z%edGdde"e Z&Gd d!eZ'Gd"d#eZ(Gd$d%eZ)gd&Z*y)'zPyTorch PLBART model.N)nn)CrossEntropyLoss)initialization)Cache)GenerationMixin)BaseModelOutputSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)auto_docstring)BartClassificationHead BartDecoder BartEncoderBartForCausalLMBartScaledWordEmbedding)'BigBirdPegasusForSequenceClassification)shift_tokens_right) PLBartConfigc eZdZy)PLBartScaledWordEmbeddingN__name__ __module__ __qualname__[/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/plbart/modular_plbart.pyrr,rrcFeZdZUeed<dZdZddgZdZdZ dZ fdZ xZ S)PLBartPreTrainedModelconfigmodelTPLBartDecoderLayerPLBartEncoderLayerct||t|tr t j |j yyN)super _init_weights isinstancePLBartForConditionalGenerationinitzeros_final_logits_bias)selfmodule __class__s r r+z#PLBartPreTrainedModel._init_weights:s2 f% f< = KK00 1 >r) rrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnr+ __classcell__r3s@r r#r#0s< &*#-/CDN22rr#c eZdZy) PLBartEncoderNrrrr r>r>@r!rr>c eZdZy) PLBartDecoderNrrrr r@r@Dr!rr@ceZdZdddZdeffd ZdZdZe dde jdzd e jdzd e jdzd e jdzd e e jdzd edzde jdzde jdzdedzdedzdedzdedzde jdzdee jezfdZxZS) PLBartModelz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightr$cJt|||j|j}}|jrt j |jnd}t||j|||_ t||_ t||_ |jy)Ng?) embed_scale)r*__init__ pad_token_id vocab_sizescale_embeddingmathsqrtd_modelrsharedr>encoderr@decoder post_init)r1r$ padding_idxrGrDr3s r rEzPLBartModel.__init__Osz  "("5"5v7H7HZ 393I3Idii/s / FNNKepq $V, $V,  rc|jSr))rL)r1s r get_input_embeddingsz PLBartModel.get_input_embeddings[s {{rc~||_|j|j_|j|j_yr))rLrM embed_tokensrN)r1values r set_input_embeddingsz PLBartModel.set_input_embeddings^s) $(KK !$(KK !rN input_idsattention_maskdecoder_input_idsdecoder_attention_maskencoder_outputspast_key_values inputs_embedsdecoder_inputs_embeds use_cacheoutput_attentionsoutput_hidden_states return_dictcache_positionreturnc | | n|jj} | | n|jj} | | n|jj} | | n|jj} |"| t ||jj }||j|||| | | }nI| rGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|j|||d|||| | | | |  }| s||zSt|j|j|j|j |j"|j|j|j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)rWrXr]r`rarbrrr)last_hidden_state hidden_states attentions) rWrXencoder_hidden_statesencoder_attention_maskr\r]r_r`rarbrc)rfr\decoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_stateriencoder_attentions)r$r`rar_use_return_dictrrFrMr,r lenrNr rfr\rgrhrm)r1rWrXrYrZr[r\r]r^r_r`rarbrckwargsdecoder_outputss r forwardzPLBartModel.forwardcsP2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK)F 29dkk>V>V W   ""ll#-+"3%9' +OO_!M-"1!"4474H14Loa0RV14_1E1I?1-tO,,'1"1!"4#1+//!5#)'  "_4 4!-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r) NNNNNNNNNNNNN)rrr_tied_weights_keysrrErRrVr torch LongTensorTensorlist FloatTensorrbooltupler rtr;r<s@r rBrBHs~(7'6 | 0 .226596::>(,26:>!%)-,0#'26_ ##d*_ ((4/_ !++d2 _ !& t 3 _ e//047 _ _ ((4/_  %0047_ $;_  $;_ #Tk_ D[_ ((4/_ u|| 1 1!_ _ rrBzv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introc!:eZdZdZdgZddiZdeffd Z dded edzd e d e jffd Z ded dfd Z e ddej dzdej dzdej dzdej"dzdeej&dzdedzdej&dzdej&dzdej"dzde dzde dzde dzde dzdej dzd eej"ezfdZdej"fdZxZS) r-r%r0zlm_head.weightzmodel.shared.weightr$cxt||t||_|j dt j d|jjjftj|j|jjjd|_ |jy)Nr0rF)bias)r*rErBr%register_bufferrvzerosrLnum_embeddingsrLinearrKlm_headrO)r1r$r3s r rEz'PLBartForConditionalGeneration.__init__s~   (  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  rNnew_num_tokenspad_to_multiple_of mean_resizingrdczt||||}|j|jjd|S)Nr)r*resize_token_embeddings_resize_final_logits_biasweightshape)r1rrrnew_embeddingsr3s r rz6PLBartForConditionalGeneration.resize_token_embeddingss?8I[]jk &&~'<'<'B'B1'EFrc6|jjd}||kr|jddd|f}nStjd||z f|jj}tj |j|gd}|j d|y)Nr)device)dimr0)r0rrvrrcatr)r1rold_num_tokensnew_bias extra_biass r rz8PLBartForConditionalGeneration._resize_final_logits_biass//55b9 ^ +--a..@AHa.)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;rrWrXrYrZr[r\r]r^labelsr_r`rarbrcc | | n|jj} | $|"| t| |jj}|j ||||||||| | | | | }|j |d}||j j|jz}d}| Ft}||jd|jj| jd}| s|f|ddz}||f|zS|St|||j|j|j|j |j"|j$|j& S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N) rXrYr[rZr\r]r^r_r`rarbrcrrr) losslogitsr\rkrlrmrnriro)r$rprrFr%rr0torrviewrGr r\rkrlrmrnriro)r1rWrXrYrZr[r\r]r^rr_r`rarbrcrroutputs lm_logitsmasked_lm_lossloss_fctoutputs r rtz&PLBartForConditionalGeneration.forwardstD&1%(,26:>&*!%)-,0#'26l ##d*l ((4/l !++d2 l !& t 3 l e//047 l l ((4/l  %0047l  t#l $;l  $;l #Tkl D[l ((4/l " u||  .#l l \DELLDrr-c eZdZy)PLBartClassificationHeadNrrrr rr]r!rrceZdZfdZxZS)PLBartForSequenceClassificationc :tjdi|y)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr*rt super_kwargsr3s r rtz'PLBartForSequenceClassification.forwardbs4 ','r)rrrrtr;r<s@r rras ((rrc(eZdZefdZxZS)PLBartForCausalLMc :tjdi|y)aF labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nrrrs r rtzPLBartForCausalLM.forwards0 ','r)rrrr rtr;r<s@r rrs((rr)rr-rrBr#)+__doc__rIrvrtorch.nnrrr. cache_utilsr generationrmodeling_outputsr r r modeling_utilsr utilsr bart.modeling_bartrrrrr(bigbird_pegasus.modeling_bigbird_pegasusrmbart.modeling_mbartrconfiguration_plbartrrr#r>r@rBr-rrr__all__rrr rs %& ) .#_5. 7  2O 2 2 K  K z 'z z z OD%:OOD  ODd 5 (&M(<((8 r