qiXPdZddlZddlmZmZmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZddlmZd d lmZmZmZmZmZmZmZmZGd d eZGd deZGddeZ GddeZ!GddeZ"GddeZ#GddeZ$GddeZ%gdZ&y)zPyTorch CamemBERT model.N)BCEWithLogitsLossCrossEntropyLossMSELoss),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)Unpack)TransformersKwargsauto_docstring)can_return_tuple)RobertaForCausalLMRobertaForMaskedLMRobertaForMultipleChoiceRobertaForQuestionAnswering RobertaForSequenceClassificationRobertaForTokenClassification RobertaModelRobertaPreTrainedModelceZdZdZy)CamembertPreTrainedModelrobertaN)__name__ __module__ __qualname__base_model_prefixa/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/camembert/modular_camembert.pyrr,s!r#rc eZdZy)CamembertModelN)rrr r"r#r$r&r&0sr#r&cteZdZdddZfdZee ddejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzde e de ejezfdZxZS)CamembertForMaskedLMz)roberta.embeddings.word_embeddings.weightz lm_head.bias)zlm_head.decoder.weightzlm_head.decoder.biascLt|||`t|d|_yNF)add_pooling_layersuper__init__ camembertr&rselfconfig __class__s r$r.zCamembertForMaskedLM.__init__:#  N%fF r#N input_idsattention_masktoken_type_ids position_ids inputs_embedsencoder_hidden_statesencoder_attention_masklabelskwargsreturnc t|j|f||||||dd| } | d} |j| } d} |a|j| j}t }|| j d|j j|j d} t| | | j| jS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)r6r7r8r9r:r; return_dictrNlosslogits hidden_states attentions) rlm_headtodevicerviewr2 vocab_sizer rErF)r1r5r6r7r8r9r:r;r<r=outputssequence_outputprediction_scoresmasked_lm_lossloss_fcts r$forwardzCamembertForMaskedLM.forward@s:$,,   ))%'"7#9    "!* LL9  YY0778F')H%&7&<&;?*.5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 5 r#r(c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) "CamembertForSequenceClassificationcLt|||`t|d|_yr*r,r0s r$r.z+CamembertForSequenceClassification.__init__{r4r#Nr5r6r7r8r9r<r=r>c |j|f||||dd|}|d} |j| } d} ||j| j}|jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |j dk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd |j |jd } n,|jj dk(rt} | | |} t!| | |j"|j$ S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Tr6r7r8r9r@rN regressionsingle_label_classificationmulti_label_classificationrArB)r classifierrHrIr2 problem_type num_labelsdtyperSlongintrsqueezerrJrr rErF r1r5r6r7r8r9r<r=rLrMrDrCrPs r$rQz*CamembertForSequenceClassification.forwards6$,,  ))%'   "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r#NNNNNN)rrr r.rrrSrTrUrrrVrWr rQrXrYs@r$r[r[zsG .237260426*.C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C C r#r[c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) CamembertForMultipleChoicecLt|||`t|d|_yr*r,r0s r$r.z#CamembertForMultipleChoice.__init__r4r#Nr5r7r6r<r8r9r=r>c "||jdn|jd}|!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |1|jd|jd|jdnd} |j| f| | | | dd|}|d}|j |}|j |}|jd|}d}|.|j |j}t}|||}t|||j|jS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nr_rAT)r8r7r6r9r@rB) shaperJsizerdropoutrcrHrIrr rErF)r1r5r7r6r<r8r9r= num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrL pooled_outputrDreshaped_logitsrCrPs r$rQz"CamembertForMultipleChoice.forwardsV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  $,,  *..,      ]3 / ++b+6  YY556F')HOV4D("!//))   r#rk)rrr r.rrrSrTrUrrrVrWr rQrXrYs@r$rmrmsG .22637*.0426P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P P r#rmc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) CamembertForTokenClassificationcLt|||`t|d|_yr*r,r0s r$r.z(CamembertForTokenClassification.__init__&r4r#Nr5r6r7r8r9r<r=r>c ~|j|f||||dd|}|d} |j| } |j| } d} |W|j| j}t } | | j d|j|j d} t| | |j|jS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Tr^rNrArB) rrsrcrHrIrrJrer rErFrjs r$rQz'CamembertForTokenClassification.forward,s2$,,  ))%'   "!*,,71  YYv}}-F')HFKKDOOc |j|f||||dd|} | d} |j| } | jdd\} } | jdj } | jdj } d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|| |}||zd z }t|| | | j| j S) a[ token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Tr^rr_rA)dimN) ignore_indexr)rC start_logits end_logitsrErF) r qa_outputssplitri contiguouslenrrclamprr rErF)r1r5r6r7r8r9rrr=rLrMrDrr total_loss ignored_indexrP start_lossend_losss r$rQz%CamembertForQuestionAnswering.forwardjs0$,,  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   r#)NNNNNNN)rrr r.rrrSrTrUrrrVrWr rQrXrYs@r$rrcsG .2372604263715> ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> > r#rc eZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd e e ejdzd e dzd ejdzde ejzde ede ejezfdZxZS)CamembertForCausalLMcLt|||`t|d|_yr*r,r0s r$r.zCamembertForCausalLM.__init__r4r#Nr5r6r7r8r9r:r;r<past_key_values use_cachecache_positionlogits_to_keepr=r>c  |d} |j|f||||||| | | dd | }|j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t|||j|j|j|jS)aq token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, CamembertForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("almanach/camembert-base") >>> config = AutoConfig.from_pretrained("almanach/camembert-base") >>> config.is_decoder = True >>> model = CamembertForCausalLM.from_pretrained("almanach/camembert-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) r6r7r8r9r:r;rrrr@)rDr<rK)rCrDrrErFcross_attentionsr")rlast_hidden_state isinstancerhslicerG loss_functionr2rKrrrErFr)r1r5r6r7r8r9r:r;r<rrrrr=rLrE slice_indicesrDrCs r$rQzCamembertForCausalLM.forwardsb  I@L  A ))%'"7#9+) A  A  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD0#33!//))$55   r#) NNNNNNNNNNNr)rrr r.rrrSrTrUrVboolrWrhrrrrQrXrYs@r$rrs}G .237260426:>;?*.BF!%.2-.Q ##d*Q ))D0Q ((4/ Q &&- Q ((4/ Q  %0047Q !& 1 1D 8Q   4'Q uU%6%6784?Q $;Q  t+Q ell*Q +,Q  u|| @ @Q Q r#r)rr(rmrr[r}r&r)'__doc__rStorch.nnrrrmodeling_outputsrrr r r r r processing_utilsrutilsrr utils.genericrroberta.modeling_robertarrrrrrrrrr&r(r[rmr}rr__all__r"r#r$rs AA'7-   "5" \ C -C LL )IL ^Y !9Y x; &C; |G $?G TZ -Z z r#