qi dZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZmZmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd lmZdd l m!Z!ddl"m#Z#m$Z$ddl%m&Z&e$jNe(Z)GddejTZ+GddejTZ,GddejTZ-GddejTZ.GddejTZ/GddejTZ0GddejTZ1Gdd eZ2Gd!d"ejTZ3Gd#d$ejTZ4Gd%d&ejTZ5Gd'd(ejTZ6e#Gd)d*eZ7e#d+,Gd-d.e7Z8e#Gd/d0e7Z9e#d1,Gd2d3e7eZ:e#d4,Gd5d6e7Z;e#Gd7d8e7Zgd=Z?y)>zPyTorch RemBERT model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging) RemBertConfigc eZdZdZfdZ d dej dzdej dzdej dzdejdzded ejf d Z xZ S) RemBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c|t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)ddy)N) padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_sizeinput_embedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandselfconfig __class__s ^/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/rembert/modeling_rembert.pyr(zRemBertEmbeddings.__init__1s !||   v::H[H[ $&<<0N0NPVPkPk#l %'\\&2H2H&JeJe%f"f&A&AvG\G\]zz&"<"<=   ELL)G)GHOOPWXej  N input_idstoken_type_idsr# inputs_embedspast_key_values_lengthreturnc||j}n|jdd}|d}||jdd|||zf}|:tj|tj|jj }||j |}|j|}||z} |j|} | | z } |j| } |j| } | S)Nr%rdtypedevice) sizer#r8zeroslongrIr-r1r/r2r6) r<rArBr#rCrD input_shape seq_lengthr1 embeddingsr/s r?forwardzRemBertEmbeddings.forwardAs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !"[[EJJtO`O`OgOghN   00;M $ : :> J"%:: "66|D)) ^^J/ \\*- r@)NNNNr) __name__ __module__ __qualname____doc__r(r8 LongTensor FloatTensorintTensorrP __classcell__r>s@r?rr.sQ $.2260426&' ##d*((4/&&-  ((4/  !$  r@rcVeZdZfdZdej dej fdZxZS) RemBertPoolerct|tj|j|j|_tj |_yN)r'r(rLinear hidden_sizedenseTanh activationr;s r?r(zRemBertPooler.__init__ds9 YYv1163E3EF '')r@ hidden_statesrEc\|dddf}|j|}|j|}|S)Nr)rarc)r<rdfirst_token_tensor pooled_outputs r?rPzRemBertPooler.forwardis6+1a40 #56  6 r@rQrRrSr(r8rXrPrYrZs@r?r\r\cs#$ U\\ellr@r\ceZdZd fd Z d dej dej dzdej dzdedzdedej dzd e fd Z xZ S) RemBertSelfAttentionNct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|j"|_||_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r'r(r`num_attention_headshasattr ValueErrorrWattention_head_size all_head_sizerr_querykeyvaluer4attention_probs_dropout_probr6 is_decoder layer_idxr<r=rxr>s r?r(zRemBertSelfAttention.__init__ss0     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F  &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$2D.- /"=*,33DNNCHHI.55dnnELLK(j"d&>&>@X@XY1a   >*j"d&>&>@X@XY1a  *7It)=)D)D{DNN=M~<^*& ;&*_FY*ZAEO..t~~>!<< Y5H5HR5PQ+dii8P8P.QQ  %/.@ --//0@b/I,,7 _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD o--r@r^NNNFN rQrRrSr(r8rXrVr booltuplerPrYrZs@r?rjrjrs#048:>(,"'.2M.||M.))D0M. %0047 M.  M.  M. t+M. M.r@rjcneZdZfdZdej dej dej fdZxZS)RemBertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr!) r'r(rr_r`rar2r3r4r5r6r;s r?r(zRemBertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r@rd input_tensorrEcr|j|}|j|}|j||z}|Sr^rar6r2r<rdrs r?rPzRemBertSelfOutput.forward7 =1  ]3 }|'CD r@rhrZs@r?rr1> U\\RWR^R^r@rceZdZd fd Z d dej dej dzdej dzdedzdedzdej dzd e ej fd Z xZ S) RemBertAttentionNcft|t|||_t ||_y)Nrx)r'r(rjr<routputrys r?r(zRemBertAttention.__init__s( (9E '/ r@rdrzr{r|r}r~rEcp|j||||||}|j|d|}|f|ddz} | S)Nrzr{r|r}r~rr)r<r) r<rdrzr{r|r}r~ self_outputsattention_outputoutputss r?rPzRemBertAttention.forwardsYyy )"7+/) !  ;;|A F#% QR(88r@r^rrrZs@r?rrs048:>(,)..2||))D0 %0047     $;  t+ u|| r@rcVeZdZfdZdej dej fdZxZS)RemBertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr^) r'r(rr_r`intermediate_sizerar hidden_actstrr intermediate_act_fnr;s r?r(zRemBertIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r@rdrEcJ|j|}|j|}|Sr^)rarr<rds r?rPzRemBertIntermediate.forwards& =1 00? r@rhrZs@r?rrs#9U\\ellr@rcneZdZfdZdej dej dej fdZxZS) RemBertOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r'r(rr_rr`rar2r3r4r5r6r;s r?r(zRemBertOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r@rdrrEcr|j|}|j|}|j||z}|Sr^rrs r?rPzRemBertOutput.forwardrr@rhrZs@r?rrrr@rceZdZd fd Z ddej dej dzdej dzdej dzdedzdedzd ej dzd e ej fd Z d Z xZ S) RemBertLayerNcht||j|_d|_t |||_|j |_|j|_|jr,|j st|dt |||_ t||_ t||_ y)Nrz> should be used as a decoder model if cross attention is addedr)r'r(chunk_size_feed_forward seq_len_dimr attentionrwadd_cross_attentionrpcrossattentionr intermediaterrrys r?r(zRemBertLayer.__init__$s '-'E'E$)&)< ++#)#=#=  # #?? D6)g!hii"26Y"OD /7#F+ r@rdrzr{encoder_attention_maskr|r}r~rEcD|j|||||}|d} |dd} |jrA|?t|dstd|d|j | |||||} | d} | | ddz} t |j |j|j| } | f| z} | S)N)rzr}r|r~rrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrwrorprrfeed_forward_chunkrr) r<rdrzr{rr|r}r~self_attention_outputsrrcross_attention_outputs layer_outputs r?rPzRemBertLayer.forward3s"& )/+) "0" 2!4(, ??4@4!12 =dVDDD '+&9&9 5&; /"3- ':' # 7q9  7 ;;G0  # #T%A%A4CSCSUe  /G+r@cL|j|}|j||}|Sr^)rr)r<rintermediate_outputrs r?rzRemBertLayer.feed_forward_chunkas,"//0@A{{#68HI r@r^)NNNNFN) rQrRrSr(r8rXrVr rrrPrrYrZs@r?rr#s ,$48:>;?(,)..2+||+))D0+ %0047 + !& 1 1D 8 +  + $;+ t++ u|| +\r@rceZdZfdZ ddej dej dzdej dzdej dzdedzdedzd ed ed ed ej dzd e e zfdZ xZ S)RemBertEncoderc 2t|||_tj|j |j |_tjt|jDcgc]}t||c}|_ d|_ ycc}w)NrF)r'r(r=rr_r+r`embedding_hidden_mapping_in ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r<r=ir>s r?r(zRemBertEncoder.__init__hsq  +-99V5P5PRXRdRd+e(]]uU[UmUmOn#o!L1$E#op &+#$ps,BNrdrzr{rr| use_cacher}output_hidden_states return_dictr~rEc n|jr%|jr|rtjdd}|r6|4t t |j t |j }|j|}|rdnd} |rdnd} |r|j jrdnd} t|jD]K\}}|r| |fz} |||||||}|d}|s#| |dfz} |j jsC| |dfz} M|r| |fz} | std||| | | fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r=rrrc3$K|]}|| ywr^r).0vs r? z)RemBertEncoder.forward..s = s)last_hidden_stater|rd attentionscross_attentions) rtraininglogger warning_oncer r r=rr enumeraterrr)r<rdrzr{rr|rr}rrr~all_hidden_statesall_self_attentionsall_cross_attentionsr layer_module layer_outputss r?rPzRemBertEncoder.forwardps  & &4==##p" 01,dkk2RT`hlhshsTtuO88G "6BD$5b4%64;;;Z;Zr`d(4 VOA|#$58H$H!(%&! M*!,M &9]1=M;?(,!%"'%* .2D ||D ))D0D  %0047 D !& 1 1D 8 D  D $;D  D #D D  t+D  : :D r@rcVeZdZfdZdej dej fdZxZS)RemBertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r'r(rr_r`rarrrr transform_act_fnr2r3r;s r?r(z'RemBertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr@rdrEcl|j|}|j|}|j|}|Sr^)rarr2rs r?rPz&RemBertPredictionHeadTransform.forwards4 =1 --m< }5 r@rhrZs@r?rrs$UU\\ellr@rcVeZdZfdZdej dej fdZxZS)RemBertLMPredictionHeadcnt|tj|j|j |_tj|j |j|_t|j|_ tj|j |j|_ yr)r'r(rr_r`output_embedding_sizerar*decoderr rrcr2r3r;s r?r(z RemBertLMPredictionHead.__init__sz YYv1163O3OP yy!=!=v?P?PQ  !2!23f&B&BH]H]^r@rdrEc|j|}|j|}|j|}|j|}|Sr^)rarcr2rrs r?rPzRemBertLMPredictionHead.forwards@ =1  6 }5  ]3 r@rhrZs@r?rrs$_U\\ellr@rcVeZdZfdZdej dej fdZxZS)RemBertOnlyMLMHeadcBt|t||_yr^)r'r(r predictionsr;s r?r(zRemBertOnlyMLMHead.__init__s 26:r@sequence_outputrEc(|j|}|Sr^)r)r<rprediction_scoress r?rPzRemBertOnlyMLMHead.forwards ,,_=  r@rhrZs@r?rrs#;!u||! !r@rc2eZdZUeed<dZdZfdZxZS)RemBertPreTrainedModelr=rembertTct||t|trZt j |j tj|j jdjdyy)Nr%r$) r' _init_weightsrrinitcopy_r#r8r9rr:)r<moduler>s r?rz$RemBertPreTrainedModel._init_weightss[ f% f/ 0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h 1r@) rQrRrSr__annotations__base_model_prefixsupports_gradient_checkpointingrrYrZs@r?r r s! !&*#iir@r a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introceZdZdfd ZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzde dzde dzde dzdejdzde ezfdZxZS) RemBertModelct||||_t||_t ||_|r t|nd|_|jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r'r(r=rrOrencoderr\pooler post_init)r<r=add_pooling_layerr>s r?r(zRemBertModel.__init__sM   +F3%f- /@mF+d  r@c.|jjSr^rOr-r<s r?get_input_embeddingsz!RemBertModel.get_input_embeddings s...r@c&||j_yr^r)r<rus r?set_input_embeddingsz!RemBertModel.set_input_embeddingss*/'r@NrArzrBr#rCr{rr|rr}rrr~rEc | | n|jj} | | n|jj} | | n|jj} |jjr| | n|jj } nd} | | t d|#|j|||j}n!||jdd}n t d|\}}| |jn |j}|dn|j}|tj|||zf|}|&tj|tj|}|j||}|jjrE|C|j\}}}||f}|tj||}|j!|}nd}|j#|||||}|j%|||||| | | | | }|d}|j&|j'|nd}| s ||f|d dzSt)|||j*|j,|j.|j0 S) NFzDYou cannot specify both input_ids and inputs_embeds at the same timer%z5You have to specify either input_ids or inputs_embedsr)rIrG)rAr#rBrCrD) rzr{rr|rr}rrr~r)r pooler_outputr|rdrr)r=r}ruse_return_dictrwrrp%warn_if_padding_and_no_attention_maskrJrIget_seq_lengthr8onesrKrLget_extended_attention_maskinvert_attention_maskrOrrrr|rdrr)r<rArzrBr#rCr{rr|rr}rrr~kwargsrMrrNrIrDextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrrgs r?rPzRemBertModel.forwards$2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T&5&=?CaCaCc  !"ZZ*jCY6Y)ZdjkN  !"[[EJJvVN150P0PQ_al0m ;; ! !&;&G=R=W=W=Y :  7$68O#P %-).4HQW)X&.2.H.HI_.` +.2 +??%)'#9 + ,, 2"7#B+/!5#)'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   r@)T) NNNNNNNNNNNNN)rQrRrSr(r r"rr8rUrVr rrXrrrPrYrZs@r?rrsY /0.226260426:>;?(,!%)-,0#'.2_ ##d*_ ((4/_ ((4/ _ &&- _ ((4/ _  %0047_ !& 1 1D 8_ _ $;_  $;_ #Tk_ D[_  t+_ = =!_ _ r@rcleZdZfdZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde e zfdZ xZS)RemBertForMaskedLMct|||jrtj dt |d|_t||_|jy)NznIf you want to use `RemBertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr r'r(rwrwarningrr rclsrr;s r?r(zRemBertForMaskedLM.__init__xsR     NN1  $FeD %f- r@cB|jjjSr^r9rrrs r?get_output_embeddingsz(RemBertForMaskedLM.get_output_embeddingsxx##+++r@c:||jj_yr^r;r<new_embeddingss r?set_output_embeddingsz(RemBertForMaskedLM.set_output_embeddings'5$r@NrArzrBr#rCr{rlabelsr}rrrEc | | n|jj} |j|||||||| | |  } | d}|j|}d}|Ft }||j d|jj |j d}| s|f| ddz}||f|zS|St||| j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) rzrBr#rCr{rr}rrrr%rlosslogitsrdr) r=r%r r9rrr*rrdr)r<rArzrBr#rCr{rrCr}rrr+rrr masked_lm_lossloss_fctrs r?rPzRemBertForMaskedLM.forwards,&1%;?*.)-,0#'5 ##d*5 ((4/5 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5  $;5 #Tk5 D[5   5 5 r@r4zS RemBERT Model with a `language modeling` head on top for CLM fine-tuning. c!eZdZfdZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd ejdzde dzde dzde dzde dzde ejzdeezfdZxZS)RemBertForCausalLMct|||jstj dt |d|_t||_|jy)NzOIf you want to use `RemBertForCausalLM` as a standalone, add `is_decoder=True.`Fr6r7r;s r?r(zRemBertForCausalLM.__init__sL    NNl m#FeD %f- r@cB|jjjSr^r;rs r?r<z(RemBertForCausalLM.get_output_embeddingsr=r@c:||jj_yr^r;r?s r?rAz(RemBertForCausalLM.set_output_embeddingsrBr@NrArzrBr#rCr{rr|rCrr}rrlogits_to_keeprEc | | n|jj} |j||||||||| | | |  }|d}t|tr t | dn|}|j |dd|ddf}d}| *|jd|| |jjd|}| s|f|ddz}||f|zS|St|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, RemBertForCausalLM, RemBertConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/rembert") >>> config = RemBertConfig.from_pretrained("google/rembert") >>> config.is_decoder = True >>> model = RemBertForCausalLM.from_pretrained("google/rembert", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```N) rzrBr#rCr{rr|rr}rrr)rGrCr*r)rFrGr|rdrrr)r=r%r rrWslicer9 loss_functionr*rr|rdrr)r<rArzrBr#rCr{rr|rCrr}rrrOr+rrd slice_indicesrGrFrs r?rPzRemBertForCausalLM.forwards)R&1%SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopDY,F)-)9TGf$ Ev E0#33!//))$55   r@)NNNNNNNNNNNNNr)rQrRrSr(r<rArr8rUrVr rrWrXrrrPrYrZs@r?rKrKsr ,6.226260426:>;?(,*.!%)-,0#'-.M ##d*M ((4/M ((4/ M &&- M ((4/ M  %0047M !& 1 1D 8M M   4'M $;M  $;M #TkM D[M ell*M " 2 2#M M r@rKz RemBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c eZdZfdZe ddej dzdej dzdejdzdej dzdej dzdejdzd edzd edzd edzd e e zfd Z xZ S) RemBertForSequenceClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr^ r'r( num_labelsrr rr4classifier_dropout_probr6r_r` classifierrr;s r?r(z)RemBertForSequenceClassification.__init__6si   ++#F+ zz&"@"@A ))F$6$68I8IJ r@NrArzrBr#rCrCr}rrrEc >| | n|jj} |j|||||||| } | d} |j| } |j | } d}||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&|| j|j}n|| |}n|jj dk(r=t}|| jd|j |jd}n,|jj dk(rt}|| |}| s| f| ddz}||f|zS|St!|| | j"| j$ S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrzrBr#rCr}rrr regressionsingle_label_classificationmulti_label_classificationr%rrE)r=r%r r6rZ problem_typerXrHr8rLrWrsqueezerrrrrdr)r<rArzrBr#rCrCr}rrr+rrgrGrFrIrs r?rPz(RemBertForSequenceClassification.forward@s(&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ,, ))%'/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   r@rb) rQrRrSr(rr8rVrUrrrrPrYrZs@r?rdrds/337261526*.)-,0#'W $$t+W ))D0W ((4/ W ''$. W ((4/ W   4'W  $;W #TkW D[W  * *W W r@rdc eZdZfdZe ddej dzdej dzdejdzdej dzdej dzdejdzd edzd edzd edzd e e zfd Z xZ S)RemBertForTokenClassificationc0t|||j|_t|d|_t j |j|_t j|j|j|_ |jyNFr6rWr;s r?r(z&RemBertForTokenClassification.__init__sk   ++#FeD zz&"@"@A ))F$6$68I8IJ r@NrArzrBr#rCrCr}rrrEc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|s&1%rsE AA&!CC)9   .6,0   H %1 1j BII c.299c.N  ryy< "))  BII A-AHM RYYM bRYY" bii "!!i_ii  w )w  w tL /L L ^ a /a  a H P '=P  P fc 5c c L> $:> > BJ "8J J Z r@