qidZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZmZmZmZmZmZmZdd lmZdd lmZmZd d lmZd d lmZmZmZm Z m!Z!m"Z"ejFe$Z%GddejLZ'GddejLZ(GddejLZ)GddejLZ*GddejLZ+GddejLZ,GddejLZ-GddejLZ.GddejLZ/eGd d!eZ0eGd"d#e0Z1eGd$d%e0Z2Gd&d'ejLZ3ed()Gd*d+e0Z4eGd,d-e0Z5eGd.d/e0Z6Gd0d1ejLZ7eGd2d3e0Z8d6d4Z9gd5Z:y)7zPyTorch I-BERT model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)gelu))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) IBertConfig)IntGELU IntLayerNorm IntSoftmaxQuantActQuantEmbedding QuantLinearc2eZdZdZfdZ ddZdZxZS)IBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. czt||j|_d|_d|_d|_d|_d|_t|j|j|j|j|j|_ t|j|j|j|j|_|jdt!j"|j$j'dd |j|_t|j$|j|j(|j|j|_t-|j|j |_t-|j|j |_t3|j|j4|j|j|j6 |_t-|j |j |_t=j>|j@|_!y) N ) padding_idx weight_bit quant_mode)r$r% position_idsrF) persistentr%eps output_bitr% force_dequant)"super__init__r% embedding_bitembedding_act_bitact_bit ln_input_bit ln_output_bitr vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddingsregister_buffertorcharangemax_position_embeddingsexpandr#position_embeddingsrembeddings_act1embeddings_act2rlayer_norm_epsr. LayerNormoutput_activationrDropouthidden_dropout_probdropoutselfconfig __class__s Z/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/ibert/modeling_ibert.pyr0zIBertEmbeddings.__init__2s  ++!# -      ++))  &4  " "F$6$64CUCUbfbqbq& "  ELL)G)GHOOPWXej  "..#1  * *   (()) $   ((>(>4??['(>(>4??[%   %%)) ..  "*$,,4??!Szz&"<"<= c|D|1t||j|j|j}n|j |}||j }n|j dd}|:t j|t j|jj}||j|\}}nd}|j|\}} |j|||| \} } |j|\} } |j| | | | \} } |j| | \} } |j| } |j!| | \} } | | fS)Nr(dtypedeviceidentityidentity_scaling_factor)"create_position_ids_from_input_idsr#torS&create_position_ids_from_inputs_embedssizer=zeroslongr&r9r;rBrArErIrF)rK input_idstoken_type_idsr& inputs_embedspast_key_values_length input_shapeinputs_embeds_scaling_factorr;$token_type_embeddings_scaling_factor embeddingsembeddings_scaling_factorrA"position_embeddings_scaling_factors rNforwardzIBertEmbeddings.forwardcs  $At//1G "Y%%& $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN  :>:N:Ny:Y 7M7+/ (FJF`F`aoFpCC040D0D  (*$H 1E1 - -CGBZBZ[gBh??040D0D  %($F 1E1 - -15zKd0e- -\\*- 040F0FzSl0m- -444rOc|jdd}|d}tj|jdz||jzdztj|j }|j dj|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr(rrQr)rZr=r>r#r\rS unsqueezer@)rKr_rasequence_lengthr&s rNrYz6IBertEmbeddings.create_position_ids_from_inputs_embedss$((*3B/ %a.||   q /D4D4D"Dq"HPUPZPZcpcwcw %%a(// <drs,5\=rOrc*eZdZfdZ ddZxZS)IBertSelfAttentionc Ft||j|jzdk7r2t |ds&t d|jd|jd|j |_d|_d|_d|_ |j|_t|j|jz |_ |j|jz|_ t|j|jd|j|j|j d |_t|j|jd|j|j|j d |_t|j|jd|j|j|j d |_t#|j|j |_t#|j|j |_t#|j|j |_t#|j|j |_t-j.|j0|_t5|j|j |j6 |_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rr"Tbiasr$bias_bitr% per_channelr*r%r.)r/r0r7num_attention_headshasattr ValueErrorr%r$rxr3intattention_head_size all_head_sizerquerykeyvaluerquery_activationkey_activationvalue_activationrFrrGattention_probs_dropout_probrIrr.softmaxrJs rNr0zIBertSelfAttention.__init__s     : : :a ?PVXhHi#F$6$6#78 445Q8 !++  #)#=#= #&v'9'9Fu;;:>((=Rn:o77>BjjXt>u;;372G2G ?3 / //3.A.A/Sq.r+ +262G2G ?3 / / %2$7$7! J!&&z2t7O7OQUQiQijtt q NN:r43K3KTMeMefppqrtuv !&&z2t7O7OQUQiQijtt q !<< Y5H5HR5PQ $223+e3 ??.HKc.cfk.k +.2 +  %/.@ ;?,, =; 77 ,,7 _kB ) 5+ILf+f (+/ (%--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD 7;6L6L 77 3 37H=/2mM]!*+I J.0  ---rONFrkrlrmr0rgrorps@rNrrrrs5pv K.rOrrc$eZdZfdZdZxZS)IBertSelfOutputc t||j|_d|_d|_d|_d|_d|_t|j|jd|j|j |jd|_ t|j |j|_ t|j|j|j|j|j|_t|j|j|_t%j&|j(|_yNrr"r!Trvr*r+)r/r0r%r3r$rxr4r5rr7denser ln_input_actrrDr.rErFrrGrHrIrJs rNr0zIBertSelfOutput.__init__*s  ++        ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rOc|j||\}}|j|}|j||||\}}|j||\}}|j ||\}}||fSNrTrrIrrErFrKrr input_tensorinput_tensor_scaling_factors rNrgzIBertSelfOutput.forwardG6:jjPl6m3 3 ]3 6:6G6G  (!$? 7H7 3 3 7;nn]Tp6q3 36:6L6L 77 3 3:::rOrrps@rNrr) >:;rOrc*eZdZfdZ ddZxZS)IBertAttentionct||j|_t||_t ||_yN)r/r0r%rrrKroutputrJs rNr0zIBertAttention.__init__Ys3  ++&v. %f- rOc|j||||\}}|j|d|d||\}}|f|ddz} |f|ddz} | | fS)Nrr)rKr) rKrrrr self_outputsself_outputs_scaling_factorattention_outputattention_output_scaling_factorroutputs_scaling_factors rNrgzIBertAttention.forward_s59II  (   5 1 1 =AKK O8;]Lh= 99$% QR(88"A!CFabcbdFe!e...rOrrrps@rNrrXs. /rOrc$eZdZfdZdZxZS)IBertIntermediatec t||j|_d|_d|_d|_t |j|jd|j|j |jd|_ |jdk7r tdt|j|j|_t|j|j|_y) Nrr"Trvr z3I-BERT only supports 'gelu' for `config.hidden_act`rzr*)r/r0r%r3r$rxrr7intermediate_sizer hidden_actr}rr.intermediate_act_fnrrFrJs rNr0zIBertIntermediate.__init__us  ++       $ $]]      &RS S#*dooU[UiUi#j !)$,,4??!SrOc|j||\}}|j||\}}|j||\}}||fSr)rrrF)rKrrs rNrgzIBertIntermediate.forwardsa6:jjPl6m3 36:6N6N 77 3 3 7;6L6L 77 3 3:::rOrrps@rNrrtsT( ;rOrc$eZdZfdZdZxZS) IBertOutputc t||j|_d|_d|_d|_d|_d|_t|j|jd|j|j |jd|_ t|j |j|_ t|j|j|j|j|j |_t|j|j|_t'j(|j*|_yr)r/r0r%r3r$rxr4r5rrr7rrrrrDr.rErFrrGrHrIrJs rNr0zIBertOutput.__init__s  ++    $ $   ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rOc|j||\}}|j|}|j||||\}}|j||\}}|j ||\}}||fSrrrs rNrgzIBertOutput.forwardrrOrrps@rNrrrrOrc0eZdZfdZ ddZdZxZS) IBertLayercXt||j|_d|_d|_t ||_t||_t||_ t|j|j|_ t|j|j|_ y)Nrrr*)r/r0r%r3 seq_len_dimr attentionr intermediaterrrpre_intermediate_actpre_output_actrJs rNr0zIBertLayer.__init__s}  ++ '/-f5!&) $,T\\doo$V!&t||PrOc|j||||\}}|d}|d}|dd} |j||\} } | f| z} | S)N)rrr)rfeed_forward_chunk) rKrrrrself_attention_outputs%self_attention_outputs_scaling_factorrrr layer_outputlayer_output_scaling_factors rNrgzIBertLayer.forwardsIM  ( / IWI E E 2!4*OPQ*R'(,484K4K =5 1 1 /G+rOc|j||\}}|j||\}}|j||\}}|j||||\}}||fSr)rrrr)rKrrintermediate_output"intermediate_output_scaling_factorrrs rNrzIBertLayer.feed_forward_chunks<@z'IBertEncoder.forward..$s  = s)last_hidden_stater attentionscross_attentions) enumeratertupler ) rKrrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsall_cross_attentionsi layer_module layer_outputss rNrgzIBertEncoder.forwards#7BD$5b4#(4 POA|#$58H$H!(,! M*!,M &9]1=Mt j|jt j|jt|dd t j|jyyt|ttj frt j |j d|jj|j"Ct|j dds,t j|j |j"t|dd?t j|j$t j|jyyt|t&tj(frlt j|jt j*|j t|d d t j|j,yyt|t.r t j|jyt|t0rZt j2|j4t7j8|j4j:d j=d yt|t>r`t j@|jBd t j@|jDd t j|jFyy)zInitialize the weightsg)meanstdNweight_integer bias_integer_is_hf_initializedFweight_scaling_factorshiftr(r'gh㈵gh㈵>)$ isinstancerrrinitnormal_weightrLinitializer_rangerwzeros_getattrrfc_scaling_factorrr Embeddingr#rrrEones_r IBertLMHeadrcopy_r&r=r>rr@r constant_x_minx_maxact_scaling_factor)rKmodules rN _init_weightsz"IBertPreTrainedModel._init_weightsKsJ f{BII6 7 LLSdkk6S6S T{{& FKK(v/6B F112 F445v~t4@ F//0A  > ? LLSdkk6S6S T!!-gfmmMach6i FMM&*<*<=>v6=I F889 F112Jr|| < = KK $ JJv}} %vw-9 FLL):  , KK $  0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  ) NN6<< / NN6<< . KK11 2*rONctd)Nz6`resize_token_embeddings` is not supported for I-BERT.)NotImplementedError)rKnew_num_tokenss rNresize_token_embeddingsz,IBertPreTrainedModel.resize_token_embeddingsms!"Z[[rOr) rkrlrmr__annotations__base_model_prefixr=no_gradr(r,rrOrNr r Fs/ U]]_33B\rOr c,eZdZdZdfd ZdZdZe ddejdzdejdzdejdzd ejdzd ejdzd e dzd e dzd e dzde e ejzfdZxZS) IBertModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. ct||||_|j|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r/r0rLr%rrdrencoderrpooler post_init)rKrLadd_pooling_layerrMs rNr0zIBertModel.__init__|sZ    ++)&1#F+ ->k&)D  rOc.|jjSrrdr9rKs rNget_input_embeddingszIBertModel.get_input_embeddingss...rOc&||j_yrr8)rKrs rNset_input_embeddingszIBertModel.set_input_embeddingss*/'rONr]rr^r&r_rrrreturnc  J||n|jj}||n|jj}||n|jj}| | t d|#|j |||j } n!||j dd} n t d| \} } | |jn |j} |tj| | f| }|&tj| tj| }|j|| }|j||||\}}|j||||||}|d}|j|j|nd}|s ||f|d dzSt!|||j"|j$|j& S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer(z5You have to specify either input_ids or inputs_embeds)rSrQ)r]r&r^r_)rrrrrr)r pooler_outputrrr)rLrruse_return_dictr}%warn_if_padding_and_no_attention_maskrZrSr=onesr[r\get_extended_attention_maskrdr3r4r rrr)rKr]rr^r&r_rrrkwargsrarrrSextended_attention_maskembedding_outputembedding_output_scaling_factorencoder_outputssequence_outputr s rNrgzIBertModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T  !"ZZ*j)A6RN  !"[[EJJvVN150P0PQ_al0m<@OO%)' =L= 99 ,,  +2/!5# ' *!,8< 8O O4UY #]3oab6II I;-')77&11,==   rO)T)NNNNNNNN)rkrlrmrnr0r:r<rr= LongTensor FloatTensorboolr rrgrorps@rNr1r1qs"/0.237260426)-,0#'B ##d*B ))D0B ((4/ B &&- B ((4/ B  $;B #TkB D[B  6e>O>O8P PB B rOr1cPeZdZdddZfdZdZdZe ddejdzd ejdzd ejdzd ejdzd ejdzd ejdzde dzde dzde dzde e ejzfdZxZS)IBertForMaskedLMz(ibert.embeddings.word_embeddings.weight$z lm_head.bias)zlm_head.decoder.weightzlm_head.decoder.biasct||t|d|_t ||_|j yNF)r6)r/r0r1rr!lm_headr5rJs rNr0zIBertForMaskedLM.__init__s6  %@ "6*  rOc.|jjSr)rQdecoderr9s rNget_output_embeddingsz&IBertForMaskedLM.get_output_embeddingss||###rOc\||j_|j|j_yr)rQrSrw)rKnew_embeddingss rNset_output_embeddingsz&IBertForMaskedLM.set_output_embeddingss - *// rONr]rr^r&r_labelsrrrr=c | | n|jj} |j|||||||| } | d} |j| } d}|Ft }|| j d|jj |j d}| s| f| ddz}||f|zS|St|| | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr^r&r_rrrrr(rlosslogitsrr) rLr@rrQrrr6r rr)rKr]rr^r&r_rXrrrrDrrIprediction_scoresmasked_lm_lossloss_fctrs rNrgzIBertForMaskedLM.forwards(&1%->!?@ rOc |j|}t|}|j|}|j|}|Sr)rr rerS)rKfeaturesrDxs rNrgzIBertLMHead.forward/s; JJx  G OOA  LLOrOrkrlrmrnr0rgrorps@rNr!r!$s3ArOr!z I-BERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introc:eZdZfdZe ddej dzdejdzdej dzdej dzdejdzdej dzd edzd edzd edzd e e ejzfd Z xZ S)IBertForSequenceClassificationct|||j|_t|d|_t ||_|jyrP)r/r0 num_labelsr1rIBertClassificationHead classifierr5rJs rNr0z'IBertForSequenceClassification.__init__AsC   ++%@ 1&9 rONr]rr^r&r_rXrrrr=c | | n|jj} |j|||||||| } | d} |j| } d}||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt}|j dk(r&|| j|j}n|| |}n|jjdk(r=t}|| jd|j |jd}n,|jjdk(rt}|| |}| s| f| d dz}||f|zS|St|| | j | j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrZrr regressionsingle_label_classificationmulti_label_classificationr(rr[)rLr@rrq problem_typerorRr=r\r~rsqueezerrrrrrrKr]rr^r&r_rXrrrrDrrIr]r\r`rs rNrgz&IBertForSequenceClassification.forwardKs(&1%?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ** *..,/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rOra) rkrlrmr0rr=rJrKrLr rrgrorps@rNrzrzs.22637*.0426)-,0#'V ##d*V ((4/V ))D0 V   4' V &&- V ((4/V  $;V #TkV D[V  #U5+<+<%= =V V rOrzc:eZdZfdZe ddej dzdejdzdej dzdej dzdejdzdej dzd edzd edzd edzd e e ejzfd Z xZ S)IBertForTokenClassificationc0t|||j|_t|d|_t j |j|_t j|j|j|_ |jyrP) r/r0ror1rrrGrHrIrr7rqr5rJs rNr0z$IBertForTokenClassification.__init__sk   ++%@ zz&"<"<= ))F$6$68I8IJ rONr]rr^r&r_rXrrrr=c | | n|jj} |j|||||||| } | d} |j| } |j | } d}|rs" AA&.,,cc   H %s=biis=lC.C.L,;bii,;^/RYY/8; ;D,;")),;^5959p6 2996 r "))  '\?'\'\Td %d d NG +G G T")), M %9M  M `b 1b b J> "6> > Bbii&I 4I I X4" rO