qi\ldZddlZddlmZddlmZmZmZddlmZ ddl m Z ddl m Z dd lmZmZmZmZmZdd lmZdd lmZmZd d lmZej6eZGddej<ZGddej<Z ejBjDdZ#ejBjDdZ$ejBjDdZ%ejBjDdZ&ejBjDdejNde(fdZ)ejBjDdejNdejNfdZ*ejBjDdejNdejNde(fdZ+ejBjDdejNdejNfdZ,Gddej<Z-Gd d!ej<Z.Gd"d#ej<Z/Gd$d%ej<Z0Gd&d'ej<Z1Gd(d)e Z2Gd*d+ej<Z3eGd,d-eZ4eGd.d/e4Z5Gd0d1ej<Z6Gd2d3ej<Z7Gd4d5ej<Z8Gd6d7ej<Z9Gd8d9ej<Z:eGd:d;e4Z;Gd<d=ej<Z<ed>?Gd@dAe4Z=eGdBdCe4Z>eGdDdEe4Z?gdFZ@y)GzPyTorch DeBERTa model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) DebertaConfigc*eZdZdZdfd ZdZxZS)DebertaLayerNormz2LayerNorm module (epsilon inside the square root).ct|tjt j ||_tjt j||_||_ yN) super__init__r Parametertorchonesweightzerosbiasvariance_epsilon)selfsizeeps __class__s ^/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/deberta/modeling_deberta.pyrzDebertaLayerNorm.__init__)sH ll5::d#34 LLT!23 #cX|j}|j}|jdd}||z jdjdd}||z t j ||j zz }|j|}|j|z|jz}|S)NT)keepdim) dtypefloatmeanpowrsqrtr!torr )r" hidden_states input_typer.varianceys r&forwardzDebertaLayerNorm.forward/s"(( %++- !!"d!3!D(--a055b$5G&-HtG\G\<\1]] %((4 KK- '$)) 3r')g-q=__name__ __module__ __qualname____doc__rr6 __classcell__r%s@r&rr&s<$ r'rc$eZdZfdZdZxZS)DebertaSelfOutputct|tj|j|j|_t |j|j|_tj|j|_ yr) rrrLinear hidden_sizedenserlayer_norm_eps LayerNormDropouthidden_dropout_probdropoutr"configr%s r&rzDebertaSelfOutput.__init__;s\ YYv1163E3EF )&*<*S>STzz&"<"<= r'cr|j|}|j|}|j||z}|SrrCrHrEr"r2 input_tensors r&r6zDebertaSelfOutput.forwardA7 =1  ]3 }|'CD r'r8r9r:rr6r<r=s@r&r?r?:s > r'r?c|jd}|jd}tj|tj|j}tj|tj|j}|dddf|j ddj |dz }|d|ddf}|jd}|S)a Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] r,deviceNrr)r)r#rarangelongrTviewrepeat unsqueeze) query_layer key_layer query_sizekey_sizeq_idsk_ids rel_pos_idss r&build_relative_positionraHs$!!"%J~~b!H LL5::k>P>P QE LLI>;++A. 0@0@0C[EUEUVWEXZfZkZklnZop qqr'c|j|jd|jd|jd|jdgS)NrrrRrc)rerZr[s r&p2c_dynamic_expandrijsG >>;++A. 0@0@0CY^^TVEWYbYgYghjYkl mmr'c|j|jdd|jd|jdfzS)Nr+rRrc) pos_indexp2c_attr[s r&pos_dynamic_expandrmos=   GLLN2A.)..2DinnUWFX1YY ZZr'rZ scale_factorctjtj|jdtj|zS)Nr)r,)rr0tensorr#r-)rZrns r&scaled_size_sqrtrrws0 ::ell;#3#3B#7u{{KlZ [[r'r[cd|jd|jdk7r t||S|SNrR)r#ra)rZr[rfs r& build_rposru|s1y~~b11&{I>>r'max_relative_positionsc tjtt|j d|j d|Srt)rrqminmaxr#)rZr[rvs r&compute_attention_spanrzs4 <<C 0 0 4innR6HIKab ccr'c |jd|jdk7rA|dddddddfjd}tj|dt |||S|S)NrRrr)r+dimindex)r#rYrgatherrm)rlrZr[rfrks r&uneven_size_correctedrs_y~~b11 Aq!,66r: ||G2DYPWYb2cddr'cpeZdZdZfdZdZ ddejdejdedejdzd ejdzd ejdzd e ejejdzffd Z d ejdejd ejd ejde f dZ xZ S)DisentangledSelfAttentiona Disentangled self-attention module Parameters: config (`str`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaConfig`] cZt||j|jzdk7r&t d|jd|jd|j|_t |j|jz |_|j|j z|_tj|j|jdzd|_ tjtj|jtj|_tjtj|jtj|_|j" |j"ng|_t%|d d|_t%|d d|_|j(rctj|j|jd|_tj|j|jd|_nd|_d|_|j&rt%|d d |_|j.d kr|j0|_tj2|j4|_d|j"vr1tj|j|jd|_d|j"vr/tj|j|j|_tj2|j<|_y)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()rFr rprelative_attention talking_headrvr)rc2pp2c) rrrBnum_attention_heads ValueErrorintattention_head_size all_head_sizerrAin_projrrrr-q_biasv_bias pos_att_typegetattrrrhead_logits_projhead_weights_projrvmax_position_embeddingsrFrG pos_dropoutpos_proj pos_q_projattention_probs_dropout_probrHrIs r&rz"DisentangledSelfAttention.__init__sm     : : :a ?#F$6$6#78 445Q8 $*#=#= #&v'9'9FP>PW\ ] )))"$))F,>,>@R@R"Szz&"E"EF r'c|jdd|jdfz}|j|}|jddddS)Nr)rr+rr)r#rrWpermute)r"x new_x_shapes r&transpose_for_scoresz.DisentangledSelfAttention.transpose_for_scoressFffhsmt'?'?&DD FF; yyAq!$$r'Nr2attention_maskoutput_attentions query_statesrfrel_embeddingsreturnc L|9|j|}|j|jdd\}} } n|jjj|jdzd} t dD cgc]C} t jt |jD cgc] } | | dz| zc} dE}} } t j|d|jj|dj}t j|d|jj|dj} t j|d|jj|dj}|| |fDcgc]}|j|c}\}} } ||j|jddddfz}| |j|jddddfz} d}dt|jz}t!||}||j|jz }t j|| j#dd }|j$r*|(|&|j'|}|j)|| |||}|||z}|j*5|j+|j-ddddj-dddd}|j/}|j1|t j2|jj4}t6j8j;|d}|j=|}|j>5|j?|j-ddddj-dddd}t j|| }|j-ddddjA}|jCdd d z}|jE|}|s|dfS||fScc} wcc} } wcc}w) a Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.BoolTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, *optional*): Whether return the attention matrix. query_states (`torch.FloatTensor`, *optional*): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. Nrr)r}rrprr+rR)r))#rrchunkrrrangercatmatmultr1r,rrlenrrr transposerrdisentangled_att_biasrrbool masked_fillfinforxr functionalsoftmaxrHr contiguousr#rW)r"r2rrrrfrqprZr[ value_layerwskiqkvwqvrrel_attrnscaleattention_scoresattention_probs context_layernew_context_layer_shapes r&r6z!DisentangledSelfAttention.forwardsL  m,B262K2KB2O2U2UVW]_2U2` /KK$$**4+C+Ca+GQ*OBhmnohpqcdEIIeDX$YY !D$=$=dkk$PTVW->X$YY 3t0011  l;!EHH;3D3DH$EE  << Y5H5HR5PQ  " "~'AlF^!--n=N00iWegstG  /'9   ,#445E5M5MaQRTUWX5YZbbcdfgijlmn ',,.+77.8I5;;WbWhWhKiKmKmn--//0@b/I,,7  ! ! -"44_5L5LQPQSTVW5XYaabcefhiklmO _kB %--aAq9DDF "/"4"4"6s";e"C%**+BC  !4( (//UWq3ds>+P)P; PP!PrZr[rnc $|t|||j}|jdk(r!|jdjd}nT|jdk(r|jd}n/|jdk7rt d|jt |||j }|j}||j |z |j |zddfjd}d}d|jvr|j|}|j|}tj||jdd } tj||zd|dzdz } tj| dt!| || } || z }d |jvr|j#|} |j| } | t%| |z} t'|||} tj| |zd|dzdz } tj|| jdd j)|j* }tj|dt-| || jdd }t/||||}||z }|S) Nr+rrrz2Relative position ids must be of dim 2 or 3 or 4. rr)rRr|rrp)rarTr}rYrrzrvrVrrrrrrclamprrgrrrrur1r,rir)r"rZr[rfrrnatt_spanscore pos_key_layerc2p_attrepos_query_layerr_posp2c_posrls r&rz/DisentangledSelfAttention.disentangled_att_bias"s  2; ;K]K]^L     "'11!4>>qAL    1 $'11!4L    1 $QR^RbRbRdQefg g)+y$B]B]^#((* '  ' '( 2T5P5PS[5[ []^ ^ )A,  D%% % MM.9M 55mDMll; 0G0GB0OPGkk,"91hlQ>NOGll7:LWVaco:pqG W E D%% %"oon=O"77HO /N NOE kk5&8"3Q1 q8HIGll9o.G.GB.O.R.RYbYhYh.R.ijGllR'9';PY'ZiB ,G[)\ZG W E r'FNNN)r8r9r:r;rrrTensorrtupler6rrr<r=s@r&rrs$GL%#(,0,0.2U0||U0 U0 U0 llT) U0 llT) U0 t+U0 u||U\\D00 1U0n6\\6<<6ll 6  6  6r'rc*eZdZdZfdZddZxZS)DebertaEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|t|dd}t|d|j|_t j |j|j||_t|dd|_ |jsd|_ n/t j |j|j|_ |jdkDr0t j |j|j|_ nd|_ |j|jk7r2t j|j|jd|_nd|_t!|j|j"|_t j&|j(|_||_|j/d t1j2|jj5d d y) N pad_token_idrembedding_size) padding_idxposition_biased_inputTFr position_idsrr)) persistent)rrrrBrr Embedding vocab_sizeword_embeddingsrposition_embeddingsrtype_vocab_sizetoken_type_embeddingsrA embed_projrrDrErFrGrHrJregister_bufferrrUrd)r"rJrr%s r&rzDebertaEmbeddings.__init__^sy v~q9 %f.>@R@RS!||F,=,=t?R?R`lm%,V5Ld%S"))'+D $')||F4R4RTXTgTg'hD $  ! !A %)+f6L6LdNaNa)bD &)-D &   &"4"4 4 ii(;(;V=O=OV[\DO"DO)&*<*S>STzz&"<"<=    ELL)G)GHOOPWXej  r'c||j}n|jdd}|d}||jddd|f}|:tj|tj|jj }||j |}|j |j|j }ntj|}|} |jr| |z} |j|j|} | | z} |j|j| } |j| } ||j| jk7rD|jdk(r |jdjd}|jd}|j!| j"}| |z} |j%| } | S)Nr)rrSrr+)r#rrrrVrTrr zeros_likerrrrEr}squeezerYr1r,rH) r" input_idstoken_type_idsrmask inputs_embeds input_shape seq_lengthr embeddingsrs r&r6zDebertaEmbeddings.forward}s  #..*K',,.s3K ^  ,,Q  ^$>~$N !#&;;J ?? &4J^^J/  xxzZ^^--88:?<<?2215D~~a(77:++,D#d*J\\*- r')NNNNNr7r=s@r&rr[sQ >,r'rcpeZdZfdZ ddedeejejdzffdZxZ S)DebertaAttentioncpt|t||_t ||_||_yr)rrrr"r?outputrJrIs r&rzDebertaAttention.__init__s- -f5 '/  r'Nrrcv|j||||||\}}||}|j||} |r| |fS| dfS)N)rrfr)r"r) r"r2rrrrfr self_output att_matrixattention_outputs r&r6zDebertaAttention.forwardse#'))   %%) #,#  Z  (L;;{LA $j1 1$d+ +r'r r8r9r:rrrrrr6r<r=s@r&rrsF#(, , u||U\\D00 1,r'rcVeZdZfdZdej dej fdZxZS)DebertaIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rrrrArBintermediate_sizerC isinstance hidden_actstrr intermediate_act_fnrIs r&rzDebertaIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r'r2rcJ|j|}|j|}|Sr)rCrr"r2s r&r6zDebertaIntermediate.forwards& =1 00? r'r8r9r:rrrr6r<r=s@r&rrs#9U\\ellr'rc$eZdZfdZdZxZS) DebertaOutputc t|tj|j|j |_t|j |j|_ tj|j|_ ||_ yr)rrrrArrBrCrrDrErFrGrHrJrIs r&rzDebertaOutput.__init__sc YYv779K9KL )&*<*S>STzz&"<"<=  r'cr|j|}|j|}|j||z}|SrrLrMs r&r6zDebertaOutput.forwardrOr'rPr=s@r&rrs r'rcpeZdZfdZ ddedeejejdzffdZxZ S) DebertaLayerct|t||_t ||_t ||_yr)rrr attentionr intermediaterrrIs r&rzDebertaLayer.__init__s3 )&1/7#F+ r'Nrrc|j||||||\}}|j|} |j| |} |r| |fS| dfS)Nrrrfr)rrr) r"r2rrrfrrrrintermediate_output layer_outputs r&r6zDebertaLayer.forwardsn(,~~  /%%) (6( $*#//0@A{{#68HI  *- - $' 'r')NNNFrr=s@r&rrsF,"'( ( u||U\\D00 1(r'rc eZdZdZfdZdZdZd dZ d dejdejde d e d e f d Z xZ S)DebertaEncoderz8Modified BertEncoder with relative position bias supportct|tjt |j Dcgc] }t |c}|_t|dd|_ |jrdt|dd|_ |jdkr|j|_ tj|jdz|j|_d|_ycc}w)NrFrvr)rr+)rrr ModuleListrnum_hidden_layersrlayerrrrvrrrBrgradient_checkpointing)r"rJ_r%s r&rzDebertaEncoder.__init__s ]]%H`H`Ba#bQL$8#bc ")&2F"N  " "*1&:RTV*WD '**Q..4.L.L+"$,,t/J/JQ/NPVPbPb"cD &+#$csCcR|jr|jj}|Sd}|Sr)rrr)r"rs r&get_rel_embeddingz DebertaEncoder.get_rel_embeddings07;7N7N,,33UYr'c|jdkrE|jdjd}||jdjdz}|S|jdk(r|jd}|S)Nr+rrRr)r)r}rYr)r"rextended_attention_masks r&get_attention_maskz!DebertaEncoder.get_attention_mask s    1 $&4&>&>q&A&K&KA&N #47N7V7VWY7Z7d7deg7hhN   !Q &+55a8Nr'cZ|jr||t||}|St||}|Sr)rra)r"r2rrfs r& get_rel_poszDebertaEncoder.get_rel_pos)s>  " "|';'6|]S  7}mT r'r2routput_hidden_statesr return_dictc ^|j|}|j|||}|r|fnd}|rdnd} |} |j} t|jD].\} } | | |||| |\}}|r||fz}||}n|} |s)| |fz} 0|st d||| fDSt ||| S)N)rrfrrc3&K|] }|| ywrr).0rs r& z)DebertaEncoder.forward..ZshqZ[Zghslast_hidden_stater2 attentions)rrr enumeraterrr )r"r2rrrrrfrall_hidden_statesall_attentionsnext_kvrr layer_moduleatt_ms r&r6zDebertaEncoder.forward1s00@'' |\R L`8Hfj0d//1(4 ;OA|#/))-"3 $ M5$$58H$H!', ' !/5(!:' ;*h]4E~$Vhh h+;LYg  r')NN)TFNNT) r8r9r:r;rrrrrrrr6r<r=s@r&rrshB ,&*"' , ||,  , # ,  , , r'rc`eZdZUeed<dZdgZdZejfdZ xZ S)DebertaPreTrainedModelrJdebertarTct||t|tr?t j |j t j |jyt|ttfr t j |jyt|trZt j|jtj|jj dj#dyy)zInitialize the weights.r)rN)r _init_weightsrrinitzeros_rrLegacyDebertaLMPredictionHeadDebertaLMPredictionHeadr rcopy_rrrUshaperd)r"moduler%s r&r0z$DebertaPreTrainedModel._init_weightsgs f% f7 8 KK & KK & !>@W X Y KK $  1 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h3r') r8r9r:r__annotations__base_model_prefix"_keys_to_ignore_on_load_unexpectedsupports_gradient_checkpointingrno_gradr0r<r=s@r&r-r-`s: !*?)@&&*#U]]_ i ir'r-c eZdZfdZdZdZe ddejdzdejdzdejdzdejdzd ejdzd e dzd e dzd e dzd e e zfdZ xZ S) DebertaModelct||t||_t ||_d|_||_|jyNr) rrrrrencoderz_stepsrJ post_initrIs r&rzDebertaModel.__init__vs@  +F3%f-    r'c.|jjSrrrr"s r&get_input_embeddingsz!DebertaModel.get_input_embeddingss...r'c&||j_yrrEr"new_embeddingss r&set_input_embeddingsz!DebertaModel.set_input_embeddingss*8'r'Nrrrrrrrrrc |||n|jj}||n|jj}||n|jj}| | t d|#|j |||j } n!||j dd} n t d| |jn |j} |tj| | }|&tj| tj| }|j|||||} |j| |d||} | d }|jd kDr|d }t|jDcgc]}|jj d}}|d}|jj#}|jj%|}|jj'| }|d dD]!}|||d ||| }|j)|#|d}|s|f| |rd dzSd dzSt+||r | j,nd| j.Scc}w)NzDYou cannot specify both input_ids and inputs_embeds at the same timer)z5You have to specify either input_ids or inputs_embeds)rTrS)rrrrrT)rrrrrRFr r+r#)rJrruse_return_dictr%warn_if_padding_and_no_attention_maskr#rTrrrrVrrArBrrrrrappendr r2r%)r"rrrrrrrrkwargsrrTembedding_outputencoder_outputsencoded_layersr2rlayersrrrel_posrsequence_outputs r&r6zDebertaModel.forwardsz2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ FCN  !"[[EJJvVN??)%' + ,,  !%/# ' )+ <R8\(]] ]XY8\(]] ]-;O/77UY&11  +Ks H9)NNNNNNNN)r8r9r:rrGrKrrrrrr r6r<r=s@r&r>r>ts/9*..2.2,0-1)-,0#'O <<$&O  t+O  t+ O llT) O ||d* O  $;O #TkO D[O   O O r'r>c$eZdZfdZdZxZS)$LegacyDebertaPredictionHeadTransformct|t|d|j|_t j |j|j|_t|jtrt|j|_ n|j|_ t j|j|j|_ y)Nr)r$)rrrrBrrrArCrrrr transform_act_fnrErDrIs r&rz-LegacyDebertaPredictionHeadTransform.__init__s %f.>@R@RSYYv1143F3FG f'' -$*6+<+<$=D !$*$5$5D !d&9&9v?T?TUr'cl|j|}|j|}|j|}|Sr)rCrZrErs r&r6z,LegacyDebertaPredictionHeadTransform.forwards4 =1 --m< }5 r'rPr=s@r&rXrXs Vr'rXc$eZdZfdZdZxZS)r3cJt|t||_t |d|j |_tj|j |jd|_ tjtj|j|_y)NrTr)rrrX transformrrBrrrArdecoderrrrr rIs r&rz&LegacyDebertaLMPredictionHead.__init__ss =fE%f.>@R@RSyy!4!4f6G6GdS LLV->->!?@ r'cJ|j|}|j|}|Sr)r^r_rs r&r6z%LegacyDebertaLMPredictionHead.forwards$}5  ]3 r'rPr=s@r&r3r3s Ar'r3cVeZdZfdZdej dej fdZxZS)LegacyDebertaOnlyMLMHeadcBt|t||_yr)rrr3 predictionsrIs r&rz!LegacyDebertaOnlyMLMHead.__init__s 8@r'rVrc(|j|}|Sr)rd)r"rVprediction_scoress r&r6z LegacyDebertaOnlyMLMHead.forwards ,,_=  r'rr=s@r&rbrbs$A!u||! !r'rbc(eZdZdZfdZdZxZS)r4zMhttps://github.com/microsoft/DeBERTa/blob/master/DeBERTa/deberta/bert.py#L270ct|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|jd|_ tjtj|j |_y)NT)r$elementwise_affine)rrrrArBrCrrrr rZrErDrrrrr rIs r&rz DebertaLMPredictionHead.__init__ s YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>ShlmLLV->->!?@ r'c|j|}|j|}|j|}tj||j j |jz}|Sr)rCrZrErrrrr )r"r2rs r&r6zDebertaLMPredictionHead.forwardsd =1 --m<    ]O4J4J4L4L4NORVR[R[[ r'r7r=s@r&r4r4 sW Ar'r4c$eZdZfdZdZxZS)DebertaOnlyMLMHeadcBt|t||_yr)rrr4lm_headrIs r&rzDebertaOnlyMLMHead.__init__%s .v6 r'c*|j||}|Sr)rn)r"rVrrfs r&r6zDebertaOnlyMLMHead.forward*s LL/J  r'rPr=s@r&rlrl$s 7 !r'rlc6eZdZdddZfdZdZdZe ddejdzd ejdzd ejdzd ejdzd ejdzd ejdzde dzde dzde dzde e zfdZ xZS)DebertaForMaskedLMzcls.predictions.bias)deberta.embeddings.word_embeddings.weight)zcls.predictions.decoder.biaszcls.predictions.decoder.weightct|||j|_t||_|jrt ||_nddi|_t||_ |jy)Nzlm_predictions.lm_head.weightrr) rrlegacyr>r.rbcls_tied_weights_keysrllm_predictionsrCrIs r&rzDebertaForMaskedLM.__init__6sg  mm #F+ ;;/7DH01\'D ##5V"D   ' ' -/=/B/BD   ' ' ,r'Nrrrrrlabelsrrrrc | | n|jj} |j|||||||| } | d} |jr|j | } n0|j | |jj j} d}|Ft}|| jd|jj|jd}| s| f| ddz}||f|zS|St|| | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrrrrrrrr)rlosslogitsr2r%)rJrMr.rtrurwrrrrWrr r2r%)r"rrrrrr|rrrrPoutputsrVrfmasked_lm_lossloss_fctrs r&r6zDebertaForMaskedLM.forwardSs!*&1%r.rpoolerrrrA classifierrJrGrFrHrC)r"rJrrdrop_outr%s r&rz)DebertaForSequenceClassification.__init__s  V\15 $#F+ #F+ [[++ ))J ;6=$76>6F4;;22Hzz(+  r'c6|jjSr)r.rGrFs r&rGz5DebertaForSequenceClassification.get_input_embeddingss||0022r'c:|jj|yr)r.rKrIs r&rKz5DebertaForSequenceClassification.set_input_embeddingss )).9r'Nrrrrrr|rrrrc | | n|jj} |j|||||||| } | d} |j| } |j | } |j | }d}||jj |jdk(rXtj}|jdj|j}|||jd}n_|jdk(s|jddk(r|dk\j}|j!}|jddkDrt#j$|d|j'|jd|jd}t#j$|d|jd}t)}||jd|jj+|jd}nIt#j,dj|}n#tj.d}|||zj1dj3 }n|jj dk(rIt}|jdk(r&||j5|j5}n|||}n|jj dk(r=t)}||jd|j|jd}n,|jj dk(rt7}|||}| s|f| ddz}||f|zS|St9||| j:| j< S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrrrrrrr) regressionsingle_label_classificationmulti_label_classificationr)rJrMr.rrHr problem_typerrrrWr1r,r}r#nonzerorVrrrdrr-rq LogSoftmaxsumr.rrrr2r%)r"rrrrrr|rrrrPr encoder_layerrrrloss_fn label_indexlabeled_logitsr log_softmaxrs r&r6z(DebertaForSequenceClassification.forwards(&1%#)Q;"7"7"9K#[[]F"''*Q.)."A{'9'9+:J:J1:Mv{{[\~'^*"'fa9I9I"9M!N#3#5'(;(;B(P(V(V(XZ`ZeZefhZij$||A11&9"$--"3K)&1F:??CIIKKD))\9"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./Y,F)-)9TGf$ Ev E'fG4I4IV]VhVh  r'r)r8r9r:rrGrKrrrrrrr6r<r=s@r&rrs$3:*..2.2,0-1&*)-,0#'N <<$&N  t+N  t+ N llT) N ||d* N  t#N  $;N #TkN D[N  ) )N N r'rc eZdZfdZe ddej dzdej dzdej dzdej dzdej dzdej dzd edzd edzd edzd ee zfd Z xZ S)DebertaForTokenClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr) rrrr>r.rrFrGrHrArBrrCrIs r&rz&DebertaForTokenClassification.__init__si   ++#F+ zz&"<"<= ))F$6$68I8IJ r'Nrrrrrr|rrrrc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|r.rrArB qa_outputsrCrIs r&rz$DebertaForQuestionAnswering.__init__TsS   ++#F+ ))F$6$68I8IJ r'Nrrrrrstart_positions end_positionsrrrrc &| | n|jj} |j||||||| | } | d} |j| }|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| jS) Nr~rrr)r) ignore_indexr+)r start_logits end_logitsr2r%)rJrMr.rsplitrrrr#rrr r2r%)r"rrrrrrrrrrrPrrVrrr total_loss ignored_indexr start_lossend_lossrs r&r6z#DebertaForQuestionAnswering.forward^s&1%r-)Ar;rrtorch.nnrrrrr1 activationsr modeling_layersr modeling_outputsr r r rrmodeling_utilsrutilsrrconfiguration_debertar get_loggerr8loggerModulerr?jitscriptrargrirmrrrrrurzrrrrrrrrr-r>rXr3rbr4rlrqrrrr__all__rr'r&rsg AA&!9.,0   H %ryy(  8rrnn[[\%,,\c\\ELLU\\d ddgjdd C CLN Nb,ryy,F "))  BII (-(BO RYYO di_ii&a )a a H299&BII&!ryy!bii6!!Y /Y Y x'BII', h '=h  h V; $:; ; |I "8I I X r'