qiM%dZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'jVe,Z-Gddej\Z/Gddej\Z0Gddej\Z1Gddej\Z2Gddej\Z3Gddej\Z4Gddej\Z5Gdd eZ6Gd!d"ej\Z7e%Gd#d$e!Z8Gd%d&e8Z9e%Gd'd(e8Z:e%d)*Gd+d,e8eZ;e%Gd-d.e8Z<e%d/*Gd0d1e8Z=e%Gd2d3e8Z>e%Gd4d5e8Z?gd6Z@y)7zPyTorch UMT5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torchdynamo_compilingloggingtorch_compilable_check) UMT5Configc&eZdZdfd ZdZxZS) UMT5LayerNormct|tjt j ||_||_y)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s X/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/umt5/modeling_umt5.pyr$zUMT5LayerNorm.__init__7s1 ll5::k#:; #c|jtjjdj dd}|tj ||j zz}|jjtjtjfvr%|j|jj}|j|zS)NT)keepdim) tor&float32powmeanrsqrtr)r(dtypefloat16bfloat16)r* hidden_statesvariances r.forwardzUMT5LayerNorm.forward?s !##EMM266q9>>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r/)gư>)__name__ __module__ __qualname__r$r> __classcell__r-s@r.r!r!6s $ +r/r!c*eZdZdeffd ZdZxZS)UMT5DenseActDenseconfigc^t|tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_ yNFbias)r#r$rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr*rFr-s r.r$zUMT5DenseActDense.__init__Qsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r/c|j|}|j|}|j|}t|jj t jr|j|jj jk7r`|jj jt jk7r/|j|jj j}|j |}|SN) rNrTrR isinstancerOr(r&Tensorr9int8r4r*r<s r.r>zUMT5DenseActDense.forwardXs . /  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r/r?r@rArr$r>rBrCs@r.rErEPs/z/ r/rEc*eZdZdeffd ZdZxZS)UMT5DenseGatedActDenserFct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yrH)r#r$rrKrLrMwi_0wi_1rOrPrQrRr rSrTrUs r.r$zUMT5DenseGatedActDense.__init__hs IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r/c,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|SrW) rTr`rarRrXrOr(r&rYr9rZr4)r*r< hidden_gelu hidden_linears r.r>zUMT5DenseGatedActDense.forwardpshhtyy78  -0 #m3  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r/r\rCs@r.r^r^gs/z/r/r^c*eZdZdeffd ZdZxZS) UMT5LayerFFrFct||jrt||_nt ||_t |j|j|_ tj|j|_ y)Nr,)r#r$ is_gated_actr^DenseReluDenserEr!rLlayer_norm_epsilon layer_normrrPrQrRrUs r.r$zUMT5LayerFF.__init__s_    "8"@D "3F";D 'FzUMT5LayerFF.forwards=??=9../?@% 5E(FF r/r\rCs@r.rfrfs7z7r/rfc eZdZdZddedzffd ZdejdejfdZdZ dd Z dd ejd ejdzd e dzd ejdzdejdzf dZ xZ S) UMT5Attentionz7 T5's attention using relative_attention_bias. N layer_idxct||j|_||_|j|_|j |_|j |_|j|_|j|_ |j|_ |j|jz|_ ||_|9|jr-tj!d|j"j$dt'j(|j |jd|_t'j(|j |jd|_t'j(|j |jd|_t'j(|j|j d|_|jr0t'j2|j|j|_yy)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrI)r#r$ is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerLd_kvkey_value_proj_dim num_headsn_headsrQrR inner_dimrqlogger warning_oncer-r?rrKqkvo Embeddingrelative_attention_bias)r*rFrtrqr-s r.r$zUMT5Attention.__init__sd  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<FFq!QPQRr/cd}|j}|j}|jsC|dz}||dkDjtj |zz }t j |}n*t j|t j| }|dz}||k}t j|j|z tj||z z }|||z z}||jtj z}t j|t j||dz }|t j|||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr1r)rurvrsr4r&longabsmin zeros_likelogfloatmath full_likewhere) r*relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r._relative_position_bucketz'UMT5Attention._relative_position_bucketsA*99 ;;  A K !2Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $y0II/557)CDtxxP\_hPhGii y!89 %.ejj1I%I"%*YY &8RT_bcTc(d& " EKK2CE_``r/c| |jjj}|.tj|tj |dddf}n |dddf}tj|tj |dddf}||z }|j |}|j|} | jgdjd} | S)z%Compute binned relative position biasN)r9device)r1rrr) rr(rr&arangerrr unsqueeze) r* query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluess r. compute_biaszUMT5Attention.compute_biass >1188??F  !$||L SYZ[\^b[bc -ag6 ,,zFSTXZ[T[\+.>>#'#A#ABS#T --.FG *44Q7 r/r<encoder_hidden_statespast_key_valuesattention_maskrc,|jdd\}}|du}|j|} | j|d|j|jj dd} d} |Qt |trA|jj|j} |r |j} n|j} n|} |r|n|} |rK|I| rG| j|jj} | j|jj}n|j!| } |j#| }| j|d|j|jj dd} |j|d|j|jj dd}|T|s|nd}| j%| ||jd|i\} }|r)t |trd|j|j<t'j(| | j dd}|||j+zn|}| jd}|j,s;t'j.d|j||f|j0|j2 }n1|j5|||j0| }|dddd| dddf}|||z}|}||z }t6j8j;|j=d j?|}t6j8jA||j@|jB }t'j(||}|j ddjE}|j||d}|jG|}||fS) Nr1r2rFrTr)rr9)rrdim)ptraining)$shaper~rrzrx transposerXr is_updatedgetrqcross_attention_cacheself_attention_cachelayerskeysrrrupdater&matmulget_seq_lengthrtzerosrr9rr functionalsoftmaxrtype_asrRr contiguousr)r*r<rrrr batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_statesscoresreal_seq_lengthr position_biasposition_bias_masked attn_weights attn_outputs r.r>zUMT5Attention.forwards_"/!4!4Ra!8 J3$>vvm, #((RtG^G^_iijkmno   &:oGZ+[(3377GJ!'6'L'L$'6'K'K$#2 2D.- /"=*-44T^^DIIJ/66t~~FMML/J66.1L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL*7It+?+F+F dnn?OQ_>`,( L&*_FY*ZAEO..t~~>lJ,@,@A,FGL[Kf*'E'E'GGlv%%b) //!KKDLL*j9&--W]WcWcM!--FMMR`.M*!Q a*?@M  %)N:M,&&}},,V\\^,DLLVT }},,\T\\TXTaTa,b ll<> !++Aq1<<> !&&z:rB ff[) L((r/)FN)NNNNNN)r?r@rA__doc__intr$r&rYrrrr r>rBrCs@r.rprpskSSWZk6%,, - ^ $6:(,.2.2 Q)||Q) %||d2Q) Q)  t+ Q)  t+ Q)r/rpc:eZdZddedzffd Z ddZxZS)UMT5LayerSelfAttentionNrqct|t|d||_t |j |j |_tj|j|_ y)NTrtrqrh) r#r$rp SelfAttentionr!rLrkrlrrPrQrRr*rFrqr-s r.r$zUMT5LayerSelfAttention.__init__PsN *6t_hi'FzUMT5LayerSelfAttention.forwardVsj $}=-- )+) .  & 5Ea5H(II  "%5ab%99r/rW)NNNr?r@rArr$r>rBrCs@r.rrOs#7#*7 r/rc<eZdZddedzffd Z ddZxZS)UMT5LayerCrossAttentionNrqct|t|d||_t |j |j |_tj|j|_ y)NFrrh) r#r$rpEncDecAttentionr!rLrkrlrrPrQrRrs r.r$z UMT5LayerCrossAttention.__init__jsO ,VQVbkl'FzUMT5LayerCrossAttention.forwardpsl $}=// "7)+) 0 %t||4DQ4G'HH /$4QR$88r/rWrrrCs@r.rris&7#*7# r/rcBeZdZddedzffd Z ddZxZS) UMT5BlockNrqcnt||j|_tj|_|j j t|||jr&|j j t|||j j t|y)Nrq) r#r$rsr ModuleListlayerappendrrrfrs r.r$zUMT5Block.__init__sz  ++]]_  09MN ?? JJ  5f R S +f-.r/c |jd||||\}} |jtjk(r}tj|jj } tj tj|j| dz | } tj|| | }d} |jxr|du} | r|jd|||||\}} |jtjk(r}tj|jj } tj tj|j| dz | } tj|| | }|jd|}|jtjk(r}tj|jj } tj tj|j| dz | } tj|| | }|f}|r|| | fz }|S)Nrri)rmaxrrr2) rr9r&r:finforrisinfanyclamprs)r*r<rrencoder_attention_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrs r.r>zUMT5Block.forwards,94::a= )+) , ( (   %-- / M$7$78< >Gr/rW)NNNNFFNrrCs@r.rrs//#*/"#7r/rc`eZdZdZdeffd ZdejdejfdZxZ S)UMT5ClassificationHeadz-Head for sentence-level classification tasks.rFc(t|tj|j|j|_tj |j|_tj|j|j|_ y)N)r) r#r$rrKrLdenserPclassifier_dropoutrR num_labelsout_projrUs r.r$zUMT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r/r<rc|j|}|j|}tj|}|j|}|j |}|SrW)rRrr&tanhrr[s r.r>zUMT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r/) r?r@rArrr$r&rYr>rBrCs@r.rrs/7EzE U\\ellr/rcteZdZUeed<dZdZdZdgZdgZ e dZ e jdZdZy ) UMT5PreTrainedModelrF transformerTrrOcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r&tensorrr)r*r  input_mask dummy_inputss r.rz UMT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r/cr|jj}t|tr$t j |j |dzy t|ttttfrt j|jj d|dzt|drE|jjs/t j|jj d|dzt|drpt j|j j d||jj"dzzt j$|j j&y y t|t(rft|drYt j|j*j d|dzt j$|j*j&y y t|t,r9t j|j.j d||jj"dzzt|j.dr?|j.j&)t j$|j.j&t j|j0j d||jj"dzzt|j0drA|j0j&*t j$|j0j&y y y t|t2r9t j|j4j d||jj"dzzt|j4dr?|j4j&)t j$|j4j&t j|j6j d||jj8dzzt|j6drA|j6j&*t j$|j6j&y y y t|t:rt j|j<j d||jj"dzzt|j<dr?|j<j&)t j$|j<j&t j|j>j d||jj"dzzt|j>dr?|j>j&)t j$|j>j&t j|j6j d||jj8dzzt|j6drA|j6j&*t j$|j6j&y y y t|t@rP|jj"}|jjB}|jjD}t j|jFj d|||zdzzt j|jHj d||dzzt j|jJj d||dzzt j|jLj d|||zdzz|jNr3t j|jPj d||dzzy y y ) zInitialize the weights?g)r7stdlm_head qa_outputs classifierrJN))rFinitializer_factorrXr!init constant_r( UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrLzeros_rJUMT5ForTokenClassificationrrrrrErNrOrMr^r`rarprwryr~rrrrtr)r*modulefactorrLrxrzs r. _init_weightsz!UMT5PreTrainedModel._init_weightss>// fm , NN6==&3, 7  , (    LL--CVc\ Jvy)$++2Q2Q V^^22&3,Ov|, V..55CVPTP[P[PcPchlOlEmn F--223- : ;v|, V..55CVc\R F--223- 6 7 LL,,3Ft{{GZGZ_cFc%>%@#qr'"$:&!  PQ Q&&'8D'@,O  r/N)r?r@rAr__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr&no_gradr'r1r/r.rrsb %&*#!$ !F U]]_@p@pD!r/rcBeZdZfdZdZ ddZxZS) UMT5Stackc t||tj|j|j |_|j|_tjt|jDcgc]}t||c}|_ t|j |j|_tj |j"|_d|_|j)ycc}w)NrrhF)r#r$rr vocab_sizerL embed_tokensrsrrange num_layersrblockr!rkfinal_layer_normrPrQrRgradient_checkpointing post_init)r*rFir-s r.r$zUMT5Stack.__init__Ls  LL):):FNNK ++]]ERXRcRcLd#eqIf$B#ef -fnn&B[B[ \zz&"5"56 ',#  $fs7C9c||_yrW)r?r*new_embeddingss r.set_input_embeddingszUMT5Stack.set_input_embeddingsXs *r/c 6||n|jj}||n|jj}| | n|jj} | | n|jj} |$|"|j rdnd} t d| d| d|&|j}|jd|d}n8||jdd}n"|j rdnd} t d| d| d |jr%|jr|rtjd d }|(|j t d |j|}|\}}|d ur|j st d|d|j rf|rr|p|jjr5tt!|jt!|j}n%t!|j}n|j sd}||j#nd}| %t%j&|||z|j(} |1t+s'||z}t%j,|||j(}|j rt/|j||| |}n]|Y|ddddddf}|j1|j2}d|z t%j4|j2j6z}nd}|j rO|M|j\}}}||f}|!t%j,||j(}|j9|}nd}| rdnd}|rdnd}|r|j rdnd}|j;|}t=|j>D]D\}}| r||fz}||||||||| }|d}|s&||dfz }|j s<||dfz }F|jA|}|j;|}| r||fz}| stCd|||||fDStE|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer2zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz$UMT5Stack.forward..s = s)last_hidden_staterr< attentionscross_attentions)#rFrroutput_hidden_statesuse_return_dictrsr,rrrDrr|r}r?is_encoder_decoderr r rr&rrrr'rr4r9rrinvert_attention_maskrR enumeraterBrCtupler) r*r rrrrNrrrrU return_dictrkwargserr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_length causal_maskencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsr<rF layer_module layer_outputss r.r>zUMT5Stack.forward[s"+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii ??_4;;11&9$DKK8,dkk:Z'O'3$++&FO#OETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D4zAO"ZZ OML`L`aN ??,{{+-- / K '(D$)9:K%..}/B/B.CK, MrBrCs@r.r<r<Ks6 + "#!^ r/r<ceZdZUdZdZeed<dddZfdZdZ dZ e dd e jd zd e jd zd e jd zd e jd zdeee jd zded zde j$d zde j$d zded zded zded zded zde jd zdee jezfdZxZS)rao Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rF shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightct||tj|j|j |_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_|j!yNFT)r#r$rrr>rLr copydeepcopyrsrr<encodernum_decoder_layersrAdecoderrEr*rFencoder_configdecoder_configr-s r.r$zUMT5Model.__init__s  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0  r/c|jSrWr r*s r.get_input_embeddingszUMT5Model.get_input_embeddings( {{r/c~||_|jj||jj|yrWr rvrJrxrHs r.rJzUMT5Model.set_input_embeddings,-$  )).9 )).9r/Nr rr r encoder_outputsrrNdecoder_inputs_embedsrrrUr[rrc F| | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}|j||||||| | | | |  }| s||zSt|j|j|j|j|j|j|j|jS) ah input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr rrNrrUr[rrr1rRr<rS r rrNrrrrrrUr[r)rRrdecoder_hidden_statesdecoder_attentionsrTencoder_last_hidden_staterencoder_attentions)rFrrVrvrXrlenrxrrRrr<rSrT)r*r rr r rrrNrrrrUr[rr\r<decoder_outputss r.r>zUMT5Model.forward1sRF"+!6IDKKrBrCs@r.rrs "J '6'6 $: .23759:>BF(,-159!%)-,0#'26s ##d*s ))D0s !++d2 s !& 0 04 7 s uU%6%6784? s s ||d*s  %||d2s $;s  $;s #Tks D[s ((4/s u  !$6 6!s s r/rz< UMT5 Model with a `language modeling` head on top. ) custom_introc!eZdZdZdZddddZfdZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzdedzde jdzde jdzde jdzdedzdedzdedzdedzde jdzdee jezfdZde jfdZxZS)ra Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```rmrn)rprqzlm_head.weightct|||j|_t j |j |j|_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_t j"|j|j d|_|j'y)NFTrI)r#r$rL model_dimrrr>r rtrursrr<rvrwrArxrKrrErys r.r$z%UMT5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0 yy1B1BO  r/c|jSrWr}r~s r.rz1UMT5ForConditionalGeneration.get_input_embeddingsrr/c~||_|jj||jj|yrWrrHs r.rJz1UMT5ForConditionalGeneration.set_input_embeddingsrr/Nr rr r rrrNrlabelsrrrUr[rrc | | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}| |||j| }|j||||||| | | | | }|d}|jjr||jdzz}|j|}d}| ^td }| j|j} ||jd |j!d | jd }| s|f|ddz|z}||f|zS|St#|||j$|j&|j(|j*|j,|j&|j( S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrrr1rrrr) ignore_indexr2 losslogitsrrrrTrrr)rFrrVrvrXrrr1rxr"rrrr4rrrrrr<rSrTrR)r*r rr r rrrNrrrrrUr[rr\r<rsequence_output lm_logitsrloss_fctoutputs r.r>z$UMT5ForConditionalGeneration.forwards"N"+!6IDKKrrBrCs@r.rrs  J'6'6) ,: .23759:>=A(,26:>*.!%)-,0#'26N ##d*N ))D0N !++d2 N !& 0 04 7 N uU\\23d: N N ((4/N  %0047N   4'N $;N  $;N #TkN D[N ((4/N " u  !O 3#N N b)ELL)r/rceZdZdZdZddiZfdZdZdZe dd e jdzd e jdzd e jdzd e dzd e dzde dzdee jezfdZxZS)ra Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rmrprnct||tj|j|j |_tj|}d|_ d|_ t||_ |jy)NF)r#r$rrr>rLr rtrurrWr<rvrE)r*rFrzr-s r.r$zUMT5EncoderModel.__init__sb  ll6#4#4fnnE v.#( ,1) 0  r/c|jSrWr}r~s r.rz%UMT5EncoderModel.get_input_embeddingsrr/cH||_|jj|yrW)r rvrJrHs r.rJz%UMT5EncoderModel.set_input_embeddingss$  )).9r/Nr rrNrrUr[rc h||n|jj}|j||||||}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rFrVrv) r*r rrNrrUr[r\rs r.r>zUMT5EncoderModel.forwardsHF&1%rBrCs@r.rrzs  J & :.23726)-,0#',##d*,))D0,((4/ ,  $; , #Tk ,D[, u  !O 3,,r/rz UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c~eZdZdgZdeffd Ze ddejdzdejdzdejdzdejdzd e ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde dzde ezfdZxZS)UMT5ForSequenceClassificationFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrFct||t||_t ||_|j yrW)r#r$rr rclassification_headrErUs r.r$z&UMT5ForSequenceClassification.__init__s6  $V,#9&#A  r/Nr rr r rrNrrrrrUr[rc | | n|jj} |d} |$|"td|jj| || t d|j |}|j|||||||| | | |  }|d}|j|jjj|j}ttj|jdj!dk(d|j"\}}}||ddfj%|d |ddd ddf}|j'|}d}||j|j}|jj(|jj*dk(rd |j_nv|jj*dkDrL|j,tj.k(s|j,tj0k(rd |j_nd |j_|jj(d k(rSt3}|jj*dk(r&||j5|j5}n|||}n|jj(d k(rGt7}||j%d |jj*|j%d }n,|jj(d k(rt9}|||}| s|f|ddz}||f|zS|St;|||j<|j>|j@|jB|jD|jF|jH S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rr r rrNrrrrUr[rrz7All examples must have the same number of tokens.r2 regressionsingle_label_classificationmulti_label_classificationr)%rFrVNotImplementedErrorr-r?r,r1r eq eos_token_idr4rrr&unique_consecutivesumnumelrrr problem_typerr9rrrsqueezerrrrrrrTrrr)r*r rr r rrNrrrrrUr[r\rreos_maskrrdr+sentence_representationrrrrs r.r>z%UMT5ForSequenceClassification.forwardsN`&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9+'"7/!5##  "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r/) NNNNNNNNNNNN)r?r@rA"_keys_to_ignore_on_load_unexpectedrr$rr&rrYlistrrrZrr>rBrCs@r.rrsU+s)s&z.2.259:>:>26:>*.!%)-,0#'A ##d*A  t+A !++d2 A !& 0 04 7 A e//047 A ((4/A  %0047A   4'A $;A  $;A #TkA D[A  0 0A A r/rceZdZdgZdeffd Ze ddejdzdejdzdejdzdejdzd e dzd e dzd e dzd e eje zfd Z xZ S)r$rrFc,t|||j|_t||_t j |j|_t j|j|j|_ |jyrW) r#r$rrr rrPrrRrKr+rrErUs r.r$z#UMT5ForTokenClassification.__init__rsj   +++F3zz&";";< ))F$6$68I8IJ r/Nr rrNrrrUr[rc ||n|jj}|j||||||} | d} |j| } |j | } d} |z"UMT5ForTokenClassification.forward}s6&1%rBrCs@r.r$r$ms*r)s& z *..2-1&*)-,0#'6 <<$&6  t+6 ||d* 6  t# 6  $; 6 #Tk6 D[6  u|| 4 46 6 r/r$ceZdZdddZfdZdZdZe ddejdzdejdzd ejdzd ejdzd e e ejdzd ejdzd ejdzdejdzdejdzdedzdedzdedzdedzde ejezfdZxZS)rrnroc$t|||j|_t j |j |j|_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_|j"|_t j$|j|j"|_|j)yrs)r#r$rLrrrr>r rtrursrr<rvrwrArxrrKrrErys r.r$z!UMT5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0  ++))FNNF4E4EF r/c|jSrWr}r~s r.rz-UMT5ForQuestionAnswering.get_input_embeddingsrr/c~||_|jj||jj|yrWrrHs r.rJz-UMT5ForQuestionAnswering.set_input_embeddingsrr/Nr rr r rstart_positions end_positionsrNrrrrUr[rc | | n|jj} | | n|jj} ||d} | | | td|j |}| | n|jj} | | n|jj} ||j |||| | | }nI| rGt |ts7t|dt|dkDr|dndt|dkDr|dnd}|d}|j||| d||| | | | }|d}|j|}|jdd \}}|jd j}|jd j}d}||t|jdkDr*|jd j|j }t|jdkDr*|jd j|j }|jd}|j#d|}|j#d|}t%| }|||}|||}||zdz }| s||f|ddz|z}||f|zS|St'||||j(|j*|j,|j.|j0|j*|j, S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrrrrr1r) r rrNrrrrrrUr[r2rr) r start_logits end_logitsrrrrTrrr)rFrVrr,r1rvrXrrrxrsplitrrrr4rrrrrr<rSrTrR)r*r rr r rrrrNrrrrUr[r\r<rrrrr total_loss ignored_indexr start_lossend_lossrs r.r>z UMT5ForQuestionAnswering.forwards0\&1%)F  U !% 1 1) < !*!6IDKKrBrCs@r.rrs(7'6 .: .23759:>=A371526:>!%)-,0#'I ##d*I ))D0I !++d2 I !& 0 04 7 I uU\\23d: I ))D0I ''$.I ((4/I  %0047I $;I  $;I #TkI D[I u  !$G G!I I r/r)rrrrr$rr)Arrtrr&rtorch.nnrrrrMrr activationsr cache_utilsr r r generationr masking_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrconfiguration_umt5r get_loggerr?r|Moduler!rEr^rfrprrrrrr<rrrrr$r__all__r:r/r.rs AA&!CC)/9.+   H %+BII+4 .RYY<"))$u)BIIu)pRYY4bii8B*BLRYY$j!/j!j!Zn #n bh #h h V J)#6J)  J)ZX*XXv N $7N  N bG !4G G Tp 2p p f r/