qiddlZddlmZmZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZmZdd lmZmZdd lmZmZdd lmZdd lmZmZmZmZmZm Z dd l!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6e,jne8Z9GddeZ:GddejvZ<GddejvZ=Gdde5Z>edd:dZ?ee?Gd d!ejvZ@Gd"d#eZAe+Gd$d%e&ZBe+Gd&d'eBZCGd(d)ejvZDe+d*+Gd,d-eBZEe+d.+Gd/d0eBZFe+d1+Gd2d3eBZGe+Gd4d5eBZHe+d6+Gd7d8eBZIgd9ZJy);N)LiteralOptional)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)PreTrainedConfiglayer_type_validation)use_kernel_func_from_hubuse_kernelized_func)create_bidirectional_mask(create_bidirectional_sliding_window_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ROPE_INIT_FUNCTIONSRopeParameters)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)eager_attention_forward)Gemma3RotaryEmbedding rotate_halfcF8eZdZdZdZdgZdddZfdZ d2d edzd edzd edzd edzd edzde dzdedzde dzde dzde dzde dzdedzdedzdedzdedzdedzde dzde dzde e dzde edefdzdedzde dzd e dzd!e dzd"e dzd#ed$d%e dzd&e dzd'e dzd(e dzd)e dzd*edzd+e dzd,e dzfDfd- Zd3d.Zfd/Zed0Zej*d1ZxZS)4ModernBertConfigaO This is the configuration class to store the configuration of a [`ModernBertModel`]. It is used to instantiate an ModernBert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ModernBERT-base. e.g. [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the ModernBert model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`ModernBertModel`] hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 22): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu"` if not specified. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_cutoff_factor (`float`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. norm_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the normalization layers. pad_token_id (`int`, *optional*, defaults to 50283): Padding token id. eos_token_id (`int`, *optional*, defaults to 50282): End of stream token id. bos_token_id (`int`, *optional*, defaults to 50281): Beginning of stream token id. cls_token_id (`int`, *optional*, defaults to 50281): Classification token id. sep_token_id (`int`, *optional*, defaults to 50282): Separation token id. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. layer_types (`list`, *optional*): Attention pattern for each layer. rope_parameters (`dict`, *optional*): Dictionary mapping attention patterns (`"full_attention"`, `"sliding_attention"`) to `RopeParameters`. Each value should be a dictionary containing `rope_type` and optional scaling parameters. local_attention (`int`, *optional*, defaults to 128): The window size for local attention. embedding_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the embeddings. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layers. mlp_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the MLP layers. decoder_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the decoder layers. classifier_pooling (`str`, *optional*, defaults to `"cls"`): The pooling method for the classifier. Should be either `"cls"` or `"mean"`. In local attention layers, the CLS token doesn't attend to all tokens on long sequences. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the classifier. classifier_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the classifier. classifier_activation (`str`, *optional*, defaults to `"gelu"`): The activation function for the classifier. deterministic_flash_attn (`bool`, *optional*, defaults to `False`): Whether to use deterministic flash attention. If `False`, inference will be faster but not deterministic. sparse_prediction (`bool`, *optional*, defaults to `False`): Whether to use sparse prediction for the masked language model instead of returning the full dense logits. sparse_pred_ignore_index (`int`, *optional*, defaults to -100): The index to ignore for the sparse prediction. reference_compile (`bool`, *optional*): Whether to compile the layers of the model which were compiled during pretraining. If `None`, then parts of the model will be compiled if 1) `triton` is installed, 2) the model is not on MPS, 3) the model is not shared between devices, and 4) the model is not resized after initialization. If `True`, then the model may be faster in some scenarios. This argument is deprecated and will be removed in a future version. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Examples: ```python >>> from transformers import ModernBertModel, ModernBertConfig >>> # Initializing a ModernBert style configuration >>> configuration = ModernBertConfig() >>> # Initializing a model from the modernbert-base style configuration >>> model = ModernBertModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` modernbertpast_key_valuesgAg@)globallocalcb|dk(r|tjdd}t| ||y)Nreference_compilezThe `reference_compile` argument is deprecated and will be removed in `transformers v5.2.0`Use `torch.compile()` directly on the model instead.)logger warning_oncesuper __setattr__)selfnamevalue __class__s c/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/modernbert/modular_modernbert.pyr3zModernBertConfig.__setattr__s; & &5+<   G E D%(N vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headshidden_activationmax_position_embeddingsinitializer_rangeinitializer_cutoff_factornorm_eps norm_bias pad_token_id eos_token_id bos_token_id cls_token_id sep_token_idattention_biasattention_dropout layer_typesrope_parametersfull_attentionsliding_attentionlocal_attentionembedding_dropoutmlp_bias mlp_dropout decoder_biasclassifier_poolingclsmeanclassifier_dropoutclassifier_biasclassifier_activationdeterministic_flash_attnsparse_predictionsparse_pred_ignore_indexr/tie_word_embeddingsc# b| |_||_| |_||_||_|"|_||_||_||_||_ ||_ ||_ ||_ | |_ | |_| |_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_|!|_|j0dvrtAd|j0d||_!|#jEdd|_#|jBBtI|jD$cgc]}$tK|$|jFzrdnd c}$|_!tM|jB|j||_'tQ%|di|#ycc}$w) NrWzQInvalid value for `classifier_pooling`, should be either "cls" or "mean", but is .global_attn_every_n_layersr rPrO)*rErGrFrHrIr`r:r@r;r<r=r>rArBrCrDrJrKr?rQrRrSrTrUrVrZr[r\r]r^r_r/ ValueErrorrLgetrcrangeboolr rMr2__init__)&r4r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrQrRrSrTrUrVrZr[r\r]r^r_r/r`kwargsir7s& r8rizModernBertConfig.__init__sL)((((#6 $'>$&!2!2#6 !2)B&  ",!2!2.!2  &("4"4.%:"(@%!2(@%!2  " "/ 9cdhd{d{c||}~ '+1**5QST*U'    #t556 (,A0O0O,O'P#Vff D  d..0F0FG. "6" s #F,c |jdd}ddiddid}|j |jn||_|<|jdj||jdj||jjdddi|jd<|jdj d|jd|j d |jjdddi|jd<|jdj d|jd |j d |j |j| |S) N rope_scaling rope_typedefault)rPrOrOrP rope_thetaglobal_rope_thetar,local_rope_thetar-) ignore_keys)poprMupdaterf setdefault default_thetastandardize_rope_params validate_rope)r4ignore_keys_at_rope_validationrjrmdefault_rope_paramss r8convert_rope_params_to_dictz,ModernBertConfig.convert_rope_params_to_dictsuzz.$7 #.y!9*I6 8<7K7K7Wt33]p  #  !1 2 9 9, G  !4 5 < <\ J    # #$4 5 =6A95MD !1 2 -.99 &**%8$:L:LX:VW     # #$7 8 @9Di8PD !4 5 01<< &**%79K9KG9TU $$& 'EF r9cHt|}|jdd|S)Nr/)r2to_dictrt)r4outputr7s r8r~zModernBertConfig.to_dicts#" &- r9c |jdzS)zKHalf-window size: `local_attention` is the total window, so we divide by 2.r$rQr4s r8sliding_windowzModernBertConfig.sliding_window$s##q((r9c|dz|_y)zFikjirrFNNrFrTrXrFrFFiNTN)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencerwr3intstrfloatrhlistdictrrrir|r~propertyrsetter __classcell__r7s@r8r)r)2sc JJ#4"5(8#'+?Y#@ $;AY#B#&*CY#D $;EY#F"D[GY#v> ))))r9r)ceZdZdZdeffd Z d dejdzdejdzdejfdZ xZ S) ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcdt|||_tj|j |j |j|_tj|j |j|j|_ tj|j|_y)N) padding_idxepsbias)r2rirr Embeddingr:r;rEtok_embeddings LayerNormrCrDnormDropoutrRdropr4rr7s r8rizModernBertEmbeddings.__init__4sw   ll6+<+P>P^d^q^qrLL!3!3vO_O_` JJv778 r9N input_ids inputs_embedsreturnc|"|j|j|}|S|j|j|j|}|Sr)rrr)r4rr hidden_statess r8forwardzModernBertEmbeddings.forward;sS  $ IIdii &>?M!IIdii0C0CI0N&OPMr9NN) rrrrr)ritorch LongTensorTensorrrrs@r8rr/sR9/9_c))D0HM W[H[ r9rc`eZdZdZdeffd ZdejdejfdZxZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. rct|||_tj|j t |jdz|j|_ t|j|_ tj|j|_tj|j|j |j|_y)Nr$r)r2rirrLinearr;rr<rSWir r?actrrTrWors r8rizModernBertMLP.__init__Ls  ))F..F4L4L0MPQ0QX^XgXgh&223JJv112 ))F44f6H6Hv_r9rrc|j|jdd\}}|j|j|j ||zS)Nr$dim)rchunkrrr)r4rinputgates r8rzModernBertMLP.forwardTsIggm,221"2= twwtyy%4!7899r9) rrrrr)rirrrrrs@r8rrEs2 `/`:U\\:ell:r9rc|eZdZd deffd Ze d dedzdeddedzdedzde d e ff fd Z xZ S) ModernBertRotaryEmbeddingNrc&t|||yr)r2ri)r4rdevicer7s r8riz"ModernBertRotaryEmbedding.__init__Zs (r9rz torch.deviceseq_len layer_typerz torch.Tensorc(t|||||Sr)r2compute_default_rope_parameters)rrrrr7s r8rz9ModernBertRotaryEmbedding.compute_default_rope_parameters]sw6vvwPZ[[r9rNNNN) rrrr)ri staticmethodrrrtuplerrrrs@r8rrYs)/)*.+/"!% \ 4'\(\t\$J \ ~u$ % \\r9rrotary_pos_embcb|j}|j|}|j|}|j|zt|j|zz}|j|zt|j|zz}|j ||j |fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )dtype unsqueezerr'to)qkcossin unsqueeze_dimoriginal_dtypeq_embedk_embeds r8apply_rotary_pos_embrgs&WWN -- &C -- &Cwwy3;qwwy#9C#?@Gwwy3;qwwy#9C#?@G ::n %wzz.'A AAr9ceZdZdZd dededzffd Z d dejde ejejfdzdejdzd e e d e ejejdzff d Z xZ S)ModernBertAttentionaPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. Nr layer_idxcRt|||_||_|j|j zdk7r&t d|jd|j d|j|_|j|_|j|j z|_ tj|jd|jz|j z|j|_ |j|dk(r|jdz|_nd|_d |_tj|j|j|j|_|jd kDr%tj$|j|_ytj&|_y) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r rrPFr)r2rirrr;r>rerKr]head_dimrrrJWqkvrLr is_causalrrIdentityout_dropr4rrr7s r8rizModernBertAttention.__init__sz  "    : : :a ?#F$6$6#77mntoIoInJJKL "(!9!9(.(G(G%**f.H.HH II   DMM 1F4N4N NU[UjUj     i (,? ?#)"7"7!";D "&D ))F..0B0BI^I^_@F@X@X[^@^ 6#;#;< dfdododq r9rposition_embeddingsattention_maskrjrc |jdd}|j|}|jg|dd|j}|j d\}}} |j dd}|j dd}| j dd} |\} } t ||| | d\}}t} |jjdk7rt|jj} | |||| |f|jr |jnd |jd z|j|jd |\} }| jg|dj!} |j#|j%| } | |fS) Nrr rrr$)reagerr)dropoutscalingr deterministic)shaperviewrunbind transposerr%r_attn_implementationrtrainingrKrr]reshape contiguousrr)r4rrrrj input_shapeqkv query_states key_states value_statesrrattention_interface attn_output attn_weightss r8rzModernBertAttention.forwards$))#2. ii &chh: :Q::DMM:141C. j,#--a3 ))!Q/ #--a3 &S#7 jRUWZjk#l j5 ;; + +w 6"9$++:Z:Z"[ $7      % /3mmD**MM4'..77 %  % ! \*k));;;;FFH mmDGGK$89 L((r9rr)rrrrr)rrirrrrrrrrs@r8rrsr/rC$Jr@IM.2 ')||')#5<<#=>E') t+ ') +, ') u||U\\D00 1 ')r9rc eZdZd dededzffd Z d dejdejdzdejdzdee d ejf d Z xZ S) ModernBertEncoderLayerNrrct|||_||_|dk(rt j |_n;t j|j|j|j|_t|||_ t j|j|j|j|_ t||_|j ||_y)Nrr)rr)r2rirrrr attn_normrr;rCrDrattnmlp_normrmlprLattention_typers r8rizModernBertEncoderLayer.__init__s  " >[[]DN\\&*<*<&//X^XhXhiDN'vK  V%7%7V__SYScScd  ($00;r9rrrrjrc |j|j|f||d|\}}||z}||j|j|z}|S)N)rr)rrr r )r4rrrrjr_s r8rzModernBertEncoderLayer.forwardsg# NN= ) 3)    Q & 3 %}1M(NN r9rr) rrrr)rrirrrrrrrs@r8rrsx </  t9j@|j>yyt|tBr|jDD]}|jF}|jH|d k7rtJ|jH|}||j| \}}t9jLtO||d|t9jLtO||d|yy)Nr rstdctj|jd| |z|zt|tj r-|j tj|j yyy)Nr)rYrab)init trunc_normal_weight isinstancerrrzeros_)rr cutoff_factors r8 init_weightz.init_weight sd     .3&#%  &")),;;*KK ,+-r9rr)inout embedding final_outr!rr r"ro)r _inv_freq_original_inv_freq)(rrBrModulerrAmathsqrtr=r;rrrrrrrrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassificationModernBertForMultipleChoice ModernBertForTokenClassificationModernBertForQuestionAnswering classifierrrones_rrrrrLrrnrcopy_getattr) r4rrstdsr rope_init_fn curr_inv_freqr rs @r8 _init_weightsz'ModernBertPreTrainedModel._init_weightssd ==  M - - -++//;;00499S4;;C`C`=`3aa6600$6   f2 3 --tK/@ A  .  4: .  4; /  3 4  T$Z 0  4; /  8 9  d5k 2  5 6 U 4  3+0.    ))4 +< =  - JJv}} %{{& FKK('  9 :$00 ^ %EE ##J/9<#6v7G7G 7S#TL#/ *#U q 76j\+CDmT 76j\9K+LM}]  ^;r9N)rrrr)__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsrno_gradrr%r7rdr9r8rrsq &*#/1IJN"&0) U]]_:^BII:^:^r9rceZdZdeffd ZdZdZeee d de jdzde jdzde jdzd e jdzd e ed ef d ZxZS)ModernBertModelrc t||||_t||_t j t|jDcgc]}t||c}|_ t j|j|j|j|_t!||_d|_|j'ycc}w)Nr)rF)r2rirr embeddingsr ModuleListrgr=rlayersrr;rCrD final_normr rotary_embgradient_checkpointing post_initrs r8rizModernBertModel.__init__Ds   .v6mmHMfNfNfHg h9 #FI 6 h  ,,v'9'9vU[UeUef36B&+#  isCc.|jjSrrErrs r8get_input_embeddingsz$ModernBertModel.get_input_embeddingsPs---r9c&||j_yrrMrs r8set_input_embeddingsz$ModernBertModel.set_input_embeddingsSs).&r9Nrr position_idsrrjrc |du|duz r td||jdn|jd}| |jn |j}|&tj||j d}|j ||}t|x} ts'|j||d} td i| td i| d} i} |jjD]} |j||| | | <|jD](} | |f| | j| | jd|}*|j!|}t#| S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)rr)rrrrN)rr)last_hidden_staterd)rerrrarangerrErrrrrrLrIrGr rHr)r4rrrQrrjrrrattention_mask_mapping mask_kwargsrr encoder_layers r8rzModernBertModel.forwardVsw  -t"; <YZ Z,9,E-%%a(9??[\K]%.%:!! @T@T   <<?II!LL)=Y NB0DI++!."0K #<"Jk"J%M%\P[%\& " !++11 gJ.2oom\[e.f  + g"[[ M)5m6R6RS$7 8T8T$U M  6 ??r9r)rrrr)rirNrPr"r#rrrrrrrrrrs@r8rCrCBs / ./ .2.204-1 ,@##d*,@ t+,@&&- ,@ ||d* ,@ +, ,@ ,@ ,@r9rCc\eZdZdeffd Zdej dej fdZxZS)r(rcJt|||_tj|j |j |j |_t|j|_ tj|j |j|j|_y)Nr)r2rirrrr;r[r)r r\rrrCrDrrs r8riz!ModernBertPredictionHead.__init__sq  YYv1163E3EvG]G]^ &667LL!3!3vO_O_` r9rrc`|j|j|j|Sr)rrr))r4rs r8rz ModernBertPredictionHead.forwards#yy$**]";<==r9) rrrr)rirrrrrs@r8r(r(s-a/a>U\\>ell>r9r(zd The ModernBert Model with a decoder head on top that is used for masked language modeling. ) custom_introc>eZdZddiZdeffd ZdZdejfdZ e e dd e jdzd e jdzd e jdzd e jdzd e jdzdeedee jezfdZxZS)r*zdecoder.weightz&model.embeddings.tok_embeddings.weightrctt||||_t||_t ||_tj|j|j|j|_ |jj|_ |jj|_|jy)Nr)r2rirrCrr(headrrr;r:rUr+r^r_rKrs r8rizModernBertForMaskedLM.__init__s   $V, ,V4 yy!3!3V5F5FVM`M`a !%!>!>(, (L(L% r9c|jSrr+rs r8get_output_embeddingsz+ModernBertForMaskedLM.get_output_embeddingss ||r9new_embeddingsc||_yrra)r4rcs r8set_output_embeddingsz+ModernBertForMaskedLM.set_output_embeddingss % r9NrrrQrlabelsrjrc |jd||||d|}|d}|jrK|I|jd}|j|jdd}||jk7} || }|| }|j |j |} d} |*|j| |fd|jji|} t| | |j|jS)NrrrQrrrr:losslogitsrrrd) rr^rrr_r+r_ loss_functionrr:rrr) r4rrrQrrfrjoutputsrT mask_tokensrkrjs r8rzModernBertForMaskedLM.forwards $** )%'    $AJ  ! !f&8[[_F 1 6 6v||A K !D$A$AAK 1+ > K(Fdii(9:;  %4%%ffbAWAWb[abD!//))   r9NNNNN)rrr_tied_weights_keysr)rirbrrrer!rrrrrrrrrrrs@r8r*r*s +,TU / &BII&.2.2,0-1&* ' ##d*'  t+' llT) ' ||d* '  t# ' +,'  u|| ~ -' ' r9r*z` The ModernBert Model with a sequence classification head on top that performs pooling. ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) r,rcnt|||j|_||_t ||_t ||_tjj|j|_ tj|j|j|_|j!yr)r2ri num_labelsrrCrr(r_rrrrZrrr;r0rKrs r8riz,ModernBertForSequenceClassification.__init__s   ++ $V, ,V4 HH$$V%>%>? ))F$6$68I8IJ r9NrrrQrrfrjrc d|jd||||d|}|d}|jjdk(r |dddf}n|jjdk(rw|=tj|j dd|j tj}||jdzjd |jd d z }|j|} |j| } |j| } d} ||jj|jd k(rd |j_ nl|jd kDrL|jtj k(s|jtj"k(rd|j_ nd|j_ |jjd k(rIt%} |jd k(r&| | j'|j'} n| | |} n|jjdk(r=t)} | | j+d|j|j+d} n,|jjdk(rt-} | | |} t/| | |j0|j2S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rhrrXNrYr$)rrrrrTrkeepdim regressionsingle_label_classificationmulti_label_classificationrird)rrrVronesrrrhrsumr_rr0 problem_typersrlongrrsqueezerrrrrr) r4rrrQrrfrjrmrT pooled_outputrkrjloss_fcts r8rz+ModernBertForSequenceClassification.forwardse"$** )%'    $AJ ;; ) )U 2 1!Q$ 7  [[ + +v 5%!&%++BQ/8I8P8PX]XbXb""3^5M5Mb5Q!Q V V[\ V ]`n`r`rtasa!  "34  -0 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r9ro)rrrr)rir!rrrrrrrrrrrs@r8r,r,s / .2.2,0-1&* C ##d*C  t+C llT) C ||d* C  t# C +,C  u|| 7 7C C r9r,zv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) r.rc`t|||j|_t||_t ||_tjj|j|_ tj|j|j|_|jyrr2rirsrCrr(r_rrrrZrrr;r0rKrs r8riz)ModernBertForTokenClassification.__init__<s{   ++$V, ,V4 HH$$V%>%>? ))F$6$68I8IJ r9NrrrQrrfrjrc f|jd||||d|}|d}|j|}|j|}|j|} d} |%>? ))F$6$68I8IJ r9NrrrQstart_positions end_positionsrjrc |j|f||d|}|d}|j|}|j|}|j|} | j dd\} } | j dj } | j dj } d} |||j| | ||fi|} t| | | |j|jS)N)rrQrrrr)rj start_logits end_logitsrr) rr_rr0splitr~rrlrrr) r4rrrQrrrjrmrTrkrrrjs r8rz&ModernBertForQuestionAnswering.forward~s$**  )%    $AJ II&78 II&78!23#)<<r<#: j#++B/::< ''+668   &=+D%4%%lJQ^ibhiD+%!!//))   r9ro)rrrr)rir!rrrrrrrrrrs@r8r/r/qs / *..2,0/3-1 # <<$&#  t+# llT) # , # ||d* # +,#  u|| ; ;# # r9r/z The ModernBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) r-rc8t||||_t||_t ||_tjj|j|_ tj|jd|_|jy)Nr)r2rirrCrr(r_rrrrZrrr;r0rKrs r8riz$ModernBertForMultipleChoice.__init__so   $V, ,V4 HH$$V%>%>? ))F$6$6: r9NrrrQrrfrjrc ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|jd||||d|}|d} |jj dk(rt j| jd| j} |,|jd j| j} n0t jdt j| j } | | | f} nS|jj d k(r:|jdd } | |jdzjd | z } |j| } |j!| } |j#| }|jd|}d}|t%j&}|||}t)|||j*|j,S)a& labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. NrrrhrrXrSr)rrrYTrurird)rrsizerrrVrrUrargmaxrtensorr}r{rr_rr0rrrrr)r4rrrQrrfrj num_choicesrmrT indices_0cls_masknum_non_pad_tokensrrkreshaped_logitsrjrs r8rz#ModernBertForMultipleChoice.forwardsn -6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $** )%'    $AJ ;; ) )U 2 %6%<%rs  $ AA&!JI`9GF&@@I5:G   H %z)'z)z299,:BII:( \ 5 \*+B,B4)*N)"))N)+N)b7@J^J^J^ZB@/B@B@J >ryy > ? 5?  ? D S *CS  S l 3 '@3  3 l1 %>1 1 h R ";R  R j r9