qi9xddlmZddlmZmZddlmZddlmZeje Z GddeZ dgZ y) )Literal)PreTrainedConfiglayer_type_validation)RopeParameters)loggingcF8eZdZdZdZdgZdddZfdZ d2d edzd edzd edzd edzd edzde dzdedzde dzde dzde dzde dzdedzdedzdedzdedzdedzde dzde dzde e dzde edefdzdedzde dzd e dzd!e dzd"e dzd#ed$d%e dzd&e dzd'e dzd(e dzd)e dzd*edzd+e dzd,e dzfDfd- Zd3d.Zfd/Zed0Zej*d1ZxZS)4ModernBertConfigaO This is the configuration class to store the configuration of a [`ModernBertModel`]. It is used to instantiate an ModernBert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ModernBERT-base. e.g. [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the ModernBert model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`ModernBertModel`] hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 22): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu"` if not specified. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_cutoff_factor (`float`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. norm_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the normalization layers. pad_token_id (`int`, *optional*, defaults to 50283): Padding token id. eos_token_id (`int`, *optional*, defaults to 50282): End of stream token id. bos_token_id (`int`, *optional*, defaults to 50281): Beginning of stream token id. cls_token_id (`int`, *optional*, defaults to 50281): Classification token id. sep_token_id (`int`, *optional*, defaults to 50282): Separation token id. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. layer_types (`list`, *optional*): Attention pattern for each layer. rope_parameters (`dict`, *optional*): Dictionary mapping attention patterns (`"full_attention"`, `"sliding_attention"`) to `RopeParameters`. Each value should be a dictionary containing `rope_type` and optional scaling parameters. local_attention (`int`, *optional*, defaults to 128): The window size for local attention. embedding_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the embeddings. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layers. mlp_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the MLP layers. decoder_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the decoder layers. classifier_pooling (`str`, *optional*, defaults to `"cls"`): The pooling method for the classifier. Should be either `"cls"` or `"mean"`. In local attention layers, the CLS token doesn't attend to all tokens on long sequences. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the classifier. classifier_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the classifier. classifier_activation (`str`, *optional*, defaults to `"gelu"`): The activation function for the classifier. deterministic_flash_attn (`bool`, *optional*, defaults to `False`): Whether to use deterministic flash attention. If `False`, inference will be faster but not deterministic. sparse_prediction (`bool`, *optional*, defaults to `False`): Whether to use sparse prediction for the masked language model instead of returning the full dense logits. sparse_pred_ignore_index (`int`, *optional*, defaults to -100): The index to ignore for the sparse prediction. reference_compile (`bool`, *optional*): Whether to compile the layers of the model which were compiled during pretraining. If `None`, then parts of the model will be compiled if 1) `triton` is installed, 2) the model is not on MPS, 3) the model is not shared between devices, and 4) the model is not resized after initialization. If `True`, then the model may be faster in some scenarios. This argument is deprecated and will be removed in a future version. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Examples: ```python >>> from transformers import ModernBertModel, ModernBertConfig >>> # Initializing a ModernBert style configuration >>> configuration = ModernBertConfig() >>> # Initializing a model from the modernbert-base style configuration >>> model = ModernBertModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` modernbertpast_key_valuesgAg@)globallocalcb|dk(r|tjdd}t| ||y)Nreference_compilezThe `reference_compile` argument is deprecated and will be removed in `transformers v5.2.0`Use `torch.compile()` directly on the model instead.)logger warning_oncesuper __setattr__)selfnamevalue __class__s i/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/modernbert/configuration_modernbert.pyrzModernBertConfig.__setattr__s; & &5+<   G E D%(N vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headshidden_activationmax_position_embeddingsinitializer_rangeinitializer_cutoff_factornorm_eps norm_bias pad_token_id eos_token_id bos_token_id cls_token_id sep_token_idattention_biasattention_dropout layer_typesrope_parameters)full_attentionsliding_attentionlocal_attentionembedding_dropoutmlp_bias mlp_dropout decoder_biasclassifier_poolingclsmeanclassifier_dropoutclassifier_biasclassifier_activationdeterministic_flash_attnsparse_predictionsparse_pred_ignore_indexrtie_word_embeddingsc# b| |_||_| |_||_||_|"|_||_||_||_||_ ||_ ||_ ||_ | |_ | |_| |_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_|!|_|j0dvrtAd|j0d||_!|#jEdd|_#|jBBtI|jD$cgc]}$tK|$|jFzrdnd c}$|_!tM|jB|j||_'tQ%|di|#ycc}$w) Nr7zQInvalid value for `classifier_pooling`, should be either "cls" or "mean", but is .global_attn_every_n_layersrr0r/)*r&r(r'r)r*r@rr!rrrrr"r#r$r%r+r,r r1r2r3r4r5r6r:r;r<r=r>r?r ValueErrorr-getrCrangeboolrr.r__init__)&rrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r1r2r3r4r5r6r:r;r<r=r>r?rr@kwargsirs& rrIzModernBertConfig.__init__sL)((((#6 $'>$&!2!2#6 !2)B&  ",!2!2.!2  &("4"4.%:"(@%!2(@%!2  " "/ 9cdhd{d{c||}~ '+1**5QST*U'    #t556 (,A0O0O,O'P#Vff D  d..0F0FG. "6" s #F,c |jdd}ddiddid}|j |jn||_|<|jdj||jdj||jjdddi|jd<|jdj d|jd|j d |jjdddi|jd<|jdj d|jd |j d |j |j| |S) N rope_scaling rope_typedefault)r0r/r/r0 rope_thetaglobal_rope_thetar local_rope_thetar) ignore_keys)popr.updaterF setdefault default_thetastandardize_rope_params validate_rope)rignore_keys_at_rope_validationrJrMdefault_rope_paramss rconvert_rope_params_to_dictz,ModernBertConfig.convert_rope_params_to_dictsuzz.$7 #.y!9*I6 8<7K7K7Wt33]p  #  !1 2 9 9, G  !4 5 < <\ J    # #$4 5 =6A95MD !1 2 -.99 &**%8$:L:LX:VW     # #$7 8 @9Di8PD !4 5 01<< &**%79K9KG9TU $$& 'EF rcHt|}|jdd|S)Nr)rto_dictrT)routputrs rr^zModernBertConfig.to_dict s#" &- rc |jdzS)zKHalf-window size: `local_attention` is the total window, so we divide by 2.r1)rs rsliding_windowzModernBertConfig.sliding_windows##q((rc|dz|_y)zFikjirirhFNNrjFrjTr8rjFrgFFiNT)N)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencerWrintstrfloatrHlistdictrrrIr\r^propertyrcsetter __classcell__)rs@rr r sc JJ#4"5(8#'+?Y#@ $;AY#B#&*CY#D $;EY#F"D[GY#v> ))))rr N)typingrconfiguration_utilsrrmodeling_rope_utilsrutilsr get_loggerrlrr __all__rDrrrsA,J1   H %z)'z)z  r