qi(dZddlmZddlmZddlmZddlmZeje Z GddeZ Gd d eZ Gd d eZd gZy )zDBRX model configuration)Any)PreTrainedConfig)RopeParameters)loggingc FeZdZdZdZ d dededzdedeffd ZxZ S) DbrxAttentionConfigaConfiguration class for Dbrx Attention. [`DbrxAttention`] class. It is used to instantiate attention layers according to the specified arguments, defining the layers architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: attn_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the attention layers. clip_qkv (`float`, *optional*): If set, clip the queries, keys, and values in the attention layer to this value. kv_n_heads (`int`, *optional*, defaults to 1): For grouped_query_attention only, allow user to specify number of kv heads. attn_configN attn_pdropclip_qkv kv_n_headskwargsc Nt|di|||_||_||_y)N)super__init__r r r )selfr r r r __class__s ]/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/dbrx/configuration_dbrx.pyrzDbrxAttentionConfig.__init__.s* "6"$  $)N) __name__ __module__ __qualname____doc__base_config_keyfloatintrr __classcell__rs@rr r sN"$O !% % %$, % %  % %rr cleZdZdZdZ d dedzdededededzd ed edzd effd Z xZ S) DbrxFFNConfiga|Configuration class for Dbrx FFN. [`DbrxFFN`] class. It is used to instantiate feedforward layers according to the specified arguments, defining the layers architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: ffn_act_fn (`dict`, *optional*, defaults to `None`): A dict specifying activation function for the FFN. The dict should have a key 'name' with the value being the name of the activation function along with any additional keyword arguments. If `None`, then set to `{"name": "silu"}`. ffn_hidden_size (`int`, *optional*, defaults to 3584): The hidden size of the feedforward network. moe_num_experts (`int`, *optional*, defaults to 4): The number of experts in the mixture of experts layer. moe_top_k (`int`, *optional*, defaults to 1): The number of experts to use in the mixture of experts layer. moe_jitter_eps (`float`, *optional*, defaults to `None`): If not `None`, the jitter epsilon for the mixture of experts layer. moe_loss_weight (`float`, *optional*, defaults to 0.01): The loss weight for the mixture of experts layer. moe_normalize_expert_weights (`float`, *optional*, defaults to 1.0): The normalization factor for the expert weights. ffn_configN ffn_act_fnffn_hidden_sizemoe_num_experts moe_top_kmoe_jitter_epsmoe_loss_weightmoe_normalize_expert_weightsrc  t ||ddi}||_||_||_||_||_||_||_||_ dD]} | | vs| j| t| dk7rtd| y)Nnamesilu) model_typeattn_implementationexperts_implementationtransformers_version _commit_hash torch_dtypedtyperzFound unknown kwargs=) rr hidden_sizer%r&r'r(r)r*r+poplen ValueError) rr6r%r&r'r(r)r*r+rkrs rrzDbrxFFNConfig.__init__Rs    &)J&$..",.,H) AF{ 1   v;! 5fY78 8 r)iNirNg{Gz?g?) rrrrrdictrrrrr r!s@rr#r#;s(#O"&# '+!%58$94K$9 $9  $9  $9 $9$9',dl$9$9$9rr#c%"eZdZdZdZeedZdddddZ dde d zde d zde d zde d zd e d zd e d zd e d zd ed zded zde d zde d zde d zde e ee fzd zde d zde d zde d zde d zdef$fd ZxZS) DbrxConfiga This is the configuration class to store the configuration of a [`DbrxModel`]. It is used to instantiate a Dbrx model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a different configuration to that of the [transformers-community/dbrx-instruct](https://huggingface.co/transformers-community/dbrx-instruct) architecture. Note: this link points to a re-upload; the original repository is closed. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: d_model (`int`, *optional*, defaults to 2048): Dimensionality of the embeddings and hidden states. n_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. n_layers (`int`, *optional*, defaults to 24): Number of hidden layers in the Transformer encoder. max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. vocab_size (`int`, *optional*, defaults to 32000): Vocabulary size of the Dbrx model. Defines the maximum number of different tokens that can be represented by the `inputs_ids` passed when calling [`DbrxModel`]. resid_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability applied to the attention output before combining with residual. emb_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the embedding layer. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. ffn_config (`dict`, *optional*): A dictionary used to configure the model's FFN module. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. output_router_logits (`bool`, *optional*, defaults to `False`): Whether or not the router logits should be returned by the model. Enabling this will also allow the model to output the auxiliary loss. See [here]() for more details. Example: ```python >>> from transformers import DbrxConfig, DbrxModel >>> # Initializing a Dbrx configuration >>> configuration = DbrxConfig(n_layers=2, d_model=256, n_heads=8, vocab_size=128) >>> # Initializing a model (with random weights) from the configuration >>> model = DbrxModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` dbrx)r r$n_headsd_modeln_layers max_seq_len)num_attention_headsr6num_hidden_layersmax_position_embeddingsN vocab_size resid_pdrop emb_pdropr r$ use_cacheinitializer_rangeoutput_router_logitsrope_parameters pad_token_id bos_token_id eos_token_idtie_word_embeddingsrc .|t|_n(t|trtdi||_n||_| t |_n(t| trt di| |_n| |_||_||_||_||_ ||_ ||_ ||_ | |_ | |_| |_|jj |_|r t%d||_||_||_||_| |_t1|ddi|y)Nz5tie_word_embeddings is not supported for DBRX models.r)r r isinstancer<r#r$rAr@rBrCrGrHrIrJrKrLr num_key_value_headsr9rNrOrPrQrMrr)rrAr@rBrCrGrHrIr r$rJrKrLrMrNrOrPrQrrs rrzDbrxConfig.__init__s *  24D   T *2A[AD *D   +oDO  D )+9j9DO(DO    &$&""!2$8!#'#3#3#>#> TU U(((#6 . "6"r)rUi}rrNNTg{Gz?FNNNNF)rrrrr/r r# sub_configs attribute_maprrboolrr<strrrr r!s@rr>r>yst5nJ"5]SK( '#0 M# !"&!&$'"%26+/!%*.,1MQ#'#'#'+0%7#t7#t7#* 7# 4Z 7# $J 7#T\7#4<7#)4/7#"D(7#$;7#!4<7##Tk7#($sN/B*CCdJ7#Dj7# Dj!7#"Dj#7#$"D[%7#&'7#7#rr>N)rtypingrconfiguration_utilsrmodeling_rope_utilsrutilsr get_loggerrloggerr r#r>__all__rrrrcs_31   H %%*%B;9$;9|x#!x#v .r