qi8dZddlmZddlmZddlmZejeZ GddeZ GddeZ dgZ y ) zEvolla model configuration)PreTrainedConfig)RopeParameters)loggingcFeZdZdZ dfd ZxZS) SaProtConfiga This is the configuration class to store the configuration of a [`EvollaSaProtProteinEncoder`]. It is used to instantiate a SaProt model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 446): Vocabulary size of the protein sequence model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`EvollaModel`]. mask_token_id (`int`, *optional*, defaults to 4): The id of the *mask* token in the protein sequence model. pad_token_id (`int`, *optional*, defaults to 1): The id of the *padding* token in the protein sequence model. hidden_size (`int`, *optional*, defaults to 1280): Dimensionality of the protein sequence model layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 33): Number of hidden layers in the protein sequence model. num_attention_heads (`int`, *optional*, defaults to 20): Number of attention heads for each attention layer in the protein sequence model. intermediate_size (`int`, *optional*, defaults to 5120): Dimensionality of the intermediate layers in the protein sequence model. hidden_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the hidden layers in the protein sequence model. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities in the protein sequence model. max_position_embeddings (`int`, *optional*, defaults to 1026): The maximum sequence length that the protein sequence model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). layer_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon value for the layer normalization layer in the protein sequence model. position_embedding_type (`str`, *optional*, defaults to `"rotary"`): The type of position embedding to use in the protein sequence model. Currently only `"rotary"` is supported. emb_layer_norm_before (`bool`, *optional*, defaults to `False`): Whether to apply layer normalization before the position embedding in the protein sequence model. token_dropout (`bool`, *optional*, defaults to `True`): Whether to apply dropout to the tokens in the protein sequence model.c t|di|||_||_||_||_||_||_||_||_ ||_ ||_ | |_ | |_ | |_| |_| |_||_||_y)N)super__init__ pad_token_id mask_token_id is_decoderadd_cross_attention vocab_size hidden_sizenum_hidden_layersnum_attention_headsintermediate_sizehidden_dropout_probattention_probs_dropout_probmax_position_embeddingsinitializer_rangelayer_norm_epsposition_embedding_typeemb_layer_norm_before token_dropout)selfrr r rrrrrrrrrrrrrrkwargs __class__s a/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/evolla/configuration_evolla.pyr zSaProtConfig.__init__?s* "6"(*$#6 $&!2#6 !2#6 ,H)'>$!2,'>$%:"*)ii!i皙?r&i{Gz?h㈵>rotaryFTFF)__name__ __module__ __qualname____doc__r __classcell__rs@r rrsI$UP%( $ (#!%'+'+r!rc@eZdZdZdZdeiZdZ d%dedzde dzde dzde dzd e dzd e dzd e dzd e dzd e dzde dzde ee e fzdzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzd e dzd!e dzd"e dzd#e dzf>fd$ ZxZS)& EvollaConfiga This is the configuration class to store the configuration of a [`EvollaModel`]. It is used to instantiate an Evolla model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Evolla-10B. e.g. [westlake-repl/Evolla-10B-hf](https://huggingface.co/westlake-repl/Evolla-10B-hf) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: protein_encoder_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`SaProtConfig`]. vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the Evolla llama model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`EvollaModel`]. hidden_size (`int`, *optional*, defaults to 4096): Dimensionality of the llama layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 14336): Dimensionality of the intermediate layers in the llama model. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the llama model. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the llama model. num_key_value_heads (`int`, *optional*, defaults to 8): Number of key-value pairs for each attention layer in the llama model. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the llama model. If string, `"gelu"`, `"relu"`, `"selu"` and `"silu"` are supported. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon value for the RMS-norm layer in the llama model. rope_parameters (`float`, *optional*): The scaling factor for the RoPE layer in the llama model. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the attention layer. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention layer. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layer. aligner_ffn_mult (`int`, *optional*, defaults to 4): The FFN multiplier for the aligner layer. aligner_enable_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the aligner layer. aligner_attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities in the aligner layer. aligner_num_add_layers (`int`, *optional*, defaults to 8): The number of additional layers for the aligner layer. resampler_depth (`int`, *optional*, defaults to 6): The depth of the resampler layer in the llama model. resampler_dim_head (`int`, *optional*, defaults to 64): The dimension of the heads in the resampler layer in the llama model. resampler_heads (`int`, *optional*, defaults to 8): The number of heads in the resampler layer in the llama model. resampler_num_latents (`int`, *optional*, defaults to 64): The number of latents in the resampler layer in the llama model. resampler_ff_mult (`int`, *optional*, defaults to 4): The FFN multiplier for the resampler layer. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*, defaults to 128000): The id of the *beginning-of-sequence* token. eos_token_id (`int`, *optional*, defaults to 128009): The id of the *end-of-sequence* token. use_cache (`bool`, *optional*, defaults to `False`): Whether or not the model should return the last key/values attentions (not used by all models). tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether or not to tie the input and output word embeddings. is_decoder (`bool`, *optional*, defaults to `False`): Whether to only use the decoder in an encoder-decoder architecture, otherwise it has no effect on decoder-only or encoder-only architectures. add_cross_attention (`bool`, *optional*, defaults to `False`): Whether cross-attention layers should be added to the model. Example: ```python >>> from transformers import EvollaModel, EvollaConfig >>> # Initializing a Evolla evolla-10b style configuration >>> configuration = EvollaConfig() >>> # Initializing a model from the evolla-10b style configuration >>> model = EvollaModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` EvollaModelprotein_encoder_configgANrrrrrnum_key_value_heads hidden_actr rms_norm_epsrope_parametersattention_biasattention_dropoutmlp_biasaligner_ffn_multaligner_enable_bias$aligner_attention_probs_dropout_probaligner_num_add_layersresampler_depthresampler_dim_headresampler_headsresampler_num_latentsresampler_ff_multrr bos_token_id eos_token_id use_cachetie_word_embeddingsrrc  (||_||_||_||_||_||_||_||_||_| |_ | |_ ||_ | |_ | |_ ||_||_||_||_||_||_||_||_||_||_||_||_| |_|i}t6j9dt;di||_||_ ||_||_ ||_!tE!|di| y)NzX`protein_encoder_config` is `None`. Initializing the `SaProtConfig` with default values.r )$rrrrrrrr4r5rr6rGr8r9r:r;r<r=r>rFrr?r@rArBrCr7loggerinforr3r rDrEr r )"rr3rrrrrr4r5rr6r7r8r9r:r;r<r=r>r?r@rArBrCrr rDrErFrGrrrrs" r r zEvollaConfig.__init__s5F%#6 $&!2!2#6 #6 $'>$(#6 ,!2  0#6 4X1&<#"!2."4.%:"!2. " )%' " KKr s&2&L5K&L##6 ((( "6"r!)Niii8 rKsilui r(NFgFr"Tr&rL@rLrOr"r'Nii FFFF)r*r+r,r- model_typer sub_configs default_thetadictintstrrboolfloatr r.r/s@r r1r1ism[ zJ+\:KM/3!'"&(-(**,*+!'.2#(MQ&+*- %'(+/=@-.&')+&',.()*.#'#)#)!&+0"'+0AJ# $t J#$JJ#4Z J# : J# : J#!4ZJ#!4ZJ#$JJ#"%tJ#DjJ#($sN/B*CCdJJ#t J#!4<J#+J# *!J#""D[#J#$/4dl%J#&!$d 'J#(t)J#* $J+J#,t-J#. #Tz/J#0:1J#2!4<3J#4Dj5J#6Dj7J#8Dj9J#:$;;J#<"D[=J#>4K?J#@"D[AJ#J#r!r1N) r-configuration_utilsrmodeling_rope_utilsrutilsr get_loggerr*rIrr1__all__r r!r r]sQ!31   H %N+#N+bl##l#^  r!