qi }ddlmZddlmZmZddlmZddlmZddl m Z eje Z Gdd eZGd d eZGd d eZGddeZgdZy))Any)PreTrainedConfiglayer_type_validation)RopeParameters)logging)SiglipVisionConfigc2eZdZdZdZdgZdddddddddd Zdgd gfd d gd gfd gd gfd Zd ddZ d*de dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzd e dzd!e dzd"e dzd#e dzd$e e dzd%e dzd&e dzd'eee efzdzf.fd( Zd+d)ZxZS),T5Gemma2TextConfiga This is the configuration class to store the configuration of a [`T5Gemma2TextModel`]. It is used to instantiate the encoder's text model portion of the T5Gemma2 Model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the T5Gemma2Text-7B. e.g. [google/t5gemma2_text-7b](https://huggingface.co/google/t5gemma2_text-7b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 262208): Vocabulary size of the T5Gemma2Text model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`T5Gemma2TextModel`] hidden_size (`int`, *optional*, defaults to 2304): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 9216): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 26): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 4): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 256): The attention head dimension. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"` if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. eos_token_id (`int`, *optional*, defaults to 1): End of stream token id. bos_token_id (`int`, *optional*, defaults to 2): Beginning of stream token id. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. query_pre_attn_scalar (`float`, *optional*, defaults to 256): Scaling factor used on the attention scores sliding_window (`int`, *optional*, defaults to 4096): In T5Gemma2Text, every other layer uses sliding window attention. This is the size of the sliding window. layer_types (`list`, *optional*): Attention pattern for each layer. final_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the logits. attn_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the attention scores. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. t5gemma2_textpast_key_valuescolwisereplicated_with_grad_allreducerowwise zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.q_normzlayers.*.self_attn.k_normzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask embed_tokenslayersnorm.A@globallocalN vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_headshead_dimhidden_activationmax_position_embeddingsinitializer_range rms_norm_eps use_cache pad_token_id eos_token_id bos_token_idattention_biasattention_dropoutquery_pre_attn_scalarsliding_window layer_typesfinal_logit_softcappingattn_logit_softcappingrope_parametersc | |_||_||_||_| |_||_||_||_||_||_ ||_ | |_ | |_ | |_ ||_||_||_||_||_||_||_||_|j-dd|_|j*Et1|jDcgc]!}t3|dz|j.zrdnd#c}|_t5|j*|j||_t9|tdi|ycc}wNsliding_window_patternsliding_attentionfull_attentionr,r.r-r r(r!r"r#r$r&r%r)r*r+r/r0r'r1r2r4r5r3get_sliding_window_patternrangeboolrr6super__init__selfr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6kwargsi __class__s e/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/t5gemma2/configuration_t5gemma2.pyrEzT5Gemma2TextConfig.__init__yN6)(($'>$&!2!2#6   #6 !2(",!2!2%:",'>$&<#&(.zz2JA'N$    #t556 (,QUd6R6R,R'S#Yii D  d..0F0FG. "6" &D;c |jdd}ddiddid}|j |jn||_||jdj||jjdddi|jd<|jdj d|jd|j d|jjdddi|jd<|jdj d|jd |j d |j |j| |S N rope_scaling rope_typedefault)r<r=r= rope_thetarr<rope_local_base_freqr) ignore_keyspopr6updater@ setdefault default_thetastandardize_rope_params validate_roperGignore_keys_at_rope_validationrHrPdefault_rope_paramss rKconvert_rope_params_to_dictz.T5Gemma2TextConfig.convert_rope_params_to_dict[zz.$7 #.y!9*I6 8<7K7K7Wt33]p  #  !1 2 9 9, G    # #$4 5 =6A95MD !1 2 -.99 &**\43E3Eh3OP     # #$7 8 @9Di8PD !4 5 01<< &**%;T=O=OPW=XY $$& 'EF i@i i$gelu_pytorch_tanhi{Gz?gư>Trr;r FrgiNNNNN__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planrZintstrfloatrClistrdictrEr` __classcell__rJs@rKr r s&BH!J#4"5%.%.%.%E%E%."+ )"+ &(9:#%568IJ!"_$56 +X>M")"&(,(**+*+"(;.5*.#'!%#$#$#$&+*-,/%)(,04/3MQ1=#$J=#4Z=#: =# : =# !4Z =#!4Z=#*=#:=#"%t=#!4<=#Dj=#$;=#Dj=#Dj=# Dj!=#"t #=#$!4<%=#& #Tz'=#(d )=#*#Y%+=#,"'-=#.!& /=#0($sN/B*CCdJ1=#~rbr ceZdZdZdZddddZeedZ dd ee e e fzdzd ee e e fzdzd e dzde dzde dzde dzd e dzd edzffd ZxZS)T5Gemma2EncoderConfigaq This is the configuration class to store the configuration of a [`T5Gemma2EncoderForConditionalGeneration`]. It is used to instantiate an T5Gemma2EncoderForConditionalGeneration according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the PaliGemma-2B. e.g. [google/gemma-3-4b](https://huggingface.co/google/gemma-3-4b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: text_config (`Union[T5Gemma2EncoderTextConfig, dict]`, *optional*): The config object of the text backbone. vision_config (`Union[AutoConfig, dict]`, *optional*): Custom vision config or dict. mm_tokens_per_image (`int`, *optional*, defaults to 256): The number of tokens per image embedding. boi_token_index (`int`, *optional*, defaults to 255999): The begin-of-image token index to wrap the image prompt. eoi_token_index (`int`, *optional*, defaults to 256000): The end-of-image token index to wrap the image prompt. image_token_index (`int`, *optional*, defaults to 262144): The image token index to encode the image prompt. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Example: ```python >>> from transformers import T5Gemma2EncoderForConditionalGeneration, T5Gemma2EncoderConfig, SiglipVisionConfig, T5Gemma2EncoderTextConfig >>> # Initializing a Siglip-like vision config >>> vision_config = SiglipVisionConfig() >>> # Initializing a T5Gemma2Encoder Text config >>> text_config = T5Gemma2EncoderTextConfig() >>> # Initializing a T5Gemma2Encoder gemma-3-4b style configuration >>> configuration = T5Gemma2EncoderConfig(vision_config, text_config) >>> # Initializing a model from the gemma-3-4b style configuration >>> model = T5Gemma2EncoderTextConfig(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```t5gemma2_encoderimage_token_indexboi_token_indexeoi_token_index)image_token_id boi_token_id eoi_token_id) text_config vision_configNrrmm_tokens_per_imager)tie_word_embeddingsc  | t}tjdnt|tr tdi|}t|tr t di|}n!|t }tjd||_||_||_||_ ||_ ||_ ||_ ||_ t |<di| y)NzItext_config is None, using default T5Gemma2EncoderTextConfig text config.zFvision_config is None, using default SiglipVisionConfig vision config.r>)r loggerinfo isinstanceryr rrrrrrr)rrDrE) rGrrrrrrr)rrHrJs rKrEzT5Gemma2EncoderConfig.__init__s  ,.K KKc d  T *,;{;K mT *.??M  ".0M KK` a&*#6 ..!2!2#6  "6"rb)NNrgiiiriT)rmrnrorprq attribute_mapr r sub_configsryrvrrurwrCrErzr{s@rKr}r}s/ b$J-))M*+KCGDH*-&-&-(/*.+/!#'$sCx.84?!#*DcN:TA!#!4Z !# t !# t !#:!#!4<!#"D[!#!#rbr}c2eZdZdZdZdgZdddddddddd Zdgd gfd d gd gfd gd gfd Zd ddZ d*de dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzd e dzd!e dzd"e dzd#e dzd$e e dzd%e dzd&e dzd'eee efzdzf.fd( Zd+d)ZxZS),T5Gemma2DecoderConfiga  This is the configuration class to store the configuration of a [`T5Gemma2DecoderModel`]. It is used to instantiate the decoder text model portion of the T5Gemma2 Model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the T5Gemma2Decoder-7B. e.g. [google/t5gemma2_text-7b](https://huggingface.co/google/t5gemma2_text-7b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 262208): Vocabulary size of the T5Gemma2Decoder model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`T5Gemma2DecoderModel`] hidden_size (`int`, *optional*, defaults to 2304): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 9216): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 26): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 4): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 256): The attention head dimension. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"` if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. eos_token_id (`int`, *optional*, defaults to 1): End of stream token id. bos_token_id (`int`, *optional*, defaults to 2): Beginning of stream token id. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. query_pre_attn_scalar (`float`, *optional*, defaults to 256): Scaling factor used on the attention scores sliding_window (`int`, *optional*, defaults to 4096): In T5Gemma2Decoder, every other layer uses sliding window attention. This is the size of the sliding window. layer_types (`list`, *optional*): Attention pattern for each layer. final_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the logits. attn_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the attention scores. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. t5gemma2_decoderrrrrrrrrrrrrrNr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6c | |_||_||_||_| |_||_||_||_||_||_ ||_ | |_ | |_ | |_ ||_||_||_||_||_||_||_||_|j-dd|_|j*Et1|jDcgc]!}t3|dz|j.zrdnd#c}|_t5|j*|j||_t9|tdi|ycc}wr8r?rFs rKrEzT5Gemma2DecoderConfig.__init__rLrMc |jdd}ddiddid}|j |jn||_||jdj||jjdddi|jd<|jdj d|jd|j d|jjdddi|jd<|jdj d|jd |j d |j |j| |SrOrVr]s rKr`z1T5Gemma2DecoderConfig.convert_rope_params_to_dictrarbrcrkrlr{s@rKrr9s&BH$J#4"5%.%.%.%E%E%."+ )"+ &(9:#%568IJ!"_$56 +X>M")"&(,(**+*+"(;.5*.#'!%#$#$#$&+*-,/%)(,04/3MQ1=#$J=#4Z=#: =# : =# !4Z =#!4Z=#*=#:=#"%t=#!4<=#Dj=#$;=#Dj=#Dj=# Dj!=#"t #=#$!4<%=#& #Tz'=#(d )=#*#Y%+=#,"'-=#.!& /=#0($sN/B*CCdJ1=#~rbrceZdZdZdZdgZeedZdddZ dd ee e e fzdzd ee e e fzdzd e d ed edededede dzffd ZxZS)T5Gemma2ConfigaV This is the configuration class to store the configuration of a [`T5Gemma2Model`]. It is used to instantiate an T5Gemma2 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to a hypothetical balanced Gemma3 encoder-decoder model. e.g. [google/t5gemma-2-270m-270m](https://huggingface.co/google/t5gemma-2-270m-270m) Configuration objects inherit from [PreTrainedConfig] and can be used to control the model outputs. Read the documentation from [PreTrainedConfig] for more information. Args: encoder (`Union[T5Gemma2EncoderConfig, dict]`, optional, *optional*): Configuration for the encoder. decoder (`Union[T5Gemma2DecoderConfig, dict]`, optional, *optional*): Configuration for the decoder. is_encoder_decoder (bool, optional, *optional*, defaults to `True`): Whether the model is used as an encoder/decoder or not. dropout_rate (`float`, *optional*, defaults to 0.0): The ratio for all dropout layers (following T5). attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for attention. classifier_dropout_rate (`float`, *optional*, defaults to 0.0): The dropout ratio for classifier (following T5). initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. image_token_index (`int`, *optional*, defaults to 256001): The image token index to encode the image prompt. Defaults to 256001, which is right after the eoi_token_index. Note this is different from Gemma 3. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings ```python >>> from transformers import T5Gemma2Config, T5Gemma2Model >>> t5gemma2_config = T5Gemma2Config.from_pretrained("google/t5gemma-270m-270m") >>> model = T5Gemma2Model(t5gemma2_config) ``` t5gemma2r)encoderdecoderrr)rrNrris_encoder_decoder dropout_rater0classifier_dropout_rater)rc  ^t|tr td i|}nI| t}tj dn't|tst t |dt|tr td i|}nI| t}tj dn't|tst t |d|jj|jk7r0t d|jjd|jd|s t d|jj|jk7r0t d|jjd|jd||j_ ||j_ ||j_ ||_||_||_ ||_ ||_d D]} | | vst#|| | | <||_||_|j(|_||_| |_t- |\d d |i| y) NzDencoder is None, using default T5Gemma2EncoderConfig encoder config.z is not supported.zDdecoder is None, using default T5Gemma2DecoderConfig decoder config.zBImbalanced encoder-decoder is not supported in T5Gemma2: encoder (z) vs decoder (z).z4T5Gemma2Model only support encoder-decoder modeling.zRImbalanced encoder-decoder vocabulary size is not supported in T5Gemma2: encoder ()r.r,r-r rr>)rryr}rr ValueErrortyperrr!r rr0rrrrgetattrrr)rrrDrE) rGrrrrr0rr)rrrHspecial_token_keyrJs rKrEzT5Gemma2Config.__init__"s+ gt $+6g6G _+-G KK^ _g'<= DM?2D!EFF gt $+6g6G _+-G KK^ _g'<= DM?2D!EFF    * *g.A.A A#//;;'J\J\I]]_a  ,8(0A-2C/$5!  ,$5! !_ P  .,3G=N,O() P(?$!2&66!2#6  I,>I&Irb) NNTrjrjrjriiT)rmrnrorprqrrr}rrrryrvrrCrwrurErzr{s@rKrrs"HJ#4"5)(K .)MBFAE#'!#&),#'!(+/DJ&c3h7$>DJ'c3h7$>DJ! DJ  DJ ! DJ"'DJ!DJDJ"D[DJDJrbr)rr r}rN)typingrconfiguration_utilsrrmodeling_rope_utilsrutilsrsiglipr get_loggerrmrr r}rr__all__r>rbrKrst*J1'   H %t)tn_#,_#Dt,tnvJ%vJr erb