qi3GddlmZddlmZddlmZddlmZeje Z GddeZ Gdd eZ dd gZ y ) )PreTrainedConfig)RopeParameters)logging) AutoConfigc,LeZdZdZdZdZdgZddiZdZ dd e dzd e dzde dzd e dzd e dzd e dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde dzde e ee fzdzde dzde dzde dzde dzf*fd ZxZS)CsmDepthDecoderConfiga| This is the configuration class to store the configuration of a [`CsmDepthDecoderModel`]. It is used to instantiate an CSM depth decoder model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the csm-1b. e.g. [sesame/csm-1b](https://huggingface.co/sesame/csm-1b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: num_codebooks (`int`, *optional*, defaults to 32): Number of codebooks used in the underlying codec model responsible for tokenizing the audio. backbone_hidden_size (`int`, *optional*, defaults to 2048): Dimension of the hidden representations of the backbone model used with this depth decoder. vocab_size (`int`, *optional*, defaults to 2051): Vocabulary size of the CsmDepthDecoder model. Defines the number of different audio tokens that can be represented by each codebook. hidden_size (`int`, *optional*, defaults to 1024): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 8192): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 4): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 2): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 33): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 2050): Padding token id. bos_token_id (`int`, *optional*): Beginning of stream token id. eos_token_id (`int`, *optional*): End of stream token id. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. head_dim (`int`, *optional*): The attention head dimension. If None, it will default to hidden_size // num_attention_heads ```python >>> from transformers import CsmDepthDecoder, CsmDepthDecoderConfig >>> # Initializing a CsmDepthDecoder >>> configuration = CsmDepthDecoderConfig() >>> model = CsmDepthDecoderModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```csm_depth_decoder_modeldepth_decoder_configpast_key_values codebook_size vocab_sizeAN num_codebooksbackbone_hidden_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_heads hidden_actmax_position_embeddingsinitializer_range rms_norm_eps use_cache pad_token_id bos_token_id eos_token_idrope_parametersattention_biasattention_dropoutmlp_biashead_dimc |jddr td||_||_||_||_||_||_| |_||_ ||_ ||_ ||_ ||}||_ | |_| |_| |_| |_||_||_||_||n|j|jz|_||_t/|`di|y)Ntie_word_embeddingsFzE`tie_word_embeddings=True` is not supported for CsmDepthDecoderConfig)pop ValueErrorrrrrrrrrrrrrrrrrr r!r"r#rsuper__init__)selfrrrrrrrrrrrrrrrrrr r!r"r#kwargs __class__s [/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/csm/configuration_csm.pyr*zCsmDepthDecoderConfig.__init__ms2 ::+U 3de e(((*$$8!'>$&!2!2#6   &"5 #6 $!2(",!2  $,$8d>N>NRVRjRj>j . "6") i rsilu!{Gz?h㈵>TNNNNFFN)__name__ __module__ __qualname____doc__ model_typebase_config_keykeys_to_ignore_at_inference attribute_map default_thetaintfloatboolrdictstrr* __classcell__r-s@r.r r sI V+J,O#4"5MM%'+/!%"&(,()*+*+!'.0*.#'!%#'#'#'MQ&+*- %#-6#Tz6#"Dj6#$J 6# 4Z 6# : 6#:6#!4Z6#!4Z6#$J6#"%t6#!4<6#Dj6#$;6#Dj6# Dj!6#"Dj#6#$($sN/B*CCdJ%6#&t '6#(!4<)6#*++6#,*-6#6#r/r c:eZdZdZdZdZdgZdZee dZ ddiZ d&d e d zde d zd e d zd e d zd e d zde d zde d zde d zde d zde d zded zde d zded zde d zde d zde d zde d zde d zde d zde d zdeee efzd zded zded zd ed zd!e d zd"ed zd#ed zd$ed zf8fd% ZxZS)' CsmConfigaW This is the configuration class to store the configuration of a [`CsmForConditionalGeneration`]. It is used to instantiate an CSM model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the csm-1b. e.g. [sesame/csm-1b](https://huggingface.co/sesame/csm-1b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: num_codebooks (`int`, *optional*, defaults to 32): Number of codebooks used in the underlying codec model responsible for tokenizing the audio. vocab_size (`int`, *optional*, defaults to 2051): Vocabulary size of the Csm model. Defines the number of different audio tokens that can be represented by each codebook. text_vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the text input for the Csm model. Defines the number of different text tokens that can be represented. hidden_size (`int`, *optional*, defaults to 2048): Dimension of the hidden representations of the backbone model. intermediate_size (`int`, *optional*, defaults to 8192): Dimension of the MLP representations of the backbone model. num_hidden_layers (`int`, *optional*, defaults to 16): Number of hidden layers in the backbone model Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the backbone model Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 8): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the backbone model Transformer decoder. max_position_embeddings (`int`, *optional*, defaults to 2048): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 128002): Padding token id. codebook_pad_token_id (`int`, *optional*, defaults to 2050): Padding token id for codebook tokens. codebook_eos_token_id (`int`, *optional*, defaults to 0): End of stream token id for codebook tokens. bos_token_id (`int`, *optional*, defaults to 128000): Beginning of stream token id. eos_token_id (`int`, *optional*): End of stream token id. audio_token_id (`int`, *optional*, defaults to 128002): Audio token id in the text input. audio_eos_token_id (`int`, *optional*, defaults to 128003): End of stream token id for audio in the text input. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. head_dim (`int`, *optional*): The attention head dimension. If None, it will default to hidden_size // num_attention_heads tie_codebooks_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie the codebook tokens embeddings of the backbone model to the codebook tokens embeddings of the depth decoder. depth_decoder_config (`CsmDepthDecoderConfig`, *optional*): Configuration for the depth decoder. codec_config (`PreTrainedConfig`, *optional*): Configuration for the codec. ```python >>> from transformers import CsmForConditionalGeneration, CsmConfig >>> # Initializing a CsmConfig >>> configuration = CsmConfig() >>> # Initializing a model >>> model = CsmForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```csm csm_configr r) codec_configr r rNrtext_vocab_sizerrrrrrrrrrrcodebook_pad_token_idcodebook_eos_token_idrraudio_token_idaudio_eos_token_idrr r!r"r#tie_codebooks_embeddingsr rOc |jddr td|%t|_tj dn8t |trtdi||_nt |tr||_|0tjd|_ tj dnBt |trtjdi||_ nt |tr||_ ||_ ||_ ||_||_||_||_||_||_| |_||_||_||_||_||}||_| |_| |_| |_| |_||_||_||_ ||n|j*|j0z|_!||_"||_#||_$||_%d|_&tO|di|y)Nr%Fz9`tie_word_embeddings=True` is not supported for CsmConfigzAdepth_decoder_config is None, using default depth decoder config.mimiz9codec_config is None, using default audio encoder config.r&))r'r(r r loggerinfo isinstancerGr for_modelrOrrPrrSrTrQrRrUrrrrrrrrrrrr r!r"r#rrrrr%r)r*)r+rrrPrrrrrrrrrrrrQrRrrrSrTrr r!r"r#rUr rOr,r-s r.r*zCsmConfig.__init__ s@ ::+U 3XY Y  '(=(?D % KK[ \ ,d 3(=(U@T(UD % ,.C D($&!2!2#6   &"5 #6 $!2(",!2  $,$8d>N>NRVRjRj>j .(((#(  "6"r/)r0r2ir1r3r0r5r6r1r8r9TiiNr]iNFr:FNTNN)r;r<r=r>r?r@rArCrr sub_configsrBrDrHrErFrrGr*rIrJs@r.rLrLsSW rJ"O#4"5M" 5K M %'!%&,"&(,(**,*+!'.2*.#'!%#),0,-#)#'%+)/MQ&+*- %#04,0$(;U#TzU#$JU#t U# 4Z U# : U#:U#!4ZU#!4ZU#$JU#"%tU#!4<U#DjU#$;U#DjU# #Tz!U#" #Tz#U#$Dj%U#&Dj'U#(d )U#* $J+U#,($sN/B*CCdJ-U#.t /U#0!4<1U#2+3U#4*5U#6#'+7U#8#Tk9U#:Tk;U#U#r/rLN)configuration_utilsrmodeling_rope_utilsrutilsrauto.configuration_autor get_loggerr;rXr rL__all__r&r/r.rfsV 410   H %J#,J#Z{# {#~ r/