qiDdZddlmZddlmZddlmZddlmZeje Z GddeZ Gd d eZ d dgZy ) zMoshi model configuration)PreTrainedConfig)RopeParameters)logging) AutoConfigcXeZdZdZdZdgZ dfd ZxZS)MoshiDepthConfiga. This is the configuration class to store the configuration of a [`MoshiDepthDecoder`]. It is used to instantiate a Moshi depth decoder model according to the specified arguments, defining the Moshi depth decoder config. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 32000): Vocabulary size of the MoshiDepthDecoder model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`MoshiDepthDecoder`]. hidden_size (`int`, *optional*, defaults to 1024): Dimensionality of the layers and the pooler layer of the depth decoder. input_size (`int`, *optional*, defaults to 4096): Dimensionality of the input hidden states. Used to connect the main decoder to the depth decoder. num_hidden_layers (`int`, *optional*, defaults to 6): Number of depth decoder layers. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the depth decoder block. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. audio_vocab_size (`int`, *optional*, defaults to 2048): Vocabulary size of the audio part of model. Defines the number of different tokens that can be represented by the `audio_codes` passed when calling the Moshi models. max_position_embeddings (`int`, *optional*, defaults to 9): The maximum sequence length that this model might ever be used with. Typically, set this to something large just in case (e.g., 512 or 1024 or 2048). hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the depth decoder. head_dim (`int`, *optional*, defaults to `hidden_size // num_attention_heads`): The attention head dimension. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. sliding_window (`int`, *optional*, defaults to 8): Sliding window attention window size. If not specified, will default to `8`. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. ffn_dim (`int`, *optional*, defaults to 5632): Dimensionality of the "intermediate" (often named feed-forward) layer in the depth decoder block. Must be even. rms_norm_eps (`float`, *optional*, defaults to 1e-08): The epsilon used by the rms normalization layers. num_codebooks (`int`, *optional*, defaults to 8): The number of audio codebooks for each audio channels. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings pad_token_id (`int`, *optional*): Padding token id. bos_token_id (`int`, *optional*): Beginning of stream token id. eos_token_id (`int`, *optional*): End of stream token id. kwargs (*optional*): Dictionary of keyword arguments. Notably: - **audio_encoder_config** ([`PreTrainedConfig`], *optional*) -- An instance of a configuration object that defines the audio encoder config. Example: ```python >>> from transformers import ( ... MoshiDepthConfig, ... MoshiDepthDecoder, ... ) >>> configuration = MoshiDepthConfig() >>> # Initializing a MoshiDepthDecoder (with random weights) from the kmhf/hf-moshiko style configuration >>> model = MoshiDepthDecoder(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` moshi_depthpast_key_valuesc ||_||_||_||_||_||n||_||_| |_| xs||z|_| |_ | |_ | |_ ||_ |dzdk(rtd|d||_||_||_||_||_||_||_||_t-|\di|y)Nr `ffn_dim=` must be even.) vocab_size hidden_size input_sizenum_hidden_layersnum_attention_headsnum_key_value_headsmax_position_embeddings hidden_acthead_diminitializer_range use_cachesliding_windowattention_dropout ValueErrorffn_dim rms_norm_eps num_codebooksaudio_vocab_sizetie_word_embeddings pad_token_id bos_token_id eos_token_idsuper__init__)selfrrrrrrr"rrrrrrrrr r!r#r$r%r&kwargs __class__s _/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/moshi/configuration_moshi.pyr(zMoshiDepthConfig.__init__ns2%&$!2#6 :M:Y#6_r '>$$ FK3F$F !2",!2 Q;! y AB B (* 0#6 ((( "6")}iNi siluN{Gz?Ti:0yE>r5FNNN)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencer( __classcell__r+s@r,r r seO bJ#4"5  !!-1#1#r-r c,leZdZdZdZdgZeedZ dde dzde dzde dzd e dzd e dzd e dzd e dzd e e e e fzdzde dzde dzde dzdedzde dzde dzde dzde dzde dzdedzde dzde dzde dzf*fd ZedZedefdZxZS) MoshiConfiga This is the configuration class to store the configuration of a [`MoshiModel`]. It is used to instantiate a Moshi model according to the specified arguments, defining the audio encoder, Moshi depth decoder and Moshi decoder configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the Moshiko model, e.g. [kmhf/hf-moshiko](https://huggingface.co/kmhf/hf-moshiko) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 32000): Vocabulary size of the MoshiDecoder model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`MoshiDecoder`]. hidden_size (`int`, *optional*, defaults to 4096): Dimensionality of the layers and the pooler layer of the main decoder. num_hidden_layers (`int`, *optional*, defaults to 32): Number of decoder layers. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the main decoder block. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. audio_vocab_size (`int`, *optional*): Vocabulary size of the audio part of model. Defines the number of different tokens that can be represented by the `audio_codes` passed when calling the Moshi models. max_position_embeddings (`int`, *optional*, defaults to 3000): The maximum sequence length that this model might ever be used with. Typically, set this to something large just in case (e.g., 512 or 1024 or 2048). rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. head_dim (`int`, *optional*, defaults to `hidden_size // num_attention_heads`): The attention head dimension. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. sliding_window (`int`, *optional*, defaults to 3000): Sliding window attention window size. If not specified, will default to `3000`. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. ffn_dim (`int`, *optional*, defaults to 22528): Dimensionality of the "intermediate" (often named feed-forward) layer in the main decoder block. Must be even. rms_norm_eps (`float`, *optional*, defaults to 1e-08): The epsilon used by the rms normalization layers. num_codebooks (`int`, *optional*, defaults to 8): The number of audio codebooks for each audio channels. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings pad_token_id (`int`, *optional*): Padding token id. bos_token_id (`int`, *optional*): Beginning of stream token id. eos_token_id (`int`, *optional*): End of stream token id. kwargs (*optional*): Dictionary of keyword arguments. Notably: - **audio_encoder_config** ([`PreTrainedConfig`], *optional*) -- An instance of a configuration object that defines the audio encoder config. - **depth__config** ([`PreTrainedConfig`], *optional*) -- An instance of a configuration object that defines the depth decoder config. Example: ```python >>> from transformers import ( ... MoshiConfig, ... MoshiForConditionalGeneration, ... ) >>> configuration = MoshiConfig() >>> # Initializing a MoshiForConditionalGeneration (with random weights) from the kmhf/hf-moshiko style configuration >>> model = MoshiForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> # Saving the model, including its configuration >>> model.save_pretrained("kmhf/hf-moshiko") >>> # loading model and config from pretrained folder >>> moshi_config = MoshiConfig.from_pretrained("kmhf/hf-moshiko") >>> model = MoshiForConditionalGeneration.from_pretrained("kmhf/hf-moshiko", config=moshi_config) ```moshir )audio_encoder_configdepth_decoder_configNrrrrrr"rrope_parametersrrrrrrrr r!r#r$r%r&c R||_||_||_||_||n||_||_| |_| xs||z|_| |_| |_ | |_ ||_ |dzdk(rtd|d||_ ||_||_||_|j#di}|j#dd}t%j&|fi||_|j|j(jkDr&td|d |j(jd ||j(j*n||_|j#d i}|j/|j,|||d t1d i||_||_||_||_||_t=||d i|y)Nrr rrrCr<mimiz`num_codebooks=zX` is greater than the maximum number of codebooks that the audio encoder can deal with (z). Please lower it.rD)r"rrr!r) rrrrrrrrrrrrrrr r!rEpopr for_modelrC codebook_sizer"updater rDr#r$r%r&r'r()r)rrrrrr"rrErrrrrrrr r!r#r$r%r&r*rCaudio_encoder_model_typerDr+s r,r(zMoshiConfig.__init__s2%&!2#6 :M:Y#6_r '>$$ FK3F$F !2",!2 Q;! y AB B (*.%zz*@"E#7#;#;L&#Q $.$8$89Q$jUi$j!    9 9 G G G!-1IJNJcJcJqJqIrrEF  8H7OD % % 3 3Ue  &zz*@"E##$($9$9)(!.   %5$L7K$L!#6 ((( "6"r-c.|jjS)N)rC sampling_rate)r)s r,rNzMoshiConfig.sampling_rateRs((666r-rCc 2|dd|ji|S)z Instantiate a [`MoshiConfig`] (or a derived class) from an audio encoder configuration. Returns: [`MoshiConfig`]: An instance of a configuration object rCr)to_dict)clsrCr*s r,from_audio_encoder_configz%MoshiConfig.from_audio_encoder_configVs* !5!=!=!?   r-)r.r/ rSNN Nr3Nr4TrTr6iXr7r5FNNN)r8r9r:r;r<r=rr sub_configsintrdictstrfloatboolr(propertyrN classmethodrrRr>r?s@r,rArAs] ~J#4"5+5O_`K"'"&(**,*.'+.2MQ!'#*.!%%)*-##'$%+0#'#'#'-J#$JJ#4ZJ#: J# !4Z J# !4Z J#*J#"%tJ#($sN/B*CCdJJ#$JJ#*J#!4<J#$;J#d J#!4<J# t!J#"Dj#J#$Tz%J#&"D['J#(Dj)J#*Dj+J#,Dj-J#X77 .  r-rAN)r;configuration_utilsrmodeling_rope_utilsrutilsrauto.configuration_autor get_loggerr8loggerr rA__all__rr-r,rdsW 310   H %F#'F#RD "D N , -r-