qiMdZddlmZddlmZddlmZejeZ GddeZ GddeZ Gd d eZ Gd d eZ Gd deZgdZy)zBlt model configuration)PreTrainedConfig)RopeParameters)loggingc eZdZdZdZdZ ddedzdedzdedzdedzd edzd edzd edzd edzd edzdedzdedzde e e e fzdzde dzdedzdedzffd Z xZ S)BltLocalEncoderConfigzB Configuration class for the Blt Local Encoder component. blt_local_encoderAN vocab_sizecross_attn_all_layers cross_attn_khidden_size_global hidden_sizenum_attention_headsnum_key_value_headsnum_hidden_layers rms_norm_epsdropoutmax_position_embeddingsrope_parameters hidden_actintermediate_sizeinitializer_rangec d||_||_||_||_||_||_|xs||_||z|_|xstd|zdz |_ ||_ | |_ | |_ | |_ | |_||_| |_|j#ddt%|Ldi|ddiy)Nrtie_word_embeddingsF)r r r r rrrhead_dimintrrrrrrrrpopsuper__init__)selfr r r r rrrrrrrrrrrkwargs __class__s [/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/blt/configuration_blt.pyr!zBltLocalEncoderConfig.__init__ s&%%:"("4&#6 #6#M:M #':: !2!Nc!k/A:M6N!2( '>$$!2.  ($/ =6=u=)FNh㈵>`Nsilu {Gz?__name__ __module__ __qualname____doc__ model_type default_thetarboolfloatrdictstrr! __classcell__r$s@r%rrs-%JM"%-2#$)-"&*,*.()%) #.3MQ!'(,*.!&>$J&> $d{&>Dj &>  $J &> 4Z &>!4Z&>!4Z&>:&>dl&>&>"%t&>($sN/B*CCdJ&>$J&>:&> !4&>r&rc("eZdZdZdZdZ ddedzdedzdedzdedzd edzd edzd edzd edzd edzdedzdedzde e e e fzdzde dzdedzdedzdedzdedzdedzdedzf&fd Z xZ S)BltLocalDecoderConfigzB Configuration class for the Blt Local Decoder component. blt_local_decoderr Nr r r r rrrrrrrrrrr pad_token_id bos_token_id eos_token_idrc p||_||_||_||_||_||_|xs||_||z|_|xstd|zdz |_ ||_ | |_ | |_ | |_ | |_||_||_||_||_d|_| |_t+|Xdi|yNrrFr)r r r r rrrrrrrrrrrrrCrDrErrr r!)r"r r r r rrrrrrrrrrrrCrDrErr#r$s r%r!zBltLocalDecoderConfig.__init__Qs.%%:"("4&#6 #6#M:M #':: !2!Nc!k/A:M6N!2( '>$$!2(((#( . "6"r&)r'Tr(r)r*r+N r-r.r/Nr0r1r2NNNFr3r?s@r%rArAIsq%JM"%-1#$)-"&*,*.()%) #.3MQ!'(,*.#'#'#'+0),#$J,# $d{,#Dj ,#  $J ,# 4Z ,#!4Z,#!4Z,#:,#dl,#,#"%t,#($sN/B*CCdJ,#$J,#:,# !4$$!2#( . "6"r&) r)r+Nr-r.Nr0ir2F)r4r5r6r7r8r9rr;rr<r=r:r!r>r?s@r%rJrJs*JM#'*,*.(*%) #.2MQ!'(,*.+0#4Z#!4Z#!4Z # : # dl ##"%t#($sN/B*CCdJ#$J#:#!4<#"D[##r&rJceZdZdZdZ ddedzdedzdedzdedzdedzd edzd edzd edzd edzd eee efzdzdedzde dzffd Z xZ S)BltPatcherConfiga Configuration class for the Blt Patcher/Entropy model component. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt patcher model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling the patcher model. hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. num_hidden_layers (`int`, *optional*, defaults to 14): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. intermediate_size (`int`, *optional*, defaults to 2048): Dimension of the MLP representations. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. blt_patcherNr rrrrrrrrrrrc  0||_||_||_||_||z|_||n||_||_||_||_d|_ | xstd|jzdz |_ | |_ | |_ d|_t|@di| y)Nr0rrFr)r rrrrrrrrrrrrrrr r!)r"r rrrrrrrrrrrr#r$s r%r!zBltPatcherConfig.__init__s %&!2#6 #':: :M:Y#6_r '>$(  !2!Sc!d>N>N:NQR:R6S!2.#(  "6"r&) r'i Ni r-r.r)Nr2F) r4r5r6r7r8rr;rr<r=r:r!r>r?s@r%rPrPs#JJ"%"%(**,*..2%) #(,MQ*.+0#$J#4Z#: # !4Z # !4Z #"%t#dl##:#($sN/B*CCdJ#!4<#"D[##r&rPc0ZeZdZdZdZdgZdZeee e dZ dde dzde dzd e dzd e dzd edzd edzd e dzde dzde dzde dzde dzde dzdedzdedzdedzdedzde dzde dzde dzde dzdedzdeeeefzdzf,fd ZxZS) BltConfiga< This is the configuration class to store the configuration of a [`BltModel`]. It is used to instantiate a Blt model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`BltModel`]. max_position_embeddings (`int`, *optional*, defaults to 4096): The maximum sequence length that this model might ever be used with. patch_in_forward (`bool`, *optional*, defaults to `True`): Whether to perform patching during the forward pass. patch_size (`int`, *optional*, defaults to 4): Size of the patches used in the patching mechanism. patching_mode (`str`, *optional*, defaults to `"entropy"`): The mode used for patching, such as entropy-based patching. patching_threshold (`float`, *optional*, defaults to 1.34): Threshold value used for determining when to apply patches. patching_batch_size (`int`, *optional*, defaults to 1): Batch size used during the patching process. max_patch_length (`int`, *optional*): Maximum length of patches that can be generated. cross_attn_k (`int`, *optional*, defaults to 2): Number of cross-attention heads used in the model. encoder_hash_byte_group_size (`list`, *optional*): List of byte group sizes used in the encoder hash function. encoder_hash_byte_group_vocab (`int`, *optional*, defaults to 500002): Vocabulary size for the encoder hash byte groups. encoder_hash_byte_group_nb_functions (`int`, *optional*, defaults to 1): Number of hash functions used in the encoder byte grouping. patcher_config (`BltPatcherConfig`, *optional*): Configuration for the patcher component of the model. encoder_config (`BltLocalEncoderConfig`, *optional*): Configuration for the local encoder component of the model. decoder_config (`BltLocalDecoderConfig`, *optional*): Configuration for the local decoder component of the model. global_config (`BltGlobalTransformerConfig`, *optional*): Configuration for the global transformer component of the model. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. ```python >>> from transformers import BltModel, BltConfig >>> # Initializing a Blt configuration >>> configuration = BltConfig() >>> # Initializing a model from the configuration >>> model = BltModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` Checkpoint: [facebook/blt](https://huggingface.co/facebook/blt) bltpast_key_valuesr )patcher_configencoder_configdecoder_config global_configNr rpatch_in_forward patch_size patching_modepatching_thresholdpatching_batch_sizemax_patch_lengthr encoder_hash_byte_group_sizeencoder_hash_byte_group_vocab$encoder_hash_byte_group_nb_functionsrYrZr[r\rrCrDrErrc ||_||_||_||_||_||_||_||_||_|jdd|_ |jdd|_ |jd|_ |jdd|_ | |_| xsgd|_| |_| |_| 't%| |_t(j+d nJt-| t.r#| j1d |t%di| |_nt-| t$r| |_|'t3| |_t(j+d nJt-|t.r#|j1d |t3di||_nt-|t2r||_|'t7| |_t(j+d nJt-|t.r#|j1d |t7di||_nt-|t6r||_|'t;| |_t(j+dnJt-|t.r#|j1d |t;di||_nt-|t:r||_|j4j>|jz}||j<j>k7r|nd|j<_ ||_!||_"||_#||_$||_%tM|di|y)Npatching_devicecudarealtime_patchingTpatching_threshold_add monotonicityF)rr)rz8patcher_config is None, using default Blt patcher configrz8encoder_config is None, using default Blt encoder configz8decoder_config is None, using default Blt decoder configz6global_config is None, using default Blt global configr)(r rrr]r^r_r`rarbgetrgrirjrkr rcrdrerPrYloggerinfo isinstancer< setdefaultrrZrAr[rJr\rencoder_cross_output_sizerCrDrErrr r!)r"r rr]r^r_r`rarbr rcrdrerYrZr[r\rrCrDrErrr#rur$s r%r!zBltConfig.__init__@s6%'>$!2!1$*"4#6 0%zz*;VD!',?!F&,jj1I&J#"JJ~u=)-I,^L^)-J*4X1  !"2EV"WD  KKR S  -  % %&9;L M"2"D^"DD  (8 9"0D   !"7J["\D  KKR S  -  % %&9;L M"7"I."ID  (= >"0D   !"7J["\D  KKR S  -  % %&9;L M"7"I."ID  (= >"0D   !;N_!`D  KKP Q  t ,  $ $%8:K L!;!Lm!LD   'A B!.D %)$7$7$C$CdFWFW$W!)BdFXFXFdFd)d %jn 4)((#6 . "6"r&)r'rNTrlentropyg]?r,Nr(Ni"r,NNNNFNNNr2N)r4r5r6r7r8keys_to_ignore_at_inferencer9rPrrArJ sub_configsrr:r=r;r<rr!r>r?s@r%rVrVs@DJ#4"5M*//3 K"%.2(,!"$-+<*+'+#$374:;<&*&*&*%)+0#'#'#'*.MQ/d#$Jd#"%td#+ d# $J d# Tz d#"DLd#!4Zd#*d#Djd#'*Djd#(+Tzd#/2Djd#t d#t d# t !d#"d{#d#$"D[%d#&Dj'd#(Dj)d#*Dj+d#,!4<-d#.($sN/B*CCdJ/d#d#r&rV)rVrPrrArJN)r7configuration_utilsrmodeling_rope_utilsrutilsr get_loggerr4rqrrArJrPrV__all__rr&r%r~sx31   H %.>,.>b4#,4#n&#!1&#RG#'G#Tq# q#h r&