qidZddlmZddlmZddlmZejeZ GddeZ Gdd eZ Gd d eZ Gd d eZ GddeZgdZy)zFLAVA model configurations)Any)PreTrainedConfig)loggingceZdZdZdZdZ ddededededed ed ed ed ed edededededeffd Z xZ S)FlavaImageConfiga This is the configuration class to store the configuration of a [`FlavaImageModel`]. It is used to instantiate an FLAVA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. image_size (`int`, *optional*, defaults to 224): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 16): The size (resolution) of each patch. num_channels (`int`, *optional*, defaults to 3): The number of input channels. qkv_bias (`bool`, *optional*, defaults to `True`): Whether to add a bias to the queries, keys and values. mask_token (`bool`, *optional*, defaults to `True`): Whether to use a mask token or not. Used in MIM (Masked Image Modeling) loss for FLAVA. vocab_size (`int`, *optional*, defaults to 8192): Vocabulary size of the [`FlavaImageCodebook`] used in conjunction with [`FlavaImageModel`] for MIM (Masked Image Modeling) loss for FLAVA. Example: ```python >>> from transformers import FlavaImageConfig, FlavaImageModel >>> # Initializing a FlavaImageModel with style configuration >>> configuration = FlavaImageConfig() >>> # Initializing a FlavaImageModel model (with random weights) from the style configuration >>> model = FlavaImageModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```flava_image_model image_config hidden_sizenum_hidden_layersnum_attention_headsintermediate_size hidden_acthidden_dropout_probattention_probs_dropout_probinitializer_rangelayer_norm_eps image_size patch_size num_channelsqkv_bias mask_token vocab_sizec t|di|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_||_yN)super__init__r r r rrrrrrrrrrrr)selfr r r rrrrrrrrrrrrkwargs __class__s _/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/flava/configuration_flava.pyrzFlavaImageConfig.__init__Ys& "6"&!2#6 !2$#6 ,H)!2,$$(  $$) r% gelur({Gz?-q=rTT __name__ __module__ __qualname____doc__ model_typebase_config_keyintfloatboolr __classcell__r!s@r"rrs: x%J$O!##%!% %(.1#' %!#%#%#%! #%  #%  #%##%',#%!#%#%#%#%#%#%#% !#%#%r#rceZdZdZdZdZ ddededededed ed ed ed ed edededede ffd Z xZ S)FlavaTextConfiga This is the configuration class to store the configuration of a [`FlavaTextModel`]. It is used to instantiate an FLAVA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 30522): Vocabulary size of the BERT model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`FlavaTextModel`]. type_vocab_size (`int`, *optional*, defaults to 2): The vocabulary size of the `token_type_ids` passed when calling [`FlavaTextModel`]. Note that even though text encoder allows `token_type_ids`'s value as 2, for text-only pretraining and fine-tuning, only 1 is used similar to RoBERTa. max_position_embeddings (`int`, *optional*, defaults to 512): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). For VL, max_length passed to model is 77. hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. image_size (`int`, *optional*, defaults to 224): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 16): The size (resolution) of each patch. num_channels (`int`, *optional*, defaults to 3): The number of input channels. qkv_bias (`bool`, *optional*, defaults to `True`): Whether to add a bias to the queries, keys and values. Example: ```python >>> from transformers import FlavaTextConfig, FlavaTextModel >>> # Initializing a FlavaTextModel with style configuration >>> configuration = FlavaTextConfig() >>> # Initializing a FlavaTextModel model (with random weights) from the style configuration >>> model = FlavaTextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```flava_text_model text_configrtype_vocab_sizemax_position_embeddingsr r r rrrrrr pad_token_idrc t|di|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ ||_| |_yr)rrrr>r?r r r rrrrrrrr@)rrr>r?r r r rrrrrrr@rr r!s r"rzFlavaTextConfig.__init__s$ "6"$.'>$&!2#6 !2$#6 ,H)!2,  (r#)i:wir$r%r%r&r'r(r(r)r*rT) r/r0r1r2r3r4r5strr6r7rr8r9s@r"r;r;s? B$J#O  '*!##%!% %(.1#' %!)!)!)"% !)  !)  !)!!)!)!)#!)',!)!!)!)!)!)!)r#r;cpeZdZdZdZdZ ddededededed ed ed ed ed edeffd Z xZ S)FlavaMultimodalConfiga This is the configuration class to store the configuration of a [`FlavaMultimodalModel`]. It is used to instantiate an FLAVA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 6): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. qkv_bias (`bool`, *optional*, defaults to `True`): Whether to add a bias to the queries, keys and values. use_cls_token (`bool`, *optional*, defaults to `True`): Whether to use an extra CLS token for multimodal settings. Usually needed by the FLAVA model. Example: ```python >>> from transformers import FlavaMultimodalConfig, FlavaMultimodalModel >>> # Initializing a FlavaMultimodalModel with style configuration >>> configuration = FlavaMultimodalConfig() >>> # Initializing a FlavaMultimodalModel model (with random weights) from the style configuration >>> model = FlavaMultimodalModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```flava_multimodal_modelmultimodal_configr r r rrrrrrr use_cls_tokenc  t |di| ||_||_||_||_||_||_||_||_ | |_ | |_ | |_ yr) rrr r r rrrrrrrrH)rr r r rrrrrrrrHr r!s r"rzFlavaMultimodalConfig.__init__ sk "6"&!2#6 !2$#6 ,H)!2,  *r#) r$r%r&r'r(r(r)r*TTr.r9s@r"rErEs2 h*J)O!"#%!% #&,/#' %"+++! +  +  +!+'*+!+++++r#rEcVeZdZdZdZ d dedededededed effd ZxZS) FlavaImageCodebookConfigflava_image_codebookimage_codebook_config num_groupsinput_channelsnum_blocks_per_groupr rfreezerc t |di|||_||_||_||_||_||_||_yr) rrrOrPrQr rrRr) rrOrPrQr rrRrr r!s r"rz!FlavaImageCodebookConfig.__init__msJ "6"$,$8!&$ !2r#)rrBr-Tr)) r/r0r1r3r4r5r6rr8r9s@r"rLrL>sx'J-O)Z$%#'333" 3  3  33!33r#rLc,eZdZdZdZeeeedZ dde e e fdzde e e fdzde e e fdzde e e fdzd e d ed e d ed edede dededededededededededzf*fd ZxZS) FlavaConfiga$ [`FlavaConfig`] is the configuration class to store the configuration of a [`FlavaModel`]. It is used to instantiate FLAVA model according to the specified arguments, defining the text model, image model, image codebook and multimodal model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: text_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`FlavaTextConfig`]. image_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`FlavaImageConfig`]. multimodal_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`FlavaMultimodalConfig`]. hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. projection_dim (`int`, *optional*, defaults to 512): Dimensionality of text and image projection layers. logit_scale_init_value (`float`, *optional*, defaults to 2.6592): The initial value of the *logit_scale* parameter. Default is used as per the original FLAVA/CLIP implementation. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. ce_ignore_index (`int`, *optional*, defaults to -100): Cross entropy index to ignore. mim_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MIM (Masked Image Modeling) unimodal loss mlm_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MLM (Masked Language Modeling) unimodal loss global_contrastive_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to global contrastive cross-alignment loss. itm_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to image-text matching multimodal loss. mmm_image_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MMM loss's image part. mmm_text_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MMM loss's text part. global_backprop_contrastive (`bool`, *optional*, defaults to `True`): Whether to use global backpropgation through all workers in contrastive loss. skip_unmasked_multimodal_encoder (`bool`, *optional*, defaults to `True`): Whether to skip running unmasked multimodal encoder whose outputs are not used by FLAVA losses. return_loss (`bool`, *optional*, defaults to `True`): Whether to return loss or not tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Example: ```python >>> from transformers import FlavaConfig, FlavaModel, FlavaForPreTraining >>> # Initializing a FlavaConfig with style configuration >>> configuration = FlavaConfig() >>> # Initializing a FlavaModel and FlavaForPreTraining model (with random weights) from the style configuration >>> model = FlavaModel(configuration) >>> model_pre = FlavaForPreTraining(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> configuration_pre = model_pre.config ``` flava)r=r rGrNNr r=rGrNr rprojection_dim init_codebooklogit_scale_init_valuerce_ignore_index mim_weight mlm_weightglobal_contrastive_weight itm_weightmmm_image_weightmmm_text_weightglobal_backprop_contrastive skip_unmasked_multimodal_encoder return_losstie_word_embeddingsc r|jdd}|jdd}|jdd}|jdd}||i}tdi|j}|jD]B\}}||vs |||k7s|dk7s||vr d|d|d}nd |d }tj |D|j |||i}tdi|j}d |vr3|d jDcic]\}}t||c}}|d <|jD]B\}}||vs |||k7s|dk7s||vr d|d |d}nd |d }tj |D|j |||i}tdi|j} | jD]B\}}||vs |||k7s|dk7s||vr d|d|d}nd|d }tj |D|j | ||i}tdi|j}!|!jD]B\}}||vs |||k7s|dk7s||vr d|d|d}nd|d }tj |D|j |!| t}tj dnt|tr tdi|}| t}tj dnt|tr tdi|}| t}tj dnt|tr tdi|}| t}tj dnt|tr tdi|}||_ ||_||_||_||_||_||_||_| |_| |_d|_| |_| |_| |_||_||_||_||_||_||_ ||_!||_"tG"|di|ycc}}w)Ntext_config_dictimage_config_dictmultimodal_config_dictimage_codebook_config_dicttransformers_version`zp` is found in both `text_config_dict` and `text_config` but with different values. The value `text_config_dict["z"]` will be used instead.zk`text_config_dict` is provided which will be used to initialize `FlavaTextConfig`. The value `text_config["z"]` will be overridden.id2labelzs` is found in both `image_config_dict` and `image_config` but with different values. The value `image_config_dict["zn`image_config_dict` is provided which will be used to initialize `FlavaImageConfig`. The value `image_config["z` is found in both `multimodal_config_dict` and `multimodal_config` but with different values. The value `multimodal_config_dict["z}`multimodal_config_dict` is provided which will be used to initialize `FlavaMultimodalConfig`. The value `multimodal_config["z` is found in both `image_codebook_config_dict` and `image_codebook_config` but with different values. The value `image_codebook_config_dict["z`image_codebook_config_dict` is provided which will be used to initialize `FlavaImageCodebookConfig`. The value `image_codebook_config["zP`text_config` is `None`. initializing the `FlavaTextConfig` with default values.zR`image_config` is `None`. initializing the `FlavaImageConfig` with default values.zW`image_config` is `None`. initializing the `FlavaMultimodalConfig` with default values.zZ`image_config` is `None`. initializing the `FlavaImageCodebookConfig` with default values.?r)%popr;to_dictitemsloggerinfoupdaterrCrErL isinstancedictr=r rGrNrYrZr rrr[initializer_factorr\r]r^r_r`rarbrcrdrerfrr)#rr r=rGrNr rrYrZr[rr\r]r^r_r`rarbrcrdrerfr rhrirjrk_text_config_dictkeyvaluemessage_image_config_dict_multimodal_config_dict_image_codebook_config_dictr!s# r"rzFlavaConfig.__init__sG8"::&8$?"JJ':DA!',Dd!K%+ZZ0Ld%S"  '" !0 C2B C K K M 0557 ) U+%%;s3C*COeHe..u%<T&U&]&]&_ #6;;= ) U++9J39O0OTW[qTq44u%TTWSXXqs VVYUZZqs KK( )"  $ $%< = % 1$,(*%+C*`E_*`*h*h*j ':??A ) U00!6s!;;5588u%]]`\ab'' ]]`\aaxz KK(' ), " ( ()D E  )+K KKj k  T *)8K8K  +-L KKl m  d ++;l;L  $ 5 7  KKq r )4 0 5 J8I J  ($<$> ! KKt u -t 4$<$U?T$U !&(!2%:",*&,!2&<#"%.$$)B&$ 0.+F(0P-&#6  "6"K2sP3)NNNNr$r*r$Tg/L F@r)irorororororoTTTT)r/r0r1r2r3r;rrErL sub_configsrwrCrr5r6r7rr8r9s@r"rWrWsBHJ&(2!9 K/3-1377; %!"(.#'#+."%!$,015 +/-L#38nt+L##s(^d*L# S>D0 L# $CH~4 L#  L#L#L#L#!&L#!L#L#L#L#$)L# !L#" #L#$%L#&&*'L#(+/)L#*+L#,"D[-L#L#r#rW)rWrLrrEr;N)r2typingrconfiguration_utilsrutilsr get_loggerr/rsrr;rErLrW__all__rr#r"rs!3   H %c%'c%Lf)&f)RS+,S+lA3/A3HY#"Y#x vr#