qiLddlmZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZmZmZdd lmZmZmZmZmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&dd l'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;mm?Z?ddl@mAZAmBZBmCZCmDZDmEZEmFZFmGZGddlHmIZImJZJmKZKmLZLddlMmNZNe9jePZQdZRGdde?ZSGddeZTGddeGZUGdd ejZWGd!d"ejZXGd#d$ejZYGd%d&eZZGd'd(e,d)*Z[Gd+d,e.d)*Z\e7Gd-d.e/Z]Gd/d0eDZ^Gd1d2ejZ_Gd3d4ejZ`Gd5d6ejZaGd7d8eAZbGd9d:eBZce7Gd;deFZeGd?d@eEZfGdAdBedeCZgGdCdDeIZhGdEdFeLZiGdGdHeKZje7dIJGdKdLeJZkgdMZly)N)IterableN)nn)initialization)ACT2FN)Cache)PreTrainedConfig)BaseImageProcessor BatchFeatureget_patch_output_sizeselect_best_resolution) PaddingModeconvert_to_rgbpadresizeto_channel_dimension_format) ChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imagemake_flat_list_of_imagesto_numpy_array valid_imagesvalidate_preprocess_arguments)FlashAttentionKwargs)BaseModelOutputWithPooling)PreTrainedModel) ImagesKwargsMultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput) TensorTypeTransformersKwargsauto_docstringcan_return_tuplelogging)CONFIG_MAPPING AutoConfig AutoTokenizer) LlamaConfig)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModelLlamaPreTrainedModel LlamaRMSNorm)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPast)divide_to_patchesc|jd}|jd}tj|||j|j}tj |d}tjdtj |j}tj||f}t|jdD]2}||} ||dz} || | } tj| ||} | || | 4|S)a* Compute the matrix multiplication (GEMM) for each expert sequentially. This approach is computationally inefficient, especially when dealing with a large number of experts. Args: token_states (torch.Tensor): Input tensor of shape (num_tokens, in_features). expert_weights (torch.Tensor): Weight tensor of shape (num_experts, in_features, out_features). tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). rdtypedevicedim) shapetorchzerosr@rAcumsumlongcatrangematmul) token_statesexpert_weightstokens_per_expert num_tokens out_featuresoutputcumsum_num_tokens zero_tensor expert_numstartendtokensouts W/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/aria/modular_aria.pysequential_experts_gemmr[Cs##A&J!''+L [[\9K9KT`TgTg hF %6A>++auzz:K:R:RSK ;0A"BCN0034 !*- Q/eC(ll6>*#=>uS   Mc \eZdZdZdZdZddddddddZ d deded ed effd ZxZ S) AriaTextConfigau This class handles the configuration for the text component of the Aria model. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the Aria [rhymes-ai/Aria](https://huggingface.co/rhymes-ai/Aria) architecture. This class extends the LlamaConfig to include additional parameters specific to the Mixture of Experts (MoE) architecture. Args: vocab_size (`int`, *optional*, defaults to 32000): Vocabulary size of the LLaMA model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`LlamaModel`] hidden_size (`int`, *optional*, defaults to 4096): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 4096): The size of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 2048): The maximum sequence length that this model might ever be used with. Llama 1 supports up to 2048 tokens, Llama 2 up to 4096, CodeLlama up to 16384. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 2): Padding token id. bos_token_id (`int`, *optional*, defaults to 1): Beginning of stream token id. eos_token_id (`int`, *optional*, defaults to 2): End of stream token id. pretraining_tp (`int`, *optional*, defaults to 1): Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this document](https://huggingface.co/docs/transformers/main/perf_train_gpu_many#tensor-parallelism) to understand more about it. This value is necessary to ensure exact reproducibility of the pretraining results. Please refer to [this issue](https://github.com/pytorch/pytorch/issues/76232). tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. head_dim (`int`, *optional*): The attention head dimension. If None, it will default to hidden_size // num_heads moe_num_experts (`int`, *optional*, defaults to 8): The number of experts in the MoE layer. moe_topk (`int`, *optional*, defaults to 2): The number of top experts to route to for each token. moe_num_shared_experts (`int`, *optional*, defaults to 2): The number of shared experts. aria_text text_configcolwiserowwise)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projz%layers.*.mlp.shared_experts.gate_projz#layers.*.mlp.shared_experts.up_projz%layers.*.mlp.shared_experts.down_projintermediate_sizemoe_num_expertsmoe_topkmoe_num_shared_expertsc `||_||_||_||_t |dd|i|y)N pad_token_id)rcrdrerfsuper__init__)selfrcrdrerfrh super_kwargs __class__s rZrkzAriaTextConfig.__init__s9"3.  &<# ClClCr\)ir,r,r,) __name__ __module__ __qualname____doc__ model_typebase_config_keybase_model_tp_planintrk __classcell__rns@rZr^r^bszENJ#O%.%.%.%.1:/81:"& &' D D D D !$ D Dr\r^czeZdZdZdZddiZeedZ d de ded e dzde dzd e dzd e dzf fd Z xZS) AriaConfiga This class handles the configuration for both vision and text components of the Aria model, as well as additional parameters for image token handling and projector mapping. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the Aria [rhymes-ai/Aria](https://huggingface.co/rhymes-ai/Aria) architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vision_config (`AriaVisionConfig` or `dict`, *optional*): Configuration for the vision component. vision_feature_layer (`int`, *optional*, defaults to -1): The index of the layer to select the vision feature. text_config (`AriaTextConfig` or `dict`, *optional*): Configuration for the text component. projector_patch_to_query_dict (`dict`, *optional*): Mapping of patch sizes to query dimensions. image_token_index (`int`, *optional*, defaults to 9): Index used to represent image tokens. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated normal initializer for initializing all weight matrices. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings ariaimage_token_idimage_token_index)r` vision_configNvision_feature_layerr`projector_patch_to_query_dictinitializer_rangetie_word_embeddingsc ||_|ddd}|jD cic]\} } t| t| c} } |_t |jj |_||_t|trd|d<t|ddi|}n| td}||_ ||_ t|trd|vr tdi|}n | t}||_||_t! |Ddi|ycc} } w)N)ii$idefics3_visionrtri)r~itemsrwrmaxvalues'max_value_projector_patch_to_query_dictr isinstancedictr-rrr^r`rrjrk) rlrrr`rr~rrkwargskvrns rZrkzAriaConfig.__init__s"3 ) 0- )JgIlIlIn-oAc!fc!fn-o*7:4;];];d;d;f7g4$8! mT **;M, '*=+FGX-XM  "*+<=?M*!2 k4 (\[-H(7;7K  (*K&#6  "6").psD)Nr>NN g{Gz?F)rprqrrrsrt attribute_mapr^r. sub_configsrwrfloatboolrkrxrys@rZr{r{s4J-M#1:NK$&&*59()*.+0(#"(#$ (# (,d{ (# : (#!4<(#"D[(#(#r\r{c eZdZy)AriaTextRMSNormNrprqrrrir\rZrrr\rc(eZdZdZfdZdZxZS)AriaProjectorMLPa! Feed-Forward Network module for the Aria Projector. Args: in_features (`int`): Input embedding dimension. hidden_features (`int`): Hidden dimension of the feed-forward network. output_dim (`int`): Output dimension. ct|tj||d|_tj||d|_t d|_y)NFbiasgelu_new)rjrkrLinear linear_in linear_outract)rl in_featureshidden_features output_dimrns rZrkzAriaProjectorMLP.__init__#sB ;eL))OZeL*%r\ch|j|j|}|j|}|SN)rrr)rl hidden_statess rZforwardzAriaProjectorMLP.forward)s- !>?  6 r\rprqrrrsrkrrxrys@rZrrs & r\rc6eZdZdZddedeffd ZddZxZS)AriaCrossAttentionzv Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. config dropout_ratecBt||jj}|jj}||_t j||d|_t j||d|_ t j||d|_ t j||d|_ t j|||_ t j||_t j ||_t j ||_y)NFrT) batch_first)rjrkr hidden_sizenum_attention_heads num_headsrrq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)rlrrrrrns rZrkzAriaCrossAttention.__init__8s **66 ((<< "ii [uE ii [uE ii [uE !33KX\]ii [9 zz,/ ,,{3\\+6r\c|j|j|}|j|}|j|}|j |}|j ||||\}}|j |j|}|S)a Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)rrrrrrrr) rlkey_value_statesrrquerykeyvalue attn_output_s rZrzAriaCrossAttention.forwardIs  DOOM:;--.>?kk*+ ,-,,UC),T Qll4;;{#;< r\)rr) rprqrrrsr{rrkrrxrys@rZrr/s 7z77"r\rcheZdZdZdeffd ZddejdejdzfdZxZ S) AriaProjectora  Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. rct||j|_|jj |_|jj|_|jj |_ |jj |_ |jj |_ tjtj |j"|j |_t'||_tj*|j |_t/|j |j|j|_yr)rjrkrpatch_to_query_dictrrrrrkv_dimr`rrr ParameterrFrGrrr cross_attnrrr feed_forwardrlrrns rZrkzAriaProjector.__init__qs #)#G#G !//;;--AA**66 %11== ,,88\\%++f.\.\^b^n^n"op ,V4,,t'7'78,T-=-=t?S?SUYUdUder\Nrrc4|jd|jd}}||jvr*td|d|jjd|j|}|jd|j dj |dd}|M|j|jd}|j djd|jdd}|j|||}|j|j|}|S) a Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). rrDzNumber of patches z: not found in patch_to_query_dict amongst possible values .Nr>r)rErKeyErrorkeysr unsqueezerepeatrepeat_interleaverexpandsizerrr) rlrr batch_size num_patches query_numqueries attention_outrYs rZrzAriaProjector.forwards6#3"8"8";=M=S=STU=VK d66 6$[M1klpmEmEmJmJmLlMMNO ,,[9 **Zi(2215<? r\r) rprqrrrsr{rkrFTensorrrxrys@rZrrfs8ff( PTATr\rcpeZdZdZgdZddddddddddej f d eedzd eedzd e d e d ee e e fdzde dzde dzde de ezde dzdeffd Z ddddddddddde jdf deeezd eeezdzd eeezdzd e dzd e dzde dzde dzde dzdedzde dzdedzdeezdzde dzdee zdzfdZdej(de de dej(fdZde de fd Zdej(de de dej(fd!Zej2d"ddfdej(d#e e e e fzee e e fzd$ed%eeezdee zdzdee zdzdej(fd&Zdej(d'ee e e fd(e dede de deej(fd)Zd-d*e d+e fd,ZxZS).AriaImageProcessoraG A vision processor for the Aria model that handles image preprocessing. Initialize the AriaImageProcessor. Args: image_mean (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Mean values for normalization. image_std (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Standard deviation values for normalization. max_image_size (`int`, *optional*, defaults to 980): Maximum image size. min_image_size (`int`, *optional*, defaults to 336): Minimum image size. split_resolutions (`list`, *optional*, defaults to a list of optimal,resolutions as tuples): The optimal resolutions for splitting the image. split_image (`bool`, *optional*, defaults to `False`): Whether to split the image. do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. resample (PILImageResampling, *optional*, defaults to `BICUBIC`): The resampling filter to use if resizing the image.  pixel_values pixel_mask num_cropsNiPFTgp? image_mean image_stdmax_image_sizemin_image_sizesplit_resolutions split_imagedo_convert_rgb do_rescalerescale_factor do_normalizeresamplec  &t|di| |gd}|gd}||_||_||_||_||_|!gd}|D cgc]} | ddz| ddzf}} ||_||_||_ | |_ | |_ | |_ ycc} w)N)?rr))rDr,)rDr)rD)rD)rD)rD)rDro)r,r)r,r)r,r,)r,rD)rrD)rr,)rrD)rr,)rrD)rrD)rrD)rorDrrDri) rjrkrrrrrrrrrrr)rlrrrrrrrrrrrrelrns rZrkzAriaImageProcessor.__init__s "6"  (J  'I,,$"&  $!y FW X"Q%#+r!us{!; X  X!2,$,(  !Ys Bptimagesreturn_tensors data_formatinput_data_formatc P||n |j}||n |j}||n |j}||n |j}||n |j}||n |j }||n |j }| | n |j} | | n |j} | | n |j} |dvr td|j|}t|}t|s tdt| ||| || |r|Dcgc] }t|}}|Dcgc] }t!|}}|r#t#|drt$j'd|t)|d}g}g}d}|D]}}|r"|j+||j,|| ||}n|g}|t/||kDr t/|}|D]2}t1|\}}|t3||z }||k\rt3t5||z||f}n|t3t5||z|f}t7||| || }||dz ||d z }}t9|d|fd|ff||}t;j<||ft> }d |d|dd|d f<|jA||r|jC|| | }| r;|jE||j|j||}| tG|| |n|}|jA|5tIt;jJ|d t;jJ|d |d| Scc}wcc}w)aI Process a list of images. Args: images (ImageInput or list of ImageInput): The input image or a list of images. image_mean (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Mean values for normalization. image_std (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Standard deviation values for normalization. max_image_size (`int`, *optional*, defaults to `self.max_image_size` (980)): Maximum image size. min_image_size (`int`, *optional*, defaults to `self.min_image_size` (336)): Minimum image size. split_image (`bool`, *optional*, defaults to `self.split_image` (False)): Whether to split the image. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb` (True)): Whether to convert the image to RGB. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize` (True)): Whether to normalize the image. resample (PILImageResampling, *optional*, defaults to `self.resample` (BICUBIC)): The resampling filter to use if resizing the image. return_tensors (`str` or `TensorType`, *optional*, defaults to "pt"): The type of tensor to return. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use same as the input image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use the inferred format of the input image. Returns: BatchFeature: A BatchFeature object containing: - 'pixel_values': Tensor of processed image pixel values. - 'pixel_mask': Boolean pixel mask. This mask is a 2D tensor of shape (max_image_size, max_image_size) where: - True (1) values indicate pixels that belong to the original resized image. - False (0) values indicate pixels that are part of the padding. The mask helps distinguish between actual image content and padded areas in subsequent processing steps. - 'num_crops': The maximum number of crops across all images. Nrrz(max_image_size must be either 490 or 980zSInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, or torch.Tensor)rrrrrrrzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.)rr)rrrrD)r@)imagescaler)axisrdata tensor_type)&rrrrrrrrrr ValueError fetch_imagesrrrrrrlogger warning_oncerget_image_patchesrlenrrrwrrnprGrappendrescale normalizerr stack)rlrrrrrrrrrrrrrrrr pixel_masksr crop_images crop_imagehwrnew_sizecrop_image_resizedpadding_bottom padding_rightcrop_image_paddedrs rZ preprocesszAriaImageProcessor.preprocesssR$.#9Zt !*!6IDNN +9+E4K^K^+9+E4K^K^%0%vay I    @ 7E"44**" 1&7 5  %g  C $4y$@ , )1 7 %j11&Q26 #CE NN C^TH .CE NN0STH%+% 1&7 &"1?!0Ln_ghi_jNj $'&(1m*<= 1&7 %! XX~~&FdS ;< =Xa[=-HQK-78"":.(, /~Yj)5)% (,)$5*; )7)%'244E{Tef.& ##$56c1 7@ 7B "A > hh{;& '   iA=s 9LL#rtarget_resolutionreturncHt|||\}}t|||f||}|S)aG Resizes an image to a target resolution while maintaining aspect ratio. Args: image (np.ndarray): The input image. target_resolution (tuple): The target resolution (height, width) of the image. resample (`PILImageResampling`): Resampling filter to use if resizing the image. input_data_format (`ChannelDimension` or `str`): The channel dimension format of the input image. Returns: np.ndarray: The resized and padded image. rr)r r)rlrrrr new_height new_width resized_images rZ_resize_for_patchingz'AriaImageProcessor._resize_for_patchings7&!6e=NPa b Iuz9&=duv r\original_resolutioncz|\}}|\}}t||z d\}}t||z d\} } | | | zf|||zffSNr,)divmod) rlr&roriginal_heightoriginal_width target_height target_widthpaste_xr_xpaste_yr_ys rZ_get_padding_sizez$AriaImageProcessor._get_padding_sizes]*='&7# |l^;Q? mo=qA 3''7S=)AAAr\cjt|||}|j||}|j||}|S)zU Pad an image to a target resolution while maintaining aspect ratio. )padding)r r2r)rlrrrnew_resolutionr4 padded_images rZ_pad_for_patchingz$AriaImageProcessor._pad_for_patchings? /u6GIZ[((9JKxxwx7 r\r4modeconstant_valuesc^t|tst|dk7rt||||||S| t |}t j dt jdt jdt jdi}tj|||||}|t|||}|S|}|S)a Pads the `image` with the specified `padding` and `mode`. Padding can be in the (`height`, `width`) dimension of in the (`num_patches`) dimension. In the second case an iterable if tuples is expected as input. Args: image (`np.ndarray`): The image to pad. padding (`int` or `tuple[int, int]` or `Iterable[tuple[int, int]]`): Padding to apply to the edges of the height, width axes. Can be one of three formats: - `((before_height, after_height), (before_width, after_width))` unique pad widths for each axis. - `((before, after),)` yields same before and after pad for height and width. - `(pad,)` or int is a shortcut for before = after = pad width for all axes. mode (`PaddingMode`): The padding mode to use. Can be one of: - `"constant"`: pads with a constant value. - `"reflect"`: pads with the reflection of the vector mirrored on the first and last values of the vector along each axis. - `"replicate"`: pads with the replication of the last value on the edge of the array along each axis. - `"symmetric"`: pads with the reflection of the vector mirrored along the edge of the array. constant_values (`float` or `Iterable[float]`, *optional*): The value to use for the padding if `mode` is `"constant"`. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use same as the input image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use the inferred format of the input image. Returns: `np.ndarray`: The padded image. rconstantreflectedge symmetric)r9r:) rrwr rrrCONSTANTREFLECT REPLICATE SYMMETRICrr)rlrr4r9r:rrpadding_mode_mappings rZrzAriaImageProcessor.pads` gs #s7|q'8ugt_kK\] ]  $ >u E   *     ! !6  ! !;   ug,@,FXghR]Ri '{PTPfPf(g% *a 1?.0PS`0`dr0r r\r)rprqrrrsmodel_input_namesrBICUBICrJrrwtuplerrkrFIRSTrstrr'rrndarrayr%r2r7rr@rrr rXrxrys@rZrrs>D*.(,!!:>#(&*&-$('9'A'A"!K$&"!;%"! "!  "!  c3h047 "!D["!t "!"!e "!Tk"!%"!N2604%)%)#'&*"&'+$(.226/?/E/E;?B T*--B DK'$.B 4;&- B d B d B D[B t B 4KB  B TkB %t+B j(4/B &,B !11D8B HZZ49Xh 4BUBuB ZZ 49 N^   "(003659;?@zz@uS#X&%S/)BB@ @ %0 @ ++d2 @!11D8@ @D0zz0U38_-0 0 % 0 & 0,0 bjj 0d#cr\rc0eZdZUdZeed<eed<eed<y)AriaImagesKwargsa split_image (`bool`, *optional*, defaults to `False`): Whether to split large images into multiple crops. When enabled, images exceeding the maximum size are divided into overlapping crops that are processed separately and then combined. This allows processing of very high-resolution images that exceed the model's input size limits. max_image_size (`int`, *optional*, defaults to `980`): Maximum image size (in pixels) for a single image crop. Images larger than this will be split into multiple crops when `split_image=True`, or resized if splitting is disabled. This parameter controls the maximum resolution of individual image patches processed by the model. min_image_size (`int`, *optional*): Minimum image size (in pixels) for a single image crop. Images smaller than this will be upscaled to meet the minimum requirement. If not specified, images are processed at their original size (subject to the maximum size constraint). rrrN)rprqrrrsr__annotations__rwrir\rZr`r`is r\r`F)totalcDeZdZUeed<ddddddej dZy)AriaProcessorKwargsrUF)r4return_mm_token_type_idsr)rr) text_kwargsrUrN)rprqrrr`rar'PYTORCH _defaultsrir\rZrdrd~s4##(- "  %,, Ir\rdc eZdZ d deezdedzdeeezefdzffd Ze dde e ze e ze e zde dzdeed efd Zdd Zed ZxZS) AriaProcessorN tokenizer chat_templatesize_conversionc*|ddd}|jDcic]\}}t||c}}|_|j|_|j|_||j |j |_t|!|||ycc}}w)zx size_conversion (`Dict`, *optional*): A dictionary indicating size conversions for images. Nrrr)rl) rrwrm image_tokenr} pad_token unk_tokenrjrk)rlimage_processorrkrlrmrrrns rZrkzAriaProcessor.__init__s  "$'c2O6E6K6K6MNdaA N$00'66  Y%8%8%@"+"5"5I  )=Q OsBtextrrrc |jtfd|jji|}t |t r|g}n.t |t st |dt s td||j|fi|d}|j|jjd}g}|jd|z}|D]P} | j|jj|jj|z} |j| Rni}|}|djd d} |djd d } |j|fi|dd di} |j!|| d g | rUt#j$| d} t#j&| d}d|| |j(k(<|j+| d<t-i| || S)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **pixel_mask** -- Pixel mask to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringsNrUr,rrfrreFr) modalities input_idsrDmm_token_type_idsr) _merge_kwargsrdrk init_kwargsrr]rJrKrrrmrrEpopreplaceror_check_special_mm_tokensrarray zeros_liker}tolistr )rlrsrr output_kwargs image_inputstokens_per_imageprompt_stringsrsamplerre text_inputs array_idsrxs rZ__call__zAriaProcessor.__call__s"+**  "&.."<"<  dC 6DD$' 47C0H_` `  /4//Y-:XYL#33L4M4M4S4STU4VW N$((58HHI .(B(BDNND^D^ajDjk%%f- . L!N&}599:JDQ#0#?#C#CD^`e#f $dnn^i}]7Sidhi  %%nkwi%X #[!9:I " k+.F G BC i4+>+>> ?/@/G/G/IK+ ,!@K!@ Lr r   4D[lm n, ,,! ss $%C Cc|jj}|jj}|Dcgc] }|dk7s | }}ttj ||zScc}w)Nr)rkrYrrrJrfromkeys)rltokenizer_input_namesimage_processor_input_namesnames rZrYzAriaProcessor.model_input_namesse $ @ @&*&:&:&L&L#9T&kW[_jWjt&k#&kDMM"7:U"UVWW'ls A#A#)NNNNr)rprqrrr/r]rrrwrkr)r&r%rJrr$rdr rrpropertyrYrxrys@rZrjrjs)-$(9= R!3&RTz R eck3./$6 R,%)4^++d9o=EV@WW4^T!4^,- 4^  4^4^l-4XXr\rjc(eZdZdZdeffd ZxZS)AriaSharedExpertsMLPa/ Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. rc`t|||j|jz|_yr)rjrkrcrfrs rZrkzAriaSharedExpertsMLP.__init__ s)  !'!9!9FV>VYZ>Z\b\r\rs)&*B*BFDVDVX^XnXnor\ctj||jjd\}}tj j |d}||fS)NrD)rrCr>rB)rFtopkrrer functionalsoftmax)rl router_logits top_logits top_indicesscoress rZroute_tokens_to_expertsz#AriaExperts.route_tokens_to_expertsEsH"'**]dkk>R>RXY"Z K&&zr&:F""r\cB|j|\}}|j}tj|j j tj |jjd|jjdz j |}|}|jd}tj|} |jd| |jjz} |j| |} tj| dd\} } tj j#| | z} |j%| |}tj&|j(d|jjz|j+df|j|j,}|j/d| ||jd|jj|j+d}||j1dzj3d}|S)NrrD)binsminrr>r,rBr?)rr@rFhistcflattentofloat32rrdviewargsort index_selectrerchunkrrsilurrGrErrA index_copy_rsum)rlrr top_k_index top_k_weightsoriginal_dtyperOindicesflatten_indicessorted_indicespermuted_tokens fc1_output projectiongate expert_outputunpermuted_tokensrRs rZrzAriaExperts.forwardJs%)%A%A-%P" ]$**!KK    ! $ $U]] 3,, ++a/  "^   !,,r*7'44Q$++J^J^8^_XXo/@A  ;;z1"= D]]'' 3d: ->? !KK   #dkk&:&: :Ms.p~p$p # u||r\rc\eZdZdeffd Zdej dej fdZxZS)AriaTextMoELayerrct|tj|j|j d|_t||_t||_ ||_ yNFr) rjrkrrrrdrouterrexpertsrshared_expertsrrs rZrkzAriaTextMoELayer.__init__ksO ii 2 2F4J4JQVW "6* 26: r\rrc |j}|jd|jd}|j|}|j ||j|}|j |j|}||zSNr>)rErrrrr)rlroriginal_shaperrshared_expert_outputs rZrzAriaTextMoELayer.forwardrs{&,,%**2}/A/A"/EF  M2  ]MBGGW #22=3E3En3UV333r\) rprqrrr^rkrFrrrxrys@rZrrjs*~4U\\4ell4r\rceZdZdZy)AriaTextAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN)rprqrrrsrir\rZrr{sGr\rc,eZdZdZdedeffd ZxZS)AriaTextDecoderLayerag Aria Text Decoder Layer. This class defines a single decoder layer in the language model, incorporating self-attention and Mixture of Experts (MoE) feed-forward network. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. layer_idx (`int`): Index of the layer. r layer_idxcFt|||t||_yr)rjrkrmlprlrrrns rZrkzAriaTextDecoderLayer.__init__s +#F+r\)rprqrrrsr^rwrkrxrys@rZrrs  ,~,#,,r\rceZdZUeed<dZdZddgZdZdZ dZ dZ dZ e edZej"fd ZxZS) AriaTextPreTrainedModelrmodel)rrsrrTpast_key_values)r attentionsct||t|tr7t j |j d|jjyy)Nr8)meanstd) rj _init_weightsrrinitnormal_rrr)rlmodulerns rZrz%AriaTextPreTrainedModel._init_weightss> f% f4 5 LLSdkk6S6S T 6r\)rprqrrr^rabase_model_prefixinput_modalities_no_split_modulessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_attention_backendrr_can_record_outputsrFno_gradrrxrys@rZrrsj (/1IJ&*#"3N"&-' U]]_UUr\rcReZdZUeed<dZdZdZejdZ y)AriaPreTrainedModelrrFTctj||t|tr6t j |j |jjyy)N)r) rrrrr trunc_normal_rrr)rlrs rZrz!AriaPreTrainedModel._init_weightss?%%dF3 fm ,   v||1N1N O -r\N) rprqrrr{rar_can_compile_fullgraphrrFrrrir\rZrrs6 ""&U]]_PPr\rc$eZdZdeffd ZxZS) AriaTextModelrc t||tjt |j Dcgc]}t ||c}|_d|_|jycc}w)NF) rjrkr ModuleListrKnum_hidden_layersrlayersgradient_checkpointing post_initrs rZrkzAriaTextModel.__init__s[  mmFKFLdLdFe f !&) 4 f  ',#  gsA.)rprqrrr^rkrxrys@rZrrs~r\rc@eZdZddiZdeffd ZefdZxZS)AriaTextForCausalLMlm_head.weightzmodel.embed_tokens.weightrct||t||_|j|_t j |j|jd|_|jyr) rjrkrr vocab_sizerrrlm_headrrs rZrkzAriaTextForCausalLM.__init__sU  "6*  ++yy!3!3V5F5FUS  r\c &t||fi|yr)rjr)rlrmrns rZrzAriaTextForCausalLM.forwards - -r\) rprqrr_tied_weights_keysr^rkr)rrxrys@rZrrs/*,GH~..r\rc eZdZy)AriaCausalLMOutputWithPastNrrir\rZr r rr\r c eZdZy)AriaModelOutputWithPastNrrir\rZr r rr\r ceZdZdeffd ZdZ ddejdejdzdede dzd e e d e e zf d Z dd ejdzdejdzdejdzd ej dzdejdzdedzdejdzde dzdejdzd e ed e ezfdZxZS) AriaModelrcDt||t||_yr)rjrkrmulti_modal_projectorrs rZrkzAriaModel.__init__s  %26%:"r\c|y|jd|jjj|jjj}|jd|jjj|jjj}|j ddkDj S)NrD) dimensionrstepr,)r>rBr)unfold vision_towerrrFrr)rlrpatches_subgrids rZ_create_patch_attention_maskz&AriaModel._create_patch_attention_masks  $++""))44""))44,  *00""))44""))441   ###1A5;;==r\Nrrroutput_hidden_statesrrc |j|}|j|f|ddd|}d}|&|jd} tj| }|j |} |j | ||_|S)NT)patch_attention_maskr return_dictrDr)rrrrF logical_notrr pooler_output) rlrrrrrr image_outputsimage_attn_maskflattened_maskselected_image_features rZget_image_featureszAriaModel.get_image_featuress $@@L)))  !5!%      +199!!(C!44)%%)[[%E%E 5 m  ,..}/C/C]EXEXYN!%!:!:~";" *889K^\M%$%% )%+')   '%777@G33d!//))2>2J   QU   r\)Nr>N) NNNNNNNNN)rprqrrr{rkrrF FloatTensorrwrr$r(r[rr# LongTensorrrrr rrxrys@rZrrsl;z;>&04$&,0 ''%%,"  #Tk  +,  + +8.215.2.204(,26!%26. ##d*. ''$.. $$t+ .  t+ . &&- . . ((4/. $;. ((4/. -..  ( (. r\rz Aria model for conditional generation tasks. This model combines a vision tower, a multi-modal projector, and a language model to perform tasks that involve both image and text inputs. ) custom_introceZdZddiZe ddej dej dzdedee de e zf d Z e e dd ejdzdej dzdejdzd ejdzd ejdzd edzdej dzdejdzdedzdeejzdejdzdee de ezfdZ dfd ZxZS)AriaForConditionalGenerationrz(model.language_model.embed_tokens.weightNrrrrrc B|jjd|||d|S)N)rrrri)rr#)rlrrrrs rZr#z/AriaForConditionalGeneration.get_image_featuresDs5-tzz,, %!!5    r\rwr$r%rr&labelsr'logits_to_keepr(c |jd|||||||| | d | } | d}t| tr t| dn| }|j |dd|ddf}d}|4|j d|||j jjd| }t||| j| j| jS)a{ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or `model.image_token_id` (where `model` is your instance of `AriaForConditionalGeneration`). Tokens with indices set to `model.image_token_id` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import httpx >>> from io import BytesIO >>> import torch >>> from PIL import Image >>> from io import BytesIO >>> from transformers import AutoProcessor, AutoModel >>> from transformers.image_utils import load_image >>> # Note that passing the image urls (instead of the actual pil images) to the processor is also possible >>> image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg") >>> image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg") >>> image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg") >>> processor = AutoProcessor.from_pretrained("Rhymes-AI/Aria") >>> model = AutoModel.from_pretrained("Rhymes-AI/Aria", dtype=torch.bfloat16, device_map="auto") >>> # Create inputs >>> messages = [ ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."}, ... {"type": "image"}, ... {"type": "text", "text": "What can we see in this image?"}, ... ] ... }, ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In which city is that bridge located?"}, ... ] ... } ... ] >>> prompts = [processor.apply_chat_template([message], add_generation_prompt=True) for message in messages] >>> images = [[image1, image2], [image3]] >>> inputs = processor(text=prompts, images=images, padding=True, return_tensors="pt").to(model.device) >>> # Generate >>> generated_ids = model.generate(**inputs, max_new_tokens=256) >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> print(generated_texts[0]) Assistant: There are buildings, trees, lights, and water visible in this image. >>> print(generated_texts[1]) Assistant: The bridge is in San Francisco. ```) rwrrr$r%rr&r'r(rN)logitsr9r)lossr<rrrri) rrrwslicer loss_functionrr`rr rrr)rlrwrrr$r%rr&r9r'r:r(rr2r slice_indicesr<r=s rZrz$AriaForConditionalGeneration.forwardSs\$**  %!)%+')      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD*#33!//))   r\c rt ||f|||||| d| } | s| jdds || d<|| d<| S)N)rr&r$r(r:is_first_iterationr'Trr)rjprepare_inputs_for_generationrT) rlrwrr&rrr$r(r:rBr model_inputsrns rZrCz:AriaForConditionalGeneration.prepare_inputs_for_generationsfw<   +')))1     VZZ T%B ,8L ()3L &r\r) NNNNNNNNNrN)NNNNNNNF)rprqrrrr)rFr3rwr$r(r[rr#r*r4rrrr rrCrxrys@rZr7r79s+,VW04$&  ''  %%,  "  +,  + +    .215.2.204(,26*.!%-.26j ##d*j ''$.j $$t+ j  t+ j &&- j j ((4/j   4'j $;j ell*j ((4/j +,j  + +j j ^   r\r7) r{r^rrjr7rrrrr)mcollections.abcrnumpyrrFrrr activationsr cache_utilsrconfiguration_utilsr image_processing_utilsr r r r image_transformsrrrrr image_utilsrrrrrrrrrrmodeling_flash_attention_utilsrmodeling_outputsrmodeling_utilsrprocessing_utilsr r!r"r#r$tokenization_pythonr%r&utilsr'r(r)r*r+autor-r.r/llama.configuration_llamar0llama.modeling_llamar1r2r3r4r5r6r7llava.modeling_llavar8r9r:r;&llava_next.image_processing_llava_nextr< get_loggerrpr r[r^r{rModulerrrrr`rdrjrrrrrrrrrrr r rr7__all__rir\rZr\sB% &! 3uuee   C:-ff?^^<<3 G   H %>aD[aDHI#!I#X l ryy244n>BII>B+D|5* *%  pXNpXpXf Z8 Z ) RYY) X)")))X4ryy4"HH,,,$UoUU. P. PJ.13C." !<  6 ] ] @`#@``F r\