qitOddlmZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZmZmZmZdd lmZdd lmZddlmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(ejRe*Z+Gdde%Z,Gdde(dZ-edeGdde'Z.eGdde#Z/eed !Gd"d#eZ0ed$!Gd%d&e"Z1gd'Z2y)()deepcopy) dataclass)AnyOptionalUnionN)PreTrainedConfig) BatchFeature) ImageInputis_valid_image)Unpack) TextInput) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuple)requires)CONFIG_MAPPING) AutoModel)ColPaliForRetrievalColPaliPreTrainedModel)ColQwen2Config)Idefics3ProcessorIdefics3ProcessorKwargscJeZdZUdZdZdeiZeee fe d< d de de fdZ y) ColModernVBertConfiga Configuration class to store the configuration of a [`ColModernVBertForRetrieval`]. It is used to instantiate an instance of `ColModernVBertForRetrieval` according to the specified arguments, defining the model architecture following the methodology from the "ColPali: Efficient Document Retrieval with Vision Language Models" paper. Instantiating a configuration with the defaults will yield a similar configuration to the vision encoder used by the pre-trained ColModernVBert model, e.g. [ModernVBERT/colmodernvbert-merged](https://huggingface.co/ModernVBERT/colmodernvbert-merged). Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vlm_config (`PreTrainedConfig`, *optional*): Configuration of the VLM backbone model. embedding_dim (`int`, *optional*, defaults to 128): Dimension of the multi-vector embeddings produced by the model. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. Example: ```python from transformers import ColModernVBertConfig, ColModernVBertForRetrieval config = ColModernVBertConfig() model = ColModernVBertForRetrieval(config) ``` colmodernvbert vlm_config sub_configsN embedding_diminitializer_rangec |#td}tjdndt|tr,t |}d|vr t dt|ddi|}n(t|tstdt|dt|ds|jj|_ ||_ ||_||_tj di|y) N modernvbertzc`vlm_config` is `None`. Initializing `vlm_config` with the `ModernVBertConfig` with default values. model_typez^The `model_type` key is missing in the `vlm_config` dictionary. Please provide the model type.zWInvalid type for `vlm_config`. Expected `PreTrainedConfig`, `dict`, or `None`, but got . vocab_size)rloggerinfo isinstancedictrKeyErrorr TypeErrortypehasattrget_text_configr(r r"r#__init__)selfr r"r#kwargss k/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/colmodernvbert/modular_colmodernvbert.pyr3zColModernVBertConfig.__init__Gs  ' 68J KKu  D )!*-J:-t( <(@AOJOJJ(89ijnoyjzi{{|} z<0$.$>$>$@$K$KJ !$*!2!!+F+)Ng{Gz?)__name__ __module__ __qualname____doc__r&r r!r-strr__annotations__intfloatr3r)r7r6rr'sJ8"J#/1A"BKc3hB #' ,,! ,r7rc(eZdZddiddddddidZy ) ColModernVBertProcessorKwargspaddinglongestTchannels_first)return_row_col_info data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)r9r:r; _defaultsr)r7r6rBrBhs/ y $(+" +D1 Ir7rBF)total)torch)backendsc eZdZdZ ddededzdedzffd Z ddedzdee d e fd Z d e e e zdee d e fd Z dd ede dfdede dfdededdedefd df dZxZS)ColModernVBertProcessora! Constructs a ColModernVBert processor which wraps a ModernVBertProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score. [`ColModernVBertProcessor`] offers all the functionalities of [`ModernVBertProcessor`]. See the [`~ModernVBertProcessor.__call__`] for more information. Args: image_processor ([`Idefics3ImageProcessor`]): An instance of [`Idefics3ImageProcessor`]. The image processor is a required input. tokenizer (`PreTrainedTokenizerFast`, *optional*): An instance of [`PreTrainedTokenizerFast`]. This should correspond with the model's text model. The tokenizer is a required input. image_seq_len (`int`, *optional*, defaults to 64): The length of the image sequence i.e. the number of tokens per image in the input. visual_prompt_prefix (`Optional`, *optional*): A prefix to be prepended to visual prompts. query_prefix (`Optional`, *optional*): A prefix to be prepended to query prompts. N image_seq_lenvisual_prompt_prefix query_prefixc d}t|||f||d||xsd|jd|_|xsd|_|j |_y)a image_seq_len (`int`, *optional*, defaults to 64): The length of the image sequence i.e. the number of tokens per image in the input. visual_prompt_prefix (`str`, *optional*): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*): A prefix to be used for the query. N) chat_templaterTz<|begin_of_text|>User:z0Describe the image. Assistant:)superr3 image_tokenrUrVend_of_utterance_tokenquery_augmentation_token) r4image_processor tokenizerrXrTrUrVr5 __class__s r6r3z ColModernVBertProcessor.__init__sw$     ('    %9% $T%5%5$66g h !).B(,(C(C%r7imagesr5returnc v|jtfd|jji|}|dj dd}|du}t |r|g}n^t |trt |drn?t |tr$t |dtrt |dds td|Dcgc]}|jd}}|j|jgt|z||d|d }|r.|d j|d dk(d }|jd |i|Scc}w)a Prepare for the model one or several image(s). Handles input validation, RGB conversion, and prepends the `visual_prompt_prefix` to each image. Optionally computes labels from `token_type_ids` when a `suffix` is provided in `text_kwargs`. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrKsuffixNrzAimages must be an image, list of images or list of list of imagesRGBrL)textrarLrK input_idstoken_type_idsilabels) _merge_kwargsrBr_ init_kwargspopr r,list ValueErrorconvert__call__rUlen masked_fillupdate) r4rar5 output_kwargsrereturn_token_type_idsimage batch_docrjs r6process_imagesz&ColModernVBertProcessor.process_imagess[<+** ) "&.."<"<  }-11(DA &d 2 & !XF  %.*C VT*z&)T/J~^def^ghi^jOk`a a5;;5%--&;;MM++,s6{:'8%m4 "  !{+77 BR8SWX8XZ^_F   h/ 022R7FSW!W$"3"3e";f"D!W !Wmm"'%m4$ "XsCquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec t|dk(r tdt|dk(r td|dj|djk7r td|dj|djk7r td||dj}g}t dt||D]%}g}t j jjj||||zdd} t dt||D]} t j jjj|| | |zdd} |jt jd| | jd djd |jt j|d j|j|(t j|d S) a[ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColQwen2, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeT) batch_first padding_valuez bnd,csd->bcnsr)dimr)rrrodevicedtyperangerPnnutilsrnn pad_sequenceappendeinsummaxsumcatto) r4rrrrrscoresi batch_scores batch_queriesjbatch_passagess r6score_retrievalz'ColModernVBertProcessor.score_retrieval%s@  A %23 3 ! "a '34 4 A  % %);A)>)E)E ENO O A  $ $(:1(=(C(C CLM M  +A.44L%'q#./< ]A/1L!HHNN..;; Q^4$VW<M1c"45zB !&!3!3!@!@&q1z>:\]"A"##LL-PTTYZT[\]^bbghbi   MM%))La8;;LILL][ \ ]yyQ''r7)NN@NNN)r8Ncpu)r9r:r;r<r?r=r3r r rBr ryrrnrrrr __classcell__r`s@r6rSrSvs' $+/#' D  D "Dj DDj DH%)@T!@67@  @D7$y/)7677  7z0449 >(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(r7rSceZdZUeed<y)ColModernVBertPreTrainedModelconfigN)r9r:r;rr>r)r7r6rrfs  r7rz: Base class for ColModernVBert embeddings output. ) custom_introceZdZUdZdZej dzed<dZejdzed<dZ e ej dzed<dZ e ej dzed<dZ e ej dzed<y) ColModernVBertForRetrievalOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): The embeddings of the model. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True` and `pixel_values` are provided): Tuple of `torch.FloatTensor` (one for the output of the image modality projection + one for the output of each layer) of shape `(batch_size, num_channels, image_size, image_size)`. Hidden-states of the image encoder at the output of each layer plus the initial modality projection outputs. Nloss embeddings hidden_statesimage_hidden_states attentions)r9r:r;r<rrP FloatTensorr>rTensorrtuplerrr)r7r6rrks &*D%  d ")&*J t#*59M5**+d29;?u001D8?26Je''(4/6r7ru Following the ColPali approach, ColModernVBert leverages VLMs to construct efficient multi-vector embeddings directly from document images (“screenshots”) for document retrieval. The model is trained to maximize the similarity between these document embeddings and the corresponding query embeddings, using the late interaction method introduced in ColBERT. Using ColModernVBert removes the need for potentially complex and brittle layout recognition and OCR pipelines with a single model that can take into account both the textual and visual content (layout, charts, ...) of a document. ColModernVBert is trained on top of ModernVBert, and was introduced in the following paper: [*ModernVBERT: Towards Smaller Visual Document Retrievers*](https://arxiv.org/abs/2510.01149). ColModernVBert is part of the ColVision model family, which was introduced with ColPali in the following paper: [*ColPali: Efficient Document Retrieval with Vision Language Models*](https://huggingface.co/papers/2407.01449). c eZdZiZdeffd Zee d dejdzdejdzdejdzde e def d ZxZS) ColModernVBertForRetrievalrct||tj|j|_|j yr)rZr3r from_configr vlm post_init)r4rr`s r6r3z#ColModernVBertForRetrieval.__init__s2  (():):; r7Nrh pixel_valuesattention_maskr5rbc |jd|||d|}|d}|jjj}|j|j |}||j ddz }|;|j |j|j }||jdz}t||j|j|jS) N)rhrrrT)rkeepdim)rr)rrrrr)) rembedding_proj_layerweightrrnormr unsqueezerrrr) r4rhrrr5 vlm_outputlast_hidden_states proj_dtypers r6forwardz"ColModernVBertForRetrieval.forwardsTXX )%   (]..55;; ../A/D/DZ/PQ  *//b$/"GG  %+..Z5E5EjN_N_.`N#n&>&>r&BBJ/!$22!,, * > >   r7)NNN)r9r:r;_checkpoint_conversion_mappingrr3rrrP LongTensorrrr rrrrrs@r6rrs$&("3 .215.2  ##d* ''$.  t+  +,  *   r7r)rrrrS)3copyr dataclassesrtypingrrrrPconfiguration_utilsr feature_extraction_utilsr image_utilsr r processing_utilsr tokenization_utils_baserrrrrr utils.genericrutils.import_utilsrautorauto.modeling_autorcolpali.modeling_colpalirrcolqwen2.configuration_colqwen2ridefics3.processing_idefics3rr get_loggerr9r*rrBrSrrr__all__r)r7r6rs!'' 345&0MM-*!*R<U   H %>,>>,B $;5  :k(/k(k(\!$:!!  7{7  7&"( !4( #"( V r7