qil@NddlmZmZddlmZddlmZmZddlm Z m Z m Z m Z ddl mZmZmZddlmZmZerddlZGd d e d Zd ZedDcgc] }d|dd c}edDcgc] }d|dd c}zZdZeGdde ZdgZycc}wcc}w))OptionalUnion) BatchFeature) ImageInputmake_flat_list_of_images)MultiModalDataProcessingKwargsProcessorMixinUnpack) AddedTokenPreTokenizedInput TextInput)auto_docstringis_torch_availableNc&eZdZddidddddidZy ) ColPaliProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaults`/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/colpali/processing_colpali.pyrr#s, y ," +D1 Ir#rF)totalziz4>z3c ||z|z||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  r"prompt bos_token image_seq_len image_token num_imagess r$build_string_from_inputr04s$&M)J6 7 {6(" MMr#c VeZdZ ddedeffd Ze ddedzdeeze eze ezde e de fd Z dd Zed Zedefd Z ddedzde e de fd Zdee ezde e de fdZ ddede dfdede dfdededdedefddf dZxZS)ColPaliProcessorNvisual_prompt_prefix query_prefixc||_||_t|ds td|j|_t|dsNt t dd}d|gi}|j||jt |_ t |_ n"|j|_ |j|_ |jtd|_ d|_t|A|||y ) a! visual_prompt_prefix (`str`, *optional*, defaults to `"Describe the image."`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Question: "`): A prefix to be used for the query. image_seq_lengthz;Image processor is missing an `image_seq_length` attribute.r.FT) normalizedspecialadditional_special_tokens) chat_templateN)r3r4hasattr ValueErrorr6r IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idr. add_tokens EXTRA_TOKENS add_bos_token add_eos_tokensuper__init__) selfimage_processor tokenizerr:r3r4r. tokens_to_add __class__s r$rFzColPaliProcessor.__init__Ls%9!((:;Z[ [ / @ @y-0$[UDQK8;-HM  ( ( 7"+"A"A+"ND *D "+":":D (44D \*"' "'  )=Qr#imagestextkwargsreturnc X|jtfd|jji|}|dj dd}d}| | t d| | t d|j|j j|}t|}|jgt|z}|Dcgc]}|jd}}t||D cgc]R\} } t| |jj|jt t#| t$r t| nd T} } } |j |fi|d d } |dj'd d|dd xx|jz cc<|j| fd|i|d} i| d | i}|r.| dj)| ddk(d}|j+d|it-|S|t#|t.r|g}n.t#|t$rt#|dt.s t d||j0dz}g}|D]?}|jj|j2z|z|zdz}|j5|A|dj'd d|dd <|j|fd|i|d}|Sycc}wcc} } w)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNTz&Either text or images must be providedz5Only one of text or images can be processed at a timeRGBr*r pixel_values max_lengthreturn_token_type_ids input_idstoken_type_idsrilabels)dataz*Text must be a string or a list of strings r)2) _merge_kwargsrrI init_kwargspopr<rH fetch_imagesrr3lenconvertzipr0r,r6r= isinstancelistget masked_fillupdaterstrquery_augmentation_tokenr4append)rGrLrMrN output_kwargsrRrW texts_docimager+ image_list input_stringsrUinputs return_datarZ texts_queryquery batch_querys r$__call__zColPaliProcessor.__call__qs"+** " "&.."<"<  }-11(DA $ F-f5F223c&kAI8>?uemmE*?F?+.i*@ 'FJ(!"nn66"&"7"7 +2S>SS:#T^^&; .F CVB^\BK$ ,88@P9QUV9VX\]""Hf#56[1 1  $$v t,DGS1I !MNN~66;%'K *0043D3DDuLvUX\\""5) *:G}9U9Y9YZfhj9kM- ( 6($..&; .K  -A@ s )J!AJ&c i}|<|jgt|z}dgt|z}|j||dtdi|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. rT)num_image_tokensnum_image_patchesr")r6rbrir )rG image_sizesrN vision_dataryrzs r$_get_num_multimodal_tokensz+ColPaliProcessor._get_num_multimodal_tokenss]  " $ 5 56[9II !"c+&6 6    4D[lm n, ,,r#c|jjddgz}|jj}t||zS)NrYrZ)rImodel_input_namesrHrf)rGtokenizer_input_namesimage_processor_input_namess r$rz"ColPaliProcessor.model_input_namessB $ @ @DTV^C_ _&*&:&:&L&L#),GGHHr#c.|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )rI pad_token)rGs r$rkz)ColPaliProcessor.query_augmentation_tokens~~'''r#c *|jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. rLr"rw)rGrLrNs r$process_imageszColPaliProcessor.process_imagess>t}}5F5f55r#c *|jdd|i|S)ag Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). rMr"r)rGrMrNs r$process_queriesz ColPaliProcessor.process_queriess<t}}1$1&11r#query_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec t|dk(r tdt|dk(r td|dj|djk7r td|dj|djk7r td||dj}g}t dt||D]%}g}t j jjj||||zdd} t dt||D]} t j jjj|| | |zdd} |jt jd| | jd djd |jt j|d j|j|(t j|d S) aZ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColPali, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeT) batch_first padding_valuez bnd,csd->bcnsr)dimrT)rbr<devicedtyperangetorchnnutilsrnn pad_sequencerleinsummaxsumcatto) rGrrrrrscoresi batch_scores batch_queriesjbatch_passagess r$score_retrievalz ColPaliProcessor.score_retrieval.s@  A %23 3 ! "a '34 4 A  % %);A)>)E)E ENO O A  $ $(:1(=(C(C CLM M  +A.44L%'q#./< ]A/1L!HHNN..;; Q^4$VW<M1c"45zB !&!3!3!@!@&q1z>:\]"A"##LL-PTTYZT[\]^bbghbi   MM%))La8;;LILL][ \ ]yyQ''r#)NNNzDescribe the image.z Question: )NN)N)r'Ncpu)rrr rjrFrrrrrfr rrrwr}propertyrrkrrrintrr __classcell__)rKs@r$r2r2Js$9( #R " #R  #RJ%)Z^XT!X++d9o=EV@WWX/0 X  XXt-$II (#((%)6T!6/06  6B2$y/)2/02  2H0449 >(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(r#r2)typingrrfeature_extraction_utilsr image_utilsrrprocessing_utilsr r r r tokenization_utils_baser rrrrrrrr=rrBr0r2__all__)rs0r$rs,#4?XXOO7 -U  ).t5A$qgQ5RWX[R\8]Q4#wa8]] N,a(~a(a(H  } 68]s B3B"