qiAddlmZmZddlmZddlmZmZddlm Z m Z m Z m Z ddl mZmZddlmZmZerddlZGd d e d ZeGd de ZdgZy))OptionalUnion) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringis_torch_availableNc&eZdZddidddddidZy ) ColQwen2ProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsb/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/colqwen2/processing_colqwen2.pyrr"s, y ," +D1 Ir"rF)totalc beZdZ ddedzdedzffd Ze ddedzdeeze eze ezde e de fd Z dd Zed Zedefd Z ddedzde e de fd Zdee ezde e de fdZ ddede dfdede dfdededdedefddf dZxZS)ColQwen2ProcessorNvisual_prompt_prefix query_prefixc t||||t|dsdn |j|_t|dsdn |j|_|xsd|_|xsd|_y) ar visual_prompt_prefix (`str`, *optional*, defaults to `"<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>"`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Query: "`): A prefix to be used for the query. ) chat_template image_tokenz <|image_pad|> video_tokenz <|video_pad|>zf<|im_start|>user <|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>zQuery: N)super__init__hasattrr+r,r'r()selfimage_processor tokenizerr*r'r(kwargs __class__s r#r.zColQwen2Processor.__init__1so )=Q29)]2S?YbYnYn29)]2S?YbYnYn$8% u !)5Ir"imagestextr3returnc v|jtfd|jji|}|dj dd}|du}| | t d| | t d|7t |r|g}n^t|trt |drn?t|tr$t|dtrt |dds t d|jgt|z}|jdd |i|d }|d } | |jjd z} d} tt|D]} |j|| vrQ|| j|jd | | j!| zzd|| <| dz } |j|| vrQ|| jd |j|| <|j|fddi|d} t#i| |}|d dddf|d ddd fz}tt%j&|d|j)}t$j*j,j.j1|d|d<|r.|dj3|ddk(d}|j5d|i|S|t|t6r|g}n.t|trt|dt6s t d||j8dz}g}|D]%}|j:|z|z}|j=|'|j|fddi|d}|Sy)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNz&Either text or images must be providedz5Only one of text or images can be processed at a timerzAimages must be an image, list of images or list of list of imagesr5rimage_grid_thwz<|placeholder|>return_token_type_idsF)data pixel_valuesT) batch_first input_idstoken_type_idsilabelsz*Text must be a string or a list of strings r!) _merge_kwargsrr2 init_kwargspop ValueErrorr isinstancelistr'lenr1 merge_sizeranger+replaceprodrtorchsplittolistnnutilsrnn pad_sequence masked_fillupdatestrquery_augmentation_tokenr(append)r0r5r6r3 output_kwargsr:r> texts_doc image_inputsr; merge_lengthindexi text_inputs return_dataoffsetsr@rD texts_queryqueryaugmented_query batch_querys r#__call__zColQwen2Processor.__call__Is"+** # "&.."<"<  }-11(DA &d 2 >A s9~.]A**il:'0|';'; ,,.?>RWCXC]C]C_coCo.prs( !   **il: $-Q<#7#78I4K[K[#\IaL ])$..&+ .K ',K{,Kl,KLK""23AqD9KHX[_`>`bfg""Hf#56   $$v t,DGS1I !MNN~66;%'K 4"&"3"3e";f"D""?3 4)$..&+ .K  +r"c i}|tjjdi}|j||jddxs|jj }|Dcgc] }|jj g||"}}|Dcgc] }||dzz } }|j| |dtdi|Scc}wcc}w)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. NrrMr<)num_image_tokensnum_image_patchesr!)rr getrYr1rMget_number_of_image_patchesr ) r0 image_sizesr3 vision_datarrM image_sizerm num_patchesrls r#_get_num_multimodal_tokensz,ColQwen2Processor._get_num_multimodal_tokenss  "3==AA/SUVM   (&**<>a$BVBVBaBaJ#.!A$$@@\*\m\! !Sdd; A !=d d   4D[lm n, ,,! es $%B?Cc|jj}|jj}|Dcgc] }|dvs| }}||zScc}w)N)pixel_values_videosvideo_grid_thw)r2model_input_namesr1)r0tokenizer_input_namesimage_processor_input_namesnames r#rxz#ColQwen2Processor.model_input_namess\ $ @ @&*&:&:&L&L# 9' DHqt}}5F5f55r"c *|jdd|i|S)ai Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColQwen2Processor's [`ColQwen2Processor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). r6r!r)r0r6r3s r#process_queriesz!ColQwen2Processor.process_queriess<t}}1$1&11r"query_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec t|dk(r tdt|dk(r td|dj|djk7r td|dj|djk7r td||dj}g}t dt||D]%}g}t j jjj||||zdd} t dt||D]} t j jjj|| | |zdd} |jt jd| | jd djd |jt j|d j|j|(t j|d S) a[ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColQwen2, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeT)rA padding_valuez bnd,csd->bcnsr)dimr<r=)rLrIdevicedtyperNrQrTrUrVrWr\einsummaxsumcatto) r0rrrrrscoresrb batch_scores batch_queriesjbatch_passagess r#score_retrievalz!ColQwen2Processor.score_retrieval"s@  A %23 3 ! "a '34 4 A  % %);A)>)E)E ENO O A  $ $(:1(=(C(C CLM M  +A.44L%'q#./< ]A/1L!HHNN..;; Q^4$VW<M1c"45zB !&!3!3!@!@&q1z>:\]"A"##LL-PTTYZT[\]^bbghbi   MM%))La8;;LILL][ \ ]yyQ''r")NNNNN)NN)N)Ncpu)rrrrZr.rrrr rKr rrrjrtpropertyrxr[rrrintrr __classcell__)r4s@r#r&r&/s+/#' 6 "Dj 6 Dj 60%)Z^fT!f++d9o=EV@WWf01 f  ffP-4 C C(#((%)6T!6016  6B2$y/)2012  2H0449 >(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(r"r&)typingrrfeature_extraction_utilsr image_utilsrrprocessing_utilsr r r r tokenization_utils_baser rrUrrrQrr&__all__r!r"r#rs_*#45XXC7 .e p(p(p(f  r"