qivf,ddlZddlmZddlmZmZmZddlZddl Z ddl m Z ddl m Z mZmZddlmZmZmZmZddlmZmZmZdd lmZdd lmZerdd lmZGd d edZdefdZ dZ!dZ"dZ#dZ$edeGddeZ%dgZ&y)N) accumulate) TYPE_CHECKINGOptionalUnion) BatchFeature) ImageInputis_valid_image load_image)MultiModalDataProcessingKwargsProcessorMixinUnpack) AddedToken BatchEncoding TextInput)auto_docstring)requires)PreTokenizedInputc(eZdZddiddddddidZy ) ColModernVBertProcessorKwargspaddinglongestTchannels_first)return_row_col_info data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaultsn/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/colmodernvbert/processing_colmodernvbert.pyrr(s/ y $(+" +D1 Ir(rF)totalreturncHt|txr|jdS)Nhttp) isinstancestr startswith)vals r)is_urlr26s c3  :CNN6$::r(c2t|xs t|SN)r2r )elems r)is_image_or_image_urlr6:s $< />$//r(c d}t|D]4}t|D]}||d|dzd|dzdz||zzz }!|dz }6|d||z||zz|zz }|S)zKPrompt with expanded image tokens for when the image is split into patches. )range) image_seq_len image_rows image_colsfake_token_around_image image_tokenglobal_img_tokentext_split_imagesn_hn_ws r)_prompt_split_imagerH>sZ "$ C *+sQwiuS1WIQ/OOU`TaerRrr    T! " $ %&   M] * +% % ' r(c&||z||zz|zS)z5Prompt with expanded image tokens for a single image.r')r?rBrCrDs r)_prompt_single_imagerJQs6 # #   M] * +% % 'r(cT|dk(r|dk(rt||||St||||||S)Nr)rBrCrD)rJrH)r@rAr?rBrCrDs r)get_image_prompt_stringrL[sHQ:?# $;#-   z:/F Ue r()torch)backendsc jeZdZdZ ddededzdedzffd ZdZe dde e e ze e e zd e e d e e e d fdedzd e ed ef d ZddZ dde dzd e ed efdZd e e e zd e ed efdZ dde de dfde de dfdededde defd df dZxZS)ColModernVBertProcessora! Constructs a ColModernVBert processor which wraps a ModernVBertProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score. [`ColModernVBertProcessor`] offers all the functionalities of [`ModernVBertProcessor`]. See the [`~ModernVBertProcessor.__call__`] for more information. Args: image_processor ([`Idefics3ImageProcessor`]): An instance of [`Idefics3ImageProcessor`]. The image processor is a required input. tokenizer (`PreTrainedTokenizerFast`, *optional*): An instance of [`PreTrainedTokenizerFast`]. This should correspond with the model's text model. The tokenizer is a required input. image_seq_len (`int`, *optional*, defaults to 64): The length of the image sequence i.e. the number of tokens per image in the input. visual_prompt_prefix (`Optional`, *optional*): A prefix to be prepended to visual prompts. query_prefix (`Optional`, *optional*): A prefix to be prepended to query prompts. Nr?visual_prompt_prefix query_prefixc d}tdddj|_tdddj|_tdddj|_d|_||_|j|j|_|j|j|_ |j|j |_ td D cgc]0}td D] } |jd |d zd | d zd "2c} }|_ tjd|_d|j|j|jgi} |j!| |j|j|_t# |H||fd|i||xsd|jd|_|xsd|_|j|_ycc} }w)a image_seq_len (`int`, *optional*, defaults to 64): The length of the image sequence i.e. the number of tokens per image in the input. visual_prompt_prefix (`str`, *optional*): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*): A prefix to be used for the query. NzFT) normalizedspecialzzz r9r:r;r<z*(\n?\n?|\n?)+additional_special_tokens chat_templatez<|begin_of_text|>User:z0Describe the image. Assistant:r8)rcontentfake_image_tokenrCend_of_utterance_tokenglobal_image_tagr?convert_tokens_to_idsimage_token_idfake_image_token_idglobal_image_token_idr> row_col_idsrecompile%_regex_to_remove_extra_special_tokensadd_special_tokenssuper__init__rQrRquery_augmentation_token) selfimage_processor tokenizerrXr?rQrRkwargsij tokens_to_add __class__s r)rgz ColModernVBertProcessor.__init__|s$ *+FSXbf g o o%iE4PXX&01ERWae&f&n&n# .*'==d>N>NO#,#B#B4CXCX#Y %.%D%DTEZEZ%["SXYZS[ NOejklem `aI + +eAE7%Awa,H I I  68ZZ@m5n2 (%%  ++*  $$]3'==d>N>NO )[=[TZ[$8% $T%5%5$66g h !).B(,(C(C%1 s5Gcg}|D]_}g}|D]E}t|r|j| t|s,|jt|G|j|a|Sr4)r appendr2r )riprompts prompt_imagespromptimagesr5s r)_extract_images_from_promptsz4ColModernVBertProcessor._extract_images_from_promptssn  )FF 4!$'MM$'D\MM*T"23  4   ( )r(rvtextrrlr+c  | | td|jtfd|jji|}||n |j }|dj dd}|dj dd}g}g} i} |jt|tr|g}n.t|tst|dts td |D cgc]} | j|j}} |Wt|r|gg}nt|ttfrt|dr|t|t|k7r>td |jd t|d |jd t|d dgtt!|z} t#t|D cgc]} || | | | dz}} nO|g}nKt|ttfs5t|dttfst|dds td|D cgc] } t| } } |D cgc](} | Dcgc]}t%|r t'|n|c}*}} }|j(|fi|d}| j+||| |k7rtd|d| d| j d|Dcgc]}dg|z c}}| j d|Dcgc]}dg|z c}}|j,}|j}|j.}g}g}t1|||D]\} }}g}g}t1||D]`\}}t3||||||}|j dz|zdz}|j5|j dz||zz|j5|b|j5|| j7|} t| dk(r td| d} t9|D]\} }| || | dzzz } |j5| |j|fi|d}!|j;||!dg| j+|!nZ|Xt=|r%tdt|d |jd|jd#d|i|d}!| j+|!|rt?j@| d }"t?jB|"}#t9D]o\} }$t?jD|"| |jFk(d}%d}&|$D];}'|&t|%k\rE|%|&}(|(|'z})d|#| |(|)f<t?jH|%|)}&=q|#jK| d!<tM| |"Scc} wcc} wcc} wcc}wcc}} wcc}wcc}w)$a  image_seq_len (`int`, *optional*): The length of the image sequence. If not provided, the default value of self.image_seq_len is used. image_seq_len should be equal to int(((image_size // patch_size) ** 2) / (scale_factor**2)) Nz+You must provide either `text` or `images`.tokenizer_init_kwargsr return_mm_token_type_idsFrrzAInvalid input text. Please provide a string, or a list of stringszThe total number of zP tokens in the prompts should be the same as the number of images passed. Found  z tokens and z images.r:zdInvalid input images. Please provide a single image or a list of images or a list of list of images.r!z!The number of images in the text z and images z should be the same.rowscols)rCrBrDrz.The image token should be present in the text.image) modalitieszFound z. tokens in the text but no images were passed.rx input_idsmm_token_type_ids)data tensor_typer')' ValueError _merge_kwargsrrk init_kwargsr?popr.r/listcountrCr6tuplesumlenrr>r2r rjupdaterZr\ziprLrrsplit enumerate_check_special_mm_tokensanynparray zeros_likewherer_ searchsortedtolistr)*rirvrxr?rl output_kwargsr{rn_images_in_textn_images_in_imagesinputssamplecumsum_images_in_textrmim image_inputsn_imagesr@rArZrCrDprompt_stringsbatch_image_seq_lengths sample_rows sample_colsimage_prompt_stringsimage_seq_lengthsn_rowsn_colsimage_prompt_string row_length split_sample text_inputs array_idsr seq_lengthsimage_start_positionsrnseq_lenstartends* r)__call__z ColModernVBertProcessor.__call__sO R R  $V,!(FT5M27LVTUY7W#+,F ;(243C3C2DE&&)*:&;%N>N=O|\_`f\g[hhpr ./C$zBR7S2T,T)"'s+;'22R7FSW!W$"3"3e";f"D!W !Wmm"'%m4$ "XsCquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec t|dk(r tdt|dk(r td|dj|djk7r td|dj|djk7r td||dj}g}t dt||D]%}g}t j jjj||||zdd} t dt||D]} t j jjj|| | |zdd} |jt jd| | jd djd |jt j|d j|j|(t j|d S) a[ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColQwen2, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeT) batch_first padding_valuez bnd,csd->bcnsr)dimrr:)rrdevicedtyper>rMnnutilsrnn pad_sequencerreinsummaxrcatto) rirrrrrscoresrm batch_scores batch_queriesrnbatch_passagess r)score_retrievalz'ColModernVBertProcessor.score_retrievals@  A %23 3 ! "a '34 4 A  % %);A)>)E)E ENO O A  $ $(:1(=(C(C CLM M  +A.44L%'q#./< ]A/1L!HHNN..;; Q^4$VW<M1c"45zB !&!3!3!@!@&q1z>:\]"A"##LL-PTTYZT[\]^bbghbi   MM%))La8;;LILL][ \ ]yyQ''r()NN@NN)NNNr4)Ncpu)r#r$r%__doc__intr/rgrwrr rrrrrrrrrrrrr __classcell__)rps@r)rPrPjs $+/#'3D  3D "Dj 3DDj3Dj JNbf$( NET*--T*5E0FFNEI2DOTJ]E^^_NETz NE 67 NE  NENE`#-N%)@T!@67@  @D7$y/)7677  7z0449 >(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(r(rP)'rb itertoolsrtypingrrrnumpyrrMfeature_extraction_utilsr image_utilsr r r processing_utilsr r rrtokenization_utils_baserrrrrutils.import_utilsrrrboolr2r6rHrJrLrP__all__r'r(r)rs* 11 4AAXXKK#*< $4E ;4;0&  :@(n@(@(F % %r(