qi-dZddlZddlmZddlmZmZddlm Z m Z m Z m Z m Z ddlmZmZmZddlmZmZej*eZd Zed Dcgc] }d |d d  c}edDcgc] }d|dd  c}zZGdde ZGdde dZdefdZdZdZ dZ!eGdde Z"dgZ#ycc}wcc}w)z Processor class for PaliGemma. N) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixin TextKwargsUnpack) AddedTokenPreTokenizedInput TextInput)auto_docstringloggingziz4>z3c@eZdZUdZeezeezeezdzed<y)PaliGemmaTextKwargsa suffix (`str`, `list[str]`, `list[list[str]]`): The suffixes or batch of suffixes to be encoded. Only necessary for finetuning. See https://github.com/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/README.md for more information. If your prompt is " What is on the image", the suffix corresponds to the expected prediction "a cow sitting on a bench". Nsuffix)__name__ __module__ __qualname____doc__rr list__annotations__d/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/paligemma/processing_paligemma.pyrr's/ ) )DO ;dCT>U UX\ \\rrc,eZdZUeed<dddddidZy)PaliGemmaProcessorKwargs text_kwargsF)paddingreturn_mm_token_type_ids data_formatchannels_first)r! images_kwargsN)rrrrr _defaultsrrrr r 1s)$$(- + Irr F)totalreturncHt|txr|jdS)Nhttp) isinstancestr startswith)vals ris_urlr0?s c3  :CNN6$::rc2t|xs t|SN)r0relems ris_image_or_image_urlr5Ds $< />$//rc<t|txs t|Sr2)r,r-r5r3s r_is_str_or_imager7Hs dS " A&;D&AArc ||z|z||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_token image_seq_len image_token num_imagess rbuild_string_from_inputr@Ls$&M)J6 7 {6(" MMrc eZdZ d fd Ze d dedzdeezeezeezde e de fdZ d dZ ed ZxZS) PaliGemmaProcessorNc t|ds td|j|_t|dsNttdd}d|gi}|j ||j t|_t|_n"|j|_|j|_|jtd|_ d|_ t|9|||y) Nimage_seq_lengthz;Image processor is missing an `image_seq_length` attribute.r>FT) normalizedspecialadditional_special_tokens) chat_template)hasattr ValueErrorrDr IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idr> add_tokens EXTRA_TOKENS add_bos_token add_eos_tokensuper__init__)selfimage_processor tokenizerrHkwargsr> tokens_to_add __class__s rrTzPaliGemmaProcessor.__init__ds(:;Z[ [ / @ @y-0$[UDQK8;-HM  ( ( 7"+"A"A+"ND *D "+":":D (44D \*"' "'  )=QrimagestextrXr)c h|jtfd|jji|}|dj dd}d}| t d|t jdd}t|r|g}nt|trt|d r |#| td |Dsjt jd t|trKt|tr;t|t|k7r$t d t|d t|dt|r|gg}nt|ttfrt|d r|Dcgc]}|g}}nKt|ttfr*t|d ttfrt|d d s t dt!||D cgc]R\}} t#||jj$|j&t(t| tr t| ndT} }} ng} |D]} | j+t(t(|j&z} | j-t(}|dk7r|tt(znd }| d||jj$z| |dz} | j/| | D cgc]} | d } } |t|r|g}|&|Dcgc]}||jj0z}}|j2|fi|dd}|dj dd}|dj dd}|j f||d|d}|j5| |dgi|d|i}|rIt7j8|d}d|t7j8|dd k(<|j;d|i|rUt7j8|d}t7j<|d}d|||j>k(<|jA|d<tC|| Scc}wcc} }wcc} wcc}w)!a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. If `suffix` is provided, the `input_ids` will also contain the suffix input ids. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **labels** -- Labels compatible with training if `suffix` is not None tokenizer_init_kwargsr!rNTzF`images` are expected as arguments to a `PaliGemmaProcessor` instance.z]You are using PaliGemma without a text prefix. It will perform as a picture-captioning model.rc3,K|] }t|vywr2)rK).0samples r z.PaliGemmaProcessor.__call__..s@{f,@saLYou are passing both `text` and `images` to `PaliGemmaProcessor`. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add `` tokens in the very beginning of your text. For this call, we will infer how many images each text has and add special tokens.z Received z images for zK prompts. Each prompt should be associated with an image or list of images.zAimages must be an image, list of images or list of list of imagesr:r9r& pixel_valuesreturn_tensorsr#) text_pairreturn_token_type_idsimage) modalities input_idsitoken_type_idslabelsmm_token_type_ids)data tensor_type)" _merge_kwargsr rW init_kwargspoprJlogger warning_oncer7r,ranywarninglenrtuplezipr@r<rDrKreplacerfindappend eos_tokenrV_check_special_mm_tokensnparrayupdate zeros_likerNtolistr)rUr[r\rX output_kwargsrrirjr; image_list input_stringsexpanded_samplesrbexpanded_samplebos_rfind_index bos_indexsfxrfrgr#inputs return_datarn array_idsros r__call__zPaliGemmaProcessor.__call__se(+** $ "&.."<"<  }-11(DA $ >ef f <   o D D !6D d #(8a(A    2@4@@<dD)j.F6{c$i/('F }LT LWX "&)%hZFu 6>&QR);T39:%ug:F:ve}5"6!9tUm<&vay|4$%hii/2$.? !+ ,%"&..":":&*&;&;$/6@T6R3z?XY  ! !$& "=F&,nn[+PTPePeBe&fO&5&;&;K&HOFUY[F[#k2B BabI' 3dnn6N6NNQ`ajakQll$%++O<=>N N6F82 N N  "26":XF  @FGcDNN444GFG+t++FUmO6TUVde &}599:JDQ#0#?#C#CD^`d#e   "7 M*   %%mV %R>>>  !XXf[12F>BF288F#345: ;   &1 2 #[!9:I " k+.F G BC i4+>+>> ?/@/G/G/IK+ ,.IIy; !(!O Hs PAP$1 P* P/c i}|<|jgt|z}dgt|z}|j||dtdi|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. rd)num_image_tokensnum_image_patchesr)rDryrr)rU image_sizesrX vision_datarrs r_get_num_multimodal_tokensz-PaliGemmaProcessor._get_num_multimodal_tokenss]  " $ 5 56[9II !"c+&6 6    4D[lm n, ,,rc|jjddgz}|jj}t||zS)Nrmrn)rWmodel_input_namesrVr)rUtokenizer_input_namesimage_processor_input_namess rrz$PaliGemmaProcessor.model_input_names sB $ @ @DTV^C_ _&*&:&:&L&L#),GGHHr)NNN)NNr2)rrrrTrrrr rr r rrrpropertyr __classcell__)rZs@rrBrBbs R8%)Z^xJT!xJ++d9o=EV@WWxJ12 xJ  xJxJt-$IIrrB)$rnumpyrfeature_extraction_utilsr image_utilsrrprocessing_utilsrrr r r tokenization_utils_baser r rutilsrr get_loggerrrurKrangerPrr boolr0r5r7r@rB__all__)is0rrs45PO,   H % ).t5A$qgQ5RWX[R\8]Q4#wa8]] ]*] /u ;4; 0BN,nInInIb a68]s B;-C