qiBdZddlmZddlmZddlmZddlmZm Z m Z m Z ddl m Z mZddlmZmZerdd lZd ZGd d e d ZGdded ZddZdZdZdZeGdde ZdgZy )z Processor class for IDEFICS. )urlparse) BatchFeature) ImageInput)ProcessingKwargsProcessorMixin TextKwargsUnpack)PreTokenizedInput TextInput)auto_docstringis_torch_availableNc2eZdZUdZedzed<edzed<y)IdeficsTextKwargsaW add_eos_token (`bool`, *optional*, defaults to `False`): Whether to add an end-of-sequence token at the end of the text input. When enabled, an EOS token is appended to mark the end of the text sequence, which is useful for generation tasks. add_end_of_utterance_token (`bool`, *optional*): Whether to add an end-of-utterance token to mark the end of a user's message in conversational contexts. This token helps the model distinguish between different utterances in a multi-turn conversation and is particularly important for chat-based models. N add_eos_tokenadd_end_of_utterance_token)__name__ __module__ __qualname____doc__bool__annotations__`/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/idefics/processing_idefics.pyrr's$; $t +rrF)totalc.eZdZUeed<ddddddidZy) IdeficsProcessorKwargs text_kwargsFlongest)add_special_tokenspaddingrreturn_tensorspt)r common_kwargsN)rrrrr _defaultsrrrrr6s)""#( " +D1 Irrc|dk7r |dk(rd|||k\<|dk(r>|dk(}d||<tjjj||}d||ddf<S)Nr%r num_classes)torchnn functionalone_hot)incremental_maskr$r+ negatives attn_masks r$incremental_to_binary_attention_maskr3Csvb T !@B -< =$* &'#HH''//0@k/Z "# )Q, rc&|dk(r t||Sy)Nr%),image_attention_mask_for_packed_input_ids_pt) input_ids tokenizerr$s r)image_attention_mask_for_packed_input_idsr8Ts;IyQQrctj|d}tj|d}|jt}|j}t |j dD]K}d}d}t||D]4\} } | |k(r|dz }|||| <d}n|||| <|rd||| <| |k(s3d}6Mt |j dD]}d}d}t ||j ddz ddD]9} ||| } | |k(r|dz }|||| <d}n|||| <| |k(rd}|s2d||| <;||dk7} ||| xx|zcc<||| xxdzcc<||fS)Nr)) fill_valuerFT)r, full_likeconvert_tokens_to_ids IMAGE_TOKEN eos_token_idrangesize enumerate) r6r7image_attention_masknext_image_attention_maskimage_token_id eod_token_id batch_idxcountseen_eodidxtoken_idnon_negative_indicess rr5r5Ys ??9D % b I44[AN))L9>>!,- &y';< MC>) 7<$Y/4 7<$Y/479$Y/4<'  "9>>!,-I 9-221592rB ?C +C0H>) ))4S9 ? 9CrI!),-ABeKB!),-ABbHB)I, !: ::rcdd|vryt|}t|j|jgS)zChecks if the passed string contains a valid url and nothing else. e.g. if space is included it's immediately invalidated the url F)rallschemenetloc)stringresults ris_urlrTs0 f} f F  v}}- ..rc eZdZdfd Ze d deeezezeezeeezdee zeezee zeeezeee zde e de fdZ edZxZS) IdeficsProcessorc pt|||t|dr |jn|j t |_|j j|j j|j jf|_ d|jjjdgv|_ y)z image_size (int, *optional*, defaults to 224): The size of the image to be processed. add_end_of_utterance_token (bool, *optional*, defaults to None): Whether to add the end of utterance token to the text. image_tokenadditional_special_tokensN)super__init__hasattrrEr=r>image_processorimage_num_channels image_sizedefault_image_dimsr7special_tokens_mapget1tokenizer_was_trained_with_end_of_utterance_token)selfr^r7r`rkwargs __class__s rr\zIdeficsProcessor.__init__s )4y-0  $ $00=   3 3  + +  + +#  !DNN$E$E$I$IJegi$j j >rimagestextrfreturnc ()| | td||}n|t|ttfs|g}t|tr|g}t|ttfr"t |t |k7r tdt d|Ds tdt|dttfr"t||Dcgc] \}}|g| }}}ntt||}|jtfd|jji|}|djd d }|djd d} | |j} td Ds|g}d (d)d} ()fd} g} g} |D],}|jj}g}d }d }t!|D]\}}|dkDr t#| }t|trg|j%d}t'|r:|j(j+|}|| |z }|j-|d}}| r|r|| z }||z }d }|| |z }|j-|d}|r||jj.z }|j(|fi|d}| j-|| j-|/|djdd}|j| fi|d}|d}|d}t1d| D}t1d|}t3d| DdkD}g}g}g}t||| D]\}}} |}!|!j5|j6}"t9|"|}#| d|#}$t |$dkDrA|dk(rat;j<|g|$j?dd}%|$|%d|$j?dn%|dk(r t;j<|g|j@}%|j-%|dk(s|j-t;jB|!|j-t;jB| |dk(r?t;jD|}t;jD|}t;jD|}|r)tG||j|\}&}'tI|&||}&nF|dk(rAt;j<|jJd|jJddt:j"}&tM|||&dScc}}w)a Returns: a dict with entries: `input_ids`, `attention_mask`, `pixel_values`, `image_attention_mask` which can be directly passed to `model.generate` Detailed explanation: Each entry in `text` is either a text to be passed as is or an image that will be processed. An image can be either an image object (`PIL.Image`) or a url from which the image can be retrieved. When the processor encounters an image it'll inject `` entry into the prompt. Example: ```python checkpoint = "HuggingFaceM4/idefics-9b" processor = AutoProcessor.from_pretrained(checkpoint) url = "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg" img = processor.image_processor.fetch_images([url])[0] prompts = [ "User:", img, "Describe this image.\nAssistant: An image of two kittens in grass.\n", "User:", "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg", "Describe this image.\nAssistant:", ] inputs = processor(text=prompts, return_tensors="pt") generated_ids = model.generate(**inputs, max_length=100) generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ``` In this example the `prompts` will be converted into: ``` User:Describe this image. Assistant: An image of two kittens in grass. User:Describe this image. Assistant:' ``` and the two images will be massaged using [`IdeficsImageProcessor.__call__`] method and placed inside the `pixel_values` dict entry of the return value. This example also exemplifies that images can be passed as objects or as text urls. It can be seen that the first image is passed as object and the second one as a url. To do training do: ```python image_transform = transforms.Compose( [ transforms.RandomResizedCrop( (w, h), scale=(0.9, 1.0), interpolation=transforms.InterpolationMode.BICUBIC ), transforms.ToTensor(), transforms.Normalize(mean=self.image_mean, std=self.image_std), ] ) inputs = processor(text=prompts, transform=image_transform, return_tensors="pt") ``` In order to help debug prompt generation enable `debug=True` which will show you what's happening. Nz9You need to specify either `text` or `images` and `text`.aWhen providing both images and text arguments, the number of text prompts should be the same as the number of images.If you want to have several images per prompt, images should be nested as such: images=[[img1, img2], [img3, img4], ...] for text=[prompt1, prompt2, ...].c3<K|]}t|tywN) isinstancestr.0is r z,IdeficsProcessor.__call__..s8az!S)8szQWhen using the image-text-to-text behavior, the prompts should only contain text.rtokenizer_init_kwargsr rFrc3HK|]}t|ttfywrm)rnlisttuplerps rrsz,IdeficsProcessor.__call__..%sAA:a$/As "zrrYc"|rzSzzSrmr)last_was_image fake_tokenrXs r image_tokensz/IdeficsProcessor.__call__..image_tokens,s!"Z//!K/*<.bs8SV8r;c32K|]}t|ywrmrrs rrsz,IdeficsProcessor.__call__..es dlXY Y >G  ftUm4 $$v$u .3t9F 3K q 8488 !tuu&)dE]3KNvW[K\]5GZF0Z0]]s6401*** " "&.."<"<  &m488%P %2=%A%E%EFbdh%i" & -)-)_)_ &AAAiG0  !5 =   % -F>>334IM"N!M$V, *4q5$(^);$"Q&!T)*/++n*a~GZGZG\]^]_G`*a'DR'(@.*=*=a*@A!T)*/++n*_tG^G^*_'  !4 5% '' 5E(FG&--ell>.JK% L( T !${{+;< !KK 6M%*[[1G%H " &O $...' # !$H$n.$ %',{{$**1-/?/E/Ea/H!SXS]S]($-"8 -(<   y^sS3c~|jj}|jj}t||zdgzS)NrC)r7model_input_namesr^rv)retokenizer_input_namesimage_processor_input_namess rrz"IdeficsProcessor.model_input_namess? $ @ @&*&:&:&L&L#),GGKaJbbccr)NN)NN)rrrr\r rrvror r r rrrpropertyr __classcell__)rgs@rrVrVs 0UY +/k T*--3d3i?$tCy/Qk   y/  !" tI   t%& ' (k /0k  k k ZddrrV)r))r urllib.parserfeature_extraction_utilsr image_utilsrprocessing_utilsrrr r tokenization_utils_baser r utilsr rr,r>rrr3r8r5rTrV__all__rrrrs"4% D7  , % , -U  "R ,;^/Kd~KdKd\  r