qi:(dZddlZddlmZddlmZmZddlZddl m Z ddl m Z m Z mZddlmZmZmZmZdd lmZmZmZdd lmZmZerdd lmZej8eZd efd Z dZ!dZ"dZ#dZ$GddedZ%eGddeZ&dgZ'y)z Processor class for Idefics3. N) accumulate) TYPE_CHECKINGUnion) BatchFeature) ImageInputis_valid_image load_image)MultiModalDataProcessingKwargsProcessorMixinUnpack) AddedToken BatchEncoding TextInput)auto_docstringlogging)PreTokenizedInputreturncHt|txr|jdS)Nhttp) isinstancestr startswith)vals b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/idefics3/processing_idefics3.pyis_urlr%s c3  :CNN6$::c2t|xs t|SN)rr )elems ris_image_or_image_urlr")s $< />$//rc d}t|D]4}t|D]}||d|dzd|dzdz||zzz }!|dz }6|d||z||zz|zz }|S)zKPrompt with expanded image tokens for when the image is split into patches. )range) image_seq_len image_rows image_colsfake_token_around_image image_tokenglobal_img_tokentext_split_imagesn_hn_ws r_prompt_split_imager4-sZ "$ C *+sQwiuS1WIQ/OOU`TaerRrr    T! " $ %&   M] * +% % ' rc&||z||zz|zS)z5Prompt with expanded image tokens for a single image.)r+r.r/r0s r_prompt_single_imager7@s6 # #   M] * +% % 'rcT|dk(r|dk(rt||||St||||||S)Nr)r.r/r0)r7r4)r,r-r+r.r/r0s rget_image_prompt_stringr9JsHQ:?# $;#-   z:/F Ue rc$eZdZdddddddidZy)Idefics3ProcessorKwargsTF)add_special_tokenspaddingis_split_into_wordsreturn_mm_token_type_idsreturn_row_col_info) text_kwargs images_kwargsN)__name__ __module__ __qualname__ _defaultsr6rrr;r;Ys(#'#((-   "4  Irr;F)totalc eZdZ d dededzffd ZdZe ddee eze e ezde e de e e dfdedzd e e d ef d Zdd ZxZS)Idefics3ProcessorNr+ chat_templatec :tdddj|_tdddj|_tdddj|_d|_||_|j|j|_|j|j|_ |j|j |_ tdDcgc]0}tdD] }|jd |d zd |d zd "2c}}|_ tjd |_d|j|j|jgi}|j!||j|j|_t# |H||fd|i|ycc}}w)a image_seq_len (`int`, *optional*, defaults to 169): The length of the image sequence i.e. the number of tokens per image in the input. This parameter is used to build the string from the input prompt and image tokens and should match the value the model used. It is computed as: image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2)) zFT) normalizedspecialzzz r%r&r'r(z*(\n?\n?|\n?)+additional_special_tokensrJN)rcontentfake_image_tokenr/end_of_utterance_tokenglobal_image_tagr+convert_tokens_to_idsimage_token_idfake_image_token_idglobal_image_token_idr* row_col_idsrecompile%_regex_to_remove_extra_special_tokensr<super__init__) selfimage_processor tokenizerr+rJkwargsij tokens_to_add __class__s rr]zIdefics3Processor.__init__is!++FSXbf g o o%iE4PXX&01ERWae&f&n&n# .*'==d>N>NO#,#B#B4CXCX#Y %.%D%DTEZEZ%["SXYZS[ NOejklem `aI + +eAE7%Awa,H I I  68ZZ@m5n2 (%%  ++*  $$]3'==d>N>NO )[=[TZ[% s5Fcg}|D]_}g}|D]E}t|r|j| t|s,|jt|G|j|a|Sr )r appendrr )r^prompts prompt_imagespromptimagesr!s r_extract_images_from_promptsz.Idefics3Processor._extract_images_from_promptssn  )FF 4!$'MM$'D\MM*T"23  4   ( )rrktextrrarc  | | td|jtfd|jji|}||n |j }|dj dd}|dj dd}g}g} i} |jt|tr|g}n.t|tst|dts td |D cgc]} | j|j}} |Wt|r|gg}nt|ttfrt|dr|t|t|k7r>td |jd t|d |jd t|d dgtt!|z} t#t|D cgc]} || | | | dz}} nO|g}nKt|ttfs5t|dttfst|dds td|D cgc] } t| } } |D cgc](} | Dcgc]}t%|r t'|n|c}*}} }|j(|fi|d}| j+||| |k7rtd|d| d| j d|Dcgc]}dg|z c}}| j d|Dcgc]}dg|z c}}|j,}|j}|j.}g}g}t1|||D]\} }}g}g}t1||D]`\}}t3||||||}|j dz|zdz}|j5|j dz||zz|j5|b|j5|| j7|} t| dk(r td| d} t9|D]\} }| || | dzzz } |j5| |j|fi|d}!|j;||!dg| j+|!nZ|Xt=|r%tdt|d |jd|jd#d|i|d}!| j+|!|rt?j@| d }"t?jB|"}#t9D]o\} }$t?jD|"| |jFk(d}%d}&|$D];}'|&t|%k\rE|%|&}(|(|'z})d|#| |(|)f<t?jH|%|)}&=q|#jK| d!<tM| |"Scc} wcc} wcc} wcc}wcc}} wcc}wcc}w)$a  image_seq_len (`int`, *optional*): The length of the image sequence. If not provided, the default value of self.image_seq_len is used. image_seq_len should be equal to int(((image_size // patch_size) ** 2) / (scale_factor**2)) Nz+You must provide either `text` or `images`.tokenizer_init_kwargsrAr?Freturn_tensorsrzAInvalid input text. Please provide a string, or a list of stringszThe total number of zP tokens in the prompts should be the same as the number of images passed. Found  z tokens and z images.r&zdInvalid input images. Please provide a single image or a list of images or a list of list of images.rBz!The number of images in the text z and images z should be the same.rowscols)r/r.r0rz.The image token should be present in the text.image) modalitieszFound z. tokens in the text but no images were passed.rm input_idsmm_token_type_ids)data tensor_typer6)' ValueError _merge_kwargsr;r` init_kwargsr+poprrlistcountr/r"tuplesumlenrr*rr r_updaterQrSzipr9rgsplit enumerate_check_special_mm_tokensanynparray zeros_likewhererV searchsortedtolistr)*r^rkrmr+ra output_kwargsr?rpn_images_in_textn_images_in_imagesinputssamplecumsum_images_in_textrbim image_inputsn_imagesr,r-rQr/r0prompt_stringsbatch_image_seq_lengths sample_rows sample_colsimage_prompt_stringsimage_seq_lengthsn_rowsn_colsimage_prompt_string row_length split_sample text_inputs array_idsrx seq_lengthsimage_start_positionsrcseq_lenstartends* r__call__zIdefics3Processor.__call__sO R R  $V,!(FT5M27LVTUY7W#+,F ;(243C3C2DE&&)*:&;%N>N=O|\_`f\g[hhpr ./C$zBR7S2T,T)"'s+;'A#\X[^bXb#\J JNbf$( NET*--T*5E0FFNEI2DOTJ]E^^_NETz NE 01 NE  NENE`#-rrI)(__doc__rY itertoolsrtypingrrnumpyrfeature_extraction_utilsr image_utilsrr r processing_utilsr r r rtokenization_utils_baserrrutilsrrr get_loggerrCloggerboolrr"r4r7r9r;rI__all__r6rrrs '4AAXXKK,<   H %;4;0&  .e f-f-f-R  r