qi,ddlZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z mZddlmZmZddlmZerd d lmZmZGd d e d ZGdded ZeedGdde ZdgZy)N) BatchFeature) ImageInput)MultiModalDataProcessingKwargsProcessorMixin TextKwargsUnpack)PreTokenizedInput TextInput)auto_docstringis_vision_available)requires)Emu3ImageProcessorKwargs smart_resizeceZdZUdZeed<y)Emu3TextKwargsa return_for_image_generation (`bool`, *optional*, defaults to `False`): Whether the processed text is intended for image generation tasks. When `True`, the processor prepares inputs for image generation by appending image start tokens and size information to the prompt, and images should not be provided. When `False`, the processor prepares inputs for text generation from images and text, requiring both inputs to be provided. return_for_image_generationN)__name__ __module__ __qualname____doc__bool__annotations__Z/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/emu3/processing_emu3.pyrrs"&%rrF)totalc8eZdZUeed<eed<dddddddZy ) Emu3ProcessorKwargs text_kwargs images_kwargsF)rreturn_mm_token_type_idsz1:1i)ratio image_area)r"r#N)rrrrrr _defaultsrrrr!r!+s/++,1(-   Irr!)vision)backendsc eZdZ d fd Ze d dedzdeezeezeezdzde e de fdZ d dZ d Zdefd Z dd ZxZS) Emu3ProcessorNc &|j|_|j|_|j|_|j|_|j |_|j|_|j|_ d|_ t|1|||y)N) chat_template) image_tokenimage_token_id boi_tokenimage_start_token eoi_tokenimage_end_tokenimage_wrapper_tokenfake_token_around_image eof_token bos_tokendownsample_ratiosuper__init__)selfimage_processor tokenizerr.kwargs __class__s rr;zEmu3Processor.__init__=s%00'66!*!4!4(22'0'D'D$",,",, ! )=Qrimagestextr?returnc |t|tr|g}n.t|tst|dts td|jt fd|j ji|}|djdd}|djdd }|djd d }|r | td |s| | td i}|j} |j|j} |s| |j|fi|d}t|j} g} |D]} |j | vrt#| }|\}}||j$z}||j$z}||d zz}| |d||j&d|z| }| j)|j |d } |j*| } |j | vr| j-| | D cgc]} | j)d|j  }} nj|rh|j/|||j$\}}| |d||j&}|D cgc]} |j*| |}} ||ggt1|z|d<|djdd }|djdd}|j |fi|ddd i}|j3||dg|rUt5j6|d}t5j8|d}d |||j:k(<|j=|d<t?i|||Scc} wcc} w)aA Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. rzAInvalid input text. Please provide a string, or a list of stringstokenizer_init_kwargsr"rFr#r%Nr&zGYou should not provide `images` when `return_for_image_generation=True`zOYou must provide either text or images when `return_for_image_generation=False`r*z image_sizesreturn_tensorsr$image) modalities input_idsmm_token_type_ids)data tensor_type) isinstancestrlist TypeError _merge_kwargsr!r> init_kwargspop ValueErrorr2r7r4r=iterrGr/nextr9r6replacer8appendcalculate_generate_sizelen_check_special_mm_tokensnparray zeros_liker0tolistr)r<rArBr? output_kwargsrr%r&image_featuresimage_start_tokensimage_end_tokensrGprompt_stringssample image_sizeheightwidthimage_seq_lengthimage_placeholder image_promptrHr$ text_inputs array_idsrLs r__call__zEmu3Processor.__call__Ns& dC 6DD$' 47C0H_` `***  "&.."<"<  '4M&B&F&FGdfk&l#o.227DA"?377 dK &6+=fg g*t|no o $ 6 67"nn-d.B.B-CD+v/A1T11&[M/vhawtOkOkNlm|@PnPmQRbQc)d%#^^D,<,<>OQRSF $/x8F&&&0%%f- .Ucc&FNN?D4D4DEcDc) 88 DLaLabMFE01&5'$B^B^A_`LLPQ&t~~&vh|n=QDQ.4e_,=D ,IN= )'}599:JDQ#0#?#C#CD^`e#f $dnnT_]=-I_Z^_  %%dKWI%N #[!9:I " k+.F G BC i4+>+>> ?/@/G/G/IK+ ,!BK!B>!BP^__+d Rs ##L4L9c i}|g}|D]\}}t|||jj|jj|jj\}}||j z}||j z}||dzz}|j |dgt|z}|j||dtdi|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. r)num_image_tokensnum_image_patchesr) rr=spatial_factor min_pixels max_pixelsr9rZr\updater) r<rGr? vision_datarrrirjrkrss r_get_num_multimodal_tokensz(Emu3Processor._get_num_multimodal_tokenss  "! !, :  ,((77((33((33 !  4#8#88!6!66#)UQY#7  ''(89 :"#c+&6 6    4D[lm n, ,,rctt|jd\}}||z}||z dz}tt||z|z }tt||z|z } || fS)N:g?)mapintsplitround) r<r%r&rtrjri current_area target_ratio token_height token_widths rr[z%Emu3Processor.calculate_generate_sizespCS!12 vv~ "\1c9 5,!6!GHI % 4~ EFG [((rc <|jj|fi|SN)r= postprocess)r<rAr?s rrzEmu3Processor.postprocesss /t##//A&AArc ||dk(r|j|fd|i|S|dk(r|j|d}|dSt|jjd|d) a  Post-process the output of a multimodal model to return the requested modality output. If the model cannot generated the requested modality, an error will be raised. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. generation_mode (`str`, *optional*): Generation mode indicated which modality to output and can be one of `["text", "image", "audio"]`. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `list[Union[str, PIL.Image.Image]]`: The decoded text or generated image. rBskip_special_tokensrIzPIL.Image.Image)rH pixel_valuesz# got an unexpected generation_mode=z.. Supported options are only `text` and `image)post_process_image_text_to_textrrVr@r)r<generated_outputsrgeneration_moder?rAs rpost_process_multimodal_outputz,Emu3Processor.post_process_multimodal_outputs*  "o&?7477!7JNT  '%%&7HY%ZF.) )>>**++NN_`NO rr)NN)TN)rrrr;r rr r rQr r!rrpryr[rr __classcell__)r@s@rr+r+:s R"%)aeQ`T!Q`++d9o=EV@WWZ^^Q`,- Q`  Q`Q`f -D)B*BLP!rr+)numpyr^image_processing_utilsr image_utilsrprocessing_utilsrrrr r tokenization_utils_baser r utilsr rutils.import_utilsrimage_processing_emu3rrrr!r+__all__rrrrs}"2%ddC8*M &Zu & *%  ;uNu up  r