qi6 (dZddlZddlmZddlmZmZddlm Z m Z m Z ddl m Z mZddlmZGd d e d Zd eededeeefdZdeeeedeeedededej*f dZdedededefdZeGdde ZdgZy)zProcessor class for Mllama.N) BatchFeature) ImageInputmake_nested_list_of_images)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringceZdZdddiiZy)MllamaProcessorKwargs image_kwargsmax_image_tilesN)__name__ __module__ __qualname__ _defaults^/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/mllama/processing_mllama.pyrrs q IrrF)total input_idsimage_token_idreturnct|Dcgc] \}}||k(s |}}}t|dk(rgSt|dk(r|ddggSt|dd|ddDcgc] \}}||g }}}|j|dt|g|dd}|dddD]} | d| ddz k(r|| d<| d}|Scc}}wcc}}w)a Generate a cross-attention token mask for image tokens in the input sequence. This function identifies the positions of image tokens in the input sequence and creates a mask that defines which subsequent tokens each image token should attend to. Args: input_ids (list[int]): A list of token ids representing the input sequence. image_token_id (int): The id of the token used to represent images in the sequence. Returns: list[list[int]]: A list of [start, end] pairs, where each pair represents the range of tokens an image token should attend to. Notes: - If no image tokens are present, an empty list is returned. - For a single image token, it attends to all subsequent tokens until the end of the sequence. - For multiple image tokens, each attends to tokens up to the next image token or the end of the sequence. - Consecutive image tokens are treated as a group and attend to all subsequent tokens together. rN) enumeratelenzipappend) rritokenimage_token_locationsloc1loc2 vision_masks last_mask_end vision_masks rget_cross_attention_token_maskr,"s,09/C_81euP^G^Q__ !Q&  !Q&&q)2.//367LSb7QShijikSl3mnZT4T4LnLn.r2C NCD !$Q'M#DbD)' q>[^a/ /*KN#A ' /`os B>B>Ccross_attention_token_mask num_tiles max_num_tileslengthc zt|}td|D}tj||||ftj}t t ||D]\\}\}} t t || D]<\} \} } t| dk(s| \} }t||}|dk(r|}d||| || d| f<>^|S)a Convert the cross attention mask indices to a cross attention mask 4D array. This function takes a sparse representation of cross attention masks and converts it to a dense 4D numpy array. The sparse representation is a nested list structure that defines attention ranges for each image in each batch item. Args: cross_attention_token_mask (list[list[list[int]]]): A nested list structure where: - The outer list represents the batch dimension. - The middle list represents different images within each batch item. - The inner list contains pairs of integers [start, end] representing token ranges for each image. num_tiles (list[list[int]]): A nested list structure specifying the number of tiles for each image in each batch item. max_num_tiles (int): The maximum possible number of tiles. length (int): The total sequence length of the input. Returns: np.ndarray: A 4D numpy array of shape (batch_size, length, max_num_images, max_num_tiles) The array contains `1` where attention is allowed and `0` where it is not. Note: - Special handling is done for cases where the end token is -1, which is interpreted as attending to the end of the sequence. c32K|]}t|ywNr!).0maskss r z?convert_sparse_cross_attention_mask_to_dense..psLUL)shapedtyperrN)r!maxnpzerosint64r r"min)r-r.r/r0 batch_sizemax_num_imagescross_attention_mask sample_idx sample_maskssample_num_tilesmask_idx locationsmask_num_tilesstartends r,convert_sparse_cross_attention_mask_to_denserLRs:/0JL1KLLN886>=Ahh 9B#F`bkBl8m[4 4\#35>s>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' >>> build_string_from_input("<|image|>Hello world", "", "<|image|>") '<|image|>Hello world' >>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' rNr) startswithr!)rMrNrOnum_image_tokens_on_starts rbuild_string_from_inputrSsn4F !   K (K(*+!Q&!   K (55 6yk& JJrc eZdZd fd Ze d dedzdeezeezeezdzde e de fdZ d dZ ed ZxZS) MllamaProcessorNcFt|ds(d|_|j|j|_n"|j|_|j|_d|_|j|j|_|j |_t|!|||y)NrOz <|image|>z<|python_tag|>) chat_template) hasattrrOconvert_tokens_to_idsr python_tokenpython_token_idrNsuper__init__)selfimage_processor tokenizerrW __class__s rr]zMllamaProcessor.__init__sy-0*D "+"A"A$BRBR"SD (44D "+":":D ,(>>t?P?PQ",, )=Qrimagestextkwargsrc | | td|jtfd|jji|}|dj dd}i}|t |tr|g}n3t |ttfrtd|Ds td|Dcgc]}|j|j}}|D cgc]#} t| |j|j%}} |j|fi|d} |j|| dg | d D cgc]} | j|j } } |j#| d g} |>|j$j'|}t)|}|Dcgc] }t+|} }|t-d Drtd |Ds tdt/|d kDrS| |k7s | k7rI| tdd}t/| t/|k(r| |k7rd}n | k7rd}td|d| d||8|j$|fi|d}|j d}|j#||c|a d D cgc]} t1| |j }} t3||j$j4t7d| d D}||d<t9||Scc}wcc} wcc} wcc}wcc} w)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. TODO: add aspect_ratio_ids and aspect_ratio_mask and cross_attention_mask Nz'You must specify either text or images.tokenizer_init_kwargs text_kwargsreturn_tensorsc3<K|]}t|tywr3) isinstancestr)r5ts rr7z+MllamaProcessor.__call__..s=_UVjC>P=_szAInvalid input text. Please provide a string, or a list of stringsimage) modalitiesrrc3&K|] }|dk( ywrNrr5 batch_imgs rr7z+MllamaProcessor.__call__..sDi9>Dc3&K|] }|dk( ywrprrqs rr7z+MllamaProcessor.__call__..sQ#, QQrszaIf a batch of text is provided, there should be either no images or at least one image per samplez@No image were provided, but there are image tokens in the promptzZMake sure to pass your images as a nested list, where each sub-list holds images per batchzhIf you activated truncation with `max_length`, increase the `max_length` so image tokens aren't cropped.z)The number of image tokens in each text (zA) should be the same as the number of provided images per batch (z). images_kwargsr.c32K|]}t|ywr3r4)r5rs rr7z+MllamaProcessor.__call__.. sQi3y>Qr8)r.r/r0rC)data tensor_type) ValueError _merge_kwargsrr` init_kwargspoprjrklisttupleallcountrOrSrN_check_special_mm_tokensrupdater_ fetch_imagesrr!anysumr,rLrr<r)r^rbrcrd output_kwargsrhrxrln_images_in_text text_itemencoding token_idsn_images_in_idsn_images_in_imagessample add_messageimage_featuresr.r-rCs r__call__zMllamaProcessor.__call__sd$ F/7F$%ghh"$K-.#6F2GGL^brLr'C (,>>'Q $CDTCUV@@R?SSVWbVce  1T11&[M/BBo Iol"DB*s""K (K!&"K& K+5K0c B|jj|f||d|S)a Post-process the output of the model to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `list[str]`: The decoded text. )skip_special_tokensclean_up_tokenization_spaces)r` batch_decode)r^generated_outputsrrrds rpost_process_image_text_to_textz/MllamaProcessor.post_process_image_text_to_texts5(+t~~**  3)E    rc|jj}|jj}|Dcgc] }|dk7s | }}t||zdgzScc}w)Nr.rC)r`model_input_namesr_r~)r^tokenizer_input_namesimage_processor_input_namesnames rrz!MllamaProcessor.model_input_names,se $ @ @&*&:&:&L&L#9T&kW[_jWjt&k#&k),GGKaJbbcc'ls AAr3)NN)TF)rrrr]r rr r r~r rrrrpropertyr __classcell__)ras@rrUrUs R%)aeXCT!XC++d9o=EV@WWZ^^XC./ XC  XCXCvY^ 6ddrrU)__doc__numpyr=feature_extraction_utilsr image_utilsrrprocessing_utilsrrr tokenization_utils_baser r utilsr rr~intr,ndarrayrLrkrSrU__all__rrrrs"4AHHC#,E-d3i--QUVZ[^V_Q`-`- $T$s)_ 5- DI- -   - ZZ - `"KC"KC"Kc"Kc"KJLdnLdLd^  r