qil`xddlZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZddlmZmZmZmZmZdd lmZd d lmZmZerddlZdd lmZd d lmZerddl m!Z!ejDe#Z$GddejJZ&GddZ'eedGddeZ(y)N)AnyUnionoverload) AudioInput)GenerationConfig) ImageInput)ProcessingKwargsUnpack)add_end_docstringsis_torch_availableis_vision_availableloggingrequires_backends) VideoInput)Pipelinebuild_pipeline_init_args)%MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES) KeyDataset)ImageceZdZdZdZdZy) ReturnTyperrrN)__name__ __module__ __qualname__TENSORSNEW_TEXT FULL_TEXTS/opt/pipecat/venv/lib/python3.12/site-packages/transformers/pipelines/any_to_any.pyrr/sGHIr!rc"eZdZdZdeefdZy)Chata This class is intended to just be used internally in this pipeline and not exposed to users. We convert chats to this format because the rest of the pipeline code tends to assume that lists of messages are actually a batch of samples rather than messages in the same conversation.messagescF|D]}d|vrd|vr td||_y)NrolecontentzQWhen passing chat dicts as input, each dict must have a 'role' and 'content' key.) ValueErrorr%)selfr%messages r"__init__z Chat.__init__:s7 vGg%)w*> !tuu v! r!N)rrr__doc__listdictr,r r!r"r$r$5sR!d!r!r$T) has_processorc|eZdZdZdZdZdZdZdZe dZ fdZ dde e fd Ze dd edzd eed fdzd eeddfdzdeedfdzdedeeeeff dZe dd eedzd eeed zdzd eeedzedzdzdeeedzdzdedeeeeeff dZ dd eeezeezd eeezeeezezdzd eeezezdzdeeezezdzdeeeefeeeeefzf fd ZddZddZ ddZxZS)AnyToAnyPipelinea Multimodal Generation pipeline using an `AutoModelForMultimodalLM`. This pipeline generates text given any combination of multimodal data and text.When the underlying model is a conversational model, it can also accept one or more chats, in which case the pipeline will operate in chat mode and will continue the chat(s) by adding its response(s). Each chat takes the form of a list of dicts, where each dict contains "role" and "content" keys. Unless the model you're using explicitly sets these generation parameters in its configuration files (`generation_config.json`), the following default values will be used: - max_new_tokens: 256 Example: ```python >>> from transformers import pipeline >>> pipe = pipeline(task="any-to-any", model="google/gemma-3n-E4B-it") >>> pipe("https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", text="A photo of") [{'generated_text': 'a photo of two birds'}] ``` ```python >>> from transformers import pipeline >>> pipe = pipeline("any-to-any", model="google/gemma-3n-E4B-it") >>> messages = [ >>> { >>> "role": "user", >>> "content": [ >>> { >>> "type": "image", >>> "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg", >>> }, >>> {"type": "text", "text": "Describe this image."}, >>> ], >>> }, >>> { >>> "role": "assistant", >>> "content": [ >>> {"type": "text", "text": "There is a dog and"}, >>> ], >>> }, >>> ] >>> pipe(text=messages, max_new_tokens=20, return_full_text=False) [{'input_text': [{'role': 'user', 'content': [{'type': 'image', 'url': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'}, {'type': 'text', 'text': 'Describe this image.'}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': 'There is a dog and'}]}], 'generated_text': ' a person in the image. The dog is sitting on the sand, and the person is sitting on'}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This multimodal pipeline can currently be loaded from pipeline() using the following task identifier: "any-to-any". See the list of available models on [huggingface.co/models](https://huggingface.co/models?pipeline_tag=any-to-any). TF)max_new_tokensc&t||i|d|jjvsd|jjvrt |dt |dd|jjvr t |d|j t y)Nimagevideovision torchvisionaudiolibrosa)superr,modelinput_modalitiesrcheck_model_typer)r*argskwargs __class__s r"r,zAnyToAnyPipeline.__init__sw $)&) djj11 1W @[@[5[ dH - dM 2 djj11 1 dI . CDr!NrAc  i} i}i}|j| |||d<| | |d<|xsi| d<| | dk7r| | dd<| jdrd| dd<|7t|tr|g}|| dd <|jj | dd <||d |vr t d || dd <|2|0| t d |rtjntj}n7||tj}n"| | dvrt|drtj}| dvr|t d|d| d| dvrt d| d| ^| |jjvrFt d| d|jjjd|jj|||d<| | |d<|||d<| | |d<| |d<|| |fS)Ntimeoutcontinue_final_messagegenerate_kwargstextgeneration_modeload_audio_from_videoTuse_audio_in_video stop_strings tokenizerr4zp'max_new_tokens' is defined twice, once in 'generate_kwargs' and once as a direct argument. Please use only one.z>`return_full_text` is mutually exclusive with `return_tensors`)NrG_postprocess_paramsz`return_type` cannot be set to z when generation_mode=z2. Set `return_type=None` or generation_mode='text')NrGr6r:z[`generation_mode` can be only one of the `text`, `audio`, `image` but got generation_mode[=]z`generation_mode=z` is not supported for z6. The model can only output the following modalities: return_typeclean_up_tokenization_spacesskip_special_tokens)updateget isinstancestr processorrLr)rrrrhasattrr=output_modalitiesrBr)r*r4rFrDreturn_full_textreturn_tensorsrOrP stop_sequencerErQrHrAforward_kwargspreprocess_paramspostprocess_paramss r"_sanitize_parametersz%AnyToAnyPipeline._sanitize_parameterss   (  +2 i ( ! -:P 6 7-<,Ar()  &?f+DCRN, -.? @ ::- .FJN, -.B C  $--!. @MN, -n ==A^^=U=UN, -k :  %*/??/R FCQN, -.> ?  'K,?) !abb2B*.. H[H[K  'K,?$,,K _%F7SWYnKo$..K . 0[5L1+>TUdTefCC  "B Bmn}m~~A  (_DJJD`D`-`#O#44KDJJL`L`LiLiKjkGGKzzGcGcFdf   "0; } - ! -;Q 7 8 ' 3A] = >  *8K 4 50?,- .2DDDr!rGimagesz Image.Imagevideosz np.ndarrayz torch.Tensorr:returnc yNr r*rGr`rar:rAs r"__call__zAnyToAnyPipeline.__call__s #r!c yrdr res r"rfzAnyToAnyPipeline.__call__s&)r!c | | tdt|tttfrt|dttt frt|dt r d|dvrt |t|fi|St|dttfrHt|ddt r2d|ddvr(|Dcgc] }t|}}t ||fi|S|Dt|ts4t|trt|dts t ||fi|St|jddtjdt |||||dfi|Scc}w)a Generate a text given text and optionally multimodal data passed as inputs. Args: text (`str`, `list[str]`, `list[dict]`): The text to be used for generation. If a list of strings is passed, the length of the list should be the same as the number of images. Text can also follow the chat format: a list of dictionaries where each dictionary represents a message in a conversation. Each dictionary should have two keys: 'role' and 'content'. 'role' should be one of 'user', 'system' or 'assistant'. 'content' should be a list of dictionary containing the text of the message and the type of the message. images (`str`, `list[str]`, `ImageInput`): The pipeline handles three types of images: - A string containing a HTTP(s) link pointing to an image - A string containing a local path to an image - An image loaded in PIL directly The pipeline accepts either a single image or a batch of images. Finally, this pipeline also supports the chat format (see `text`) containing images and text in this argument. videos (`str`, `list[str]`, `VideoInput`): The pipeline handles three types of videos: - A string containing a HTTP(s) link pointing to a video - A string containing a local path to a video - A video loaded and decoded to array format The pipeline accepts either a single video or a batch of videos. Finally, this pipeline also supports the chat format (see `text`) containing videos and text in this argument. audio (`str`, `list[str]`, `AudioInput`): The pipeline handles three types of audios: - A string containing a HTTP(s) link pointing to an audio - A string containing a local path to an audio - An audio loaded in PIL directly The pipeline accepts either a single audios or a batch of audios. Finally, this pipeline also supports the chat format (see `text`) containing audios and text in this argument. return_tensors (`bool`, *optional*, defaults to `False`): Returns the tensors of predictions (as token indices) in the outputs. If set to `True`, the decoded text is not returned. return_text (`bool`, *optional*): Returns the decoded texts in the outputs. return_full_text (`bool`, *optional*, defaults to `True`): If set to `False` only added text is returned, otherwise the full text is returned. Cannot be specified at the same time as `return_text`. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not to clean up the potential extra spaces in the text output. continue_final_message( `bool`, *optional*): This indicates that you want the model to continue the last message in the input chat rather than starting a new one, allowing you to "prefill" its response. By default this is `True` when the final message in the input chat has the `assistant` role and `False` otherwise, but you can manually override that behaviour by setting this flag. Return: A list or a list of list of `dict`: Each result comes as a dictionary with the following key (cannot return a combination of both `generated_text` and `generated_token_ids`): - **generated_text** (`str`, present when `return_text=True` and `generation_mode="text"`) -- The generated text. - **generated_audio** (`np.ndarray`, present when `generation_mode="audio"`) -- The generated audio. - **generated_image** (`PIL.Image.Image`, present when `generation_mode="image"`) -- The generated image. - **generated_token_ids** (`torch.Tensor`, present when `return_tensors=True` and `generation_mode="text"`) -- The token ids of the generated text. - **input_text** (`str`) -- The input text. Nz0You must at least provide either text or images.rr' chat_templatea The input data was not formatted as a chat with dicts containing 'role' and 'content' keys, even though this model supports chat. Consider using the chat format for better results. For more information, see https://huggingface.co/docs/transformers/en/chat_templating)rGr`r7r:)r)rTr.tuplerr/r<rfr$rUgetattrrVlogger warning_once) r*rGr`rar:rAchatchatsrBs r"rfzAnyToAnyPipeline.__call__spN >dlOP P dT5*5 6:d1gPTV[]aOb;c$q'4(VtAw->w'T =f==DGdE]3 471:t8TY_cghicjklcmYm045d55w'888  Zc%:z$PT?UZdeijkelnqZr 7#D3F3 3 4>>?D 9 E   _  w&[` alekll+6s:E%c t|tr||jdddk(}d| i|}|jjj j dk(r(|jDcic] \}}|dvs ||}}}|jj|jf|dddd |j|j }||d <|St|tttfr|} i}nt|j}|jd } |j!d dAt#|jd r+|jj$j'|d |d <t| ttfr t)| dkDr|j+dd|jdd | i|ddi|j|j }| |d <|Scc}}w)Nr' assistantadd_generation_promptMistralCommonBackend)padding truncation max_lengthptT)rErZtokenize return_dict)dtyperGr:feature_extractorrrurZr )rTr$r%rVrLrBritemsapply_chat_templatetor{r.rjrUcopypoprSrWr| fetch_audiolen setdefault) r*inputsrDrEprocessing_kwargschat_template_kwargskv model_inputsrGs r" preprocesszAnyToAnyPipeline.preprocesscs fd #&-)/)TT%9%?%?%A(!QQJqEqAqD($(>4>>=='=#  ' btzzb" $*L   ftUC0 1DF[[]F::f%Dzz'4(4Qd9e"&.."B"B"N"NvV]"_w dT5M *s4y1}  ( (D 9&t~~d4d6d$dRcdgg**h  $ VI(s / G<Gc|in|}|jd}|jd|jd}d|vr|j|d<|jjdi||}|||dS)NrG input_idsdecoder_input_idsgeneration_config)generated_sequence prompt_textrr )rrSrr=generate)r*rrFrrrs r"_forwardzAnyToAnyPipeline._forwards / 7"_"&&v.  $$[,2B2BCV2WX  o 5373I3IO/ 00TZZ00S<S?S&8clmmr!c |d}t|ttfr|gn|}|d}|d}|tjk(r+t t |D cgc] } || || dc} S||nd}|dxsd} | dk(rTt|jd r>|jj|j|jj}|jj|fd |i|} |tjtjhvrg} d|d<|jj|fd |i|} t!| | D]V\}}|j#|}d |cxkrd kr$nn!| j%||t |zdF| j%|X| } |tjk(rg}t!|| D]\}}t|tr||z}nt|tr||j&d ddk(}|rt)|j&d dd j+}|dxx|z cc<t-|j&dd |j&d d|j&d ddd |gzdgz}nt-|j&d|dgz}|j%||} t!|| Dcgc]*\}}dt|tr |j&n|d| |i,}}}|Scc} wcc}}w)Nrrr) input_textgenerated_token_idsTrHrGr6decode_image_tokensrQrrrqr'rrr()r'r(r generated_)rTrUr$rrrangerrWr=rrdevicerVpost_process_multimodal_outputrrzipfindappendr%r/r}r.)r* model_outputsrOrErQpostprocess_kwargs input_textsrrirHgenerated_outputsnew_generated_textsdecoded_inputstext_generated decoded_inputindex_input_text full_textsrgenerated_textnew_textrgenerated_outputrecordss r" postprocesszAnyToAnyPipeline.postprocesss$M2 '1+T{'K{mQ\ *+?@!+. *,, ,s;/0 +1~FXYZF[\  6I5T1Z^,->?I6 g %'$**>S*T!%!?!?@R@U@UVZV`V`VgVg@h!i IDNNII  4G K]  :.. 0D0DE E#% 4: 0 1JT^^JJ/BFXN255F1W ?- #1#6#6}#E (-A-'..~>NQTUbQc>c>e/fg'..~> ?!4  *.. .J/2;@Q/R 2+ ^k3/%0>%AN T2-52=1E1Eb1I&1QU`1`.-#' (<(" #))IM6:59 dmDIoT *dmd3i$tCy/1J>Edmd3i*,t3 dm T#Y+d2 dm d38n T$sCx.%9 : :dmL.` n# Rr!r2))enumtypingrrrnumpynp audio_utilsr generationr image_utilsr processing_utilsr r utilsr r rrr video_utilsrbaserrtorchmodels.auto.modeling_autorpt_utilsrPILr get_loggerrrlEnumrr$r2r r!r"rs ''$)$7%4Q$   H % ! !,4@AoxoBor!