qi]$dZddlZddlZddlmZmZddlmZm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZddlmZddlmZmZdd lmZej>e Z!Gd d ed Z" dde#de#de#de#de#de#de#fdZ$GddeZ%dgZ&y)z#Image processor class for GLM-4.1V.N)BaseImageProcessor BatchFeature)convert_to_rgbresizeto_channel_dimension_format) OPENAI_CLIP_MEANOPENAI_CLIP_STDChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imagemake_flat_list_of_imagesto_numpy_array valid_imagesvalidate_preprocess_arguments) ImagesKwargs) TensorTypelogging) VideoInputc0eZdZUdZeed<eed<eed<y)Glm4vImageProcessorKwargsaR patch_size (`int`, *optional*, defaults to 14): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to 2): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to 2): The merge size of the vision encoder to llm encoder. patch_sizetemporal_patch_size merge_sizeN)__name__ __module__ __qualname____doc__int__annotations__b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/glm4v/image_processing_glm4v.pyrr0sOOr%rF)total num_framesheightwidthtemporal_factorfactor min_pixels max_pixelscR||krtd|d|||ks||kr.t||z ||z }t||z}t||z}t||t||z dkDr%tdt||t||z t ||z |z}t ||z |z} t ||z |z} | |z| z|kDrrt j ||z|z|z } t|t j|| z |z |z}t|t j|| z |z |z} || fS| |z| z|krZt j |||z|zz } t j|| z|z |z}t j|| z|z |z} || fS)Nzt:z% must be larger than temporal_factor:z4absolute aspect ratio must be smaller than 200, got ) ValueErrormaxr"minroundmathsqrtfloorceil) r(r)r*r+r,r-r.scaleh_barw_bart_barbetas r& smart_resizer>?sO#2j\)NN_`aa %&.FVOVe^4Ve^$EEM" 65C..4B3vuCUX[\bdiXjCjBk l   &6/ "V +E %&. !F *E *. // AE u}uz)yy*v-5CDFDJJv}v'=>GHFDJJut|f'<=FG %<   +yyzF':U'BCD &4-&01F: %$,/069 %<r%c"eZdZdZddgZeZddejdddddddddf d e d e e e fdzd ed e d e e zde de ee zdzde ee zdzde de de de ddffd Zddddddddddddej"dfdeezd e dzd e e e fdzd edzd e dzd e dzde dzde ee zdzde ee zdzde dzde dzde dzde dzdedzde ezdzfdZdddddddddddddej"dfded e dzd e e e fdzd edzd e dzd e dzde dzde ee zdzde ee zdzde dzde dzde dzde dzde ezdzdedzde ezdzf dZd de de fdZxZS)!Glm4vImageProcessora Constructs a GLM-4V image processor that dynamically resizes images based on the original images. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image's (height, width) dimensions. size (`Dict[str, int]` *optional*, defaults to `{"shortest_edge": 112 * 112, "longest_edge": 28 * 28 * 15000}`): Size of the image's `(height, width)` dimensions after resizing. Can be overridden by the `size` parameter in the `preprocess` method. Available options are: - `{"height": int, "width": int}`: The image will be resized to the exact size `(height, width)`. Do NOT keep the aspect ratio. - `{"shortest_edge": int, "longest_edge": int}`: The image will be resized to a maximum size respecting the aspect ratio and keeping the shortest edge less or equal to `shortest_edge` and the longest edge less or equal to `longest_edge`. - `{"max_height": int, "max_width": int}`: The image will be resized to the maximum size respecting the aspect ratio and keeping the height less or equal to `max_height` and the width less or equal to `max_width`. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`): Resampling filter to use when resizing the image. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `[0.48145466, 0.4578275, 0.40821073]`): Mean to use if normalizing the image. This is a float or list of floats for each channel in the image. image_std (`float` or `List[float]`, *optional*, defaults to `[0.26862954, 0.26130258, 0.27577711]`): Standard deviation to use if normalizing the image. This is a float or list of floats for each channel in the image. do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. patch_size (`int`, *optional*, defaults to 14): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to 2): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to 2): The merge size of the vision encoder to llm encoder. pixel_valuesimage_grid_thwTNgp? do_resizesizeresample do_rescalerescale_factor do_normalize image_mean image_stddo_convert_rgbrrrreturnc  $t|di| |d|vsd|vr td|ddd}||_||_||_||_||_||_||nt|_ ||nt|_ | |_ | |_| |_| |_y)N shortest_edge longest_edge:size must contain 'shortest_edge' and 'longest_edge' keys.1iq)rPrQr$)super__init__r1rFrErGrHrIrJr rKr rLrrrrM)selfrErFrGrHrIrJrKrLrMrrrkwargs __class__s r&rUzGlm4vImageProcessor.__init__s "6"  !<VZ@ZYZ Z \%.PD "  $,((2(>*DT&/&;$#6 $,r%images data_formatinput_data_formatc t|}| r|Dcgc] }t|}}|Dcgc] }t|}}|r#t|drtj d|t |d}t|d|\}}||}}g}|D]}|r/t| ||| | | z|d|d\}}t|||f||}|r|j|||}|r|j||| | }t||| }|j|tj|}|t j"k(r|j%dd d d }|j&d| zdk7rXtj(|dtj*| |j&d| zz d}tj,||gd}|j&d }|j&d| z}|| z|| z}}|j/|| ||| z| | || z| | }|j%dd dddd d dd }|j/||z|z|| z| z| z}||||ffScc}wcc}w)a~ Preprocess an image or batch of images. Copy of the `preprocess` method from `CLIPImageProcessor`. Args: images (`ImageInput`): Image or batch of images to preprocess. Expects pixel values ranging from 0 to 255. If pixel values range from 0 to 1, set `do_rescale=False`. vision_info (`List[Dict]`, *optional*): Optional list of dictionaries containing additional information about vision inputs. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image after resizing. `shortest_edge` and `longest_edge` keys must be present. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the `PILImageResampling` enums. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Mean to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Standard deviation to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. patch_size (`int`, *optional*, defaults to `self.patch_size`): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to `self.temporal_patch_size`): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to `self.merge_size`): The merge size of the vision encoder to llm encoder. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. data_format (`ChannelDimension`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. rzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.) channel_dimrPrQ)r(r)r*r+r,r-r.)rFrGr[)r9r[)imagemeanstdr[)input_channel_dimrrD)axis)rrrrlogger warning_oncerrr>rrescale normalizerappendnparrayr LAST transposeshaperepeatnewaxis concatenatereshape)rVrYrErFrGrHrIrJrKrLrrrrMrZr[r^r)r*resized_height resized_widthprocessed_imagespatchesrepeatschannelgrid_tgrid_hgrid_wflatten_patchess r& _preprocesszGlm4vImageProcessor._preprocesssz*&1 9?@nU+@F@6<vay I &vay>OP (.  +E0<2!$7% 2#O4#N31-  ?(fw U.Te fjiSd'0{VghE  # #E *1 +4((+, *// /''1a3G == 1 1Q 6ii BJJ')< a@PSf@f)gnoGnngw%7a@G--"q!%88':5} 7R//    j   j    ##Aq!Q1aA>!// VOf $g0C&Cj&PS]&]  888MA=s II return_tensorscz||n |j}|d|vsd|vr td||n |j}||n |j}||n |j}||n |j }||n |j }||n |j}| | n |j} | | n |j} | | n |j} | | n |j} | | n |j} ||j|}t|}|t|s tdt!|||| |||i}|gg}}|D]G}|j#||||||||| | | | || |\}}|j%||j'|It)j*|}t)j*|}|j-||dt/||S) a Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image after resizing. Shortest edge of the image is resized to size["shortest_edge"], with the longest edge resized to keep the input aspect ratio. resample (`int`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`. patch_size (`int`, *optional*, defaults to `self.patch_size`): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to `self.temporal_patch_size`): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to `self.merge_size`): The merge size of the vision encoder to llm encoder. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. rPrQrRzSInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, or torch.Tensor)rIrJrKrLrErFrG)rErFrGrHrIrJrKrLrrrrZrMr[)rArB)data tensor_type)rFr1rErGrHrIrJrKrLrrrrM fetch_imagesrrrrextendrnrorpupdater)rVrYrErFrGrHrIrJrKrLrrrrMrrZr[rrAvision_grid_thwsr^r{rBs r& preprocesszGlm4vImageProcessor.preprocess:sJ'tTYY  !<VZ@ZYZ Z!*!6IDNN '38#-#9Zt +9+E4K^K^'3'?|TEVEV #-#9Zt !*!6IDNN #-#9Zt 5H5T1Z^ZrZr#-#9Zt +9+E4K^K^  &&v.F-f5F  l6&:rs s%)%!   -/*L 8*.*:*:'%)#1!-)')(;) +#1&7+;+'"##G, ''7' 8(88L1L!xx(89  KKIYZ [>BBr%r)r*c 0|jd|j}|jd|j}|jd|j}||z}t |j ||||d|d|j \}} ||z| |z} } | | zS)a A utility that returns number of image patches for a given image size. Args: height (`int`): Height of the input image. width (`int`): Width of the input image. images_kwargs (`dict`, *optional*) Any kwargs to override defaults of the image processor. Returns: `int`: Number of image patches per image. rrrFrPrQ)r(r)r*r,r-r.r+)getrrrFr>r) rVr)r* images_kwargsrrrFr,rxryrrs r&get_number_of_image_patchesz/Glm4vImageProcessor.get_number_of_image_patchess#&&|T__E "&&|T__E   3j((4//O,N+ 44) % (:5} 7Rr%)N)rrr r!model_input_namesr valid_kwargsr BICUBICbooldictstrr"floatlistrUr FIRSTr rrrrr __classcell__)rXs@r&r@r@cs%N()9:,L&*'9'A'A&-!1504##$"-"-38nt#"-% "-  "- e "-"-DK'$."-4;&-"-"-"-!"-"- "-N"&&*.2"&'+$(1504!%*.!%&*/?/E/E;?!F9Z'F9$;F938nt# F9 %t+ F9 4K F9 F9TkF9DK'$.F94;&-F9$JF9!4ZF9$JF9t F9&,F9 !11D8!F9V"&&*.2"&'+$(1504!%*.!%&*26/?/E/E;?#ACAC$;AC38nt# AC %t+ AC 4K AC ACTkACDK'$.AC4;&-AC$JAC!4ZAC$JACt ACj(4/AC &,!AC"!11D8#ACF#cr%r@)rDrSi)'r!r5numpyroimage_processing_utilsrrimage_transformsrrr image_utilsr r r r r rrrrrrrprocessing_utilsrutilsrr video_utilsr get_loggerrrjrr"r>r@__all__r$r%r&rs* F    -(%   H %  E &0!! ! ! !  !  !!Hw,wt ! !r%