qildZddlmZddlZddlmZddlmZddl m Z m Z m Z ddl mZmZmZmZdd lmZej(eZerddlZd Zd Zd Zd ZdZdZdZedeGddeZdgZ y)z Processor class for SAM3. )deepcopyN) ImageInput)ProcessorMixin) BatchEncodingPreTokenizedInput TextInput) TensorTypeauto_docstringis_torch_availablelogging)requiresc|jd\}}}}|d|zz |d|zz |d|zz|d|zzg}tj|dSN?dimunbindtorchstackxx_cy_cwhbs Z/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/sam3/processing_sam3.pybox_cxcywh_to_xyxyr!#sVXXb\NCa a-3q=C#'MS37]LA ;;qb !!c|jd\}}}}|d|zz |d|zz ||g}tj|dSrrrs r box_cxcywh_to_xywhr$)sFXXb\NCa a-3q=A4A ;;qb !!r"ct|jd\}}}}||||z||zg}tj|dSNrrrryrrrs r box_xywh_to_xyxyr)/>"JAq!Q qAEQU$A ;;qb !!r"c|jd\}}}}|d|zz|d|zz||g}tj|dSrrr's r box_xywh_to_cxcywhr,5sF"JAq!Q cAg+S1WQ0A ;;qb !!r"ct|jd\}}}}||||z ||z g}tj|dSr&r)rr(XYrs r box_xyxy_to_xywhr0;r*r"c|jd\}}}}||zdz ||zdz ||z ||z g}tj|dS)Nrrr)rx0y0x1y1rs r box_xyxy_to_cxcywhr7AsNXXb\NBB r'QbA R27)selfrD tokenizerr>r?kwargs __class__s r rCzSam3Processor.__init__XsG )>v>.*5*A;tG[G[G`G`aiGjr"imagestextsegmentation_maps input_boxesinput_boxes_labelsoriginal_sizesreturn_tensorsreturnc d} ||j|f||d|} nW|Ht|tjr|j j }t d|i|} n | td|j||}|+|j||dd} | | j| n| } |,| d}|j|d d d d } |j|ddd} | |j| dd} | |j| dd}| |  k7r td| n|j|  d gz}tj|tj}|j!||ddt#|}| jd|i| J|j| }tj|tj$}| jd|i| S)a} images (`ImageInput`, *optional*): The image(s) to process. text (`str`, `list[str]`, `list[list[str]]`, *optional*): The text to process. segmentation_maps (`ImageInput`, *optional*): The segmentation maps to process. input_boxes (`list[list[list[float]]]`, `torch.Tensor`, *optional*): The bounding boxes to process. input_boxes_labels (`list[list[int]]`, `torch.Tensor`, *optional*): The labels for the bounding boxes. original_sizes (`list[list[float]]`, `torch.Tensor`, *optional*): The original sizes of the images. Returns: A [`BatchEncoding`] with the following fields: - `pixel_values` (`torch.Tensor`): The processed image(s). - `original_sizes` (`list[list[float]]`): The original sizes of the images. - `labels` (`torch.Tensor`): The processed segmentation maps (if provided). - `input_boxes_labels` (`torch.Tensor`): The processed labels for the bounding boxes. - `input_boxes` (`torch.Tensor`): The processed bounding boxes. N)rLrPrO) tensor_typezKEither images or original_sizes must be provided if input_boxes is not None max_length )rPpaddingrTrr9z)[image level, box level, box coordinates])expected_depth input_nameexpected_formatexpected_coord_sizer2labelsz[image level, box level])rXrYrZzaInput boxes and labels have inconsistent dimensions. Please ensure they have the same dimensions.)dtypeT)is_bounding_boxpreserve_paddingrMrN)rD isinstancerTensorcputolistr ValueError_resolve_text_promptsrGupdate_validate_single_input_get_nested_dimensions_pad_nested_listtensorfloat32_normalize_tensor_coordinatesr7int64)rFrJrKrLrMrNrOrPrHencoding text_inputsprocessed_boxesprocessed_boxes_labelsboxes_max_dimsboxes_labels_max_dims padded_boxes final_boxespadded_boxes_labelsfinal_boxes_labelss r __call__zSam3Processor.__call__esPD  +t++"3- H  '.%,,7!/!3!3!5!$&6%GUcdH  $jk k))$ <  ..nVboq.rK# ,&  "%&67N"99 " K$% :O&*%@%@" # : &A& "*!%!=[\K\]oo bjj ) 22c$**o6J{{}$jnobf44T>=[\K\]oo d #. !%^b]mD00~}WXGXYswwsEl +K6tDzlCD D'p ps EE! !E&c||rdSdSt|ttfs|St|}|r;t|t|k7r$t dt|dt|dt |D]\}}| |s ||d||<|S)zQ Resolve text prompts by setting defaults based on prompt types. NvisualzEThe number of text prompts must match the number of input boxes. Got z text prompts and z input boxes.)r`rtuplerrd enumerate)rFrKrMi text_values r rez#Sam3Processor._resolve_text_promptss <*8 4 4$u .KDz 3t9K(884yk!3C 4D3E]T  't_ #MAz!kk!n6P"Q # r"c|g}t|ts|St|dk(r|jt|nt |dt||d<t|dkDrz|D]u}|t|ts|j |}t |D]@\}}|dzt|k\r|j|)t ||dz|||dz<Bw|S)a Get the maximum dimensions at each level of nesting, skipping None values. Args: nested_list (`list`): Nested list structure (may contain None values). max_dims (`list`, *optional*): Current maximum dimensions (for recursion). Returns: `list`: A list of maximum dimensions for each nesting level. rrz)r`rrappendmaxrhr)rF nested_listmax_dimsrsub_dimsrrs r rhz$Sam3Processor._get_nested_dimensions.s  H+t,O x=A  OOC , -hqk3{+; template_dimstemplate_rs r rizSam3Processor._pad_nested_listVs"  ,,I C , , +t,&-K;' !-0  C ,q0 0    {kL.HI Ja 3{#3a#77$/ 0A0C$DM#BB=R[\H!*{[9J-KKH""kT`F`@a#b1HX$6#bc!,MA,=,? @ >>}iX""k@R#S1HX$6#ST 3{+a/ /3{+, vq>)$/ 0A0C$DM%)%H%HXa%bKN A5%)%:%:;q>;XehiXikt%uKN  v#$c $Ts F)F.ct|dk(r |g|dzSt|dDcgc]}|j|dd|c}Scc}w)a Create an empty nested structure with given dimensions filled with pad_value. Args: dims (`list`): The dimensions of the nested structure. pad_value (`int`): The value to fill the structure with. rzrN)rrr)rFdimsrrs r rz,Sam3Processor._create_empty_nested_structuresT t9>;a( (V[\`ab\cVdeQRD77QR)Le eesAct|tr/t|dk(ry|D]}|d|j|zcSyt|tj t jfrt|jSy)z Get the nesting level of a list structure, skipping None values. Args: input_list (`list`): The list to get the nesting level of. rrz) r`rr_get_nesting_levelrrrrar)rF input_listrs r rz Sam3Processor._get_nesting_levelsx j$ ':!#" =#t66t<<< =  RZZ$> ?z''( (r"rrXrYrZr[c |yt|tjtjfry|j |k7r"t d|d|d|d|j d |4|jd|k7r"t d|d|d|jdd |j||St|tr@|j|}||k7rt d|d |d |d|d |j||Sy) a Validate a single input by ensuring proper nesting and raising an error if the input is not valid. Args: data (`torch.Tensor`, `np.ndarray`, or `list`): Input data to process. expected_depth (`int`): Expected nesting depth. input_name (`str`): Name of the input for error messages. expected_format (`str`): The expected format of the input. expected_coord_size (`int`, *optional*): Expected coordinate size (4 for boxes, None for labels). . NzInput z must be a tensor/array with z, dimensions. The expected nesting format is z. Got z dimensions.rz as the last dimension, got .z must be a nested list with z( levels. The expected nesting format is z levels.) r`rrarrndimrdrrrr)rFrrXrYrZr[rs r rgz$Sam3Processor._validate_single_inputs0 < dU\\2::6 7yyN* ZL(EnEUVBCRBSSYZ^ZcZcYddpq%0::b>%88$  ,IJ]I^^z{|F|FGI|J{KKLM//nE E dD ! 33D9M. ZL(D^DTT|~M}NNTUbTcckl//nE E "r"cz|r"||jk7}|jdd}tt|D]}||jdks|t|kr||n|d}|j ||||} |r5|} t j| j||| ||||<|| ||<y)a Helper method to normalize coordinates in a tensor across multiple images. Args: tensor (`torch.Tensor`): Input tensor with coordinates. original_sizes (`list`): Original image sizes. is_bounding_box (`bool`, *optional*, defaults to `False`): Whether coordinates are bounding boxes. preserve_padding (`bool`, *optional*, defaults to `False`): Whether to preserve padding values (for boxes). rT)rkeepdimr)r^N) r?allrrrrrwhere expand_as) rFrjrOr^r_mask coord_maskimg_idxr~normalized_coordsimg_masks r rlz+Sam3Processor._normalize_tensor_coordinatess T111Db$7JS01 8Ga(;BSEX;Xw 7^lmn^o $($?$?7O]O%@%!$)'2H&+kk **6'?;=NPVW^P_'F7O'8F7O 8r"c<|jj|||S)a Converts the output of [`Sam3Model`] into semantic segmentation maps. Args: outputs ([`Sam3ImageSegmentationOutput`]): Raw outputs of the model containing semantic_seg. target_sizes (`list[tuple]` of length `batch_size`, *optional*): List of tuples corresponding to the requested final size (height, width) of each prediction. If unset, predictions will not be resized. threshold (`float`, *optional*, defaults to 0.5): Threshold for binarizing the semantic segmentation masks. Returns: semantic_segmentation: `list[torch.Tensor]` of length `batch_size`, where each item is a semantic segmentation map of shape (height, width) corresponding to the target_sizes entry (if `target_sizes` is specified). Each entry is a binary mask (0 or 1). )rD"post_process_semantic_segmentation)rFoutputs target_sizes thresholds r rz0Sam3Processor.post_process_semantic_segmentations!$##FFwP\^ghhr"c<|jj|||S)a Converts the raw output of [`Sam3Model`] into final bounding boxes in (top_left_x, top_left_y, bottom_right_x, bottom_right_y) format. This is a convenience wrapper around the image processor method. Args: outputs ([`Sam3ImageSegmentationOutput`]): Raw outputs of the model containing pred_boxes, pred_logits, and optionally presence_logits. threshold (`float`, *optional*, defaults to 0.3): Score threshold to keep object detection predictions. target_sizes (`list[tuple[int, int]]`, *optional*): List of tuples (`tuple[int, int]`) containing the target size `(height, width)` of each image in the batch. If unset, predictions will not be resized. Returns: `list[dict]`: A list of dictionaries, each dictionary containing the following keys: - **scores** (`torch.Tensor`): The confidence scores for each predicted box on the image. - **boxes** (`torch.Tensor`): Image bounding boxes in (top_left_x, top_left_y, bottom_right_x, bottom_right_y) format. Example: ```python >>> from transformers import AutoModel, AutoProcessor >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> model = AutoModel.from_pretrained("facebook/sam3-base") >>> processor = AutoProcessor.from_pretrained("facebook/sam3-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text="cat", return_tensors="pt") >>> outputs = model(**inputs) >>> # Post-process to get bounding boxes >>> results = processor.post_process_object_detection(outputs, threshold=0.3, target_sizes=[image.size[::-1]]) >>> boxes = results[0]["boxes"] >>> scores = results[0]["scores"] ``` )rDpost_process_object_detection)rFrrrs r rz+Sam3Processor.post_process_object_detection&s!V##AA'9Vbccr"c>|jj||||S)ay Converts the raw output of [`Sam3Model`] into instance segmentation predictions with bounding boxes and masks. This is a convenience wrapper around the image processor method. Args: outputs ([`Sam3ImageSegmentationOutput`]): Raw outputs of the model containing pred_boxes, pred_logits, pred_masks, and optionally presence_logits. threshold (`float`, *optional*, defaults to 0.3): Score threshold to keep instance predictions. mask_threshold (`float`, *optional*, defaults to 0.5): Threshold for binarizing the predicted masks. target_sizes (`list[tuple[int, int]]`, *optional*): List of tuples (`tuple[int, int]`) containing the target size `(height, width)` of each image in the batch. If unset, predictions will not be resized. Returns: `list[dict]`: A list of dictionaries, each dictionary containing the following keys: - **scores** (`torch.Tensor`): The confidence scores for each predicted instance on the image. - **boxes** (`torch.Tensor`): Image bounding boxes in (top_left_x, top_left_y, bottom_right_x, bottom_right_y) format. - **masks** (`torch.Tensor`): Binary segmentation masks for each instance, shape (num_instances, height, width). Example: ```python >>> from transformers import AutoModel, AutoProcessor >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> model = AutoModel.from_pretrained("facebook/sam3-base") >>> processor = AutoProcessor.from_pretrained("facebook/sam3-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text="cat", return_tensors="pt") >>> outputs = model(**inputs) >>> # Post-process to get instance segmentation >>> results = processor.post_process_instance_segmentation( ... outputs, threshold=0.3, target_sizes=[image.size[::-1]] ... ) >>> masks = results[0]["masks"] >>> boxes = results[0]["boxes"] >>> scores = results[0]["scores"] ``` )rD"post_process_instance_segmentation)rFrrmask_thresholdrs r rz0Sam3Processor.post_process_instance_segmentationSs'r##FF Y   r")Ni)NNNNNNN)F)r} torch.TensorrQr)r)N)rN)FF)Nr)333333?N)rrN) __name__ __module__ __qualname__rrCr rr rrr{rrastrr rrxrrrerhrirrrrrgrlrrr __classcell__)rIs@r r=r=Usbe k7:Tz k[^ k%)ae/3EIJNBF26fT!f++d9o=EV@WWZ^^f&, f $tE{+,u||;dB f !d3i1ELL@4G fT%[)ELL84?fj(4/f ffP6(ET6&P@D f8+/ 0FllRZZ'$.0F0F 0F  0F !4Z 0F 0Fd!8Fi(+d` ; r"r=)!__doc__copyrrr image_utilsrprocessing_utilsrtokenization_utils_baserrr utilsr r r r utils.import_utilsr get_loggerrloggerrr!r$r)r,r0r7r:r=__all__r"r rs%.RRLL*   H %" " " " " " ! :w Nw w t  r"