qiddlZddlZddlmZddlmZddlZddlZddl m cm Z ddlm Z m Z ddlmZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'erddl(m)Z)e!r ddl*m+Z+ddl,m-Z-ee dGddeZ. dPdejdejdejfdZ/de de de fdZ0dejdejdejfdZ1Gd d!e jdZ3de de d"e4de fd#Z5dejdejd"e4dejfd$Z6Gd%d&e jdZ7Gd'd(e jdZ8Gd)d*e jdZ9 dQd+e jdd,ejd-ejd.ejd/ejdzd0e:d1e:fd2Z;Gd3d4e jdZ<Gd5d6e jdZ=dRd7ejd8e:d9e>dejfd:Z?Gd;de jdZAGd?d@e jdZBGdAdBeZCGdCdDe jZEGdEdFe jdZFGdGdHe jdZGGdIdJe jdZHe GdKdLeZIe dMGdNdOeIZJdLdOgZKy)SN)Callable) dataclass)Tensornn)initialization)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)merge_with_config_defaults)capture_outputs) EomtConfig)linear_sum_assignment) PartialState)reducea Class for outputs of [`EomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~EomtImageProcessor.post_process_semantic_segmentation`] or [`~EomtImageProcessor.post_process_instance_segmentation`] or [`~EomtImageProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~EomtImageProcessor] for details regarding usage. ) custom_introc<eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<dZeej dzed <y) "EomtForUniversalSegmentationOutputa* loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_state hidden_states attentions patch_offsets)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rr r!r"tupler#r$listrX/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/eomt/modeling_eomt.pyrr3s*&*D%  d ")59%++d2959%++d2926u((4/659M5**+d2926Je''(4/6/3M4 %,3r/rinput_featurespoint_coordinatesreturnc |jdk(rd}|jd}tjjj |d|zdz fi|}|r|j d}|S)a( A wrapper around `torch.nn.functional.grid_sample` to support 3D point_coordinates tensors. Args: input_features (`torch.Tensor` of shape (batch_size, channels, height, width)): A tensor that contains features map on a height * width grid point_coordinates (`torch.Tensor` of shape (batch_size, num_points, 2) or (batch_size, grid_height, grid_width,: 2)): A tensor that contains [0, 1] * [0, 1] normalized point coordinates add_dim (`bool`): boolean value to keep track of added dimension Returns: point_features (`torch.Tensor` of shape (batch_size, channels, num_points) or (batch_size, channels, height_grid, width_grid): A tensor that contains features for points in `point_coordinates`. rTg@?)dim unsqueezer)r functional grid_samplesqueeze)r1r2add_dimkwargspoint_featuress r0 sample_pointr?^st(!#-77:XX((44^SK\E\_bEbmflmN'//2 r/inputslabelsc|jjd}dtj||jz}|j ddddf|j ddddfz}d|dz|dzz z }|S)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. rr5N)sigmoidflattenr)matmulTsum)r@rA numerator denominatorrs r0pair_wise_dice_lossrK~s|^^  % %a (FELL22I**R.D)FJJrN47,CCK A +/2 2D Kr/c\|jd}tjd}||tj|}||tj |}tj ||z |j}tj ||z d|z j}||z}|S)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnone reduction)shaperBCEWithLogitsLossr) ones_like zeros_likerFrG) r@rAheight_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negrs r0$pair_wise_sigmoid_cross_entropy_lossrZs||A$$v6I&vuv/FG&vu/?/?/GH||25EEvxxPH||25EEF ~~VH h D Kr/c eZdZdZ d dedededeffd Zejdejdejd ejd ejd e e e f d Z xZ S)EomtHungarianMatcheraqThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). cost_class cost_mask cost_dice num_pointsct||dk(r|dk(r|dk(r td||_||_||_||_y)aHCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)super__init__ ValueErrorr`r]r^r_)selfr]r^r_r` __class__s r0rczEomtHungarianMatcher.__init__sK"  ?yA~)q.34 4$$""r/r r mask_labels class_labelsr3c g}|jd}t|D]}||jd}||} |dd||f } ||j| } | dddf} | dddf} t j d|j d| j} | j| jddd} t| | djd} | j| jddd}t| |djd} t| | }t| | }|j|z|j| zz|j|zz}t j |t j"d }t j$|t j"d }t j&|d}t)|j+}|j-||Dcgc]O\}}t j.|tj0 t j.|tj0 fQ}}}|Scc}}w) ao Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rrCNrr5deviceF align_cornersg _Bg _©dtype)rPrangesoftmaxtor)randr`rkrepeatr?r;rZrKr^r]r_minimumtensormaximum nan_to_numrcpuappend as_tensorint64)rer rrgrhindices batch_sizei pred_probs pred_maskr] target_maskr2target_coordinatespred_coordinatesr^r_ cost_matrixassigned_indicesjmatched_indicess r0forwardzEomtHungarianMatcher.forwards/8*,*//2 z" -A-a088ejjY_` 36wP]3^ G /FM< +FLLO+->v||A->@Q&,,q/@QQ R X XY_ `AFL*6<<?*,=fll1o,== > G},,&PsD+rrhr}c |}|j\}}}tj|j}|j |} t j t||D  cgc] \} \}} | | c} }} } t j||f|jt j|j} | | | <|jdd}||| }d|i}|Scc} }} w)aCompute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )weight) fill_valuerorkrr5loss_cross_entropy) rPrCrossEntropyLossr$_get_predictions_permutation_indicesr)catrfullrr|rk transpose)rerrhr} pred_logitsr~ num_queriesrrUidxtargetrtarget_classes_otarget_classespred_logits_transposedloss_celossess r0 loss_labelszEomtLoss.loss_labelss"+ %0%6%6" K''t/@/@A 77@ 99-0w-G H H>66AqVAY H   %$//]h]o]o /s!,!6!6q!!<2NC&0  Is#C! r rgrcj|}j|}||}j|\}} ||}|dddf}|dddf}tj5j |fdj jj} t|| djd} dddt| djd} t|  |t| | |d} ~~| S#1swYExYw)aCompute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Nc&j|SN)calculate_uncertainty)logitsres r0z%EomtLoss.loss_masks..st99&Ar/Frlr) loss_mask loss_dice) r _get_targets_permutation_indicesrr)rsample_points_using_uncertaintyr`rrr?r;rr)rer rgr}rsrc_idxtgt_idx pred_masks target_masksrr2 point_labels point_logitsrs` r0 loss_maskszEomtLoss.loss_maskss,4;;GD77@)'2 ::;G a#G,  4( #AtG, ]]_ i $ D DA%%,, ! ( 6GW\]eefghL i$J0AQVW__`ab 4L,PYZ"<yI    ) i is (ADD c tjt|Dcgc]\}\}}tj|| c}}}}tj|Dcgc]\}}| c}}}||fScc}}}wcc}}wrr)rr full_like)rer}rsrcr batch_indicespredictions_indicess r0rz-EomtLoss._get_predictions_permutation_indicessj iX_N`"a"a{q(35??3#:"ab #iiW(E#q(EF111#b(E #A7  A> c tjt|Dcgc]\}\}}tj|| c}}}}tj|Dcgc]\}}| c}}}||fScc}}}wcc}}wrr)rer}rrtgtrtarget_indicess r0rz)EomtLoss._get_targets_permutation_indicessh iX_N`"a"a{q(1c5??3#:"ab #@HQC#@An,,#b#@rrc2tj| }|S)a In Eomt paper, uncertainty is estimated as L1 distance between 0.0 and the logit prediction in 'logits' for the foreground class in `classes`. Args: logits (`torch.Tensor`): A tensor of shape (R, 1, ...) for class-specific or class-agnostic, where R is the total number of predicted masks in all images and C is: the number of foreground classes. The values are logits. Returns: scores (`torch.Tensor`): A tensor of shape (R, 1, ...) that contains uncertainty scores with the most uncertain locations having the highest uncertainty score. )r)abs)reruncertainty_scoress r0rzEomtLoss.calculate_uncertaintys %yy01!!r/r`rrc |jd}t||z}tj||d|j}t ||d} || } t||z} || z } tj | dddddf| dd} |tj|tj|j z}| |dddfz } |jd d| jd ddfj|| d}| dkDr:tj|tj|| d|jgd }|S) a This function is meant for sampling points in [0, 1] * [0, 1] coordinate space based on their uncertainty. The uncertainty is calculated for each point using the passed `uncertainty function` that takes points logit prediction as input. Args: logits (`float`): Logit predictions for P points. uncertainty_function: A function that takes logit predictions for P points and returns their uncertainties. num_points (`int`): The number of points P to sample. oversample_ratio (`int`): Oversampling parameter. importance_sample_ratio (`float`): Ratio of points that are sampled via importance sampling. Returns: point_coordinates (`torch.Tensor`): Coordinates for P sampled points. rr5rjFrlNr)kr7rrCr7) rPrr)rsrkr?topkarangelongviewr)reruncertainty_functionr`rr num_boxesnum_points_sampledr2rpoint_uncertaintiesnum_uncertain_pointsnum_random_pointsrshifts r0rz(EomtLoss.sample_points_using_uncertaintysI<LLO  .>!>?"JJy2DaPVP]P]^#F,=US 2<@"#:Z#GH&)==jj,Q1W59MSTUVWX"U\\)5::V\VcVc%dd uQW~-222q9#((2,/JOOPY[oqrs q % "EJJy:KQW]WdWd$ef! ! r/Nauxiliary_predictionsc|j||||}|j||dj}i|j|||||j |||}|jt |D]\\} } | d}| d}|j ||||} | jD cic] \} } | d| | } } } |j| ^|Scc} } w)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], then it contains the logits from the inner layers of the EomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrjr rr) r get_num_masksrkrrrritemsupdate)rer rrgrhrr}rrr aux_outputs loss_dictkeyvalues r0rzEomtLoss.forward=s H,,35I;Xde&&|LOz)EomtLoss.get_num_masks..xsAG Asrr)min) rHr)r{rrr _shared_stater num_processesclamp)rerhrkr world_sizes r0rzEomtLoss.get_num_maskstsuALAA OOIU[[P  " $))R/"9- )^99 KK J 6A> r/r)r%r&r'rdictstrrrcr-rrrr,rnparrayrr)rrrrrrrkrrrs@r0rrLs! z! S%Z8H! F$tCy/d3i-4<-E&RX.DY-" $* :>v, QVWYW_W_Q`  c6k  D<#ll<%,,'<rxx <  < c5<< <|2 - "ELL"U\\""5! 5! 5!  5! "' 5! 5!zAE 5#ll5$ll5%,,' 5 5<<( 5 $C$56= 5 c5<< 5n %,,  QVQ]Q] r/rcZeZdZdZfdZdej dej fdZxZS)EomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr kernel_sizestride)rbrc image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)rerr!r"r#r$r)rfs r0rczEomtPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir/ pixel_valuesr3c|jd}||jk7rtd|jd|d|j|j dj dd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r5)rPr#rdr+rEr)rer,r# embeddingss r0rzEomtPatchEmbeddings.forwardsz#))!, 4,, ,!../yaI __\2::1=GG1M r/) r%r&r'r(rcr)rrrrs@r0rrs) jELLU\\r/rcdeZdZdZdeddffd ZdejdejfdZxZ S)EomtEmbeddingszM Construct the CLS token, mask token, position and patch embeddings. rr3Nct|||_|j|_t j t jdd|j|_ t j t jd|j|j|_ t||_|jj}t j |j"|_d|jz|_t j(||j|_|j-dt j.|j1ddy)Nr position_idsrrCF) persistent)rbrcrr"r Parameterr)randnr$ cls_tokenrnum_register_tokensregister_tokensrpatch_embeddingsr)Dropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsrrexpand)rerr)rfs r0rczEomtEmbeddings.__init__s   ++ekk!Q8J8J&KL!||EKK6;U;UW]WiWi,jk 3F ;++77 zz&"<"<= !"V%?%?!?#%<< V=O=O#P  ^U\\+-F-M-Mg-Vchir/r,c|j\}}}}|jjjj}|j|j |}|j j|dd}|jj|dd}||j|jz}tj|||gd}|j|}|S)NrnrCrr)rPr;r+rrorrr8rBr:rAr3r)rr>)rer,r~r target_dtyper/ cls_tokensr:s r0rzEomtEmbeddings.forwards*00 Aq!,,77>>DD ** ..55j"bI$":":4;L;L"MM YY OZHaP \\*- r/ r%r&r'r(rrcr)rrrrs@r0r1r1s9jzjdj ELL U\\ r/r1modulequeryr r attention_maskscalingr>c tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)NrC)r7ro)ptrainingrr5) r)rFrrr9rqfloat32rrror>rN contiguous) rGrHr r rIrJr>r= attn_weights attn_outputs r0eager_attention_forwardrSs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r/c eZdZdZfdZ ddej dej dzdeej ej dzffdZxZ S) EomtAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)rbrcrr$ embed_dimnum_attention_heads num_headshead_dimrdscaleattention_dropoutr> is_causalrLineark_projv_projq_projout_projrerrfs r0rczEomtAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r/Nr"rIr3c |j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} tj|jjt} | |||| ||j|j|jsdn |j \} } | j#|||j%} |j'| } | | fS)z#Input shape: Batch x Time x Channelrr5)r]rJr>)rPrar_r`rrYrZrr get_interfacer_attn_implementationrSr]r[rNr>reshaperPrb) rer"rIr=r~ seq_lengthrWquerieskeysvaluesattention_interfacerRrQs r0rzEomtAttention.forwardsW-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(?(M(M KK , ,.E) %8     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0 L((r/r) r%r&r'r(rcr)rr,rrrs@r0rUrUsVGB./3$)||$) t+$) u||U\\D00 1 $)r/rUcXeZdZdfd Zdej dej fdZxZS)EomtLayerScaler3ct|tj|jt j |jz|_yr) rbrcrr6layerscale_valuer)rr$lambda1rcs r0rczEomtLayerScale.__init__ s8 ||F$;$;ejjI[I[>\$\] r/ hidden_statec ||jzSr)rrrerss r0rzEomtLayerScale.forward$sdll**r/r3Nr%r&r'rcr)rrrrs@r0roros$^+ELL+U\\+r/roinput drop_probrNc|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)zc Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). rerr)rr)rPndimr)rsrorkfloor_div)rxryrN keep_probrP random_tensoroutputs r0 drop_pathr(s Cx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr/cxeZdZdZd dedzddffd ZdejdejfdZde fdZ xZ S) EomtDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nryr3c0t|||_yr)rbrcry)reryrfs r0rczEomtDropPath.__init__:s "r/r"cDt||j|jSr)rryrNrer"s r0rzEomtDropPath.forward>s FFr/c d|jS)Nzp=)ryres r0 extra_reprzEomtDropPath.extra_reprAsDNN#$$r/r) r%r&r'r(rrcr)rrrrrrs@r0rr7sGb#%$,#$#GU\\GellG%C%r/rcXeZdZdfd Zdej dej fdZxZS)EomtMLPr3c~t||jx}}t|j|jz}t j ||d|_t|jtrt|j|_ n|j|_ t j ||d|_ y)NTbias)rbrcr$r mlp_ratiorr^fc1r% hidden_actrr activationfc2rer in_features out_featureshidden_featuresrfs r0rczEomtMLP.__init__Fs %+%7%77 lf0063C3CCD99[/E f'' -$V%6%67DO$//DO99_lFr/rscl|j|}|j|}|j|}|Sr)rrrrus r0rzEomtMLP.forwardQs2xx - |4 xx - r/rvrwrs@r0rrEs$ GELLU\\r/rcXeZdZdfd Zdej dej fdZxZS) EomtSwiGLUFFNr3c0t||jx}}t|j|jz}t|dzdz dzdzdz}t j |d|zd|_t j ||d|_y)Nr5rTr) rbrcr$rrrr^ weights_in weights_outrs r0rczEomtSwiGLUFFN.__init__Ys %+%7%77 lf0063C3CCD2Q67!;AAE))K_1D4P99_lNr/rsc|j|}|jdd\}}tjj ||z}|j |S)Nr5rCr)rchunkrr9silur)rersx1x2hiddens r0rzEomtSwiGLUFFN.forwardbsS|4 ##A2#.B##B'",''r/rvrwrs@r0rrXs$O(ELL(U\\(r/rceZdZdZdeddffd Z d dejdejdzdejfdZxZ S) EomtLayerzCThis corresponds to the Block class in the original implementation.rr3Nct|tj|j|j |_t||_t||_ |jdkDrt|jntj|_tj|j|j |_|j rt#||_nt'||_t||_y)Nepsre)rbrcr LayerNormr$layer_norm_epsnorm1rU attentionro layer_scale1drop_path_raterIdentityrnorm2use_swiglu_ffnrmlpr layer_scale2rcs r0rczEomtLayer.__init__ls \\&"4"4&:O:OP &v.*62@F@U@UX[@[f&;&;>2Dn#U q $ 1 12G H'<= M zz-0 xx - ((6 ~~l3mC r/rrFrs@r0rrisPM3z3d3&/3|| t+  r/rcXeZdZdfd Zdej dej fdZxZS)EomtLayerNorm2dc*t||||y)N)relementwise_affine)rbrc)rer#raffinerfs r0rczEomtLayerNorm2d.__init__s 36Jr/rsr3c|jdddd}tj||j|j|j |j }|jdddd}|S)Nrr5rr)permuteF layer_normnormalized_shaperrrrus r0rzEomtLayerNorm2d.forwardsb#++Aq!Q7 ||L$2G2GVZV_V_aeaiaij #++Aq!Q7 r/)gư>Trwrs@r0rrs$KELLU\\r/rc\eZdZdeffd Zdej dej fdZxZS)EomtScaleLayerrct||j}tj||dd|_t |j|_tj||dd|d|_ t||_ y)Nr5rrrF)rpaddinggroupsr) rbrcr$rConvTranspose2dconv1r rrr*conv2r layernorm2drerr$rfs r0rczEomtScaleLayer.__init__su (( '' [aXYZ  !2!23YY     +;7r/r"r3c|j|}|j|}|j|}|j|}|Sr)rrrrrs r0rzEomtScaleLayer.forwardsB =1  6  =1 ((7 r/ r%r&r'rrcr)rrrrs@r0rrs*8z8 U\\ellr/rc\eZdZdeffd Zdej dej fdZxZS)EomtScaleBlockrct||j|_t j t |jDcgc] }t|c}|_ycc}wr) rbrcnum_upscale_blocks num_blocksr ModuleListrprblockrerrrfs r0rczEomtScaleBlock.__init__sG  33]]E$//DZ#[qN6$:#[\ #[sA&r"r3c8|jD] }||} |Sr)r)rer"rs r0rzEomtScaleBlock.forwards%ZZ 1E!-0M 1r/rrs@r0rrs,]z] U\\ellr/rc\eZdZdeffd Zdej dej fdZxZS) EomtMaskHeadrc t||j}tj|||_tj|||_tj|||_t|j|_ yr) rbrcr$rr^rrfc3r rrrs r0rczEomtMaskHead.__init__sa (( 99[+699[+699[+6 !2!23r/r"r3c|j|j|}|j|j|}|j|}|Sr)rrrrrs r0rzEomtMaskHead.forwardsD(?@ (?@ / r/rrs@r0rrs*4z4U\\ellr/rceZdZUdZeed<dZdZdZdZ dgZ dZ e e d Zej d ej$d d fd Zy )EomtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. reomtr,)imageFrT)r"r#rGr3Nc|jj}t|tjtj tj frtj|jtjd|jxtjjj|j\}}|dkDrdtj|z nd}tj|j| |yyt|tj r?tj"|jtj$|jyt|tj&rttj(|jdd|j*Et-|jdds-tj$|j|j*yyyt|t.rBt1|d r5tj2|j4|jj6yyt|t8rtj:|j<d|tj$|j>tj@|jBtjD|jBjFd jId yt|tJrRtjL|jNdz}|jP|d <tj@|jR|yt|tTr tj"|jVyy) N)arrre)rstd_is_hf_initializedFrrrCr4),rinitializer_ranger%rr^r*rinitkaiming_uniform_rmathsqrtrr)_calculate_fan_in_and_fan_outuniform_rones_zeros_r@normal_ padding_idxgetattrrohasattr constant_rrrqr1 trunc_normal_r8r:rr3rrPrBrrrrrEomtForUniversalSegmentationattn_mask_probs)rerGrfan_inrboundrs r0 _init_weightsz!EomtPreTrainedModel._init_weightssJkk++ fryy"))R5G5GH I  ! !&--499Q< @{{&!HHMMGG V 17!DIIf-- fkkE659' - JJv}} % KK $  - LLSa 8!!-gfmmMach6i FMM&*<*<=>7j-  /vy)v~~t{{/K/KL*  /   v//cs C KK.. / JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  ) ::f&7&7!&;r/)r%r&r'r(rr+base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdparrU_can_record_outputsr)rrModulerr.r/r0rrso $O!&+#$ N"# U]]_/BII/$//r/rzV The EoMT Model with head on top for instance/semantic/panoptic segmentation. ceZdZdZdeffd Zdededededeeefd eeeff d Z d eeefd efd Z e e e ddedeed zdeed zdeed zdeed ef dZdZdej fdZedZxZS)rr,rct||||_|j|_t ||_t j|j|j|_ t j|j|j|_ t jt|jDcgc] }t!|c}|_t%||_t)||_t j,|j|j.dz|_|j2|j4z|j2|j4zf|_|j8|j:|j<d|_tA||j>|_!|jEdtGjH|jJ|jMycc}w)Nrr)rrr)rrr)'rbrcrnum_hidden_layersr1r/rrr$r layernormr@rrHrrprlayersr upscale_blockr mask_headr^rclass_predictorr!r" grid_sizerrrrrrUrr)rr post_initrs r0rcz%EomtForUniversalSegmentation.__init__sp   !'!9!9(0f&8&8f>S>ST\\&"4"4f6H6HI mmfF^F^@_$`1Yv%6$`a +F3%f-!yy););V=N=NQR=RS ++v/@/@@&BSBSW]WhWhBhi"("5"5++++.  "T=M=MN . 6;L;L0MN %%as>G*r rrgrhrr3c|j|||||}|jjD]'\}}|jD]\} } || vs | |z} )|S)Nr rrgrhr)rUrr) rer rrgrhrr r rloss_keyrs r0 get_loss_dictz*EomtForUniversalSegmentation.get_loss_dict+s|(,~~!5!5#%"7 (6(  ++113 #KC"+//"3 #$(?FND # # r/r c4t|jSr)rHrl)rer s r0get_lossz%EomtForUniversalSegmentation.get_lossCs9##%&&r/Nr$r=c (d\}}d}| td|j|} t|jD]\} } | |j|j j z k(rp|jjdddddfj| jdddj| j} tj| | fd} | |j|j j z k\r|js7|j | |jz |j j zdkDr|j#| } |j%| \}}||fz }||fz }tj&| jd| jd| jd| jtj(}t+j,||j.d }|j1|j3d|j3dd}|j j4}||jj6z}|dkD|ddd||df<|j9||j | |jz |j j z|||j }|dddd fjd|j j:dd}|j=j?|d }| | |} |j#| }|j%|\}}||fz }||fz }d}|B|@d}tA||D]/\}}|jC||||d}||jE|z }1tG|||||S)ag mask_labels (`list[torch.Tensor]`, *optional*): list of mask labels of shape `(num_labels, height, width)` to be fed to a model class_labels (`list[torch.LongTensor]`, *optional*): list of target class labels of shape `(num_labels, height, width)` to be fed to a model. They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segmentation. )r.r.Nz You have to specify pixel_valuesrrCrr)rkrobilinear)sizemode)probnum_query_tokensencoder_start_tokensrk.gerer)rr rr!r$)$rdr/rrrrrrHrrBrPrrrkr)rrNrrpredictrrr interpolater rrrr?_disable_attention_maskrXr masked_fillrrrr)rer,rgrhr$r=masks_queries_logits_per_layerclass_queries_logits_per_layerrIr"r layer_modulerHnorm_hidden_statesr rinterpolated_logitsrrsequence_outputrr s r0rz$EomtForUniversalSegmentation.forwardFs*JPF&(F  ?@ @ 5 !*4;;!7. H Cd,,t{{/E/EEE ))$1*5<<]=P=PQR=SUWY[\__`m`t`tu % 5-*@a H d,,t{{/E/EEE !5!5cD&>',,Q/1D1I1I!1Lb'#$(;;#:#: '7$//:[:['[$ObdeNeq"3#3"35I5JJK"&!=!="--cD4J4J.JT[[McMc.cd%5)=)00 ">""04 !=!D!DRIhIhjlnp!q!/!5!5!7!C!C^OUY!Z(GM]. H`..759\\/5R22&+?*AA&&+?*AA&  "|'?D>A.0N? 1:$&:!..)=)= +!-*. /  i00 12!5!5-'   r/c.|jjSr)r/r;rs r0get_input_embeddingsz1EomtForUniversalSegmentation.get_input_embeddingss///r/rc|ddd|jjddf}|j|}|dd|jj|jjzdddf}|j dd}|j |jddg|j}|j|}|j|}tjd||}||fS)Nrr5rrCzbqc, bchw -> bqhw) rrr r/r?rrhrPr r rr)einsum)rer query_tokens class_logits prefix_tokens mask_logitss r0rz$EomtForUniversalSegmentation.predictsa!:4;;#:#:!:A=> ++L9 q$++"9"9DOO<]<]"]"_abbc %//15 - --m.A.A!.DbZ4>>Z ~~l3 **=9 ll#6 mT L((r/c|dkr9tj|jd|||kD}d|ddd||df|<|S)Nrrrj)r)rsrP) attn_maskrrrrkrandom_queriess r0rz4EomtForUniversalSegmentation._disable_attention_masksW !8"ZZ (: Rr/)NNN)r%r&r'rrrcrrrrrrrrr-rrrrr%r)r staticmethodrrrs@r0rrs? %Oz8$%    $CK0  c6k 0'$sF{"3'' ,0,0-1 e e &\D(e 6lT) e F|d* e +, e  ,e  e N0)ell) r/r)F)re)reF)Lcollections.abcr&rr dataclassesrnumpyrr)torch.nn.functionalrr9rrrr activationsr file_utilsr r r modeling_layersr modeling_utilsrrprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrconfiguration_eomtrscipy.optimizer accelerateraccelerate.utilsrrr?rKrZrr\rrrrrr1rrSrUrorrrrrrrrrrrrr__all__r.r/r0rBs* $! &!LL9F&PP75*4''  44  4BLQLL5:\\ \\@6,u||X]XdXd8g299gTff<u||U\\VY^c^j^j(uryyup "))B"RYY"X% II% <<% % << % LL4' %  %%.;)BII;)|+RYY+ U\\ e T V[VbVb  %299 %bii&(BII("'*'TbllRYY2 RYY 299"///////d @#6@  @F !"@ Ar/