qiDddlmZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZmZdd lmZdd lmZmZmZdd lmZddlmZddlmZeedGddeZGddej@Z!dZ"dAdZ#dejHde%dejHfdZ& dBdej@dejHd ejHd!ejHd"ejHdzd#e'd$e'd%eefd&Z(e e#Gd'd(ej@Z)Gd)d*ej@Z*Gd+d,ej@Z+d-ejHd.ejHd/ejHdejHfd0Z,Gd1d2ej@Z-Gd3d4ej@Z.eGd5d6eZ/d7ejHd8e'de0ejHejHffd9Z1d:ejHd;e%dGd?d@e/Z3d6d@gZ4y)C)Callable) dataclassN)nn pad_sequence)ACT2FN)use_kernelized_func)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstring)can_return_tuple)AutoModelForKeypointDetection)LightGlueConfiga Base class for outputs of LightGlue keypoint matching models. Due to the nature of keypoint detection and matching, the number of keypoints is not fixed and can vary from image to image, which makes batching non-trivial. In the batch of images, the maximum number of matches is set as the dimension of the matches and matching scores. The mask tensor is used to indicate which values in the keypoints, matches, matching_scores and prune tensors are keypoint matching information. ) custom_introc^eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ejdzed<dZ ej dzed<dZeej dzed <dZeej dzed <y) LightGlueKeypointMatchingOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*): Loss computed during training. matches (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Index of keypoint matched in the other image. matching_scores (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Scores of predicted matches. keypoints (`torch.FloatTensor` of shape `(batch_size, num_keypoints, 2)`): Absolute (x, y) coordinates of predicted keypoints in a given image. prune (`torch.IntTensor` of shape `(batch_size, num_keypoints)`): Pruning mask indicating which keypoints are removed and at which layer. mask (`torch.BoolTensor` of shape `(batch_size, num_keypoints)`): Mask indicating which values in matches, matching_scores, keypoints and prune are keypoint matching information. hidden_states (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(batch_size, 2, num_channels, num_keypoints)` returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True` attentions (`Tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, 2, num_heads, num_keypoints, num_keypoints)` returned when `output_attentions=True` is passed or when `config.output_attentions=True` Nlossmatchesmatching_scores keypointsprunemask hidden_states attentions)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrr IntTensorrr tupler!b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/lightglue/modeling_lightglue.pyrr's0&*D%  d ")(,GU   %,04OU&&-4*.Iu  4'.$(E5??T !(%)D%  d ")59M5**+d2926Je''(4/6r,rc eZdZdeffd Z ddej dedzdeej eej ej fzfdZ xZ S) LightGluePositionalEncoderconfigct|tjd|j|j zdzd|_y)NrFbias)super__init__rLineardescriptor_dimnum_attention_heads projectorselfr0 __class__s r-r5z#LightGluePositionalEncoder.__init__Us: 1f&;&;v?Y?Y&Y]^&^ejkr,routput_hidden_statesNreturnc|j|}|jdd}tj|}tj|}||f}|r||f}|S|f}|S)Nrdim)r9repeat_interleaver&cossin)r;rr=projected_keypoints embeddingscosinessinesoutputs r-forwardz"LightGluePositionalEncoder.forwardYsq#nnY7(::1":E ))J' *%u% 6J*12 R\P] r,F) r"r#r$rr5r&Tensorboolr*rK __classcell__r<s@r-r/r/Ts]ll LQ  =AD[  u|| uU\\5<<%?@ @ r,r/c|ddddf}|ddddf}tj| |gdjd}|S)N.rrr@rA)r&stackflatten)xx1x2rot_xs r- rotate_halfrYesL 3!8B 319B KK"b r * 2 22 6E Lr,c6|j}|j}|j}|j|}|j|}||zt||zz}||zt||zz}|j ||j |fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. dtype)r\float unsqueezerYto)qkrDrE unsqueeze_dimr\q_embedk_embeds r-apply_rotary_pos_embrems$ GGE  A  A -- &C -- &C3w;q>C/0G3w;q>C/0G ::E: "GJJUJ$; ;;r,r n_repr>c|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)shapeexpandreshape)r rfbatchnum_key_value_headsslenhead_dims r- repeat_kvroso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr,modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nrrr@)rBr\)ptrainingr)ronum_key_value_groupsr&matmul transposer functionalsoftmaxfloat32r_r\rvrz contiguous) rprqrrrsrtrurvrw key_states value_states attn_weights attn_outputs r-eager_attention_forwardrs3 ; ;$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r,Nr position_embeddingsrtencoder_hidden_statesencoder_attention_maskrwr>c |jdd}g|d|j}|j|j|j dd} |du} | r|n|} | r|n|} |j | j|j dd} |j | j|j dd}||\}}t| | ||\} } tj|jjt}||| | || f|jsdn |j|jd|\}}|j g|dj#}|j%|}||fS)Nr@rr)rvru)rhrnrviewr}rrrer get_interfacer0_attn_implementationrrzrrurjrr)r;r rrtrrrw input_shape hidden_shape query_statesis_cross_attentioncurrent_statescurrent_attention_maskrrrDrEattention_interfacerrs r-rKzLightGlueAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST 2$>2D.-;M!7Sa[[055lCMMaQRS {{>277 EOOPQSTU  **HC';L*VY[^'_ $L*(?(M(M KK , ,.E) %8     " % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r,)NNNN)r"r#r$r%rintr5r&rMr*rr rKrOrPs@r-rrsG  3 4IM.2596: *)||*)#5<<#=>E*) t+ *) %||d2 *) !& t 3 *)-.*) u||U\\D00 1*)r,rc\eZdZdeffd Zdej dej fdZxZS) LightGlueMLPr0cht|||_t|j|_t j|j|j|_ t j|j|j|_ t j|jd|_ y)NT)elementwise_affine)r4r5r0r hidden_act activation_fnrr6intermediate_sizefc1rfc2 LayerNorm layer_normr:s r-r5zLightGlueMLP.__init__s}  #F$5$5699V55v7O7OP99V55v7I7IJ,,v'?'?TXYr,r r>c|j|}|j|}|j|}|j|}|SN)rrrr)r;r s r-rKzLightGlueMLP.forwardsB/  6 **=9 / r, r"r#r$rr5r&rMrKrOrPs@r-rrs,ZZU\\ellr,rceZdZdedeffd Z d dejdejdejdedzd edzd e eje ejdze ejdzff d Z xZ S) LightGlueTransformerLayerr0rct|t|||_t ||_t|||_t ||_yr)r4r5rself_attentionrself_mlpcross_attention cross_mlprs r-r5z"LightGlueTransformerLayer.__init__sD 0C$V, 1&)D%f-r, descriptorsrrtr=Noutput_attentionsr>c|rdnd}|rdnd}|r||fz}|j\}} } |j||||\} } tj|| gd} |j | }||z}|r| |f}|j dd| | j dj || | }|6|j dddd| j dj |dd| nd}|j||||\}}tj||gd}|j|}||z}|r6||f}||j || | fzz|j || | fz|z}|r || fz|fz}|||fS)Nr+)rrtrr@rArr)rrr) rhrr&catrrjfliprr)r;rrrtr=rall_hidden_statesall_attentions batch_size num_keypointsr7attention_outputself_attentionsintermediate_states output_statesself_attention_descriptorsself_attention_hidden_statesrrcross_attention_outputcross_attentionscross_intermediate_statescross_output_statescross_attention_hidden_statess r-rKz!LightGlueTransformerLayer.forwards #7BD0d  1[N B 4?4E4E1 M>-1,?,?  ))/ -@- )/ $ii6F(GRP &9: %0=%@" ,?+O ( ' . .r1m^ T T!W WZ ? )  " "2q!Q > C CA F N Nz[\^_an o 483G3G &"7#9/ 4H4 0 0 %*II/IKa.bhj$k!"nn-FG03FF -FH[,\ )!-55j-Q_`bc./&&z=.QST0 0  +.@@DTCVVN-~==r,)FF) r"r#r$rrr5r&rMrNr*rKrOrPs@r-rrs..3.-2). H>\\H><<H> H> #Tk H>  $; H> u||U5<<047u||9Lt9SS TH>r,r similarity matchability0 matchability1c|j\}}}tjj|tjj|j ddz}tjj |d}tjj |j ddj dj dd}|j||dz|dzfd} ||z|z| ddd|d|f<tjj|jd | dddddf<tjj|jd | dddddf<| S)z;create the log assignment matrix from logits and similarityrrr@rRrN) rhrr~ logsigmoidr} log_softmaxrnew_fullsqueeze) rrrrnum_keypoints_0num_keypoints_1 certaintiesscores0scores1scoress r-sigmoid_log_double_softmaxrZsS4>3C3C0J--**=9BMMc|j\}}}|j|}|tj|j|j dzz }|j |dzd||}|dddf}|dddf}||jddz} ||j |dzd|}|dddfjd} |dddfjdjdd} | | z}| j|dk(tj| jj} |j|} | j |dzd|d} | dddf} | dddf}t| | |}|S)Ndeviceg?rrrr@rR)rhrr&tensorr7rrjr}r^ masked_fillfinfor\minrr)r;rrrrr7 m_descriptorsm_descriptors0m_descriptors1rmask0mask1rmatchability_0matchability_1rs r-rKz%LightGlueMatchAssignmentLayer.forwardqs4?4E4E1 M>--k: % T5H5HQ^QeQe(fjn(nn %--jAoq-Q_` &q!t,&q!t,#n&>&>r2&FF  << aMBDAJ((,EAJ((,66r2>E5=D#// 5;;zGWGW;X;\;\]J((5 #++J!OQ qQ %ad+%ad+,JW r,c|j|}tjj|j d}|S)z0Get matchability of descriptors as a probabilityr@)rrr~sigmoidr)r;rrs r-get_matchabilityz.LightGlueMatchAssignmentLayer.get_matchabilitys7((5 }},,\:BB2F r,) r"r#r$rr5r&rMrKrrOrPs@r-rrisRII5<<u|| 4ELLU\\r,rc\eZdZdeffd Zdej dej fdZxZS)LightGlueTokenConfidenceLayerr0clt|tj|jd|_y)Nr)r4r5rr6r7tokenr:s r-r5z&LightGlueTokenConfidenceLayer.__init__s& YYv44a8 r,rr>c|j|j}tjj |j d}|S)Nr@)rdetachrr~rr)r;rrs r-rKz%LightGlueTokenConfidenceLayer.forwards= ;--/0 %%e,44R8 r,rrPs@r-rrs*99 5<<ELLr,rc4eZdZUdZeed<dZdZdZdZ dZ dZ y) LightGluePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r0 lightglue pixel_values)imageFTN) r"r#r$r%rr(base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpar+r,r-rrs1 #$O!&+#Nr,rr thresholdc6|j\}}}|ddddddfjd}|ddddddfjd}|j}|j}tj|jd|j d}tj|jd|j d} ||j d|k(} | |j d|k(} |jj}|jd} tj| || } tj| | j d|| }| | |kDz}| |j d|z}tj||d}tj||d}tj||gjddj|dzd}tj| |gjddj|dzd}||fS)z1obtain matches from a score matrix [Bx M+1 x N+1]Nr@rrrr)rhmaxindicesr&arangergathervaluesexp new_tensorwhererSr}rj)rrr_max0max1matches0matches1indices0indices1mutual0mutual1zeromatching_scores0matching_scores1valid0valid1rrs r-get_matches_from_scoresrs||J1 !SbS#2#+  " "1 %D !SbS#2#+  " "1 %D||H||H||HNN1-hooFtLH||HNN1-hooFtLH(//!X66G(//!X66G ;;?? D ??1 D{{7D$7{{7,<,C,CAx,PRVW (94 5F v}}Q1 1F{{68R0H{{68R0Hkk8X./99!Q?GG UVXZ[Gkk#35E"FGQQRSUVW__`jmn`nprsO O ##r,rheightwidthctj||g|j|jd}|dz }|j dj dz }||ddddfz |dz }|S)a Normalize keypoints locations based on image image_shape Args: keypoints (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`): Keypoints locations in (x, y) format. height (`int`): Image height. width (`int`): Image width. Returns: Normalized keypoints locations of shape (`torch.Tensor` of shape `(batch_size, num_keypoints, 2)`). rr\Nrr@.).NN)r&rrr\rr )rrrsizeshiftscales r-normalize_keypointsr$sp << 0@0@  XY] ^D 1HE HHRL  ! #EU3a<00E/4JJI r,zV LightGlue model taking images as inputs and outputting the matching of them. ceZdZdZdeffd ZdedefdZ d#de jde jd e d zde e je e je jfffd Z d e jded e jde jde jf dZd$dZde jde jdede jfdZde jde jd e jde jde jd e jdefdZdZde jde jde jde jde e je jff dZ d%de jde jdeded e jd zde d zd e d zde e je je je e ffdZee d%d e j.d!e j0d zde d zd e d zde ezf d"ZxZS)&LightGlueForKeypointMatchingar LightGlue is a model matching keypoints in images by leveraging detections from a keypoint detector such as SuperPoint. It is based on the SuperGlue architecture and is designed to be lightweight and efficient. It consists of : 1. Keypoint Encoder 2. A Graph Neural Network with self and cross attention layers 3. Matching Assignment layers The correspondence ids use -1 to indicate non-matching points. Philipp Lindenberger, Paul-Edouard Sarlin and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV 2023. https://huggingface.co/papers/2306.13643 r0c ,t||tj|j|j |_|jj|_|j|_ |j|_ |j|_ |j|_ |j|_|j|jk7r2tj |j|jd|_ntj$|_t'||_tj*t-|jDcgc]}t/||c}|_tj*t-|jDcgc] }t3|c}|_tj*t-|jdz Dcgc] }t7|c}|_|j;ycc}wcc}wcc}w)N)trust_remote_codeTr2)rr)r4r5r from_configkeypoint_detector_configr(keypoint_detectordescriptor_decoder_dim keypoint_detector_descriptor_dimr7num_hidden_layers num_layersfilter_thresholddepth_confidencewidth_confidencerr6input_projectionIdentityr/positional_encoder ModuleListrangertransformer_layersrmatch_assignment_layersrtoken_confidence post_init)r;r0irr<s r-r5z%LightGlueForKeypointMatching.__init__s  !>!J!J  + +v?W?W" 170O0O0f0f-$33 22 & 7 7 & 7 7 & 7 7   $"G"G G$&IId.S.SUYUhUhos$tD !$&KKMD !"cddtjd|z|jz zz}tj|ddS)z-scaled confidence threshold for a given layerg?g?grr)npr r/clip)r;r=rs r-_get_confidence_thresholdz6LightGlueForKeypointMatching._get_confidence_thresholds;#tk'9DOO'K LLL wwy!Q''r,rrr=Nc|jj}|j|}|j||}||fS)Nr=)rrr3r5)r;rrr=projected_descriptorskeypoint_encoding_outputs r-_keypoint_processingz1LightGlueForKeypointMatching._keypoint_processingsO"((*557 $ 5 5k B#'#:#:9[o#:#p $&>>>r,keypoint_confidencesr num_pointsc||j\}}||jdz kru|j|dk(d}|j|dzd}|j |}d||kj j d|z z }||jkD} | Stj|tj} | S)zRevaluate whether we should stop inference based on the confidence of the keypointsrrrr@g?rAr[) rhr/rrjrAr]sumr1r&onesrN) r;rGr=rrHrrrratio_confidentearly_stopped_pairss r-_get_early_stopped_image_pairsz;LightGlueForKeypointMatching._get_early_stopped_image_pairs%s A 1, ,$8#C#CDAIq#Q #7#?#? aQS#T 66{CI!%9I%E$L$L$N$R$RWX$R$Y\f$ffO"1D4I4I"I  #"#(**Zuzz"J ""r,c| ||}||}|j|||}t||j\}}||fSr)r9rr0)r;rrr= early_stopsrrrs r-_get_keypoint_matchingz3LightGlueForKeypointMatching._get_keypoint_matching8sW  "%k2K $D:--k:;M#:64CXCX#Y ''r, confidencesrc\|d|jz kD}||||j|kz}|S)z#mask points which should be removedr)r2rA)r;rRrr=keeps r-_get_pruning_maskz.LightGlueForKeypointMatching._get_pruning_mask@s<T2223  " K4#A#A+#NN ND r,r prune_outputc|j\}} } |j|j|} |j|| |j |dk(t j dfd||d|d|fD\} } } }}t|D]}||||fxxdz cc<d| | | |fD\} } } }| | f}t|dd}| ||||fS) z For a given layer, prune keypoints based on the confidence of the keypoints and the matchability of the descriptors. rFc3nK|]&}t|Dcgc] \}}|| c}}(ycc}}wwr)zip).0rtrpruned_keypoints_masks r- zJLightGlueForKeypointMatching._do_layer_keypoint_pruning..[s9c %(0E$F GDQtW Gc Gs 5/5rc36K|]}t|dyw)T) batch_firstNr)rZ pruned_tensors r-r]zJLightGlueForKeypointMatching._do_layer_keypoint_pruning..cs$S  D 9 9S sTr@r_ padding_value) rhr9rrUrr&rr7r)r;rrrrrVrGr=rrdescriptors_matchabilitypruned_descriptorspruned_keypoints_0pruned_keypoints_1 pruned_maskpruned_indicesr<pruned_keypointsr\s @r-_do_layer_keypoint_pruningz7LightGlueForKeypointMatching._do_layer_keypoint_pruningGs+',, Aq#'#?#? #L#]#]^i#j $ 6 67KMegr s 5 A A$!)U\\Z_M` ac & ! ilDY[bcc _.0BKQ_z" 4A N1-- .! 3 . 4S "46HJ\^i!jS O.0BK/0BC%n$VXY!#3^[R^^^r,ctjtjjd}|}|d||fD\}}d||fD\}}fd||||fD\}}}}||||fS)Nrc38K|]}t|ddyw)Tr@raNrrZrs r-r]zMLightGlueForKeypointMatching._concat_early_stopped_outputs..ys$3  T D D3 c38K|]}t|ddyw)TrraNrrms r-r]zMLightGlueForKeypointMatching._concat_early_stopped_outputs..}s$>  T C C> rnc3(K|] }| ywrr+)rZrearly_stops_indicess r-r]zMLightGlueForKeypointMatching._concat_early_stopped_outputs..s!g  & 'g s)r&rSrrh)r;rqfinal_pruned_keypoints_indices!final_pruned_keypoints_iterationsrrids order_indicess ` r-_concat_early_stopped_outputsz:LightGlueForKeypointMatching._concat_early_stopped_outputsls$kk*=>ll.44Q78+C0 1-@3 "$BC3 //> *,MN> ::g .1 g c"@Bc./PRY[jjjr,rrrc |j\}fd|||fD\}}}|dddf}|dddf}|dddf}|dddf} |dddf} |dddf} tjdzd|fd|j|j} tj dzd|f|j|j} t dzD]}tj||dk(d||jd||jd| |d||f<tj| |dk(d||jd| |jd| |d||f<| || |d||f<| || |d||f<| | fS)Nc3JK|]}|jdzddyw)rr@N)rj)rZrrs r-r]zJLightGlueForKeypointMatching._do_final_keypoint_pruning..s'- 7=FNN:?Ar 2- s #rrrr@r )r) rhr&fullrr\zerosr7r r clamp)r;rrrrrrrrrrr_matches_matching_scoresr<rs @r-_do_final_keypoint_pruningz7LightGlueForKeypointMatching._do_final_keypoint_prunings A- BI7TcAd- )/1a4=1a4=1a4=1a4=*1a40*1a40::zQ=A2gnndkdqdqr ;; 1_a /oNcNc zQ' FA*/++ r!2x{'9'9!Xa[=N=NST=N=U'V+HQ8A;& '+0++ r!2x{'9'9!Xa[=N=NST=N=U'V+HQ8A;& '3C12E Q8A;. /2B12E Q8A;. / F)))r,rrrc & (|rdnd}|rdnd} |jddk(rT|jdd} |j| dtj|j | |j | || fS|j } |j\} } }} tj |j| dd}|j| dz|d}||j| dz|nd}|j| dz||j}tj| dz| }t|||}|j||| \}}|d}|jdkD}|jdkD}g}g}g}g}g}tjd|| j| dzd}tj|}t!|j"D]3}|j%}||j'||}n&tj(| |d f|j }|j*|||||| }|\}}} |r||z}|r| | z} |r||j"dz kr+|j,||}!|j/|!||| }"n%tj(| tj0}"tj2|"r|"j5d(|(}#|j7|||( \}$}%|j9t;|#|j9t;|$|j9t;|%|r:|j9t;|(|j9t;|(||"}t=(fd||d|d||fD\}}&}'}}|&|'f}|rt=(fd||!fD\}}}!tj>|"rn$|s|jA|||||!|\}}}}}6|r4|r2|jC|||||\}}}}|jE||||\}}nE|j7|||j"dz \}}tj||j"z}|j| d|}||||| fS)Nr+rrr@r[rrArrCrR)rtr=r)rH)rPc3*K|] }| ywrr+rZrrPs r-r]zALightGlueForKeypointMatching._match_image_pair..s"V" |,Vc3*K|] }| ywrr+rs r-r]zALightGlueForKeypointMatching._match_image_pair..!s"l &#K<0lr)#rhrr&r new_zerosrrJrjr-rr$rFr1r2ri ones_liker7r/r!get_extended_attention_maskrKr8r:rNrNanyrCrQextendlistr*allrjrvr~))r;rrrrrrr=rrrhrrrinitial_num_keypointsnum_points_per_pair image_indicesrE do_early_stopdo_keypoint_pruningrqrrrrrspruned_keypoints_indicespruned_keypoints_iterationsr=rextended_attention_mask layer_outputr attentionrGrMearly_stopped_image_indicesearly_stopped_matchesearly_stopped_matching_scores keypoints_0 keypoint_1rPs) @r-_match_image_pairz.LightGlueForKeypointMatching._match_image_pairsg#7BD0d ??1  "OOCR(E""5"EII">##E*##E*!  !!2;/// A,a#ii Z(D!L%%j1n6KQO FJFVt||JN,AB\`!))*q.:OQUQvQvw  Z!^FC ' 65A 040I0I 9M1J1 - --Q/ --1 #33a7 )+&,.)#(<<3HQW#X#_#_`jmn`npr#s &+oo6N&O# 1R K%**,K*.*J*J4Q\*]'*/**j+b/5R[d[k[k*l'?422;?6%9"3 L5A 1K #$5 $E! !/)!;1!44+M4+@+@+Mk+Z(+/*M*M,k4L_+N+' +0**Zuzz*R'9901#6"G"G"JK2? 2L/KOKfKf#T;KLgLH)+H(..t4O/PQNN4(=#>?#**40M+NO*6==dC[\gCh>ij9@@FabmFnAop+>?R>R*S'PUV'2IaL)A,PTVc&dVQMKj$ "-j 9I*fkl!9 ; 4+lgc02MOc9901"33#!03,#d Y(@$HcQR h 022'25#  h *,MwXg(,'F'F.% ( $G_(,'B'B;PTVZVeVehiVi'j $G_050PSWSbSb0b -,M,U,U 0- )   -     r,rlabelsc 0d}| td||n|jj}||n|jj}|jdk7s|j ddk7r td|j \}}} } } |j|dz| | | }|j|} | dd\} }}}| j|dddj|} |j|dd|jj|}|j|dd}| j}|dddddddf| z|dddddddf<|dddddddf| z|dddddddf<|j||| | ||| \}}}}}t|||| |||| S) Nz9LightGlue is not trainable, no labels should be provided.rrzOInput must be a 5D tensor of shape (batch_size, 2, num_channels, height, width)r@r)rrr=)rrrrrrr r!) ValueErrorr0rr=ndimr!rhrjr+r_r-clonerr)r;rrrr=rwrrrchannelsrrkeypoint_detectionsrrrabsolute_keypointsrrrr r!s r-rKz$LightGlueForKeypointMatching.forward^s  XY Y1B1N-TXT_T_TqTq$8$D $++JjJj     !\%6%6q%9Q%>no o1=1C1C. Ax#++JNHfeT "44\B*=bq*A' 1k4%%j!R;>>|L !))*aT=b=bcffgst ||J2.&__.);Aq!QJ)G%)O1aA:&);Aq!QJ)G&)P1aA:&EIE[E[    /!5F\F B% /+'!  r,rLr)NNN)r"r#r$r%rr5rr]rAr&rMrNr*rFrNrQrUrjrvr~rrrr' LongTensorrrKrOrPs@r-r&r&s @(S(U( gl? <<?49LL?X\_cXc? u||U5<<#=>> ??#$)LL#?B#JO,,#didpdp# #&(U\\5<<^afkfrfr#_\\#_<<#_ll #_  #_ ll #_$ll#_#_Jk@#*#*#* #* || #* u||U\\) * #*V%))-,0k <<k \\k  k  k llT! k  $;k #Tkk  u||U\\5<<E Fk Z+/)-,0 4 ''4   4'4  $; 4 #Tk 4  0 04 4 r,r&)r)r)5collections.abcr dataclassesrnumpyr?r&rtorch.nn.utils.rnnr activationsr integrationsr modeling_flash_attention_utilsr modeling_utilsr r processing_utilsrutilsrrr utils.genericrauto.modeling_autorconfiguration_lightgluerrModuler/rYrerMrror]rrrrrrrrr*rr$r&__all__r+r,r-rsk(%! +!/BF&DD->4  7k 7  7F"<8 UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%'(%2)*D)D)+D)N299"P> P>f   -2\\ JO,,  \\ &BII&R BII    $ELL$U$uU\\[`[g[gMgGh$@5<<SU\\, k #;k  k \ &'E Fr,