qi dZddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZmZmZmZd dlmZej8eZeedGddeZeedGddeZ eedGddeZ!eedGddeZ"dZ#dZ$GddejJZ&Gd d!ejJZ'Gd"d#ejJZ(dId$ejRd%e*d&e+d'ejRfd(Z,Gd)d*ejJZ-Gd+d,ejJZ.Gd-d.ejJZ/Gd/d0ejJZ0Gd1d2ejJZ1Gd3d4ejJZ2Gd5d6ejJZ3Gd7d8eZ4Gd9d:ejJZ5eGd;de6Z7ed?Gd@dAe6Z8edBGdCdDe6Z9edEGdFdGee6Z:gdHZ;y)JzPyTorch Swin Transformer model.N) dataclass)nn)initialization)ACT2FN) BackboneMixin)GradientCheckpointingLayer)BackboneOutput)PreTrainedModel) ModelOutputauto_docstringlogging torch_int) SwinConfigzN Swin encoder's outputs, with potential hidden states and attentions. ) custom_introceZdZUdZdZej dzed<dZe ej dfdzed<dZ e ej dfdzed<dZ e ej dfdzed<y)SwinEncoderOutputa reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nlast_hidden_state. hidden_states attentionsreshaped_hidden_states) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rtuplerrX/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/swin/modeling_swin.pyrr's}37u((4/6:>M5**C/047>7;Je'',-4;CGE%"3"3S"89D@Gr"rzV Swin model's outputs that also contains a pooling of the last hidden states. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<dZ e ej dfdzed<y) SwinModelOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, *optional*, returned when `add_pooling_layer=True` is passed): Average pooling of the last layer hidden-state. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr pooler_output.rrr) rrrrrrrrr&rr rrr!r"r#r%r%=s 37u((4/6.2M5$$t+2:>M5**C/047>7;Je'',-4;CGE%"3"3S"89D@Gr"r%z* Swin masked image model outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<dZ e ej dfdzed<y) SwinMaskedImageModelingOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `bool_masked_pos` is provided): Masked image modeling (MLM) loss. reconstruction (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Reconstructed pixel values. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nlossreconstruction.rrr) rrrrr)rrrr*rr rrr!r"r#r(r(Vs &*D%  d ")/3NE%%,3:>M5**C/047>7;Je'',-4;CGE%"3"3S"89D@Gr"r(z0 Swin outputs for image classification. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<dZ e ej dfdzed<y) SwinImageClassifierOutputa7 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr)logits.rrr) rrrrr)rrrr-rr rrr!r"r#r,r,qs &*D%  d ")'+FE   $+:>M5**C/047>7;Je'',-4;CGE%"3"3S"89D@Gr"r,c|j\}}}}|j|||z|||z||}|jddddddjjd|||}|S)z2 Partitions the given input into windows. rrrshapeviewpermute contiguous) input_feature window_size batch_sizeheightwidth num_channelswindowss r#window_partitionr?s}/<.A.A+J|!&&Fk);8Lk[gM##Aq!Q15@@BGGKYdfrsG Nr"c|jd}|jd||z||z|||}|jddddddjjd|||}|S)z? Merges windows to produce higher resolution features. r2rrrr/r0r1r3)r>r9r;r<r=s r#window_reverserAsn==$Lll2v4e{6JKYdfrsGooaAq!Q/::<AA"feUabG Nr"c eZdZdZd fd Zdej dededej fdZ dd ejdzd ejdzd e de ej fd Z xZS)SwinEmbeddingszW Construct the patch and position embeddings. Optionally, also the mask token. c~t|t||_|jj}|jj |_|r4tjtjdd|jnd|_ |jr=tjtjd|dz|j|_nd|_tj|j|_tj"|j$|_|j(|_||_y)Nr)super__init__SwinPatchEmbeddingspatch_embeddings num_patches grid_size patch_gridr Parameterrzeros embed_dim mask_tokenuse_absolute_embeddingsposition_embeddings LayerNormnormDropouthidden_dropout_probdropout patch_sizeconfig)selfrXuse_mask_tokenrI __class__s r#rFzSwinEmbeddings.__init__s  3F ;++77 //99O]",,u{{1a9I9I'JKcg  ) )')||EKK;QR?TZTdTd4e'fD $'+D $LL!1!12 zz&"<"<=  ++ r" embeddingsr;r<returnc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr2?rrr/bicubicF)sizemode align_cornersdim)r4rQrjit is_tracingrWrreshaper6r functional interpolater5cat) rYr\r;r<rI num_positionsclass_pos_embedpatch_pos_embedre new_height new_widthsqrt_num_positionss r#interpolate_pos_encodingz'SwinEmbeddings.interpolate_pos_encodings`!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr"N pixel_valuesbool_masked_posrrc|j\}}}}|j|\}} |j|}|j\} } }|K|jj | | d} |j dj| } |d| z z| | zz}|j(|r||j|||z}n||jz}|j|}|| fS)Nr2g?) r4rHrSrarOexpand unsqueezetype_asrQrrrV)rYrsrtrr_r=r;r<r\output_dimensionsr:seq_len mask_tokensmasks r#forwardzSwinEmbeddings.forwards *6););&<(,(=(=l(K% %YYz* !+!2 GQ  &//00WbIK",,R088ED#sTz2[45GGJ  # # /''$*G*G TZ\a*bb '$*B*BB \\*- ,,,r")FNF)rrrrrFrTensorintrrr BoolTensorboolr r~ __classcell__r[s@r#rCrCs&&D5<<&D&DUX&D]b]i]i&DV48). -''$.-))D0-#' - u||  -r"rCcveZdZdZfdZdZdejdzdeejee ffdZ xZ S)rGz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ |d|dz|d|dzf|_ tj|||||_y)Nrr) kernel_sizestride)rErF image_sizerWr=rN isinstance collectionsabcIterablerIrJrConv2d projection)rYrXrrWr= hidden_sizerIr[s r#rFzSwinPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79I9Ik #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&$Q-:a=8*Q-:VW=:XY))L+:^hir"cn||jdzdk7rDd|jd||jdzz f}tjj||}||jdzdk7rFddd|jd||jdzz f}tjj||}|S)Nrr)rWrripad)rYrsr;r< pad_valuess r# maybe_padzSwinPatchEmbeddings.maybe_pads 4??1% % *T__Q/%$//!:L2LLMJ==,,\:FL DOOA& &! +Q4??1#5QRAS8S#STJ==,,\:FLr"rsNr]c|j\}}}}|j|||}|j|}|j\}}}}||f}|jdj dd}||fS)Nr/r)r4rrflatten transpose)rYrsryr=r;r<r\rzs r#r~zSwinPatchEmbeddings.forwards})5););&<~~lFEB __\2 (..1fe#UO''*44Q: ,,,r") rrrrrFrrrr rrr~rrs@r#rGrGsF j -E$5$5$< -u||UZ[^U_G_A` -r"rGc eZdZdZej fdeededejddffd Z dZ d e jd eeefde jfd Z xZS) SwinPatchMerginga' Patch Merging Layer. Args: input_resolution (`tuple[int]`): Resolution of input feature. dim (`int`): Number of input channels. norm_layer (`nn.Module`, *optional*, defaults to `nn.LayerNorm`): Normalization layer class. input_resolutionre norm_layerr]Nct|||_||_t j d|zd|zd|_|d|z|_y)Nr0r/Fbias)rErFrrerLinear reductionrS)rYrrerr[s r#rFzSwinPatchMerging.__init__6sI  01s7AG%@q3w' r"c|dzdk(xs|dzdk(}|r.ddd|dzd|dzf}tjj||}|S)Nr/rr)rrir)rYr8r;r< should_padrs r#rzSwinPatchMerging.maybe_pad=sUqjAo:519> Q519a!