qixbdZddlZddlmZddlmZddlmZddlZ ddl Z ddl m Z ddl m Z dd lmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZmZmZddlm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ejRe*Z+eedGddeZ,eedGddeZ-dZ.Gdde j^Z0Gdde j^Z1 dDd e j^d!e jdd"e jdd#e jdd$e jddzd%e3dzd&e3d'eefd(Z4Gd)d*e j^Z5Gd+d,e j^Z6Gd-d.e j^Z7Gd/d0e j^Z8Gd1d2e j^Z9Gd3d4eZ:Gd5d6e j^Z;eGd7d8eZed=Gd>d?e<Z?ed@GdAdBe<Z@gdCZAy)Ez,PyTorch VideoMAE (masked autoencoder) model.N)Callable)deepcopy) dataclass)nn)MSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)can_return_tuplemerge_with_config_defaults)capture_outputs)VideoMAEConfigz[ Class for VideoMAEDecoder's outputs, with potential hidden states and attentions. ) custom_introceZdZUdZdZej dzed<dZe ej dzed<dZ e ej dzed<y)VideoMAEDecoderOutputz logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlogits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rtupler `/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/videomae/modeling_videomae.pyrr)sR (,FE   $+59M5**+d2926Je''(4/6r*rzb Class for VideoMAEForPreTraining's outputs, with potential hidden states and attentions. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dzed<dZ e ej dzed<y)VideoMAEForPreTrainingOutputz loss (`torch.FloatTensor` of shape `(1,)`): Pixel reconstruction loss. logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlossrrr ) r!r"r#r$r.r%r&r'rrr(r r)r*r+r-r-:sg&*D%  d ")'+FE   $+59M5**+d2926Je''(4/6r*r-chfd}tjt|Dcgc] }|| c}}tj|dddddf|dddddf<tj|dddddf|dddddf<t j |jdScc}w)z Sinusoid position encoding tablec tDcgc]$}|tjdd|dzzz z &c}Scc}w)Ni')rangenppower)positionhid_jd_hids r+get_position_angle_vecz;get_sinusoid_encoding_table..get_position_angle_vecTs;RWX]R^_288E1 +;e+CDD___s);Nrr1r)r3arrayr2sincosr%r& unsqueeze) n_positionr7r8pos_isinusoid_tables ` r+get_sinusoid_encoding_tabler@Ps`XX%PZJ[\5e<\]N ff^Aqt!tG%<=N1add7 ff^Aqt!tG%<=N1add7   ^ , 6 6q 99 ]sB/c(eZdZdZfdZdZxZS)VideoMAEEmbeddingsz7 Construct the patch and position embeddings. ct|t||_|jj|_t |j|j |_||_yN) super__init__VideoMAEPatchEmbeddingspatch_embeddings num_patchesr@ hidden_sizeposition_embeddingsconfigselfrL __class__s r+rFzVideoMAEEmbeddings.__init__dsR  7 ?00<<#>t?O?OQWQcQc#d  r*c |j|}||jjj|j |j dz}|)|j \}}}||}|j|d|}|S)NTdevicecopy)rHrKdetachtype_astorRshapereshape)rN pixel_valuesbool_masked_pos embeddings batch_size_ num_channelss r+forwardzVideoMAEEmbeddings.forwardms**<8  $":":"A"A"C"K"KJ"W"Z"Z$$4#[#   &*4*:*: 'J<#_$45J#++JLIJr*r!r"r#r$rFr` __classcell__rOs@r+rBrB^s r*rBc(eZdZdZfdZdZxZS)rGaw Video to Patch Embedding. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size) * (height // patch_size) * (width // patch_size). c t ||j}|j}|j}|j }|j }|j}t|tjjr|n||f}t|tjjr|n||f}||_||_t||_|d|dz|d|dzz||jzz}||_||_ tj|||j|d|df|j|d|df|_y)Nrr) in_channels out_channels kernel_sizestride)rErF image_size patch_sizer_rJ num_frames tubelet_size isinstance collectionsabcIterableintrIrConv3d projection) rNrLrjrkr_rJrlrmrIrOs r+rFz VideoMAEPatchEmbeddings.__init__s> && && ** (( && ** #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq $$ - ]jm + 1 A0N OS]aeararSr s )&))$$**JqM:a=I%%z!}jmD  r*c |j\}}}}}||jk7r td||jdk7s||jdk7r2td|d|d|jdd|jdd |j dddd d }|j |j djdd}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).r1r)rXr_ ValueErrorrjpermutertflatten transpose)rNrZr]rlr_heightwidthr\s r+r`zVideoMAEPatchEmbeddings.forwards>J>P>P; J fe 4,, ,w  T__Q' '5DOOA4F+F$VHAeW4KDOO\]L^K__`aeapapqras`ttvw $++Aq!Q: __\2::1=GG1M r*rarcs@r+rGrGs 6 r*rGmodulequerykeyvalueattention_maskscalingdropoutkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)NrTr1rdim)ptrainingr) sizer%matmulr{r functionalsoftmaxrr contiguous) r~rrrrrrr attn_weights attn_outputs r+eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r*ceZdZdeddffd Zddej dzdeej ej ffdZxZ S)VideoMAESelfAttentionrLreturnNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |j|_ |jdz|_ d|_ tj|j|jd|_tj|j|jd|_tj|j|jd|_|j&rotj(t+j,|j|_tj(t+j,|j|_yd|_d|_y) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .rFbias)rErFrJnum_attention_headshasattrrxrLrrattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearrrrqkv_bias Parameterr%zerosq_biasv_biasrMs r+rFzVideoMAESelfAttention.__init__s     : : :a ?PVXhHi"6#5#5"67334A7  #)#=#= #&v'9'9FS"S%--.EF o--r*rD) r!r"r#rrFr%Tensorr(r`rbrcs@r+rrsG~$4.U\\D%8.E%,,X]XdXdJdDe.r*rcxeZdZdZdeffd ZdejdejdejfdZxZ S)VideoMAESelfOutputz The residual connection is defined in VideoMAELayer instead of here (as is the case with other models), due to the layernorm applied before each block. rLct|tj|j|j|_tj |j|_yrD) rErFrrrJdenseDropouthidden_dropout_probrrMs r+rFzVideoMAESelfOutput.__init__sB YYv1163E3EF zz&"<"<= r*r input_tensorrcJ|j|}|j|}|SrDrrrNrrs r+r`zVideoMAESelfOutput.forwards$ =1  ]3 r* r!r"r#r$rrFr%rr`rbrcs@r+rrs= >~> U\\RWR^R^r*rc\eZdZdeffd Zdej dej fdZxZS)VideoMAEAttentionrLcbt|t||_t ||_yrD)rErFr attentionroutputrMs r+rFzVideoMAEAttention.__init__!s& .v6(0 r*rrcR|j|\}}|j||}|SrD)rr)rNrself_attn_outputr^rs r+r`zVideoMAEAttention.forward&s,"nn];!-}= r* r!r"r#rrFr%rr`rbrcs@r+rr s*1~1 U\\ellr*rc\eZdZdeffd Zdej dej fdZxZS)VideoMAEIntermediaterLct|tj|j|j |_t|jtrt|j|_ y|j|_ yrD) rErFrrrJintermediate_sizerrn hidden_actstrr intermediate_act_fnrMs r+rFzVideoMAEIntermediate.__init__.s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r*rrcJ|j|}|j|}|SrD)rr)rNrs r+r`zVideoMAEIntermediate.forward6s& =1 00? r*rrcs@r+rr-s*9~9U\\ellr*rcteZdZdeffd Zdej dej dej fdZxZS)VideoMAEOutputrLct|tj|j|j |_tj|j|_ yrD) rErFrrrrJrrrrrMs r+rFzVideoMAEOutput.__init__>sB YYv779K9KL zz&"<"<= r*rrrcT|j|}|j|}||z}|SrDrrs r+r`zVideoMAEOutput.forwardCs. =1  ]3 % 4 r*rrcs@r+rr=s8>~> U\\RWR^R^r*rc`eZdZdZdeffd ZdejdejfdZxZ S) VideoMAELayerz?This corresponds to the Block class in the timm implementation.rLcrt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nreps)rErFchunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormrJlayer_norm_epslayernorm_beforelayernorm_afterrMs r+rFzVideoMAELayer.__init__Ns '-'E'E$*6208$V, " V-?-?VEZEZ [!||F,>,>FDYDYZr*rrc|j|}|j|}||z}|j|}|j|}|j ||}|SrD)rrrrr)rNrhidden_states_normattention_output layer_outputs r+r`zVideoMAELayer.forwardXsi!22=A>>*<=)=8 ++M: ((6 {{<? r*rrcs@r+rrKs/I[~[U\\ellr*rcHeZdZdeffd Zdej defdZxZ S)VideoMAEEncoderrLct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rErFrLr ModuleListr2num_hidden_layersrlayergradient_checkpointing)rNrLr^rOs r+rFzVideoMAEEncoder.__init__ksN  ]]5IaIaCb#caM&$9#cd &+#$dsA#rrcdt|jD] \}}||}t|S)Nlast_hidden_state) enumeraterr )rNri layer_modules r+r`zVideoMAEEncoder.forwardqs5(4 8OA|(7M 8??r*) r!r"r#rrFr%rr r`rbrcs@r+rrjs),~, @U\\@o@r*rcJeZdZUeed<dZdZdZdZddgZ dZ dZ dZ dZ eedZy ) VideoMAEPreTrainedModelrLvideomaerZvideoTrBr)rr N)r!r"r#rr'base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsr)r*r+rrxsO "$O&*#-?N"&&+r*rc eZdZfdZdZeede d dejdejdzde e d e fd ZxZS) VideoMAEModelct||||_t||_t ||_|jrd|_n0tj|j|j|_|jy)Nr)rErFrLrBr\rencoderuse_mean_pooling layernormrrrJr post_initrMs r+rFzVideoMAEModel.__init__si   ,V4&v.  " "!DN\\&*<*<&BWBWXDN r*c.|jjSrD)r\rH)rNs r+get_input_embeddingsz"VideoMAEModel.get_input_embeddingss///r*F)tie_last_hidden_statesNrZr[rrc |j||}|j|}|j}|j|j|}t |S)aB bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. If `None`, then all patches are considered. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> import torch >>> from transformers import VideoMAEVideoProcessor, VideoMAEModel >>> from huggingface_hub import hf_hub_download >>> # replace this with your own video file >>> video_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> video_processor = VideoMAEVideoProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base") >>> # prepare video for the model >>> inputs = video_processor(video_path, return_tensors="pt") >>> # forward pass >>> with torch.no_grad(): ... outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1568, 768] ```r)r\rrrr )rNrZr[rembedding_outputencoder_outputssequence_outputs r+r`zVideoMAEModel.forwardsST ??<I+/<<8H+I);; >> %"nn_=OAAr*rD)r!r"r#rFrrrrr%r& BoolTensorrrr r`rbrcs@r+r r s~ 0 E248.B''.B))D0.B+, .B  .B3 .Br*r cHeZdZdeffd Zdej defdZxZ S)VideoMAEDecoderrLct||j|jz|jdzz}t |}|j |_|j|_ |j|_ |j|_ tjt!|jDcgc] }t#|c}|_tj&|j |_|dkDr tj*|j |ntj,|_d|_||_ycc}w)Nr1rF)rErFr_rmrkrdecoder_hidden_sizerJdecoder_num_hidden_layersrdecoder_num_attention_headsrdecoder_intermediate_sizerrrr2rdecoder_layersrnormrIdentityheadrrL)rNrLdecoder_num_labelsdecoder_configr^rOs r+rFzVideoMAEDecoder.__init__s #0063F3FFIZIZ\]I]]!&)%+%?%?"+1+K+K(-3-O-O*+1+K+K( mm49&:Z:Z4[ \q]> * \ LL!;!;< I[^_I_BIIf002D Eegepeper ',#$  ]s.D=rreturn_token_numc|jD] }||} |dd| df}|j|}|j|}t|S)N)r)r!r"r$r)rNrr'rrs r+r`zVideoMAEDecoder.forwardsb // 8L(7M 8&a*:):);&;<  -0 =)$F33r*) r!r"r#rrFr%rrrr`rbrcs@r+rrs&%~%, 4U\\ 4S 4r*rzb The VideoMAE Model transformer with the decoder on top for self-supervised pre-training. c xeZdZfdZeedejdejde e de fdZ xZ S)VideoMAEForPreTrainingct||||_t||_t j |j|jd|_ t jtjdd|j|_ t|jjj |j|_t%||_|j)y)NFrr)rErFrLr rrrrJrencoder_to_decoderrr%r mask_tokenr@r\rIrKrdecoderrrMs r+rFzVideoMAEForPreTraining.__init__s   %f- "$))F,>,>@Z@Zaf"g,,u{{1a9S9S'TU#> MM $ $ 0 0&2L2L$  'v.  r*rZr[rrc *|j|fd|i|}|j}|j|}|j\}}}| t d|j j |ddj|} | jj|jd} | |j|d|} | |j|d|} tj|| z|j| zgd} |j| | jd} | j }d}tj"5|j$j&d k7r|}n|j}|j(}tj*t,j|| ddddddf}tj*t.j|| ddddddf}||z|z}|j\}}}}}|j$j0|j$j2}}|j$j4r|j7|||z||||z|||z|}|j9d dd d dddd j;}|j7|||z|z|z|z|z||z|z|}||j=ddz |j?dddjAdzz }|j7|||z|z|z|z|z||z|z|z}n|j$j&d k7r t d|j7|||z||||z|||z|}|j9d dd d dddd j;}|j7|||z|z|z|z|z||z|z|z}|j\}}}||j|d|}dddtC}||}tE|||jF|jHS#1swY?xYw)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> from transformers import AutoImageProcessor, VideoMAEForPreTraining >>> import numpy as np >>> import torch >>> num_frames = 16 >>> video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224))) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base") >>> pixel_values = image_processor(video, return_tensors="pt").pixel_values >>> num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2 >>> seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame >>> bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss = outputs.loss ```r[Nz!One must provided a boolean mask rTTrQrrr)rRdtyperrwr1r)rkeepdim)runbiasedr4gư>zQCan't unnormalize non-RGB images. Consider setting config.norm_pix_loss to False.r.rrr )%rrr,rXrxrKexpandrVrUrWrRrYr%catr-r.rno_gradrLr_r0 as_tensorrrrmrk norm_pix_lossrryrmeanvarsqrtrr-rr )rNrZr[routputsrr]r^r_expanded_position_embeddingspos_emb_visible pos_emb_maskx_fulldecoder_outputsrr.framesrRr0r<stdtimer|r}rmrk frames_norm videos_patchlabelsloss_fcts r+r`zVideoMAEForPreTraining.forwardsF$14==#i#ibh#i!3311/B'6&;&;# A|  "@A A'+'?'?'F'FzSUWY'Z'b'bco'p$'C'J'J'L'O'OWcWjWjqu'O'v$67GHPPQ[]_amn3ODLLZY[]ij Oo=tQ]?]^def26flFXFXYZF[1\ '' ]]_H Y{{''1,%&,,$**'<=@@V[@\]acgijlprv]vwoo&:;>>fTY>Z[_aeghjnpt[tu%+d2J P  P)r!r"r#rFrrr%r&rrrr-r`rbrcs@r+r*r*sb "L ''L ))L +, L & L L r*r*z VideoMAE Model transformer with a video classification head on top (a linear layer on top of the average pooled hidden states of all tokens) e.g. for ImageNet. c eZdZfdZee ddejdzdejdzdee de fdZ xZ S) VideoMAEForVideoClassificationct|||j|_t||_|j rt j|jnd|_ |jdkDr*t j|j|jnt j|_ |jy)Nr)rErF num_labelsr rrrrrJfc_normrr# classifierrrMs r+rFz'VideoMAEForVideoClassification.__init__s   ++%f-  1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import torch >>> from transformers import VideoMAEVideoProcessor, VideoMAEForVideoClassification >>> from huggingface_hub import hf_hub_download >>> # replace this with your own video file >>> video_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> video_processor = VideoMAEVideoProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> inputs = video_processor(video_path, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) eating spaghetti ```Nrrr6) rrrPr<rQ loss_functionrLr rr ) rNrZrJrr?rrrr.s r+r`z&VideoMAEForVideoClassification.forwardsR$14==#H#H!33 << #$))!,F\\&)F$QT*F(  %4%%ffdkkLVLD$!//))   r*)NN) r!r"r#rFrrr%rrrr r`rbrcs@r+rMrMsj -1&*; llT);  t#; +, ;  ; ; r*rM)r*r rrM)Nr)Br$collections.abcrorrSr dataclassesrnumpyr3r%rtorch.nnr activationsr modeling_layersr modeling_outputsr r modeling_utilsr rprocessing_utilsrutilsrrrrutils.constantsrr utils.genericrrutils.output_capturingrconfiguration_videomaer get_loggerr!loggerrr-r@ModulerBrGrfloatrrrrrrrrrr rr*rM__all__r)r*r+rgso3$! !9FF&MMJI52   H %  7K7  7  7; 7  7 :B2bii2x!% II% <<% % << % LL4' % T\ %%'(%89.BII9.z$   299  RYY .> @bii @o"DB+DBDBN"4bii"4J ` 4`  ` F K %<K  K \ sr*