qiddlZddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z dd lmZdd lmZdd lmZdd lmZmZmZmZmZmZdd lmZmZddlmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5e'jle7Z8GddejrZ:GddejrZ;e%GddeZ< dLdejrdejzdejzd ejzd!ejzdzd"e>d#e>fd$Z?Gd%d&ejrZ@Gd'd(ejrZAGd)d*eZBGd+d,ejrZCGd-d.e<ZDGd/d0ejrZEGd1d2ejrZFGd3d4ejrZGGd5d6ejrZHGd7d8ejrZIGd9d:eZJGd;de<ZLee%d?@GdAdBe#ZMe%dC@GdDdEe<ZNee%GdFdGeZOe%dH@GdIdJe1D!--1"$,,u{{1d>P>PRVR`R`/a"b embeddingsheightwidthreturnc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r$Ng?rrr bicubicF)sizemode align_cornersdim)shaper>r7jit is_tracingr5rreshapepermuter functional interpolateviewcat) r@rDrErFr<r=class_pos_embedpatch_pos_embedrO new_height new_widthsqrt_num_positionss rBinterpolate_pos_encodingz:InstructBlipVideoVisionEmbeddings.interpolate_pos_encodingOs`!&&q)A- //55a81< yy##%+*F6UZ?** *11!RaR%811!QR%8r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCrC pixel_valuesr^c|j\}}}}|jjj}|j|j |}|j dj dd}|jj|ddj |} tj| |gd} |r|j| ||} n |j} | | ddd| jdddfj |z} | S)Ndtyper r$rIrN)rPr;weightrbtoflatten transposer9expandr7rXr^r>rK) r@r_r^ batch_size_rErF target_dtype patch_embeds class_embedsrDr>s rBforwardz)InstructBlipVideoVisionEmbeddings.forwardws'3'9'9$ Avu++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY l;C #!%!>!>z6SX!Y !%!8!8 "4Q8L*//!:L8La5O"P"S"ST`"aa rCF) __name__ __module__ __qualname__r'r1r7Tensorintr^ FloatTensorboolrm __classcell__rAs@rBr)r)<smc<c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn rCr)c2eZdZdZfdZ ddZxZS)"InstructBlipVideoQFormerEmbeddingsz;Construct the embeddings from word and position embeddings.c,t|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|j!dt#j$|jj'dd||_y)N) padding_idxeps position_idsr$rIF) persistent)r0r1r Embedding vocab_sizer2 pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_eps layernormDropouthidden_dropout_probdropoutregister_bufferr7arangergr*r?s rBr1z+InstructBlipVideoQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej   rCc||jd}nd}|&|jdd|||zfj}|\|j|}|j |j |j }||z}|tj||fd}n|}|j |jjj}|j|}|j|}|S)Nr$rrN) rKr~clonerrrddevicer7rXrrcrbr)r@ input_idsr~ query_embedspast_key_values_length seq_lengthrDrs rBrmz*InstructBlipVideoQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J"&":":>#8#8#>#>? ^^J/ \\*- rCNNNr)rorprq__doc__r1rmrvrws@rBryrysE " rCryczeZdZUeed<dZdZdZdZdZ dZ dZ dZ gdZ ejfdZxZS) InstructBlipVideoPreTrainedModelr*blip)videotextT)ryInstructBlipVideoAttention*InstructBlipVideoQFormerMultiHeadAttention"InstructBlipVideoQFormerSelfOutputc:t|||jj}t |t rEt j|jd|t j|jd|yt |ttfr t j|jyt |trZt j|j t#j$|j j&dj)dyy)zInitialize the weights)meanstdrIrN)r0 _init_weightsr*initializer_range isinstancer)init trunc_normal_r>r9)InstructBlipVideoForConditionalGenerationInstructBlipVideoModelzeros_ query_tokensrycopy_r~r7rrPrg)r@modulefactorrAs rBrz.InstructBlipVideoPreTrainedModel._init_weightss f%.. f? @   v88s O   v55CV L !JLb c d KK++ ,  B C JJv**ELL9L9L9R9RSU9V,W,^,^_f,g hDrC)rorprqr%__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesr7no_gradrrvrws@rBrrs^ ##(&*#"&N!U]]_ i irCrrquerykeyvalueattention_maskscalingrc ptj||jdd|z}|||z}tjj |d}tjj |||j}tj||} | jddj} | |fS)NrIrN)ptrainingr$r ) r7matmulrfrrUsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs rBeager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rCceZdZdZfdZdej dedefdZdej de ej ej d ze ej d zffd Z xZ S) rz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ d|_ |j|_ tj|jd|jzd|_|jretj t#j$|j}tj t#j$|j}nd}d}|Qt#j&|t#j(|d|f}tj ||j_tj|j|j|_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr)bias) requires_grad)r0r1r*r2r3num_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasr6r7zerosrX zeros_liker projection)r@r*q_biasv_biasrrAs rBr1z#InstructBlipVideoAttention.__init__su  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCrCtensorseq_lenbszc|j|||j|jjddj S)Nr$r )rWrrrfr)r@rrrs rB_shapez!InstructBlipVideoAttention._shapes7{{3GQQRSUVWbbddrC hidden_statesrGNc |j\}}}|j|}|j||d|j||jzj ddddd}|d|d|d} }}t j |jjt} | |||| fd|jsdn |j|jd|\} } | j||d j} |j| } | | fS) z#Input shape: Batch x Time x Channelrr rr$Nr)rrrrI)rKrrSrrTr get_interfacer*_attn_implementationrrrrrr) r@rrrtgt_lenr3 mixed_qkv query_states key_states value_statesattention_interfacerrs rBrmz"InstructBlipVideoAttention.forwards##0"4"4"6WiHH]+ %%c7At~~yTXTbTbGbckk q!Q 2;1y|YWX\,j (?(M(M KK , ,.E) %8     %  #}}C$2H2HJJ %  % ! \"))#w;FFH ook2 L((rC) rorprqrr1r7rrrsrtuplermrvrws@rBrrspGD>eU\\eCece")||") u||U\\D0% 2E2LL M ")rCrcVeZdZfdZdej dej fdZxZS)InstructBlipVideoMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) r0r1r*r hidden_act activation_fnrrr2intermediate_sizefc1fc2r?s rBr1zInstructBlipVideoMLP.__init__=sd  #F$5$5699V//1I1IJ99V55v7I7IJrCrrGcl|j|}|j|}|j|}|Sr)rrrr@rs rBrmzInstructBlipVideoMLP.forwardDs4/ **=9 / rCrorprqr1r7rrrmrvrws@rBrr<s$KU\\ellrCrcpeZdZdeffd Zedejdee dejfdZ xZ S)InstructBlipVideoEncoderLayerr*cDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNr|) r0r1r2r3r self_attnrrr layer_norm1rmlp layer_norm2r?s rBr1z&InstructBlipVideoEncoderLayer.__init__Lsm ++3F;<<F> '   q&0  ((7 / %0 rC) rorprqr%r1rr7rrrrrtrmrvrws@rBrrKsRS6S||+,    rCrcNeZdZdZdeffd Zedeede e zfdZ xZ S)InstructBlipVideoEncodera" Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipVideoEncoderLayer`]. Args: config (`InstructBlipVideoConfig`): The corresponding vision configuration for the `InstructBlipVideoEncoder`. r*ct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wNF) r0r1r*r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r@r*rirAs rBr1z!InstructBlipVideoEncoder.__init__usQ  mmTYZ`ZrZrTs$tq%B6%J$tu &+#%usA#rrGc R|}|jD] }||fi|} t|S)Nlast_hidden_state)r r )r@ inputs_embedsrr encoder_layers rBrmz InstructBlipVideoEncoder.forward{s> & ![[ M)M  ??rC) rorprqrr%r1rrrrr rmrvrws@rBrrksL,6,  @+, @  @ @rCrc eZdZUdZdZeed<eedZ deffd Z e e de ddejdzd ed eed eezfd Zd ZxZS)InstructBlipVideoVisionModelr_rr*)r attentionsct||||_|j}t ||_t ||_tj||j|_ |jyr) r0r1r*r2r)rDrencoderrrrpost_layernorm post_init)r@r*r3rAs rBr1z%InstructBlipVideoVisionModel.__init__s]   && ;FC/7  ll9&:O:OP rCF)tie_last_hidden_statesNr^rrGc | td|j||}|jdd|i|}|j}|j |}|dddddf}|j |}t ||S)Nz You have to specify pixel_values)r^rrr pooler_outputr)rrDrrrr)r@r_r^rrencoder_outputsr pooled_outputs rBrmz$InstructBlipVideoVisionModel.forwards  ?@ @ Ogh +74<<, ', ,  ,== //0AB)!Q'2 ++M: )/'  rCc|jSr)rDr@s rBget_input_embeddingsz1InstructBlipVideoVisionModel.get_input_embeddingss rCr )rorprqmain_input_namerr'rrr_can_record_outputsr1rrrr7rtrurrrrrmr#rvrws@rBrrs$O ))60 <  E226). ''$. #' +,  + +  3  6rCrcXeZdZd fd ZdZdZdZdZdZ d de e fdZ xZ S) rct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj |j"|_d|_y)Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)F)r0r1r*r2rhasattrrrsattention_head_size all_head_sizerrrencoder_hidden_sizerrrattention_probs_dropout_probrsave_attentionr@r*is_cross_attentionrAs rBr1z3InstructBlipVideoQFormerMultiHeadAttention.__init__sa     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9FInstructBlipVideoQFormerMultiHeadAttention.save_attn_gradientss ,rCc|jSrr2r"s rBget_attn_gradientsz=InstructBlipVideoQFormerMultiHeadAttention.get_attn_gradientss"""rCc||_yr attention_map)r@r9s rBsave_attention_mapz=InstructBlipVideoQFormerMultiHeadAttention.save_attention_maps *rCc|jSrr8r"s rBget_attention_mapz 11$((;P2QRI33DJJ?T4UVK3N11$((=2IJI33DJJ}4MNK JJ}5//0AB !<< Y5H5HR5PQ+dii8P8P.QQ!1!7!7  %/.@ -"**,-=>AABXY $"5"5  # #O 4  ) )$*B*B C#',,"? %S"S* **,CD o--rCrnNNN) rorprqr1r4r6r:r<r@rrrmrvrws@rBrrsA$0-#+"%"# 4. +, 4.rCrcneZdZfdZdej dej dej fdZxZS)rc(t|tj|j|j|_tj |j|j|_tj|j|_ yr) r0r1rrr2denserrrrrr?s rBr1z+InstructBlipVideoQFormerSelfOutput.__init__#s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rCr input_tensorrGcr|j|}|j|}|j||z}|SrrUrrr@rrVs rBrmz*InstructBlipVideoQFormerSelfOutput.forward)7 =1  ]3 }|'CD rCrrws@rBrr"1> U\\RWR^R^rCrc eZdZd fd Z d dej dej dzdej dzdej dzdeedej f d Z xZ S) !InstructBlipVideoQFormerAttentioncdt|t|||_t ||_yr)r0r1r attentionroutputr/s rBr1z*InstructBlipVideoQFormerAttention.__init__1s) CFL^_8@ rCNrrrFrGrrGc ^|jd||||d|\}}|j||}|S)N)rrrFrGr)r_r`) r@rrrFrGrrriattention_outputs rBrmz)InstructBlipVideoQFormerAttention.forward6sL( ')"7#9     Q ;;{MBrCrnrR) rorprqr1r7rrrtrrrmrvrws@rBr]r]0sA48:>;?  || ))D0  %0047  !& 1 1D 8  +,    rCr]cVeZdZfdZdej dej fdZxZS)$InstructBlipVideoQFormerIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r0r1rrr2rrUrrstrr intermediate_act_fnr?s rBr1z-InstructBlipVideoQFormerIntermediate.__init__Js] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rCrrGcJ|j|}|j|}|Sr)rUrgrs rBrmz,InstructBlipVideoQFormerIntermediate.forwardRs& =1 00? rCrrws@rBrdrdIs#9U\\ellrCrdcneZdZfdZdej dej dej fdZxZS)InstructBlipVideoQFormerOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r0r1rrrr2rUrrrrrr?s rBr1z'InstructBlipVideoQFormerOutput.__init__Ys` YYv779K9KL f&8&8f>S>STzz&"<"<= rCrrVrGcr|j|}|j|}|j||z}|SrrXrYs rBrmz&InstructBlipVideoQFormerOutput.forward_rZrCrrws@rBrjrjXr[rCrjcFeZdZfdZ ddeefdZdZdZxZ S)InstructBlipVideoQFormerLayercft||j|_d|_t ||_||_||jzdk(rt |d|_d|_ nd|_ t||_ t||_ t||_t||_y)Nr$rT)r0F)r0r1chunk_size_feed_forward seq_len_dimr]r_ layer_idxcross_attention_frequencycrossattentionhas_cross_attentionrd intermediaterjr`intermediate_query output_queryr@r*rrrAs rBr1z&InstructBlipVideoQFormerLayer.__init__gs '-'E'E$:6B" v77 71 <"CF_c"dD '+D $',D $@H4V< "Fv"N:6BrCrc b|j|fd|i|}|dkDr|ddd|ddf}|jr$| td|j|f|||d|}t |j |j |j|} |jd|kDrjt |j|j |j|dd|dddfj| j} tj| | gd} | St |j|j |j|} | S)Nrrz>encoder_hidden_states must be given for cross-attention layers)rrFrGr$rN)r_rurrtrfeed_forward_chunk_queryrprqrPfeed_forward_chunkrdrr7rX) r@rrrFrG query_lengthrrbquery_attention_output layer_outputlayer_output_texts rBrmz%InstructBlipVideoQFormerLayer.forward{sp*4>>  )   ! %5a,6I%J "''(0$%eff)<)<)<**#1*?+A *  *&5--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,8I)JPQR  5'',,   L rCcL|j|}|j||}|Sr)rvr`r@rbintermediate_outputrs rBr|z0InstructBlipVideoQFormerLayer.feed_forward_chunks,"//0@A{{#68HI rCcL|j|}|j||}|Sr)rwrxrs rBr{z6InstructBlipVideoQFormerLayer.feed_forward_chunk_querys."556FG(()<>NO rCr) rorprqr1rrrmr|r{rvrws@rBrnrnfs7C."# 3+,3j rCrncDeZdZfdZe ddeefdZxZS)InstructBlipVideoQFormerEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wr ) r0r1r*rr r r rnlayerrrys rBr1z(InstructBlipVideoQFormerEncoder.__init__sW  ]]OTU[UmUmOn o) *69 = o  ',# psA$rc t|jjD]}|j|}||||f||d|}!t |S)N)rGr}r)r r*r rr) r@rrrFrGr}ri layer_modules rBrmz'InstructBlipVideoQFormerEncoder.forwardskt{{445 A::a=L(%(>)   M 9+  rCr) rorprqr1rrrrmrvrws@rBrrs:,"#  +,  rCrceZdZdZdZdZdZdZee e ddge e ddgdZ de ffd Z d Zd Z dd ej"d eedej(dedej"f dZeee ddej4d ej6dzdej4dzdej"dzdej6dzdej6dzdeedeej6ezfdZxZ S)InstructBlipVideoQFormerModelz Querying Transformer (Q-Former), used in InstructBlipVideo. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr$z .attention)index layer_namez.crossattention)rrcross_attentionsr*ct||||_t||_t ||_|jyr)r0r1r*ryrDrrrr?s rBr1z&InstructBlipVideoQFormerModel.__init__s9     rCc.|jjSrrDrr"s rBr#z2InstructBlipVideoQFormerModel.get_input_embeddingss...rCc&||j_yrrr@rs rBset_input_embeddingsz2InstructBlipVideoQFormerModel.set_input_embeddingss*/'rCr input_shaper has_queryrGc|jdk(r|dddddddf}n=|jdk(r|ddddddf}ntd|d|jd|j|j}d|z d z}|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. rNr z!Wrong shape for input_ids (shape z) or attention_mask (shape )rag?g)rOrrPrdrb)r@rrrrextended_attention_masks rBget_extended_attention_maskz9InstructBlipVideoQFormerModel.get_extended_attention_masks.    1 $&4Qa]&C #    !Q &'5QdA5E&F #3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&rCNrr~rrFrGrc | | td||jdnd}|j|||} | jdd} | \} } | j} |t j | | f| }|j|| | }|t|tr|dj\}}}n|j\}}}||f}t|tr|Dcgc]}|j|}}n?|)t j || }|j|}n|j|}nd}|j| f||||d|}|j}|dddddf}t|| Scc}w) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner$r)rr~rrI)r)rrFrGr}r)rrPrDrKrr7onesrrlistinvert_attention_maskrrr)r@rrr~rrFrGrr}embedding_outputrrhrrrencoder_batch_sizeencoder_sequence_lengthriencoder_hidden_shapemaskencoder_extended_attention_maskrsequence_outputr s rBrmz%InstructBlipVideoQFormerModel.forward/s$  !5VW W0<0H|))!,a ??%%+  '++-cr2 !, J!((  !"ZZ*j)A6RN#'"B"B>S^`f"g ! ,/6AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2wX\43M3Md3S2w/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 ++74<< , 2"7#B% ,  , *;;'1a0 ;-'  )3xsE6rn)NNNNN)!rorprqrrrrrrnrrr%r&r1r#rr7rrrrsrrurrrr LongTensorrtrrrrmrvrws@rBrrs #( N7 EQ[g h  EQ[l m =/0 )' )'3Z)' )'  )'  )'V 4804,0:>;?F ##F ))D0F &&- F llT) F %0047 F !& 1 1D 8F +,F  u  !$P PF  F rCrzV Class defining the outputs of [`InstructBlipVideoForConditionalGeneration`]. ) custom_introceZdZUdZdZeejdzed<dZ eejdzed<dZ e dzed<dZ e dzed<dZeezdzed<deefd Zy) 4InstructBlipVideoForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsrGcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)rrrN)getattrto_tuple).0kr@s rB zPInstructBlipVideoForConditionalGenerationModelOutput.to_tuple..sB WW Gq!**, - s-0)rkeysr"s`rBrz=InstructBlipVideoForConditionalGenerationModelOutput.to_tuples% YY[    rC)rorprqrrrr7rtrrrrrrrrrrrrrCrBrr{s -1D%!! "T )0.2FE%## $t +28D   ( ( 7 4rCrrc|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j|j }|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. rbrrI) r#r7rr*image_token_idlongrall unsqueeze expand_asrdr@rrspecial_image_masks rBget_placeholder_maskz+InstructBlipVideoModel.get_placeholder_mask  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H /99"=GG VYYZgZnZno!!rCNqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskoutput_attentionsoutput_hidden_states return_dictr^ use_cacherrGc | | n|jj} |j\}}}}}|j||z|||}|j || | | | }|d}t j |jddt j|j}|jj|jddd}t j |jddt j|j}|t j|}|j|d}|j|d}t j||gd}|j|||||| | | }|dddd|jdddf}|j!|}|j||jj"|zd}|Q|j$j'|}||jj(k(}|t j|}nl||j't j*|jj(t j|jk(}|j-d}|j/dj1|j3|j}|j3|j|j4}|j7||}|jj8r|j$d ||| | | | d |}n|j$d ||||| | | | d |}t;||| S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)r_rrrr^rrIrrNr$)rrrrFrGrrrrrrrrr)rrrrrrrrrr)r*use_return_dictrPrSrr7rrKrrrrg ones_likerepeat_interleaverXrrrrr#video_token_idrrrrrdrbmasked_scatteruse_decoder_only_language_modelr)r@r_rrrrrrrrrrr^rrrhframeschannelrErFr image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputss rBrmzInstructBlipVideoModel.forwardsR&1%->q-A+A1(DE !% 8 8 F!6 = =j$++JfJfioJoqs t  FD//DDFyQM!*dkk.H.H!H %!&!;!.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; /99"=GG VYYZgZnZno 5 8 89M9M}ObOb c%445GI^_ ;; 6 6)d))+-"3%9'# G*d)) +-"3'="3%9'#  GD))#*  rC) NNNNNNNNNFN)rorprqr$_keep_in_fp32_modulesr%r1r#rrr7rrtrrrrrrurr rrrmrvrws@rBrrs %O+, 6 :8?( "e.>.> "uO`O` " ;?.22659:>-1)-,0#').!% '' !,, !& 0 04 7  $$t+  ((4/  !++d2 !& 0 04 7 ||d*  $; #Tk D[ #' $; -. E E!  rCrc:eZdZUdZdZedzed<dZedzed<y)'BaseModelOutputWithVisionQformerOutputsz vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). Nrr) rorprqrrrrrrrrCrBrras)9=N.5P>PQN2>>v?Q?QRN, rCc6|jjSrrr"s rBr#z>InstructBlipVideoForConditionalGeneration.get_input_embeddingsrrCc:|jj|yrrrs rBrz>InstructBlipVideoForConditionalGeneration.set_input_embeddingsrrCc:|jj|yr)rset_output_embeddings)r@new_embeddingss rBrz?InstructBlipVideoForConditionalGeneration.set_output_embeddingss 11.ArCrGc6|jjSr)rget_output_embeddingsr"s rBr z?InstructBlipVideoForConditionalGeneration.get_output_embeddingss""88::rCNc\||jjSt||S)N)modality)r get_encoderr0)r@r rAs rBrz5InstructBlipVideoForConditionalGeneration.get_encoders1  &&224 47&&9 9rCc6|jjSr)r get_decoderr"s rBrz5InstructBlipVideoForConditionalGeneration.get_decoders""..00rCc|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyrrrs rBrz@InstructBlipVideoForConditionalGeneration._preprocess_acceleraterrCrrc|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j|j }|Sr) r#r7rr*rrrrrrrdrs rBrz>InstructBlipVideoForConditionalGeneration.get_placeholder_maskrrCrrrrrrrlabelsrr^rrc <| | n|jj} |j|f||| dd|}|j}|j}|j }||j |}|tj|}|j|j|j}|j||}|j||}|jjre|jd ||| | | |d|}| r |j n|d}d}| w|j"d || |jj$j&d|}nB|jd ||||| | | | |d |}| r |j(n|d}| r |j n|d }t+||||| S) a qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length)): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. Examples: ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTrrr^rrrr)rrr) rrrrrrrrrr$)rrrrrr)r*rget_video_featuresrrrr#r7rrdrrbrrrrr loss_functionrrrr)r@r_rrrrrrrrrrrr^rrvideo_featuresrrrrrrrs rBrmz1InstructBlipVideoForConditionalGeneration.forwardsf&1%>(66(66%00)  &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "-??A?N!7!I!I&VW!I!X!&,@BX+Y_`!a&$,, '1%".#7   *9&&q)!-C|/@/@/C-CQ*FG 11,?(// DKK.> "uO`O` " ;?.22659:>26)-,0*.#').!%P ''P !,,P !& 0 04 7 P $$t+ P ((4/ P !++d2P !& 0 04 7P ((4/P  $;P #TkP   4'P D[P #'P $;P +,!P " E E#P P dU]]_6::>-12626).D''D!++d2D!& 0 04 7 D ##d* D ((4/ D((4/D#'D   DDL ;?05 E''E!++E!& 0 04 7 E #'+ E +, E 8 8EErCr)rrrrr)r)RrBcollections.abcr dataclassesrtypingrr7rrr activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsr rrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrrr utils.genericrutils.output_capturingrrautor!r"r#configuration_instructblipvideor%r&r' get_loggerrorr%r)ryrrrfloatrrrrrrrrr]rdrjrnrrrrrr__all__rrCrBr:s, $! &!)B9G&6jj7EII   H %G GT//diiiR% II% <<% % << % LL4' %  %%.G)G)T 299 $>@@ryy@@3#C3l^.^.B     2 299  RYY R$>Rj! bii! HY $DY x   ;   : { ={  { |  P.H P  Pu0PRauup  rC