qi\:dZddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZd dlmZej>e Z!eedGddeZ"GddejFZ$GddejFZ%GddejFZ&GddejFZ'GddejFZ(GddejFZ)Gdd ejFZ*Gd!d"ejFZ+Gd#d$eZ,Gd%d&ejFZ-eGd'd(eZ.eGd)d*e.Z/Gd+d,ejFZ0ed-Gd.d/e.Z1Gd0d1ejFZ2Gd2d3ejFZ3ed4Gd5d6e.Z4ed7Gd8d9e.Z5ed:Gd;de.Z7gd?Z8y)@zPyTorch ViLT model.N) dataclass)nn)CrossEntropyLoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) ViltConfigzF Class for outputs of [`ViltForImagesAndTextClassification`]. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ e e ej dzed<dZ e e ej dzed<y)(ViltForImagesAndTextClassificationOutputa7 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`list[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. Nlosslogits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrlisttuplerX/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/vilt/modeling_vilt.pyrr+sq &*D%  d ")'+FE   $+;?M4e//01D8?8r6 __class__s r'r-zViltEmbeddings.__init__Ls  .f5ekk!Q8J8J&KL 3F ;++77 #%<< A{QPVPbPb0c#d %'\\&2Q2QSYSeSe%f"zz&"<"<=  r&c` |jjjj\}}}}|j|}|dddddddfj }t j j||jd|jdfj}|dddfjddddf} |dddfjddddf} |j\} } } }|jj|jjz}|jddddddfjddjd| ||}t!j"t%| | Dcgc]R\}}t j j't j j|||fdd d||z d| |z fTc}}d}|j)djdd}|j)djdd}t!j*t!j,t!j.|jd t!j.|jd d d j1|j2}|ddddddddf}|j5|jd|jdd d d }|j)dd}|j)d}|dks|t7|t8s| | z}|j;}n| | z}t=|j;|}|j?d}d|z j?d}|dddfjA}|Dcgc]}||dddf|k(}}|Dcgc]}||dddf|k(}}|Dcgc]}|jCd}}|Dcgc]}|jCd}}|Dcgc]}||z  }}g} tEt%|||D]\}!\}}"}#|#dkrOt!jFt!jH|j |}$| jK||!|$^t!jFt!jH|"j |#d}%| jKt!j"||!||!|%gdt!j"| d} || dddf| dddffj| d | }|| dddf| dddffj| d }|| dddf| dddffj| d d}|| dddf| dddffj| d | }|jLj5| d d }&t!j"|&|fd}t!j"|jdddddfdddddfj5| d d |fd}||z}|jO|}t!j"t!jH|jddj1||gd}|||| |fffScc}}wcc}wcc}wcc}wcc}wcc}w)Nr)sizerrdimbilinearT)rCmode align_cornersij)indexingdeviceF)as_tuple) replacement)(r5 projectionweightshapefloatr functional interpolatelongsumr> image_size patch_sizer7 transposeviewr catzippadflattenstackmeshgridarangetorNexpand isinstanceintmaxminnonzerouniquerC enumerate multinomialonesappendr3r=)'r? pixel_values pixel_maskmax_image_length_phpwxx_maskx_hx_w batch_size num_channelsheightwidth patch_dim spatial_poshw pos_embed patch_indexeffective_resolution valid_idx non_valid_idx unique_rowsu valid_row_idxnon_valid_row_idxv valid_numsnon_valid_numspad_numsselectinvp valid_choice pad_choice cls_tokenss' r' visual_embedzViltEmbeddings.visual_embed[sW,,77>>DD 1b"  ! !, /AtQM*002**6QWWQZ8P*QVVXQTl1%ad+QTl1%ad+23''/ L&%KK**dkk.D.DD ..q!"ax8BB1aHMMaQ]_hjst II SM Aq !!MM--#V'&* .  1fqj1  %%a(221a8 IIaL " "1a (kk NN5<< R(895<< UWHX;Ydh ioq "FMM" " "$aA"56 !((a&,,q/2rSUV !))!Q/ " a #3#;:N^`cCd $'9 3779 #&9 "#7#;#;#=?OP NNEN2 V,,e,< 1o,,. BMNQ9QT?a#78N NNYZ]=A+>!+CDZZ)67AaffQi7 7->?!&&)??2<=Q$q(==&s:~x'PQ fMAz2qAv$00A1D1D1FHXY  mA.|<="..uzz"~/C/C/EqVZ[  eiiq)9;LQ;OPZ;[(\bcde  f6q) fQTlF1a4L( ) . .z2| Lq!t fQTl2388RH!&A,q!t "<=BB:rSTU fQTlF1a4L89>>z2|\ ^^**:r2> IIz1o1 -II  % %aAg .q$z : A A*bRT UW` agh   M LLOEJJv||A:==fEvNTUV&;888S POZ7?=s%?AZ Z5ZZ!/Z& Z+c (|j|||} |-|j|||jj\}} } n|j d} |d}| |j t j|t j| jz} ||j t j| |t j| jz}t j| |gd} t j|| gd} | | fS)N) input_idstoken_type_ids inputs_embeds)rrrdtyperNrD) r/rr>rrr`r:r zeros_likerWrN full_liker])r?rattention_maskrrprqr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmaskss r'forwardzViltEmbeddings.forwards**m+  595F5Fj4;;;W;W6G6 2L+{%,,Q/K '#$ !D$>$>   ^5::kFXFX Y%  $d&@&@ OOK)=UZZXcXjXj k'  YY \:B  >;7Q?5  r&))r)rrrrr-rr __classcell__r@s@r'r)r)Cs V9B'!r&r)c*eZdZdZfdZddZxZS)r.zGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_idsrrJF) persistentrr)r,r-rr8 vocab_sizer2 pad_token_idword_embeddingsmax_position_embeddingsr7type_vocab_sizer: LayerNormlayer_norm_epsr;r<r=register_bufferr rcrer1rrCrWr?r>r@s r'r-zTextEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r&c6||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} || z} |j|} | | z } |j| } |j| } | S)NrJrrrr)rCrhasattrrrer r1rWrNrr:r7rr=) r?rrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr:rr7s r'rzTextEmbeddings.forwards/  #..*K',,.s3K ^  ,,Q  ^ J"%:: "66|D)) ^^J/ \\*- r&)NNNNrrrrr-rrrs@r'r.r.sQ r&r.c(eZdZdZfdZdZxZS)r4z# Image to Patch Embedding. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)r,r-rYrZr{r2rf collectionsabcIterabler6rConv2drQ)r?r>rYrZr{r2r6r@s r'r-zViltPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir&c|j\}}}}||jk7r td|jjj }|j|j |}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rSr{ ValueErrorrQrRrrd)r?rprzr{r|r} target_dtypervs r'rzViltPatchEmbeddings.forward'si2>2D2D/ L&% 4,, ,w --33 OOLOO,O? @r&rrs@r'r4r4s jr&r4c&eZdZfdZddZxZS)ViltSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)r,r-r2num_attention_headsrrrgattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer;attention_probs_dropout_probr=rs r'r-zViltSelfAttention.__init__3s.     : : :a ?PVXhHi"6#5#5"67334A7  $*#=#= #&v'9'9F,,7 _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD 6G=/2O\M]r&NFrrrr-rrrs@r'rr2sG$(r&rcxeZdZdZdeffd ZdejdejdejfdZxZ S)ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r>ct|tj|j|j|_tj |j|_yN) r,r-rrr2denser;r<r=rs r'r-zViltSelfOutput.__init__wsB YYv1163E3EF zz&"<"<= r&r input_tensorreturncJ|j|}|j|}|Srrr=r?rrs r'rzViltSelfOutput.forward|s$ =1  ]3 r&) rrrrrr-r Tensorrrrs@r'rrqs= >z> U\\RWR^R^r&rc&eZdZfdZddZxZS) ViltAttentioncbt|t||_t ||_yr)r,r-r attentionroutputrs r'r-zViltAttention.__init__s& *62$V, r&ch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r?rrr self_outputsattention_outputrs r'rzViltAttention.forwardsC~~m^EVW ;;|A F#% QR(88r&rrrs@r'rrs - r&rc\eZdZdeffd Zdej dej fdZxZS)ViltIntermediater>ct|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r,r-rrr2intermediate_sizerrf hidden_actstrrintermediate_act_fnrs r'r-zViltIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r&rrcJ|j|}|j|}|Sr)rrr?rs r'rzViltIntermediate.forwards& =1 00? r& rrrrr-r rrrrs@r'rrs*9z9U\\ellr&rcteZdZdeffd Zdej dej dej fdZxZS) ViltOutputr>ct|tj|j|j |_tj|j|_ yr) r,r-rrrr2rr;r<r=rs r'r-zViltOutput.__init__sB YYv779K9KL zz&"<"<= r&rrrcT|j|}|j|}||z}|Srrrs r'rzViltOutput.forwards. =1  ]3 % 4 r&r rs@r'r r s8>z> U\\RWR^R^r&r c*eZdZdZfdZddZxZS) ViltLayerz?This corresponds to the Block class in the timm implementation.crt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nrr)r,r-chunk_size_feed_forward seq_len_dimrrr intermediater rrrr2rlayernorm_beforelayernorm_afterrs r'r-zViltLayer.__init__s '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr&c|j|j|||}|d}|dd}||j|jz}|j |}|j |}|j ||}|f|z}|S)N)rrr)rrrdrNrrr)r?rrrself_attention_outputsrr layer_outputs r'rzViltLayer.forwards!%  ! !- 0 /"0"  2!4(,)=+;+;r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r?r>rsr@s r'r-zViltEncoder.__init__sN  ]]uVE]E]?^#_!If$5#_` &+#$`sA#c|rdnd}|rdnd}t|jD](\}} |r||fz}| |||} | d}|s || dfz}*|r||fz}|std|||fDSt|||S)Nr%rrc3&K|] }|| ywrr%).0rs r' z&ViltEncoder.forward..smq_`_lms)last_hidden_staterr)rlr!r$r ) r?rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_module layer_outputss r'rzViltEncoder.forwards#7BD$5b4(4 POA|#$58H$H!(HYZM)!,M &9]1=MeZdZUeed<dZdZdZddgZfdZ xZ S)ViltPreTrainedModelr>vilt)imagetextTr)rc6t||t|tryt j |j tj|j jdjdt j|jyy)NrJr) r, _init_weightsrfr.initcopy_rr rcrSrezeros_r)r?moduler@s r'r4z!ViltPreTrainedModel._init_weightssl f% fn - JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- . .r&) rrrrr"base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modulesr4rrs@r'r/r/s1 (&*#)+>?//r&r/cteZdZdfd ZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzde dzde dzde e ejzfdZxZS) ViltModelct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r,r-r>r)rrencoderrrr2r layernorm ViltPoolerpooler post_init)r?r>add_pooling_layerr@s r'r-zViltModel.__init__sk   (0"6* f&8&8f>S>ST,=j(4  r&cB|jjjSrrr/rr?s r'get_input_embeddingszViltModel.get_input_embeddingss..>>>r&c:||jj_yrrG)r?rs r'set_input_embeddingszViltModel.set_input_embeddings"s:?''7r&Nrrrrprqrrrrr(r)rc 0| | n|jj} | | n|jj} | | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \}}| |jn |j}|tj||f|}| | t d| | t d||jdn|jd}||k7r t d |Btj||jj|jjf|}|j|||||||| \}}|j|| }|j||| | | }|d}|j|}|j |j!|nd}| s ||f|d dzSt#|||j$|j& S)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerJz5You have to specify either input_ids or inputs_embedsrMzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrr(r)r)r' pooler_outputrr)r>rr(use_return_dictr%warn_if_padding_and_no_attention_maskrCrNr rnrSrYrget_extended_attention_maskr@rArCr rr)r?rrrrprqrrrrr(r)kwargsrtext_batch_sizerrNimage_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputs r'rzViltModel.forward%shX2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U&1#%.%:!! @T@T  !"ZZ/:)FPVWN  # (@ef f  !l&:VW W4@4L<--a0R^RdRdefRg  .`a a  %5t{{7M7Mt{{OeOe$fouvJ+/??       !5,; , (.150P0PQ_al0m,, 2/!5# ' *!,..98< 8O O4UY #]3oab6II I)-')77&11   r&)T NNNNNNNNNNN)rrrr-rIrKrr LongTensorr!rgboolr r$rrrs@r'r>r> s;"?@.2372615.22615+/)-,0#'n ##d*n ))D0n ((4/ n ''$. n $$t+ n ((4/n ''$.n "Djn  $;n #Tkn D[n  $eE,=,=&> >n n r&r>c$eZdZfdZdZxZS)rBct|tj|j|j|_tj |_yr)r,r-rrr2rTanh activationrs r'r-zViltPooler.__init__s9 YYv1163E3EF '')r&c\|dddf}|j|}|j|}|S)Nr)rr_)r?rfirst_token_tensorrXs r'rzViltPooler.forwards6+1a40 #56  6 r&rrs@r'rBrBs $ r&rBzU ViLT Model with a language modeling head on top as done during pretraining. ceZdZddiZfdZdZdZe ddejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzdejdzde dzde dzde dzde e ejzfdZxZS)ViltForMaskedLMzmlm_score.decoder.weightz6vilt.embeddings.text_embeddings.word_embeddings.weightct||t||_t ||_|j yr)r,r-r>r0 ViltMLMHead mlm_scorerDrs r'r-zViltForMaskedLM.__init__s4  f% $V, r&c.|jjSr)rfdecoderrHs r'get_output_embeddingsz%ViltForMaskedLM.get_output_embeddingss~~%%%r&c\||j_|j|j_yr)rfrhr)r?new_embeddingss r'set_output_embeddingsz%ViltForMaskedLM.set_output_embeddingss !/,11r&Nrrrrprqrrlabelsrr(r)rc D| | n|jj} |j|||||||| | |  } | dd\}}||jdn|jd}|ddd|f|dd|df}}|j |}d}|at }|j |j}||jd|jj|jd}| s|f| ddz}||f|zS|St||| j| jS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrprqrrrr(r)rBrrJrrrr) r>rNr0rSrfrrdrNr\rr rr)r?rrrrprqrrrmrr(r)rQrrWrX text_seq_len text_featuresrs mlm_logitsmasked_lm_lossloss_fctrs r'rzViltForMaskedLM.forwardsZV&1%S>STr&cl|j|}|j|}|j|}|Sr)rrzrr s r'rz#ViltPredictionHeadTransform.forward>s4 =1 --m< }5 r&rrs@r'rxrx4sUr&rxc$eZdZfdZdZxZS)rect|||_t||_t j |j|j|_ yr) r,r-r>rx transformrrr2rrhrs r'r-zViltMLMHead.__init__Fs?  4V<yy!3!3V5F5FG r&cJ|j|}|j|}|Sr)r~rh)r?rvs r'rzViltMLMHead.forwardLs" NN1  LLOr&rrs@r'rereEsH r&rez Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. czeZdZfdZe ddej dzdejdzdej dzdejdzdej dzdejdzd ejdzd ej dzd edzd edzd edzde e ejzfdZ xZ S)ViltForQuestionAnsweringc t|||j|_t||_t j t j|j|jdzt j|jdzt jt j|jdz|j|_ |jy)NrB) r,r- num_labelsr>r0r Sequentialrr2rGELU classifierrDrs r'r-z!ViltForQuestionAnswering.__init__Ys   ++f% -- IIf((&*<*>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrorrBrp)r>rNr0rMrrdrNrrU binary_cross_entropy_with_logitsrSrrr)r?rrrrprqrrrmrr(r)rQrrMrrrs r'rz ViltForQuestionAnswering.forwardjsf&1%