qi*ddlZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZmZmZddlmZdd lmZdd lmZmZmZdd lmZd d lmZddl m!Z!eGddeZ"eGddeZ#GddejHZ%eGddeZ&edGdde&Z'GddejHZ(eGdde&Z)ed Gd!d"e&Z*ed#Gd$d%e&Z+gd&Z,y)'N) dataclass)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringtorch_compilable_check)can_return_tuple) AutoModel)ModernVBertConfigceZdZUdZdZej ed<dZe ej dzed<dZ e ej dzed<dZ e ej dzed<y)ModernVBertBaseModelOutputaY Base class for ModernVBERT model's outputs. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlast_hidden_state hidden_states attentionsimage_hidden_states) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rtuplerrf/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/modernvbert/modeling_modernvbert.pyrr-si,,0u((/59M5**+d2926Je''(4/6;?u001D8?r(rceZdZUdZdZej dzed<dZej ed<dZ e ej dfdzed<dZ e ej dfdzed<dZ ej dzed<y) ModernVBertMaskedLMOutputaG Base class for ModernVBERT model's outputs with masked language modeling loss. Args: loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided): Masked language modeling (MLM) loss. logits (`torch.FloatTensor`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlosslogits.rrr) rr r!r"r,r#r$r%r-rr&rrr'r(r)r+r+Ks,&*D%  d ") $FE  $:>M5**C/047>7;Je'',-4;48**T18r(r+c.eZdZdZfdZdZdZxZS)ModernVBertConnectorz Connector module for ModernVBERT. It performs a pixel shuffle operation followed by a linear projection to match the text model's hidden size. Based on https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html ct||j|_tj|j j |jdzz|jj d|_y)NrFbias) super__init__pixel_shuffle_factornnLinear vision_config hidden_size text_configmodality_projectionselfconfig __class__s r)r4zModernVBertConnector.__init__ps^ $*$?$?!#%99  , ,0K0KQ0N O    * *$  r(c|j\}}}t|dzx}}|j||||}|j||t||z ||z}|jdddd}|j |t||z t||z ||dzz}|jdddd}|j |t||dzz ||dzzS)Ng?rrrr)sizeintviewpermutereshape)r=rr5 batch_size seq_length embed_dimheightwidths r) pixel_shufflez"ModernVBertConnector.pixel_shuffleys,?,D,D,F) J Z_--166z65R[\166 E,@$@ A9OcCc 299!Q1E199  ,, - -- . -q0 1   299!Q1E"** J*>*ABCYRfhiRiEj  r(c\|j||j}|j|SN)rKr5r;)r=rs r)forwardzModernVBertConnector.forwards."001DdF_F_`''(;<model)imagetextTpast_key_valuesc t|dtjdtffd }t |t rajjtjdjjjzz }||j|yt |trajjtjdjjjzz }||j|yt |t t"fr^jjtjjjj$z }||j&|yy)Nmodulestdc8tjdd}tj|jd|| |z||zt |t jt jfr-|j tj|jyyy)Ninitializer_cutoff_factor@)meanrYab) getattrr>init trunc_normal_weight isinstancer6r7Conv2dr2zeros_)rXrY cutoff_factorr=s r) init_weightz=ModernVBertPreTrainedModel._init_weights..init_weights#DKK1LcRM     .3&#%  &299bii"89;;*KK ,+:r(r\)r3 _init_weightsr6Modulefloatrer/r>initializer_rangemathsqrtr:num_hidden_layersr;ModernVBertForMaskedLMlm_head$ModernVBertForSequenceClassification!ModernVBertForTokenClassificationr9 classifier)r=rXriout_std final_out_stdr?s` r)rjz(ModernVBertPreTrainedModel._init_weightss f% - - - f2 3kk33diidkkF]F]FoFo@o6ppG 22G <  6 7kk33diidkkF]F]FoFo@o6ppG  0  41  !KK99DIIdkkF]F]FiFicZt|||jjj|_|jjj |_tj|j|_ t||_ tj|j|_ t|jj|jj zdz|j"dzz |_|jj&|_|j)y)Nr)r3r4r>r: pad_token_id padding_idx vocab_sizer from_configr8 vision_modelr/ connector text_modelrB image_size patch_sizer5 image_seq_lenimage_token_id post_initr<s r)r4zModernVBertModel.__init__s  ;;22??++11<<%11&2F2FG.f5#//0B0BC ""--1E1E1P1PPUV V**A- / #kk88 r(c6|jjSrM)rget_input_embeddingsr=s r)rz%ModernVBertModel.get_input_embeddingss3355r(c:|jj|yrM)rset_input_embeddings)r=values r)rz%ModernVBertModel.set_input_embeddingss ,,U3r( input_ids inputs_embedsrc |j\}}}|a||jtj|jj tj |jk(}|d}n||jj k(}|jd}ttj||zdk(d||z}tjjj|jddd } | dd } |jd } | dz |z} | dz |z} | jd| z}tj |}|||| |ddf||<tj"|jd ||}|S) as This method aims at merging the token embeddings with the image hidden states into one single sequence of vectors that are fed to the transformer LM. The merging happens as follows: - The text token sequence is: `tok_1 tok_2 tok_3 ... tok_4`. - We get the image hidden states for the image through the vision encoder and that hidden state, after a pixel shuffle operation, is then projected into the text embedding space. We thus have a sequence of image hidden states of size (1, image_seq_len, hidden_dim), where 1 is for batch_size of 1 image and hidden_dim is the hidden_dim of the LM transformer. - The merging happens so that we obtain the following sequence: `vector_tok_1 vector_tok_2 vector_tok_3 vector_fake_tok_around_image {sequence of image_seq_len image hidden states} vector_fake_toke_around_image vector_tok_4`. That sequence is fed to the LM. - To fit the format of that sequence, `input_ids`, `inputs_embeds`, `attention_mask` are all 3 adapted to insert the image hidden states. Ndtypedevice).rrdimrzCAt least one sample has tokens not divisible by patch_size.)rr)r)shaperr#tensorr>rlongrsumrallr6 functionalpadcumsum unsqueeze zeros_likewhere)r=rrr_r image_masknum_image_tokensblocks_per_sampleoffsets block_offsetrow_cum chunk_idx local_idx block_idx image_embeds merged_embedss r) inputs_mergerzModernVBertModel.inputs_mergers/44:q  &*E$*C*C*E T[[77uzzR_RfRfg+J$F+J"dkk&@&@@J%>>a>0 II&3q8 9 Q - :((%%))*;*B*Bq*B*I6YZ)[s| ###+q[Z/ q[J.  **1- 9 '' 6 #6y7LiXbNcef7f#g Z  J$8$8$r8runfoldrrr pooler_output)r=rrrrF num_images num_channelsrIrJnb_values_per_imagereal_images_indsirpatches_subgridr image_outputsrimage_featuress r)get_image_featuresz#ModernVBertModel.get_image_featuress ?K>P>P; J fe#TZZ8 (|((j)@Z?l((+?jj#**$ $=#7#<# (batch_size, max_num_images, 3, max_heights, max_widths) where max_num_images is the maximum number of images among the batch_size samples in the batch. Padding images are not needed beyond padding the pixel_values at the entrance of the model. For efficiency, we only pass through the vision_model's forward the real images by discarding the padding images i.e. pixel_values of size (image_batch_size, 3, height, width) where image_batch_size would be 7 when num_images_per_sample=[1, 3, 1, 2] and max_num_images would be 3. ModernVBERT/modernvbertr checkpointattention_mask position_idsc |9|jj|j|j}||j ||j }|;|j|j |j}|j|||}|jd|||d|} t| j| j| j|S)a| pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. )rrr)rrr)rrr)rrrrr') rrrrrrrrrrrr) r=rrrrrrrroutputss r)rNzModernVBertModel.forwardJs>  BDOO@@B9MPPQZQaQabM  #"&"9"9)@T#:#m  *"5"8"8}?R?R[h[o[o"8"p  ..#=Vi/M "$// ')%   *%77!//)) 3   r(rM)NNNNNNN)rr r!r"rr4rrr# LongTensorTensorrrrr$rrr&r r BoolTensorrrNrOrPs@r)rrs 0$64())(:?,,(]b]i]i(Tm 9=2''2$..52+, 2 + + 22h - '+.20426158<8</ ##/  t+/ &&- / ((4/ / ''$. / $..5/ #..5/ +,/  + +/  / r(rc\eZdZdeffd Zdej dej fdZxZS)ModernVBertPredictionHeadr>cJt|||_tj|j |j |j |_t|j|_ tj|j |j|j|_y)N)epsr2)r3r4r>r6r7r9classifier_biasdenser classifier_activationact LayerNormnorm_eps norm_biasnormr<s r)r4z"ModernVBertPredictionHead.__init__sq  YYv1163E3EvG]G]^ &667LL!3!3vO_O_` r(rrc`|j|j|j|SrM)rrr)r=rs r)rNz!ModernVBertPredictionHead.forwards#yy$**]";<==r() rr r!rr4r#rrNrOrPs@r)rrs-a0a>U\\>ell>r(rcneZdZddiZfdZdZdZeedd dd e jd e jd zd e jd zd e jd zde jd zde jd zde jd zde jd zdeedeezfdZxZS)rqzlm_head.weightz1model.text_model.embeddings.tok_embeddings.weightclt|||jj|_t ||_t |j|_tj|jj|j|jj|_ |jy)Nr1)r3r4r:rrrSrprojection_headr6r7r9 decoder_biasrrrr<s r)r4zModernVBertForMaskedLM.__init__s   ,,77%f- 89K9KLyy!3!3!?!?W]WiWiWvWvw  r(c|jSrMrrrs r)get_output_embeddingsz,ModernVBertForMaskedLM.get_output_embeddingss ||r(c||_yrMr)r=new_embeddingss r)set_output_embeddingsz,ModernVBertForMaskedLM.set_output_embeddingss % r(rrrNrrrrrrrlabelsrrc \|jd|||||||d| } | d} |j|j| } d} |