qi=!ddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z mZdd lmZd d lmZd d lmZeGddeZeedGdde ZedGddeZddgZy)) dataclassN)nn)initialization)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstring)can_return_tuple) AutoModel)ColModernVBertConfigc^eZdZUeed<dZdZgZdZdZ dZ e jdZ y)ColModernVBertPreTrainedModelconfigmodel)imagetextTct|jdr|jjn)|jjjj}t |t jt jfrOtj|jd||j tj|jyyt |t jrttj|jd||jEt!|jdds-tj|j|jyyyy)Ninitializer_rangeg)meanstd_is_hf_initializedF)hasattrrr vlm_config text_config isinstancerLinearConv2dinitnormal_weightbiaszeros_ Embedding padding_idxgetattr)selfmodulers l/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/colmodernvbert/modeling_colmodernvbert.py _init_weightsz+ColModernVBertPreTrainedModel._init_weights-st{{$78 KK ) )''33EE fryy"))4 5 LLSc :{{& FKK('  - LLSc :!!-gfmmMach6i FMM&*<*<=>7j-.N)__name__ __module__ __qualname__r__annotations__base_model_prefixinput_modalities_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attntorchno_gradr-r.r,rr#sE  (NU]]_??r.rz: Base class for ColModernVBert embeddings output. ) custom_introceZdZUdZdZej dzed<dZejdzed<dZ e ej dzed<dZ e ej dzed<dZ e ej dzed<y) ColModernVBertForRetrievalOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): The embeddings of the model. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True` and `pixel_values` are provided): Tuple of `torch.FloatTensor` (one for the output of the image modality projection + one for the output of each layer) of shape `(batch_size, num_channels, image_size, image_size)`. Hidden-states of the image encoder at the output of each layer plus the initial modality projection outputs. Nloss embeddings hidden_statesimage_hidden_states attentions)r/r0r1__doc__r?r9 FloatTensorr2r@TensorrAtuplerBrCr;r.r,r>r>@s &*D%  d ")&*J t#*59M5**+d29;?u001D8?26Je''(4/6r.r>u Following the ColPali approach, ColModernVBert leverages VLMs to construct efficient multi-vector embeddings directly from document images (“screenshots”) for document retrieval. The model is trained to maximize the similarity between these document embeddings and the corresponding query embeddings, using the late interaction method introduced in ColBERT. Using ColModernVBert removes the need for potentially complex and brittle layout recognition and OCR pipelines with a single model that can take into account both the textual and visual content (layout, charts, ...) of a document. ColModernVBert is trained on top of ModernVBert, and was introduced in the following paper: [*ModernVBERT: Towards Smaller Visual Document Retrievers*](https://arxiv.org/abs/2510.01149). ColModernVBert is part of the ColVision model family, which was introduced with ColPali in the following paper: [*ColPali: Efficient Document Retrieval with Vision Language Models*](https://huggingface.co/papers/2407.01449). c eZdZiZdeffd Zee ddejdzdejdzdejdzde e def d Zd Zd Zd Zd Z ddedzdedzdedej.fdZxZS)ColModernVBertForRetrievalrct||||_|jjj |_t j|j|_|jj|_ tj|jjjj|j|_ |jyN)super__init__rrr vocab_sizer from_configvlm embedding_dimrr hidden_sizeembedding_proj_layer post_init)r*r __class__s r,rMz#ColModernVBertForRetrieval.__init__ms    ++77BB(():):;![[66$&II KK " " . . : :   % ! r.N input_ids pixel_valuesattention_maskkwargsreturnc |jd|||d|}|d}|jjj}|j|j |}||j ddz }|;|j |j|j }||jdz}t||j|j|jS) N)rVrXrWrT)dimkeepdim)dtypedevice)r@rArCrBr;) rPrSr$r_tonormr` unsqueezer>rArCrB) r*rVrWrXrY vlm_outputlast_hidden_states proj_dtyper@s r,forwardz"ColModernVBertForRetrieval.forward{sTXX )%   (]..55;; ../A/D/DZ/PQ  *//b$/"GG  %+..Z5E5EjN_N_.`N#n&>&>r&BBJ/!$22!,, * > >   r.c6|jjSrK)rPget_input_embeddingsr*s r,riz/ColModernVBertForRetrieval.get_input_embeddingssxx,,..r.c:|jj|yrK)rPset_input_embeddings)r*values r,rlz/ColModernVBertForRetrieval.set_input_embeddingss %%e,r.c6|jjSrK)rPget_output_embeddingsrjs r,roz0ColModernVBertForRetrieval.get_output_embeddingssxx--//r.c:|jj|yrK)rPset_output_embeddings)r*new_embeddingss r,rqz0ColModernVBertForRetrieval.set_output_embeddingss &&~6r.new_num_tokenspad_to_multiple_of mean_resizingcB|jj|||}|j|jjj _|j|jj_|j|j_|j|_|S)N)rsrtru)rPresize_token_embeddingsnum_embeddingsrrrrN)r*rsrtru model_embedss r,rwz2ColModernVBertForRetrieval.resize_token_embeddingss xx77)1'8 9E8S8S **5,8,G,G )*99&55r.)NNN)NNT)r/r0r1_checkpoint_conversion_mappingrrMr r r9 LongTensorrErFrr r>rgrirlrorqintboolrr'rw __classcell__)rUs@r,rIrIYs$&(" 3 .215.2  ##d* ''$.  t+  +,  *   @/-07 &*)-" d  $J   r.rI) dataclassesrr9rrr"modeling_utilsrprocessing_utilsrutilsr r r utils.genericr auto.modeling_autorconfiguration_colmodernvbertrrr>rI__all__r;r.r,rs*" &-&DD-*>?O??8  7{7  7&"P!>P#"Pf ()H Ir.