qidZddlmZddlmZddlmZddlZddlmZddl m Z dd l m Z dd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddlmZmZmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ejTe+Z,dejZdejZfdZ.dejZdejZfdZ/dejZdejZfdZ0eedGddeZ1eedGd d!eZ2eeGd"d#eZ3Gd$d%ejhZ5Gd&d'ejhZ6 dOd(ejhd)ejZd*ejZd+ejZd,ejZdzd-e7d.e7d/eefd0Z8Gd1d2ejhZ9Gd3d4ejhZ:Gd5d6eZ;eGd7d8eZ<Gd9d:ejhZ=Gd;ded=Gd>d?e<Z?Gd@dAe<Z@edBGdCdDe<ZAeGdEdFe<ZBeGdGdHe<ZCeGdIdJe<ZDedKGdLdMe<ZEgdNZFy)PzPyTorch CLIP model.)Callable) dataclass)AnyN)nn)initialization)ACT2FN)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) CLIPConfigCLIPTextConfigCLIPVisionConfiglogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr")rs X/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/clip/modeling_clip.pycontrastive_lossr)/s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r)t)r+ caption_loss image_losss r( clip_lossr03s,#J/L!*,,.1J : % ,,r*tensorctj|d}tj|dd}tj|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 T)dimkeepdim?)r%powsum)r1 square_tensor sum_tensor normed_tensors r(_get_vector_normr=9s< IIfa(M=b$?JIIj#.M r*z} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<y)CLIPVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rAr% FloatTensor__annotations__rBrCtuplerDr*r(r@r@Dsr .2L%##d*126u((4/6:>M5**C/047>7;Je'',-4;r*r@ze Base class for text model's outputs that also contains a pooling of the last hidden states. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<y)CLIPTextModelOutputz text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. N text_embedsrB.rCrD) rErFrGrHrOr%rIrJrBrCrKrDrLr*r(rNrNVsr -1K""T)026u((4/6:>M5**C/047>7;Je'',-4;r*rNceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ e ed<dZe ed <d eefd Zy) CLIPOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. Nlosslogits_per_imagelogits_per_textrOrAtext_model_outputvision_model_outputrcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))rUrVN)getattrto_tuple).0kselfs r( z&CLIPOutput.to_tuple..s=  LLDGRYZ^`aRbRkRkRm m s-0)rKkeysr]s`r(rZzCLIPOutput.to_tuples# YY[   r*)rErFrGrHrRr%rIrJrSrTrOrArUr rVrKrrZrLr*r(rQrQhs&&*D%  d ")15e''$.504OU&&-4,0K""T)0-1L%##d*148186:3: %* r*rQceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) CLIPVisionEmbeddingsconfigct|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_tj"|j |j|_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasr3r position_idsrr4 persistent)super__init__rc hidden_size embed_dim image_size patch_sizer Parameterr%randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr&expandr]rc __class__s r(rozCLIPVisionEmbeddings.__init__s   ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr* embeddingsheightwidthrc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr4r7rr3bicubicF)sizemode align_cornersr5)shaper}weight unsqueezer%jit is_tracingrjrsrreshapepermuterr# interpolateviewcat) r]rrrrzr}r{class_pos_embedpatch_pos_embedr5 new_height new_widthsqrt_num_positionss r(interpolate_pos_encodingz-CLIPVisionEmbeddings.interpolate_pos_encodingsv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr* pixel_valuesc `|j\}}}}|sJ||jk7s||jk7r,td|d|d|jd|jd |jjj }|j|j |}|jdjdd}|jj|dd} tj| |gd } |r| |j| ||z} | S| |j|jz} | S) NzInput image size (*z) doesn't match model ().)dtyper3rr4r)rrr ValueErrorryrrtoflatten transposervrr%rrr}rj) r]rr batch_size_rr target_dtype patch_embeds class_embedsrs r(forwardzCLIPVisionEmbeddings.forwards6'3'9'9$ Avu'Vt-F%SWSbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr*F) rErFrGrror%TensorintrrIr __classcell__rs@r(rbrbseq/q,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr*rbc eZdZdeffd Z d dej dzdej dzdejdzdejfdZ xZ S) CLIPTextEmbeddingsrccNt||j}tj|j ||_tj|j||_|jdtj|jjddy)NrjrkFrl) rnrorprr| vocab_sizetoken_embeddingmax_position_embeddingsr}r~r%r&rr]rcrqrs r(rozCLIPTextEmbeddings.__init__s && !||F,=,=yI"$,,v/M/My"Y  ELL)G)GHOOPWXej  r*N input_idsrj inputs_embedsrc8||jdn|jd}|jjjd}||kDrtd|d|||jddd|f}||j |}|j|}||z}|S)Nr4rzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rr}rrrjr)r]rrjr seq_lengthmax_position_embeddingposition_embeddingsrs r(rzCLIPTextEmbeddings.forwards -6,AY__R(}GZGZ[]G^ !%!8!8!?!?!E!Ea!H . .d,<=S> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r*NrCrrrc |j\}}}|j|}|j|}|j|} |j ||d|j j dd}|j ||d|j j dd}| j ||d|j j dd} tj|jjt} | |||| |f|j|jsdn |jd|\} } | j||dj!} |j#| } | | fS)z#Input shape: Batch x Time x Channelr4rr3)rr)rrrrrrrr get_interfacerc_attn_implementationrrrrrrr) r]rCrrrrrqqueriesr_valuesattention_interfacerrs r(rzCLIPAttention.forward4sT-:,?,?) J ++m,{{=)]+,,z:r4==ISSTUWXYyyZT]]CMMaQRSZRGQQRSUVW(?(M(M KK , ,.E) %8      % JJ#}}C$,, %  % ! \"))*j"EPPR mmK0 L((r*N)rErFrGrHrrror%rrrrKrrrs@r(rrstGB/.@B./3$)||$) t+$)+, $) u||U\\D00 1 $)r*rcVeZdZfdZdej dej fdZxZS)CLIPMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) rnrorcr hidden_act activation_fnrrrpintermediate_sizefc1fc2rs r(rozCLIPMLP.__init__\sd  #F$5$5699V//1I1IJ99V55v7I7IJr*rCrcl|j|}|j|}|j|}|Sr)rrr)r]rCs r(rzCLIPMLP.forwardcs4/ **=9 / r*)rErFrGror%rrrrs@r(rr[s$KU\\ellr*rceZdZdeezffd Zdejdejdee dejfdZ xZ S)CLIPEncoderLayerrccDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yN)eps) rnrorprqr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rs r(rozCLIPEncoderLayer.__init__ksl ++&v.<<F> ')   q !=0  ((7 /  =0 r*) rErFrGrrror%rrrrIrrrs@r(rrjsVS/.@S|| +,     r*rcleZdZUeed<dZdZdZdZdZ dZ dZ e e dZej dZy)CLIPPreTrainedModelrcclip)imagetextT)rCrDc |jj}t|trt j |j jd|dzt j |jjd|dzt j|jtj|jjdjdnt|tr|jj}t j |j d|j"dz|zt j |j$j|jj&|zt j |jj|jj&|zt j|jtj|j(jdnt|t*r|jj}|j"dzd|jj,zdzz|z}|j"dz|z}t j |j.j|t j |j0j|t j |j2j|t j |j4j|nt|t6r|jj}|jj8dzd|jj,zdzz|z}d|jj8zdz|z}t j |j:j|t j |j<j|nt|t>rt j |j@j|jBdz|jjzt j |jDj|jFdz|jjznGt|tHrZt j |jDj|jj8dz|jjznt|tJrZt j |j@j|jj8dz|jjznst|tLrct j |jNj|jjPj8dz|jjzt|tRjTr>t jV|jXt jZ|jt|tRj\r-|jX t jV|jXy y y ) zInitialize the weightsrg{Gz?)meanstdr4rkr)rr3N)/rcinitializer_factor isinstancerinitnormal_rrr}copy_rjr%r&rrrbrvrqryinitializer_ranger{rnum_hidden_layersrrrrrrprr CLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimCLIPVisionModelWithProjectionCLIPTextModelWithProjectionCLIPForImageClassification classifier vision_configrrzeros_riones_r)r]rfactor in_proj_std out_proj_stdfc_stds r( _init_weightsz!CLIPPreTrainedModel._init_weightssu// f0 1 LL//66Sftm T LL2299RV W JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  4 5[[33F LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d LL((//KK++T1DKK4R4RR  ; < LL&&--KK++T1DKK4R4RR  : ; LL!!((KK--994?$++B`B``  fbll + KK $ JJv}} % fbii (V[[-D KK $.E (r*N)rErFrGrrJbase_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsr%no_gradrrLr*r(rrs[ (&*#N"&)# U]]_8%8%r*rc`eZdZdZdeffd Z d dejdzdee de fdZ xZ S) CLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig rcct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wNF) rnrorcr ModuleListranger rlayersgradient_checkpointing)r]rcrrs r(rozCLIPEncoder.__init__sO  mmuVMeMeGf$g!%5f%=$gh &+#%hsA#Nrrrc T|}|jD] }|||fi|}t|S)a7 Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )rB)r-r )r]rrrrC encoder_layers r(rzCLIPEncoder.forwardsH(& ![[ M)M +  r*r) rErFrGrHrror%rrrr rrrs@r(r(r(sK,z,/3  t+ +,    r*r(ceZdZUeed<dZddgZdeffd Zee de dd e jdzd e jdzd e jdzd e ed ef dZxZS)CLIPTextTransformerrcrrrct||||_|j}t ||_t ||_tj||j|_ |j|_ |jyr)rnrorcrprrr(encoderrrrfinal_layer_norm eos_token_id post_initrs r(rozCLIPTextTransformer.__init__ sm   && ,V4"6* " YF}G[G[\    ;%+74<<, '),  , ,== 112CD    !. .44Q7@Q@X@XY 5995F5M5M NUUZ\U]_M . .44Q7@Q@X@XYEII6G6N6NOSWSdSddB !M*/'  r*r)rErFrGrrJr_no_split_modulesrorrrr%rrrr rrrs@r(r2r2s  -/AB ~  E2*..2,0 ; <<$&;  t+; llT) ; +, ; $ ; 3 ; r*r2zI The text model from CLIP without any head or projection on top. c eZdZUeed<dZddgZdeffd ZdejfdZ dZ e dd e jd zd e jd zd e jd zd eedef dZxZS) CLIPTextModelrcr3rrcdt||t||_|j yr)rnror2 text_modelr8rs r(rozCLIPTextModel.__init__ds&  -f5 r*rcB|jjjSrrIrrr`s r(get_input_embeddingsz"CLIPTextModel.get_input_embeddingsj))999r*c:||jj_yrrKr]rs r(set_input_embeddingsz"CLIPTextModel.set_input_embeddingsm5:""2r*Nrrrjrc .|jd|||d|S)a9 Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrjrL)rI)r]rrrjrs r(rzCLIPTextModel.forwardps/0t )%    r*r)rErFrGrrJrrErorModulerLrPrr%rrrr rrrs@r(rGrGYs  -/AB~ :bii:;*..2,0  <<$&  t+ llT)  +,  $   r*rGc eZdZUeed<dZdZdgZdeffd Ze e de d de jdzd edzd eed efd ZxZS)CLIPVisionTransformerrcrrrcBt||||_|j}t ||_t j||j|_ t||_ t j||j|_ |jyr)rnrorcrprbrrrr pre_layrnormr(r5post_layernormr8rs r(rozCLIPVisionTransformer.__init__sv   && .v6LL8M8MN"6*  ll9&:O:OP r*Fr9Nrrrc | td|j||}|j|}|jdd|i|}|j}|dddddf}|j |}t ||S)Nz You have to specify pixel_values)rrrr>rL)rrrYr5rBrZr )r]rrrrCrCrBrDs r(rzCLIPVisionTransformer.forwards  ?@ @ Ogh ))-8 +74<<, ', ,  ,==)!Q'2 ++M: )/'  r*r*)rErFrGrrJmain_input_namerrErorrrr%rIboolrrr rrrs@r(rVrVs $O!+, /  E22605 ''$. #'+ +,  $  3  r*rVzK The vision model from CLIP without any head or projection on top. c eZdZUeed<dZdZdgZdeffd Zde jfdZ e d de jdzd ed eedefd ZxZS) CLIPVisionModelrcrrWrcdt||t||_|j yr)rnrorV vision_modelr8rs r(rozCLIPVisionModel.__init__s'  1&9 r*rcB|jjjSrrarryr`s r(rLz$CLIPVisionModel.get_input_embeddings  ++;;;r*Nrrc ,|jd||d|S)a( Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrL)ra)r]rrrs r(rzCLIPVisionModel.forwards.<!t   %%=   r*r*)rErFrGrrJr\rrErorrTrLrr%rIr]rrr rrrs@r(r_r_s $O!+,/ Q>QX]^<< T[[5W5W(XY r*Nrrrjrrc x|jd|||dd|}|j}|j||_|S)a Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```T)rrrj return_dictrL)rIr?r)r]rrrjr text_outputsrDs r(get_text_featureszCLIPModel.get_text_features sV04C4??4 )% 4  4 %22 %)%9%9-%H "r*rrc v|jd||dd|}|j}|j||_|S)a Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```T)rrrsrL)rar?r)r]rrrvision_outputsrDs r(get_image_featureszCLIPModel.get_image_featuresDsU66GT5F5F6 %%=6  6  '44 '+'='=m'L$r* return_lossc L|jd||d|}|jd|||d|} |j} |j| } | j} |j | } | t | z } | t | z } t j| | jj| j} | |jjj| jz} | j} d}|r t| }t|| | | | | |S)a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```rfrSN)rRrSrTrOrArUrVrL)rarIr?rrr=r%rr-rr"rqexpr0rQ)r]rrrrjryrrrwrtrArOrTrSrRs r(rzCLIPModel.forwardjsIL6GT5F5F6 %%=6 6  4C4??4 )%4  4 &33 --l; "00 **;7 $&6|&DD !$4[$AA  ,,{LNN4D4G4G HZHZ4[\)D,<,<,@,@,B,E,EkFXFX,YY*,,. _-D-+#%* .  r*NNr)NNNNNF)rErFrGrrJrErorrr%rrrrKr rurIr]rxrrQrrrs@r(r r s Z!z!F/3,0 <<  t+ llT)  +,  + +  D*/"''"#'"+, " + + ""H.215.204#').M ##d*M ''$.M  t+ M &&- M D[ M #'M +,M  M M r*r c eZdZUeed<dZddgZdeffd ZdejfdZ dZ e e dd ejd zd ejd zd ejd zd eedef dZxZS)rrcr3rrct||tj|}|j|_t j |j|jd|_ |jyNFrj) rnrorGrorIrrrprnrr8)r]rcrIrs r(roz$CLIPTextModelWithProjection.__init__s[  "//7 $//!yy););V=R=RY^_ r*rcB|jjjSrrKr`s r(rLz0CLIPTextModelWithProjection.get_input_embeddingsrMr*c:||jj_yrrKrOs r(rPz0CLIPTextModelWithProjection.set_input_embeddingsrQr*Nrrrjrc |jd|||d|}|j}|j|}t||j|j |j S)a@ Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> text_embeds = outputs.text_embeds ```rS)rOrBrCrDrL)rIr?rrNrBrCrD)r]rrrjrrtrDrOs r(rz#CLIPTextModelWithProjection.forwardsu44C4??4 )%4  4 %22 **=9 "#*<<&44#..   r*r)rErFrGrrJrrErorrTrLrPrrr%rrrrNrrrs@r(rrs  -/AB ~ :bii:;*..2,0 & <<$&&  t+& llT) & +, &  & & r*rc eZdZUeed<dZdZdeffd ZdejfdZ e e d de jdzded eedefd ZxZS) rrcrrWct||tj|}|j|_t j |j|jd|_ |jyr) rnror_rorarrrprnrr8r]rcrars r(roz&CLIPVisionModelWithProjection.__init__s\  &33F; (55!#6+=+=v?T?T[`!a r*rcB|jjjSrrcr`s r(rLz2CLIPVisionModelWithProjection.get_input_embeddingsrdr*Nrrc |jd||d|}|j}|j|}t||j|j |j S)a Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection >>> from transformers.image_utils import load_image >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> image_embeds = outputs.image_embeds ```rf)rArBrCrDrL)rar?rr@rBrCrD)r]rrrrwrDrAs r(rz%CLIPVisionModelWithProjection.forwardst:6GT5F5F6 %%=6 6  '44 --m< $%,>>(66%00   r*r*)rErFrGrrJr\rrorrTrLrrr%rIr]rrr@rrrs@r(rrs $O! /