qi[kjdZddlmZddlmZddlZddlmZddlmZddl m Z dd l m Z dd l mZdd lmZdd lmZmZdd lmZddlmZddlmZmZmZmZmZmZddlm Z ddl!m"Z"ddl#m$Z$ejJe&Z'eedGddeZ(eedGddeZ)GddejTZ+dejXdzdejXdzd edzfd!Z-e d"d#d$% d7d&e d$ejXd'ejXdzd(ejXd)edzd*ejXdzdejXdzd+ej\dzd,e/dzd-e/dzd e0fd.Z1eGd/d0eZ2ed1Gd2d3e2Z3ed1Gd4d5e2e Z4gd6Z5y)8zPyTorch PaliGemmamodel.)Callable) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_masks_for_generate)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)deprecate_kwarg) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introc:eZdZUdZdZej dzed<y)PaligemmaModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr.s 59**T18r(rzU Base class for PaliGemma causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) PaliGemmaCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)r r!r"r#r,r$r%r&r-r.rr/tupler0rr'r(r)r+r+>s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r(r+c*eZdZdeffd ZdZxZS)PaliGemmaMultiModalProjectorconfigct|tj|jj |jj d|_y)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr4 __class__s r)r9z%PaliGemmaMultiModalProjector.__init__]s; ii 4 4 @ @&BVBVBeBelpq r(c(|j|}|SN)r>)r@image_featuresr/s r)forwardz$PaliGemmaMultiModalProjector.forwardas N3 r()r r!r"rr9rE __classcell__rAs@r)r3r3\srrr(r3token_type_idsimage_group_idsreturnc Zydtdtdtdtdtf fd }|S)z This function adds the correct offsets to the `q_idx` and `kv_idx` as the torch API can only accept lengths, not start and end indices. N batch_idxhead_idxq_idxkv_idxrJc:tj| jdk|d}tj| jdk|d} ||f}tj| jdk|d} ||f}tj| jdk|d} ||f}tj| jdk|d} ||f} tj| jdk| d} |dk(|dk(z} || k(} | | zS)Nrr)r$whereshape)rLrMrNrO safe_q_idx safe_kv_idxtoken_type_ids_at_q_idxtoken_type_ids_at_kv_idximage_group_ids_at_q_idximage_group_ids_at_kv_idxis_image_blocksame_image_blockrIrHs r) inner_maskz0token_type_ids_mask_function..inner_maskssM[[)=)=a)@!@%K kk&>+?+?+B"BFAN "0J1F"G"'++en6J6J16M.MOfhi"j#1)[2H#I #(;;v8L8LQ8O/OQikl#m #29j3H#I #(;;u7L7LQ7O/OQikm#n $3I{4J$K!$)KK9N9Nq9Q0QSlnp$q!1Q6;SWX;XY37PP 000r()intbool)rHrIr\s`` r)token_type_ids_mask_functionr_gs>1c1S11c1d12 r( input_embeds5.6.0 inputs_embedsversionnew_namer4attention_maskcache_positionr. position_ids pixel_values is_trainingis_first_iterationc  |r | td|j|||||d} | r| n|duxs|j xs|du} | s| jdds<|d|z }n4tj dt j|dddddf}|| r|dk(j|j} tjj| d d dddd f} | | z}t j|jd dz }t j| |t j |d }t#|j|j|| d <t%di| S)a" Overwrites the base `create_masks_for_generate` with `token_type_ids` masking to create the causal mask mapping for all kinds of forward passes. Paligemma uses a bidirectional mask on the prompt tokens. Uses `pixel_values` as an optional input to disambiguate edge cases. Nz;`token_type_ids` is required as a model input when training)r4rbrfrgr.rh use_cacheTrzIt is a prefill stage but The `token_type_ids` is not provided. We recommend passing `token_type_ids` to the model to prevent bad attention masking.r)rr)valuerQ)dimor_mask_functionr') ValueErrorget_text_configis_initializedgetlogger warning_oncer$ ones_liketodevicer functionalpadcumsumr]rR full_liker_r )r4rbrfrgr.rhrHrirjrkkwargs mask_kwargsis_imageis_previous_imagenew_image_startrIs r)create_causal_mask_mappingrs(~-VWW((*&((*$ K  %g_-K-K)Kg|cgOg K!>  %/N   Z  #__];Aq!GDN !&8 #a'++N,A,ABMM--ha-HCRCP"&7%77,,':':'!D!D!R  r(c6|jjSrC)rget_input_embeddingsr@s r)rz#PaliGemmaModel.get_input_embeddingss""7799r(c:|jj|yrC)rset_input_embeddingsr@rns r)rz#PaliGemmaModel.set_input_embeddingss 007r(zWObtains image last hidden states from the vision tower and apply multimodal projection.rrir~rJc |j|fddi|}|j}|j|}||jjj dzz }||_|S)N return_dictTg?)rlast_hidden_staterr4rr< pooler_output)r@rir~ image_outputsselected_image_featurerDs r)get_image_featuresz!PaliGemmaModel.get_image_featuressk*)),SDSFS !.!@!@334JK'4;;+B+B+N+NPS+ST&4 #r( input_idsrbrDcN|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )rryrQrrz6Image features and image tokens do not match, tokens: z , features: )rr$tensorr4image_token_idlongryallsumrS unsqueeze expand_asrxrnumel)r@rrbrDspecial_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskz#PaliGemmaModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r(Nrfrhr.rHrglabelsrmoutput_attentionsoutput_hidden_statesrc b|du|duz r td| | n|jj} | | n|jj} | | n|jj} |R|jj |j k\r/||jj k(}|j}d||<n|}||j|}|F||jnd}tj|||jdz|j}||jddz}|i|j|dj }|j#|j|j$}|j'|||}|j)||}t+|x}t,s(t/|j||||||||j0 }|j2d ||||| | | d|d |}t5|j6|j8|j:|j<| Sd S)  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrr)ryT)r)rbrD)rj) rfrhr.rbrmrrrrg)rr.r/r0rr')rqr4rruse_return_dictrrclonerget_seq_lengthr$arangerSryrrrrxrrmasked_scatter isinstancedictrtrainingrrrr.r/r0)r@rrirfrhr.rHrgrbrrmrrrr~r llm_input_idspast_seen_tokensrDcausal_mask_mappingoutputss r)rEzPaliGemmaModel.forward.s}b  -t"; <YZ Z1B1N-TXT_T_TqTq$8$D $++JjJj &1%2J   QU   r() NNNNNNNNNNNNN)r r!r"_checkpoint_conversion_mappingaccepts_loss_kwargsrr9rrrrr$r%rrr1r r LongTensorrTensorrr^r rrErFrGs@r)rrs  '=>N%O"  :8n !-- 9?@R9S  + +  "))":?:K:K"]b]n]n"0.215.204(,262626*.!%)-,0#'v ##d*v ''$.v  t+ v &&- v  v ((4/v ((4/v ((4/v   4'v $;v  $;v #Tkv D[v -.v - -!v v r(rc$eZdZdddddZddiZdeffd Zd Zd Ze d e jd e e fdZee d%de j dzd e jdzde j"dzde j dzdedzde j dzde j dzde jdzde j dzdedzdedzdedzdedzdee j"zd e e deezf dZ d&fd Zeed d!d" d'dede j"de j"dzde j"dedzde j"dzde j"dzd#edzdefd$ZxZS)(!PaliGemmaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightr4ct||t||_t j |j j|j jd|_ |jy)NFr6) r8r9rrrr:rr<rrrr?s r)r9z*PaliGemmaForConditionalGeneration.__init__sS  #F+ yy!3!3!?!?ASASA^A^ejk  r(c6|jjSrC)rrrs r)rz6PaliGemmaForConditionalGeneration.get_input_embeddingsszz..00r(c:|jj|yrC)rrrs r)rz6PaliGemmaForConditionalGeneration.set_input_embeddingss ''.r(rir~c <|jj|fi|SrC)rr)r@rir~s r)rz4PaliGemmaForConditionalGeneration.get_image_featuress,tzz,,\DVDDr(Nrrfrhr.rHrgrbrrmrrrlogits_to_keeprJc >| | n|jj} | | n|jj} | | n|jj} |jd|||||||| | | | d|d |}|d}t |t r t| dn|}|j|dd|ddf}d}| 4|jd|| |jjjd|}t|||j|j|j|j S)rNT) rrirHrfrhr.rbrmrrrrrgr)r-rr)r,r-r.r/r0rr')r4rrrrrr]slicer loss_functionrrr+r.r/r0r)r@rrirfrhr.rHrgrbrrmrrrrr~rr/ slice_indicesr-r,s r)rEz)PaliGemmaForConditionalGeneration.forwardsSb2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD/#33!//)) ' ; ;   r(c t||f|||||| | || d | }|jd |ddz|d<| s| s||d<|S)N) r.rbrfrhrgrmrrHrkrhrri)r8prepare_inputs_for_generationrt)r@rr.rbrgrhrirfrHrmrrrkr~ model_inputsrAs r)rz?PaliGemmaForConditionalGeneration.prepare_inputs_for_generation#s"w<   +')%)))1       N + 7+7+G!+KL ( Y+7L (r(r`rarcrkc t|||||||fd|i|jD cic]\} } | dk7s | | c} } Scc} } w)Nrkri)ritems) r4rbrfrgr.rhrHrkr~kvs r)r z;PaliGemmaForConditionalGeneration.create_masks_for_generatePs`*          2  !' F1!~2Eq!tF  Gs ==)NNNNNNNNNNNNNr) NNNNNNNTNNF)NF)r r!r"r_tied_weights_keysrr9rrrr$r%rrrrrrrr^r]r1r+rEr staticmethodrrrr rFrGs@r)rrs"8-"?#, &" +,VW 1/Eu/@/@EFSeLfEE.215.204(,262626*.!%)-,0#'-.X ##d*X ''$.X  t+ X &&- X  X ((4/X ((4/X ((4/X   4'X $;X  $;X #TkX D[X ell*X +,!X " 0 0#X X z +Z^WO/3*/   ||  t+     llT)   t+ !4K   P r(r)rrr)NNFN)6r#collections.abcr dataclassesrr$r cache_utilsrconfiguration_utilsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_outputsr r modeling_utilsrprocessing_utilsrutilsrrrrrrutils.deprecationrautorconfiguration_paligemmar get_loggerr rurr+Moduler3rr_r%r^rrrrr__all__r'r(r)rsO$!  3)6BS-&14   H %  9#:9  9  9k9  90299%LL4'%\\D(%_%P?K+/-1$&*G4 G4<<G4LL4'G4LL G4 T\ G4 ,,% G4LL4'G4##d*G4G4t G4 G4LG4T ' ' ' x -x  x v z (@/z  z z ^r(