qixddlZddlmZddlmZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZdd lmZdd lmZmZmZmZd d lmZd dlmZeGddeZeedGddeZeGddeZeedGddeZGddee Z gdZ!y)N) dataclass)Any)Cache)GenerationMixin)BaseModelOutputWithPooling ModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check) AutoModel) Glm46VConfigc@eZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ dZy)Glm46VPreTrainedModelconfigmodel)imagevideotextTNpast_key_values)__name__ __module__ __qualname__r__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backend_can_record_outputs\/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/glm46v/modeling_glm46v.pyrr'sC 1&*#"3N!"&r+rzJ Base class for Llava outputs, with hidden states and attentions. ) custom_introceZdZUdZdZej dzed<dZe dzed<dZ e ej dzed<dZ e ej dzed<dZ ejdzed<y)Glm46VModelOutputWithPasta[ past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlast_hidden_stater hidden_states attentions rope_deltas)rrr__doc__r0torch FloatTensorrrrr1tupler2r3 LongTensorr*r+r,r/r/7sv37u((4/6$(OUT\(59M5**+d2926Je''(4/6+/K!!D(/r+r/c jeZdZdZiZdZdZfdZdZdZ d$de de e e e fe jzd e d e d e d ee jzdzf d Z d%de j"de j$de j"dzde j"dzde jdzdee je jff dZee d&de j.de j"dzdeedeezfdZee d&de j.de j"dzdeedeezfdZ d'de j"de j.de j.dzde j.dzfdZ d(de jdzde jdzde jdzde jdzde jdzde jdzde j$dzde jdzfdZee d)de j"dzde jdzd e j"dzdedzde j.dzde jdzde j.dzde j"dzde j"dzd!e j"dzde j$dzd"e j"dzdeedee zfd#Z!xZ"S)* Glm46VModelrFNct||tj|j|_tj|j |_d|_|jyN) super__init__r from_config vision_configvisual text_configlanguage_modelr3 post_initselfr __class__s r,r>zGlm46VModel.__init__WsU  ++F,@,@A '33F4F4FG r+c6|jjSr<)rCget_input_embeddingsrFs r,rIz Glm46VModel.get_input_embeddings`s""7799r+c:|jj|yr<)rCset_input_embeddingsrFvalues r,rLz Glm46VModel.set_input_embeddingscs 007r+start_positiongrid_thwtemp_merge_sizespatial_merge_size time_intervaldevicec|dj|z|dj|z|dj|z} }}|| z|z} tj||| z|j||z} tj|||z|j | |z} tj | f||tj } | |z} tj| | | gd}|S)a Compute 3D positional indices for vision tokens derived from a single image or video input. The positions are generated from the input grid defined by temporal (T), height (H), and width (W) dimensions. Temporal and spatial dimensions can be downscaled according to the merge sizes used in the vision backbone. The resulting positions are offset by `start_position`. Args: start_position (`int`): Offset added to all computed positional indices. grid_thw (`Sequence[int]` or `torch.Tensor` of shape `(3,)`): The (T, H, W) grid representing the feature layout of the current image or video after patch embedding. temp_merge_size (`int`, *optional*): Factor by which the temporal dimension is reduced in the backbone. The temporal grid size is divided by this value. Defaults to 1. spatial_merge_size (`int`, *optional*): Factor by which the spatial dimensions (H and W) are reduced in the backbone. Both H and W are divided by this value. Defaults to 1. time_interval (`int`, *optional*): Spacing factor applied between consecutive temporal position indices.Defaults to 1. device (`str` or `torch.device`, *optional*): Device on which the resulting tensor is allocated. If `None`, uses the current default device. Returns: torch.LongTensor of shape (3, sequence_length): Positional indices for temporal, height, and width dimensions, flattened into sequence form and offset by `start_position`. rrrrT)rTdtypedim)itemr5arangerepeatrepeat_interleavefulllongstack)rFrOrPrQrRrSrT llm_grid_t llm_grid_h llm_grid_wimage_seq_lengthposition_widthposition_heightposition_temporalvision_position_idss r,get_vision_position_idsz#Glm46VModel.get_vision_position_idsfsL QK   / 1 QK   "4 4 QK   "4 4!+J & 2Z?nnz6QZ`ahh  #  ,,~~ 7R[abtt  # "JJ(8':NSYafakakl- =#kk+z,Glm46VModel.get_rope_index..s `abc`dr+rVrrX)rr@rRr5zerosshaperWrTiter enumeratebool itertoolsgroupbytolistlistappendr[viewexpandnextrimaxcatreshapetolentensor unsqueeze)rFrjrkrlrmrnkwargsrRmrope_position_deltas position_ids grid_iters batch_idxcurrent_input_idsinput_token_typeinput_type_groupkeygroup start_index end_index current_posvideo_group_indexllm_pos_ids_list modality_type start_idxend_idxtext_lenrPrQrh llm_positionss r,get_rope_indexzGlm46VModel.get_rope_indexsjb"[[66II "{{ OOA  OOA //##  (6'AtN#t'5'AtN#t -6i,@1 [ (I(0; )$5nY6O6T6T6V$W!#3N94M4R4R4T#U ! '// :J:Q:Q:S0TVde G UU #Ahqk !"IaL1,  ''k9(EF  G K ! ! 5E W1 y' A%&2H$++ Xi6F6FGLLQPRSZZ[\^`adoo 8+K %),1'+J},E'FH)Q.)1Bhqk1QAWh)#' =(A#B'/qkO*.*F*F#X@R[d[k[k+G+'%++,?@3x{HQK#@DV#VVK7 W8"II&6A>FFq"MM)O\O_O_`l`s`sOt Q >)+D+I+I+KKL-:-=-=l>Q>Q-R Q \* ! ( ():):)#?#?M '3$r+ pixel_valuesc <|j|jj}|j|f|dd|}|jd|jjdzzj }t j|j|}||_|S)T pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. Trrur) rrArWrrRr}r5rr)rFrrlrrr image_embedss r,get_image_featureszGlm46VModel.get_image_features2s$(():):; $\gNX\g`fg%**2.$++2P2PRS2SS[[] {{>#?#?M '3$r+ inputs_embedsimage_featuresvideo_featurescT|||jtj|jjtj |j k(}|jd}||jtj|jjtj |j k(}|jd}n2||jjk(}||jjk(}|j}|jdj|j|j }|At||j|jk(d|d|jd|j}|jdj|j|j }|At||j|jk(d|d|jd||fS)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rqruz6Image features and image tokens do not match, tokens: z , features: rz6Video features and video tokens do not match, tokens: )rIr5rrimage_token_idr_rTallvideo_token_idsumr expand_asrrnumelrw) rFrjrrrspecial_image_maskspecial_video_maskn_image_tokensn_video_tokenss r,get_placeholder_maskz Glm46VModel.get_placeholder_maskHs  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; "+dkk.H.H!H !*dkk.H.H!H +//1/99"=GG VYYZgZnZno  % "01779^=Q=Q=SSHHXXdeseyeyz{e|d}~  ,//1/99"=GG VYYZgZnZno  % "01779^=Q=Q=SSHHXXdeseyeyz{e|d}~ "#555r+rcL|dn|j}|duxr|duxr |duxs|du} | r3|j|dk(r"|j|||||\} } | |_| S|j5|j\} } }|u|j j ddz } | j |dk(d} | jd| djdddj|j} nVtj||| z} | jdddjd| dj|j} |jj| |jjdzd}| |j|jz} | Sd} | S)Nr)rlrmrnrkrurrrXrV)get_seq_lengthr3rrwr_cumsum masked_fillrr\rrTr5r[rr])rFrjrrlrmrnrrkpast_key_values_lengthcan_compute_mroperr3 batch_size seq_length_deltas r,compute_3d_position_idsz#Glm46VModel.compute_3d_position_idsrs'6&=?CaCaCc T ! K!- Kt+I~T/I  $"2"2":>TXY>Y(,(;(;---"3 )<) %L+ +D     )(5(;(; %J A)-224;;B?!C +77!8KQO +00JCJJ1aQRSVVWdWkWkl $||,BDZ]gDgh +00Ar:AA!ZQSTWWXeXlXlm $$66zTEUEUE[E[\]E^7^de6fE'%((-:N:N(*OOL Lr+rr3cache_positionc |du|duz r td||j|}||j||dj}t j |dj |j|j}|j|||\}}|j||}||j|| dj}t j |dj |j|j}|j|||\}}|j||}||j||| |||| }|jd d||||| d | }td i|d |jiS) a image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nz:You must specify exactly one of input_ids or inputs_embedsT)rrrX)r)r)rjrlrmrrnrrk)rjrrnrrrr3r*) ValueErrorrIrrr5rrrTrWrmasked_scatterrrrCr/r3)rFrjrnrrrrrrlrmr3rkrrr image_maskrr video_maskoutputss r,forwardzGlm46VModel.forwards4  -t"; <YZ Z  7D557 BM  #22<]a2bppL 99\q9<<]=Q=QS`SfSfgL 55i_k5lMJ)88\RM  *223Fdh2iwwL 99\q9<<]=Q=QS`SfSfgL 55i_k5lMAz)88\RM  77#--+- /"38L&$%% %)+')   )  ((  r+)rrrN)NNNr<)NN)NNNNN) NNNNNNNNNNNN)#rrrr _checkpoint_conversion_mappingaccepts_loss_kwargsr#r>rIrLintr~r5TensorstrrTrir8 IntTensorr7rrr r6r r rrrrrrr/r __classcell__rGs@r,r:r:OsQ%'":8 !"#,06#6#sC}% 46# 6#  6#  6#ell"T)6#x3726.2 s3##s3!??s3((4/ s3 ((4/ s3  t+ s3 u||U\\) *s3j37"..((4/+,  + + :37''((4/+,  + + 04837 (6##(6(((6))D0 (6 ))D0 (6\/3.2.2/348)<<$&)||d*) t+ )  t+ )  t+ ),)!??T1)  )V.2.204(,26,08<2626/34826B ##d*B  t+B &&- B  B ((4/ B llT)B #..5B ((4/B ((4/B %%,B !??T1B ((4/B +,B  * *B B r+r:zR Base class for Glm46V causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZejdzed<y) Glm46VCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlosslogitsrr1r2r3)rrrr4rr5r6rrrrr1r7r2r3r8r*r+r,rrs &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/6+/K!!D(/r+rc"eZdZiZddiZdZfdZdZdZe d de jd e jdzd e ed eezfd Ze d d e jde jdzd e ed eezfdZee d!de jdzde j(dzde jdzdedzde jdzde jdzd e j(dzde jdzde jdzd e jdzde j,dzde jdzdee j(zd e ed eezfdZ d"fd ZfdZ d de jdzde j(dzd ee j(e j(ffdZ d#dedede jdzd ee jeee fffdZ!xZ"S)$Glm46VForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightFct||t||_t j |j j|j jd|_ |jy)NF)bias) r=r>r:rnnLinearrB hidden_size vocab_sizelm_headrDrEs r,r>z'Glm46VForConditionalGeneration.__init__sS   ( yy!3!3!?!?ASASA^A^ejk  r+c6|jjSr<)rrIrJs r,rIz3Glm46VForConditionalGeneration.get_input_embeddingsszz..00r+c:|jj|yr<)rrLrMs r,rLz3Glm46VForConditionalGeneration.set_input_embeddingss ''.r+Nrrmrroc @|jjd||d|S)r)rrmr*)rr)rFrrmrs r,rz1Glm46VForConditionalGeneration.get_video_featuress/-tzz,, 3N V\  r+rrlc @|jjd||d|S)r)rrlr*)rr)rFrrlrs r,rz1Glm46VForConditionalGeneration.get_image_features%s'-tzz,,p,Wepioppr+rjrnrrrlabelsrkrlogits_to_keepc |jd|||| | | ||||| d |}|d}t| tr t| dn| }|j |dd|ddf}d}|2|j |||j jj}t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Glm46VForConditionalGeneration >>> model = Glm46VForConditionalGeneration.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) rjrrrlrmrkrrnrrrrN)rrr)rrrr1r2r3r*)r isinstancerslicer loss_functionrrBrrrr1r2r3)rFrjrnrrrrrrrlrmrkrrrrr1 slice_indicesrrs r,rz&Glm46VForConditionalGeneration.forward4sz$**  % 3))/%)+')      9C>SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD+#33!//))++   r+c  \t||f||||||| | | || d | }| s |r d|d<d|d<|S)N) rrnrrrrrrlrm use_cacheis_first_iterationrr)r=prepare_inputs_for_generation)rFrjrrnrrrrrrrlrmrr model_inputsrGs r,rz#5#5#7,:J:J:L#LRST "Q % 6),,,3 %))a)0 \))r+ expand_sizeis_encoder_decoderc dk(rfSgdfd}fd}|jd||r*jd td|dd<fS) Nr)rrlrrmsecond_per_grid_tsc4 jdd} jdd} j jdd\}}d}|D]9}|dk(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<l|dk(rt |} |||| ||<|d k(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<|dk(rt |} |||| ||<|d k(s |||t | ||<<|Scc}wcc}w) Nrlrmr)rctj||}|gdg|jdz zz}tj|Dcgc]}|j|c}d}|Scc}w)NrrrX)r5rrYrr\)rslengths repeat_timessamples repeat_argssampleresults r,_repeat_interleave_sampleszGlm46VForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samples1sa++a1+nsaeegk/BB g#VFMFMM;$?#V\]^ $WsA&rrrX)rr rr)rrr5rr~rr)dict_to_expandrlrm image_nums video_numsr%rr!r#rrrjrrFs r,"_expand_dict_for_generation_visualzhGlm46VForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual*s)--.>EN)--.>EN%)%H%H)9)9/4)P&I& "J   & .(#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'11#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'00*D&s+T*5ET_+N3'7 <" !3VVs =,F,Fc|D]{}|dk(r,||jdk(r||jd||<4|dk7s:||@t||tjs^|vsc||jd||<}|S)NrrrrXrr)ndimr]rr5r)r&rr visual_keyss r,_expand_dict_for_generationzaGlm46VForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generationWs% d.(^C-@-E-E-J*8*=*O*OP[ab*O*cN3'++&s+7">##6 E;.*8*=*O*OP[ab*O*cN3' d" !r+rrXencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)r]rr)rFrrrjrr)r-r,s`` `` @r,_expand_inputs_for_generationzrIrLr r5r6r8r r r7rrrrrrrrrrrrrrzdictrrr/rrs@r,rrs9%'"*,VW1/37 ".. ((4/ +,  + +   37 q'' q((4/ q+, q + + q q.2.204(,26*.,08<26264826-.[ ##d*[  t+[ &&- [  [ ((4/ [   4'[ llT)[ #..5[ ((4/[ ((4/[ !??T1[ ((4/[ ell*[ +,[ - -![ [ @  &P$R.26*##d*6*||d*6* u||U\\) * 6*t#(-1 W'W'!W'##d* W' uc3h/ 0 W'r+r)r:rr)"r{ dataclassesrtypingrr5torch.nnr cache_utilsr generationrmodeling_outputsrr modeling_utilsr processing_utilsr utilsr r rrautorconfiguration_glm46vrrr/r:rr__all__r*r+r,r>s,!  )G-&aa. O    0 0  0$Q 'Q Q h   0;0  0.o'%:Oo'd Ur+