qi3TddlZddlmZddlmZmZmZmZmZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZd d lmZej0eZGd deZGddeZGddej:ZGddeZGddeZ GddeZ!gdZ"y)N)nn)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)ACT2FN)Cache)BaseModelOutputWithPooling)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)VipLlavaConfigc eZdZy)VipLlavaModelOutputWithPastN__name__ __module__ __qualname___/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/vipllava/modular_vipllava.pyrr&rrc eZdZy)VipLlavaCausalLMOutputWithPastNrrrrrr*rrrc*eZdZdeffd ZdZxZS)VipLlavaMultiModalProjectorconfigcHt|t|jtrdnt |j}t j||jjz|j|_ t j||jjz|jjd|_t|j |_t j|jj|jjd|_y)Nr)epsT)bias)super__init__ isinstancevision_feature_layersintlenr LayerNorm vision_config hidden_sizeprojector_layernorm_epsprojector_layernormLinear text_configlinear_1r projector_hidden_actactlinear_2)selfr"num_feature_layers __class__s rr'z$VipLlavaMultiModalProjector.__init__/s ",V-I-I3"OQUXY_YuYuUv#%<< !5!5!A!A AvGeGe$   !5!5!A!A A    * *  &556 &"4"4"@"@&BTBTB`B`gkl rc|j|}|j|}|j|}|j|}|SN)r0r3r5r6)r7 hidden_statess rforwardz#VipLlavaMultiModalProjector.forward>sB00?  m4 /  m4 r)rrrrr'r= __classcell__)r9s@rr!r!.s m~ mrr!c eZdZy)VipLlavaPreTrainedModelNrrrrr@r@Frrr@ceZdZeed ddej deeezdzde dzde e de e zf d Ze dd ejdzdej dzd ej dzd ejdzd edzdej dzdeeezdzde dzde dzde dzde dzdejdzde ezfdZy) VipLlavaModelzWObtains image last hidden states from the vision tower and apply multimodal projection.) custom_introN pixel_valuesr)output_hidden_stateskwargsreturnc j||n|jj}|j|fddd|}t|tr|j |ddddf}n<|Dcgc]}|j |ddddf}}t j|d}|j|}||_ |Scc}w)\ pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. NT)rE return_dictr)dim) r"r) vision_towerr(r*r<torchcatmulti_modal_projector pooler_output)r7rDr)rErF image_outputsimage_featuresindexs rget_image_featuresz VipLlavaModel.get_image_featuresKs&&;%F !DKKLmLm *))  !%    +S 1*889NOPQSTSUPUVNVkkEm99%@ABGkNk"YY~2>N33NC&4 # lsB0 input_idsattention_mask position_idspast_key_values inputs_embeds use_cacheoutput_attentionsrJcache_positionc | | n|jj} | | n|jj} | | n|jj} ||n|jj}|du|duz r t d||j |}|j|j||dj}|j|j|j}|j|||}|j||}|jd|||||| | d| d | }t|j |j"|j$|j&|nd}| r|S|j)S) z vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedsT)rDr)rJ)rZrS) rWrXrYrZr[r\rErJr])last_hidden_staterYr< attentionsimage_hidden_statesr)r"r\rEuse_return_dictr) ValueErrorget_input_embeddingsrUrQtodevicedtypeget_placeholder_maskmasked_scatterlanguage_modelrr_rYr<r`to_tuple)r7rVrDrWrXrYrZr)r[r\rErJr] lm_kwargsrSspecial_image_maskoutputsoutputs rr=zVipLlavaModel.forwardts,2C1N-TXT_T_TqTq$8$D $++JjJj &1%2JPT  %v;&//*;;r)NN) NNNNNNNNNNNN)rrrrrrN FloatTensorr*listboolr rtupler rU LongTensorTensorr rr=rrrrBrBJsn 9=,0 #''# #T#Y5##Tk # +, # + + ##J.215.204(,268>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) rVrDrWrXrYrZr[r)r\rErJr]r)logitsrz vocab_size)lossr}rYr<r`rar)r"r\rErbr)ryr(r*slicelm_head loss_functionr2r~rrYr<r`ra)r7rVrDrWrXrYrZr)rzr[r\rErJr]r{rlrnr< slice_indicesr}rs rr=z(VipLlavaForConditionalGeneration.forwards[l2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD-#33!//)) ' ; ;   rr;)NNNNNNNNNNNNNr)rrrrrNrpr*rqr rrsr rUrtrur rrrr=rrrrwrws9= ''  #T#Y5 +,  + +   &.215.204(,268<*.!%)-,0#'26-._ ##d*_ ''$._  t+ _ &&- _  _ ((4/_  #T#Y5_   4'_ $;_  $;_ #Tk_ D[_ ((4/_ ell*_ " / /#_ rrw)rBrwr@)#rNr(transformers.models.llava.modeling_llavarrrrr activationsr cache_utilsr modeling_outputsr processing_utilsr utilsrrr utils.genericrconfiguration_vipllavar get_loggerrloggerrrModuler!r@rBrw__all__rrrrs " :&@@-2   H % ":  %@ "))0 2 m