qi+ddlZddlZddlmZmZmZddlmZddlmZ ddl m Z ddl m Z mZmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZddlmZej@e!Z"GddejFZ$GddejFZ%GddejFZ&GddejFZ'GddejFZ(GddejFZ)GddeZ*GddejFZ+Gd d!ejFZ,Gd"d#ejFZ-Gd$d%ejFZ.Gd&d'ejFZ/Gd(d)eZ0Gd*d+e0Z1Gd,d-e0eZ2gd.Z3y)/N)Tensordevicenn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)apply_chunking_to_forward)logging)BlipTextConfigc eZdZdZfdZ d dej dzdej dzdejdzdedejf d Z xZ S) BlipTextEmbeddingsz;Construct the embeddings from word and position embeddings.c,t|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|jdt!j"|jj%dd||_y)N) padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandconfigselfr3 __class__s ]/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/blip/modeling_blip_text.pyr"zBlipTextEmbeddings.__init__-s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej   N input_idsr inputs_embedspast_key_values_lengthreturnc*||j}n|jdd}|d}||jdd|||zf}||j|}|}|j|}||z }|j |}|j |}|S)Nrr)sizerr'r)r*r.) r5r9rr:r; input_shape seq_length embeddingsr)s r7forwardzBlipTextEmbeddings.forward<s  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL   00;M" "66|D)) ^^J/ \\*- r8)NNNr) __name__ __module__ __qualname____doc__r"r0 LongTensor FloatTensorintrrB __classcell__r6s@r7rr*ssE ".20426&' ##d*&&-((4/  !$   r8rceZdZdfd ZdZdZdZdZ ddejdejdzd ejdzd ejdzd e dzd e dzd ejdzde ejfdZxZS)BlipTextSelfAttentionNct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ ||_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj"|j$|_y)Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d))r!r"r3r%num_attention_headshasattr ValueErrorrIattention_head_size all_head_size layer_idxrLinearqueryencoder_hidden_sizekeyvaluer,attention_probs_dropout_probr.r5r3is_cross_attentionrUr6s r7r"zBlipTextSelfAttention.__init__\s`     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9Ffz}|j|}||fS) NrrFrrT)dimrr) shaperWviewrPrS transpose isinstancer is_updatedgetrUcross_attention_cacheself_attention_cachelayerskeysvaluesrYrZupdater0matmulmathsqrttorrSoftmaxr.permute contiguousr>rT)r5rlrmrnrorprqrr batch_sizer@_ query_layerr]r{curr_past_key_valuescurrent_states key_layer value_layerattention_scoresattention_probsattention_probs_dropped context_layernew_context_layer_shapes r7rBzBlipTextSelfAttention.forwards%2$7$7! J JJ} % T*b$":":D3E/>  &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$2D.- /"=*,33DNNCHHI.55dnnELLK(j"d&>&>@X@XY1a   >*j"d&>&>@X@XY1a  *7It)=)D)D{DNN=M~<^*& ;&*_FY*ZAEO..t~~>!<< Y5H5HR5PQ+dii8P8P.QQ  %/.2C2CDTD[D[2\\ -"**,-=>#',,"? %S"S* **,CD o--r8r_NNNNFN)rCrDrEr"rbrerirkr0rrHr booltuplerBrJrKs@r7rMrM[sG0-#+" 48:>;?(,)..2S.||S.))D0S. %0047 S. !& 1 1D 8 S.  S. $;S. t+S. u|| S.r8rMcneZdZfdZdej dej dej fdZxZS)BlipTextSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr) r!r"rrVr%denser*r+r,r-r.r4s r7r"zBlipTextSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r8rl input_tensorr<cr|j|}|j|}|j||z}|Sr_rr.r*r5rlrs r7rBzBlipTextSelfOutput.forward7 =1  ]3 }|'CD r8rCrDrEr"r0rrBrJrKs@r7rr1> U\\RWR^R^r8rceZdZd fd Z d dej dej dzdej dzdedzdedzdej dzd e ej fd Z xZ S) BlipTextAttentionNcht|t||||_t ||_y)NrU)r!r"rMr5routputr\s r7r"zBlipTextAttention.__init__s, )&2DPYZ (0 r8rlrmrnrprqrrr<cp|j||||||}|j|d|}|f|ddz} | S)Nrmrnrprqrrrr)r5r) r5rlrmrnrprqrr self_outputsattention_outputoutputss r7rBzBlipTextAttention.forwardsYyy )"7+/) !  ;;|A F#% QR(88r8)FN)NNNFN) rCrDrEr"r0rrHr rrrBrJrKs@r7rrs148:>(,)..2||))D0 %0047     $;  t+ u|| r8rcVeZdZfdZdej dej fdZxZS)BlipTextIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr_) r!r"rrVr%intermediate_sizerrz hidden_actstrr intermediate_act_fnr4s r7r"zBlipTextIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r8rlr<cJ|j|}|j|}|Sr_)rrr5rls r7rBzBlipTextIntermediate.forward s& =1 00? r8rrKs@r7rrs#9U\\ellr8rcneZdZfdZdej dej dej fdZxZS)BlipTextOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r!r"rrVrr%rr*r+r,r-r.r4s r7r"zBlipTextOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r8rlrr<cr|j|}|j|}|j||z}|Sr_rrs r7rBzBlipTextOutput.forwardrr8rrKs@r7rrrr8rceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd ej dzd e ej fd Z d Z xZ S) BlipTextLayercLt|||_|j|_d|_t |||_||_|jjr't ||jj||_ t||_ t||_ y)Nrr)r]rU)r!r"r3chunk_size_feed_forward seq_len_dimr attention layer_num is_decodercrossattentionr intermediaterr)r5r3rr6s r7r"zBlipTextLayer.__init__"s  '-'E'E$*6YG" ;; ! !"34;;+A+AY#D 18$V, r8Nrlrmrnrorprqrrr<c|j|||||}|d} |dd} |$|j| |||||} | d} | | ddz} t|j|j|j | } | f| zS)N)rmrqrprrrrr)rrrfeed_forward_chunkrr) r5rlrmrnrorprqrrself_attention_outputsrrcross_attention_outputs layer_outputs r7rBzBlipTextLayer.forward0s"& )/+) "0" 2!4(, ,&*&9&9 5&; /"3- ':' # 7q9  7 ;;G0  # #T%A%A4CSCSUe ((r8cL|j|}|j||}|Sr_)rr)r5rintermediate_outputrs r7rz BlipTextLayer.feed_forward_chunkTs,"//0@A{{#68HI r8r) rCrDrEr"r0rrHr rrrBrrJrKs@r7rr!s -"48:>;?(,)..2")||")))D0") %0047 ") !& 1 1D 8 ")  ") $;") t+") u|| ")Hr8rc eZdZfdZ ddej dej dzdej dzdej dzdedzdedzd edzd edzd edzd ej dzd e ej e zfdZ xZ S)BlipTextEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}w)NF) r!r"r3r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r5r3ir6s r7r"zBlipTextEncoder.__init__\sP  ]]eFLdLdFe#fM&!$<#fg &+#$gsA$Nrlrmrnrorp use_cacherqoutput_hidden_states return_dictrrr<c |jr%|jr|rtjdd}|rgt |t r!t |t |j}n6|4t t |jt |j}|rdnd} |rdnd} |r|dnd} t|jjD]D}|j|}|r| |fz} |||||||| }|d}|s0| |dfz} |<| |dfz} F|r| |fz} | std||| | | fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r3rrrtc3$K|]}|| ywr_r).0vs r7 z*BlipTextEncoder.forward..s = s)last_hidden_staterprl attentionscross_attentions) rtrainingloggerwarningrzr r r3rrrrr)r5rlrmrnrorprrqrrrrall_hidden_statesall_self_attentionsall_cross_attentionsr layer_module layer_outputss r7rBzBlipTextEncoder.forwardbs  & &4==p" /<8"5o|[_[f[fGg"h ("5  4l$++6V##7BD$5b4%6;P;\rbft{{445 VA::a=L#$58H$H!(%&!M*!,M &9]1=M;?(,!%).,1#'.2L ||L ))D0L  %0047 L !& 1 1D 8 L  L $;L  $;L #TkL D[L  t+L  u|| H HL r8rcVeZdZfdZdej dej fdZxZS)BlipTextPoolerct|tj|j|j|_tj |_yr_)r!r"rrVr%rTanh activationr4s r7r"zBlipTextPooler.__init__s9 YYv1163E3EF '')r8rlr<c\|dddf}|j|}|j|}|S)Nr)rr)r5rlfirst_token_tensor pooled_outputs r7rBzBlipTextPooler.forwards6+1a40 #56  6 r8rrKs@r7rrs#$ U\\ellr8rcVeZdZfdZdej dej fdZxZS)BlipTextPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r!r"rrVr%rrzrrr transform_act_fnr*r+r4s r7r"z(BlipTextPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr8rlr<cl|j|}|j|}|j|}|Sr_)rrr*rs r7rBz'BlipTextPredictionHeadTransform.forwards4 =1 --m< }5 r8rrKs@r7rrs$UU\\ellr8rc$eZdZfdZdZxZS)BlipTextLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r!r"r transformrrVr%r$decoder Parameterr0zerosrr4s r7r"z!BlipTextLMPredictionHead.__init__s[ 8@yy!3!3V5F5FTR LLV->->!?@ r8cJ|j|}|j|}|Sr_)rrrs r7rBz BlipTextLMPredictionHead.forwards$}5  ]3 r8)rCrDrEr"rBrJrKs@r7rrsAr8rcVeZdZfdZdej dej fdZxZS)BlipTextOnlyMLMHeadcBt|t||_yr_)r!r"r predictionsr4s r7r"zBlipTextOnlyMLMHead.__init__s 3F;r8sequence_outputr<c(|j|}|Sr_)r)r5rprediction_scoress r7rBzBlipTextOnlyMLMHead.forwards ,,_=  r8rrKs@r7rrs#<!u||! !r8rc6eZdZUdZeed<dZgZfdZxZ S)BlipTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r3bertct||t|trZt j |j tj|j jdjdyy)Nrr) r! _init_weightsrzrinitcopy_rr0r1rwr2)r5moduler6s r7r z%BlipTextPreTrainedModel._init_weightss[ f% f0 1 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h 2r8) rCrDrErFr__annotations__base_model_prefix_no_split_modulesr rJrKs@r7rrs( iir8rc eZdZdZdfd ZdZdZdedee de de d ef d Z dd e jd zde jd zd e jd zde jd zde jd zde jd zde jd zded zde d zde d zde d zde d zde d zde jd zd ee jezfdZxZS) BlipTextModela& The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and `is_decoder` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ct||||_t||_t ||_|r t|nd|_|jyr_) r!r"r3rrArencoderrpooler post_init)r5r3add_pooling_layerr6s r7r"zBlipTextModel.__init__ sI   ,V4&v. 0AnV,t  r8c.|jjSr_rAr'rds r7get_input_embeddingsz"BlipTextModel.get_input_embeddingss...r8c&||j_yr_r)r5rZs r7set_input_embeddingsz"BlipTextModel.set_input_embeddingss*/'r8rmr?rrr<c|jdk(r|dddddddf}n5|jdk(r|r|\}}tj||}|ddddfj||d|ddddfk} | j |j } | j d|j dkr[|j d| j dz } tjtj||| f|| j | gd} | dddddddf|ddddddfz}n*|ddddddf}ntd |d |j d |j |j }d |z dz}|S)a= Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. rNrtrr)rdtyper)axisz!Wrong shape for input_ids (shape z) or attention_mask (shape ))r g?g) rvr0r1repeatrr rwcatonesrR) r5rmr?rrextended_attention_maskrr@seq_ids causal_maskprefix_seq_lens r7get_extended_attention_maskz)BlipTextModel.get_extended_attention_masks&    1 $&4Qa]&C #    !Q &)4& J,,z&A%dD!m4;;J TUVZabfhikoboZpp )nn^-A-AB $$Q'.*>*>q*AA%3%9%9!%<{?P?PQR?S%SN"'))!JJ!+Z HQW_j_p_p(   #K+6aq!m*D~VWY]_cefVfGg*g'*8D$9I*J'3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&r8Nr9rr:encoder_embedsrnrorprrqrrrrc d| | n|jj} | | n|jj} | | n|jj} | r| | n|jj} nd} | | t d|4|j |||j}|\}}|j}nY|%|jdd}|\}}|j}n2|%|jdd}|\}}|j}n t d|dn|j}|)tj|||zfj|}|j|||| }|t|tr|dj\}}}n|j\}}}||f}t|tr|Dcgc]}|j!|}}n?|)tj||}|j!|}n|j!|}nd}||j#||||}n|}|j%|||||| | | | | }|d}|j&|j'|nd}| s ||f|d dzSt)|||j*|j,|j.|j0 Scc}w) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Cache`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timerzGYou have to specify either input_ids or inputs_embeds or encoder_embedsrr)r9rr:r;) rmrnrorprrqrrrrr)r pooler_outputrprlrr)r3rqruse_return_dictrrR%warn_if_padding_and_no_attention_maskr>rget_seq_lengthr0r%rr*rzlistinvert_attention_maskrArrrrprlrr) r5r9rmrr:r+rnrorprrqrrrrrkwargsr?rr@rr;r&encoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapemaskencoder_extended_attention_maskembedding_outputencoder_outputsrrs r7rBzBlipTextModel.forwardXs&H2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K%0 "J %%F  &',,.s3K%0 "J "))F  '(--/4K%0 "J #**Ffg g&5&=?CaCaCc  !"ZZZBX5X(YZ]]^deN150P0P K1  ! ,/6AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2wX\43M3Md3S2w/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +  !##)+'= /   . ,, 2"7#B+/!5#)'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   K3xs5J-)T)NNNNNNNNNNNNFN)rCrDrErFr"rrrrrIrrr*r0r rrBrJrKs@r7rrs/0<'$<'38:<'GM<'[_<' <'@*..2,0-1.2596:(,!%)-,0#'"'.2B <<$&B  t+B llT) B ||d* B  t+ B  %||d2B !& t 3B B $;B  $;B #TkB D[B 4KB  t+B " u|| K K#B r8rc&eZdZdddZfdZdZdZdZdZ dd e jd zd e jd zd e jd zd e jd zde jd zde jd zde jd zde d zde d zde d zde d zde d zde d zde d zde d zde jd zdee jzdee jezf$dZdfd ZxZS) BlipTextLMHeadModelzcls.predictions.biasz&bert.embeddings.word_embeddings.weight)zcls.predictions.decoder.biaszcls.predictions.decoder.weightct||t|d|_t ||_|j |_|jy)NF)r)r!r"rr rclslabel_smoothingrr4s r7r"zBlipTextLMHeadModel.__init__sB  !&EB &v.%55 r8c6|jjSr_)r rrds r7rz(BlipTextLMHeadModel.get_input_embeddingssyy--//r8c:|jj|yr_)r rr5new_embeddingss r7rz(BlipTextLMHeadModel.set_input_embeddingss &&~6r8cB|jjjSr_)r>rrrds r7get_output_embeddingsz)BlipTextLMHeadModel.get_output_embeddingssxx##+++r8c||jj_|j|jj_yr_)r>rrrrBs r7set_output_embeddingsz)BlipTextLMHeadModel.set_output_embeddingss,'5$$2$7$7!r8Nr9rmrr:rnrolabelsrprrqrr return_logitsr reductionrrlogits_to_keepr<c | | n|jj} |d} |j|||||||| | | | || }|d}t|tr t | dn|}|j |dd|ddf}| r|ddddddfjSd}||ddddddfj}|ddddfjj|j}t||j}||jd|jj|jd}|dk(r0|j|jddjd}| s|f|d dz}||f|zS|St!|||j"|j$|j&|j( S) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` past_key_values (`Cache`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NF) rmrr:rnrorprrqrrrrrrrr)rJr?nonert)losslogitsrprlrr)r3r.r rzrIslicer>rrrrr?rxr$r>sumrrprlrr)r5r9rmrr:rnrorHrprrqrrrIrrJrrrKr3rrl slice_indicesrlm_lossshifted_prediction_scoresloss_fctrs r7rBzBlipTextLMHeadModel.forwardsV&1%SV8W~ot4]k  HH]1mQ3F%GH $QQY/::< <  (9!SbS!)(D(O(O(Q %AqrE]--/223L3S3STF')TMaMabH8==b$++BXBXY[a[f[fgi[jkGF"!,,'8'='=a'@"EII!L')GABK7F,3,?WJ' KV K0$#33!//))$55   r8c 8t||f||d|}d|d<|S)N)rprmTr)r!prepare_inputs_for_generation)r5r9rprm model_kwargs model_inputsr6s r7rWz1BlipTextLMHeadModel.prepare_inputs_for_generationXs>w<  +)   &* \"r8)NNNNNNNNNNNNFTmeanNr)NN)rCrDrE_tied_weights_keysr"rrrErGr0rr rrrIrrrBrWrJrKs@r7r<r<s(>*R 07,8 *..2,0-1596:&*(,!%)-,0#'%*"& &.2-.%\ <<$&\  t+\ llT) \ ||d* \ %||d2 \ !& t 3\  t#\ \ $;\  $;\ #Tk\ D[\ d{\ 4K\ :!\ " t+#\ $ell*%\ ( u|| @ @)\ |  r8r<)rr<r)4rr0rrrtorch.nnrrr  activationsr cache_utilsr r r generationr modeling_layersrmodeling_outputsrrrmodeling_utilsr pytorch_utilsrutilsrconfiguration_blipr get_loggerrCrModulerrMrrrrrrrrrrrrr<__all__rr8r7rjsQ  $$%&!CC)9 .6.   H %--bx.BIIx.x   : 299  RYY 6.6tS biiS n RYY  bii$ ryy "!"))! io i"Y +Y zE1?EP Nr8