qijdZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z mZddlmZdd lmZmZmZdd lmZdd lmZmZd d lmZej8eZdZdZ d"dZ!GddejDZ#dZ$GddejDZ%eGddeZ&eGdde&Z'edGdde&eZ(edGdd e&Z)gd!Z*y)#zPyTorch CTRL model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)auto_docstringlogging) CTRLConfigcPdtjdd|dzz|z z }||zS)Nri')torchpow)posi d_model_size angle_ratess X/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr%s/eiiQ!V 'DEEK  cttj|tjj |j dtj|tjj |j d|}tj |dddddf}tj|dddddf}tj||gd}|S)Ndtyperrrdim) rrarangeint64to unsqueezesincoscat)positionrr! angle_radssinescosines pos_encodings rpositional_encodingr1*s XU[[144U;EEaH \588?II!LJ IIjADqD) *Eii 1add7+,G99eW-26L rc tj||jdddd}|jd}|t j |z }|6|j d|j d} }||| |z | d| fdzz }|||z}tj|d} tj| |} | | fS) Nrrrrr"gr#)rmatmulpermuteshapenpsqrtsizesoftmax) qkvmaskattention_mask matmul_qkdkscaled_attention_logitsndnsattention_weightsoutputs rscaled_dot_product_attentionrG9s Q !Q1 56I B'"''"+5 (--b13J3O3OPR3SB4R" crc(9#:T#AA!"9N"J &=2F \\+Q /F $ $$rc8eZdZdfd ZdZ ddZxZS)MultiHeadAttentionc^t|||_||_||_t ||jz |_tj|||_ tj|||_ tj|||_ tj|||_ yN) super__init__ num_headsr layer_idxintdepthrLinearWqWkWvdense)selfrrNrO __class__s rrMzMultiHeadAttention.__init__Ps "("67 ))L,7))L,7))L,7YY|\: rcx|j|d|j|j}|jgdS)Nr"rrrr)reshaperNrQr5)rWx batch_sizes rsplit_into_headsz#MultiHeadAttention.split_into_heads^s- IIj"dnndjj Ayy&&rc |jd} |j|}|j|}|j|}|j || }|j || }|j || }|#|j |||j d| i\}}t|||||} | djgd} | d} | j| d|j}|j|} | | fS)Nrcache_positionrZrr") r6rSrTrUr^updaterOrGr5r[rrV)rWr=r<r;r> layer_pastr? use_cacheoutput_attentionsr`r]rFscaled_attentionattnoriginal_size_attentions rforwardzMultiHeadAttention.forwardbsWWQZ GGAJ GGAJ GGAJ  ! !!Z 0  ! !!Z 0  ! !!Z 0  !$$Q4>>eps) rLrMrImulti_head_attentionrsffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)rWrrNrrraterOrXs rrMzEncoderLayer.__init__sr $6|YZc$d!2<E,,|>,,|> 4(  4( rc |j|}|j||||||||| } | d} |j| } || z} |j| } |j | } |j | } | | z} | f| ddz}|S)Nrbr?rcrdr`rr)r}rzrr~r{r)rWr\r>rbr?rcrdr`normed attn_outputs attn_outputout1out2 ffn_outputoutputss rrhzEncoderLayer.forwards#00    !)/)1  #1o mmK0 ;t$XXd^ ]]:. j 'L,,r)g?Nri)rjrkrlrMrhrmrns@rrurus )  rruc.eZdZUeed<dZfdZxZS)CTRLPreTrainedModelconfig transformerct||t|trXt j |j t|jj|jtjyyrK) rL _init_weights isinstance CTRLModelinitcopy_r0r1r n_positionsrrfloat)rWmodulerXs rrz!CTRLPreTrainedModel._init_weightssX f% fi ( JJ##%89R9RTZTgTginitit%u  )r)rjrkrlr__annotations__base_model_prefixrrmrns@rrrs %rrc^eZdZfdZdZdZe ddejdzde dzdejdzdejdzd ejdzd ejdzd e dzd e dzd e dzde dzdejdzde ejezfdZxZS)rc t|||j|_|j|_t j|j|j|_ t j|j|_ t jt|jDcgc]:}t|j|j |j"|j$|<c}|_t j(|j|j*|_|j/dt1|j2|jt4j6d|j9ycc}w)Nrwrxr0F) persistent)rLrMn_embdrn_layer num_layersr Embedding vocab_sizewr embd_pdropdropout ModuleListrangerun_headrr resid_pdrophr|layer_norm_epsilon layernormregister_bufferr1rrr post_init)rWrrrXs rrMzCTRLModel.__init__s  "MM ..f//?zz&"3"34 v~~. V]]FMM6::vGYGYefg   fmm9R9RS  /0B0BDDUDUW\WbWbcpu   s,?E9c|jSrKr)rWs rget_input_embeddingszCTRLModel.get_input_embeddingss vv rc||_yrKr)rWnew_embeddingss rset_input_embeddingszCTRLModel.set_input_embeddingss rN input_idspast_key_valuesr?token_type_ids position_ids inputs_embedsrcrdoutput_hidden_states return_dictr`returnc V||n|jj}||n|jj}| | n|jj} | | n|jj} | | t d|G|j |||j} |jd| d}|jd}n0|#|jdd} |jd}n t d| |jn |j}|r|t|j}||jnd}|>tj|| d|ztj|}|j!d}||dkr t d|j|d}|j!d j!d }|j#|j$ }d |z tj&|j$j(z}|I|jd| d}|j+|}|t-j.|j0z}nd}||j+|}| d}tj2tj4||z||zd j#|}|t-j.|j0z}|j6j#||_|j6|ddf}||z|z}|j9|}| rd nd}|rd nd}t;|j<D]-\}}| r||fz}|||||||| }|d}|s%||d fz }/|j?|}| r||fz}| stAd||||fDStC||||S)a Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer"rz5You have to specify either input_ids or inputs_embeds)r)r!devicez$batch_size has to be defined and > 0rrr g?rc3&K|] }|| ywrKr).0r=s r z$CTRLModel.forward..hsbcbos)last_hidden_stater hidden_states attentions)"rrdrcruse_return_dict ValueError%warn_if_padding_and_no_attention_maskr9viewr6rr get_seq_lengthrr%longr(r'r!finfominrr7r8rtriuonesr0r enumeraterrtupler )rWrrr?rrrrcrdrrr`kwargs input_shaper]r past_lengthtoken_type_embedsseq_lenr> pos_embedsrall_hidden_statesall_attentionsrrrs rrhzCTRLModel.forwardsH2C1N-TXT_T_TqTq!*!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T 0*$++>O:I:Uo446[\   << [_{5RZ_ZdZdmstL'11!4L  %Q !GHH+00R@N ,55a8BB1EN,..TZZ.@N!N2ekk$**6M6Q6QQN  %+00[_EN $~ 6  ):):!; ;  !    FF9-Mb/zz%**W{%:Gk>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N) rr?rrrrcrdrrr`rrr)losslogitsrrr) rrrrrPslicer loss_functionrr rrr)rWrrr?rrrrrcrdrrr`rrtransformer_outputsr slice_indicesrrrFs rrhzCTRLLMHeadModel.forwards2b&1%SV8W~ot4]k mA}a,?@A  %4%% ;;11 DY!4QR!88F)-)9TGf$ Ev E%/??-;;*55   rc Tt||f|||d|}|jdd|S)N)rrcis_first_iterationr)rLprepare_inputs_for_generationpop)rWrrrcrr model_inputsrXs rrz-CTRLLMHeadModel.prepare_inputs_for_generationsH w<  +1     )40r) NNNNNNNNNNNNr)NNF)rjrkrl_tied_weights_keysrMrrrr rrrrPrr rhrrmrns@rrrtsv+,BC.2(,37260426*.!%)-,0#'.2-.X ##d*X X ))D0 X ((4/ X &&- X ((4/X   4'X $;X  $;X #TkX D[X  t+X ell*X u|| 5 5!X X vSXrra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cReZdZfdZe ddej dzdedzdejdzdej dzdej dzdejdzd ej dzd e dzd e dzd e dzd e dzde eje zfdZ xZS)CTRLForSequenceClassificationct|||j|_t||_t j |j|jd|_|jy)NFr) rLrM num_labelsrrrrRr classifierrrs rrMz&CTRLForSequenceClassification.__init__sR   ++$V,))FMM4??O rNrrr?rrrrrcrdrrrc | | n|jj} |j|||||||| | |  } | d}|j|}||jdd\}}n|jdd\}}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d |tj||j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```N) rr?rrrrcrdrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r")rr!z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrr)rrrrr6 pad_token_idrr'rrint32r%argmaxlogger warning_oncerXrj problem_typerr!rrPrsqueezerrrrrr)rWrrr?rrrrrcrdrrrrrrr]sequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrFs rrhz%CTRLForSequenceClassification.forwardsz&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E' -;;*55   rr)rjrkrlrMrrrr rrrrrrhrmrns@rrrs(.2(,37260426*.!%)-,0#'d ##d*d d ))D0 d ((4/ d &&- d ((4/d   4'd $;d  $;d #Tkd D[d  u|| 7 7d d rr)rrrrrK)+__doc__numpyr7rrtorch.nnrrrrr cache_utilsr r generationr modeling_outputsr r rmodeling_utilsrutilsrrconfiguration_ctrlr get_loggerrjrrr1rGModulerIrsrurrrr__all__rrrrs AA&.)ii-+   H % %,11h`-299-` /  l #l l ^ v)?v  vr o $7o  o d cr