qi(PdZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z dd lmZd d lmZd d lmZmZmZmZmZmZmZddlmZej8eZGddeZGddeZ GddeZ!GddeZ"GddeZ#GddeZ$gdZ%y)zPyTorch BitNet model.)CallableN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack)logging)GemmaMLP)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward) BitNetConfigc eZdZy) BitNetRMSNormN__name__ __module__ __qualname__[/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/bitnet/modular_bitnet.pyrr)rrc*eZdZdeffd ZdZxZS) BitNetMLPconfigcpt||t|j|j|_yN)eps)super__init__rintermediate_size rms_norm_eps ffn_sub_norm)selfr" __class__s rr'zBitNetMLP.__init__.s+  )&*B*BH[H[\rc |j|j|j|j||j |z}|S)N) down_projr*act_fn gate_projup_proj)r+xr.s rforwardzBitNetMLP.forward2sFNN4#4#4T[[PQAR5SVZVbVbcdVe5e#fg r)rrrrr'r3 __classcell__r,s@rr!r!-s]|]rr!ceZdZdedeffd Z d dejdeejejfdejdzde dzd ejdzd e e d eejejdzffd Z xZS)BitNetAttentionr" layer_idxcrt|||t|j|j|_yr$)r&r'r hidden_sizer) attn_sub_norm)r+r"r8r,s rr'zBitNetAttention.__init__8s- +*6+=+=6CVCVWrN hidden_statesposition_embeddingsattention_maskpast_key_valuescache_positionkwargsreturnc :|jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } tj|jjt}||| | | |f|jsdn |j |j"d|\}}|j$g|dj'}|j)|}|j+|}||fS)Nrr )sincosr@g)dropoutscaling)shapehead_dimq_projview transposek_projv_projrupdater8r get_interfacer"_attn_implementationrtrainingattention_dropoutrHreshape contiguousr;o_proj)r+r<r=r>r?r@rA input_shape hidden_shape query_states key_states value_statesrFrE cache_kwargsattention_interface attn_output attn_weightss rr3zBitNetAttention.forward<s$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH ((5 kk+. L((r)NN)rrrrintr'torchTensortupler LongTensorr rr3r4r5s@rr7r77sX|XX)-26 *)||*)#5<<#=>*) t+ *)  *) ((4/ *)-.*) u||U\\D00 1*)rr7c eZdZy)BitNetDecoderLayerNrrrrrgrgirrrgc eZdZy) BitNetModelNrrrrririmrrric4eZdZddiZdZdZdeffd ZxZS)BitNetForCausalLMzlm_head.weightzmodel.embed_tokens.weightNrBc "t|di|S)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BitNetForCausalLM >>> model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: ' >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=100) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "User: Hey, are you conscious? Can you talk to me?Assistant: No, I'm not conscious. I'm an artificial intelligence designed to assist with information and tasks. How can I help you today?" ```r)r&r3)r+ super_kwargsr,s rr3zBitNetForCausalLM.forwardvs4w...r) rrr_tied_weights_keys_tp_plan_pp_planrr3r4r5s@rrkrkqs-*,GHHH/ //rrk)rkriBitNetPreTrainedModel)&__doc__collections.abcrrb cache_utilsrmodeling_flash_attention_utilsrmodeling_outputsrmodeling_utilsrprocessing_utilsr utilsr gemma.modeling_gemmar llama.modeling_llamar rrrrrrconfiguration_bitnetr get_loggerrloggerrr!r7rgrirk__all__rrrrs$ B65&+/   H % L /)n/)d *  * /(/D r