qi4ddlZddlmZddlmZddlmZddlmZm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*GddeZ+GddeZ,GddeZ-Gdde Z.Gddej^Z0Gdd e"Z1Gd!d"e%Z2eGd#d$e#eZ3eGd%d&e'Z4Gd'd(e&Z5gd)Z6y)*N)nn)initialization)ACT2FN)Cache DynamicCache)create_causal_mask)MoeCausalLMOutputWithPastMoeModelOutputWithPast)PreTrainedModel)Unpack)TransformersKwargsauto_docstring)can_return_tuplemerge_with_config_defaults)capture_outputs)GraniteRMSNormGraniteRotaryEmbedding)JetMoeParallelExpertsJetMoeTopKGating)LlamaAttentionLlamaPreTrainedModel)MixtralDecoderLayerMixtralForCausalLM MixtralModelload_balancing_loss_func)GraniteMoeConfigc eZdZy)GraniteMoeRMSNormN__name__ __module__ __qualname__c/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/granitemoe/modular_granitemoe.pyr!r!$r'r!c eZdZy)GraniteMoeRotaryEmbeddingNr"r&r'r(r+r+(r)r'r+c eZdZy)GraniteMoeParallelExpertsNr"r&r'r(r-r-,r)r'r-c eZdZy)GraniteMoeTopKGatingNr"r&r'r(r/r/0r)r'r/c.eZdZdZdeffd ZdZxZS) GraniteMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. configct||j|_|j|_t |j |_t|j|j|jdz|_ t|j|j|j|_ t|j|j|j|_y)Nr) input_size num_expertstop_k)super__init__ hidden_sizer4intermediate_sizer hidden_act activationr-num_local_experts input_linear output_linearr/num_experts_per_tokrouterselfr2 __class__s r(r8zGraniteMoeMoE.__init__=s  ,,!33 !2!235f6N6NPTP_P_aeaqaqtuauv6v7O7OQUQaQacgcrcrs*00,,  r'c|j\}}}|jd|}|j|\}}}}}||} |j| |} | j dd} |j | d| dz} |j | |} | |dddfz} tj||z|jf| j| j} | jd|| }|j|||j}|S)Nr)dimrr)dtypedevice)sizereshaperAr>chunkr<r?torchzerosr4rHrI index_addview)rC layer_inputbszlengthemb_size_ batch_index batch_gates expert_size expert_inputs hidden_stateschunked_hidden_statesexpert_outputsrN layer_outputs r(forwardzGraniteMoeMoE.forwardLs + 0 0 2VX!))"h7 6:kk+6N3; [!#K0 ))-E - 3 3A2 3 >(=a(@ADYZ[D\\ ++M;G'+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF r')r#r$r%__doc__rr8r^ __classcell__rDs@r(r1r14s  /  r'r1c(eZdZdedeffd ZxZS)GraniteMoeAttentionr2 layer_idxcJt|||||j|_yN)r7r8attention_multiplierscalingrCr2rdrDs r(r8zGraniteMoeAttention.__init__`s! vy122 r')r#r$r%rintr8r`ras@r(rcrc_s3/3C33r'rcc eZdZdedeffd Z d dejdejdzdedzdejdzd e ejejfdzd ejf d Z xZ S) GraniteMoeDecoderLayerr2rdcHt|||t|||_t ||_t |j|j|_ t |j|j|_ |` t ||_|j|_ y)N)r2rdeps) r7r8rc self_attnr1block_sparse_moer!r9 rms_norm_epsinput_layernormpost_attention_layernormmlpresidual_multiplierris r(r8zGraniteMoeDecoderLayer.__init__fs +,FiP -f 501C1CI\I\](9&:L:LRXReRe(f% H -f 5#)#=#= r'NrZattention_maskpast_key_valuescache_positionposition_embeddingsreturnc |}|j|}|jd|||||d|\}}|||jzz}|}|j|}|j |}|||jzz}|S)N)rZrwrxryrzr&)rsrprvrtrq) rCrZrwrxryrzkwargsresidualrUs r(r^zGraniteMoeDecoderLayer.forwardps!,,]; )4>> ')+) 3     q!=43K3K#KK  55mD --m<  =43K3K#KK r')NNNN) r#r$r%rrjr8rMTensorr LongTensortupler^r`ras@r(rlrles>/>C>/3(,26HL || t+  ((4/  #5<<#=>E  r'rlcfeZdZUeed<dZdZdgZdgZdZ dZ dZ e jdZy) GraniteMoePreTrainedModelr2modelTrlrxFctj||t|tr7t j |j d|jjyy)Ng)meanstd) r _init_weights isinstancer-initnormal_weightr2initializer_range)rCmodules r(rz'GraniteMoePreTrainedModel._init_weightss@%%dF3 f7 8 LLSdkk6S6S T 9r'N)r#r$r%r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphrMno_gradrr&r'r(rrsS &*#12#4"5N"U]]_UUr'rceZdZdeffd Zeee ddejdzdejdzdejdzde dzdejdzd e dzd ejdzd eed efd ZxZS)GraniteMoeModelr2c &t||tjt |j Dcgc]}t ||c}|_t|j|j|_ |j|_ ycc}w)Nrn) r7r8r ModuleListrangenum_hidden_layersrllayersr!r9rrnormembedding_multiplierris r(r8zGraniteMoeModel.__init__sq  mmHMfNfNfHg h9 #FI 6 h  &f&8&8f>Q>QR $*$?$?! isBN input_idsrw position_idsrx inputs_embeds use_cacheryr}r{c b|du|duz r td|r|t|j}||j|}|F||j nd} t j | | |jdz|j}||jd}t|j|||||} ||jz}|} |j| |} |jd|jjD]} | | f| | ||||d|} |j| } t!| |S) Nz:You must specify exactly one of input_ids or inputs_embeds)r2rr)rI)r2rrwryrxr)rzrwrrxrry)last_hidden_staterx) ValueErrorrr2 embed_tokensget_seq_lengthrMarangeshaperI unsqueezer r rotary_embrrrr )rCrrwrrxrrryr}past_seen_tokens causal_maskrZrz decoder_layers r(r^zGraniteMoeModel.forwardsp  -t"; <YZ Z 0*$++>O   --i8M  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L(;;'))+%  &(A(AA % #oom\J![[)H4;;+H+HI M) $7*) /#-  M  -0 %++  r')NNNNNNN)r#r$r%rr8rrrrMrrr FloatTensorboolr rr r^r`ras@r(rrs@/@ .2.204(,26!%26; ##d*;  t+; &&- ;  ; ((4/ ; $;; ((4/; +,;  ;  ; r'rcDeZdZdeffd Zee ddejdzdejdzdejdzde dzdejdzd ejdzd e dzd ejdzd e ejzd eezfdZxZS)GraniteMoeForCausalLMr2cft||t||_|j|_yrf)r7r8rrlogits_scalingrBs r(r8zGraniteMoeForCausalLM.__init__s*  $V, $33r'Nrrwrrxrlabelsoutput_router_logitsrylogits_to_keepr{c ||n|jj}|jd||||||d| } | j} t | t r t | dn| } |j| dd| ddf}||jjz }d}|*|j||fd|jji| }d}|rYt| j|j|j|}|+||j|j!|j"zz }t%|||| j&| j(| j*| jS)al labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, GraniteMoeForCausalLM >>> model = GraniteMoeForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rrwrrxrry vocab_size)lossaux_losslogitsrxrZ attentions router_logitsr&)r2rrrrrjslicelm_headr loss_functionrrrr5r@router_aux_loss_coeftorIr rxrZr)rCrrwrrxrrrryrr}outputsrZ slice_indicesrrrs r(r^zGraniteMoeForCausalLM.forwardsL%9$D $++JjJj $** )%+')    11 8B>SV8W~ot4]k mA}a,?@A$++444  %4%% ;;11 D /%%  (( H !11HKK 4LLL(#33!//))!//  r') NNNNNNNNr)r#r$r%rr8rrrMrrrrrrjrr r^r`ras@r(rrs4/4 .2.204(,26*.,026-.S ##d*S  t+S &&- S  S ((4/ S   4'S #TkS ((4/S ell*S  * *S S r'r)rrr)7rMrrr activationsr cache_utilsrr masking_utilsr modeling_outputsr r modeling_utilsr processing_utilsr utilsrr utils.genericrrutils.output_capturingrgranite.modeling_graniterrjetmoe.modeling_jetmoerrllama.modeling_llamarrmixtral.modeling_mixtralrrrrconfiguration_granitemoerr!r+r-r/Moduler1rcrlrrr__all__r&r'r(rs  &!./Q-&7I5MLGvv6   6  5  + (BII(V3.3 #0#LU 4oUU"G lG G T[ .[ | Tr'