qiVdZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z e r ddlZddlmZe j eZdZe rerdd lmZndd lmZGd d eZGd deZdadZdZdZdZdZdZ ddZ!dZ"ddZ#ddZ$ddZ%dZ&y)z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging)nnctjjddu}|r! tjjd}yy#tjj$rYywxYw)N deepspeedTF) importlibutil find_specmetadataPackageNotFoundError)package_exists_s U/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_availabler$sc^^--k:$FN ""++K8A!!66  sAA&%A&)HfDeepSpeedConfig)objectc"eZdZdZfdZxZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. cft|tdtdt| |y)N accelerater )set_hf_deepspeed_configrsuper__init__selfconfig_file_or_dict __class__s rrzHfDeepSpeedConfig.__init__Js)%,'+& ,-)__name__ __module__ __qualname____doc__r __classcell__r s@rrr9s ..r!rcXeZdZdZfdZdZdZd dZeedZ d dZ d Z xZ S) HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. c@t||d|_g|_yN)rr_dtype mismatchesrs rrz!HfTrainerDeepSpeedConfig.__init__Xs ,- r!cH|j td|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r, ValueError)rs rdtypezHfTrainerDeepSpeedConfig.dtype]s" ;; WX X{{r!c4|j|}|y|dk(S)NFauto) get_value)r ds_key_longvals ris_autoz HfTrainerDeepSpeedConfig.is_autobs"nn[) ;&= r!c |j|\}}|y|j|dk(r|||<y|sy|j|}|.||k7r(|jjd|d|d|d|yyy)a A utility method that massages the config file and can optionally verify that the values match. 1. Replace "auto" values with `TrainingArguments` value. 2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer config values and if mismatched add the entry to `self.mismatched` - will assert during `trainer_config_finalize` for one or more mismatches. Nr2z- ds =z vs hf )find_config_nodegetr-append)rr4hf_valhf_key must_matchconfigds_keyds_vals r fill_matchz#HfTrainerDeepSpeedConfig.fill_matchis..{; >  ::f  '#F6N  F#  &F"2 OO " "U;-qxqQWPX#Y Z#3 r!F)r>c|j|jz|jz}|jd|jd| |jd|jd|jd|d| |jd|jd|jd|j d |jd |j |jgd |jd |jd |jd|jd|jdd|jd|j d |jrE|jjdi|jd<|j|jdd<|jd|jxs |jd|jd|j xs |j"d|j%drt&j(|_y|j%drt&j,|_yt&j.|_y)z Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object creation. train_micro_batch_size_per_gpuper_device_train_batch_sizegradient_accumulation_stepstrain_batch_sizeztrain_batch_size (calculated)gradient_clipping max_grad_normzoptimizer.params.lr learning_ratezoptimizer.params.betaszadam_beta1+adam_beta2zoptimizer.params.eps adam_epsilonzoptimizer.params.weight_decay weight_decayzscheduler.params.warmup_min_lrrzscheduler.params.warmup_max_lr checkpointuse_node_local_storagez fp16.enabledzfp16|fp16_full_evalz bf16.enabledzbf16|bf16_full_evalN) world_sizerErFrBrIrJ adam_beta1 adam_beta2rKrL fill_onlysave_on_each_noder?r:fp16fp16_full_evalbf16bf16_full_evalis_truetorchbfloat16r,float16float32)rargsauto_find_batch_sizerGs rtrainer_config_processz/HfTrainerDeepSpeedConfig.trainer_config_processs ??T-M-MMPTPpPpp  ,  , , )$ $   )  , , )    +$ $  +T-?-?Q -t/A/A?S  $ __doo . # .0A0A>R 79J9JN[ 7; 8$:L:Lo^  ! !(,  b(IDKK %BFBXBXDKK %&> ? )Id6I6ILab )Id6I6ILab << '..DK \\. )--DK--DKr!cgd}|Dcgc]}|j|s|}}t|dkDrd}t|drt|jdr|jj}nt|jdr t |jj }nt|jdrAt|jjdr!|jjj}n_t|jdrIt|jjdr)t |jjj }|td|d |jd ||z|jr6|jd td |z|z|jd d|z|jd|d|jd|j|dt|jdkDr*dj|j}td|dycc}w)z This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. )$zero_optimization.reduce_bucket_size-zero_optimization.stage3_prefetch_bucket_size4zero_optimization.stage3_param_persistence_thresholdrNr? hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.rarbg?rc z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_steps warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.)r6lenhasattrr?rdmaxrerfr/rRis_zero3intrBget_warmup_stepsr-join) rr]modelnum_training_stepshidden_size_based_keysxhidden_size_auto_keysrdr-s rtrainer_config_finalizez0HfTrainerDeepSpeedConfig.trainer_config_finalizes "  -C VqdllSTo V V $ % )Kuh'5<<7"',,":":KU\\>:"%ell&?&?"@KU\\=9gellF^F^`m>n"',,":":"F"FKU\\=9gellF^F^`n>o"%ell&>&>&K&K"LK" 55J4KLYY NNA;Q\C\ ]}}Ck)K78J$  .  -  /  ! !"4 5  t ! #4??3J'L(oq  $a!Ws II)NTF) r"r#r$r%rr0r6rBrrRr_rvr&r's@rr)r)Rs7   ![4jU;I8(tCr!r)c.tj|ayr+)weakrefref_hf_deepspeed_config_weak_ref)hf_deepspeed_config_objs rrr s %,KK0G$H!r!cdayr+)r{r!runset_hf_deepspeed_configrs %)!r!cTt"ttjSy)NF)r{rmr~r!ris_deepspeed_zero3_enabledrs&$05R5T5`,.7799r!cLtttjSyr+)r{r?r~r!rdeepspeed_configrs#$05R5T5`,.555r!cddlddl}ddlm}ddlm|j fd|j5|5||jddddddy#1swYxYw#1swYyxYw)aA DeepSpeed ZeRO-3 variant of `PreTrainedModel.initialize_weights`. Mirrors the `smart_apply` dispatch logic but gathers each module's partitioned parameters before calling `_initialize_weights`, so initialization operates on full tensors instead of empty shards. Only rank 0 performs the actual init. rNr)guard_torch_init_functions)PreTrainedModelc|jD]+}t|r||j#||-t|j d}|rMj j |d5jjdk(r ||dddy||y#1swYyxYw)NF)recurser modifier_rank) children isinstance_initialize_weightslist parameterszeroGatheredParameterscommget_rank)model_or_modulefnchildparamsr _apply_zero3r is_remote_codes rrz.initialize_weights_zero3.._apply_zero34s$--/ (E%1UE$=$=>UB'  ( o000?@ 2262K 8>>**,17 8 8  /  8 8s :'B44B=) r rYinitializationrmodeling_utilsrrno_gradr)rqrYrrrr rs @@@@rinitialize_weights_zero3r%sv;0))+N 0 ; ' ) ;  9 9 : ;;; ; ;;;s$A; A/A;/A8 4A;;Bc!t}||jdijdd}|jdi}t|tr,t ||jdijdd}|dkDr t ddd lm}m}m !m }t|d d} |j} i} |jjD]4\} } tj | j"| j$d | | <6|Dcgc]}t||s|}}|Dcgc]}t||s|}}t'|d k(r>i}|jD]\}}|||g| | \}}|| vs|||< | | |_|S|Dcic]}|j*D]}||}}}i}i}t-|j/!fd}|D]}|j1|}||||| | \}}|| vs(|U||}||j*|j2|j4}|j7||}|j9|||||||<|jD]X\}} |j;|||j<}|jD]!\}} t| t>r| d n| } | ||<#Z| | |_|Scc}wcc}wcc}}w#t@$r} tCd|d| | d} ~ wwxYw)z Apply weight conversions (renaming and merging/splitting operations) to a state dict. This is a simplified version that handles the conversion without loading into the model. Ntensor_parallel autotp_size inferencetp_sizezWeight conversions (e.g., MoE expert fusion) with DeepSpeed Tensor Parallelism are not yet implemented but support is coming soon. Please disable tensor_parallel in your DeepSpeed config or convert your checkpoint to the expected format first.r)WeightConverterWeightRenamingdot_natural_keyrename_source_key _metadatameta)r0devicerc|Sr+r~)krs rz9_apply_weight_conversions_to_state_dict..s /!:Lr!)key)source_patternstarget_patterns operations)rqr?z'Failed to apply weight conversion for 'zb'. This likely means the checkpoint format is incompatible with the current model version. Error: )"rr:rdictrlNotImplementedErrorcore_model_loadingrrrrgetattrbase_model_prefix state_dictitemsrYemptyshaper0rjrrsortedkeyspoprr setdefault add_tensorconvertr?r Exception RuntimeError)"rqrweight_mapping ds_configrinference_configrrrrprefixmodel_state_dictrparamentry renamings convertersnew_state_dict original_keytensor renamed_keyr converterrpattern_to_converterconversion_mapping sorted_keyssource_pattern new_convertermappingrealized_value target_nameers" @r'_apply_weight_conversions_to_state_dictrHs!"I-- 126::=!L$==b9 & -'#3#7#78I2#N#R#RS\^_#`aG Q;%d  ihz;5H  $ $F&&(..0[ U % EKKu{{SY Z[%3X5j6WXIX%3ZEz%7Y%ZJZ :!$.$4$4$6 5 L&.|YFTdeNK...4{+ 5  '/N $;EhYiNgNghAyLhAhh N*0LMK#5  -&7 iQ[]ceu&v# ^ * *)1@ /$-$=$=$-$=$=(33! -77 ]S""; nfU/5{++50!3 8 8 : W $__||-N '5&:&:&< 4" U$.ud$;a.3{+ 4$#+ KYZiT 9+G   s7K K*K<KK2AK!! L*K<<Lc  t|dd |j}  |_d}| t|dd}|"t|dkDrt |||}||_g|j }t|j t|dd}|jDcic]%\}}|j|d||d|n||'}}}d dtjf fd  ||d  fScc}}w) a Loads state dict into a model specifically for Zero3, since DeepSpeed does not support the `transformers` tensor parallelism API. Nearly identical code to PyTorch's `_load_from_state_dict` Args: model_to_load: The model to load weights into state_dict: The state dict containing the weights load_config: Optional LoadStateDictConfig containing weight_mapping and other loading options rNrrr.Fmodulecrinj|ddi}||d<|||dgg f}trddl}t|j |ddd}g}|D]5} | |vs|| } d| _|j | j| 7t|dkDrV|jj|d5tjjdk(r|j|ddd|jj!D]\} } |  | ||| zdz|y#1swY@xYw) Nassign_to_params_buffersTrF)rrrr)r:rr rnamed_parameters_is_hf_initializedr;discardrjrrrY distributedr_load_from_state_dict_modulesr)rrrrlocal_metadatar]r rparams_to_gatherrrnamer error_msgsloadr missing_keyss rrz/_load_state_dict_into_zero3_model..loadsa'/X\\&"+r5R5M12FND"b*M & '  $F$;$;6#2;X]$;$^_ ! % , ?,Q/E/3E,$++E2 ((+  ,#$q(^^667GWX6Y<((113q8444d;<"??002 WKD% UJ (;=UV W <z;5H"J' N .>E!c.&9A&=<]JXfg ,: )J)446,1134L=*=tDL$$& Aq#8";";|nAaS1# dehi iJ WRYY W WD  UC | ##Ws+*Dc&ddlm}m}|j}d}d|vr ||}n:|j rt j dj}d|d<d} d |vr ||} || fSt||rfd } ||| } || fS) zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizer)rzDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)Tzero_allow_untested_optimizer schedulercftj}d|_|j|}|S)N)rrr)r lr_schedulercreate_scheduler)r trainer_copyrrrtrainers r_lr_scheduler_callablez5deepspeed_optim_sched.._lr_scheduler_callable*s=#yy1 -1 )+<<'9Y =  $#r!)lr_scheduler_callable) accelerate.utilsrrr? is_offloadloggerinfocreate_optimizerr) rhf_deepspeed_configr]rrmodel_parametersrrr?rrrs ` ` rdeepspeed_optim_schedr s< ' 'FIf&67  ) ) + KKV ,,. 26./Lf%i0 " l "" i , $*)KabL l ""r!cddlm}|j}|j}|jj j j}|j||||j|j|rH|js td|jd|jdd\}}d} ||fSd|_|jj!dij!d d } | d kDr1ddl} | j%|| |j'|j }t)t+d |j-} t/||||| \}}||fS) a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/deepspeedai/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/deepspeedai/DeepSpeed/issues/1612 r)rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configrr)NNNrrr)rqrr0r?c|jSr+) requires_grad)ps rrz deepspeed_init..rs r!)deepspeed.utilsrrqr] acceleratorstatedeepspeed_plugin hf_ds_configrvsetLevelget_process_log_levelrmr/del_config_sub_treerr?r:r tp_model_initr0rfilterrr ) rrrr ds_loggerrqr]r rrr deepspeed_tp_sizer s rdeepspeed_initr:sq*4 MME <!>I 11$8LMr!c> d|vr d|vr|d|d<|d i|}|j}|jdj}|j}tj j jj|| |ddk7jdj} tj j jj| | t fdt|D} t } | t| dz }|r||fS|S) aq Computes the loss under sequence parallelism with `sp_backend="deepspeed"` and `sp_size > 1`. Performs weighted loss aggregation across SP ranks, accounting for varying numbers of valid tokens per rank (e.g., when some ranks receive only padding or prompt tokens that are masked with -100). Args: accelerator (`Accelerator`): The accelerator instance with `torch_device_mesh` support. model (`torch.nn.Module`): The model to compute the loss for. inputs (`dict[str, torch.Tensor | Any]`): The input data for the model. Must include `"shift_labels"` key. return_outputs (`bool`): Whether to return the model outputs along with the loss. pc (`accelerate.parallelism_config.ParallelismConfig`): The parallelism configuration. Returns: The loss, or a tuple of `(loss, outputs)` if `return_outputs` is `True`. labels shift_labelssp)groupirc3DK|]}|dkDr ||zyw)rNr~).0rankgood_tokens_per_ranklosses_per_ranks r z,deepspeed_sp_compute_loss..s6   % )  4T ::s rr~) losstorch_device_mesh get_groupsp_sizerYrr functional all_gatherviewsumrangerl)rrqinputsreturn_outputspcoutputsr7sp_group sp_world_size good_tokens total_losstotal_good_tokensr4r5s @@rdeepspeed_sp_compute_lossrIs,v.F":!.1xofoG <HJJM''**55@@X@VO.)T177;??AK ,,//::EEkYaEb-(J 01 -q1 1D,D'?6$6r!r+rw)T)'r%rimportlib.metadatar importlib.utilry functoolsrdependency_versions_checkrutilsrrrrYr get_loggerr"rraccelerate.utils.deepspeedrDeepSpeedConfigbuiltinsrr)r{rrrrrrrr rr(r+rIr~r!rrSs #9HH   H % !7!9O3..2p0ph!%I)  ;FhVN$b3#l@#FP0N",7r!