qi1$ddlmZmZddlmZmZddlmZmZddl m Z m Z ddl m Z erddlmZer dd lZdd lmZneZej(eZd efd Zd ZGddeZGddeZdee j8e j:gdiZy ))ABCabstractmethod) TYPE_CHECKINGAny)is_torch_availablelogging)QuantizationConfigMixinQuantizationMethod)get_module_from_namePreTrainedModelN) ModuleListreturnc6t}t|jdkDrGt|jjt|jj z}t |j ddh}|j}|jDchc] \}}|t|t|k(r|"}}}||z|z}t |Dchc]}|jdc}}t |Scc}}wcc}w)z Function to automatically detect keys to not convert for usage like quantization. For example for CausalLM modules we may want to keep the lm_head in full precision for numerical stability reasons. rz.weight) setlenall_tied_weights_keysvalueskeyslistnamed_parametersget_output_embeddings named_modulesid removesuffix) model tied_keyslast_module_keyoutput_emb_modulenamemoduleoutput_emb_keysmodules_to_not_convertks N/opt/pipecat/venv/lib/python3.12/site-packages/transformers/quantizers/base.pyget_keys_to_not_convertr)"s I 5 & &'!+33::<=ED_D_DdDdDf@gg E2245b9!<=O335"//1 D&  (RZ2>O;P-P O '8?J!F\"]1>>)#<"]^ & ''#^s -%D&Dcvddlm}|jD] }t||sd|j_"y)NrrT)modeling_utilsrmodules isinstanceconfig _is_quantized)rrr$s r(_assign_is_quantizedr0=s00--// fo .*.FMM '/c eZdZdZdZdefdZd*dZdee e fdzdee e fdzfd Z d d d e d dde fdZ dee ee zfdee ee zffdZd d d e defdZdZdZdZdZd+d,dZd,dZd,dZdZd+dZd+dZd e de fdZe d-d d dee dzdee dzd efd!Ze defd"Z!e defd#Z"d$Z#e$d%Z%e e$d&Z&d'Z'd(Z(d)Z)y). HfQuantizera Abstract class of the HuggingFace quantizer. Supports for now quantizing HF transformers models for inference and/or quantization. This class is used only for transformers.PreTrainedModel.from_pretrained and cannot be easily used outside the scope of that method yet. Attributes quantization_config (`transformers.utils.quantization_config.QuantizationConfigMixin`): The quantization config that defines the quantization parameters of your model that you want to quantize. requires_calibration (`bool`): Whether the quantization method requires to calibrate the model before using it. Fquantization_configc ||_|jdd|_|js&|jrt d|j dyy)N pre_quantizedTzThe quantization method z does require the model to be pre-quantized. You explicitly passed `pre_quantized=False` meaning your model weights are not quantized. Make sure to pass `pre_quantized=True` while knowing what you are doing.)r4popr6requires_calibration ValueError quant_method)selfr4kwargss r(__init__zHfQuantizer.__init__Ts`#6 #ZZ>!!d&?&?*+>+K+K*LMNO '@!r1rc|S)aO Some quantization methods require to explicitly set the dtype of the model to a target dtype. You need to override this method in case you want to make sure that behavior is preserved Args: dtype (`torch.dtype`): The input dtype that is passed in `from_pretrained` )r;dtypes r( update_dtypezHfQuantizer.update_dtype_s  r1 device_mapNc|S)a Override this method if you want to pass a override the existing device map with a new one. E.g. for bitsandbytes, since `accelerate` is a hard requirement, if no device_map is passed, the device_map is set to `"auto"`` Args: device_map (`Union[dict, str]`, *optional*): The device_map that is passed through the `from_pretrained` method. r?)r;rBs r(update_device_mapzHfQuantizer.update_device_mapks r1rr param_nameparam torch.Tensorc"|jSN) element_size)r;rrErFs r(param_element_sizezHfQuantizer.param_element_sizews!!##r1 max_memoryc|S)zaadjust max_memory argument for infer_auto_device_map() if extra memory is needed for quantizationr?)r;rLs r(adjust_max_memoryzHfQuantizer.adjust_max_memoryzsr1c y)zD Check whether a given param needs to be quantized. Fr?)r;rrEr<s r(param_needs_quantizationz$HfQuantizer.param_needs_quantization~sr1cy)a& This method is used to potentially check for potential conflicts with arguments that are passed in `from_pretrained`. You need to define it for all future quantizers that are integrated with transformers. If no explicit check are needed, simply return nothing. Nr?)r;argsr<s r(validate_environmentz HfQuantizer.validate_environments r1c|Sz"updates the tp plan for the scalesr?r;r.s r(update_tp_planzHfQuantizer.update_tp_plan r1c|SrUr?rVs r(update_ep_planzHfQuantizer.update_ep_planrXr1c |SrIr?r;rr<s r($_process_model_before_weight_loadingz0HfQuantizer._process_model_before_weight_loading r1c d|_|jj|_|jr|j ||j |fi|y)aQ Setting model attributes and/or converting model before weights loading. At this point the model should be initialized on the meta device so you can freely manipulate the skeleton of the model in order to replace modules in-place. Make sure to override the abstract method `_process_model_before_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_before_weight_loading`. TN) is_quantizedr4r:quantization_methodr6_convert_model_for_quantizationr])r;rr@r<s r(preprocess_modelzHfQuantizer.preprocess_modelsL"$($<$<$I$I!     0 0 7111%B6Br1c |SrIr?r\s r(#_process_model_after_weight_loadingz/HfQuantizer._process_model_after_weight_loadingr^r1c |j|j_|jr)t|jddr|j |n t ||j |fi|S)a Post-process the model post weights loading. Make sure to override the abstract method `_process_model_after_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_after_weight_loading`. dequantizeF)r4r.r6getattrremove_quantization_configr0rer\s r(postprocess_modelzHfQuantizer.postprocess_models_,0+C+C (   '$*B*BLRW"X  + +E 2  '7t77HHHr1ct|dr|`t|jdr |j`t|dr|`d|_y)z@ Remove the quantization config from the model. hf_quantizerr4raFN)hasattrrlr.r4rar`r;rs r(riz&HfQuantizer.remove_quantization_configsF 5. )" 5< ?!%c*@&A!B%%r1cy)zUFlag indicating whether the quantized model can carry out quantization aware trainingFr?r;s r(is_qat_trainablezHfQuantizer.is_qat_trainabler1cy)z;Flag indicating whether the quantized model can be compiledFr?r~s r(is_compileablezHfQuantizer.is_compileablerr1c difS)zcGet state dict and metadata. Useful when we need to modify a bit the state dict due to quantizationNr?rns r(get_state_dict_and_metadataz'HfQuantizer.get_state_dict_and_metadatas Rxr1cyrIr?r~s r(is_serializablezHfQuantizer.is_serializable s"r1cyrIr?r~s r( is_trainablezHfQuantizer.is_trainable sr1c|jD]\}}|jj}|tvs%|jj t|dvsHt jd5t||\}}t|d|jj|j|<dddy#1swYxYw)Nquantization_methodsmeta module_name) r __class____name__!MODULES_TO_PATCH_FOR_QUANTIZATIONr4r:torchdevicer r.get_text_config_modules)r;rr#r$module_class_name parent_modules r(rbz+HfQuantizer._convert_model_for_quantizations!//1 LD& & 0 0 9 9  $EE((5545FGH^_`\\&)*>ud*K'M43TUf3ghu3v 4464M**40  s .AB??C cFt|jjd)Nz1 is not available yet and will be supported soon.rsr~s r(get_quantize_opszHfQuantizer.get_quantize_opss'!''4455f g  r1cgSrIr?r~s r(get_weight_conversionsz"HfQuantizer.get_weight_conversions"s r1)r@ torch.dtyperrrI)rr)NNF)*r __module__ __qualname____doc__r8r r=rAdictstrrrDfloatrKintrNboolrPrSrWrZr]rcrerjrirgrprv staticmethodrr|propertyrrrrrrrbrrr?r1r(r3r3Es ! ,C   DcNT,A d3PS8nW[F[ $(9$s$Sa$fk$DcCi,@T#sUXy.EY.?S_c C$I( #   *.15"' & &3i$&&#3i$.& &&*$""   r1r3c2eZdZdZfdZ ddZxZS)SequentialLlama4TextExpertsz A module that implements a compressed version of a list of expert modules. This is specifically designed to work with Llama4TextExperts in MoE layers. cddlm}t| t |j Dcgc] }|| c}|j |_ycc}w)Nr) Llama4TextMLP)*transformers.models.llama4.modeling_llama4rsuperr=rangenum_local_experts num_experts)r;r.r_rs r(r=z$SequentialLlama4TextExperts.__init__,sBL v?W?W9XYA-/YZ!33ZsAc|j|jd|jd}tj|}t |jD]}||||||<|S)Nr)reshapershaper zeros_liker)r; hidden_states routed_out expert_idxs r(forwardz#SequentialLlama4TextExperts.forward2sw&--d.>.>MDWDWXZD[\ %%m4  0 01 QJ%5T*%5mJ6O%PJz " Qr1)rrGrrG)rrrrr=r __classcell__)rs@r(rr&s$ 4 % r1rLlama4TextExperts)rr)abcrrtypingrrutilsrr utils.quantization_configr r quantizers_utilsr r+rrtorch.nnrr get_logger__file__loggerrr)r0r3rCOMPRESSED_TENSORSBITS_AND_BYTESrr?r1r(rs$%/S20#J   H %(d(6/^#^B*02  1 1  - -! %!r1