qi7ddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZddl m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlm?Z?ddl@mAZAddlBmCZCddlDmEZEddlFmGZGddlHmIZIddlJmKZKddlLmMZMide%d e)d!e+d"e9d#e!d$eCd%eEd&e5d'e7d(e/d)e;d*e=d+e-d,e1d-eKd.e'd/eMeIe3e#eAe?eGd0ZNided e d!e d(e d"ed#ed$ed%ed&ed'ed*ed+e d,e d)ed-ed.e d/eeeeeeed0ZOejeQZRGd1d2ZSGd3d4ZTd5eUfd6ZVd7eUfd8ZWd9ZXy):N) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8ConfigFourOverSixConfig FPQuantConfig GPTQConfig HiggsConfig HqqConfig MetalConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SinqConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FourOverSixHfQuantizer)FPQuantHfQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)MetalHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SinqHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerawqbitsandbytes_4bitbitsandbytes_8bitgptqaqlmquantoquark fouroversixfp_quanteetqhiggshqqzcompressed-tensors fbgemm_fp8torchaobitnetvptq)spqrfp8z auto-roundmxfp4metalsinqc6eZdZdZedefdZedZy)AutoQuantizationConfigz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictc v|jd}|jdds|jddr*|jddrdnd}tj|z}n | td|tvr,td|d t t jt|}|j|S) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsrNrPsuffix target_clss N/opt/pipecat/venv/lib/python3.12/site-packages/transformers/quantizers/auto.pyr_z AutoQuantizationConfig.from_dicts/33NC # ' ' >BZB^B^_motBu 8 < <^U SWY`F-<>>( # #rdrMcreZdZdZedeezfdZedZedeezdedzfdZ e dZ y) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. rfc Dt|trtj|}|j}|t j k(r|jr|dz }n|dz }|tvr,td|dttjt|}||fi|S)NrTrSrVrW) isinstancersrMr_rPrrYrQr]rZr\r^)r`rfrlrPrbs rc from_configzAutoHfQuantizer.from_configs )4 0"8"B"BCV"W *77  -<< <"//' ' 5 51,@/44678:  ,L9 -888rdc Ptj|fi|}|j|S)N)rMrhrx)r`rkrlrfs rcrhzAutoHfQuantizer.from_pretraineds*4DDEbmflm233rdquantization_config_from_argsNc *|d}nd}t|tr;t|trtj|}ntj|}|g|j j |j j k7r:td|j j d|j j dt|ttttttttfrW|U|j}|j!D]\}}t#||||dt%|j'dz }|dk7r2t|tttfst)j*||St,j/||S)z handles situations where both quantization_config from args and quantization_config from model config are present. zYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)rwrsrr_rM __class__rnrZrrr r rrrget_loading_attributesitemssetattrr\r^warningswarnloggerinfo)r`rfrz warning_msgloading_attr_dictattrvals rcmerge_quantization_configsz*AutoHfQuantizer.merge_quantization_configss ) 4y  K )4 07I&5&?&?@S&T#&<&F&FGZ&[# * 5#--66:W:a:a:j:jj./B/L/L/U/U.VVmoLoVoVo_o_n``FF  ###+(  .9 = T T V .446 8 c+T37 8 ?EVE[E[E]@^?_`}~ ~K " Z0CkS^`tEu%v MM+ &#" KK $""rdc ^|jdd}|jdds|jddr*|jddrdnd}tj|z}n | td|tvr8t j d|d ttjd yy ) NrPrQFrRrSrTrUrVrWz~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) rXrrYrZr[rwarningr\r]r^)rNrPras rcsupports_quant_methodz%AutoHfQuantizer.supports_quant_method s/33NDI # ' ' >BZB^B^_motBu 8 < <^U SWY`F-<.register_config_fn!sH 5 5xx/CDE E#67HI I36(0 rdrg)rrs` rcregister_quantization_configrs rdnamecfd}|S)zRegister a custom quantizer.c~tvrtddt|ts t d|t<|S)Nz Quantizer 'rz!Quantizer must extend HfQuantizer)r]rZrrr)r`rs rcregister_quantizer_fnz1register_quantizer..register_quantizer_fn1sG ) ){4&0DEF F#{+?@ @'*t$ rdrg)rrs` rcregister_quantizerr.s ! rdc"t|d}|r!tj|jsd}|s|Q|r&tj |j||_n||_tj |j|}nd}||j |||j|}|j|}|j|}t|jdds&|jj}t|d||d<|||fS)NrfF) pre_quantized) device_map weights_only dequantizevaluequant) hasattrrurrfrrxvalidate_environmentupdate_device_mapupdate_tp_planupdate_ep_planrirP)configrfrr user_agentr hf_quantizerrPs rcget_hf_quantizerr>sF$9:M_BB6C]C]^ +7 )8)S)S**,?*F &*=F &&22  & &'3  ))!% * "33J? ,,V4,,V4|77uM';;HHL"),"NJw   ++rd)Yrmodels.auto.configuration_autorutilsrutils.quantization_configrrrr r r r r rrrrrrrrrrrrrrrrbaserquantizer_aqlmr quantizer_auto_roundr! quantizer_awqr"quantizer_bitnetr#quantizer_bnb_4bitr$quantizer_bnb_8bitr%quantizer_compressed_tensorsr&quantizer_eetqr'quantizer_fbgemm_fp8r(quantizer_finegrained_fp8r)quantizer_fouroversixr*quantizer_fp_quantr+quantizer_gptqr,quantizer_higgsr- quantizer_hqqr.quantizer_metalr/quantizer_mxfp4r0quantizer_quantor1quantizer_quarkr2quantizer_sinqr3quantizer_spqrr4quantizer_torchaor5quantizer_vptqr6r]r[ get_loggerrnrrMrustrrrrrgrdrcrs74+4'/22F+6@92+-)--/-++1+ <++ O   O     )" O  >6&!  !" O#$  $$   /4$ 9$+$+$ J $  J $  J $ l$ [$$$ $ 9$1$/$ [$}$  !$" J#$$  !   /$ 4   H %&#&#Rttn   !S ! !,rd