qi8ddlZddlmZddlmZmZddlmZmZm Z m Z m Z e rddl Z e rddlZddlmZddlmZer ddlZddlmZmZe j.eZGdd eZGd d eZGd d eZGddeZ ddej j<dee dzfdZ!dddZ"dZ#ddZ$ddZ%y)N) ConversionOps)get_module_from_nameshould_convert_module)get_available_devicesis_accelerate_availableis_bitsandbytes_availableis_torch_availablelogging)Conv1D)add_hook_to_moduleremove_hook_from_modulec eZdZdZ ddeeeejfdedzdejjdzdeeejffdZ y) Bnb4bitQuantizec||_yN hf_quantizerselfrs X/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/bitsandbytes.py__init__zBnb4bitQuantize.__init__ (N input_dictfull_layer_namemodelreturnc pt|jd}|d}t||\}}t|jt r |j }|j|}tjj|fddi|jj|j} d|_|| iS)a: we need to store some parameters to create the quantized weight. For example, bnb requires 6 values that are stored in the checkpoint to recover the quantized weight. So we store them in a dict that it stored in hf_quantizer for now as we can't save it in the op since we create an op per tensor. r requires_gradFT)listvaluesr issubclass source_clsr Tget_parameter_or_bufferbnbnn Params4bit__dict__todevice_is_hf_initialized) rrrrkwargsvaluemodule_ old_value new_values rconvertzBnb4bitQuantize.convert"sZ&&()!,a)@  f'' 0GGE11/B FF%%eW5WIDVDVWZZ[`[g[gh $(!++rNN __name__ __module__ __qualname__rdictstrr!torchTensorr(Moduler4rrrrsl) '+(, ,d5<<001,t,xx% , c5<< ,rrc eZdZdZ ddeeeejfdejjdzdedzdeeejffdZ y) Bnb4bitDeserializec||_yrrrs rrzBnb4bitDeserialize.__init__>rrNrrrrc Nt|dk(r|S|jD]\}}t|ts|d||< d}|j |}t ||\} } t jjj||d|j| } d| _ || iS)z_ Deserialization of bnb keys. We need 6 keys to recreate the quantized weights rweightF)dataquantized_statsr r,r0T) lenitems isinstancer!poprr'r(r)from_prequantizedr,r-) rrrrr.keyr/ key_weightrEr0r1r3s rr4zBnb4bitDeserialize.convertAs z?a  $**, +JC%&"'( 3 +  +(@ FF%%77&== 8 %)!I&&rr5r6r?rrrArA=sl) )-&* 'd5<<001'xx%'t ' c5<< 'rrAc eZdZdZ ddeeeejfdejjdzdedzdeeejffdZ y) Bnb8bitQuantizec||_yrrrs rrzBnb8bitQuantize.__init__arrNrrrrc t|jd}t|tr|dn|}t||\}}t |j t r |j}|j}|j|j}|jddtjj|jdfddi|j|} || iS)NrSCBcpur F)r!r"rJrr#r$r r%r,r&r*rKr'r( Int8Paramsr+) rrrrr.r/r0r1 value_devicer3s rr4zBnb8bitQuantize.convertdsZ&&()!,&ud3a(@  f'' 0GGE|| ..?HH 5$FF%%ehhuoUUUfUXXYef ++rr5r6r?rrrPrP`sl) )-&* ,d5<<001,xx%,t , c5<< ,rrPc eZdZdZ ddeeeejfdejjdzdedzdeeejffdZ y) Bnb8bitDeserializec||_yrrrs rrzBnb8bitDeserialize.__init__|rrNrrrrc t|dk(r|S|jD]\}}t|ts|d||< t ||\}}d} || } |j |j }|d|d<tjj| fddi|j| j} d|_ | | iS)z. Deserialization of bnb keys. rDrrErSr FT) rHrIrJr!rr&r*r'r(rUr+r,r-) rrrrr.rMr/r0r1rNrEr3s rr4zBnb8bitDeserialize.converts z?a  $**, +JC%&"'( 3 +)@  J'..?HH"5)u FF%%fLELVLOOPVP]P]^ $(!I&&rr5r6r?rrrXrX{sl) )-&* 'd5<<001'xx%'t ' c5<< 'rrXrmodules_to_not_convertc d}|jD]\}}t||sd}tjd5t |t j tfrt |tr|jj\}} n|j}|j} |jdk(rtj j|| |jdu|j |j"}|r|jj$j'tj(|j_ntj j+|| |jdu|j,|j.|j0|j2}|r?|jj$j'|j2|j_|5t5||_|j9d|j;||d}ddd|st<j?d |S#1swY,xYw) a A helper function to replace all `torch.nn.Linear` modules by bnb modules from the `bitsandbytes` library. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. modules_to_not_convert (`list[str]`, defaults to `None`): A list of nn.Linear weights to not convert. If a parameter path is in the list (e.g. `lm_head.weight`), the corresponding module will not be converted. quantization_config (`BitsAndBytesConfig`): The quantization config object that contains the quantization parameters. pre_quantized (`book`, defaults to `False`): Whether the model is pre-quantized or not FNmetallm_int8)has_fp16_weights threshold)dtype)compress_statistics quant_type quant_storageTzYou are loading your model using eetq but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.) named_modulesrr<r,rJr(Linearr rEshape in_features out_featuresquantization_methodr' Linear8bitLtbiasllm_int8_has_fp16_weightllm_int8_thresholdrFr+int8 Linear4bitbnb_4bit_compute_dtypebnb_4bit_use_double_quantbnb_4bit_quant_typebnb_4bit_quant_storagetyper$requires_grad_ set_submoduleloggerwarning) rr[quantization_config pre_quantizedhas_been_replaced module_namer0 new_modulerhris rreplace_with_bnb_linearrs ($224,- V$[2HI  \\& !( -&299f"56ff-06 0C0C-K"("4"4K#)#6#6L&::< J!$!4!4#$ 4/)<)U)U"5"H"H "5"J%2<1B1B1G1G1J1JQVQ[Q[1J1\ )).!$!2!2#$ 4/+BB,?,Y,Y#6#J#J&9&P&P"3"J%1;1B1B1G1G1J1J"5"L"L2K2 )).),0LJ)--e4'' Z@(,%Q( -( - ,-\   La( -( -s GH55H? c^t|tjjst dt |d|j j}|dvr|S|dk(r6tjj|j|j}|S|j|j|_ ttjdr6tjj|j|j}|S|j|jj!ddzdz}|S) z Helper function to dequantize 4bit or 8bit bnb weights. If the weight is not a bnb quantized weight, it will be returned as is. z1Input weight should be of type nn.Parameter, got z instead)r)rUr)int8_vectorwise_dequantrDg@ ?)rJr<r( Parameter TypeErrorru __class__r7r' functionaldequantize_4bitrF quant_staterShasattrrview)rEstatecls_name output_tensor dequantizeds rdequantize_bnb_weightrs fehh00 1KDQWL>Yabcc((H33 <66v{{FDVDVW  yyJJ s~~89nn<FLLDE % 2 23HX]3^ _J *6==%@F}##5fll^DOP##&Efll^Sbchbiij$kl5) % 2 26 :J "& vz*!??6x@'/":x8 MM&--.. /    Z 8 $ 3%6  m  L; m ms AI  I cttdt}tt}|j |s0|rd|xsdd|d}t |t jdyy) zs Validates if the available devices are supported by bitsandbytes, optionally raising an exception if not. supported_torch_devicesz3None of the available devices `available_devices = NzY` are supported by the bitsandbytes version you have installed: `bnb_supported_devices = z`. Please check the docs to see if the backend you intend to use is available and how to install it: https://huggingface.co/docs/bitsandbytes/main/en/installationz+No supported devices found for bitsandbytesFT)rr'setr intersection RuntimeErrorrxry)raise_exceptionbnb_supported_devicesavailable_deviceserr_msgs r!validate_bnb_backend_availabilityrGs$C)BCEJ134  ) )*? @ EFWF_[_E`az{PzQQrr w' 'DE r)NNFr)rEztorch.nn.Parameterr5)F)&rcore_model_loadingrquantizers.quantizers_utilsrrutilsrrr r r bitsandbytesr'r<torch.nnr( pytorch_utilsr raccelerate.hooksr r get_loggerr7rxrrArPrXr>r!r;rrrrrr?rrrs.U&L   H %,m,> ' 'F,m,6''H04 J 88??J I,J\<"*Zr