qiddlmZddlmZmZddlmZmZer ddlZddl m Z eje Z GddeZ d deedzfd Zy) ) ConversionOps)get_module_from_nameshould_convert_module)is_torch_availableloggingNc eZdZdZ d deeeejfdejjdzdedzdeedzdeeejff dZ y) QuantoQuantizec||_y)N) hf_quantizer)selfr s R/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/quanto.py__init__zQuantoQuantize.__init__s (N input_dictmodelfull_layer_name missing_keysreturnc 6t|jd\}}|d}ddlm}||||t ||\} }t j | jj| _t j | jj| _ | jd| j_ d| _ |jddd} |j| d|j| d |j| d iS) Nrr)_load_parameter_into_modelFT.z.weightz .input_scalez .output_scale)tupleitemsmodeling_utilsrrtorchones input_scaleshape output_scalefreezeweight requires_grad_is_hf_initializedrsplitdiscard) r rrrrkwargs_valuermodule module_names rconvertzQuantoQuantize.converts))+,Q/5a?"5/5A(@ "ZZ(:(:(@(@A#jj)<)<)B)BC &+ #$(!&,,S!4Q7  }G45 }L9: }M:; r)NNN) __name__ __module__ __qualname__rdictstrlistrTensornnModuler-rrr r s) )-&*)- d5<<001xx%t  3i$&  c5<< rr modules_to_not_convertc ddlm}m}m}m}m}m}||||d} d||d} d} |jD]4\} } t| |stjd5d}t| tjr]|| j| j| j du| j"j$| |j&| |j(}ntt| tjj*rP|j(D|| j,| j.| j0| j du| |j( }|d } |j3| |ddd7| st4j7d |S#1swY[xYw) a Public method that recursively replaces the Linear layers of the given model with Quanto quantized layers. Returns the converted model and a boolean that indicates if the conversion has been successful or not. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`QuantoConfig`, defaults to `None`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*, defaults to `None`): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. r) QLayerNormQLinearqfloat8qint2qint4qint8)float8int8int4int2N)Nr@rAFmeta) in_features out_featuresbiasdtypeweights activations)rJTzYou are loading your model using quanto but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)optimum.quantor:r;r<r=r>r? named_modulesrrdevice isinstancer5LinearrErFrGr#rHrIrJ LayerNormnormalized_shapeepselementwise_affine set_submoduleloggerwarning)rquantization_configr8r:r;r<r=r>r? w_mapping a_mappinghas_been_replacedr,r+ new_modules rreplace_with_quanto_layersr\>s}$QP"E5%PIw>I$224= V$[2HI  \\& ! =J&")),$ & 2 2!'!4!4D0 ----%&9&A&AB )*=*I*I J  FEHH$6$67rbs]/U/   H % ] J/39!I,9r