qi( zdZddlmZddlmZddlmZmZer ddlZddl m Z eje Z dadZGdd e j Z dd eedzd efd Zd ej,dedefdZdej,dej,dej,dedef dZGddeZGddeZy)a Metal affine quantization integration for transformers. This module provides: - ``MetalLinear``: a drop-in replacement for ``nn.Linear`` that stores weights as affine-quantized uint32 packed tensors and uses the ``quantization-mlx`` Metal kernels for the forward pass. - ``replace_with_metal_linear``: walks a model and swaps every eligible ``nn.Linear`` with ``MetalLinear``. - ``MetalQuantize`` / ``MetalDequantize``: weight conversion operations that participate in the new ``WeightConverter`` pipeline. Weight layout (transposed, matching ``affine_qmm_t``): - ``weight``: ``[N, K_packed]`` (``uint32``) -- K is the packed dimension. - ``scales``: ``[N, K // group_size]`` (``float16 / bfloat16``) - ``qbiases``: ``[N, K // group_size]`` (same dtype as scales) The kernel call is ``affine_qmm_t(x, weight, scales, qbiases, group_size, bits)`` which computes ``y = x @ dequant(weight).T``, identical to ``nn.Linear``. ) ConversionOps)should_convert_module)is_torch_availableloggingNct ddlm}|datStS#t$r}t d|d|d}~wwxYw)z>Lazily load the quantization-mlx kernel from Hugging Face Hub.N) get_kernelz0kernels-community/mlx-quantization-metal-kernelsz9Failed to load the quantization-mlx kernel from the Hub: zm. Make sure you have `kernels` installed (`pip install kernels`) and are running on an Apple Silicon machine.) _metal_kernel hub_kernelsr Exception ImportError)r es ^/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/metal_quantization.py_get_metal_kernelr3s`  /&'YZM =  KA3O??   s" A;Ac eZdZdZdej ddfdedededed ef d Zd ejd ejfd Z y) MetalLinearz A quantized linear layer that stores weights in affine uint32 packed format and uses the ``quantization-mlx`` Metal kernels for the forward pass. Parameters match ``nn.Linear`` with additional quantization metadata. F in_features out_featuresbiasbits group_sizec:tjj|||_||_||_||_d|z}||z}||z} |tjk(rAtjtj||tjd|_ n2tjtj|||d|_ |tjk(rtjnd} tjtj|| | d|_ tjtj|| | d|_|r.tjtj||_y|j!ddy)N )dtypeF) requires_gradr)nnModule__init__rrrrtorchuint32 Parameterzerosweightfloat32scalesqbiasesrregister_parameter) selfrrrrrr elems_per_intk_packedn_groups scales_dtypes rr!zMetalLinear.__init__Qs' 4 &( $d -/*, ELL ,,u{{<QVQ]Q]'^nstDK,,u{{<TY'ZjopDK(-(=u}}4 ll5;;|X\#Zjop ||EKK hl$[kpq  U[[%>?DI  # #FD 1inputreturnc|jjtjk7r5tj j ||j|jSt}|j||j|jj|j|jj|j|j|j}|j||jz}|SN)r&rr"r#r functionallinearrr affine_qmm_tr(tor)rr)r+r1kerneloutputs rforwardzMetalLinear.forwardss ;;   ,==''t{{DIIF F"$$$  KK KKNN5;; ' LLOOEKK ( OO II   99 dii'F r0N) __name__ __module__ __qualname____doc__r"r#intboolr!Tensorr;r0rrrIsjll 2 2 2 2  2 2DU\\ellr0rmodules_to_not_convert pre_quantizedc |jr|S|j}|j}d}|jD]z\}}t ||st |t js.|rinddi} td|j|j|jdu||d| } |j|| d}||stjd|S)a` Replace every eligible ``nn.Linear`` with ``MetalLinear``. Args: model: the ``PreTrainedModel`` (on the meta device at this point). modules_to_not_convert: module names to leave untouched. quantization_config: the ``MetalConfig`` instance. pre_quantized: ``True`` when loading from a quantized checkpoint. FrN)rrrrrTzYou are loading a model with Metal quantization but no nn.Linear modules were found. Please double check your model architecture.rC) dequantizerr named_modulesr isinstancerLinearrrrr set_submoduleloggerwarning) modelrDquantization_configrErrhas_been_replaced module_namemodule module_kwargs new_modules rreplace_with_metal_linearrUs%%  # #D$//J$224% V$[2HI  fbii ("/Bgt_M$"..#00[[,%   J    Z 8 $ !%$  ; Lr0r&rrc |j\}}d|z}d|zdz }||z}|jj|||}|jdj} |j dj} | | z |z j d} | } || jdz | jdz } | jj d|jtjj||} ||z}tj||tj|j}t|D]}|| d d |d |f||zzz}|jtj| | fS) aP Quantize a 2-D float weight ``[N, K]`` into packed uint32 + scales + biases. Returns ``(w_packed, scales, biases)`` with: - ``w_packed``: ``[N, K // (32 // bits)]`` uint32 - ``scales``: ``[N, K // group_size]`` float32/float16/bfloat16 - ``biases``: ``[N, K // group_size]`` float32/float16/bfloat16 rr )dimg:0yE>)minrrdeviceN)shapefloatreshaperYvaluesmaxclamp unsqueezeroundr8r"int32r%r[ranger#)r&rrNKr,max_valr. w_groupedw_minw_maxr(biasesw_intr-w_packedis r_affine_quantize_tensorrpsm <r?r!dictrrCr0rrwrws) $ T r0rwc2eZdZdZdZddededzdefdZy) MetalDequantizez Dequantize (weight, scales, qbiases) back to a full-precision tensor. Used when ``dequantize=True`` is set in the config to fall back to a normal ``nn.Linear`` on devices without MPS. c||_yr4ryr{s rr!zMetalDequantize.__init__r|r0Nr}full_layer_namer2c 4|jjj}|jjj}t |dkr||diS|dd}|dd}|dd}t |||||} || j |jiS)Nrzweight$rr(r))rzrOrrlenrur8r) r+r}rrrr quantizedr(r)rts rrzMetalDequantize.converts  4499&&::EE z?Q #Z %:; ;y)!, H%a(Y'*))VWjRVW&,,!788r0r4)r<r=r>r?r!rstrrrCr0rrrs+) 9$ 9t 9Y] 9r0r)NNF)r?core_model_loadingrquantizers.quantizers_utilsrutilsrrr"torch.nnr get_loggerr<rLr rrJrrrrArUrBr@rprurwrrCr0rrs*/?/   H % ,;"));@04 / I,/ /d5ELL5c55Bll$)LL:?,,TW_b. M @9m9r0