qiw XdZddlmZddlmZddlmZmZmZm Z er ddl Z ddl m Z erddl mZddlmZmZerdd lmZe j(eZdd Zd ed ed dfdZdded ededefdZGdde jj6ZddeedzfdZddZ y)zZHIGGS through FLUTE (Flexible Lookup Table Engine for LUT-quantized LLMs) integration file)sqrt)should_convert_module)is_flute_availableis_hadamard_availableis_torch_availableloggingN)prepare_data_transposed) TuneMetaDataqgemm_v2)hadamard_transformctdt|jzDcgc]}d}}|D]/}|j|}|dz |zdz|z}||z } | |d|zdz <1tjj ||d|Scc}w)Nrrconstant)rangelenshapenn functionalpad) tensordimshad_block_sizevalue_pad_dimsdimsizenext_multiple_of_1024deltas Q/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/higgs.py pad_to_blockr#$s S%6!678a8H8'||C "&(~!=!A^ S%,!&cA ' ==  VXz5 AA9s Bpnreturnz torch.Tensorc||fdk(rtjgddgddgddgdd gd d gd d gddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gd.d/gd0d1gd2d3gd4d5gd6d7gd8d9gd:d;gdd?gd@dAgdBdCgdDdEgdFdGgdHdIgdJdKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZd[gd\d]gd^d_gd`dagdbdcgdddegdfdggdhdigdjdkgdldmgdndogdpdqgdrdsgdtdugdvdwgdxdygdzd{gd|d}gd~dgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgdd gd d gd d gddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gd.d/gd0d1gd2d3gd4d5gd6d7gd8d9gd:d;gd<d=gd>d?gd@dAgdBdCgdDdEgdFdGgdHdIgdJdKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZd[gd\d]gd^d_gd`dagdbdcgdddegdfdggdhdigdjdkgdldmgdndogdpdqgdrdsgdtdugdvdwgdxdygdzd{gd|d}gd~dgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgdĐdgdƐdgdȐdgdʐdgd̐dgdΐdgdАdgdҐdgdԐdgd֐dgdؐdgdڐdgdܐdgdސdgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgS||fdk(rtjgddgddgddgd d gd d gd dgddgddgddgddgddgddgddgddgdd gd!d"gd#d$gd%d&gd'd(gd)d*gd+d,gd-d.gd/d0gd1d2gd3d4gd5d6gd7d8gd9d:gd;dgd?d@gdAdBgdCdDgdEdFgdGdHgdIdJgdKdLgdMdNgdOdPgdQdRgdSdTgdUdVgdWdXgdYdZgd[d\gd]d^gd_d`gdadbgdcddgdedfgdgdhgdidjgdkdlgdmdngdodpgdqdrgdsdtgdudvgdwdxgdydzgd{d|gd}d~gddgddgS||fdk(retjddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddggS||fdk(rEtjdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdggS||fdk(r-tjdgdgdgdgdgdgdgdggS||fdk(r!tjdgdgdgdggStd|d|(N)rgg f?gBg]?g_g\?g9g@g?gg@y?gЂ @g@?g?g`Qcg2g:7gdig c?g`ֿg`ؿg?g+@g?g`(f?g@8ig=g򩣿g g+gfg gLgx?g`*(g?g/տg@/aſgEggg ?gGg{?gh?g_a?g?g?g3@g#?gygѱܿg @g׿g d,gg&gٿg`{X?gYb?g@"gيg3gTg@@gkgg`{ڿg?g>g ?gg`5g ?g?g5?g?gIؿg A?g?gѿg9?g@@gzg j g g@g@N?g}g?gZg`@gcg ҿg@=Dgg?g`gb@g{@gh@gg?g]ƿgK?ggw?g`?g 4?g b?gg`ſgp?g@?g@Q=s?g@տgg@@g1u?g͔ƿg@?g䉫?g`@g_g@g,?g`dg`T?gL?gb?g3?gȃgxg v?g`?g섥gK3g/ۿg`Eg@#?g3?g g c?g1g?g`%?g%?g@w@gI?gkg g@J߿g?gǿgFֿg9?g`g8?g` @gxg@wg x?gg@ҿg͆gvg?gn@g`9ǿgY?g g@?g[m?g F?g@ ?g`Ьg`Ggc?g@gPug@?gg@Hg4g@gwg#?gd?gMg g ?g?g@vPgſgr?gLg ?g@?g:ڿg ?g?g` ?g@_g$Rg Z?g3g`g ?g@Ogew@g LgV?gߩҿgV?g?ggŠ?g`;?g Կg`@g-g?g`gZY?gg ?g8;?g?gaοg@Rh?g`F ?gA?g@?gg`g8?g`>?g?g@pgi?g ?g`пg@@gB?gbg@@gg`\gg)W?g`?g`g`@g 0$ӿg >gi?gcg?gH?g T?g ]g.ܿgyؿg*g`F?g@g`Is?g_;?gE?g g`޿g.g@\?g`[?g #?g ?g?ggN?gjg=?gR޿g耎?g`gLg@g=g@g@?gR@grgg?g`v_?g@T?g?g@?g8?g{}gi?gSg@k?gs?g?g?g _g@3@g?g@ ?g`Xrg[gg?g`?g@?g5g?)r@ggx?g gk?g gl?gy?g6g?gпg`5?gk@g$gz?g 1J@g ?go?g.ghgӿg_?g?g$?gg6@g@T?g-?g g@ggù?g.?g?gfg?g,'?g?g^@g@Lg>ۿg4bg?g Ng޿g ?gnZgtgi~@g Jf?g^Gg`mg`6@g/?g ?gOܿg?gggg^DgFg|g1g`8g@*?gֿg%&9:?A @$%78AA B%&89CA D%&89EA F%&9:GA H$%78IA J#$67KA L#$67MA N$%78OA P&';$%89?A @%&9:AA B%&9:CA D%&89EA F%&89GA H%&89IA J$%78KA L$%89MA N$%89OA P&';>bits group_size hadamard_sizec t|jdk(sJdt|d||zzj|j}t j j|ddz}|j}|j}|jdt j}t|dg|}|jd|z} |jd| |}t j j|d} t|d| dddddfz }t|dg|j|jd| d|}t j|jdd|t j } td|jdd D]U} t j d|| | d zz|j"z|z d jt j| | | d zW~| j| jdd} | t%|z } t'| t j(| j|||zd |j||||||d \}} } }}|| | |j+t j,|dS)Nrz%Only 2D weights are supported for now)axisT)copydtyperrdevicer8r*)rF)num_bitsr2 vector_sizer8r:check_correctnessr8)weightscalestablestables2 tune_metadata)rrr/tor:r-linalgnormr8float32r#reshaper emptyuint8rargmaxTrr repeat_interleaveviewfloat16)r?r1r$r2r3grid grid_norm_2r:r8multr@codesirArBrCs r"quantize_with_higgsrUsY v||  !J#JJ ! !Q1t8_ - 0 0 ?D,,##Dr#2a7K ]]F LLE YYD Y 6F &1#} 5F <<?m +D ^^Bm 4F \\  vB  /F  *VAq$J-? ?F&1#q ) 1 1&,,q/4Q OF KK Sb)& LE 1fll1or *p!LLVAB-?)?$&&)H;)V\^_bbchcncnoa!b&p MM%++a." -E d=) )F5L   % 0-:2MSTU  62FFFG]<)r3) r#r3rc Exceptionr r?r@rArBrNr-rGrC)rdxs r"forwardzHiggsLinear.forwards RD$"4"4 5 >> !JK K KK KK KK LL  EMM  2 NN   ,,  r0)TNNr() __name__ __module__ __qualname__intr-r8r:r_ri __classcell__)res@r"rWrWsu $(&*!'0'0'0 '0 {{T! '0 t#'0'0'0R r0rWmodules_to_not_convertc d}|jD]\}}t||stjd5t |t j rt|j|j|jdu|j|j|j}t||_|j!d|j#||d}ddd|st$j'd|S#1swYxYw)a_ Public method that replaces the Linear layers of the given model with HIGGS quantized layers. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. modules_to_not_convert (`list[str]`, *optional*, defaults to `None`): A list of nn.Linear weights to not convert. If a parameter path is in the list (e.g. `lm_head.weight`), the corresponding module will not be converted. quantization_config (`HiggsConfig`): The quantization config object that contains the quantization parameters. FmetaN)r]r;r3r2TzYou are loading your model using eetq but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.) named_modulesrr-r: isinstancerLinearrWrXrYr]r1r3r2type source_clsrequires_grad_ set_submoduleloggerwarning)modelrpquantization_confighas_been_replaced module_namemodule new_modules r"replace_with_higgs_linearr&s$224) V$[2HI  \\& ! )&")),(&&''D0055"5"C"C2==  )-V %))%0##K<$(! ) ))$   L+ ) )s BC==D c Ltj5|jD]e\}}|g}|j|t |t r|j }|j}tjj|||jdu|jj|jj|j|<|tj||jj|jjj j#|j|j$_t)t+|j-dkDr t/||}|j1dh|cdddS#1swYyxYw)a Dequantizes the HiggsLinear layers in the given model by replacing them with standard torch.nn.Linear layers. Args: model (torch.nn.Module): The model containing HiggsLinear layers to be dequantized. current_key_name (list, optional): A list to keep track of the current module names during recursion. Defaults to None. Returns: torch.nn.Module: The model with HiggsLinear layers replaced by torch.nn.Linear layers. N)r]r:r8r9r)current_key_namer5)r-no_gradnamed_childrenappendrtrWrXrYrrur]r@r:r8_moduleseyerL contiguousr?datarlistchildrendequantize_higgspop)r|rnamerrXrYrs r"rrQsT !002 %LD&'#%  # #D )&+.$00 %22 ',xx D0!==// ---- (7(t$4:IIk&--2F2FfmmNaNab4!JJLt$++04)*+a/$%5   $7 %8;s E;FF#)r)r,rr(rj)NN)N)!__doc__mathrquantizers.quantizers_utilsrutilsrrrr r-torch.nnrflute.integrations.higgsr flute.tuner r fast_hadamard_transformr get_loggerrkrzr#rnr/rUModulerWrstrrrr0r"rs]?ZZ@1:   H %BE?cE?cE?nE?P 0c0#0s0ad0f9 %((//9 x(T#Y=M(V'r0