qiTGjUddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl mZmZddlmZd d lmZej(eZ dd lmZmZmZmZmZdd lmZdd lmZ  ddlm!Z! ddlm#Z$dZ%ejLjOddjQZ)dZ*e)evZ+de,fdZde,fdZ#deddideddiejZeddiejZeddiejZed d!iejZed"d!iejZeddid#ded$d%iej\ed&d'ejZed&d'iejZed(d'iejZed&d'iejZed&d)id*dejZej^zed+d,d -iidejZej^zed+d.d -iidejZej^zed+d/d -iidejZej^zed+d0d -iidejZej^zed+d1d -iidejZej^zed+d2d -iid3 Z0e1e,e1ee,zee1eefzffe2d4<e!0ejZe!d5d67iejZe!d5d67id8e0d9<d:Z3dYd;Z4dCd dDdEd dDdEd dDdFZ5e1e,e1e,e,ffe2dG<iZ6e1e,e dzfe2dH<dIe,dzdJe7fdKZ8 dZdIe,dLedzdMe7dJe dzfdNZ9e6fdOe,dPe1e,e dzffdQZ: d[dOe,dRe,dzdSe;e,zdzdMe7dJe f dTZdUeedWZ?gdXZ@y#e"$rdZ!Y wxYw#e"$rdZ%YwxYw#e"$r dZ*dZ+d<Zd=Z#Gd>d?Zd@ZdAZdBZ4YwxYw)\N)Callable)contextmanager) ModuleType)version)ENV_VARS_TRUE_VALUESlogging)is_kernels_available)flash_attention_forward)DeviceLayerRepositoryModeregister_kernel_mappingreplace_kernel_forward_from_hub) get_kernel)use_kernel_forward_from_hub)FuncRepository)use_kernel_func_from_hubTFUSE_HUB_KERNELSYES layer_namecbtr t|StjdtdS)NFkernels hub usage is disabled through the environment USE_HUB_KERNELS=c|SNclss W/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/hub_kernels.pyz-use_kernel_forward_from_hub..Fss)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)rs r rr?s2 7 C C   XYvXwx # "r" func_namectrtr t|Ststj ddStj dt dS)Nzwuse_kernel_func_from_hub is not available in the installed kernels version. Please upgrade kernels to use this feature.rc|Srrfuncs r r!z*use_kernel_func_from_hub..Usr")r#_has_use_kernel_func_from_hub!_kernels_use_kernel_func_from_hubr%r&r')r(s r rrHsV  =4Y? ?0##B% $##\]z\{|% $r"cudaz!kernels-community/deformable-detrMultiScaleDeformableAttention)repo_idrzkernels-community/moe Llama4TextMoezkernels-community/liger_kernels LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)r/rocmxpumpsnpuzmedmekk/triton-llama-mlpTritonLlamaMLPzkernels-community/megablocksMegaBlocksMoeMLPzahadnagy/megablocksCPUMegaBlocksMoeMLP)r/r5r6cpuzkernels-community/activationFastGELU)r1rr QuickGELUNewGELUSiluGeluGeluTanh) r0r2r4MLPr:r=r>r?SiLUGeLUrB_KERNEL_MAPPINGzkernels-community/rotaryapply_rotary_transformers)r1r()r6r/rotary_pos_embcT|vxs"tfd|jDS)Nc3ZK|]"}t|txr t|$ywr) isinstancedicthas_key).0vkeys r zhas_key..s&[1z!T2Fwq#F[s(+)anyvalues)drPs `r rMrMs'ax[3[PQPXPXPZ[[[r"cp|t}t|drtds tdt |y)Nr6z0.10.2) MIN_VERSIONzfkernels uses an incompatible version. Please install the latest version with `pip install -U kernels`.)rFrMr ImportErrorr)mappings r $register_kernel_mapping_transformersrYs8 ?%G 7E "+?H+Ux  (r"c d}|S)Nc|Srrrs r decoratorz.use_kernel_forward_from_hub..decoratorsJr"rargskwargsr\s r rrs r"c d}|S)Nc|Srrr+s r r\z+use_kernel_func_from_hub..decoratorsKr"rr]s r rrs r"ceZdZdZy)rctd)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeError)selfr^r_s r __init__zLayerRepository.__init__ sop pr"N)__name__ __module__ __qualname__rgrr"r rr s qr"rctd)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rdr^r_s r rr s l  r"ctd)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.rdrls r rrssttr"ctd)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.rdrls r rYrYs q  r"zkernels-community/causal-conv1d)r1rzkernels-community/mamba-ssm)z causal-conv1dz mamba-ssmzfalcon_mamba-ssm_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationreturnc>|duxrtjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rqs r is_kernelrv$s- 4' c II@BU V^b br"attention_wrapperallow_all_kernelsctddlm}ddlm}d|vr|j ddn|}t |syt s tdd|vr%|j d\}}|j}n|}d}|j}|jd \}}} |j}| r| jnd} t|| | } t| dr |t}|} n| t| |} |j!| |j!||d| S#t$r} td |d | d d} ~ wwxYw)a Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continious batching, where we want the `paged` wrapper, which calls the paged cache. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if it is a custom kernel outside of the `kernels-community` hub repository. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|r N`kernels` is either not installed or uses an incompatible version. Please install the latest version with `pip install -U kernels`.:@)revisionrxz-An error occurred while trying to load from 'z': .flash_attn_varlen_funcflash_attention_2) masking_utilsrzmodeling_utilsr{splitrv_kernels_availablerWstrip partitionr Exception ValueErrorhasattrr getattrregister) rqrwrxrzr{actual_attn_namer1 kernel_name_revkernelekernel_functions r load_and_register_attn_kernelr,si =8 ; - %--c37/**5  '/ //0F TI"))Y"E  ' (-@-B ,000O3DF'- $ $(GK ; =! (#'GK : ; 9 (#'GK 4 ; 9 ( '"&  ' ,'+ $ ;   ,s= AD>$E5EE- EE EEE*)E*rrcVddlm}ts td|j dd}|dk7r |s t dd||d }t jjd }tj|tjd k\rt|||| St||| S)Nr) __version__r}/rzkernels-communityziYou need to specify `allow_all_kernels=True` to use kernels outside of the `kernels-community` repository transformers) frameworkrr1kernelsz0.10.4)rr user_agentr) rrrWrrrmetadatar pkg_versionparseget_kernel_hub)rrrrxr repo_parentrkernels_versions r rrs   -  ##C(+K))2C w   .+R]^J((00;O)[->->x-HHkHgZdeekHgNNr" module_namesc8ttrgfd}|S)a This decorator attaches the target function as an attribute of the module. The function must already be decorated with @use_kernel_func_from_hub this decorator then wraps it as an nn.Module internally. When kernelize is later applied to the full model, the function can be accessed as a regular module attribute and kernelized just like any other layer. The kernelization is performed in place, modifying the module directly. c<|jfd}||_|S)NcF|g|i|D]}t|d|y)N rotary_fn)setattr)rfr^r_fnr orig_inits r new_initz8use_kernelized_func..decorator..new_inits0 d ,T ,V ," /k2. /r")rg)rrrrs @r r\z&use_kernelized_func..decoratorsLL  /    r")rKr)rr\s` r use_kernelized_funcrs#,)$~   r"c#,K dadday#dawxYww)z Context manager used to adjust the value of the global `ALLOW_HUB_KERNELS`. This is needed, as this argument cannot be forwarded directly to the `__init__` of the models, where we set the attention implementation. TNF)ALLOW_ALL_KERNELSrr"r allow_all_hub_kernelsrs""  "Es ) rrrrrYrrrrr)NF)NNF)Aimportlib.metadatarosrtcollections.abcr contextlibrtypesr packagingrrutilsrr utils.import_utilsr flash_attentionr get_loggerrhr%rr rrrrrrrr$rrWrr.r-environrupperr'rr#str INFERENCETRAINING TORCH_COMPILErFrL__annotations__rMrYrorpboolrvrrintlistrrr__all__rr"r rs $%,154   H %z  * .Y(,%%'JJNN3De$L$R$R$T!48LL## %C %" O;:*  O/* =-!=-!7(!;(!=-!5  D O2+  :1 :1! 11!:1!:4!/ > !3!33_:)6  !3!33_:*6  !3!33_:(6  !3!33_:vWX6  !3!33_:vWX6  !3!33_:z[\6 wBeOT#tFSL/DI^D_2_$_``aBJ!6B]! 6B]! - ()\)^"CqQ:qI$AaP2T#tCH~-. 8:S*t"33493:$di::19D:\`:$:zPf, #, S*t:K5K0L, b $# OODjO3Y O O  O:d8nx&?6 " " i  .(-%.~!   qq u =! sU N (M1/M?7I&N 1M<8N ;M<<N ?N N N  N "N21N2