qiX ddlmZddlmZddlmZddlmZddlm Z m Z m Z e rddl Z ejeZ d+d e j d e j d e j dzd ed e j f dZde j&j(de j de j de j d e j f dZd e j d e j de j d e j fdZd e j d e j de j d e j fdZdZdZe rXe j4j7dede j4j9dee j4j;deed e j d e j de j d efdZd e j d e j de j d e j fdZ d+d e j d e j de j d e j dzd ed e j f dZ de j&j(de j de j de j d e j f d Z!Gd!d"eZ"e"Z#d#e j d e j fd$Z$ d,ddd%d&d'e%e j&j(dzd ed(ed)ed e%e j&j(f d*Z&y)-)Callable)wraps)logging)GeneralInterface)is_grouped_mm_availableis_torch_availableis_torchdynamo_compilingNFinputweightbias is_transposedreturnc|r5tj|jd|jd}n4tj||jdjd}|||z}|S)aBatched linear layer supporting optional bias and transposed weights. Args: input (`torch.Tensor`): Input tensor of shape (batch_size, input_dim). weight (`torch.Tensor`): Weight tensor of shape (batch_size, output_dim, input_dim) if transposed is `False`, else of shape (batch_size, input_dim, output_dim). bias (`torch.Tensor`, *optional*): Bias tensor of shape (batch_size, output_dim). Default is `None`. is_transposed (`bool`, *optional*, defaults to `False`): Whether the weight tensor is transposed. Returns: `torch.Tensor`: Output tensor of shape (batch_size, output_dim). )torchbmm unsqueezesqueeze)r r r routs O/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/moe.py_batched_linearrDse*ii*F3;;A>ii 34<#;#;B#??Ll44R8#> '++J :NRRWXRY  ! !-"5"5 66roffsc4tj|jd|jd|j|j}d}t |j D].\}}||k(r tj|||||||||}0|S)a( Fallback grouped matrix multiplication used when `torch.nn.functional.grouped_mm` and `torch._grouped_mm` are unavailable or incompatible with `torch.compile` (e.g. non-bfloat16 weights). Args: input (`torch.Tensor`): Input of shape (S, input_dim), sorted by expert id. weight (`torch.Tensor`): Expert weights of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Cumulative token counts per expert of shape (num_experts,). Returns: `torch.Tensor`: Output of shape (S, output_dim). rrr!r9r)rzerosr%r!r9 enumeratetolistmm)r r rIoutputstartiends r_grouped_mm_fallbackrUs[[A Au||SXS^S^ _F EDKKM*3 C<  uS!6!9&s2CD  Mrcp|jdk(sJdt|j|jdk(sJdt|j|jdk(sJdt|j|jd|jdk(s+Jd|jdd |jd|jd|jdk(s+Jd |jdd |jd|jt j t jfvsJd |jt j|jd|jd|j|j S)zRShape/dtype inference stub for `_grouped_mm_fallback` required by `torch.compile`.rz+input must be 2D (S, input_dim), got shape zBweight must be 3D (num_experts, input_dim, output_dim), got shape rz*offs must be 1D (num_experts,), got shape rz offs length z must match number of experts zinput_dim mismatch: input has z , weight has z$offs must be an integer tensor, got rK) r$tupleshaper%r9rint32int64emptyr!r r rIs r_grouped_mm_fallback_faker^s 99;! _J5QVQ\Q\K]J^__  ::<1  LUSYS_S_M`Lab  88:?\HtzzIZH[\\? 99Q<6;;q> )v\$))A,Geflfqfqrsfteu+vv ) ::a=FKKN * (A}V[[QR^DTU * ::%++u{{3 3h7[\`\f\f[g5hh 3 ;;uzz!}fkk!nU\\QVQ\Q\ ]]rcH|j|d|d|d|_y)zjSaves input and weight for backward; offs is stored directly as it is a non-differentiable integer tensor.rrrN)save_for_backwardrI)ctxinputsrQs r"_grouped_mm_fallback_setup_contextrcs%&)VAY/ayCHrc|j\}}tj|}tj|}d}t|jj D]c\}}||k(r tj |||||j|||tj |||j||||||}e||dfS)zuBackward pass for `_grouped_mm_fallback`. Computes grad_input and grad_weight per expert group; offs has no gradient.rrLN) saved_tensorsr zeros_likerNrIrOrPT) ra grad_outputr r grad_input grad_weightrRrSrTs r_grouped_mm_fallback_backwardrks%%ME6!!%(J""6*K ECHHOO-.3 C<  U3'*U3:OP uS!##[s%;QP  {D ((rz!transformers::grouped_mm_fallback) mutates_args) setup_contextcftr|jtjk7ryt S)a Check if torch.nn.functional.grouped_mm or torch._grouped_mm can be used based on availability and compatibility with torch.compile. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. Returns: `bool`: True if grouped_mm can be used, False otherwise. F)r r9rbfloat16rr]s r_can_use_grouped_mmrqs% !fllenn&D " $$rct|||rttjjdrEtjjj |j |j||Sttdr1tj|j |j||Stjjj|||S)aGrouped matrix multiplication dispatcher that uses torch.nn.functional.grouped_mm if available, else falls back to torch._grouped_mm. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. Returns: `torch.Tensor`: Output tensor of shape (S, output_dim). grouped_mmrI _grouped_mm) rqhasattrrnn functionalrsr8r9ruops transformersgrouped_mm_fallbackr]s rruru s$5&$/ 588&& 588&&11%((6<<2H&W[1\ \ UM *$$UXXfll%;V$O O 99 ! ! 5 5eV$ 5 OOrcr|rt|||}nt||jdd|}|||z}|S)aGrouped linear layer supporting optional bias and transposed weights. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim) if `is_transposed`, else of shape (num_experts, output_dim, input_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. bias (`torch.Tensor`, *optional*): Bias tensor of shape (num_experts, output_dim). Default is `None`. is_transposed (`bool`, *optional*, defaults to `False`): Whether the weight tensor is transposed. Returns: `torch.Tensor`: Output tensor of shape (S, output_dim). rtr)ru transpose)r r rIr rrs r_grouped_linearr,sF0%d3%!1!1"b!9E Dj JrcX|j}|jd}|jd}|jd}tj||j dj d|j d}|j d} |j d} ||} tj| } tj| } | | }| | }| | }|jdk(r|jn|j}tj||jd|jdz }tj|dtj}|jr*|j }|j"r|j$|nd}n)|j&}|j"r|j(|nd}t+|||||j,}|jr|j/|}n|j1|}|j2}|j"r|j4|nd}t+|||||j,}||j dz}|| }|j7|||j9d }|j;|j<S) Nrrr rcpu)binsminmax)r$r9r"r#)r!r%rr&rr'r(argsorttypefloatinthistcr)cumsumrZr+r,r-r.r/r0rrr1r2r3r4r6r7r8r9)rrrrr!r:r;r<r=r>r?rAperminv_perm expert_ids_gsample_weights_gselected_hidden_states_g histc_inputnum_tokens_per_expertoffsetsrBrCrDrErFrGs rgrouped_mm_experts_forwardrRs  ! !F  $I##A&J##B'J Z7AA!DKKBPYZbbcefI"**2.N$$R(J+95 == $D}}T"Hd#L%d+5d; +1++*>,$$&LDTDTDVK!KK $:J:JPQW[WgWgjkWklll0au{{KG }},,BF--$00>UY<<=A]]$++L9PT" "2G/aeasasK  }}&&{3 kk+. ~~;?==d)),7dO$%w_TXTfTfM !#3#=#=b#AAL )L '++J :NRRWXRY  ! !-"5"5 66rc:eZdZdZeedZdededeffd Z xZ S)ExpertsInterfacez9Interface for registering custom experts implementations.) batched_mmrsexperts_implementationdefaultrc|tjdn|dk7r||vrtd|dt|||S)zfReturn the requested `experts_implementation`. Also strictly check its validity, and raise if invalid.a You tried to access the `ExpertsInterface` with a `config._experts_implementation` set to `None`. This is expected if you use an Expert Module as a standalone Module. If this is not the case, something went wrong with the dispatch of `config._experts_implementation`eager`zL` is not a valid experts implementation registered in the `ExpertsInterface`)logger warning_onceKeyErrorsuperget)rrr __class__s r get_interfacezExpertsInterface.get_interfaces` ! )   N  $w .3IQU3U*++wx w{17;;r) __name__ __module__ __qualname____doc__rHr_global_mappingstrrr __classcell__)rs@rrrs4C10O .wrapper..__init__s4 $ 8 8 8 DK$DM$DM!.D rcptj|jj}||g|i|Sr)ALL_EXPERTS_FUNCTIONSrr_experts_implementation)rrrexperts_forwardoriginal_forwards rforwardz.wrapper..forwards:3AA 335EO#49$9&9 9rr1)rrrrvrr1)rrrrrr-r+rs @@rwrapperz+use_experts_implementation..wrappersu%.. (00 }  /  /   : ! : }m4(;M %!)  ' r)rrrwModule)rrr-r+rs ``` ruse_experts_implementationrsH*tEHHOO4ehhoo9N4 }%% Nr)NFr)'collections.abcr functoolsrutilsr utils.genericrutils.import_utilsrr r r get_loggerrrTensorboolrrwrrHrUr^rcrklibrary custom_op register_fakeregister_autogradrqrurrrrrrrrlrrrse%,ff   H %Z!%  << LL ,,    \\ D@7 ((//@7<<@7@7<< @7  \\ @7L ell%,,[`[g[g4 ^U\\ ^5<< ^u|| ^`e`l`l ^ )& MM?AUdfg MM CE^_ MM##+%8$%u||%U\\%%Z^%*P <<P LLP ,,P \\ PF!% # <<# LL# ,,# ,,  #  #  \\ #LO7 ((//O7<<O7O7<< O7  \\ O7d<'<.)* "5<< "ELL "372  2(4/22 2  2  %((// 2r