qi ddlZddlmZddlmZ d dej j dejdejdejd ejdzd ed ejfd Zy)N)PagedAttentionCache)!lazy_import_paged_flash_attentionmoduleqkvattention_maskcachereturnc t|jj} t|ddsdn|jdz df} | dk(rdnd} |)|j |||j | d | d  \}}t|tr || }| | } d | vrd | jd ini}| |jdd jdj|j|j|jtj|jtjj!|| f|j"d| d|}t|t$r|d}|dfS)aPerform the forward pass of attention with paged key-value cache. This function handles the cache updates and performs the attention computation using the flash_attn_varlen_func for efficient processing. Args: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. but if there is a block table it can be the full k v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. but if there is a block table it can be the full v cu_seq_lens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seq_lens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. sliding_windowF)rrfull_attentionsliding_attentionN read_index write_index) key_states value_states layer_idxrrs_auxrT) softmax_scalecausal window_size)rconfig_attn_implementationgetattrrupdater isinstancedictget transposesqueeze contiguoustotorchint32clonescalingtuple)rrrr r r cu_seq_lens_q cu_seq_lens_k max_seqlen_q max_seqlen_kkwargsflash_attn_varlen_funcr layer_type custom_kwargs attn_outputs W/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/flash_paged.pypaged_attention_forwardr6syF?v}}?a?ab%,V5Eu%MXTZTiTilmTmopSqN%3x%?!EXJ ||&&l+}-  1-&%j1 #J/ 6=6GWfjj12RM( Aq!!!$//1  %%++- nn"   K+u%!!n  )NNNNNN) r'generation.continuous_batchingrmodeling_flash_attention_utilsrnnModuleTensorr6r7r5r>s @N+/!%H HHOOH ||H ||H || H LL4' H  H \\Hr7