qi ddlmZddlmZddlZddlmZddlmZGddZ d ed e fd Z d e d e de d e fdZ de de de d e fdZ ddejdee dee de d df dZy)) OrderedDict)ceilN)PretrainedConfig)loggerceZdZdZdeddfdZdededejjdzfdZ deded ejjddfd Z y) CudaGraphBufferz>A fixed-size dict for CUDA graphs with LRU eviction when full.max_sizereturnNcV|dkrtd|||_t|_y)Nrz#max_size must be positive, but got ) ValueErrorr r_storage)selfr s c/opt/pipecat/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/utils.py__init__zCudaGraphBuffer.__init__s, q=B8*MN N  LWM q_lenkv_lenc~|jj||f}||jj||f|SN)rget move_to_end)rrrgraphs r get_graphzCudaGraphBuffer.get_graph!s; !!5&/2   MM % %ufo 6 rrct|j|jk\rG|jjd\}}t j d||j ||j||f<y)NF)lastz!Evicting graph for evicted_key = )lenrr popitemrinforeset)rrrr evicted_key evicted_graphs r set_graphzCudaGraphBuffer.set_graph'sf t}}  .)-)>)>E)>)J &K KK ?    !). ufo&r) __name__ __module__ __qualname____doc__intrtorchcuda CUDAGraphrr#rrr r spHZZZ sCEJJ4H4H44O /s/C/ 8L8L/QU/rr configr c|jdvS)z:Checks if attention mask is needed for the given (config).)z paged|eagerz paged|sdpa)_attn_implementation)r-s rattn_mask_is_neededr0/s  & &*G GGrsize interval_size max_valuecX|dkr|S|dkDrt||z |zn|}t||S)zQReturn the smallest multiple of (interval_size) >= (size), capped at (max_value).r)rmin)r1r2r3paddeds rpad_to_intervalr74s9;?!8T$& '- 7F vy !!rx divide_byalign_tocTtt||z }||zr ||||zz z }|Sr)r(r)r8r9r:s raligned_divider<<s4 DY  A8| XX && Hrattention_maskcumulative_seqlens_qcumulative_seqlens_ksliding_windowcJtj|jj}t t |dz D]}||dz||z }||dz||z }||kr|dk\r ||z dz}nd}t ||||dz} t ||||dz} tj|d| | fj||j|j} tj| |} |dkDr"||z |z } | tj| | z } | |d| | f<y)uBuilds an attention mask inplace using the cumulative seqlens of the query and key. If given a sliding window, it will also apply a sliding window mask on top. The attention mask is not boolean, it uses zeroes and -inf (or its equivalent) so it's more of an attention score bias tensor. The attention mask is a block-diagonal matrix, with each block an attention mask for a single query-key pair. Each of those block is built from a causal mask and, if there is a sliding window, a sliding window mask. An example is represented below, with seqlen_k = 8, seqlen_q = 4 and sliding_window = 6: CAUSAL MASK: █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ █ █ █ █ █ █ █ ░ █ █ █ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 8 - 4 - 6 = -2 offset to the left <─┴─> ░ █ | █ █ █ █ █ █ █ █ ░ ░ | █ █ █ █ █ █ █ █ ░ ░ | ░ █ █ █ █ █ █ █ ░ ░ | ░ ░ █ █ █ █ █ █ ATTENTION MASK (sum of causal and sliding window masks): █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ Another example with seqlen_k = 5, seqlen_q = 3 and sliding_window = 2: CAUSAL MASK: █ █ █ ░ ░ █ █ █ █ ░ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 5 - 3 - 2 = 0 offset to the left <┴> | ░ █ █ █ █ | ░ ░ █ █ █ | ░ ░ ░ █ █ ATTENTION MASK (sum of causal and sliding window masks): ░ █ █ ░ ░ ░ ░ █ █ ░ ░ ░ ░ █ █ r.)dtypedevice)diagonalN) r)finforBr5rangerslicefullshaperCtriutril)r=r>r?r@ min_valueiseqlen_qseqlen_kcausal_diagonal query_range key_range minus_infmaskedsliding_diagonals rbuild_attention_maskrVCsSt N00155I 3+,q0 1='A.1Ea1HH'A.1Ea1HH h 8q=&1A5OO035I!a%5PQ .q13GA3NO JJ 3 Y6 7 = =  &&!((  I@ A '(2^C  ejj5EF FF6<sK23-=r)r) collectionsrmathrr) transformers.configuration_utilsrrequestsrr boolr0r(r7r<TensorlistrVr,rrr^s$ =//.H 0HTH "#"c"c"c" c c S S  Q=LLQ=s)Q=s)Q= Q=  Q=r