qi[iddlmZmZddlmZddlmZddlmZddl m Z ddl m Z e dZ d ee d eeee ffd ZGd d ZGddZGddeZGddeZGddeZy))ABCabstractmethod)deque)Iterator)ceil)TypeVar)loggerTxsreturnc#XKt|dz }|dddD] }||f|dz}yw)Nr )len)r indexxs k/opt/pipecat/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/cache_manager.pyreverse_enumeraters< GaKE "XQh  s(*cPeZdZdZdededzdeddfdZdefdZede fd Z y) BlockaA class to represent a block managed by the block manager. We say that a block is complete when the physical KV cache it points to is fully computed. A block can have a parent, which is the block that came before in the sequence. Once a block is complete, it is given a hash, which takes into account the tokens ids of the block, the layer (group_id) it belong to and its parent's hash (if there is a parent).id_ parent_idNgroup_idr cJ||_||_||_d|_d|_y)Nr idrrhash ref_count)selfrrrs r__init__zBlock.__init__'s%%.% $ c d|jd|jd|jd|jd|jd S)Nz Block(id=z , parent_id=z , group_id=z, hash=z , ref_count=)rrs r__repr__zBlock.__repr__.s_477)</?{4==/Y`aeajaj`kkwx|yGyGxHHIJ Jr!c|jduS)N)rr$s r is_completezBlock.is_complete1syy$$r!) __name__ __module__ __qualname____doc__intr strr%propertyboolr'r!rrr!sVS  C C$J # $ J#J%T%%r!rcVeZdZdZdededdfdZedefdZdedefd Z ded edzd ed ede edzf d Z de eded ed ede e e edze ee eff dZ deddfdZdeddfdZde ed eddfdZdeddfdZdede ede eddfdZdedzde ed edefdZy) BlockManagera1A class to manage the number of free blocks and block re-use. When a block becomes in use, a flag is passed to determine if the block is shareable or not. If it is, then a Block object is created and kept track of internally. It can have the following states: - in use: one or more requests references this block, thus it cannot be written over. The number of requests referencing this block is stored as ref_count in the Block object. - un-initialized: the block points to a space in the KV cache tensor that contains no data yet. Those blocks can be given as free blocks to new requests without any overhead. - initialized: the block is complete and was used by one or more request that are finished. It contains KV cache data and its hash is stored in the hash table. If a new request needs a block with the same hash, we increase the ref_count of the block and remove it from the list of initialized blocks, because it is now in use. Still, the block can be freed if no un-initialized blocks are left. In that case, we remove its hash from the hash table. If the block is not shareable, we just use the block manager as a FIFO structure where blocks are either free or in use. Sharability is determined by the type of cache allocator: blocks created for full attention layers are shareable, while blocks created for sliding window attention layers are not. There is no structure to keep track of the blocks in use: if a block is neither un-initialized nor initialized, it is in use. num_blocks block_sizer Nc|||_||_tt||_i|_i|_i|_y)z^Initializes the block manager with a given number of blocks (num_blocks) of size (block_size).N)r3r4rrange_uninit_block_ids_init_block_ids _hash_to_id _id_to_block)rr3r4s rr zBlockManager.__init__Js:$$!&uZ'8!902+-.0r!cXt|jt|jzS)zfReturns the number of free blocks left. Both initialized and uninitialized blocks are considered free.)rr7r8r$s rnum_free_blockszBlockManager.num_free_blocksSs%4))*S1E1E-FFFr!n_blocksct|j|k\ry|t|jz }t|j|kryt|D]n}|jj d}|j |}|j j|j|jj|py)zChecks if there are enough free blocks to allocate the requested number of blocks (n_blocks). If there are not enough uninitialized blocks, we uninitialize the required number of initialized blocks.TFr) rr7r8r6popitemr:r9poprappend)rr=block_to_uninitialize_id_to_uninitializeblocks rhas_enough_free_blocksz#BlockManager.has_enough_free_blocksXs t%% &( 2 (3t/E/E+F F t## $'< <,- >A!%!5!5!=!=!?!B %%&89E     ,  " " ) )*< =  > r! last_block_id shareablerc|j|syt|Dcgc]}|jj}}|r%|D] }t |||}||j |<|}"|Scc}w)aReturns a list of (n_blocks) free block and mark them as no longuer free in the internal data structures. If the (shareable) flag is set to True, a Block object is created to keep track of the block, with the (last_block_id) to indicate the last block id in the sequence, also named the parent block. If the manager cannot find enough free blocks, it returns None.N)rFr6r7popleftrr:) rr=rGrHrrCallocated_block_idsblock_idrEs rget_free_blockszBlockManager.get_free_blocksks**84INxYAt55==?YY / )h x@.3!!(+ (  ) #"Zs!A, parent_blocks num_forksc(g}|rT|D]O}|j|}|jr1|j|j|xj|z c_Ont |t |z }|dk(rt |D cgc]} |dd c} ggfSg} g} g} |r|dnd} t |D]Y} |j|| ||}|dggfcS| j||z| j|| d| j|[| | | fScc} w)u#Fork a given list of (parent_blocks) as many times as (num_forks). If the blocks are (shareable), we use reference on the blocks that are complete. Otherwise, we allocate new blocks and keep track of their indices to later copy the physical cache. For instance, when forking 4 blocks for 2 children: Parent blocks: [0, 1, 2, 3], with all blocks being complete except the last one (block 3). ----------------------------------------- IF BLOCKS ARE NOT SHAREABLE ----------------------------------------- Forked blocks lists: [[5, 6, 7, 8], [9, 10, 11, 12]] Copy source: [0, 1, 2, 3, 0, 1, 2, 3] ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ Copy destination: [5, 6, 7, 8, 9, 10, 11, 12] → 8 blocks are newly allocated and copied ----------------------------------------- IF BLOCKS ARE SHAREABLE --------------------------------------------- Forked blocks lists: [[0, 1, 2, 5], [0, 1, 2, 6]] Copy source: [ 3, 3] (block 3 is not complete so it's copied, not referenced) ↓ ↓ Copy destination: [ 5, 6] → only 2 blocks are newly allocated and copied rNr) r:r'rArrrr6rMextend)rrNrOrHrforked_by_referencerLrEblocks_to_copyrCforked_blocks_listscopy_srccopy_dstrrKs r fork_blockszBlockManager.fork_blocks~sJ0! ) ))(3$$'..uxx8OOy0O  ]+c2E.FF Q 49)4DEq'*Er2M M!0C'+ y! 1A"&"6"6~yR[]e"f "*R|#  & &':=P'P Q OOM>/*:; < OO/ 0  1#Hh66!Fs DrLc|j|}|xjdz c_|jdk(r|jj|yy)z4Increases the reference count of a given (block_id).r N)r:rr8r@rrLrEs rincrease_ref_countzBlockManager.increase_ref_countsE!!(+ 1 ??a   $ $X . r!c|j|}|xjdzc_|jdk(rS|jrd|j|<y|jj ||j j |yy)zDecreases the reference count of a given (block_id). If the reference count reaches 0, the block is no longer in use, and becomes initialized (if it was complete) or uninitialized (if it was incomplete).r rN)r:rr'r8r@r7rArYs rdecrease_ref_countzBlockManager.decrease_ref_countst!!(+ 1 ??a   15$$X.!!%%h/&&--h7 r!blockscp|r|D]}|j|y|jj|y)zMarks a list of (blocks) as free. If the blocks were not (shareable), we simply add them to the uninitialized blocks queue. Otherwise, their new state depends on whether they are complete.N)r\r7rQ)rr]rHrLs r free_blockszBlockManager.free_blockss: " 2''1 2  " " ) )& 1r!c|jj|}|jdkDrtd|d|j|jj |y)zYMarks a block as uninitialized. Raises an error if the block has more than one reference.r Block z0 has more than one reference: block.ref_count = N)r:r@r RuntimeErrorr7rArYs runinitialize_unshared_blockz(BlockManager.uninitialize_unshared_blocks[!!%%h/ ??Q z1bPUP_P_Ocde e %%h/r!num_complete_blocksallocated_blocks prompt_idscd}g}t|D]A\}}|j|}|jr|j}n|j ||fCd} |r|j \}}| | |_d} |dk(ry|dz}|||jz|dz|jz} |j|| |j|_|jj|j} | | |jk(r$tjd|jdntjd| d|j| ||<| } |j!| |j#|jn_tjd|jd |jd |j|j|j|j<|j}|ryy) zAmong the list of (allocated_blocks), mark (num_complete_blocks) incomplete blocks as now complete. The list of (prompt_ids) is used to compute the hash of the new block.Nrr raz& was marked as complete more than oncezFound existing block z for block zAdding new block z (group z ) with hash )rr:r'rrAr@rr4 compute_hashrr9getrr warningdebugrZrc) rrdrerf parent_hashincomplete_blocksirLrE new_parent_idtokensexisting_block_ids r!mark_shareable_blocks_as_completez.BlockManager.mark_shareable_blocks_as_completes  57,-=> 1KAx%%h/E  #jj   $ $aZ 0  1 (,,.HAu("/ $ #a' 1 $ DOO 3q1u6OPF**;OEJ $ 0 0 4 4UZZ @  ,$0NNVEHH:5[#\]LL#89J8K;W\W_W_V`!ab*;$Q'$5M++,=>44UXX> 0 (5>>BRR^_d_i_i^jkl/4xx  , **KI r!rlrpc0t|t||fS)zComputes the hash of a block identified by the (tokens) it contains, its (parent_hash) and the layer (group_id) it belong to. If the block has no parent, the parent hash is None.)rtuple)rrlrprs rrhzBlockManager.compute_hashs[%-:;;r!)r(r)r*r+r,r r.r<r/rFlistrMrtrWrZr\r_rcrrrhr0r!rr2r26s&131C1D1GGGst&##,/$J#CG#SV# cT #&67!#Y673667CG67SV67 tDI%tCy$s); <67p/3/4/ 83 84 82$s)2220C0D05%#&5%:>s)5%QUVYQZ5% 5%n<d 1J1J'./--[[ 2K2 .Hh  %ABSATUV V366JL^2_ B . "d&6&66 #JK^J_!`aa4AD  0 1 B!!r!)r(r)r*r+r,__annotations__dictr-rur/rr2r~r_rrrtrWr0r!rrwrws.y Kc49n%%---\-^adh^h--c,4d3dSdPSdX\]`XaddeCeceQTeY]^aYbee"!$"<@I"Vb" tCy$s)# $"r!rwc eZdZdZdedededdfdZded ed ededzfd Z d ed ed ede efdZ d ed ed ede efdZ y)FullAttentionCacheAllocatorz3Cache manager for a group of full attention layers.rr4allow_block_sharingr Nc<||_||_||_i|_y)zInitializes the cache manager for a group of full attention layers. Args: - index: the index of the associated layer group - block_size: the size of the blocks in the cache N)rxrzr4ry)rrr4rs rr z$FullAttentionCacheAllocator.__init__Ys"  "5$r!r=r{r|c|jj|g}|r|d}n||j|<d}|j|||j|j}|y|j ||S)zAllocate (n_blocks) for a given (request_id) using the (block_manager). Returns the number of blocks allocated if successful and None otherwise. For group of full attention layers, we always allocate the number of requested blocks.rN)ryrirMrzrxrQ)rr=r{r|ryrGres rr~z+FullAttentionCacheAllocator.allocate_blocksds &&**:r: 'OM+6D  Z ( M(88=RVRiRikokvkvw  #+,r!rrc|jj|}|td|||z}||jz}||jz}g}t |D]<} || |jz} |j t | | |jz>|r0|||jz} |j t | | |z|S)zReturns the physical indices of where to read request_id's cache. For a group of full attention layers, we first write the new cache to the cache tensor and then read the entire cache from the beginning to the end.rryrirr4r6rQ) rr{rrry total_lengthnum_full_blocks remainderphysical_indicesbstarts rrz,FullAttentionCacheAllocator.get_read_indicesvs&&**:6  @ MN N"\1 &$//9 4??2 ' KANT__4E  # #E%1H$I J K 04??BE  # #E%1B$C Dr!c|jj|}|td|||jz}||jz}||z}|dz |jz}g} t ||dzD]d} || |jz} | |k(r|nd} | |k(r|dz |jzdzn |j} | j t | | z| | zf| S)zReturns the physical indices for writing to the cache. For a group of full attention layers, we write the new cache as a continuation of the existing cache for the same request.rr rr)rr{rrry start_block start_offsetend_pos end_blockrr block_start local_start local_ends rrz-FullAttentionCacheAllocator.get_write_indicess&&**:6  @ MN N!T__4 "T__4  ,q[T__4 {IM2 _A%a.4??:K*+{*:,K?@I~17!;SWSbSbI  # #E+ *C[S\E\$] ^  _  r!) r(r)r*r+r,r/r r-r2r~rurrr0r!rrrVs= c s  RV \^adh^h$ 3 S PS X\]`Xa * C c QT Y]^aYb r!rc eZdZdZdedededdfdZded ed ededzfd Zd ed ed ede efdZ d ed ed ede efdZ y)SlidingAttentionCacheAllocatorz2Cache manager for sliding window attention layers.rr4sliding_windowr Nc||_d|_||_||_t |j|jz |_i|_y)aInitializes the cache manager for a group of sliding window attention layers. Args: - index: the index of the associated layer group - block_size: the size of the blocks in the cache - sliding_window: the size of the sliding window FN)rxrzr4rr_max_blocks_per_requestry)rrr4rs rr z'SlidingAttentionCacheAllocator.__init__sF "'$,'+D,?,?$//,Q'R$r!r=r{r|c^||jvrg|j|<t|j|}||jk(ryt||z|j}||z }|j |d|j |j }|y|j|j||S)aAllocate (n_blocks) for a given (request_id) using the (block_manager). Returns the number of blocks allocated otherwise. For group of sliding window attention layers, we only allocate up to the point where we can fit an entire sliding window in the cache tensor.rN)ryrrminrMrzrxrQ)rr=r{r|already_allocatedafter_allocationactual_n_blocksres rr~z.SlidingAttentionCacheAllocator.allocate_blockss T-- -+-D  Z ( 0 0 <=  < < <08;T=Y=YZ*->>(88 T4#:#:DKK   # $++,<=r!rrc|jj|}|td|||jkrdn||jz}t ||jdz }g}t |||zD]U}||jz}||j z} ||j z} || |j z| z} |j| W|dg|zzS)aReturns the physical indices of where to read request_id's cache in the cache tensor. For a group of sliding window attention layers, we read from the cache tensor before writing on it, because the new cache can overwrite the old one. To form the cache + new key / values states, we read the at most sliding_window - 1 cache page and then manually add the new key / values states after. Hence the -1 indices which indicate where to store the new key or values indices.rrr rryrirrrr6r4rA) rr{rrry start_index cache_lengthrrn block_idx block_offsetphysical_indexs rrz/SlidingAttentionCacheAllocator.get_read_indicess&&**:6  @ MN N&)<)<? 4A $$ $AT__,It.L(3dooE TN  # #N 3  4  2$"555r!c|jj|}|td|||jz}t ||j}||z }g}t |||zD]U} | |jz} | |j z} | |j z} || |j z| z} |j| W|dkDr dg|z|z}|S)aBReturns the physical indices of where to write request_id's cache in the cache tensor. For a group of sliding window attention layers, we write the new cache in rolling-buffer kind of way: if we reach the end of the allocated physical cache, we start writing from the beginning of the physical cache again.rrrr) rr{rrryrrpadding_lengthrrnrrrs rrz0SlidingAttentionCacheAllocator.get_write_indicess &&**:6  @ MN N!D$7$77 <)<)<= % 4{K,$>? 4A $$ $AT__,It.L(3dooE TN  # #N 3  4 A  "tn47GG r!) r(r)r*r+r,r r-r2r~rurrr0r!rrrs< c s C D \^adh^h,636S6PS6X\]`Xa6. C c QT Y]^aYb r!rN)abcrr collectionsrcollections.abcrmathrtypingrrequestsr r rurtr,rrr2rwrrr0r!rrs$$ CL$q'huS!V}&=%%*_<_