qiWddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl Z ddl m Z ddlmZdd lmZd d lmZd d lmZmZd d lmZd dlmZd dlmZmZmZddlmZddl m!Z!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-m.Z. dZ/dZ0dZ1Gdde jdZ3eGddZ4eGddZ5Gdd Z6y)!N)abstractmethod) Generator)contextmanager)ceil) perf_counter)nn)tqdm)logging_redirect_tqdm)PretrainedConfig) CompileConfigGenerationConfig)LogitsProcessorList)logging)ContinuousBatchProcessorMetrics attach_tracertraced)PagedAttentionCache)ContinuousBatchingAsyncIOsContinuousBatchingIOs)GenerationOutput RequestState RequestStatuslogger)SCHEDULER_MAPPING FIFOScheduler Scheduler)attn_mask_is_neededpad_to_interval@i@ ceZdZUeed<ej ed<ejed<ede ddfdZ ede de fd Z y) ProtoPretrainedModelconfigdtypedeviceattn_implementationreturnNcyN)selfr(s l/opt/pipecat/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/continuous_api.pyset_attn_implementationz,ProtoPretrainedModel.set_attn_implementationF generation_configcyr+r,)r-r2s r._get_logits_processorz*ProtoPretrainedModel._get_logits_processorJr0r1)__name__ __module__ __qualname__r __annotations__torchr&r'rstrr/rrr4r,r1r.r$r$As`  ;; LL 3 4   7G L_  r1r$c eZdZUeezed<dededede jde jde jde jd e jd ed ed ed ededededdf dZdefdZed/dZedededdfdZd/dZedefdZededdfdZed/dZedefdZedZ ededdfdZ!ee jDd e#jHd!e%d"eddfd#Z&ed e#jHd$e'd!e%d"eddf d%Z(ed&'d e#jHd$e'de jRfd(Z*ed)'d$e'd*e jRd!e%de jRfd+Z+ed,'d-e jRd$e'd"eddfd.Z,y)0ContinuousBatchProcessorinputs_and_outputscacher%r2 input_queue output_queue stop_event model_device model_dtype schedulermanual_evictionuse_cuda_graphq_padding_interval_sizekv_padding_interval_sizemax_cached_graphsuse_async_batchingr)Nc||_||_||_||_||_||_||_||_| |_| |_ t|dddn |j|_ | |_ | |_ ||_| |_t|dd|_d|_| xs'|j duxr|j j$ |_|j(|_t+|j(|_||_|j.r#t1|dz }t3||||||_yt7||||||_y)aInitialize the continuous batch processor. Args: cache: A [`PagedAttentionCache`] object config: The model configuration generation_config: The generation configuration input_queue: Queue for incoming requests output_queue: Queue for outgoing results stop_event: Event to signal processing should stop model_device: Device for model inputs/outputs model_dtype: Data type for model inputs/outputs scheduler: The [`Scheduler`] to use manual_eviction: Whether to manually evict blocks from the cache use_cuda_graph: Whether to use cuda graphs or not during CB. Check the docstring at the top of the file for more details. q_padding_interval_size: Padding granularity for queries in tokens. kv_padding_interval_size: Padding granularity for KV cache in tokens. max_cached_graphs: Maximum number of CUDA graphs to cache. Uses LRU eviction when full. sliding_windowNrcompile_configF)r>r%r2r?r@rArBrCrDrEgetattrrLrGrHrIrFrM'_forward_process_and_sample_is_compileddynamic _pad_inputsmax_batch_tokensrmetricsrJrrr=r)r-r>r%r2r?r@rArBrCrDrErFrGrHrIrJs r.__init__z!ContinuousBatchProcessor.__init__TsVJ  !2&($(&".$+63CT#J#RaX^XmXm'>$(@%!2,4;>QRVR`R`RqRqQrrs u%%668AAC D r1c|jjsU |jj}|8|jj ||jjsTyy#t j $rYyt$rQ}tjd|dtjd}||j||Yd}~d}~wwxYw)z?Pull new requests from the input queue and add to waiting list.NzError processing new request: Texc_infostate) r?empty get_nowaitrDadd_waiting_requestqueueEmpty Exceptionrerrorlocalsget_handle_request_error)r-r`es r._get_new_requestsz*ContinuousBatchProcessor._get_new_requestss""((* 9((335=2259 ""((*;;  9 =aSADQ&,hll7&;$..q%8  9s#A1A11CCACCrgr`ctj|_t||_t |j tr+|jj|j |_ ng|_ |jj|j|j |jj|jy)z(Handle general request processing error.N)rFAILEDstatusr:rg isinstance request_idrD!get_active_request_static_outputsgenerated_tokensrTrecord_request_completion created_timer@putto_generation_output)r-rgr`s r.rjz.ContinuousBatchProcessor._handle_request_errors%++ %j  e&& ,%)^^%U%UV[VfVf%gE "%'E " ..u/A/A5CSCST e88:;r1c X|jjr(|jjj\}}n9t t |jjj \}}tjd|dt|jdt|jd|j}tj|_|jj!|d|jj#|d|j_y)aSoft resets one active request by removing it from active requests and re-adding it to the waiting queue. The generated tokens are kept as part of the new request's initial prompt. When `block_new_requests` is False, the oldest request is offloaded; when True, the newest request is offloaded. This method also sets `block_new_requests` to True to prevent infinite loops of offloading and re-scheduling requests. zSoft resetting request z with z initial tokens and z generated tokensTevict_from_cacheN)rDblock_new_requestsrXpopitemnextiteritemsrinfoleninitial_tokensrs!create_equivalent_initial_requestrFINISHED_statusfinish_requestrc)r-rqr` new_states r.soft_reset_one_requestz/ContinuousBatchProcessor.soft_reset_one_requests >> , , $ > > F F H J $T$..*H*H*N*N*P%Q R J %j\E ;;= %..  %%j4%H **95-1)r1ct|j|jj|jjsy|jj t |jjt |jj|jj|j|jj}|>t |jjdkDr|jytd|sy|jj||j j#||jj%|jt'j(t*j,r|j j/dd\}}t'j0dt |dt |jjdt |jjd |d |d |jj3 y ) zPrepare tensors and metadata for the next model forward pass. Returns True if there are requests to process, False otherwise.FNrz=No requests can be scheduled and no request can be offloaded.rNz Scheduled: z , Waiting: z , Active: z . cum Q: z . cum KV: z, free blocks: T)rlrDclear_cancelled_requestshas_pending_requestsrTrecord_queue_metricsrrXrYschedule_batchrSr> num_pagesr RuntimeErrorrecord_batch_metricsr=prepare_batch_tensorsrecord_kv_cache_memory_metricsr isEnabledForrDEBUGget_actual_lengthsdebugget_num_free_blocks)r-requests_in_batchactual_query_lengthactual_key_lengths r.prepare_next_batchz+ContinuousBatchProcessor.prepare_next_batchs   //1~~224 ))#dnn.L.L*MsSWSaSaSrSrOst!NN99$:O:OQUQ[Q[QeQef  $4>>112Q6++-"#bcc  ))*;< 556GH 33DJJ?   w}} -595L5L5_5_5abdcd5e 2 !2 LLc"345[T^^EdEdAe@fgt~~==>?yI\H]^,-_TZZ=[=[=]<^`  r1c|js|jtjk(r*|jj |j yy)zCSend output to the queue based on streaming mode and request state.N) streamingrorrr@rvrw)r-r`s r._maybe_send_outputz+ContinuousBatchProcessor._maybe_send_outputs? ??ellm.D.DD    ! !%"<"<"> ?Er1cT|jj\}}d}|D]}|j}|jtj k(r7|j r|jr|dz }Ktd|jd|jr!|jdk(rE|jj|j|jtj|_||}|dz }|j|}|j j#||j$|rs|jj'|j|j|j(j+|j|j, d|j(_|j1||jtj2k(s|j j#||j$gg} }|j(j4r|j(j4j7} | j8} d| _t;| D cgc]} | jd| } } | D]*}| j=||j(j>|<,|j jA| j| \}}|jC|| jC||j(j4r|r|j jE|| yycc} w) z0Update request states based on generated tokens.rrz!Tried to update FINISHED request z in sync mode.ryFz__child#N)#r=prepare_batch_updater`rorrrJ has_new_tokenrrq generated_lenrTrecord_ttft_metricruDECODINGupdate_and_check_completionr>!mark_shareable_blocks_as_completecomplete_blocksrtrDrrEr{r PREFILLING_requests_to_forkpop num_childrenrangeforkrX fork_requestextend copy_cache)r-r new_tokenscurrent_logits_index future_stater`token is_finished copy_sourcecopy_destination state_to_forkrinew_request_idsnew_request_idcopy_srccopy_dsts r. update_batchz%ContinuousBatchProcessor.update_batchs)-(?(?(T(T(V%: - bL &&E||}555**#11,1,"%FuGWGWFXXf#ghh))&&(A-LL33E4F4FHXHXY#0#9#9EL"#78$)$$??F  < d"&!8!89Q9QSb!c Hh   x (  # #H -nn..  JJ ! !+/? @ fsL%c6|jjS)z2Check if there are any active or waiting requests.)rDrr\s r.rz-ContinuousBatchProcessor.has_pending_requestsQs~~2244r1c|jjd}|D]M}|j||j|jj |jj Oy)z&Handle errors during batch processing.rN)r=rrjr`rDrrq)r-rgfailed_future_statesrs r.handle_batch_errorz+ContinuousBatchProcessor.handle_batch_errorVsc $66KKMaP0 IL  & &ul.@.@ A NN ) ),*<*<*G*G H Ir1ct|jjj}|D]9}|j |||jj |j ;t|jjjD]9}|jjj|}|j ||;|jjjy)zFail all active requests with the given error. Args: error: The error to report in the failure message N) listrDrXvaluesrjrrqrYkeysrwaiting_requests_orderclear)r-rgrequestsr`req_ids r.fail_all_requestsz*ContinuousBatchProcessor.fail_all_requests^s66==?@ >::??AB 5FNN3377?E  & &ue 4 5 --335r1modellogit_processor do_samplecp|j|jstj|j|jj |jj |jj|jj|jj|_d|_|jrx|jj\}}}}}t||j|j}t!|}t||j"|j$j&} nd\}} |jj)|| } |jj*} |j,s>tj.j1| 5|j || ||dddn:|jj2j5|| } | 9tj.j1| 5| j7dddnt9j:d|| ftj.j1| 5|j || ||dddtj.j=} tj.j?| | 5|j || ||ddd|jj2jA|| | |jjCy#1swY$xYw#1swY0xYw#1swYxYw#1swYoxYw)z!Perform a single generation step.N) fullgraphmoderQbackendoptionsT)rrz8Creating graph for (padded_q, padded_read_index_size) = )stream)"rMrPr9compile_forward_process_and_samplerrrQrrrRr=rr rGrSmaxrHr>rrZcompute_streamrFcudargraphs get_graphreplayrr CUDAGraphgraph set_graphretrieve_device_outputs) r-rrrr_actual_read_sizespadded_qmax_read_index_sizepadded_read_index_size batch_datarrs r._generation_stepz)ContinuousBatchProcessor._generation_stepss    *43_3_/4}}00--77((--++33++33++33 0D ,<@D 8   >B>U>U>h>h>j ; A'8!&':D2 `00 OU^_ ` ` ++22<!KKM#//+@D"& !&^^-8I7PE33Vg!2#*+<>SUZ#[  !2KF48JJ4T4TUf4g&'8:PRVWhD # KKd$JaJaIefff (?'B #H_ $)A1(D $Jb %7H!6K!2Qb  # #%&PQ Q $r1rFr$rMcptjjs8|r5tjdtjjdy||S|ry||tj j j|j|j}|jdd}|r#tjd|jd| St|jj }tjd|d |jjjd |S) aReturns whether or not to use cuda graphs for continuous batching, depending on the following criteria: - (use_cuda_graph) which is the user choice - (user_specified_param): a boolean indicating if the user specified a parameter related to cuda graphs If none of the above criteria are met, we use a default heuristic based on the attention implementation: we turn on cuda graphs if and only if no attention mask is needed. z7use_cuda_graph is True but torch.cuda.is_available() = z: turning off cuda graphs.FTztriton.cudagraphsz%Compile config compile_config.mode = z uses cudagraphs, which usually does not work well with continuous batching. We recommend using mode 'default' or 'max-autotune-no-cudagraphs' instead.zINo behavior specified for use_cuda_graph, defaulting to use_cuda_graph = z2 because self.model.config._attn_implementation = zc. If you want to save memory, turn off cuda graphs, but they tend to improve performances by a lot.)r9r is_availablerwarning _inductorlist_mode_optionsrirrrrr%r%)r-rFr$rMrcompile_uses_cudagraphss r.r.z1ContinuousBatchingManager._decide_use_cuda_graphsEs$zz&&(!YUZZ=T=T=VQ>QSaSiSijG&-kk2Eu&M #&vv/. .11B1BCCX~FYZ9zz  559:: : r1c|j0|jjrtjdyt j |j |_|jjy)z'Start the background generation thread.Nz"Manager thread is already running.)target)r*is_aliverr5rThread_run_generation_loopstartr\s r.r>zContinuousBatchingManager.startqs\  " " .43J3J3S3S3U NN? @ "+"2"2$:S:S"T %%'r1cV|jduxr|jjS)z5Check if the background generation thread is running.N)r*r;r\s r. is_runningz$ContinuousBatchingManager.is_running{s'&&d2Yt7N7N7W7W7YYr1blocktimeoutc|jtjdnV|jjjr6tj d|jjj |jtjdyt}|jjs/|jjtj d|r|j||d|_y)zSignal the background thread to stop. Args: block: Whether to wait for the thread to stop timeout: Maximum time to wait for the thread to stop Nz% Batch processor was not initialized.z- Prefix sharing was on. Total prefix length: zManager not started.z'Stopping continuous batching manager...) r)rr5r>use_prefix_sharingr_total_prefix_lengthr*rrAis_setsetjoin)r-rArBstop_trigger_times r.stopzContinuousBatchingManager.stops    ' NNC D##))<< DTEYEYE_E_EtEtDuv  " " * NN1 2 (N%%' OO   ! KKA B  II' 1#r1rIc|j|jj||jjrtjd|dyt }tj d||z ddd|_yy)zWait for the background thread to finish. Args: timeout: Maximum time to wait for the thread to stop NrBz3Generation thread did not exit after join timeout (z).z*Continuous Batching Manager stopped after z.2fzs.)r*rHr;rr5rr)r-rIrBends r.rHzContinuousBatchingManager.joins  " " .  # # ( ( ( 9&&//1!TU\T]]_`a"n HO`I`adHeeghi*.' /r1rrqmax_new_tokensrrecord_timestampsc ||9|j5d|j}|xjdz c_ddd||jjn|}t |t ||j dz |||jj|}|jj|dd|S#1swYxYw)a/Add a new generation request to the queue. Args: input_ids: Input token IDs to use as prompt request_id: Optional custom request ID (auto-generated if None) **kwargs: Additional generation parameters Returns: str: The request ID Nreq_r)rqrrrOrN eos_token_idrT rArB) r-r+r2rNrrr#rRr?rv)r-rrqrNrrOr`s r. add_requestz%ContinuousBatchingManager.add_requests$  ## +#D$9$9#:; %%*% +CQBX//>>^l! ?22Q6/)//<<  U$;' + +s %B22B;inputsc|j5t|j|jt|zDcgc]}d| }}|xjt|z c_dddt t |}|j rt|dd}|D]\}} |j| ||||ycc}w#1swY^xYw)NrQc |dS)Nrr,)xs r.z8ContinuousBatchingManager.add_requests..s !A$r1T)keyreverse) r-rr+rrzipr(sortedrU) r-rVrNrrOr request_idsids_and_inputsrqrs r. add_requestsz&ContinuousBatchingManager.add_requestss   1/4T5J5JDLaLadghndoLo/pq!T!:qKq  ! !S[ 0 ! 1c+v67  # ##NPTUN%3 b !J   Y NIO` a br 1 1s/C C CCCch|j&|jjj|yy)zkCancel a request by its ID. Args: request_id: The ID of the request to cancel N)r)rDset_request_cancellationr-rqs r.cancel_requestz(ContinuousBatchingManager.cancel_requests/    +  * * C CJ O ,r1c|j|jjry |jjd|}|+|j|k7r|jj |y|S#t j$rYywxYw)zRetrieve one result from the output queue. Args: timeout: Maximum time to wait for a result Returns: Optional[GenerationOutput]: The result data or None if timeout NTrT)r*r@rarirqrvrdre)r-rqrBresults r. get_resultz$ContinuousBatchingManager.get_results  " " *t/@/@/F/F/H &&**w*GF%&*;*;z*I!!%%f-M{{  sA A53A55B  B c#K|j\|jjrA|jd}|||j|jjr?yyyyw)z.Iterate over results as they become available.N皙?rL)r*r;rh)r-rgs r.__iter__z"ContinuousBatchingManager.__iter__sf%%1d6M6M6V6V6X__S_1F! %%1d6M6M6V6V6X16X1s A%A,(A,c#PKd}|j|jjry|sv|j|d}|||j%|jjj |}|j|jjr|ssyyyyyyw)zMIterate over results matching a specific request id as they become available.FNrj)rqrB)r*r;rhr)rDrequest_is_cancelled)r-rqrequest_cancelledrgs r.request_id_iterz)ContinuousBatchingManager.request_id_iters!%%1d6M6M6V6V6Xar__ C_HF! ##/$($8$8$B$B$W$WXb$c! %%1d6M6M6V6V6Xar6X1ar6X1s BB& B&c|j td|jj|j|j|j y)z=Perform a single generation step. This is mostly cuda graphedNzNTried to perform a generation step before the batch processor was initialized.)r)rrrrrr\s r.rz*ContinuousBatchingManager._generation_stepsE    'op p --djj$:N:NPTP^P^_r1cd} t}t|jj|j|jj |jj t|jdd|j}|j|_ tjdt|z dd}t|jdrLtj|jj d}|&tj"d|dt$}nt$}t}t'||jj|j|j(|j*|j,|jj |jj |||j.|j.|j0|j2|j4|j6|j8 }||_d |_tjd t|z d|j:j8r@|j?s tAd |jC|xj<d z c_|j,jEr|jGrR|jI||xj<d z c_|j,jEsA|jGrRtK|jLtNr8d |jLjPz |jL_(|jStjZdy#tT$r6}tjVd|d|jY||Yd}~Pd}~wwxYw#tjZdwxYw)z6Main processing loop running in the background thread.N_tp_size)tp_sizerzPagedAttentionCache created in z secondsrDz Scheduler 'z ' not found. Defaulting to FIFO.)r>r%r2r?r@rArBrCrDrErFrGrHrIrJrzbatch_processor created in z$Failed to bootstrap the first batch.rzError in generation loop: Tr^zGeneration loop finished.).rrrr%r2r'r&rOr'rDr(rrrrrirDr5rr<r?r@rArErFrGrHrIrJr) current_batchrrrrFr_inner_generation_looprpr=r current_pairrrfrg_handle_critical_errorr)r-r)t0paged_attention_cacherDt1rks r.r=z.ContinuousBatchingManager._run_generation_loops;?B 5B$7 !!&& !!    J=$($=$= % !(='O'OD $ LL:<>B;N:OxX YIt--{;-11$2H2H2R2RTXY $NN[ ;[#\] -I* B6+zz(("&"8"8 ,,!..??!ZZ.. JJ,,#$94;O;OP $ 4 4#22(,(D(D)-)F)F"&"8"8#'#:#:O"$3D !"D  LL6|~7J6K8T U##66&99;&'MNN%%'""a'"--/O4X4X4Z++O<""a'"--/O4X4X4Z /<<>XYBCoFhFhFuFuBu22?,,. KK3 4  < LL5aS9D I  ' '? ; ; < KK3 4s7K&M$+M$r@floatrJrHrr:rUrarerrhrkrrorr=r<rurfrwrr,r1r.rrs!&'(()!"$(*.SR#SR,SR SR  SR "% SR#&SRSR"SR!4KSR SRj*t *"*&, *  *X ( (ZDZ $$$ $$: /e /edl /d /$"&%)"' &9&$J&d &  &  & &V&*"' bT#Ybd b b  b  b&PPPS4ZYilpYp*d#d)rrrJ) r-r2rErrGrHrrArBrJrImanagers r.#continuous_batching_context_managerz3ContinuousMixin.continuous_batching_context_managers///+)$;%=/ 310     7M LL3  LLugL 6 LL3  LLugL 6s*BA)B*BBc t|drt|dr t|ds td||n |j} | td| jt j dd| _t|| ||||||| S) aInitialize a manager for continuous batching inference. Args: generation_config: An optional generation configuration, which may contain a CompileConfig object manual_eviction: Whether to manually evict requests from the cache max_queue_size: Maximum size of the input request queue q_padding_interval_size: Padding granularity for queries in tokens. 0 uses default. kv_padding_interval_size: Padding granularity for KV cache in tokens. 0 uses default. allow_block_sharing: A flag to allow block sharing if the model has some full attention layers use_async_batching: Whether to use async API or not. If None, will be automatically detected. max_cached_graphs: Maximum number of cached CUDA graphs. 0 uses default. Returns: `ContinuousBatchingManager`: The manager instance to add requests and retrieve results. r%r'r&z;Model must have 'config', 'device', and 'dtype' attributes.z8A GenerationConfig must be provided or set in the model.zE`eos_token_id` not set in GenerationConfig. Setting to -1 (disabled).r) rr2rErrGrHrrJrI)rAttributeErrorr2 ValueErrorrRrr5r) r-r2rErrGrHrrJrI gen_configs r.rz(ContinuousMixin.init_continuous_batchings2tX&gdH.EWUY[bMc !^_ _*;*G&TMcMc  WX X  " " * NNb c&(J #)(+)$;%= 31/  r1rVrO progress_barc \|siStjtjkrtjdd}i} | |j n|} t || j | jndz} |j|||| |dd|}ttg}t| | d| d d }|5}|5|5} |j|| jd | d}|| kr|jd}|r8|j}|jrM|| |<|dz }|j!dn1|j#s!tj$dt'dn|| krdddddddddi}t+t |D]:}| jd|}| ||d|<"tj$d|d<|S#t($r$}tj$d|dYd}~d}~wwxYw#1swYxYw#1swYxYw#1swYxYw)a4Generate sequences for a batch of prompts using continuous batching. Args: inputs: List of input token sequences (prompts) generation_config: Optional generation configuration q_padding_interval_size: Padding granularity for queries in tokens. 0 uses default. kv_padding_interval_size: Padding granularity for KV cache in tokens. 0 uses default. allow_block_sharing: A flag to allow block sharing if the model has some full attention layers record_timestamps: If set to true, the requests will have a timestamp for each token generated progress_bar: If set to true, a progress bar will be displayed use_async_batching: Whether to use async double buffering or not. If None, will be automatically detected. max_cached_graphs: Maximum number of cached CUDA graphs. 0 uses default. **kwargs: Additional generation parameters Returns: `dict[str, GenerationOutput]`: a dictionary of request ids to GenerationOutput objects z=Progress bar is disabled when logger level is less than DEBUGFNrT)r2rGrHrIrrArBrJzSolving z requestsrequest)totaldisabledescunitrN)rVrNrOrrLz*Generation thread terminated unexpectedly.zCReturning results of generate_batch despite unexpected termination.zError during batch generation: r^rQz Request req_z not found in results.)rgetEffectiveLevelrrr5r2rr#rr r rarirhrqrupdater@rgprintrfr)r-rVr2rGrHrrOrrJrIkwargsresultsgen_cfg num_requests manager_cm logging_cmpbar_cmrpbarfinished_countrgrrkreordered_resultsrs r.generate_batchzContinuousMixin.generate_batchsq@I  # # % 6 NNZ [ L,=,E$((K\6{gFbFbFng&B&Btuv ==/$;%=/ 31>  +F84 %%L>3   S7J S S4 S$$!&**=M2Nbs%"#$|3$///:F!'!2!2!--/.4GFO*a/N KKN&113"LL)UV!"gh!%|3  S S S0s6{# GA[[4s,F!06!D*- |A3.DEF  G!  S >qcBTRR S+ S S S S S Ssa8H";H>H B,G-H5H" H #H =H H H  H HH H""H+) NFrrrTTNNr)NFrrrTNr)NrrTFTNr)r5r6r7rrr8rrrrrrrrrr9inference_moderrr:rrr,r1r.rrs I''6: %'(()$( $*.!"7+d277 7 "% 7 #& 7"777!4K77 , -77F6: %'(()$(*.!"/ +d2/ /  / "% / #& / "/ !4K/ /  #/ d U6:'(()$("'!*.!"[!T#Y[!,d2[!"% [! #& [! " [! [![!!4K[![! c## $[! [!r1r)7rdrabcrcollections.abcr contextlibrmathrtimerr9rr tqdm.contrib.loggingr configuration_utilsr generation.configuration_utilsr rgeneration.logits_processr utils.loggingr utils.metricsrrrr>r input_outputsrrrrrrrrDrrrutilsrr r/r0r1rr$r<rrr,r1r.rs %% 63M<$SS&LKKBB7&#  299  SGSGSGn ^F^F^FB v!v!r1