IuigS)ddlZddlZddlZddlZddlZddlmZmZmZm Z m Z m Z ddl m Z mZmZmZmZmZdZ d6dedee ede ee eded ed eef d Z d7dedee ed ee eded ed eef d Z d8dedee eded ed ee f dZ d7dedee eded ed eef dZ d9dddddddddddddd dede ede e ededededededededed ed!e e e ed"e e ee ee efd#ed$ed eef"d%Z d:d&dddddddddddd'dd(ded)e e ee e efded ed*ed+ededededededed ed!e e e ed"e e ee ee efd,e e ed-ed.eegefd eef&d/Z d:d&dddddddddddd'dd(ded)e e ee e efded ed*ed+ededededededed ed!e e e ed"e e ee ee efd,e e ed-ed.eegefd eef&d0ZGd1d2Zd3Z d4Z!d5Z"y);N) AsyncIterableCallableIterableListOptionalUnion)GenerationResultGenerationStepResult Generator ScoringResultTranslationResult Translatorc*ttdtttdtttdttt dt tt dttt dttt dty)z4Registers additional attributes to compiled modules.translate_iterablescore_iterablegenerate_tokensgenerate_iterableasync_generate_tokensN) setattrrtranslator_translate_iterabletranslator_score_iterabletranslator_generate_tokensr generator_generate_iterablegenerator_score_iterablegenerator_generate_tokensgenerator_async_generate_tokensH/opt/pipecat/venv/lib/python3.12/site-packages/ctranslate2/extensions.pyregister_extensionsr sd J,.KL J(*CD J)+EF I*,GH I')AB I(*CD I.0OPr translatorsource target_prefixmax_batch_size batch_typereturnc+~K|g}||j|t|j|||fi|Ed{y7w)a6Translates an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Translator.translate_batch` to efficiently translate an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel translations (if the translator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: source: An iterable of tokenized source examples. target_prefix: An optional iterable of tokenized target prefixes. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any translation options accepted by :meth:`ctranslate2.Translator.translate_batch`. Returns: A generator iterator over :class:`ctranslate2.TranslationResult` instances. Example: This method can be used to efficiently translate text files: .. code-block:: python # Replace by your own tokenization and detokenization functions. tokenize_fn = lambda line: line.strip().split() detokenize_fn = lambda tokens: " ".join(tokens) with open("input.txt") as input_file: source = map(tokenize_fn, input_file) results = translator.translate_iterable(source, max_batch_size=64) for result in results: tokens = result.hypotheses[0] target = detokenize_fn(tokens) print(target) N)append_process_iterabletranslate_batch)r!r"r#r$r%kwargs iterabless rrrsR`I ' ""   s 3=;=targetc+VKt|j||g||fi|Ed{y7w)a}Scores an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Translator.score_batch` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel scoring (if the translator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: source: An iterable of tokenized source examples. target: An iterable of tokenized target examples. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any scoring options accepted by :meth:`ctranslate2.Translator.score_batch`. Returns: A generator iterator over :class:`ctranslate2.ScoringResult` instances. Nr) score_batch)r!r"r-r$r%r+s rrr[s;<!    s )') generator start_tokensc+TKt|j|g||fi|Ed{y7w)acGenerates from an iterable of tokenized prompts. This method is built on top of :meth:`ctranslate2.Generator.generate_batch` to efficiently run generation on an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel generations (if the generator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: start_tokens: An iterable of tokenized prompts. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any generation options accepted by :meth:`ctranslate2.Generator.generate_batch`. Returns: A generator iterator over :class:`ctranslate2.GenerationResult` instances. N)r)generate_batch)r1r2r$r%r+s rrrs88!      (&(tokensc+TKt|j|g||fi|Ed{y7w)a9Scores an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Generator.score_batch` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel scoring (if the generator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: tokens: An iterable of tokenized examples. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any score options accepted by :meth:`ctranslate2.Generator.score_batch`. Returns: A generator iterator over :class:`ctranslate2.ScoringResult` instances. Nr/)r1r6r$r%r+s rrrs88!    r5Fi) max_decoding_lengthmin_decoding_length sampling_topk sampling_toppsampling_temperaturereturn_log_probrepetition_penaltyno_repeat_ngram_size disable_unksuppress_sequences end_tokenmax_input_lengthuse_vmapr:r;r<r=r>r?r@rArBrCrDrErFc #vKt|j|g||gnd| | | | | ||||||||Ed{y7w)a{Yields tokens as they are generated by the model. Arguments: source: Source tokens. target_prefix: Optional target prefix tokens. max_decoding_length: Maximum prediction length. min_decoding_length: Minimum prediction length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one of these tokens (defaults to the model EOS token). max_input_length: Truncate inputs after this many tokens (set 0 to disable). use_vmap: Use the vocabulary mapping file saved in this model Returns: A generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. N) r@rArBrCrDr:r;r<r=r> return_scoresrErF)_generate_tokensr*)r!r"r#r:r;r<r=r>r?r@rArBrCrDrErFs rrrs\\ "" (4$-1-//##1%)!s /979iT) max_length min_lengthr<r=r>r?r@rArBrCrD static_promptcache_static_promptcallbackpromptrJrKrLrMrNc#Kt|dkDrt|dtr|g}t|j|fid|d|d| d| d| d| d|d |d |d |d |d |d| d|d|ddd|Ed{y7w)aYields tokens as they are generated by the model. Arguments: prompt: Batch of start tokens. If the decoder starts from a special start token like , this token should be added to this input. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". max_length: Maximum generation length. min_length: Minimum generation length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one these tokens (defaults to the model EOS token). static_prompt: If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations. cache_static_prompt: Cache the model state after the static prompt and reuse it for future generations using the same static prompt. callback: Optional function that is called for each generated token when obj:`beam_size` is 1. If the callback function returns ``True``, the decoding will stop for this batch index. Returns: A generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. rr$r%r@rArBrCrDrJrKr<r=r>rHrLrMinclude_prompt_in_resultFrNN)len isinstancestrrIr4)r1rOr$r%rJrKr<r=r>r?r@rArBrCrDrLrMrNs rrrsp 6{Q:fQi5  &  .  2  .$$2& $!"0#$"'%&'sA0A:2A83A:cKt|dkDrt|dtr|g}t|j|fid|d|d| d| d| d| d|d |d |d |d |d |d| d|d|ddd|23d{}|7 6yw)aYields tokens asynchronously as they are generated by the model. Arguments: prompt: Batch of start tokens. If the decoder starts from a special start token like , this token should be added to this input. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". max_length: Maximum generation length. min_length: Minimum generation length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one of these tokens (defaults to the model EOS token). static_prompt: If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations. cache_static_prompt: Cache the model state after the static prompt and reuse it for future generations using the same static prompt. callback: Optional function that is called for each generated token when obj:`beam_size` is 1. If the callback function returns ``True``, the decoding will stop for this batch index. Returns: An async generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. rr$r%r@rArBrCrDrJrKr<r=r>rHrLrMrQFrNN)rRrSrTAsyncGeneratorr4)r1rOr$r%rJrKr<r=r>r?r@rArBrCrDrLrMrN step_results rrr_sp 6{Q:fQi5+  &  .  2  .$$2& $!"0#$"'%&'k*+s*A-B/B3A>4B7B>BBc$eZdZdZdZdZdZy)rVctj|_tj|_d|_||_||_||_ yN) asyncioQueuequeue threadingEventshutdown_event iterator_task process_funcargsr+)selfrbrcr+s r__init__zAsyncGenerator.__init__s<]]_ 'oo/!(  rczKt|jg|ji|jD]^}|jj |d{t jdd{|jjs^n|jj dd{y7c7H7 w)Ng-C6?) rIrbrcr+r]putr[sleepr`is_set)rdrWs rproducerzAsyncGenerator.producers+    $  -1[[  K**..- - ---' ' '""))+ jjnnT""" . ' #s<AB;B5B;.B7/B;!B;/B90B;7B;9B;cVtj|j|_|SrZ)r[ create_taskrjra)rds r __aiter__zAsyncGenerator.__aiter__s $00A rcJK|jjrt |jj d{}| |jj t|S7(#t j$r!|jj twxYwwrZ)r`riStopAsyncIterationr]getsetr[CancelledError)rditems r __anext__zAsyncGenerator.__anext__s    % % '$ $ %))D|##'')((K * %% %    # # %$ $ %s.!B#A,A*'A,)B#*A,,4B  B#N)__name__ __module__ __qualname__rerjrmrtrrrrVrVs # %rrVc/ Ktj tj|j dd d  fd}|j dd|d||i| fd}tj |d}|j j }|nt|tr| |+|jy#t$rjY,wxYww) NrNcy)NFr)rWs rz"_generate_tokens..src^|}j|jxs|SrZ)rgri)rWuser_callback_resultgenerator_closed step_results user_callbacks r _callbackz#_generate_tokens.._callbacks1,[9%&&(@,@@rTr9) asynchronous beam_sizerNc D]}|j jdy#t$r}j|Yd}~1d}~wwxYwrZ)result Exceptionrg)re async_resultsr~s r_catch_exceptionz*_generate_tokens.._catch_exceptionsS '       Q   s- AA  A)r-daemon) r]r\r^r_rpupdateThreadstartrSr GeneratorExitrqjoin) rbrcr+rrthreadrWrr}r~rs @@@@rrIrIs;;=L (JJz40M1 A  MM ! !$1&1M  %5d CF LLN "&&(    k9 -      KKM    "  s*B7C3>CC3C0-C3/C00C3c+"K|dkr tdt|dk(r|d}ntj|}|j ||dd|dkDr|dzn|}t j }t|||D]i}|j||i||s|djs2|jj|sV|djr8k|r%|jj|r$yyw)Nr9zmax_batch_size must be >= 1rT)r$r%r) ValueErrorrR itertools zip_longestr collectionsdeque_batch_iteratorextenddonepopleftr) rbr,r$r%r+iterableread_batch_sizer]streamss rr)r)s677 9~Q<(()4 MM,$  .z"_batch_iterator..Fs01B0s z+Input iterables do not have the same length)rStuplerRmaxrzipr()r batch_sizer%rrJexamplebatchelements rrr-sGJ"'5)jG  #3wqz?j8  8 #ZWQZ9JC Oa/:= J  _ 4zAB B ?000G!'73 "NE73wrs KKQ48 :: T#Y :HT#Y/0: :  : :B $$ T#Y $ T#Y $ $  $m$T ""49%"" "  "P "" T#Y "" " m "P*.? # "#! ! !48<@ #?? I?DI&?  ?  ??? ?????!d3i1?c49d3i789? !?"#?$"#%?J N "#! ! !48<@)- $7;'NN $s)T$s)_, -NN N  NNNN NNNNN!d3i1N c49d3i789!N"DI&#N$%N&,-t34'N("#)Nh O "#! ! !48<@)- $7;'OO $s)T$s)_, -OO O  OOOO OOOOO!d3i1O c49d3i789!O"DI&#O$%O&,-t34'O('()Od%%%%P3l'< r