qi*dZddlZddlZddlZddlmZddlZddlm Z m Z m Z m Z m Z ddlmZddlmZddlmZd d lmZmZej.eZd d d ddZiddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1id2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSidTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrdsdtduidvdwdxdydzd{d|d}d~dddddddddddddddddddddddddiddddddddddddddddddddddddddddddddddddddddddddddddddɜZiej9Dcic]\}}|| c}}dd&d(dddddBdBdddd̜ ZddgZGdτdeZ dde e!de"dzde"de"fdՄZ#ddքZ$dd؜dلZ%ddڄZ$dۄZ& dde e!de"dzde"de"fd܄Z#de e!fd݄Z'de e!fdބZ(d߄Z)dgZ*ycc}}w)z!Tokenization classes for Whisper.N) lru_cache) AddedToken Tokenizerdecoderspre_tokenizers processors)BPE)TokenizersBackend)logging)BasicTextNormalizerEnglishTextNormalizerz vocab.jsonztokenizer.jsonz merges.txtznormalizer.json) vocab_filetokenizer_file merges_filenormalizer_fileenenglishzhchinesedegermanesspanishrurussiankokoreanfrfrenchjajapanesept portuguesetrturkishplpolishcacatalannldutchararabicsvswedishititalianid indonesianhihindififinnishvi vietnamesehehebrewuk ukrainianelgreekmsmalaycsczechroromaniandadanishhu hungariantatamilno norwegianththaiururduhrcroatianbg bulgarianlt lithuanianlalatinmimaoriml malayalamcywelshskslovaktetelugufapersianlvlatvianbnbengalisrserbianaz azerbaijanisl slovenianknkannadaetestonianmk macedonianbrbretoneubasqueis icelandichyarmeniannenepalimn mongolianbsbosniankkkazakhsqalbanianswswahiliglgalicianmrmarathipapunjabisisinhalakmkhmersnshonayoyorubasosomaliaf afrikaansococcitankageorgianbe belarusiantgtajiksdsindhigugujaratiamamharicyiyiddishlolaouzuzbekfofaroesehtzhaitian creolepspashtotkturkmennnnynorskmtmaltesesanskrit luxembourgishmyanmartibetantagalogmalagasyassamesetatarhawaiianlingalahausabashkirjavanese sundanese cantonese)salbmybotlmgastthawlnhabajwsuyuerr) burmese valencianflemishhaitian letzeburgeschpushtopanjabi moldavianmoldovan sinhalese castilianmandarin translate transcribec&eZdZdZeZddgZeZ d1de e e e fzdzffd Z d2de ffd Z d3d Zed4d Zd5d efd ZdZ d6d ededzdedededededede ffd Zdddddededede ffdZdZed5dZd7de de dzdee fdZ d8de dzde dzd edzfd!Zedee fd"Zd7dee fd#Z d9d$ee d%ee dzd&edee ffd' Z!d:d(Z"d)Z#d;d*e fd+Z$d,ee d-e d.e fd/Z%ed0Z&xZ'S)<WhisperTokenizerag Construct a "fast" Whisper tokenizer (backed by HuggingFace's *tokenizers* library). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. normalizer_file (`str`, *optional*): Path to the normalizer_file file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The beginning of sequence token. The `decoder_start_token_id` is used to set the first token as `"<|startoftranscript|>"` when generating. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (Whisper tokenizer detect beginning of words by the preceding space). language (`str`, *optional*): The language of the transcription text. The corresponding language id token is appended to the start of the sequence for multilingual speech recognition and speech translation tasks, e.g. for Spanish the token `"<|es|>"` is appended to the start of sequence. This should be used for multilingual fine-tuning only. task (`str`, *optional*): Task identifier to append at the start of sequence (if any). This should be used for mulitlingual fine-tuning, with `"transcribe"` for speech recognition and `"translate"` for speech translation. predict_timestamps (`bool`, *optional*, defaults to `False`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. input_idsattention_maskNFvocabc .t|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}||ni|_||ng|_t t |j|jdddd|_tj||j_ tj|j_ t |8d ||||||| | d| |1t|d 5} t!j"| |_dddnd|_t'j(d |_||_| |_| |_|j3y#1swYIxYw) NFT)lstriprstrip normalizedspecial)rmergesdropoutcontinuing_subword_prefixend_of_word_suffixfuse_unk)add_prefix_space) unk_token bos_token eos_tokenrrlanguagetaskpredict_timestampsutf-8encodingz<\|(\d+\.\d+)\|>) isinstancestrr_vocab_mergesrr _tokenizerr ByteLevel pre_tokenizerrdecodersuper__init__openjsonloadenglish_spelling_normalizerrecompile timestamp_patrrrset_prefix_tokens)selfrrrrrrrrrrkwargs vocab_handle __class__s b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/whisper/tokenization_whisper.pyr zWhisperTokenizer.__init__s )S) yuX\ ] )S) yuX\ ] )S) yuX\ ]  %0eb !'!3v # kk||*,#%    )7(@(@Rb(c%"*"4"4"6  -+1    &o8 KL3799\3J0 K K04D ,ZZ(;<   "4   K Ks F  Freturncj|jddz}gg}d}d}d} t|D]\} } | |k\r|t| |z |z} | |kr7| dk\xr|| dz |k\xr || dz |k\ } | r |||zz }n | }|| z }|dd}|} | }|jd| |zdd |jg|dj| g}|D]X}t |t r|j|%|r!|jt ||| H|jd Zd j|S) z Timestamp tokens are above the special tokens' id range and are ignored by `decode()`. This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". r Nz<|z.2fz|>skip_special_tokensr) all_special_ids enumeratefloatappendrrr decodejoin)r token_idsr"time_precision segment_sizetimestamp_beginoutputscur_max_timestampprev_segments_lenpenultimate_timestampitoken timestamplast_was_single_endingdecoded_outputssrs r_decode_with_timestampsz(WhisperTokenizer._decode_with_timestampss..r2Q6$ #!), *HAu'!5?#:n"LM 00-.!V.!!a%(O;c !a%@PTc@c9*.)^l-JJ),A))-BB)")#2,(9%$-!Y1B%BC#HKLr" ""5)- *0 +A!S!&&q)&&uw~aM`~'ab&&r*  +ww''cg}dtt|vr1t|dr%t|jr|j }t j |}|jddkDr#t|jdkDr td|jddz}||k\}t j|dd|ddzddz}|jddk(r|jdkrgSt j|dddz|vr2t j|t j|dddz}t j|dd}d} d} |D]} ||| } t| dkDr| dj|z } | dj|z }| | kr-|dk\xr||dz |k\xr ||dz |k\ }|r| |z } n| | z } |} |j| } |j!| }|j#|}|j|| |z| |zz||z| |zzfd | }|S) a Compute offsets for a given tokenized input Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. segment_size (`int`, *optional*, defaults to 1500): The number of features in the input mel spectrogram. torchcpurr z)Can only process a single input at a timerNrtextr3)rtypehasattrcallabler;nparrayshapelen ValueErrorr#wheresumr&item_preprocess_token_ids_decode_filter_timestamp_ids)rr)r*r+offsetsr,timestamp_tokens consecutive last_slicer.r/ current_slice sliced_tokensstart_timestamp_positionend_timestamp_positionis_single_endingr=s r_compute_offsetsz!WhisperTokenizer._compute_offsetsHs c$y/* * 50IhW`WdWdNe! IHHY' ??1  !c)//&:Q&>HI I..r2Q6$7hh/47G7KKLQORSS   Q 1 $)9)=)=)?1)DI XX& ' *2 . 2+ E))K:J1KA1Nr1RUV1VWKXX./215 ( 'M%j?M=!A%+8+;+@+@+B_+T()6r):)?)?)AO)S&+.??'1Q(!*q.1_DuS]`aSaIbfuIu<$()\9))->>)$:!!% : := I ||M211$7 $4~EHY\jHjj2^CFWZhFhh&'J? 'Br8ch|jtdDcgc] }d||zz c}Scc}w)a Compute the timestamp token ids for a given precision and save to least-recently used (LRU) cache. Args: time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. iz<|%.2f|>)convert_tokens_to_idsrange)rr*r1s r timestamp_idszWhisperTokenizer.timestamp_idss5))X]^fXg*hSTJ!n:L,M*hii*hs/r"ct|r5|jd}|jd}|j|||}|S)a Pre-process the token ids for decoding by removing the prompt tokens ids and timestamp token ids. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Typically, obtained using the `__call__` method of the tokenizer. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens from the token ids. If `True`, the prompt token ids will be removed. <|startofprev|><|startoftranscript|>)rW _strip_prompt)rr)r"prompt_token_iddecoder_start_token_ids rrIz&WhisperTokenizer._preprocess_token_idssE "889JKO%)%?%?@W%X "**9oG]^Ir8cDtj|jd|S)Nr)rsubr)rr=s rrKz&WhisperTokenizer._filter_timestamp_idssvvd(("d33r8clean_up_tokenization_spacesoutput_offsetsr*decode_with_timestamps normalizebasic_normalizeremove_diacriticsc F|j||} t|| f||||| d| } |r|j| ||} n@t | t r| D cgc]} |j | } } n|j | } |r|j||}| |dS| Scc} w)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Will remove the previous tokens (pre-prompt) if present. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). output_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output the offsets of the tokens. This should only be set if the model predicted timestamps. If there are previous tokens (pre-prompt) to decode, they will only appear in the decoded text if they contain timestamp tokens. time_precision (`float`, *optional*, defaults to 0.02): The time ratio to convert from token to time. decode_with_timestamps (`bool`, *optional*, defaults to `False`): Whether or not to decode with timestamps included in the raw text. normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the English text normalizer to the decoded text. Only applicable when the target text is in English. Otherwise, the basic text normalizer should be applied. basic_normalize (`bool`, *optional*, defaults to `False`): Whether or not to apply the Basic text normalizer to the decoded text. Applicable to multilingual target text. remove_diacritics (`bool`, *optional*, defaults to `False`): Whether or not to remove diacritics when applying the Basic text normalizer. Removing diacritics may destroy information in the decoded text, hence it should be used with caution. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. r!)r"rbrerfrg)r*r")r*)r=rL)rIr r'r7rlistrKrU)rr)r"rbrcr*rdrerfrgr filtered_idsr=trLrs rr'zWhisperTokenizer.decodesd11  32 w~  3)E+/    "//^Qd0D $%?CD!2215DD11$7 ++In+UG W5 5 EsB)rerfrgc~t||i|}|r|j|}|S|r|j||}|S|S)Nrg)r rJrerf) rrerfrgargsrr= clean_textrs rrJzWhisperTokenizer._decodesSw// -J  --dFW-XJ Kr8c<t|j}||S)z Normalize a given string using the `EnglishTextNormalizer` class, which performs commons transformation on english text. )rr)rr= normalizers rrezWhisperTokenizer.normalizes +4+K+KL $r8c*t|}||S)z Normalize a given string using the `BasicTextNormalizer` class, which performs commons transformation on multilingual text. rm)r)r=rgrqs rrfz WhisperTokenizer.basic_normalizes );LM $r8save_directoryfilename_prefixc tjj|stj d|dytjj ||r|dzndt dz}tjj ||r|dzndt dz}tjj ||r|dzndt dz}t|dd 5}|jtj|jd d d dzdddt|dd 5}|jd|jd|jDddd|jLt|dd 5}|jtj|jd d d dzddd|||fS#1swYxYw#1swYrxYw#1swY&xYw)NzVocabulary path (z) should be a directory-rrrrwrrrTF)indent sort_keys ensure_ascii z#version: 0.2 c3DK|]}dj|dzyw) r{N)r().0 merge_pairs r z3WhisperTokenizer.save_vocabulary..5sYjchhz2T9Ys )ospathisdirloggererrorr(VOCAB_FILES_NAMESrwriterdumpsr writelinesrr)rrsrtr merge_filerfwriters rsave_vocabularyz WhisperTokenizer.save_vocabulary!sww}}^, LL,^,<>> # instantiate the tokenizer and set the prefix token to Spanish >>> tokenizer = WhisperTokenizerFast.from_pretrained("openai/whisper-tiny", language="spanish") >>> # now switch the prefix token from Spanish to French >>> tokenizer.set_prefix_tokens(language="french") ``` Args: language (`str`, *optional*, defaults to `None`): The language of the transcription text. task (`str`, *optional*, defaults to `None`): Task identifier to append at the start of sequence (if any). predict_timestamps (`bool`, *optional*, defaults to `None`): Whether to omit the `<|notimestamps|>` token at the start of the sequence. Nr}z:0z $A:0 z $A:0 $B:1 z:1)singlepairspecial_tokens) rrr prefix_tokensconvert_ids_to_tokensr eos_token_idr(rTemplateProcessingzipbackend_tokenizerpost_processor) rrrrprefix_token_idsprefixeseosrr2prefix_templates rrz"WhisperTokenizer.set_prefix_tokens?s,%-$8dmm  ,D$)) 8J8V"4\`\s\s----.>?nn(( ((h#GUugRL#GH0:0M0M%&fSE4#$KuB7l#X/01 -$Hs Cc |jd}|jd}|jd}|jd}ttj}|j|jj |_|jt vrt |j}n|jt jvr |j}nnt|jdk(}td|jd|rtt jntt jd|j1|jtvrtd |jd t|g}|j&|j|d z|jz|j"|j|jd k(r|n||js|j||S) Nr\z <|translate|>z<|transcribe|><|notimestamps|>rzUnsupported language: z. Language should be one of: .zUnsupported task: z. Task should be in: r r)rWtuple LANGUAGESkeysrlowerTO_LANGUAGE_CODEvaluesrDrErirTASK_IDSr&indexr) r bos_token_idtranslate_token_idtranscribe_token_idnotimestamps_token_idlangs language_idis_language_code bos_sequences rrzWhisperTokenizer.prefix_tokensgs112IJ !77H"889IJ $ : :;M Ninn&' == $ MM//1DM}} 00.t}}= "2"9"9";;"mm #&t}}#5#:  ,T]]O<;K-4467QUVfVkVkVmQnoopr 99 yy( #5dii[@UV^U_!`aa$~ == $    q 05;;{3K K L 99   tyyL7P 3Vh i&&    5 6r8c||j|z|jgzS|j|z|z|jgzS)z=Build model inputs from a sequence by appending eos_token_id.)rr)r token_ids_0 token_ids_1s r build_inputs_with_special_tokensz1WhisperTokenizer.build_inputs_with_special_tokenssK  %% 3t7H7H6II I!!K/+=ARAR@SSSr8rralready_has_special_tokensc|rt|||dSdgt|jz}dg}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr r)r get_special_tokens_maskrDr)rrrr prefix_ones suffix_onesrs rrz(WhisperTokenizer.get_special_tokens_masks& &72'[]a3 cC 2 233 c  1#K(8"89KG GqcC $445!s;?O9OPS^^^r8c|j||| |jdd}t|Dcgc] \}}|dz|f}}}|Scc}}w)N)rrrr )rrr$)rrr no_timestamps forced_tokensrankr2forced_decoder_idss rget_decoder_prompt_idsz'WhisperTokenizer.get_decoder_prompt_idssa D8TaPab **12. CL]C[\KD%tax/\\!!]sA c"t|||||S)N)return_timestampsreturn_languager*) _decode_asr)r model_outputsrrr*s rrzWhisperTokenizer._decode_asrs  /+)   r8r=cdd|jzd}|ddd}tfd|Dd}| j|}td |d |j | |dS) z`Converts prompt text to IDs that can be passed to [`~WhisperForConditionalGeneration.generate`].r[r}F)add_special_tokensrr Nc3HK|]}|jdk\s|yw)rN)r#)r~xrs rrz2WhisperTokenizer.get_prompt_ids..s$ \qqDDXDXYZD[?[ \s""zJEncountered text in the prompt corresponding to disallowed special token: r) tensor_type)stripnextrrEconvert_to_tensors)rr=return_tensorsbatch_encodingprompt_text_idsspecial_token_idr2s` rget_prompt_idszWhisperTokenizer.get_prompt_idss/tzz|1CX]^)5ab9 \O \^bc  '../?@Eijoippqrs s))n)Ek**r8r)r^r_ct|ts|j|}|s|S|d|k(}|r||vr||j|dSgS|S)Nr)rri_convert_to_listr)rr)r^r_ has_prompts rr]zWhisperTokenizer._strip_promptsc)T*--i8I q\_4 %2 1G!H!JKK r8ct|dr|jj}t|tj r|j }|S)Nnumpy)r?r;rrrAndarraytolist)r)s rrz!WhisperTokenizer._convert_to_listsD 9g &! --/I i ,!((*Ir8) NNN <|endoftext|>rrFNNF)F{Gz?)rr)r)F)FNFrFFFFN)NNN)NF)NNT)rA)(__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr modelrdictintr r7rUrrYboolrIrKr%r'rJre staticmethodrfrrrpropertyrirrrrrrr]r __classcell__)rs@rrrs$L*$&67 E.2!!!  F!T#s(^#d*F!TW[.( .(bBHjjD&4%*48$ $', %"'P"P'+Tk P  P  P!%PPP P Pf(-egl  $ ?C `d      9c9C$J9Z_`cZd9>gk& d & 14t& X\_cXc& PtCyDTQUVYQZTpu_9_379t3C_hl_ c_>"  +3 +tCy3`c$r8rtokensrprepend_punctuationsappend_punctuationsc| |j}|d}|dvrt||\}}}nt||\}}}t||||||||fSz Groups tokens by word. Returns a tuple containing a list of strings with the words, and a list of `token_id` sequences with the tokens making up each word. r>rrQrrr#rr_split_tokens_on_unicode_split_tokens_on_spaces_merge_punctuations tokenizerrrrrwords word_tokens token_indicess r_combine_tokens_into_wordsro%%QQ,DYPV,W){M,CIv,V){M{M;OQde +} ,,r8c |d}t|}g}rdg}t|ddD]w\}d}||ddf}t|} td|| zD]} | dz } td|| z t ||| z| z } t j || } td| |z t | | }t j ||t| tk7r tdr#tfdt| D}nt j| k(}|| z | z}|dkDs||kDs|}| |f}|\} }| zdz}|zdz}|j|d|||d}t|}sYjd|dz|dz|j||StdkDrj||fS|gfS)Nrr r@iThere is a bug within whisper `decode_asr` function, please report it. Dropping to prevent bad inference.c3fK|](\}}||k(r|zdz|zkrd*ywr Nr r~idxelem left_startleft_token_timestamp_sequenceright right_startseq_idxtoken_timestamp_sequencess rrz0_find_longest_common_sequence..VS!Tc *9*s:JK4Wq[A+PSBSTU .1r rDr$rXmaxminrArB RuntimeErrorrGextend sequencesr left_sequence left_lengthtotal_sequencetotal_token_timestamp_sequenceright_sequencemax_ max_indices right_lengthr1eps left_stopleft right_stopmatchesmatchingleft_mid right_midrrrrrs ` @@@@@r_find_longest_common_sequencer  vaLMm$KN (A!(D%)+&#,Yqr]#;[_"KA6 <>* q+ 45( OAg+C Q a0JK|)Ca)GHI88M*Y?@Da[1K\1-JHH^K CDE4yCJ&")%.t_&&/{S(H{x$)9k:N Q( OT.new_chunks)t bQQr8rrr Fr[r\rrwordtoken_timestampsstriderr r=rTr3rc3 K|]}|ywrr)r~ps rrz_decode_asr..Qs1Aa1s zWhisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.rc3&K|] }|d yw)r=Nr)r~chunks rrz_decode_asr..ks:%f :schunks)rWsetr#r$rr]reversedr'rgetr&r r%round_collate_word_timestampsanyrwarningr(poprr )0rrrrr*r+r%r-r, time_offsetr,previous_tokensprevious_token_timestampsskipright_stride_startr#r^r_chunk_idoutputr)r'last_timestampfirst_timestampr.r/r0 chunk_len stride_left stride_rightr2current_tokenscurrent_token_timestampsr1r=rresolved_tokens resolved_textr3r4timeresolved_token_timestamps start_timeend_time full_text new_chunksoptionalr$s0 @rrrsi$MRF KEK556HIAMOO " D)334O556GHO&<<=TU%m4l*&8$Q'..0 ++IH^_  &%&89!<CCE  )  # v 39(3C 0I{L ; &K!*\!9  "-">"P%i0 /E/+6!&!8N JM_ _!). /#% "),l LHAu ' ''0Abz$==.'%])BK\'..~>*G*X(1(8(8(I (5f  e,+-)+ ) (0E*%$,M/)"5?#:n"LM 00-.!V.!!a%(O;c !a%@PTc@c9*.)^l-JJ),A))-BB)(9%$-!/>AKORccT1~!e~&=  Do%/2I D;'*2,0E+&q)u[1!4404k*1-'..~>,65<<=UVEb+-FFB)B)2(8(8(I (5f ,6-E )? 2 F*)001IJ111KE ON(* %') $Yl*\  NN]  6S 66 22"((9 %f  &5?,E}VeE'N  e:6::IO &E$ +&%*5+=%>k"" *%  &  &J 2!!%.1 2 *-H h !&)H h  h r8c |d}t|}g}rdg}t|ddD]w\}d}||ddf}t|} td|| zD]} | dz } td|| z t ||| z| z } t j || } td| |z t | | }t j ||t| tk7r tdr#tfdt| D}nt j| k(}|| z | z}|dkDs||kDs|}| |f}|\} }| zdz}|zdz}|j|d|||d}t|}sYjd|dz|dz|j||StdkDrj||fS|gfS)Nrr rrrc3fK|](\}}||k(r|zdz|zkrd*ywrrrs rrz0_find_longest_common_sequence..rrrr rs ` @@@@@rr r r!r8c t|||\}}}|rd|ini}t||D cgc] \} } | || dd|| ddfd|"} } } | Scc} } w)Nrrrr r<)rr) rrr'rrr_roptional_language_fieldr&indicestimingss rr2r2s8FHUE1m8Gz84R! 6  D' *71:6q9;KGTVK;XYZ;[\ & G Ns%Ac| |j}|d}|dvrt||\}}}nt||\}}}t||||||||fSrrrs rrr rr8c|j|d}d}g}g}g}g}g}d} t|D]\} } |j| |j| |j|d} || vs|| | j|z|k(sZ|j| |j||j|g}g}| t | z } |||fS)zlCombine tokens into words by splitting at any position where the tokens are decoded as valid unicode points.T)rdu�r)r'r$r&rrD) rr decoded_fullreplacement_charrrrrBcurrent_indicesunicode_offset token_idxr2decodeds rrr#s##F4#HL EKMNON%f-+ 5e$y)"">$"O G +NW]];K-LLMQaa LL !   ~ .   1N O c'l *N+ +} ,,r8ct||\}}}g}g}g}t|||D]\}} } | d|jk\} |jd} |j dv} | s| s| st |dk(r4|j ||j | |j | |d|z|d<|dj| |dj| |||fS)zLCombine tokens into words by splitting at whitespace and punctuation tokens.rr}z !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~r)rrr startswithrrDr&r )rrsubwordssubword_tokens_listsubword_indices_listrrrsubwordsubword_tokenssubword_indicesr with_space punctuations rrrBs:RS\^d:e7H!#7 EKM47BUWk4l 60 #y'='=='', mmo)MM jK3u:? LL !   ~ .   1b G+E"I O " "> 2 "  $ $_ 5 6 +} ,,r8ct|dz }t|dz }|dk\rp||jdrO||j|vr:||||z||<||||z||<||||z||<d||<g||<g||<n|}|dz}|dk\rpd}d}|t|krq||jdsG|||vr@||xx||z cc<||xx||z cc<||xx||z cc<d||<g||<g||<n|}|dz }|t|krq|Dcgc]}|s| c}|dd|Dcgc]}|s| c}|dd|D cgc]} | s|  c} |ddycc}wcc}wcc} w)z1Merges punctuation tokens with neighboring words.rr rr}rN)rDr]rendswith) rrrR prependedappendedr1jr&r2rs rrrZs E QA E QA q& 8  s #a(8I(EQx%(*E!Hq F1I-F1I gaj0GAJE!HF1IGAJA Q q& A A c%j.Qx  %%(h*> !Ha H 1I "I AJ'!* $JE!HF1IGAJA Q c%j."'/$/E!H$*45e4F1I!(0#C#0GAJ040s$E!E/E7EE" E")Nu "'“¡¿([{-u"'.。,，!！?？:：”)]}、r)+rrrr functoolsrrrA tokenizersrrrrrtokenizers.modelsr tokenization_utils_tokenizersr utilsr english_normalizerrr get_loggerrrrritemsrrrrirrrr rr2rrr__all__)coders00rrus( RR!>J   H %&( e )e )e  (e  ) e   ) e   ( e  (e  *e  ,e  )e  (e  )e  'e  (e  )e   )!e " ,#e $ '%e & )'e ( ,)e * (+e , +-e . '/e 0 '1e 2 '3e 4 *5e 6 (7e 8 +9e : ';e < +=e > &?e @ &Ae B *Ce D +Ee F ,Ge H 'Ie J 'Ke L +Me N 'Oe P (Qe R (Se T )Ue V )We X )Ye Z )[e \ -]e ^ +_e ` )ae b *ce d ,ee f (ge h (ie j +ke l *me n (oe p +qe r )se t (ue v *we x )ye z *{e | )}e ~ )e @ )Ae B 'Ce D 'Ee F (Ge H (Ie J +Ke L )Me N *Oe P ,Qe R 'Se T (Ue V *We X )Ye Z )[e \ %]e ^ '_e ` )ae b ce d (ee f )ge h )ie j )ke l               Ie  P,5OO,=>.$x~>  &M (M f 0A - I-Dj- -  -4u"posyxu"p & 0A - I-Dj- -  -4-S ->-tCy-0#1L  _'?s H