qiZ<dZddlZddlZddlZddlmZddlmZmZddl m Z e je Z ddiZGd d ZGd d eZd gZy) z"Tokenization class for model MyT5.N) defaultdict) AddedTokenPreTrainedTokenizer)logging vocab_filezbyte_maps.jsonceZdZdZdZdeeeefzfdZdeeeeezfdedefdZ deeefd eeeeezffd Z d eed d eezfd Z ddeed eefdZ y ) ByteRewriteraZ Byte rewriter class for MyT5 tokenizer. This class is used to rewrite bytes using a hash tree. The hash tree is constructed from a set of rewriting rules. Args: rewriting_rules (`str` or `dict[str, str]`): A path to a json file containing the rewriting rules or a dictionary containing the rewriting rules. z[LEAF]rewriting_rulesct|tr+t|d5}tj|}dddn't|t st dt||j||_ |jDcic]\}}|| }}}|j||_ y#1swYYxYwcc}}w)NrzDrewriting_rules should be either a path to json file or a dict, got ) isinstancestropenjsonloaddict TypeErrortypeconstruct_hash_tree hash_treeitemsreverse_hash_tree)selfr fkvreverse_rewriting_ruless \/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/myt5/tokenization_myt5.py__init__zByteRewriter.__init__,s os +os+ /q"&))A, / /OT2VW[\kWlVmn 11/B4C4I4I4K"LDAq1a4"L"L!%!9!9:Q!R / /#MsB3 B?3B<rbyte_in_sequencebyte_out_sequencec|jd}|jd}|}|D]}||vri||<||}|||j<y)zL Add a leaf with the output byte sequence to the hash tree.  N)splitLEAF)rrr!r" byte_in_list byte_out_list tree_pointerbs radd_leafzByteRewriter.add_leaf9sb(--c2 )//4   +A $"$ Q'?L + #0 TYYreturnctt}dtdDD]}|g|||j<|j D]\}}|j ||||S)zE Construct a hash tree for rewritten byte sequences. c3$K|]}|d yw)02xN).0xs r z3ByteRewriter.construct_hash_tree..Ms1QsG*1s)rrranger&rr+)rr rr* in_sequence out_sequences rrz ByteRewriter.construct_hash_treeHss % 1eCj1 *A'(cIaL # **9)>)>)@ @ %K MM)[, ? @r, byte_sequenceNc\|j}|D] }||vr||} y||jS)zW Search the hash tree and return the rewritten byte sequence if found. N)rr&)rr9r)r*s rsearch_hash_treezByteRewriter.search_hash_treeUsA~~  AL +A    DII&&r,in_bytescZg}d}d}|t|kr|s |jn |j}t|t|D]?}||}||vr||}n||k(r|g} |}n$n"|j|vs/||j} |}A|j  |dz}|t|kr|S)a6 Rewrite a sequence of bytes using the hash tree. Args: in_bytes (`list[str]`): A list of bytes to be rewritten. reverse (`bool`): If True, decoding is performed with the reverse hash tree. Returns: `list[str]`: The rewritten byte sequence. r)lenrrr6r&extend) rr<reverse out_bytesb_startb_endr)jr*cur_leafs r rewrite_byteszByteRewriter.rewrite_bytesbs H %184>>d>T>TL7CM2 QK $#/?L'\ !sHE99 ,+DII6HE    X &aiG!H %$r,)F) __name__ __module__ __qualname____doc__r&rrr listr+rr;rGr1r,rr r s D Sd38n(< S 0$sD49,<'<"= 0QT 0il 0 4S> d3PTW[\_W`P`K`Fa  'd3i 'D49"`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 125): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary like in ByT5 preprocessing see [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)). additional_special_tokens (`list[str]`, *optional*): Additional special tokens used by the tokenizer. input_idsattention_maskNr-c |dkDr|t|Dcgc]}d|d }}nK|dkDrF|Dt|dkDr6tttd|} | |k7rt d|d|dt |t rt|dd n|}t |t rt|dd n|}t |t rt|dd n|}|||d |_t|j|_ d |_ tjt|d |_t|jd |_t|jd|_t% |Ld|||d|d|ycc}w)Nrz c.tdt|vS)Nextra_id)boolr)r3s rz(MyT5Tokenizer.__init__..sDs1v9M4Nr,zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to MyT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)rr>r5r decompose_map merge_map) eos_token unk_token pad_token extra_idsadditional_special_tokensr1)r6r?setfilter ValueErrorrrr_added_tokens_decoderoffset_utf_vocab_sizerrr byte_mapsr decompose_rewritermerge_rewritersuperr ) rrr\r]r^r_r`kwargsi extra_tokens __class__s rr zMyT5Tokenizer.__init__s q=6>DI)DT(Uq:aS):(U %(U ]8DMfIgjkIks6*NPi#jklLy( &yk1RSlRmn(( HRR[]`GaJydCgp GQR[]`GaJydCgp GQR[]`GaJydCgp )2yY%O"$445 #4 C#89".t~~o/N"O*4>>++FG  &?    3)Vs E1c|jSN)rf)rs r vocab_sizezMyT5Tokenizer.vocab_sizes###r,ct|j|jzDcic]}|j||}}|j |j |Scc}wrp)r6rqreconvert_ids_to_tokensupdateadded_tokens_encoder)rrlvocabs r get_vocabzMyT5Tokenizer.get_vocabsW;@SWS^S^A^;_`a++A.1`` T../ asA token_ids_0 token_ids_1already_has_special_tokensc|rt|||dS|dgt|zdgzSdgt|zdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rxryrzrr>)rjget_special_tokens_maskr?)rrxryrzrns rr|z%MyT5Tokenizer.get_special_tokens_masksy$ &72'[]a3   C#k**qc1 1c+&&1#-!s;7G1GHA3NNr, token_idsct|dkDr7|d|jk(r%tjd|jd|S||jgzS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r? eos_token_idwarningswarnr\)rr}s r_add_eos_if_not_presentz%MyT5Tokenizer._add_eos_if_not_presents] y>A )B-43D3D"D MM,T^^,<=++   1 122 2r,ct|jg}|t||zdgzSt||z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. MyT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. r)rr?)rrxryeoss r$create_token_type_ids_from_sequencesz2MyT5Tokenizer.create_token_type_ids_from_sequencessP   !  {S()QC/ /;${2S89QC??r,cX|j|}||S|j|}||zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r)rrxrys r build_inputs_with_special_tokensz.MyT5Tokenizer.build_inputs_with_special_tokenss;&22;?   66{CK, ,r,textc r|jdDcgc]}|d}}|j|}|Scc}w)zTake as input a string and return a list of strings (tokens) for words/sub-words. Represents tokens in two character hex formatutf-8r0)encodemorphological_encode)rrrkrltokenss r _tokenizezMyT5Tokenizer._tokenize4s@'+kk'&:;QsG*;;**62 ?#dF_F_B``  0E %5005==//  0 9 r,save_directoryfilename_prefixcntjj|r2tjj||r|dzndtdz}n |r|dznd|z}t |dd5}|j tj|jdd ddd|fS#1swY|fSxYw) N-rwr)encodingrYF)indent ensure_ascii) ospathisdirjoinVOCAB_FILES_NAMESrwriterdumpsrg)rrrrwriters rsave_vocabularyzMyT5Tokenizer.save_vocabularyns 77== (/3!6rUfgsUt tJ4C/C/n\J *cG 4 S LLDNN15Q R S} S}s ,2B))B4)zzz}N)r-N)NFrp)rHrIrJrKmodel_input_namesrvocab_files_namesr propertyrqrwrLrrUr|rrrrrrrrrrtupler __classcell__)rns@rrNrNs4%&67) "&,  , \$$puO9O379t3COhlO cO8 3c 3tCy 3GK@9@379t3C@ c@0GK-9-379t3C- c-4cS  DI$s) DI$s) . c C$J Z_`cZd r,rN)rKrrr collectionsrtokenization_pythonrrutilsr get_loggerrHloggerrr rN__all__r1r,rrse) #B   H %"#34ccLr'rj  r,