qiF0ddlZddlZddlZddlmZddlmZddlmZddl Z ddl m Z ddl m Z ddlmZe j eZd d d d d dZdZedGdde Zdedeeefde j0fdZdeddfdZdedeezfdZdgZy)N)Path)copyfile)Any)PreTrainedTokenizer)logging)requiresz source.spmz target.spmz vocab.jsonztarget_vocab.jsonztokenizer_config.json) source_spm target_spmvocabtarget_vocab_filetokenizer_config_fileu▁) sentencepiece)backendsc eZdZdZeZddgZ d+deee fdzddffd Z dZ d edefd Z d Z d efd Zd edeefdZdedefdZfdZfdZ d,dededzdeffd ZdeedefdZd-deefdZdZdZedefdZd-dededzdeefdZdefdZd Z d!Z!defd"Z"d#eddfd$Z#d%Z$d&Z% d.d'ed(edzd)edeefd*Z&xZ'S)/MarianTokenizeraB Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: source_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the source language. target_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the target language. source_lang (`str`, *optional*): A string representing the source language. target_lang (`str`, *optional*): A string representing the target language. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`list[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ``` input_idsattention_maskNsp_model_kwargsreturnc | in| |_t|js Jd|| |_t ||_t ||j vr td| rKt ||_|jjDcic]\}}|| c}}|_ g|_ nv|j jDcic]\}}|| c}}|_ |j Dcgc](}|jds|jds'|*c}|_ ||_||_||g|_t#||j|_t#||j|_|j$|_|j |_|j-d|_t1|dd||||| | |j|| d | ycc}}wcc}}wcc}w)Nzcannot find spm source z token must be in the vocab>>< -=> > "+,=">D -1-@-@-F-F-HITQAqDIDL,.D )-1\\-?-?-ABTQAqDBDL>Bll2vall[_N`efeoeopteu12vD )&&$j1#:t/C/CD":t/C/CD??#||  ,1)  ##- 00/+  -JC2vs G G7G"G" G"c ddlm}||jj|_y#t t f$r tjdd|_YywxYw)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.c|SNr!)xs r<z3MarianTokenizer._setup_normalizer..sQ) sacremosesr>r normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)r7r>s r<r3z!MarianTokenizer._setup_normalizersM / 7#78H8H#I#S#SD ./ / MM@ A#.D  /s'*,AArAc,|r|j|SdS)zHCover moses empty string edge case. They return empty list for '' input!)rF)r7rAs r<rEzMarianTokenizer.normalizes*+t##A&33rCcn||jvr|j|S|j|jSr@)r2r)r7tokens r<_convert_token_to_idz$MarianTokenizer._convert_token_to_ids8 D(( (''. .##DNN33rCtextcg}|jdr5|jdx}dk7r|j|d|dz||dzd}||fS)z6Remove language codes like >>fr<< before sentencepiecerrN)r+findappend)r7rPcodeend_locs r<remove_language_codez$MarianTokenizer.remove_language_codesX ??4 4&@gR%G KK]w{+ ,! &DTzrCcv|j|\}}|jj|t}||zS)N)out_type)rXr1encoder%)r7rPrVpiecess r< _tokenizezMarianTokenizer._tokenizes;..t4 d!!(((<f}rCindexc||jvr|j|S|jr |jn |j}|j |}|r|S|j S)z?Converts an index (integer) in a token (str) using the decoder.)r)r4r/r0 IdToPiecer)r7r^ spm_modelpieces r<_convert_id_to_tokenz$MarianTokenizer._convert_id_to_tokensU DLL <<& &'+'H'HDOOdoo ##E*u14>>1rCc $t||fi|S)ad Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[list[int], list[list[int]], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `list[str]`: The list of decoded sentences. )r5 batch_decode)r7 sequencesr8r;s r<rezMarianTokenizer.batch_decodes*w#I888rCc $t||fi|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, list[int], np.ndarray, torch.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )r5decode)r7 token_idsr8r;s r<rhzMarianTokenizer.decodes0w~i2622rCskip_special_tokensclean_up_tokenization_spacesc r|j }|jd||_t|d|||d|S)zCInternal decode method that handles use_source_tokenizer parameter.use_source_tokenizer)rirjrkr!)r popr4r5_decode)r7rirjrkr8default_use_sourcer;s r<rozMarianTokenizer._decodesR"&!5!55,2JJ7MOa,b)w  3)E    rCtokenscL|jr |jn |j}g}d}|D]>}||jvr||j ||zdzz }g}.|j |@||j |z }|j td}|jS)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserL ) r4r/r0all_special_tokens decode_piecesrUreplaceSPIECE_UNDERLINEstrip)r7rqsp_modelcurrent_sub_tokens out_stringrNs r<convert_tokens_to_stringz(MarianTokenizer.convert_tokens_to_strings&*&G&G4??T__  1E///h445GH5PSVVV %'""))%0  1 h,,-?@@ ''(8#> !!rCcL|||jgzS||z|jgzS)z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r7 token_ids_0 token_ids_1s r< build_inputs_with_special_tokensz0MarianTokenizer.build_inputs_with_special_tokens&s5  $"3"3!44 4[(D,=,=+>>>rCcH|j|_|j|_yr@)r/r1r$r2r7s r<_switch_to_input_modez%MarianTokenizer._switch_to_input_mode-s??#||rCcb|j|_|jr|j|_yyr@)r0r1r r'r2rs r<_switch_to_target_modez&MarianTokenizer._switch_to_target_mode1s*??   #'#6#6D  rCc,t|jSr@)lenr$rs r< vocab_sizezMarianTokenizer.vocab_size6s4<<  rCsave_directoryfilename_prefixcztjj|stj d|dyg}|j rtjj ||r|dzndtdz}tjj ||r|dzndtdz}t|j|t|j||j||j|nXtjj ||r|dzndtdz}t|j||j|ttdtdg|j|j|jgD]\}}} tjj ||r|dznd|z} tjj!|tjj!| k7r=tjj#|rt%|| |j| tjj#|rt'| d 5} | j)} | j+| ddd|j| t-|S#1swY(xYw) NzVocabulary path (z) should be a directory-rLr r r r wb)ospathisdirloggererrorr joinVOCAB_FILES_NAMES save_jsonr$r'rUzipr-r/r0abspathisfileropenserialized_model_protowritetuple) r7rr saved_filesout_src_vocab_fileout_tgt_vocab_fileout_vocab_filespm_save_filename spm_orig_pathra spm_save_pathficontent_spiece_models r<save_vocabularyzMarianTokenizer.save_vocabulary:sLww}}^, LL,^,<   1 2   1 2WW\\/3!6rUfgnUo oN dllN 3   ~ .;> | ,.? .M N NN __doo .<  2 7 }i GGLL/3!6rUf fMww}-1OOTVT[T[TbTbcpTq 6""=1WW^^M2-.3"+4+K+K+M(HH123""=1 2"[!! 33s ("J11J: c"|jSr@) get_src_vocabrs r< get_vocabzMarianTokenizer.get_vocabgs!!##rCcBt|jfi|jSr@)dictr$added_tokens_encoderrs r<rzMarianTokenizer.get_src_vocabjsDLL>D$=$=>>rCcBt|jfi|jSr@)rr'added_tokens_decoderrs r< get_tgt_vocabzMarianTokenizer.get_tgt_vocabmsD''E4+D+DEErCc|jj}|jtj gd|S)N)r/r0r1rFr )__dict__copyupdaterfromkeys)r7states r< __getstate__zMarianTokenizer.__getstate__ps4 ""$ MMm n  rCdc|_tdsi_tdsd_fdjD\__j _jy)Nrr4Fc3JK|]}t|jywr@)r.r).0fr7s r< z/MarianTokenizer.__setstate__..s+fRSHQ8L8L,M+fs #) rhasattrrr4r-r/r0r1r3)r7rs` r< __setstate__zMarianTokenizer.__setstate__wsd t./#%D t;<05D -+fW[WeWe+f(??  rCcy)zJust EOSr!)r7argsr8s r<num_special_tokens_to_addz)MarianTokenizer.num_special_tokens_to_addsrCct|j}|j|j|Dcgc] }||vrdnd c}Scc}w)Nrr)setall_special_idsremove unk_token_id)r7seqrrAs r<_special_token_maskz#MarianTokenizer._special_token_masksFd223t001:=>QQ/)q0>>>sArralready_has_special_tokensc|r|j|S||j|dgzS|j||zdgzS)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.r)r)r7rrrs r<get_special_tokens_maskz'MarianTokenizer.get_special_tokens_masksS &++K8 8  ++K8A3> >++K+,EF!L LrC) NNNzzziNF)FNr@)NF)(__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrr%rr6r3rErOrXlistr]intrcrerhboolror|rrrpropertyrrrrrrrrrrr __classcell__)r;s@r<rr,s8 t*$&6715= c3h$.=  = ~/434344cd3i 2#2#29.3:%*48  " '+Tk   ""tCy"S" ?QUVYQZ?,7 !C!!+"c+"C$J+"Z_`cZd+"Z$4$?Fd !d !t !? fk M M.2Tk M^b M c MrCrrrrcRtjdi|}|j||S)Nr!)rSentencePieceProcessorLoad)rrspms r<r.r.s%  . . A ACHHTN JrCcvt|d5}tj||ddddy#1swYyxYw)NwrS)indent)rjsondump)datarrs r<rrs2 dC%A $!$%%%s/8cpt|d5}tj|cdddS#1swYyxYw)Nr)rrload)rrs r<r#r#s- dCAyy|s,5)rrrIpathlibrshutilrtypingrrtokenization_pythonrutilsrutils.import_utilsr get_loggerrrrrwrr%rrr.rrr#__all__r!rCr<rs 6*   H % ,4   %&iM)iM'iMX 3c3hM<`<` %#%$% CD4K  rC