qiM d dl mZmZmZmZmZ d dlmZ ddlm Z ddl
m Z e j e
ZdddZ G d d
e Zd
gZy ) ) Tokenizerdecoders normalizerspre_tokenizers
processors)BPE )TokenizersBackend)loggingz
vocab.jsonz
merges.txt)
vocab_file merges_filec e Zd ZdZeZddgZeZ dde e
e e f z dz de e e z dz de de d e d
e d e d e dz d
e dz f fd
Z
xZS )HerbertTokenizera
Construct a BPE tokenizer for HerBERT (backed by HuggingFace's tokenizers library).
Peculiarities:
- uses BERT's pre-tokenizer: BertPreTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of
a punctuation character will be treated separately.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the methods. Users should refer to the
superclass for more information regarding methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The padding token.
mask_token (`str`, *optional*, defaults to `""`):
The mask token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary.
merges (`str` or `list[str]`, *optional*):
Custom merges list.
input_idsattention_maskNvocabmerges cls_token unk_token pad_token
mask_token sep_tokenr r
c
D ||n t | di| _ |xs g | _ t t | j | j d t | d | _ t
j dddd | j
_ t j | j
_ t j d | j
_ t | @ d |||||d|
t# j$ | j& d f| j( df
| j
_ y ) Nr z)r r dropoutr end_of_word_suffixFT) lowercase
strip_accents
clean_texthandle_chinese_chars)suffix)r r r r r )sepcls )str_vocab_mergesr r
_tokenizerr BertNormalizer
normalizerr BertPreTokenizer
pre_tokenizerr
BPEDecoderdecodersuper__init__r BertProcessingr r post_processor) selfr r r r r r r r r
kwargs __class__s b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/herbert/tokenization_herbert.pyr0 zHerbertTokenizer.__init__A s % 0es9~q6I | # kk|| i.#)
&1%?%?5TX\&
" )7(G(G(I%"*"5"5V"D
!
*4)B)B # #*
& ) NNzzzzzNN)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelr% dictintlistr0
__classcell__)r5 s @r6 r r s @ *$&67 E .2)- " !%"&+
T#s(^#d*+
d3i$&+
+
+
+
+
+
$J+
4Z+
+
r7 r N)
tokenizersr r r r r tokenizers.modelsr tokenization_utils_tokenizersr
utilsr
get_loggerr8 loggerr< r __all__r$ r7 r6 rK sP T S ! >
H %#/ M P
( P
f
r7