qiA d Z ddlmZmZmZmZmZmZ ddlm Z ddl
m Z ddl m
Z
e
j e ZddiZ G d d
e Zd
gZy ) z Tokenization classes for XGLM. )Regex Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )TokenizersBackend)loggingtokenizer_fileztokenizer.jsonc e Zd ZdZeZddgZeZ dde e
e e e f z dz de de de d e d
e d e d e
f fd
Z xZS )
XGLMTokenizeraW
Construct a XGLM tokenizer (backed by HuggingFace's tokenizers library). Based on BPE.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
tokenizer_file (`str`, *optional*):
Path to a tokenizers JSON file containing the serialization of a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary. If not provided, a minimal vocabulary is created.
merges (`list[tuple[str, str]]`, *optional*):
Custom merge rules for BPE. If not provided, merges are generated from the vocabulary.
add_prefix_space (`bool`, *optional*, defaults to `True`):
Whether to add a prefix space before encoding.
input_idsattention_maskNvocab bos_token eos_token sep_token cls_token unk_token pad_tokenadd_prefix_spacec d| _ t | j D
cg c] }
d|
d
} }
| j dg xs g | d< | dxx | D cg c] } | | d vs | c} z
cc< || _ ||| _ n7t | dft | dft | dft | dfg| _ t
t | j dd | _ t j t j t d d
t j t j t d d
g | j _ |rd nd
}
t j d|
| j _ t% j d|
| j _ t) | T d|||||||d| t- j. | j0 d| j0 | j0 d| j0 d
| j0 d| j0 | j2 | j4 f| j0 | j6 fg | j _ y c c}
w c c} w )N z additional_special_tokensg r
F)r unk_id
byte_fallbackz[\n\r\t] z {2,}alwaysneveru ▁) replacementprepend_scheme)r r r r r r r z $A z $B )singlepairspecial_tokens ) num_madeup_wordsrangegetr _vocabstrr r
_tokenizerr SequenceReplacer NFKC
normalizerr Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessingr r bos_token_id eos_token_idpost_processor)selfr r r r r r r r kwargsi madeup_wordswordr$ __class__s \/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/xglm/tokenization_xglm.pyr7 zXGLMTokenizer.__init__> sF !" 49$:O:O4PQq+aS*Q Q.4jj9TVX.Y._]_*+*+ )0
T@[9\-\D0
+ !1 DK Y %Y %Y %Y % DK $G$++aW\$]^%0%9%9##E+$6< "##E(OS9
&
" &67(6(@(@Ucq(r%"*"4"4We"f
-
*4)F)Fnn%T$..)9:NN#4'7q 8HT^^L\] !2!23 !2!23 *
&S R0
s
I I
I
)N