qi d Z ddlmZmZmZ ddlmZ ddlmZm Z ddl
m Z e j e
Zddd Z G d
d e Zd gZy )
z$Tokenization classes for OpenAI GPT. ) Tokenizerdecoderspre_tokenizers)BPE )
AddedTokenTokenizersBackend)loggingz
vocab.jsonz
merges.txt)
vocab_file merges_filec e Zd ZdZeZddgZeZ d
de e
e e f z dz de e e z dz de de
e z d e
e z d
e
e z d e
e z dz f fd
Z xZS )
GPT2TokenizeraF
Construct a GPT-2 tokenizer. Based on byte-level Byte-Pair-Encoding.
This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will
be encoded differently whether it is at the beginning of the sentence (without space) or not:
```python
>>> from transformers import GPT2Tokenizer
>>> tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2")
>>> tokenizer("Hello world")["input_ids"]
[15496, 995]
>>> tokenizer(" Hello world")["input_ids"]
[18435, 995]
```
You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you
call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.
When used with `is_split_into_words=True`, this tokenizer will add a space before each word (even the first one).
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
errors (`str`, *optional*, defaults to `"replace"`):
Paradigm to follow when decoding bytes to UTF-8. See
[bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The beginning of sequence token.
eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The end of sequence token.
pad_token (`str`, *optional*):
The token used for padding, for example when batching sequences of different lengths.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word. (GPT2 tokenizer detect beginning of words by the preceding space).
add_bos_token (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial beginning of sentence token to the input. This allows to treat the leading
word just as any other word.
vocab (`str` or `dict[str, int]`, *optional*):
Custom vocabulary dictionary. If not provided, vocabulary is loaded from `vocab_file`.
merges (`str` or `list[str]`, *optional*):
Custom merges list. If not provided, merges are loaded from `merges_file`.
input_idsattention_maskNvocabmergeserrors unk_token bos_token eos_token pad_tokenc
b || _ ||ni | _ |xs g | _ t t | j | j d ddd | _ t
j | | j
_ t j | j
_
t
| 0 d||||||d| y )N F)r r dropoutcontinuing_subword_prefixend_of_word_suffixfuse_unk)add_prefix_space)r r r r r r )
r _vocab_mergesr r
_tokenizerr ByteLevel
pre_tokenizerr decodersuper__init__) selfr r r r r r r r kwargs __class__s \/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/gpt2/tokenization_gpt2.pyr' zGPT2Tokenizer.__init__^ s !1 $ 0eb | # kk|| *,#%
)7(@(@Rb(c%"*"4"4"6
-
)NNreplace
<|endoftext|>r. r. NF)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelstrdictintlistr r'
__classcell__)r* s @r+ r r s 8t *$&67 E .2)- &5&5&5-1 #
T#s(^#d*#
d3i$&#
#
# #
#
#
##
#d*#
#
r, r N)r2
tokenizersr r r tokenizers.modelsr tokenization_utils_tokenizersr r utilsr
get_loggerr/ loggerr3 r __all__r r, r+ rC sW + : : ! J
H % b
% b
J
r,