qi. d d Z ddlmZmZ ddlmZ ej e Z G d de Z dgZ
y)z!Tokenization class for Perceiver. )
AddedTokenPreTrainedTokenizer)loggingc
e Zd ZdZddgZ d d fd
Zdeeef fdZ e
d Z dd e e d
e e dz d e
de e f fd
Z dd e e d
e e dz de e fd
Zdede e fdZd Zd Zd Zddededz dee fdZ xZS )PerceiverTokenizeraS
Construct a Perceiver tokenizer. The Perceiver simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
pad_token (`str`, *optional*, defaults to `"[PAD]"`):
The token used for padding, for example when batching sequences of different lengths.
bos_token (`str`, *optional*, defaults to `"[BOS]"`):
The BOS token (reserved in the vocab, but not actually used).
eos_token (`str`, *optional*, defaults to `"[EOS]"`):
The end of sequence token (reserved in the vocab, but not actually used).
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
mask_token (`str`, *optional*, defaults to `"[MASK]"`):
The MASK token, useful for masked language modeling.
cls_token (`str`, *optional*, defaults to `"[CLS]"`):
The CLS token (reserved in the vocab, but not actually used).
sep_token (`str`, *optional*, defaults to `"[SEP]"`):
The separator token, which is used when building a sequence from two sequences.
input_idsattention_maskreturnNc t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}d| _ ||||||d| _ t | j | _ t | d|||||||d| y )NF)lstriprstrip ) r ) pad_token bos_token eos_token
mask_token cls_token sep_tokenmodel_max_length )
isinstancestrr _utf_vocab_size_added_tokens_decoderlen_num_special_tokenssuper__init__)
selfr r r r r r r kwargs __class__s
f/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/perceiver/tokenization_perceiver.pyr# zPerceiverTokenizer.__init__8 s JTT]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir KUV`beKfZ
5Glv
IST]_bIcJyuEir IST]_bIcJyuEir #
6
" $'t'A'A#B
! -
c i }t | j D ] }t | }|| j z ||< ! |j | j
|S N)ranger chrr! updateadded_tokens_encoder)r$ vocabitokens r' get_vocabz PerceiverTokenizer.get_vocaba sW t++, 8AFE t 7 77E%L 8 T../ r( c | j S r* )r )r$ s r'
vocab_sizez PerceiverTokenizer.vocab_sizei s ###r( token_ids_0 token_ids_1already_has_special_tokensc |rt | ||d S |dgdgt | z z dgz S dgdgt | z z dgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r5 r6 r7 r r )r" get_special_tokens_maskr )r$ r5 r6 r7 r& s r' r9 z*PerceiverTokenizer.get_special_tokens_maskm s $ &72 '[]a 3
3!s; ///1#5 5sqcC ,,-3sS=M7MNRSQTTTr( c | | j g|z | j gz S | j g|z | j gz |z | j gz S )af
Build model inputs from a sequence or a pair of sequence for sequence classification tasks. A sequence has the
following format:
- single sequence: `[CLS] X [SEP]`
- pair of sequences: `[CLS] A [SEP] B [SEP]`
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)r$ r5 r6 s r' build_inputs_with_special_tokensz3PerceiverTokenizer.build_inputs_with_special_tokens sb & %%&48I8I7JJ J%%&48I8I7JJ[X\`\m\m[nn nr( textc ^ |j d D cg c]
}t | }}|S c c}w )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsutf-8)encoder, )r$ r> r0 tokenss r' _tokenizez PerceiverTokenizer._tokenize s, "&++g"67Q#a&77
8s *c n t | dk7 r| j }|S t | | j z }|S )z0Converts a token (str) in an id using the vocab.r )r unk_token_idordr! )r$ r1 token_ids r' _convert_token_to_idz'PerceiverTokenizer._convert_token_to_id s: u:?((H 5zD$<$< $ $ puU 9U379t3CUhlU
cU: GKo 9o379t3Co
co0c d3i
c C$J Z_`cZd r( r N) r\ tokenization_pythonr r utilsr
get_loggerrY loggerr __all__r r( r' rj s> ( B
H %k, k\
r(