qi d Z ddlmZ ddlmZ ddlmZ ddlmZ ddl m
Z
ej e Z
d d
iZ ed G d
de ZdgZy)z Tokenization class for SpeechT5. )Any )SentencePieceBackend)logging)requires )EnglishNumberNormalizer
vocab_filezspm_char.model)
sentencepiece)backendsc
e Zd ZdZeZddgZdZ ddee e
f dz ddf fd
Z dd Z e
d
Zej d Zddee fd Z dd
ee dee dz dedee f fd
Z dd
ee dee dz dee fdZ xZS )SpeechT5Tokenizera
Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The begin of sequence token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
normalize (`bool`, *optional*, defaults to `False`):
Whether to convert numeric quantities in the text to their spelt-out english counterparts.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskFNsp_model_kwargsreturnc
\ || _ d | _ |||d< t | d||||||d| y )Nr )r
bos_token eos_token unk_token pad_token normalize )r _normalizersuper__init__)
selfr
r r r r r r kwargs __class__s
d/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr zSpeechT5Tokenizer.__init__M sW # &(7F$ %
!
c v |j d| j }|rd|z }|r| j | }||fS )Nr )popr
normalizer)r textis_split_into_wordsr r s r prepare_for_tokenizationz*SpeechT5Tokenizer.prepare_for_tokenizationj s= JJ{DNN; :D ??4(Df~ r! c P | j t | _ | j S N)r r )r s r r% z SpeechT5Tokenizer.normalizerr s% # 6 8D r! c || _ y r* )r )r values r r% z SpeechT5Tokenizer.normalizerx s
r! c L ||| j gz S ||z | j gz S )z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r token_ids_0 token_ids_1s r build_inputs_with_special_tokensz2SpeechT5Tokenizer.build_inputs_with_special_tokens| s5 $"3"3!44 4[(D,=,=+>>>r! r/ r0 already_has_special_tokensc |rt | ||d S dg}|dgt | z |z S dgt | z dgt | z z |z S )NT)r/ r0 r2 r r )r get_special_tokens_masklen)r r/ r0 r2 suffix_onesr s r r4 z)SpeechT5Tokenizer.get_special_tokens_mask su &72 '[]a 3
c C#k**k9 9c+&&A3[1A+AB[PPr! c n | j g}|t ||z dgz S t ||z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. SpeechT5 does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )r. r5 )r r/ r0 eoss r $create_token_type_ids_from_sequencesz6SpeechT5Tokenizer.create_token_type_ids_from_sequences sK ! {S()QC/ /;,s23qc99r! )zzzzFN)Fr* )NF)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesis_fastdictstrr r r( propertyr% setterlistintr1 boolr4 r9
__classcell__)r s @r r r s (T *$&67G
15
c3h $.
:
! !?QUVYQZ ? pu Q 9 Q379t3C Qhl Q
c Q GK: 9:379t3C:
c:r! r N)r= typingr tokenization_utils_sentencepiecer utilsr utils.import_utilsr number_normalizerr
get_loggerr: loggerr> r __all__r r! r rR si ' D * 6
H %!#34
%&E:, E: 'E:P
r!