qi n d dl Z d dlZd dlmZmZmZmZmZ d dlm Z ddl
m Z dddZ G d d
e Z
d
gZy) N) Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )TokenizersBackendz spiece.modelztokenizer.json)
vocab_filetokenizer_filec e Zd ZdZeZddgZeZ d fd Z d Z
d Z dde e
e z d ed
edz d ed ef
fd
Z xZS )
LasrTokenizera
Construct a LASR tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dict. If not provided, a minimal vocabulary is created using the special tokens.
input_idsattention_maskNc H || _ |q|D
cg c] }
dt |
v s|
} }
t | dk r!|t | D cg c] } d| d
c} z
}nC|dkD r>|t | k7 r0t d| d| d t | D cg c] } d| d
} } | }||| _ nbt | dft | dft | dfd g| _ t |dz
d
d
D ]# } | j
j
d| ddf % t t | j
d d
| _ |$t j | | j _ t j t j t j ddd g | j _ t% j ddd | j _ t)
| T d|||||d| t- j. ddgg dd| j0 fg | j _ y c c}
w c c} w c c} w )Nz
r zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to LasrTokenizer. In this case the additional_special_tokens must include the extra_ids tokensg ) ▁g r F)unk_id
byte_fallbackr alwaysT) replacementprepend_schemesplit) eos_token unk_token pad_token extra_idsadditional_special_tokens$A)r! r" z$Br" )singlepairspecial_tokens )
_extra_idsstrlenrange
ValueError
_vocab_scoresappendr r
_tokenizerr Precompiled
normalizerr SequenceWhitespaceSplit Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessing eos_token_idpost_processor)selfr r r _spm_precompiled_charsmapr r vocabr kwargsx extra_tokensi __class__s \/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/lasr/tokenization_lasr.pyr7 zLasrTokenizer.__init__J s] $ % 0'@[!LTWXYTZDZA[L[< 1$)yIY-ZA
1#Q.?-ZZ)Q 9L0A#A &yk1RSlRm n 8=Y7GH!j1 -HLH(4 % !&D Y %Y %Y % "D 9q="b1
D""))Zs!+-**+ *
&k \-Z Is HH
H
H c T t t t d | j S )zQGet the list of sentinel tokens (extra_id tokens) from additional_special_tokens.c D t t j d| d uS )Nz)boolresearch)r? s rC z3LasrTokenizer.get_sentinel_tokens.. s bii0A1&E!Fd!R )listsetfilterr )r; s rC get_sentinel_tokensz!LasrTokenizer.get_sentinel_tokens s& RTXTrTrs t
rJ c f | j D cg c] }| j | c}S c c}w )z&Get the token IDs for sentinel tokens.)rN convert_tokens_to_ids)r; tokens rC get_sentinel_token_idsz$LasrTokenizer.get_sentinel_token_ids s* ?C?W?W?YZe**51ZZZs . token_idsskip_special_tokens clean_up_tokenization_spaces group_tokensreturnc t |t r|g}|r%t j | D cg c] }|d }}|D cg c] }|| j k7 s| }}t | d|||d|S c c}w c c}w )Nr )rS rT rU r& )
isinstanceint itertoolsgroupby pad_token_idr6 _decode) r; rS rT rU rV r> token_grouprQ rB s rC r^ zLasrTokenizer._decode s i %" I ;D;L;LY;WXKQ XIX )2PuUd>O>O5OUP Pw
3)E
Y Qs A. A3A3)r" zzNd NNN)FNT)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelr7 rN rR rZ rK rF r( r^
__classcell__)rB s @rC r r ! s "H *$&67 E "&"&K
Z
[ %*48 !
c?
"
'+Tk
rJ r )r[ rG
tokenizersr r r r r tokenizers.modelsr tokenization_utils_tokenizersr
re r __all__r& rJ rC rn sA * S S % > $2EUV U
% U
p
rJ