qiC_ 0 d Z ddlZddlZddlZddlZddlmZ ddlmZ ej e
Z dddZ d Z
G d
d e Z d Zd
Zededdddddf Z ej$ ddj' e z ej( ej* z ej, z Z ej$ d Z ej$ eej( ej* z ej, z Z ej$ d Zd dZd!dZ G d d Z d Z d Z d"d Z d gZ!y)#z!Tokenization classes for BERTweet N )PreTrainedTokenizer)logging vocab.txt bpe.codes)
vocab_file merges_filec x t }| d }| dd D ] }|j ||f |} t | }|S )z
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairs prev_charchars d/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairsr # sO
EEQIQR
9d#$
JE L c e Zd ZdZeZ d fd Zed Zd Z d Z
d Z d Z d Z
d Zd
Zd Zdd
eded z deedf fdZd Z xZS )BertweetTokenizera
Constructs a BERTweet tokenizer, using Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
normalization (`bool`, *optional*, defaults to `False`):
Whether or not to apply a normalization preprocess.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
c ddl m} | | _ || _ || _ i | _ d| j t | < d| j t | < d| j t | < d| j t | < | j | | j j D
ci c] \ }
}||
c}}
| _ t |d 5 }|j j d d d
}d d d D cg c] }t! |j d d
}}t# t% |t' t) | | _ i | _ || _ t1 | _ d d d
| _ t7 | p d||||||| |
dddd | y # t $ r t j d d | _ Y w xY wc c}}
w # 1 sw Y xY wc c}w )Nr )demojizezsemoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0r r utf-8encoding
'z...)u ’u … all_zerosTcls_double_sep)
normalization bos_token eos_token sep_token cls_token unk_token pad_token
mask_tokentoken_type_ids_pattern%token_type_ids_include_special_tokensspecial_tokens_pattern ) emojir demojizer ImportErrorloggerwarningr r encoderstr
add_from_fileitemsdecoderopenreadsplittupledictziprangelen bpe_rankscacher" TweetTokenizertweetPreprocessorspecial_punctssuper__init__)selfr r r" r# r$ r% r& r' r( r) kwargsr kv
merges_handlemergesmerge __class__s r rF zBertweetTokenizer.__init__h s " & %DN %& '( S^$'( S^$'( S^$'( S^$ :&)- ););)=>A1>
+
0 ;M"'')//5cr:F ;9?@%
cr*+@@ c&%F *< =>
*!/!1 &)% 8
' !#.26#3
= " NN(
"DN "$ ? ; ;@s)
F 3
G#G #G%GG Gc , t | j S N)r? r3 rG s r
vocab_sizez BertweetTokenizer.vocab_size s 4<< r c B t | j fi | j S rP )r< r3 added_tokens_encoderrQ s r get_vocabzBertweetTokenizer.get_vocab s DLL>D$=$=>>r c $ | j v r j | S t | }t t |d d |d dz gz }t | }|s|S t | fd }| j
vrn|\ }}g }d}|t
| k r |j || } |j ||| | }|| |k( r6|t
| dz
k r%||dz |k( r|j ||z |dz
}n|j || |dz
}|t
| k rt | }|}t
| dk( rn t | }dj | }|d d }| j |< |S # t $ r |j ||d Y nw xY w)
Nr zc N j j | t d S )Ninf)r@ getfloat)pairrG s r z'BertweetTokenizer.bpe.. s 1C1CD%PU,1W r keyr r r @@ ) rA r; listr minr@ r? indexextend
ValueErrorappendjoin)
rG tokenr r bigramfirstsecondnew_wordijs
` r bpezBertweetTokenizer.bpe s DJJ ::e$ $U|T$s)_R6(9'::;$L$WXFT^^+ "ME6HAc$i-
5!,A
OOD1I.A7e#CIM(9d1q5kV>S OOEFN3FA OODG,FA c$i- XHD4yA~!$9 : zz$ CRy
5 - " OODH-s E/ / FFc | j r| j | }g }t j d| }|D ]: }|j t | j
| j d < |S )zTokenize a string.z\S+\n? )r" normalizeTweetrefindallrd ra ro r: )rG text split_tokenswordsrh s r _tokenizezBertweetTokenizer._tokenize sm &&t,D
9d+ BE TXXe_%:%:3%? @ A Br c ( | j D ]! }|j || j | }# | j j | }dj |D cg c] }| j | c} }|j dd j dd j dd j dd j d d
}|j d d j d
d j dd j dd j dd j dd }|j dd j dd j dd j d d }dj |j
S c c}w ) z'
Normalize a raw Tweet
rq zcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rD replacerC tokenizerg normalizeTokenr: )rG tweetpuncttokensrh normTweets r rr z BertweetTokenizer.normalizeTweet se (( EEMM%)<)>@
r Nsave_directoryfilename_prefixreturn.c t j j | st j d| d yt | di }|r| dnd}t j j
|||j dd z }t |d d
5 }t | j j d
D ]" \ }}|dk\ s |j | d| d $ ddd t j j
|||j dd z } t | d d
5 }
|
j d t | j j d
D ddd || fS # 1 sw Y xY w# 1 sw Y || fS xY w)zF
Save the vocabulary and merges files to a directory.
zVocabulary path (z) should be a directoryr- vocab_files_names-r r r wr r c | d S Nr r- kvs r r\ z3BertweetTokenizer.save_vocabulary..= s rRSu r r] rq r Nr r c 3 J K | ] \ }}d j | dz yw)rq r N)rg ).0
bpe_tokens token_indexs r z4BertweetTokenizer.save_vocabulary..E s* +J $t+ s !#c | d S r r- r s r r\ z3BertweetTokenizer.save_vocabulary..G s
]_`a]b r )ospathisdirr1 errorgetattrrg rY r8 sortedr3 r6 write
writelinesr@ ) rG r r r prefixr frh token_id
merge_filewriters r save_vocabularyz!BertweetTokenizer.save_vocabulary. sy ww}}^, LL,^,<DT#U
5xq=GGugQxj 34
5 5 WW\\.&;L;P;PQ^`k;l2lm
*cG
4 /5dnn6J6J6LRb/c
J'' 5 5 J''s 2E0E0'=E<0E9<Fc t |t r* t |dd 5 }| j | ddd y|j }|D ]Z }|j }|j d }|dk( r t d |d| }t | j | j |< \ y# 1 sw Y yxY w# t $ r}|d}~wt
$ r t
d| d w xY w)
zi
Loads a pre-existing dictionary from a text file and adds its symbols to this instance.
rr r Nz Incorrect encoding detected in z , please rebuild the datasetrq r z5Incorrect dictionary format, expected ' ')
isinstancer4 r8 r5 FileNotFoundError UnicodeError Exception readlinesr rfindre r? r3 ) rG r fdfnfelineslineTmplineidxr s r r5 z BertweetTokenizer.add_from_fileL s a
c!S73 +r&&r*+
3G==?D**S/Cby !XYY:D!$T\\!2DLL
3+
%
c "A!D` abb
cs3 B7 B+B7 +B40B7 4B7 7 C C C )F