qi@! d Z ddlZddlmZmZ ddlZddlmZm Z m
Z
m Z ddl m
Z
ddlmZ ddlmZmZ er e rddlZ ej( e Zd d
d Z G d d
e Zd
gZy)z!Tokenization classes for CodeGen. N)
TYPE_CHECKINGUnion) Tokenizerdecoderspre_tokenizers
processors)BPE )TokenizersBackend)is_torch_availableloggingz
vocab.jsonz
merges.txt)
vocab_file merges_filec e Zd ZdZeZddgZeZ dde e
e e f z dz de e e z dz de de d e d
e
d e
f fd
Z dd
ee e e ej" df de
de
dz de e dz de f
fd
Zd Z xZS )CodeGenTokenizera
Construct a CodeGen tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level
Byte-Pair-Encoding.
This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will
be encoded differently whether it is at the beginning of the sentence (without space) or not:
```python
>>> from transformers import CodeGenTokenizer
>>> tokenizer = CodeGenTokenizer.from_pretrained("Salesforce/codegen-350M-mono")
>>> tokenizer("Hello world")["input_ids"]
[15496, 995]
>>> tokenizer(" Hello world")["input_ids"]
[18435, 995]
```
You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer, but since
the model was not pretrained this way, it might yield a decrease in performance.
When used with `is_split_into_words=True`, this tokenizer needs to be instantiated with `add_prefix_space=True`.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab (`str` or `dict[str, int]`, *optional*):
Custom vocabulary dictionary. If not provided, vocabulary is loaded from `vocab_file`.
merges (`str` or `list[str]`, *optional*):
Custom merges list. If not provided, merges are loaded from `merges_file`.
unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The beginning of sequence token.
eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The end of sequence token.
pad_token (`str`, *optional*):
The token used for padding, for example when batching sequences of different lengths.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word. (CodeGen tokenizer detect beginning of words by the preceding space).
return_token_type_ids (`bool`, *optional*, defaults to `False`):
Whether to return token type IDs.
input_idsattention_maskNvocabmerges unk_token bos_token eos_tokenadd_prefix_spacereturn_token_type_idsc
|| _ | j r| j j d || _ ||ni | _ |xs g | _ t
t | j | j
d ddd | _ t j | | j _ t j | j _
t j ddd | j _ t!
| D d ||||||d| y )
Ntoken_type_ids F)r r dropoutcontinuing_subword_prefixend_of_word_suffixfuse_unk)r T)r use_regex trim_offsets)r r r pad_tokenr r )r model_input_namesappendr _vocab_mergesr r
_tokenizerr ByteLevel
pre_tokenizerr decoderr post_processorsuper__init__) selfr r r r r r$ r r kwargs __class__s b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/codegen/tokenization_codegen.pyr0 zCodeGenTokenizer.__init__] s &;" % % " " ) )*: ; 0 $ 0eb | # kk|| *,#%
)7(@(@Rb(c%"*"4"4"6 )3)=)= !T*
&
-"7
token_idsz torch.Tensorskip_special_tokens clean_up_tokenization_spacestruncate_before_patternreturnc r t | d|||d|}| t | dkD r| j || }|S )a
Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
tokens and clean up tokenization spaces.
Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.
Args:
token_ids (`Union[int, List[int], np.ndarray, torch.Tensor]`):
List of tokenized input ids. Can be obtained using the `__call__` method.
skip_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not to remove special tokens in the decoding.
clean_up_tokenization_spaces (`bool`, *optional*):
Whether or not to clean up the tokenization spaces. If `None`, will default to
`self.clean_up_tokenization_spaces` (available in the `tokenizer_config`).
truncate_before_pattern (`List[str]`, *optional*, defaults to `None`):
A list of regular expression strings that will be used to truncate the returned string. This can be
used to remove extra pieces of code (e.g. truncate if observing a comment symbol "#" at the beginning
of a new line). An example pattern could be `["^#", re.escape("<|endoftext|>"), "^'''", "
"]`.
kwargs (additional keyword arguments, *optional*):
Will be passed to the underlying model specific decode method.
Returns:
`str`: The decoded sentence.
)r6 r7 r8 r r% )r/ decodelentruncate)r1 r6 r7 r8 r9 r2 decoded_textr3 s r4 r< zCodeGenTokenizer.decode sY B w~
3)E
# .37N3ORS3S == 7NOLr5 c f d }|D cg c]& }t j |t j ( }}t t j d|t j }t | dkD r|d |d j
}t t j d|t j }t | dkD r|d |d j
}d}|D cg c] } ||| | c} D
cg c]
}
|
dk7 s |
} }
t | dkD r|d t | S |S c c}w c c} w c c}
w )Nc N |j | | }|r|j S dS )N)searchstart)stringpattern start_posms r4 find_rez*CodeGenTokenizer.truncate..find_re s% vy1A !1779 )r )r5 z^print z^defr rB )recompile MULTILINElistfinditerr= rD min) r1
completionr9 rI rF terminalsprintsdefsrG terminalpos
terminals_poss r4 r> zCodeGenTokenizer.truncate s! * G^^7RZZ 6^ ^bkk(J EF v;?#$7fQioo&78JBKK
BLLAB t9q=#$5d1gmmo6J V__GJ) D_
cfjlclC
} ! 2M 23 3 + _ `
s +D$ D)5
D. D.)NN
<|endoftext|>rX rX NFF)FNN)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesr& r modelstrdictintrN boolr0 r npndarrayr< r>
__classcell__)r3 s @r4 r r % s 1f *$&67 E .2)-(((!&&+.
T#s(^#d*.
d3i$&.
.
.
.
.
$.
f %*4848 + d3i^CD+ "+ '+Tk +
"&cT!1 +
+ Z r5 r )r\ rK typingr r numpyrd
tokenizersr r r r tokenizers.modelsr tokenization_utils_tokenizersr utilsr r
torch
get_loggerrY loggerr] r __all__r% r5 r4 rq si ( ' F F ! > 0
H %#/ M o ( o d
r5