qiQ4ddlZddlmZGddeZdgZy)N)TokenizersBackendc ReZdZdZ d deeezdededzdedef fd ZxZ S) ParakeetTokenizera  Inherits all methods from [`PreTrainedTokenizerFast`]. Users should refer to this superclass for more information regarding those methods, except for `_decode` which is overridden to adapt it to CTC decoding: 1. Group consecutive tokens 2. Filter out the blank token N token_idsskip_special_tokensclean_up_tokenization_spaces group_tokensreturnc t|tr|g}|r%tj|Dcgc]}|d }}|Dcgc]}||jk7s|}}t |d|||d|Scc}wcc}w)Nr)rrr ) isinstanceint itertoolsgroupby pad_token_idsuper_decode) selfrrr r kwargs token_grouptoken __class__s d/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/parakeet/tokenization_parakeet.pyrzParakeetTokenizer._decodes i %" I ;D;L;LY;WXKQXIX)2PuUd>O>O5OUP Pw  3)E    YQs A.A3A3)FNT) __name__ __module__ __qualname____doc__rlistboolstrr __classcell__)rs@rrrsY%*48!  c? " '+Tk       r)rtokenization_utils_tokenizersrr__all__r r#rr&s%> ) @  r#