qi d Z ddlZddlmZ ddlmZmZmZ ddlm Z m
Z
ddl m Z m
Z
G d d ed
Z e
j e Ze G d d
e Zd
gZy)z
Processor class for Donut.
N )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringloggingc e Zd Zi Zy)DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults \/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/donut/processing_donut.pyr
r
s Ir r
F)totalc e Zd Zd fd Ze d dedz deee z ez e z dz de
e fd Z e
d Zd
dZ xZS ) DonutProcessorNc & t | || y )N)super__init__)selfimage_processor tokenizerkwargs __class__s r r zDonutProcessor.__init__# s
)4r imagestextr c < |
| t d | j t fd| j j i|}| | j
|fi |d }|-||d j
dd | j |fi |d }|S |S d d< |d |d< |S ) NzBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs
images_kwargs text_kwargsadd_special_tokensF input_idslabels)
ValueError
_merge_kwargsr
r init_kwargsr
setdefault)r r r! r
output_kwargsinputs encodingss r __call__zDonutProcessor.__call__& s >dl ab b***
"&.."<"<
)T))&SM/4RSF ! m,778LeT&tL}]/KLI <M
^ ( 5F8 "+K"8F; Mr c N | j j }t |ddgz S )Nr' r( )r model_input_nameslist)r image_processor_input_namess r r2 z DonutProcessor.model_input_namesF s* &*&:&:&L&L#/;2IIJJr c || j j }i }|r[t j d|t j }|n2||j d }d|vrn|d|j
d dz }|t d t d }t j | }t j d| d|t j } | |j |d }n| j } t j | }
t j | } t j |
d| |t j t j z } | | j d j } d| v r3d| v r/| j | d| }
|
rt |
dk( r|
d
}
|
||< ntg ||< | j d D ]? }|j }||v r|d
d k( r
|d
d dk( r|dd
}|| j | A t || dk( r || d
||< ||j! | t | z d j }|dd d k( r|g| j |dd d| z S |r[|r|r|gS |S |rg S d|iS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z
text_sequence)r get_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r tokensr9 r: outputpotential_start start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs r rK zDonutProcessor.token2jsonL s ..88:K ii
FO & !6!6!8!:;K+%%&B (9(9#(>(BCK c%jCH95C ))C.K T+ a"8&"--PI R8 %OO - &(ii &<#$&IIi$8! ))*+51B0CDfbmm^`^g^gNg & %mmA . 4 4 6G (W-? $Ze f "5zQ(-a*/F3K&(s $+MM)$< 5D#':: K K
8Gr r )__doc__r@ image_utilsr processing_utilsr r r tokenization_utils_baser r utilsr
r r
get_loggerr loggerr __all__r r r rf sm
% H H C ,+5
H % bG^ bG bGJ
r