qiV1<dZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z mZe r dd lmZd d lmZiZeed GddZGddZeddZddZ ddZdddZej4Zd dZddddZy)!z Contains the logic for automatic additional output capture with our forward decorators. This mostly describe the hooks used and the logic to make capture thread/context safe. ) annotationsN) ContextVar) dataclasswraps) TYPE_CHECKING)is_torchdynamo_compilingrequires)nnPreTrainedModel)torch)backendscFeZdZUdZded<dZded<dZded <dZded <y) OutputRecordera Configuration for recording outputs from a model via hooks. Attributes: target_class (Type): The class (e.g., nn.Module) to which the hook will be attached. index (Optional[int]): If the output is a tuple/list, optionally record only at a specific index. layer_name (Optional[str]): Name of the submodule to target (if needed), e.g., "transformer.layer.3.attn". class_name (Optional[str]): Name of the class to which the hook will be attached. Could be the suffix of class name in some cases. ztype[nn.Module] target_classrintindexN str | None layer_name class_name)__name__ __module__ __qualname____doc____annotations__rrrU/opt/pipecat/venv/lib/python3.12/site-packages/transformers/utils/output_capturing.pyrr's,"!E3N!J !!J !r rc(eZdZdZdZdZdZdZy)CompileableContextVara Convenience wrapper around a ContextVar for usage with `torch.compile`. This behaves exactly as a `ContextVar`, except when compilation is triggered in which case it behaves as a simple global variable. This is useful as `torch.compile` cannot trace the `get` method of `ContextVar`. This however means that the access to the underlying variable is not thread-safe when compilation is triggered. cDt|||_||_d|_y)NdefaultF)r context_var global_var compiling)selfnamer&s r!__init__zCompileableContextVar.__init__Bs%dG<!r c|jr |jStrd|_|jS|jj SNT)r)r(r is_compilingr'get)r*s r!r0zCompileableContextVar.getGsB >>?? "()$(!&''++--r cjtr||_d|_y|jj |Sr.)r r(r)r'set)r*values r!r2zCompileableContextVar.setSs0 # %#DO!DN##''. .r ct|js|d|_d|_y|jj|y)NF)r)r(r'reset)r*tokens r!r5zCompileableContextVar.reset[s/ >>U]"DO"DN    " "5 )r N)rrrrr,r0r2r5rr r!r#r#:s ./*r r#output_collectorr%c6fd}|j|y)zaInstall the forward hook needed to capture the output described by `key` and `index` in `module`.c6tj}||jvrydk(r(t|dk(r|j |dt |t s|j |y||j |yy)N hidden_statesr)_active_collectorr0keyslenappend isinstancetuple)moduleargsoutputcollected_outputsrkeys r!output_capturing_hookz;install_output_capuring_hook..output_capturing_hookjs-113  $3D3I3I3K(K  / !c*;C*@&AQ&F c " ) )$q' 2&%( c " ) )& 1 E] & c " ) )&- 8'r N)register_forward_hook)rArErrFs `` r!install_output_capuring_hookrHgs 9   !67r cddlm}|jD]6\}}t||st ||d||%t ||d|8|D]\}}|j t||j s)|j5|j|jsQ|j|j|vrlt|||jy)a Recursively install all output capturing hooks on all submodules of `parent_module`. Note that we need to use this recursive approach instead of simply iterating over all modules, because we want to respect the `capture_tasks` of all individual submodels (`PreTrainedModel` instances) in the graph. That is, once we reach a submodel in the graph, its children should use this submodel's `capture_tasks`, but other parts of the graph should not. r r.)prefixN) modeling_utilsrnamed_childrenr?recursively_install_hooks"install_all_output_capturing_hooksrrendswithrrHr) parent_module module_name capture_tasksrr+rArEspecss r!rNrN{s1&446W f&/2 %f Qtf.E} U /v Qtf>U V W$J U    *z-I[I[/\    ([-A-A%BRBR-S+0@0@ 0S (U[[ IJr ctjt|jxsi}g}|j D]\}}t |t s|g}|D]c}t |ts>d|vrdnd}t |tsdn|}t |ts|nd} t| ||}|j||fe||nd}t|||t|ddy) z Install the output recording hooks on all the modules in `model`. Tis will take care of correctly dispatching the `_can_record_outputs` property of each individual submodels in case of composite models. r:rr N)rrr!_output_capturing_hooks_installedT) _CAN_RECORD_REGISTRYr0str __class__itemsr?listrr>rNsetattr) modelrK capture_flagsrSrE layer_specsrTrrrs r!rOrOs ),,S-ABHbMM)//1 /[+t,&-K  /Ee^4,3)3E3)?TU ,6uc,Bu &LZde  #u .  / /)VrFeV]; E6=r ct|ddryt5t|ddr dddyt|dddy#1swYyxYw)z Check if the model already has output capturing hooks installed, and install them if it is not already the case. Note that this is thread-safe, in case 2 (or more) threads want to install them concurrently. rWFN)getattr_hook_installation_lockrO)r^s r!maybe_install_capturing_hooksrdsRu95A 2 5=u E  22 +51 222sA AA T)tie_last_hidden_statesc&fd}|||S|S)a Decorator to intercept specific layer outputs through hooks. The hooks are installed only once and lazily, the first time output capture is requested with the `output_xxx` kwargs/config. The implementation is fully context/thread safe, except when using `torch.compile`, as dynamo is unable to trace through `ContextVar` methods. Args: tie_last_hidden_states (`bool`, *optional*, defaults to `True`): Whether to overwrite `out.hidden_states[-1]` with the `out.last_hidden_state`. This is true for all language models and should be toggled off only if `out.hidden_states[-1]` has to be the hidden state before last layer norm, which is needed for some vision models (e.g. CLIP, SigLIP) c2tfd}|S)Nc x|jdt|jdd}tj t |j xsi}|Dcic]3}d||j d|t|jd|d5}}d|vr*|j dt|jdd|d<d|vr*|j dt|jdd|d <|jDcic]\}}|s |jdd g}}}t|d kDr t|tj|}  |g|i|} tj| |D]} | d k(r snkt| d r*|| dd|| <|| j| j n5t| dr)|| dd|| <|| j| j"t%|| | | <| dk(rht'|| t(rCt|| dk(r2t%|| d dd| | <t%|| ddd| d| z<t%|| | | <t%|| | | < |dur| j+} | Scc}wcc}}w#tj| wxYw)N return_dictToutput_Fcross_attentionsoutput_attentionsoutput_cross_attentionsmask_decoder_attentionsoutput_mask_decoder_attentionsrVrr:vision_hidden_stateslast_hidden_state attentionsr r cross_)poprbconfigrXr0rYrZr[replacer=rdr;r2r5hasattrr>rprrr@r?r\to_tuple)r*rBkwargsricapturable_flagskrecordable_keysvrD output_tokenoutputsrEfuncres r!wrapperz4capture_outputs..wrapped_fn..wrappers!**]GDKKX\4]^K 477DNN8KLRPR *! vzzGA3-PWXYWZm]b9cddO "%55=CZZ'>QSX)Y> 9:),<QSX)YE @AKZJ_J_Ja g$!Qef9b!92!= g  g$%)-d3,001BCL 6t5d5f5"'' 5) A/)1 *@A1B31G1L)#.)#.55g6R6RS *=>1B31G1L)#.)#.55g6O6OP#():3)?#@GCLL(!"23"7>3GWX[G\C]abCb',->s-CADqD-I'J 278I#8NqtRSt8T2U3/',->s-C'D #():3)?#@GCL) A,e#!**,No!h"'' 5s8J J J J""J9r)rrres` r! wrapped_fnz#capture_outputs..wrapped_fns! t=  = ~r r)rrers ` r!capture_outputsrs#AF $ r )rA nn.ModulerErYrrreturnNone)rQrrRrYrSz list[tuple[str, OutputRecorder]]rr)N)r^rrKrrr)r^rrr)r __future__r threading contextvarsr dataclassesr functoolsrtypingr import_utilsr r rr rLrrXrr#r;rHrNrOLockrcrdrrr r!rs #"! <0  :"" ""&*&*T**:))..*2&TTr