qi2dZddlZddlmZdZddZddZy) z NEFTune: Noisy Embeddings for Fine-Tuning. Implementation based on https://github.com/neelsjain/NEFTune Paper: https://huggingface.co/papers/2310.05914 N)_is_peft_modelc |jrtj|jd|jdz}|jtj |z }|tj |j| |z}|S)a Implements the NEFTune forward pass for the model using forward hooks. Note this works only for torch.nn.Embedding layers. This method is slightly adapted from the original source code that can be found here: https://github.com/neelsjain/NEFTune. Simply add it to your model as follows: ```python from transformers.integrations.neftune import neftune_post_forward_hook model = ... model.embed_tokens.neftune_noise_alpha = 0.1 model.embed_tokens.register_forward_hook(neftune_post_forward_hook) ``` Args: module (`torch.nn.Module`): The embedding module where the hook is attached. Note that you need to set `module.neftune_noise_alpha` to the desired noise alpha value. input (`torch.Tensor`): The input tensor to the model. output (`torch.Tensor`): The output tensor of the model (i.e. the embeddings). r)trainingtorchtensorsizeneftune_noise_alphasqrt zeros_likeuniform_)moduleinputoutputdimsmag_norms S/opt/pipecat/venv/lib/python3.12/site-packages/transformers/integrations/neftune.pyneftune_post_forward_hookrsp*||FKKNV[[^;<-- 40@@%**62;;XIxPP Mc||j|}n|}t|r%|jjj }n|j }||_|j t}|S)a Activates NEFTune (Noisy Embeddings for Fine-Tuning) on the model. NEFTune adds noise to embedding vectors during training, which has been shown to improve fine-tuning performance. See https://huggingface.co/papers/2310.05914 for details. Args: model (`torch.nn.Module`): The model to activate NEFTune on. neftune_noise_alpha (`float`): The noise alpha value controlling the magnitude of the noise. accelerator (`Accelerator`, *optional*): The accelerator instance. If provided, the model will be unwrapped before accessing embeddings. Required when using distributed training. Returns: `torch.utils.hooks.RemovableHandle`: The hook handle that can be used to deactivate NEFTune. ) unwrap_modelr base_modelmodelget_input_embeddingsr register_forward_hookr)rr acceleratorunwrapped_model embeddings hook_handles ractivate_neftuner!6sn&%2259o&$//55JJL $99; %8J"223LMK rc||j|}n|}t|r%|jjj }n|j }|j t |dr|`yy)a Deactivates NEFTune on the model. Args: model (`torch.nn.Module`): The model to deactivate NEFTune on. hook_handle (`torch.utils.hooks.RemovableHandle`): The hook handle returned by `activate_neftune`. accelerator (`Accelerator`, *optional*): The accelerator instance. If provided, the model will be unwrapped before accessing embeddings. Nr )rrrrrremovehasattrr )rr rrrs rdeactivate_neftuner%Ysq%2259o&$//55JJL $99; z01  *2r)N)__doc__r trainer_utilsrrr!r%rrr)s" *8 F+r