qik/ d Z ddlZddlmZ ddlmZ ddlmZ ddl m
Z
ddl m Z dd l
mZmZ dd
lmZ d d lmZmZ d d
lmZmZmZ d dlmZmZm Z m Z m Z m Z m Z m!Z! G d de Z" G d de Z# G d de Z$ G d de Z% G d de Z& G d de Z' G d d e Z( G d d e Z) G d d e Z*e G d! d"e Z+ e d#$ G d% d&e! Z, G d' d(e Z-g d)Z.y)*zPyTorch SAM 2 model. N )initialization)PreTrainedConfig)PreTrainedModel)Unpack)auto_docstring)TransformersKwargsmerge_with_config_defaults)capture_outputs )CONFIG_MAPPING
AutoConfig)
Sam2ConfigSam2MaskDecoderConfigSam2PromptEncoderConfig)
Sam2AttentionSam2FeedForward
Sam2LayerNorm Sam2ModelSam2PreTrainedModelSam2TwoWayAttentionBlockSam2VisionEncoderOutputSam2VisionModelc L e Zd ZdZdZdZdeiZ d fd Z xZ S )EdgeTamVisionConfiga
This is the configuration class to store the configuration of a [`EdgeTamVisionModel`]. It is used to instantiate a SAM
vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration
defaults will yield a similar configuration to that of SAM 2.1 Hiera-tiny
[facebook/EdgeTAM](https://huggingface.co/facebook/EdgeTAM) architecture.
Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
documentation from [`PreTrainedConfig`] for more information.
Args:
backbone_config (`Union[dict, "PreTrainedConfig"]`, *optional*, defaults to `timm/repvit_m1.dist_in1k`):
Configuration for the vision backbone. This is used to instantiate the backbone using
`AutoModel.from_config`.
backbone_channel_list (`List[int]`, *optional*, defaults to `[384, 192, 96, 48]`):
The list of channel dimensions for the backbone.
backbone_feature_sizes (`List[List[int]]`, *optional*, defaults to `[[256, 256], [128, 128], [64, 64]]`):
The spatial sizes of the feature maps from the backbone.
fpn_hidden_size (`int`, *optional*, defaults to 256):
The hidden dimension of the FPN.
fpn_kernel_size (`int`, *optional*, defaults to 1):
The kernel size for the convolutions in the neck.
fpn_stride (`int`, *optional*, defaults to 1):
The stride for the convolutions in the neck.
fpn_padding (`int`, *optional*, defaults to 0):
The padding for the convolutions in the neck.
fpn_top_down_levels (`List[int]`, *optional*, defaults to `[2, 3]`):
The levels for the top-down FPN connections.
num_feature_levels (`int`, *optional*, defaults to 3):
The number of feature levels from the FPN to use.
hidden_act (`str`, *optional*, defaults to `"gelu"`):
The non-linear activation function in the neck.
layer_norm_eps (`float`, *optional*, defaults to 1e-06):
The epsilon for the layer normalization.
initializer_range (`float`, *optional*, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
vision_configedgetam_vision_modelbackbone_configc
|g dn|}| ddgddgddggn|}|ddgn|}t |t r'|j dd |d< t |d di |}n | t j
d dd
g d d
}|| _ || _ || _ || _ || _
|| _ || _ || _
| | _ |
| _ | | _ | | _ t% | L di |
y )N)i ` 0 @ r r
model_type timm_wrapperztimm/repvit_m1.dist_in1kT)r r r )in_chans
features_only out_indices)
model_args )
isinstancedictgetr
r from_pretrainedr backbone_channel_listbackbone_feature_sizesfpn_hidden_sizefpn_kernel_size
fpn_stride fpn_paddingfpn_top_down_levelsnum_feature_levels
hidden_actlayer_norm_epsinitializer_rangesuper__init__)selfr r2 r3 r4 r5 r6 r7 r8 r9 r: r; r< kwargs __class__s ]/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/edgetam/modular_edgetam.pyr> z EdgeTamVisionConfig.__init__V s 7L6S 2Yn 2H2Pc3Z#sb"X .Vl )<(Cq!fI\ ot ,,;,?,? n,]OL ) ,_\-J K ^o ^O
$ ( 8 8*()DQ]^ O
/ &;"&<# . . $ "4 $ ,!2
"6" ) NNNr# r( r( r Nr gelugư>g{Gz?)
__name__
__module__ __qualname____doc__base_config_keyr& r sub_configsr>
__classcell__)rA s @rB r r ) sQ $L &O'J:K " # .# .#rC r c e Zd Zy)EdgeTamPromptEncoderConfigNrE rF rG r- rC rB rM rM rC rM c e Zd Zy)EdgeTamMaskDecoderConfigNrN r- rC rB rQ rQ rO rC rQ c e Zd ZdZy)
EdgeTamConfiga
[`EdgeTamConfig`] is the configuration class to store the configuration of a [`EdgeTamModel`]. It is used to instantiate a
EDGETAM model according to the specified arguments, defining the memory attention, memory encoder, and image encoder
configs. Instantiating a configuration defaults will yield a similar configuration to that of the SAM 2.1 Hiera-tiny
[facebook/edgetam.1-hiera-tiny](https://huggingface.co/facebook/edgetam.1-hiera-tiny) architecture.
Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
documentation from [`PreTrainedConfig`] for more information.
EdgeTAM checkpoints with `model_type="edgetam_video"` are compatible with `EdgeTamModel` since the video variant
weights are a superset of the image-only model weights. You may see a warning about model type mismatch when
loading such checkpoints, which can be safely ignored in this case.
Args:
vision_config (Union[`dict`, `EdgeTamVisionConfig`], *optional*):
Dictionary of configuration options used to initialize [`EdgeTamVisionConfig`].
prompt_encoder_config (Union[`dict`, `EdgeTamPromptEncoderConfig`], *optional*):
Dictionary of configuration options used to initialize [`EdgeTamPromptEncoderConfig`].
mask_decoder_config (Union[`dict`, `EdgeTamMaskDecoderConfig`], *optional*):
Dictionary of configuration options used to initialize [`EdgeTamMaskDecoderConfig`].
initializer_range (`float`, *optional*, defaults to 0.02):
Standard deviation for parameter initialization.
Example:
```python
>>> from transformers import (
... EdgeTamVisionConfig,
... EdgeTamPromptEncoderConfig,
... EdgeTamMaskDecoderConfig,
... EdgeTamModel,
... )
>>> # Initializing a EdgeTamConfig with `"facebook/edgetam.1_hiera_tiny"` style configuration
>>> configuration = EdgeTamConfig()
>>> # Initializing a EdgeTamModel (with random weights) from the `"facebook/edgetam.1_hiera_tiny"` style configuration
>>> model = EdgeTamModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
>>> # We can also initialize a EdgeTamConfig from a EdgeTamVisionConfig, EdgeTamPromptEncoderConfig, and EdgeTamMaskDecoderConfig
>>> # Initializing EDGETAM vision encoder, memory attention, and memory encoder configurations
>>> vision_config = EdgeTamVisionConfig()
>>> prompt_encoder_config = EdgeTamPromptEncoderConfig()
>>> mask_decoder_config = EdgeTamMaskDecoderConfig()
>>> config = EdgeTamConfig(vision_config, prompt_encoder_config, mask_decoder_config)
```
N)rE rF rG rH r- rC rB rS rS s
6p rC rS c e Zd Zy)EdgeTamLayerNormNrN r- rC rB rU rU rO rC rU c e Zd Zy)EdgeTamVisionEncoderOutputNrN r- rC rB rW rW rO rC rW c e Zd Zy)EdgeTamAttentionNrN r- rC rB rY rY rO rC rY c e Zd Zy)EdgeTamTwoWayAttentionBlockNrN r- rC rB r[ r[ rO rC r[ c e Zd Zy)EdgeTamFeedForwardNrN r- rC rB r] r] rO rC r] c > e Zd ZdZ ej
d Zy)EdgeTamPreTrainedModelNc t j | | t |t r-|j t j |j y y t |d r,t j |j |j y y )Npositional_embedding)std) r
_init_weightsr. EdgeTamModelno_memory_embeddinginitzeros_hasattrnormal_ra scale)r? modules rB rc z$EdgeTamPreTrainedModel._init_weights sg %%dF3 fl +))5 F 6 67 6
V 3
4 LL44&,, G 5rC )rE rF rG "_keys_to_ignore_on_load_unexpectedtorchno_gradrc r- rC rB r_ r_ s$ )-&U]]_H HrC r_ zN
The vision model from EdgeTAM without any head or projection on top.
) custom_introc
p e Zd ZeZdZi Zd Zee dde
j dz de e
deez fd Zy)EdgeTamVisionModel pixel_valuesc t d Nz2Can't get input embeddings from timm wrapper modelNotImplementedErrorr? s rB get_input_embeddingsz'EdgeTamVisionModel.get_input_embeddings !"VWWrC Nr@ returnc ^ | t d | j |fi |}|j }|D cg c] }|j dddd }}| j | \ }}|| j
d d d d }|| j
d d d d }t
|d |||j S c c}w )Nz You have to specify pixel_valuesr r r r( )last_hidden_statefpn_hidden_statesfpn_position_encoding
hidden_states)
ValueErrorbackboner} permuteneckr9 rW r )r? rr r@ backbone_outputintermediate_hidden_states hidden_stater~ r s rB forwardzEdgeTamVisionModel.forward s ?@ @ ($-- ??%4%F%F"[u%vr s & 3 - & L 5 - ` ` [#* [#| !8 4 9 J 9 x } !8 } ":
H 0
H
H
#
#
#
L
X9
X rC