qi,dZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z dd lmZdd lmZmZmZmZdd lmZd dlmZej2eZeedGddeZeGdde ZddZGddej>Z Gddej>Z!edGddeZ"ddgZ#y)zPyTorch VitPose model.) dataclassN)nn)initialization) load_backbone)BackboneOutput)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuple) VitPoseConfigz6 Class for outputs of pose estimation models. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<y)VitPoseEstimatorOutputaH loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Loss is not supported at this moment. See https://github.com/ViTAE-Transformer/ViTPose/tree/main/mmpose/models/losses for further detail. heatmaps (`torch.FloatTensor` of shape `(batch_size, num_keypoints, height, width)`): Heatmaps as predicted by the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage. Nlossheatmaps. hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrtupler^/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/vitpose/modeling_vitpose.pyrr$sq &*D%  d "))-He$&-:>M5**C/047>7;Je'',-4;r"rceZdZUeed<dZdZdZdZe jde je jze jzfdZy) VitPosePreTrainedModelconfigvit pixel_values)imageTmodulect|tjtjfrct j |j d|jj|j t j|jyyt|tjr?t j|jt j|j yy)zInitialize the weightsg)meanstdN) isinstancerLinearConv2dinit trunc_normal_weightr&initializer_rangebiaszeros_ LayerNormones_)selfr*s r# _init_weightsz$VitPosePreTrainedModel._init_weightsDs fryy"))4 5   v}}3DKKQ ]b]i]i r"rYcheZdZdZdeffd ZddejdejdzfdZxZ S) VitPoseClassicDecoderz Classic decoding head consisting of a 2 deconvolutional blocks, followed by a 1x1 convolution layer, turning the feature maps into heatmaps. r&ct|tj|jj ddddd|_tjd|_tj|_ tjdddddd|_ tjd|_ tj|_ tjd|jddd|_y) NrCrF)r`rarbr5rr_)rcrdrConvTranspose2drirjdeconv1 BatchNorm2d batchnorm1rerelu1deconv2 batchnorm2relu2r0rkrlrms r#rdzVitPoseClassicDecoder.__init__s ))  " " . .1VW^c  ..-WWY ))#s!UV]bc ..-WWY IIc6#4#4!AWXY r"NrorMc|j|}|j|}|j|}|j|}|j |}|j |}|j |}| t||}|Srr)rrrrrrrlrWrss r#rtzVitPoseClassicDecoder.forwards{||L1 |4 zz,/ ||L1 |4 zz,/ 99\*  ! :6Hr"rrrurxs@r#rzrzs6 Z} ZELLellT>Qr"rzz? The VitPose model with a pose estimation head on top. ceZdZdeffd Zee d dejdejdzdejdzdejdzde e d e f d Z xZ S) VitPoseForPoseEstimationr&ct||t||_t |jj ds t dt |jj ds t dt |jj ds t d|jr t|n t||_ |jy)Nrjz0The backbone should have a hidden_size attribute image_sizez0The backbone should have an image_size attribute patch_sizez/The backbone should have a patch_size attribute) rcrdrbackbonehasattrr&rEuse_simple_decoderrYrzhead post_initrms r#rdz!VitPoseForPoseEstimation.__init__s  %f- t}}++];OP Pt}}++\:OP Pt}}++\:NO O4:4M4M(0ShioSp  r"Nr( dataset_indexrMlabelskwargsrpc d}| td|jj|fd|i|}|jd}|jd} |j j jd|j j jdz} |j j jd|j j jdz} |jddd}|j| d| | j}|j||} t|| |j|j S) a dataset_index (`torch.Tensor` of shape `(batch_size,)`): Index to use in the Mixture-of-Experts (MoE) blocks of the backbone. This corresponds to the dataset index used during training, e.g. For the single dataset index 0 refers to the corresponding dataset. For the multiple datasets index 0 refers to dataset A (e.g. MPII) and index 1 refers to dataset B (e.g. CrowdPose). flip_pairs (`torch.tensor`, *optional*): Whether to mirror pairs of keypoints (for example, left ear -- right ear). Examples: ```python >>> from transformers import AutoImageProcessor, VitPoseForPoseEstimation >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> processor = AutoImageProcessor.from_pretrained("usyd-community/vitpose-base-simple") >>> model = VitPoseForPoseEstimation.from_pretrained("usyd-community/vitpose-base-simple") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> boxes = [[[412.8, 157.61, 53.05, 138.01], [384.43, 172.21, 15.12, 35.74]]] >>> inputs = processor(image, boxes=boxes, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> heatmaps = outputs.heatmaps ```NzTraining is not yet supportedrrDrrr})rM)rrrr)NotImplementedErrorrforward_with_filtered_kwargs feature_mapsrGr&rirrpermuterH contiguousrrrr) r9r(rrMrrroutputssequence_outputrO patch_height patch_widthrs r#rtz VitPoseForPoseEstimation.forwards>R  %&EF F"L$--"L"L # '# # "..r2$**1- {{22==a@DKKD_D_DjDjklDmm kk11<rs! &+.-&MM-0   H %   <[<  <$&_&&&%P2996#BII#L U 5U  U p $%? @r"