PaperlayerView exact revision on arXiv

Source-linked AI summary

ViTPose++: Vision Transformer for Generic Body Pose Estimation

Yufei Xu, Jing Zhang, Qiming Zhang, Dacheng Tao

arXiv:2212.04246v3cs.CV

TL;DR

Body pose estimation needs models that handle varied body-keypoint tasks without elaborate, task-specific architectures. The paper proposes plain vision-transformer ViTPose and ViTPose++ with lightweight decoding, scalable model capacity, flexible training, and knowledge factorization. The models achieve state-of-the-art results across human, whole-body, and animal keypoint benchmarks, while the study remains limited to pose estimation and visual-modality training.

  • Problem

    Existing methods often use CNN backbones or specialized transformer structures and mainly address one body-keypoint type, leaving generic heterogeneous-task modeling underexplored.

  • Method

    ViTPose uses a plain non-hierarchical vision transformer with lightweight decoders, while ViTPose++ factorizes feed-forward networks into shared and task-specific components for multiple keypoint tasks.

  • Results

    ViTPose achieves 80.9 AP on MS COCO test-dev, while ViTPose++ sets state-of-the-art results across multiple human, whole-body, and animal keypoint benchmarks.

  • Takeaways & Limitations

    Plain vision transformers provide a simple, scalable, flexible, and transferable baseline for generic body keypoint detection.

  • Takeaways & Limitations

    The study focuses only on pose estimation and uses visual-modality training, leaving unified detection-tracking and language-assisted zero-shot generalization for future work.

Abstract

from arXiv · show

In this paper, we show the surprisingly good properties of plain vision transformers for body pose estimation from various aspects, namely simplicity in model structure, scalability in model size, flexibility in training paradigm, and transferability of knowledge between models, through a simple baseline model dubbed ViTPose. Specifically, ViTPose employs the plain and non-hierarchical vision transformer as an encoder to encode features and a lightweight decoder to decode body keypoints in either a top-down or a bottom-up manner. It can be scaled up from about 20M to 1B parameters by taking advantage of the scalable model capacity and high parallelism of the vision transformer, setting a new Pareto front for throughput and performance. Besides, ViTPose is very flexible regarding the attention type, input resolution, and pre-training and fine-tuning strategy. Based on the flexibility, a novel ViTPose+ model is proposed to deal with heterogeneous body keypoint categories in different types of body pose estimation tasks via knowledge factorization, i.e., adopting task-agnostic and task-specific feed-forward networks in the transformer. We also empirically demonstrate that the knowledge of large ViTPose models can be easily transferred to small ones via a simple knowledge token. Experimental results show that our ViTPose model outperforms representative methods on the challenging MS COCO Human Keypoint Detection benchmark at both top-down and bottom-up settings. Furthermore, our ViTPose+ model achieves state-of-the-art performance simultaneously on a series of body pose estimation tasks, including MS COCO, AI Challenger, OCHuman, MPII for human keypoint detection, COCO-Wholebody for whole-body keypoint detection, as well as AP-10K and APT-36K for animal keypoint detection, without sacrificing inference speed.

1 INTRODUCTION

Existing pose-estimation methods often rely on CNN backbones or specialized transformer designs and mainly target human pose. ViTPose uses a plain vision transformer with lightweight decoding, while ViTPose++ extends this approach to heterogeneous body-keypoint tasks.

  • Motivation: Existing transformer-based pose methods commonly retain CNN backbones or use pose-specific transformer structures, limiting exploration of plain transformers and cross-type keypoint modeling.The stated gap concerns both architectural simplicity and generic body keypoint detection.
  • Approach: ViTPose uses a plain, non-hierarchical vision transformer backbone for both top-down and bottom-up human pose estimation.The decoder and prediction design vary with the adopted estimation paradigm.
  • Results: 80.9 AP on MS COCO test-dev is achieved by ViTPose for human keypoint detection without elaborate task-specific structural designs.The result is reported for the human keypoint detection benchmark.
  • Properties: ViTPose combines structural simplicity, model-size scalability, training flexibility, and knowledge transferability.The model can use simple decoding, vary transformer depth and feature dimensions, adapt training settings, and transfer knowledge through a learnable token.
  • Contributions: ViTPose and ViTPose++ establish simple baseline models with state-of-the-art performance across representative human and generic body keypoint benchmarks.The paper positions these models as a step toward foundation models for generic body keypoint detection.

2 RELATED WORK

Related work spans CNN-based, transformer-based, top-down, bottom-up, and foundation-model-oriented pose estimation. ViTPose differs by using plain vision transformers across paradigms and by demonstrating scalable throughput-performance trade-offs and broader task coverage.

  • Transformer-based methods: Many prior transformer pose methods use CNN feature extractors or specialized transformer modules to model keypoint relationships.Examples include transformer encoders and decoders, token-based representations, and multi-resolution transformer modules.
  • Pose estimation paradigms: Top-down methods estimate poses from individual human instances, whereas bottom-up methods localize and associate keypoints for all people in an image.The related work includes regression, multi-branch, and association-based bottom-up designs.
  • Pre-training: Masked-image-modeling pre-training and smaller unlabeled pose datasets are explored as alternatives to fully supervised ImageNet pre-training for pose estimation.The paper focuses on plain vision transformers with MIM pre-training and tests whether ImageNet pre-training is necessary.
  • Foundation models: Vision transformers are increasingly used in foundation-model research because of their representation and scalability properties across diverse tasks and modalities.The related work connects these properties to unified modeling of detection, pose estimation, and other tasks.
  • ViTPose extension: 24M-parameter ViTPose-S reaches 73.8 AP at 1,432 fps on MS COCO, outperforming ResNet-50's 71.8 AP at 1,351 fps and matching HRFormer-S's 73.8 AP at 269 fps.Different ViTPose sizes form a new throughput-performance Pareto front, and the paper extends the model from top-down to bottom-up estimation.
  • Generic keypoint detection: ViTPose++ is evaluated across MS COCO, OCHuman, MPII, AI Challenger, COCO-Wholebody, AP-10K, and APT-36K for varied body keypoint tasks.The broader evaluation supports the paper's generic body keypoint detection direction.

3 VITPOSE

ViTPose uses a plain vision transformer backbone with lightweight decoder variants for keypoint heatmap regression, emphasizing simplicity, scalability, training flexibility, and knowledge transfer. ViTPose++ extends this design to heterogeneous body-keypoint tasks by factorizing shared and task-specific knowledge without added inference cost.

  • 3 VITPOSE: ViTPose uses a plain, non-hierarchical vision transformer to extract features and supports classic, simple, and minimal decoders for keypoint estimation.The model is compatible with top-down and bottom-up pose-estimation pipelines.
  • 3 VITPOSE: The backbone keeps spatial and channel dimensions constant across transformer layers, with each layer combining multi-head self-attention and an FFN.Features begin with patch embedding and are then processed by stacked transformer layers.
  • 3 VITPOSE: The simple decoder upsamples backbone features fourfold with bilinear interpolation before ReLU and 3×3 convolution, while the minimal decoder uses only a linear projection.These simplified designs exploit the representation capability of the vision-transformer backbone.
  • 3 VITPOSE: ViTPose’s structural simplicity improves computational parallelism and establishes a new inference-speed/performance Pareto front.The paper also links the simple structure to scalability as model capacity increases.
  • 3 VITPOSE: ViTPose trained only on pose data achieves competitive MS COCO performance, demonstrating flexibility in pre-training data.This avoids relying exclusively on ImageNet pre-training data.
  • 3 VITPOSE: ViTPose++ factorizes knowledge through shared and task-specific experts, and merges them during inference without extra parameters or computational cost.The shared expert uses all data, while each task-specific expert uses data for its corresponding pose-estimation task.

4 EXPERIMENTS

The experiments evaluate ViTPose and ViTPose++ across human, whole-body, and animal keypoint datasets using standard pose-estimation metrics and top-down or bottom-up settings.

  • Datasets: ViTPose is evaluated on MS COCO, while ViTPose++ is evaluated across human, whole-body, and animal pose datasets.The datasets include MS COCO, AIC, MPII, COCO-Wholebody, AP-10K, and APT-36K; Interhand2.6M is used for transfer-learning data-efficiency evaluation.
  • Metrics: Average precision is the primary metric for most datasets, with AP50 and AP75 providing loose and strict OKS-based evaluations.MPII uses PCKh instead of AP.
  • ViTPose: Top-down ViTPose uses a person detector to produce instances, then estimates keypoints for each detected person.This follows the standard top-down human pose-estimation setting.
  • ViTPose: ViTPose experiments use ViT-S, ViT-B, ViT-L, ViT-H, and ViTAE-G backbones with MAE initialization and MMPose-based training.Top-down models use 256×192 inputs, while bottom-up models use whole images resized to 512×512.
  • ViTPose++: ViTPose++ experiments use ViT-S, ViT-B, ViT-L, and ViT-H backbones initialized with MAE weights and trained with AdamW for 210 epochs.Training includes a 500-iteration linear warmup and learning-rate reductions at epochs 170 and 200.

4.3 Ablation studies of ViTPose and analysis

Ablations show that ViTPose’s plain transformer backbone supports simple decoders, scales with model size, adapts across training choices, and benefits from targeted attention and distillation strategies.

  • Structural simplicity and scalability: ViTPose’s simple decoder causes less than 0.3 AP loss, whereas the same simplification causes about 18 AP drops for ResNet-50 and ResNet-152.Performance also improves consistently as ViTPose model size increases.
  • Decoder analysis: ViTPose exceeds 75.4 AP even with simplified decoders using frozen projection layers, indicating that a linear layer can serve as the task-specific pose module.The evaluated decoder variants include classic-FP, simple, and minimal designs.
  • Pre-training data: Pose-domain pre-training improves data efficiency when the pre-training data match the fine-tuning task type.The comparison includes ImageNet-1k, MS COCO, and combined MS COCO plus AIC data, including cropped person instances.
  • Attention type: Window attention lowers memory use but drops performance from 77.4 AP to 66.4 AP; shifted and pooling windows recover about 10 AP with less than 10% memory increase.Combining both mechanisms reaches 76.8 AP with 22.9G memory versus 76.9 AP and 28.6G for the ViTDet-style setting.
  • Partially fine-tuning: Freezing MHSA reduces performance from 75.8 AP to 75.1 AP, while freezing FFN causes a 3.0 AP drop.The results associate FFN with task-specific modeling and MHSA with task-insensitive token relationships.
  • Transferability: Output distillation from ViTPose-L to ViTPose-B gains 0.5 AP, while token distillation gains 0.2 AP with a marginal extra memory cost.Output distillation incurs a moderate extra memory cost.

4.4 Ablation studies of ViTPose++ and analysis

ViTPose++ addresses conflicts among heterogeneous pose tasks by factorizing shared and task-specific knowledge within transformer feed-forward networks.

  • Baseline method: The multi-task baseline shares a backbone and uses an individual decoder for each pose-estimation task.Images from different tasks are sampled into batches, encoded jointly, and routed to task-specific decoders.
  • ViTPose++ variants: ViTPose++ varies feed-forward experts to separate independent task knowledge from shared knowledge across tasks.The evaluated variants include independent FFNs and independent-plus-shared FFNs.
  • Experimental setting: The multi-task comparison trains ViTPose-B and ViTPose++-B on combined MS COCO, AIC, and AP-10K data.Performance is reported on the MS COCO validation set, with inference parameter counts included for ViTPose++-B settings.
  • Results: ViTPose++ with PS-FFN reaches 77.0 AP without extra computations or parameters at partition ratio α=0.25, improving the trade-off over the multi-task baseline’s 76.8 AP.Independent FFN reaches 75.8 AP, while PS-FFN preserves common and task-specific knowledge.

4.5 Comparison with SOTA methods

ViTPose and ViTPose++ achieve strong performance across top-down and bottom-up MS COCO evaluations and multiple human, whole-body, and animal pose benchmarks. The results also show favorable throughput–accuracy trade-offs, scaling behavior, and robustness across datasets.

  • MS COCO: ViTPose++-H reaches 79.4 AP, improving ViTPose-H from 79.1 AP while preserving the benefits of larger-scale training.ViTPose++-S obtains 75.8 AP at 1,439 fps, comparable to ViTPose-B but substantially faster.
  • MS COCO: 80.9 AP on MS COCO test-dev makes ViTPose-G outperform UDP++ at 80.8 AP, while a three-model ensemble reaches 81.1 AP.ViTPose-G uses the ViTAE-G backbone with about 1B parameters and a larger 576×432 input resolution.
  • MS COCO: 70.1 AP with ViTPose-L surpasses HigherHRNet’s 68.6 AP in bottom-up MS COCO evaluation.ViTPose-B obtains 68.5 AP, demonstrating performance scaling within the plain vision-transformer design.
  • Other datasets: ViTPose++ gains over prior methods on OCHuman, including more than 10 AP on its validation set despite lacking specialized occlusion modules.The evaluation uses ground-truth boxes to isolate pose-estimation performance from person-detection errors.
  • Other datasets: ViTPose++-H obtains 34.8 AP on AIC, exceeding HRNet-w48 at 33.5 AP and HRFormer-B at 34.4 AP.The authors note that the AIC score remains limited, indicating room for further improvement.
  • Other datasets: ViTPose++-B obtains 74.5 AP on AP-10K, while ViTPose++-S reaches 74.2 AP on APT-36K versus 71.4 AP on AP-10K.The APT-36K comparison is attributed to its more balanced data distribution.

4.6 Subjective results

ViTPose++-H is evaluated visually across human, whole-body, and animal pose datasets. The examples include challenging occlusions, blur, scale changes, appearance variation, unusual postures, and complex backgrounds.

  • Subjective results: ViTPose++-H produces visual pose estimates across seven datasets, including MS COCO, AIC, OCHuman, MPII, COCO-W, AP-10K, and APT-36K.The displayed cases cover occlusions, blur, scale changes, appearance variance, odd postures, and complex backgrounds.

4.7 Data efficiency analysis

ViTPose’s transfer performance improves with model scale and multi-task training, enabling strong results after fine-tuning with limited labeled data. Large models also outperform smaller baselines under reduced-data settings.

  • InterHand2.6M: ViTPose-H consistently outperforms smaller ViTPose models on InterHand2.6M at the same training-data percentage.Increasing model size improves data efficiency across the evaluated transfer-learning settings.
  • InterHand2.6M: Multi-task training improves 10%-data InterHand2.6M transfer by 0.4, 0.3, and 0.2 AUC for ViTPose-B, ViTPose-L, and ViTPose-H, respectively.The comparison is against single-task training.
  • InterHand2.6M: From 10% to 100% transfer data, ViTPose-B improves 1.3 AUC, compared with 0.8 AUC for SimpleBaseline with ResNet-50.The result indicates larger gains for the transformer-based model in this comparison.
  • AP-10K: With only 10% AP-10K training data, ViTPose-H outperforms ResNet-50, HRNet-32, HRNet-48, ViTPose-B, and ViTPose-S.This experiment evaluates the data efficiency of the largest ViTPose model against smaller models.

4.8 Visualization and analysis

The analyses examine ViTPose++ representations, attention behavior, transfer-learning figures, and task-specific parameter relationships. The visualizations suggest progressive body-focused representations, stable attention under masking, and greater task specificity in deeper layers.

  • Visualization of feature maps: ViTPose++-B progressively focuses on the human body from shallow to deep layers and can infer masked-head keypoints from visible regions.The 9th- and 12th-layer feature maps retain information useful for localizing masked keypoints.
  • Visualization of attention maps: Deep-layer attention to the right wrist remains similar whether the wrist is visible or masked.The comparison uses attention maps from the 3rd, 6th, 9th, and 12th layers.
  • Experimental visualizations: Figures 5–7 compare ViTPose++ results across seven datasets and transfer-learning performance under different training-data percentages.Figure 6 separates multi-task and single-task training for ViTPose-B, ViTPose-L, and ViTPose-H, while Figure 7 focuses on ViTPose-H on AP-10K.
  • Task correlation: Task-specific FFN weights are more similar between MS COCO and AIC than between MS COCO and AP-10K, especially in deeper layers.The analysis measures cosine similarity and finds shallower-layer FFNs more similar across tasks than deeper-layer FFNs.

4.9 Failure case analysis

ViTPose still faces challenges on extreme postures such as skiing. Larger backbones and more varied training data can mitigate these limitations.

  • Extreme postures, including skiing, remain challenging for ViTPose.
  • Larger backbones and expanded pose data can mitigate these challenges.
  • ViTPose++-H correctly estimates the pose after training on a dataset with greater pose variety.

5 LIMITATIONS AND DISCUSSION

The discussion identifies remaining room for improvement and examines feature and attention visualizations alongside task-specific FFN-weight similarities. It also notes that the study covers pose estimation but not broader unified body understanding or language-assisted generalization.

  • 5 LIMITATIONS AND DISCUSSION: The models retain room for improvement despite their good properties and performance.
  • 5 LIMITATIONS AND DISCUSSION: Feature maps and attention maps are visualized for ViTPose++-B with and without manual masks.
  • 5 LIMITATIONS AND DISCUSSION: Task-specific FFN-weight similarity is examined for ViTPose++-B between COCO and AIC, and between COCO and AP-10K.
  • 5 LIMITATIONS AND DISCUSSION: The study focuses on pose estimation using visual training data, leaving unified detection or tracking and language-assisted zero-shot generalization for future exploration.

6 CONCLUSION

The paper presents ViTPose and ViTPose++ as simple baselines with strong properties across body pose estimation benchmarks. The models set new performance records on the listed datasets.

  • 6 CONCLUSION: ViTPose and ViTPose++ demonstrate simplicity, scalability, flexibility, and transferability across representative body pose benchmarks.
  • 6 CONCLUSION: The proposed models set new performance records on MS COCO, AIC, MPII, OCHuman, COCO-W, AP-10K, and APT-36K.
Loading 2212.04246v3…