Source-linked AI summary
ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
Yufei Xu, Jing Zhang, Qiming Zhang, Dacheng Tao
TL;DR
Pose estimation had seen limited investigation of whether simple plain vision transformers could serve effectively without specialized structures. ViTPose uses a plain transformer backbone and lightweight decoder, and demonstrates strong performance alongside simplicity, scalability, flexibility, and transferability. The paper reports state-of-the-art MS COCO performance, while noting that more advanced decoder and feature-pyramid technologies remain unexplored.
Problem
Existing pose-estimation transformer methods often use CNN feature extractors or elaborate transformer designs, leaving the potential of plain vision transformers insufficiently explored.
Method
ViTPose uses plain, non-hierarchical vision transformers as backbones and a lightweight decoder that upsamples features and predicts keypoint heatmaps.
Results
ViTPose demonstrates simplicity, scalability, flexibility, and transferability, while a large-backbone model achieves the best 80.9 AP on the MS COCO Keypoint test-dev set.
Takeaways & Limitations
ViTPose provides a strong baseline for vision-transformer-based pose estimation and offers insights for exploring plain vision transformers in other computer-vision tasks.
Takeaways & Limitations
The potential of ViTPose with advanced technologies such as complex decoders or FPN structures is not fully explored, and further work is suggested on prompt-based tuning and other pose datasets.
Abstract
from arXiv · showhide
Although no specific domain knowledge is considered in the design, plain vision transformers have shown excellent performance in visual recognition tasks. However, little effort has been made to reveal the potential of such simple structures for pose estimation tasks. In this paper, we show the surprisingly good capabilities of plain vision transformers for pose estimation from various aspects, namely simplicity in model structure, scalability in model size, flexibility in training paradigm, and transferability of knowledge between models, through a simple baseline model called ViTPose. Specifically, ViTPose employs plain and non-hierarchical vision transformers as backbones to extract features for a given person instance and a lightweight decoder for pose estimation. It can be scaled up from 100M to 1B parameters by taking the advantages of the scalable model capacity and high parallelism of transformers, setting a new Pareto front between throughput and performance. Besides, ViTPose is very flexible regarding the attention type, input resolution, pre-training and finetuning strategy, as well as dealing with multiple pose tasks. We also empirically demonstrate that the knowledge of large ViTPose models can be easily transferred to small ones via a simple knowledge token. Experimental results show that our basic ViTPose model outperforms representative methods on the challenging MS COCO Keypoint Detection benchmark, while the largest model sets a new state-of-the-art. The code and models are available at https://github.com/ViTAE-Transformer/ViTPose.
1 Introduction
ViTPose asks how well a plain vision transformer can perform human pose estimation without extra CNN feature extractors or elaborate transformer designs. It answers with a simple baseline that demonstrates strong benchmark performance and capabilities across simplicity, scalability, flexibility, and transferability.
- Human pose estimation localizes anatomical keypoints but remains challenging because of occlusion, truncation, scale, and appearance variation.
- Existing transformer-based pose methods either retain CNN backbones or require carefully designed transformer structures, motivating evaluation of plain vision transformers.
- ViTPose combines plain, non-hierarchical vision-transformer backbones with a lightweight decoder that upsamples features and regresses keypoint heatmaps.
- 80.9 AP on the MS COCO Keypoint test-dev set establishes ViTPose as a state-of-the-art result despite its lack of elaborate structural designs.
- ViTPose demonstrates structural simplicity, model-size scalability, training flexibility, and knowledge transferability, while its simple structure supports a favorable throughput-performance trade-off.
2 Related Work
Related work has progressed from CNN-based pose estimation toward transformer-based feature extraction and refinement. ViTPose builds on plain vision-transformer backbones, including those initialized through masked image modeling.
- Earlier pose-estimation methods commonly used transformers as decoders or refiners over CNN-extracted features, including cascade and token-based designs.
- HRFormer removes CNN feature extraction by using transformers to directly produce high-resolution features.
- Plain vision-transformer backbones have been trained with supervised ImageNet-1K learning and, more recently, self-supervised masked image modeling pretext tasks.
3 ViTPose
ViTPose uses a plain, non-hierarchical transformer backbone with lightweight decoders for keypoint heatmap estimation. The design emphasizes structural simplicity, scalability, flexibility across settings, and knowledge transfer from larger to smaller models.
- Framework: ViTPose processes person-instance features with transformer layers whose spatial and channel dimensions remain constant, followed by lightweight keypoint decoders.Each transformer layer combines multi-head self-attention and a feed-forward network; the initial feature is produced by patch embedding.
- Decoder design: The classic decoder uses two deconvolution blocks with batch normalization and ReLU, followed by a 1 × 1 convolution producing keypoint heatmaps.Each deconvolution block upsamples feature maps by 2 times.
- Decoder design: The simpler decoder upsamples feature maps 4 times with bilinear interpolation, then applies ReLU and a 3 × 3 convolution to produce heatmaps.Despite less non-linear capacity, it achieves competitive performance against the classic decoder and carefully designed transformer-based decoders.
- Scalability: ViTPose supports model-size control by stacking different numbers of transformer layers and changing feature dimensions, enabling deployment choices along a throughput–performance Pareto front.This scalability allows the framework to benefit from scalable pre-trained vision transformers without major modifications to other components.
- Flexibility: The framework is flexible across pre-training data, input and feature resolution, attention type, finetuning, and multiple pose datasets.With shared backbone features and dataset-specific decoders, multiple pose datasets can be handled without much extra cost; freezing MHSA can retain comparable performance to full finetuning.
- Knowledge transfer: Token-based distillation transfers knowledge from a frozen teacher to a student by learning a knowledge token, freezing it, and concatenating it with student visual tokens during training.The teacher token is optimized using ground-truth heatmaps before being used in student training.
4 Experiments
Experiments show that ViTPose remains effective across simplified decoders, model scales, pre-training data, resolutions, attention mechanisms, partial finetuning, multi-dataset training, and knowledge transfer. On MS COCO, larger ViTPose models achieve strong accuracy-throughput trade-offs and the 1B-parameter ViTPose-G reaches 80.9 AP on test-dev.
- Structure simplicity and scalability: ViTPose loses less than 0.3 AP with its simple decoder, whereas ResNet-50 and ResNet-152 lose about 18 AP under the same simplification.The result supports the claim that plain vision-transformer backbones provide strong representations without complex decoders.
- Structure simplicity and scalability: ViTPose performance improves consistently as model size increases, demonstrating scalability across ViT-B, ViT-L, and ViT-H backbones.The models use MAE-pre-trained backbones and are evaluated on the MS COCO keypoint validation set.
- Attention type: Window attention reduces memory but drops performance from 77.4 AP to 66.4 AP; shifted and pooled windows restore global information, reaching 76.8 AP with 22.9G memory.Increasing the window size further raises performance to 77.1 AP, while the joint full-and-window setting reaches 76.9 AP with 28.6G memory.
- Finetuning flexibility: Freezing MHSA reduces performance from 75.8 AP to 75.1 AP, while freezing FFN causes a 3.0 AP drop, indicating stronger task-specific modeling in FFN.AP50 remains almost unchanged when MHSA is frozen, supporting greater task-agnostic behavior for MHSA.
- Transferability: Token-based distillation adds 0.2 AP and output distillation adds 0.5 AP when transferring ViTPose-L knowledge to ViTPose-B; together they reach 76.6 AP.The complementary gains support transferability between ViTPose model sizes.
- Comparison with state of the art: ViTPose-L reaches 78.3 AP and 83.5 AR on the validation set, while ViTPose-H reaches 79.1 AP at 241 fps versus HRFormer-B’s 75.6 AP at 158 fps.These comparisons indicate a favorable throughput-accuracy trade-off for larger ViTPose backbones.
- Comparison with state of the art: ViTPose-G, using a 1B-parameter ViTAE-G backbone, reaches 80.9 AP on COCO test-dev, exceeding UDP++ at 80.8 AP; three-model ensembling reaches 81.1 AP.ViTPose-G uses 576 × 432 inputs, MS COCO and AI Challenger training data, and stronger person detections.
5 Limitation and Discussion
ViTPose achieves strong pose-estimation performance with a simple vision-transformer baseline, but its potential with more advanced technologies and broader tasks remains unexplored.
- Advanced decoders or FPN structures may further improve ViTPose’s performance.
- Further research could explore prompt-based tuning to demonstrate ViTPose’s flexibility more fully.
- The paper suggests applying ViTPose to animal pose estimation and face keypoint detection as future work.
6 Conclusion
The paper presents ViTPose as a simple, effective baseline for vision-transformer-based human pose estimation. Extensive experiments support its capabilities, including a best 80.9 AP on MS COCO test-dev with a large ViTAE-G backbone.
- ViTPose provides a simple yet effective baseline without elaborate structural designs or complex frameworks.
- ViTPose demonstrates simplicity, scalability, flexibility, and transferability across vision-transformer-based pose-estimation tasks.
- 80.9 AP is achieved on the MS COCO test-dev set by a single ViTPose model with a large ViTAE-G backbone.
- The authors hope ViTPose will inspire further study of plain vision transformers in additional computer-vision tasks.
A Additional results of multi-dataset training
Additional experiments evaluate ViTPose under multi-dataset training across OCHuman, MPII, and AI Challenger. The models perform strongly across these benchmarks, including new reported best results on MPII and AI Challenger.
- ViTPose variants are trained with multi-dataset training and tested directly without dataset-specific finetuning.
- OCHuman evaluation uses ground-truth bounding boxes to measure performance on heavily occluded people without extra human detectors.
- 94.3 PCKh is achieved by ViTPose-G on the MPII val set, setting a new state-of-the-art with a larger backbone and 576x432 input resolution.
- 35.4 AP from ViTPose-H exceeds 33.5 AP from HRNet-w48 and 34.4 AP from HRFormer base on AI Challenger.
- 43.2 AP is achieved by ViTPose-G on AI Challenger, although the reported precision remains insufficient for the dataset.
B Detailed dataset details.
The experiments use MS COCO, AI Challenger, MPII, and CrowdPose for training and evaluation, while OCHuman is reserved for evaluating performance on occluded people.
- MS COCO, AI Challenger, MPII, and CrowdPose are used for training and evaluation, whereas OCHuman is used only for evaluation.OCHuman measures performance on occluded people.
- MS COCO contains 118K training images and 150K human instances with up to 17 keypoint annotations per instance.
- The MS COCO dataset is released under the CC-BY-4.0 license, while MPII is released under the BSD license.
C Subjective results
ViTPose visual results are shown for AI Challenger, OCHuman, and MPII, including challenging cases such as occlusion, blur, appearance variation, and irregular postures.
- The results include challenging cases involving occlusion, blur, appearance variance, and irregular body postures.
- Visual pose estimation results are presented on the AI Challenger dataset.
- Visual pose estimation results are presented on the OCHuman dataset.
- Visual pose estimation results are presented on the MPII dataset.