Source-linked AI summary
Effective Whole-body Pose Estimation with Two-stages Distillation
Zhendong Yang, Ailing Zeng, Chun Yuan, Yu Li
TL;DR
Whole-body pose estimation needs efficient, accurate localization across body parts despite fine-grained regions and limited annotations. DWPose uses two-stage distillation and UBody data, raising RTMPose-l from 64.8% to 66.5% whole-body AP and surpassing the RTMPose-x teacher at 65.3% AP.
Problem
Whole-body pose estimation must localize body, hand, face, and foot keypoints across scales, while existing annotations are limited for fine-grained regions.
Method
DWPose combines first-stage teacher feature-and-logit distillation, second-stage student head self-distillation, and UBody training data.
Results
66.5% whole-body AP improves RTMPose-l from 64.8% and surpasses the RTMPose-x teacher at 65.3% AP.
Takeaways & Limitations
DWPose provides real-time whole-body pose estimators with different model sizes and supports downstream human-centric applications.
Abstract
from arXiv · showhide
Whole-body pose estimation localizes the human body, hand, face, and foot keypoints in an image. This task is challenging due to multi-scale body parts, fine-grained localization for low-resolution regions, and data scarcity. Meanwhile, applying a highly efficient and accurate pose estimator to widely human-centric understanding and generation tasks is urgent. In this work, we present a two-stage pose \textbf{D}istillation for \textbf{W}hole-body \textbf{P}ose estimators, named \textbf{DWPose}, to improve their effectiveness and efficiency. The first-stage distillation designs a weight-decay strategy while utilizing a teacher's intermediate feature and final logits with both visible and invisible keypoints to supervise the student from scratch. The second stage distills the student model itself to further improve performance. Different from the previous self-knowledge distillation, this stage finetunes the student's head with only 20% training time as a plug-and-play training strategy. For data limitations, we explore the UBody dataset that contains diverse facial expressions and hand gestures for real-life applications. Comprehensive experiments show the superiority of our proposed simple yet effective methods. We achieve new state-of-the-art performance on COCO-WholeBody, significantly boosting the whole-body AP of RTMPose-l from 64.8% to 66.5%, even surpassing RTMPose-x teacher with 65.3% AP. We release a series of models with different sizes, from tiny to large, for satisfying various downstream tasks. Our codes and models are available at https://github.com/IDEA-Research/DWPose.
1. Introduction
The paper targets efficient and precise whole-body pose estimation, motivated by performance limitations, data scarcity, and the needs of human-centric applications. It proposes two-stage distillation and broader training data, achieving improved RTMPose performance.
- Whole-body pose estimation supports human-centric perception, understanding, generation, and virtual content applications, but existing tools remain performance-limited.
- Limited holistic annotations impair localization of fine-grained fingers and face landmarks across different body-part scales.
- DWPose introduces two-stage pose knowledge distillation for efficient and precise whole-body pose estimation.The first stage uses teacher intermediate features and logits, while the second stage applies head-aware self-distillation.
- UBody adds comprehensive training data with diverse hand gestures and facial expressions from real-life scenes.
- 66.5% AP improves RTMPose-l from 64.8% AP and surpasses the RTMPose-x teacher at 65.3% AP.
2. Related work
Related work covers whole-body pose estimation and knowledge distillation, while prior pose distillation studies largely target body-only heatmaps or combine body-part experts. DWPose is presented as the first efficient KD strategy for 2D whole-body pose estimation.
- Whole-body pose estimation jointly localizes expressive body, hand, feet, and face keypoints for all persons in an image.
- Knowledge distillation transfers teacher knowledge through soft output labels or intermediate features to compress models across tasks.
- Prior pose distillation studies target heavy body-only heatmaps or combine separate body-part experts into a whole-body network.
- DWPose is presented as the first work to explore efficient knowledge-distillation strategies for 2D whole-body pose estimation.
3. Method
DWPose uses two distillation stages: a teacher guides a student through feature and logit supervision, then the trained model refines the student head through self-distillation. The design combines complete teacher logits, scheduled distillation, and a short head-only training phase.
- First-stage distillation: The first stage guides a student trained from scratch using the teacher’s intermediate features and final logits.Feature distillation matches backbone representations, while logit distillation transfers the teacher’s outputs.
- Logit-based distillation: Logit distillation removes the target visibility mask, allowing teacher supervision for both visible and invisible keypoints.The method treats the teacher’s values for invisible keypoints as useful supervision rather than ignoring them.
- Weight-decay strategy: A weight-decay schedule gradually reduces the distillation penalty so the student focuses more on labels during training.The time function r(t) controls the distillation terms across training epochs.
- Second-stage distillation: The second stage freezes a trained student backbone and trains only its untrained head against logits from the same model’s trained head.Because teacher and student share an architecture, the backbone feature is extracted once and passed through both heads; the original label loss is dropped.
- Second-stage distillation: Head-aware self-distillation improves localization capability with only 20% training time as a plug-and-play strategy.The approach also improves students trained either from scratch with distillation or without it.
4. Experiments
Experiments evaluate DWPose across datasets, training settings, model sizes, and qualitative comparisons. The results show improved accuracy and fine-grained localization, with competitive efficiency and broader robustness.
- Datasets and Details: The experiments use COCO and UBody, with UBody providing over 1M frames from 15 real-life scenarios and 133 2d keypoints.
- Datasets and Details: Two-stage distillation requires about 1/5 of the whole training epochs, while the remaining training settings stay unchanged.The implementation uses 8 GPUs and fixed loss-balancing hyperparameters across COCO and UBody.
- Main Results: 66.5 whole AP is achieved by DWPose-l at one input resolution, surpassing the RTMPose-x teacher with fewer parameters and flops.DWPose-l also reaches 63.1 whole AP under another input resolution.
- Main Results: 60.6 whole AP is achieved by DWPose-m with 2.2 GFLOPs, improving 4.1% over the baseline while retaining the same inference consumption.
- Qualitative Comparisons: Qualitative comparisons report more accurate predictions, fewer false poses, more true poses, and improved finger-keypoint localization with TPD.
- Qualitative Comparisons: DWPose surpasses OpenPose and MediaPipe, especially under truncation and occlusion and for fine-grained localization.The paper connects these improvements to downstream applications.
5. Analysis
Experiments show that UBody data and both distillation stages improve whole-body pose estimation, while the second stage offers an efficient self-distillation option. The resulting pose estimator also improves controllable image generation and runs substantially faster than OpenPose.
- 5.1. Effects of TPD Method and UBody Data: Adding UBody data improves whole-body and hand AP, while TPD further raises RTMPose-l performance.RTMPose-l reaches 62.1 whole-body AP and 55.1 Hand AP with UBody, improving by 1.0 and 3.2 points over COCO-only training; TPD then reaches 63.1 whole-body AP.
- 5.3. Effects of First and Second Stage Distillation: Both distillation stages help, and their combination achieves 63.1 whole AP, exceeding either distillation loss alone.The ablation uses RTMPose-x as teacher and RTMPose-l as student on COCO + UBody.
- 5.3. Effects of First and Second Stage Distillation: The second-stage distillation fine-tunes only the head, saves training time, and lets the student surpass RTMPose-x with 63.0% AP.It is also applicable to models trained without first-stage distillation, including when a larger teacher is unavailable.
- 5.5. Ablation Study of the First-stage Distillation: Feature and logit distillation are complementary, and weight decay adds 0.3% AP, producing 62.3% AP.Feature distillation contributes 1.4% AP, while adding logit distillation raises the gain to 1.6% before weight decay.
- 5.5. Ablation Study of the First-stage Distillation: Teacher logits alone achieve 60.9% AP, 0.5% above ground-truth supervision, but combining feature distillation makes ground truth, features, and logits preferable together.The authors suggest teacher labeling can reduce manual annotation effort, while noting that feature distillation changes the comparison.
- 5.6. Application to Controllable Image Generation: DWPose produces higher-quality ControlNet images with more precise skeletons and infers a single-person image in 0.068 s versus OpenPose’s 5.78 s.With nine persons, OpenPose inference triples while DWPose becomes only about 1.5 times longer.
6. Conclusion
The paper presents DWPose, a two-stage distillation framework for efficient and effective whole-body pose estimation, augmented with UBody data. Experiments support its effectiveness and include an application to controllable image generation.
- 6. Conclusion: DWPose combines two-stage pose distillation with UBody data to improve lightweight RTMPose-based whole-body pose estimation.The second stage can operate without a larger teacher and requires only short training, while experiments also examine improved pose estimation for controllable image generation.