Source-linked AI summary
3D Human Pose Estimation in the Wild by Adversarial Learning
Wei Yang, Wanli Ouyang, Xiaolong Wang, Jimmy Ren, Hongsheng Li, Xiaogang Wang
TL;DR
3D human pose estimation in the wild lacks abundant 3D annotations, limiting the transfer of models trained in constrained laboratories. The paper uses adversarial learning with a multi-source discriminator and geometric descriptor to transfer pose structure from annotated data to in-the-wild images. Experiments report improvements over prior state-of-the-art methods, while high-angle views remain a documented failure case.
Problem
3D pose annotations are difficult to obtain for in-the-wild images, whereas existing lab datasets have limited environmental variation and may not generalize across the domain shift.
Method
An adversarial framework transfers 3D pose structures using a pose-estimator generator and a multi-source discriminator informed by images, geometric descriptors, heatmaps, and depth maps.
Results
The approach improves state-of-the-art 3D pose estimation performance qualitatively and quantitatively, including 37.7mm error under Protocol #2 versus 45.7mm for the previous best result.
Takeaways & Limitations
Adversarial learning with geometric and visual information transfers human pose structure to in-the-wild images without requiring 3D annotations there.
Takeaways & Limitations
A documented failure case involves high-angle shots not covered by the four cameras in the 3D pose dataset; the authors suggest adding more camera views.
Abstract
from arXiv · showhide
Recently, remarkable advances have been achieved in 3D human pose estimation from monocular images because of the powerful Deep Convolutional Neural Networks (DCNNs). Despite their success on large-scale datasets collected in the constrained lab environment, it is difficult to obtain the 3D pose annotations for in-the-wild images. Therefore, 3D human pose estimation in the wild is still a challenge. In this paper, we propose an adversarial learning framework, which distills the 3D human pose structures learned from the fully annotated dataset to in-the-wild images with only 2D pose annotations. Instead of defining hard-coded rules to constrain the pose estimation results, we design a novel multi-source discriminator to distinguish the predicted 3D poses from the ground-truth, which helps to enforce the pose estimator to generate anthropometrically valid poses even with images in the wild. We also observe that a carefully designed information source for the discriminator is essential to boost the performance. Thus, we design a geometric descriptor, which computes the pairwise relative locations and distances between body joints, as a new information source for the discriminator. The efficacy of our adversarial learning framework with the new geometric descriptor has been demonstrated through extensive experiments on widely used public benchmarks. Our approach significantly improves the performance compared with previous state-of-the-art approaches.
1. Introduction
3D human pose estimation remains difficult in the wild because 3D annotations are scarce and lab-trained models face domain shift. The paper addresses this with adversarial learning and a multi-source discriminator that transfers pose structure without hard-coded rules.
- Motivation: 3D pose estimation is limited by the difficulty of obtaining ground-truth 3D joint locations in unconstrained environments.Existing 3D datasets are collected in constrained labs, while in-the-wild datasets typically provide only 2D annotations.
- Motivation: Lab-trained DCNN pose estimators may generalize poorly to in-the-wild images because of large domain shifts in background, viewpoint, and lighting.Human3.6M is cited as a constrained lab dataset, whereas MPII represents an in-the-wild setting with 2D annotations.
- Motivation: Human perception evaluates predicted poses using image-pose correspondence and articulation-constrained body configurations.The paper uses this observation to motivate a discriminator that distinguishes ground-truth poses from estimations.
- Approach: The proposed adversarial framework distills 3D pose structures from fully annotated constrained data to in-the-wild images without 3D pose annotations.A pose estimator generates predictions while a discriminator distinguishes predicted poses from ground-truth poses.
- Approach: The multi-source discriminator combines image information with relative offsets and distances between body parts to model correspondence and articulation constraints.These sources provide visual information and domain prior knowledge for improving generalization.
2. Related Work
Prior work includes tree-structured and DCNN-based approaches for 2D and 3D pose estimation, with one-stage methods relying heavily on fully annotated data. Two-stage lifting, weak supervision, transfer learning, and adversarial methods address generalization and annotation limitations in different ways.
- 2D Pose Estimation: Conventional 2D pose methods use tree-structured models with unary joint-detection terms and pairwise body-part relationship terms.Pairwise terms can encode relative locations, distances, and limb symmetry.
- DCNN-Based Methods: Recent DCNN methods commonly use heatmaps rather than directly regressing joint coordinates, and this paper adopts stacked hourglass features as its backbone.Heatmaps are generated by 2D Gaussians centered on body-joint locations.
- 3D Pose Estimation: One-stage 3D methods learn poses directly from monocular images but depend heavily on fully annotated datasets.Examples include multi-task regression, autoencoder-based joint dependency modeling, and voxel representations.
- 3D Pose Estimation: Two-stage methods first estimate 2D poses and then lift them to 3D, often generalizing better because 2D estimators can use in-the-wild training data.Representative approaches regress 3D coordinates or learn pairwise distance matrices from 2D inputs.
- Generalization and Adversarial Learning: Prior efforts for in-the-wild 3D prediction include geometric losses, transfer learning, kinematic fitting, and adversarial discrimination of ground-truth versus fake poses.The proposed framework is described as complementary to these approaches and uses in-the-wild predictions to improve the estimator.
3. Framework
The framework trains a 3D pose estimator and discriminator adversarially, using lab and in-the-wild images to transfer human-body structure. Its discriminator combines image, geometric, heatmap, and depth information, with the geometric descriptor encoding pairwise 3D relationships.
- 3. Framework: The framework uses a GAN formulation in which a 3D pose estimator generates predictions and a discriminator distinguishes them from ground-truth 3D poses.Predictions from lab and in-the-wild images allow body structures learned from annotated data to be adapted to in-the-wild images.
- 3. Framework: Training first pretrains the pose estimator on a 3D pose dataset, then alternately optimizes the generator and discriminator; the discriminator is discarded at test time.
- 3.1. Generator: 3D Pose Estimator: The generator is a two-stage estimator with a stacked hourglass 2D module followed by a depth regression module.The first stage outputs P heatmaps, and the second stage predicts a P × 1 depth vector for the body joints.
- 3.1. Generator: 3D Pose Estimator: The paper omits a previously proposed geometric loss for concise adversarial-learning analysis, while describing the method as complementary to that loss.
- 3.2. Discriminator: The discriminator uses original images, pairwise relative locations and distances, and concatenated heatmap-depth representations as three information sources.Each source is separately embedded before concatenation for ground-truth-versus-estimated pose classification.
- 3.2.2 Geometric Descriptor: The geometric descriptor explicitly encodes pairwise relative locations and distances between body parts to represent articulation priors such as limb lengths, joint-angle limits, and symmetry.It extends pictorial-structure deformation constraints from 2D to 3D and is defined as a 6D vector for pairs of joints.
- 3.2.2 Geometric Descriptor: Computing the descriptor for every joint pair produces a 6 × P × P matrix for P body joints.
4. Learning
The learning procedure first pretrains a 3D pose estimator, then alternates discriminator and generator optimization so predicted poses become harder to distinguish from ground truth while retaining pose accuracy.
- 4.1. Pretraining of the Generator: The generator is pretrained with a 2D pose estimation module and a depth regression module using squared-error supervision.The 2D module predicts joint heatmaps, while the depth module predicts joint depth values.
- 4.1. Pretraining of the Generator: A pretrained stacked hourglass network provides the 2D pose estimation module, which is jointly fine-tuned with depth regression.
- 4.2. Adversarial Learning: The discriminator classifies ground-truth poses as real and generator predictions as fake using encoded heatmaps, depth maps, and geometric descriptors.Real samples come from the 3D pose dataset, while generated samples are produced from images in the 3D or 2D pose datasets.
- 4.2. Adversarial Learning: The generator minimizes a classification loss that encourages anthropometrically plausible, image-conditioned poses capable of fooling the discriminator.
- 4.2. Adversarial Learning: Direct adversarial training can reduce pose accuracy, so the generator loss incorporates the regression loss as a regularizer.The classification and regression terms are balanced by a hyperparameter set to 1e −4 in the experiments.
- 4.2. Adversarial Learning: Figure 3 shows predicted 3D poses becoming more accurate during adversarial learning, as the generator improves after receiving larger errors from the discriminator.The initial predictions are described as anthropometrically invalid and easily distinguishable from ground-truth poses.
5. Experiments
Experiments evaluate the framework on 3D, cross-domain, and in-the-wild benchmarks, showing improved accuracy, ablation benefits from geometric descriptors and pretraining, and qualitative correction of implausible poses. A high-angle camera view remains a common failure case.
- Results on Human3.6M: The method achieves 58.6 mm MPJPE under Protocol #1 on Human3.6M, improving 9.7% over its backbone and 3.0% over the recent best result.Under Protocol #2, it obtains 37.7 mm versus the previous best 45.7 mm, a 17.5% improvement.
- Ablation Study: All adversarial-learning variants outperform the baseline, while geometric descriptors reduce error by 4.5 mm and outperform heatmap-based information by 1 mm.Combining images, heatmaps, depth maps, and geometric descriptors achieves the lowest error.
- Ablation Study: Pretraining the generator lowers prediction error from 63.4 to 59.7 mm and produces substantially faster convergence.The comparison evaluates the full model with and without pretraining the depth regressor.
- Ablation Study: End-to-end adversarial training reduces error by 3.4 mm, or around 5%, compared with fixing the 2D pose module.The experiment concludes that end-to-end training is necessary to boost adversarial-learning performance.
- Ablation Study: Adversarial learning reduces the 2D pose-estimation error rate by 8.1% on the MPII validation set.The comparison uses PCKh@0.5 scores between the pretrained baseline module and the adversarially trained model.
- Cross-Domain Generalization: Qualitative results show refinement of anatomically implausible poses, while unseen high-angle camera views remain a common failure case.The high-angle view is not covered by the four cameras in the 3D pose dataset and may require more training views.
6. Conclusion
The paper transfers 3D pose structures to in-the-wild images using adversarial learning, a multi-source discriminator, and a geometric descriptor. Experiments show improved 3D pose estimation accuracy, while future work targets broader camera-view augmentation.
- The framework transfers 3D human pose structures from fully annotated data to in-the-wild images using only 2D pose annotations.
- A multi-source discriminator and geometric descriptor encode pairwise relative joint locations and distances to bridge predicted poses and ground-truth poses.
- Experiments validate improved pose-estimation accuracy on a 3D human pose dataset.
- Future work will augment camera views to improve generalization ability.