Source-linked AI summary
Adversarial PoseNet: A Structure-aware Convolutional Network for Human Pose Estimation
Yu Chen, Chunhua Shen, Xiu-Shen Wei, Lingqiao Liu, Jian Yang
TL;DR
Human pose estimation can produce biologically implausible predictions when joints are occluded or bodies overlap, despite geometric constraints in human vision. Adversarial PoseNet uses a multi-task pose generator with pose and confidence discriminators to learn human-body priors, achieving better benchmark results and mostly human-manifold poses with small computation overhead.
Problem
Occlusions and overlapping bodies can cause deviated, biologically implausible pose predictions, while explicitly learning human-joint geometric constraints is challenging.
Method
Adversarial PoseNet trains a multi-task pose generator with pose and confidence discriminators that distinguish reasonable poses and confidence predictions from unreasonable ones.
Results
The method outperforms previous methods across all body joints on LSP, improving average PCK by 2.4%, and produces mostly human-body-manifold poses on two benchmark datasets.
Takeaways & Limitations
Adversarial training incorporates human-body structure priors and makes pose estimation more robust to occlusions, overlapping, and twisting bodies.
Takeaways & Limitations
The method may fail on challenging cases involving twisted limbs at image edges, overlapping people, and occluded body parts.
Abstract
from arXiv · showhide
For human pose estimation in monocular images, joint occlusions and overlapping upon human bodies often result in deviated pose predictions. Under these circumstances, biologically implausible pose predictions may be produced. In contrast, human vision is able to predict poses by exploiting geometric constraints of joint inter-connectivity. To address the problem by incorporating priors about the structure of human bodies, we propose a novel structure-aware convolutional network to implicitly take such priors into account during training of the deep network. Explicit learning of such constraints is typically challenging. Instead, we design discriminators to distinguish the real poses from the fake ones (such as biologically implausible ones). If the pose generator (G) generates results that the discriminator fails to distinguish from real ones, the network successfully learns the priors.
1. Introduction
Human pose estimation is difficult under occlusion, overlapping bodies, and clutter because standard DCNNs struggle to enforce human-body structure. Adversarial PoseNet implicitly learns these priors through discriminators that distinguish plausible from implausible poses.
- 1. Introduction: Occlusions, overlapping bodies, clutter, flexible limbs, and viewpoint variation make monocular human pose estimation challenging.These conditions can make DCNNs regress inaccurate heatmaps for affected body parts.
- 1. Introduction: Human observers can infer plausible poses under severe occlusion by exploiting geometric constraints among body joints.The paper identifies these structural priors as difficult to incorporate explicitly into DCNNs.
- 1. Introduction: Standard DCNNs may produce unreasonable poses because explicit learning of the real human-joint distribution is difficult.Without body-structure priors, predictions can follow locally matched features rather than a coherent human configuration.
- 1. Introduction: Adversarial PoseNet uses a regression network as a generator and trains it against discriminators that assess whether predicted poses are geometrically reasonable.This adversarial strategy implicitly learns the distribution of human body structures.
- 1. Introduction: The generator uses stacked multi-task learning to predict pose and occlusion heatmaps, which are provided to pose and confidence discriminators.The pose discriminator evaluates body configuration, while the confidence discriminator evaluates confidence in predicted heatmaps.
- 1. Introduction: The work reports significant improvements over state-of-the-art methods and more plausible predictions on two public human-pose datasets.Its contributions include incorporating joint-connectivity constraints and using GANs to exploit the constrained human-pose distribution.
2. The Proposed Adversarial PoseNet
Adversarial PoseNet combines a stacked multi-task pose generator with pose and confidence discriminators to incorporate human-body structure priors during training. The generator predicts pose and occlusion heatmaps, while adversarial feedback encourages plausible, confident predictions.
- The model comprises a pose generator G, pose discriminator P, and confidence discriminator C.G generates pose outputs; P evaluates body-shape plausibility, while C evaluates confidence in body-part localization.
- G takes RGB images and outputs 32 heatmaps: 16 pose estimates and 16 corresponding occlusion predictions.Heatmap values are confidence scores in [0, 1], with Gaussian blur around ground-truth positions.
- Without discriminators, G can produce low-confidence or incorrect locations; adversarial gradients from C and P correct these predictions and exploit body priors.Training with C produces more confident results, while training with both P and C implicitly exploits human-body priors and improves prediction confidence.
- 2.1. Multi-Task Generative Network: The stacked multi-task generator jointly models pose and occlusion heatmaps to use occlusion information when inferring geometric pose structure.Its encoder-decoder design uses large receptive fields, skip connections, and stacking to combine contextual and local information and re-evaluate estimates.
- 2.2. Pose Discriminator: P receives the RGB image together with G’s pose and occlusion maps to judge whether a pose is reasonable for that image.This conditional design addresses cases where a predicted pose may be plausible for a different body shape, and G is also supervised toward ground-truth poses with an ℓ2 objective.
- 2.3. Confidence Discriminator: C distinguishes high-confidence from low-confidence predictions, targeting the reduced confidence that standard estimators often assign to occluded body parts.The confidence discriminator is motivated by human ability to locate occluded parts using geometric priors.
3. Experiments
Experiments on LSP and MPII show that the proposed method improves pose-estimation accuracy, including under occlusion, while producing more plausible difficult-case predictions. Ablations attribute gains to the multi-task generator and discriminators.
- Datasets and Metrics: The method is evaluated on the LSP and MPII Human Pose benchmarks using PCK@0.2 and PCKh@0.5.LSP contains 11k training and 1k testing images; MPII contains around 25k images with 40k annotated samples.
- Quantitative Results: 2.4% improvement in average PCK is achieved over previous methods on LSP, with gains of 1.5% for wrists and 1.9% for ankles over the closest competitor.The comparisons use a normalized distance of 0.2.
- Quantitative Results: 92.1% PCKh@0.5 is achieved on MPII, the best score reported in the comparison.The baseline is a four-stacked single-task network without multi-task learning or discriminators.
- Qualitative Comparisons: Qualitative comparisons show fewer strange locations and better recovery of difficult poses than a 2-stacked hourglass network under twisting and occlusion.The proposed network succeeds where the baseline places parts nearby or produces deviated twisting across limbs.
- Qualitative Comparisons: The method still fails in cases involving twisted limbs at the image edge, overlapping people, and occluded body parts.Even in these cases, the reported outputs are more reasonable than those of the previous method.
- Occlusion Analysis: Under occlusion, gains over the stacked hourglass baseline are about 0.8% for visible parts and 3.5% and 3.6% for invisible wrists and elbows.About 25% of annotated MPII elbows and wrists are labeled invisible.
- Ablation Study: Using both multi-task learning and discriminators improves accuracy by 1.5%, compared with 0.5% and 0.6% when each is used separately.The ablation uses the MPII validation set and a four-stacked single-task generator as baseline.
4. Conclusions
Adversarial PoseNet uses a multi-task pose generator and two discriminators to make predictions more robust to occlusions, overlap, and body twisting. It produces mostly human-shaped poses, requires only the generator at test time, and may extend to other structured prediction tasks.
- Adversarial PoseNet trains a multi-task pose generator with two discriminator networks that distinguish reasonable from unreasonable poses.The generator is trained to deceive the discriminators into judging generated poses as real.
- The method is more robust to occlusions, overlapping, and twisting of human bodies while reducing reliance on matched image features alone.
- Only the generator network is used during testing, despite training three sub-networks, with a small computation overhead.
- The network produces poses that are mostly within the manifold of human body shape.
- The approach may also apply to face landmark detection and broader structured output learning problems.