Source-linked AI summary
Self Adversarial Training for Human Pose Estimation
Chia-Jung Chou, Jui-Ting Chien, Hwann-Tzong Chen
TL;DR
Human pose estimation from a single image is difficult because visual information is limited and body configurations vary. The paper uses adversarial training with matched generator and discriminator networks to distinguish plausible from implausible heatmaps, and reports improved prediction accuracy across three benchmark datasets. The discriminator is removed after training, so it does not affect inference time.
Problem
Single-image human pose estimation is challenging because 2D images provide limited information and body parts vary in configuration and appearance.
Method
The method uses a generator to predict keypoint heatmaps and a same-architecture discriminator to distinguish real from generated heatmaps during adversarial training.
Results
The approach improves prediction accuracy on LSP, MPII, and LIP, with the reported method achieving the best result in the LIP challenge.
Takeaways & Limitations
The discriminator can be removed after training, so adversarial training does not affect inference time.
Takeaways & Limitations
The LSP dataset contains noisy labels because some occluded joints lack location information or have incorrect locations.
Abstract
from arXiv · showhide
This paper presents a deep learning based approach to the problem of human pose estimation. We employ generative adversarial networks as our learning paradigm in which we set up two stacked hourglass networks with the same architecture, one as the generator and the other as the discriminator. The generator is used as a human pose estimator after the training is done. The discriminator distinguishes ground-truth heatmaps from generated ones, and back-propagates the adversarial loss to the generator. This process enables the generator to learn plausible human body configurations and is shown to be useful for improving the prediction accuracy.
1. Introduction
Single-image human pose estimation is difficult because 2D images provide limited information and body parts vary widely in configuration and appearance. The paper addresses these challenges by using adversarial training to learn structural constraints and reports improved accuracy across evaluated datasets.
- Single-image pose estimation is challenging because 2D images contain limited information and body parts vary in configuration and appearance.
- Stacked hourglass networks learn useful multiscale correlations but may still produce implausible poses.
- Adversarial training adds a discriminator that checks human-body structural constraints while the pose estimator remains the generator.
- The proposed approach uses generative-adversarial training to learn human body structure and configuration.
- Experiments on LSP, MPII, and LIP report improved accuracy across all three datasets.
2. Related Work
Prior human pose estimation methods progressed from handcrafted graphical-model approaches toward deep networks that predict structural heatmaps and capture spatial relationships. GAN research developed increasingly stable training strategies and was later adapted to supervised vision tasks through conditional losses.
- Early pose estimation methods used graphical models and random-field inference with handcrafted image features.
- Recent pose estimators commonly predict heatmaps that represent the probability of each keypoint at different locations.
- Multi-stage networks and intermediate supervision help capture long-range spatial relationships and refine confidence maps.
- GAN research introduced architectures and objectives such as DCGAN, WGAN, and gradient penalties to improve training stability.
- Conditional GAN losses combined with L1 or L2 distances have been applied to super-resolution, inpainting, and image translation.
3. Adversarial Training with the Stacked Hourglass Networks
The model combines a stacked-hourglass pose generator with an equally structured discriminator that evaluates whether predicted heatmaps form plausible poses. Training combines supervised heatmap reconstruction with adversarial feedback, while intermediate supervision and adaptive loss balancing support optimization.
- Network architecture: The framework uses two networks with the same architecture: a generator predicts keypoint heatmaps, while a discriminator evaluates them with the input image.The discriminator encodes heatmaps and RGB information, then decodes them into heatmaps for distinguishing real from generated poses.
- Network architecture: The generator is a fully convolutional stacked-hourglass network designed to preserve local evidence while modeling long-range relationships between body joints.Residual modules, pooling, upsampling, and skip connections support multiscale processing; transition blocks produce intermediate heatmaps between stacks.
- Training the generator: Generator training combines mean squared error against ground-truth heatmaps with an adversarial loss based on the discrepancy between generated and discriminator-reconstructed heatmaps.The generator receives gradients from both its supervised loss and the discriminator’s adversarial loss, weighted by λG.
- Training the discriminator: The discriminator reconstructs ground-truth heatmaps similarly but drives reconstruction errors on generated heatmaps higher, producing per-pixel structural feedback rather than only a global real-or-fake judgment.For example, it can assign a larger error where a predicted right-knee heatmap conflicts with the left-knee location.
- Training the generator: Intermediate supervision applies mean squared error at the end of each hourglass module, encouraging successive stacks to refine keypoint heatmaps.Early stacks use more local evidence, while later stacks incorporate enlarged receptive fields and longer-range spatial relationships.
- Training the discriminator: An adaptive variable kt balances generator and discriminator training by increasing emphasis on generated-heatmap discrimination when the generator begins to fool the discriminator.The variable is bounded between 0 and 1 and is updated according to the difference between γLreal and Lfake.
4. Experiments
Experiments evaluate the method on LSP, MPII, and LIP using PCK or PCKh, with qualitative and component analyses. The method improves reported accuracy across datasets, while adversarial training provides its clearest gains in selected configurations.
- Datasets and metrics: The method is evaluated on LSP, MPII, and LIP using PCK for LSP and LIP and PCKh for MPII.The experiments use common preprocessing and augmentation, including horizontal flipping, rotation, and scaling.
- Datasets and metrics: LSP contains noisy labels and varied sports poses, making it a challenging evaluation dataset.Some occluded joints lack location information or have incorrect annotations.
- Results: The method refines stacked-hourglass heatmaps that are mostly accurate but uncertain, producing more confident qualitative predictions.This comparison is illustrated in the heatmap visualization.
- Results: On LSP, the approach achieves the highest detection rate across all tolerance ranges, with larger improvement between 0.05 and 0.1.The reported LSP score is computed at r = 0.2, using external MPII training data.
- Results: The method achieves the best result in the LIP Human Pose Estimation Challenge and reports PCKh comparisons on MPII at r = 0.5.The MPII model uses external LSP training data.
- Component analysis: Adversarial training produces significant gains for a 1-stack hourglass, while gains are less obvious for 2-stack models and a 4-stack hourglass with a discriminator performs better than the released 8-stack setting.Learning-rate decay helps both methods, but the adversarial model is reported as more stable and slightly better at the end.
5. Conclusion
The adversarial network improves human pose estimation by using a generator–discriminator framework to produce more plausible heatmaps and poses. Evaluation includes standard benchmark datasets, with qualitative evidence of refined heatmaps and structurally improved poses.
- Method: The adversarial network uses a generator to predict keypoint heatmaps and a discriminator to identify implausible poses and guide refinement.The discriminator can be removed after training, so it does not affect inference time.
- Evaluation: The approach is evaluated on human pose estimation benchmarks including LSP and MPII.
- Qualitative results: Our method generates more plausible and structural poses than [28] in qualitative results.
- Heatmap refinement: Our method further refines stacked-hourglass heatmaps and corrects the position of the right shoulder on LSP.