Source-linked AI summary
Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation
Xi Peng, Zhiqiang Tang, Fei Yang, Rogerio Feris, Dimitris Metaxas
TL;DR
Random augmentation treats data preparation and network training as isolated processes, limiting adaptation to network weaknesses and training status. The paper jointly trains an augmentation network that generates hard online augmentations against a pose network, using reward/penalty training, and reports improved benchmark performance.
Problem
Random augmentation is usually isolated from network training, while scarce effective examples and long-tail data make bounded-data training difficult.
Method
An augmentation network generates hard adversarial augmentations online against a target pose network, using adversarial distributions, U-net features, and a reward/penalty policy.
Results
The method improves the baseline by 0.6% in PCKh@0.5 on MPII and by 1.5% in PCK@0.2 on LSP.
Takeaways & Limitations
Jointly optimizing augmentation and network training improves human pose estimation on public benchmarks without requiring additional data collection.
Abstract
from arXiv · showhide
Random data augmentation is a critical technique to avoid overfitting in training deep neural network models. However, data augmentation and network training are usually treated as two isolated processes, limiting the effectiveness of network training. Why not jointly optimize the two? We propose adversarial data augmentation to address this limitation. The main idea is to design an augmentation network (generator) that competes against a target network (discriminator) by generating `hard' augmentation operations online. The augmentation network explores the weaknesses of the target network, while the latter learns from `hard' augmentations to achieve better performance. We also design a reward/penalty strategy for effective joint training. We demonstrate our approach on the problem of human pose estimation and carry out a comprehensive experimental analysis, showing that our method can significantly improve state-of-the-art models without additional data efforts.
1. Introduction
The paper jointly optimizes data augmentation and pose-network training by generating adversarial augmentations online, addressing weaknesses of isolated random augmentation. Its augmentation network competes with and supports the target network, with reward/penalty training yielding strong benchmark results.
- Limited data, long-tail distributions, and costly annotation make effective training examples scarce for deep networks.
- Random augmentation applies static transformations uniformly, producing ineffective examples, mismatching training status, and rarely sampling useful long-tail augmentations.
- The proposed augmentation network generates hard augmentations online while the pose network evaluates and learns from them.
- Joint training uses adversarial distributions over operations rather than direct pixels or deformations, together with a reward and penalty policy.
- The work claims the first joint optimization of data augmentation and network training for human pose estimation.
- The method reports strong performance on MPII and LSP benchmarks and intensive ablation validation.
2. Related Work
Prior work includes adversarial learning, hard example mining, and deep human pose estimation architectures. This paper differs by generating new adversarially augmented data through an augmentation network rather than selecting existing hard examples or using pose estimation as the generator.
- Adversarial learning: Adversarial learning commonly uses generator–discriminator minimax games, while prior pose-estimation applications use the pose network as the generator.
- Hard example mining: Hard example mining alternates model training and data selection but cannot create data absent from the training set.
- Hard example mining: The proposed augmentation network actively generates adversarial augmentations and new data points to improve pose-network training.
- Human pose estimation: Human pose estimation methods progressed from DeepPose and feature-fusion models to cascaded architectures with intermediate supervision and stacked hourglass networks.
- Proposed approach: The approach overview depicts an augmentation network creating hard augmentations, with the pose network learning from them and producing reward/penalty updates.
3. Adversarial Data Augmentation
The paper jointly optimizes augmentation and pose-network training by using an augmentation generator to create hard augmentations and a pose network to learn from and evaluate them. Unlike static random augmentation and prior adversarial pose methods, the approach conditions generation on the input image and current network status, using sampled augmentation distributions and reward/penalty updates for joint training.
- The method targets improved training of a pre-designed pose network without collecting additional data, addressing the limitations of static random augmentation.Static strategies can produce ineffective augmentations that are too hard or too easy for the network's changing training status.
- An augmentation network G generates hard augmentations that may increase the pose network D's loss, while D learns from and evaluates those generations.The generator's output is conditioned on both the input image and D's current status.
- The generator predicts distributions over augmentation operations, from which scaling and rotating are sampled to augment training images.Figure 3 depicts mixed-Gaussian distributions used to sample scaling and rotating operations.
- A reward and penalty policy supplies online ground truth for G because nondifferentiable augmentation operations prevent gradients from flowing directly from D to G.This lets G track D's changing training status during joint optimization.
- Sampling from distributions is more robust to outliers than directly generating operations or adversarial pixels, reducing upside-down augmentations and the risk of local optima.The paper contrasts distribution sampling with direct generation, which experiments found could collapse training.
4. Adversarial Human Pose Estimation
The method jointly trains an augmentation network and a pose network by generating hard scaling, rotating, and occluding augmentations from predicted distributions. Pre-training, online reward/penalty updates, and alternating augmentation tasks address convergence, missing supervision, and excessively difficult combinations.
- Adversarial augmentation design: The augmentation network takes U-net bridge features and predicts distributions for scaling, rotating, and hierarchical occluding operations.Scaling and rotating are applied to images, while occlusion masks are applied to deep bridge features.
- Adversarial Scaling and Rotating (ASR): Adversarial scaling and rotating divide augmentation ranges into bounded Gaussian bins, sampling operations from the predicted bin distributions.This avoids direct generation that can collapse toward predominantly upside-down, hardest-case augmentations.
- Pre-training: ASR and AHO are pre-trained before joint optimization, using distributions derived from augmentation difficulty and spatial joint statistics.The AHO ground-truth heat map is formed by voting joint coordinates into grid cells and normalizing the counts.
- Adversarial Hierarchical Occluding (AHO): Hierarchical occluding predicts a heat map over a low-resolution feature grid, samples one or two cells, and scales the mask upward across bridge-feature resolutions.The approach uses a 4 × 4 mask scaled to 64 × 64, preserving joint positions while occluding deep features.
- Joint training: The reward/penalty policy updates augmentation probabilities by comparing adversarial and random augmentation losses, rewarding harder samples and penalizing easier ones.This provides online supervision without exhaustively repeating pre-training over the many possible augmentation combinations.
- Joint training: Joint training splits each mini-batch among ordinary random augmentation, ASR, and AHO, training the pose and augmentation networks on their respective shares.Alternating ASR and AHO prevents simultaneously applied adversarial augmentations from becoming too hard for the pose network.
5. Experiments
Experiments evaluate adversarial augmentation across hourglass designs, training dynamics, components, and benchmarks. The method consistently improves pose estimation over random augmentation and the stacked-hourglass baseline.
- Experimental setup: The experiments evaluate adversarial augmentation using residual and dense hourglasses on MPII and LSP pose benchmarks.The study also visualizes training states and compares method components and qualitative predictions.
- Training-status visualization: Adversarial augmentation makes the target-network loss distribution flatter during training, indicating improved handling of rotated images.The augmentation network’s generated rotation distribution tracks the pose network’s loss distribution and training status.
- Component evaluation: ASR improves average PCKh@0.5 accuracy by 0.5% on both residual and dense hourglasses, while AHO improves it by 0.4% on each.ASR denotes adversarial scaling and rotating, whereas AHO denotes adversarial occluding.
- Architecture comparison: The dense hourglass achieves comparable pose-estimation accuracy to the residual design with 18M rather than 38M parameters.Direct connections among dense feature blocks facilitate gradient flow without sacrificing estimation accuracy.
- Benchmark comparison: 0.6% improvement in PCKh@0.5 on MPII and 1.5% improvement in PCK@0.2 on LSP over baseline stacked HGs produce state-of-the-art performance.The quantitative comparison uses eight stacked hourglass networks.
- Qualitative comparison: Qualitative comparisons show improvements on challenging joints and left-right confusions after adversarial training.The same stacked hourglass network is compared under random and adversarial augmentation.
6. Conclusion
The paper concludes that jointly optimizing augmentation and network training with an augmentation network improves human pose-estimation performance. It reports clear improvements over baseline stacked hourglass models and identifies broader visual and language understanding as future application areas.
- The proposed method jointly optimizes data augmentation and target-network training through an augmentation network that generates adversarial augmentations.
- Clear improvements are observed over baseline stacked HGs(8) [24] on LSP using PCK@0.2.
- The authors plan to further improve the method for more general applications in visual and language understanding.