Source-linked AI summary
PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation
Kehong Gong, Jianfeng Zhang, Jiashi Feng
TL;DR
3D human pose estimators generalize poorly to new datasets because training pose pairs have limited diversity. PoseAug jointly learns differentiable augmentation with estimation and discrimination, and achieves broad improvements, including 88.6% 3D PCK on MPI-INF-3DHP, 9.1% above the previous best data-augmentation method.
Problem
3D human pose estimators generalize poorly to new datasets because training data contain limited variation in posture, body size, viewpoint, and human position.
Method
PoseAug jointly trains a differentiable augmentor, pose estimator, and discriminator, using estimation-error feedback to generate diverse poses while enforcing plausibility with part-aware Kinematic Chain Space.
Results
88.6% 3D PCK on 3DHP under cross-dataset evaluation improves on the previous best data-augmentation method by 9.1%, while PoseAug improves performance across evaluated estimators and settings.
Takeaways & Limitations
PoseAug provides a generic framework for generating diverse and plausible training poses that improves generalization across intra-scenario and cross-scenario evaluations.
Takeaways & Limitations
Augmented poses require discrimination because lacking augmentation priors can produce implausible joint angles, positions, or viewpoints that do not benefit generalization.
Abstract
from arXiv · showhide
Existing 3D human pose estimators suffer poor generalization performance to new datasets, largely due to the limited diversity of 2D-3D pose pairs in the training data. To address this problem, we present PoseAug, a new auto-augmentation framework that learns to augment the available training poses towards a greater diversity and thus improve generalization of the trained 2D-to-3D pose estimator. Specifically, PoseAug introduces a novel pose augmentor that learns to adjust various geometry factors (e.g., posture, body size, view point and position) of a pose through differentiable operations. With such differentiable capacity, the augmentor can be jointly optimized with the 3D pose estimator and take the estimation error as feedback to generate more diverse and harder poses in an online manner. Moreover, PoseAug introduces a novel part-aware Kinematic Chain Space for evaluating local joint-angle plausibility and develops a discriminative module accordingly to ensure the plausibility of the augmented poses. These elaborate designs enable PoseAug to generate more diverse yet plausible poses than existing offline augmentation methods, and thus yield better generalization of the pose estimator. PoseAug is generic and easy to be applied to various 3D pose estimators. Extensive experiments demonstrate that PoseAug brings clear improvements on both intra-scenario and cross-scenario datasets. Notably, it achieves 88.6% 3D PCK on MPI-INF-3DHP under cross-dataset evaluation setup, improving upon the previous best data augmentation based method by 9.1%. Code can be found at: https://github.com/jfzhang95/PoseAug.
1. Introduction
PoseAug addresses limited training-pose diversity by jointly learning differentiable augmentation, pose estimation, and plausibility discrimination. It improves generalization across intra- and cross-dataset evaluations while producing diverse, realistic pose pairs.
- 3D human pose estimators struggle to generalize to cross-scenario datasets because training data lack variation in posture, body size, viewpoint, and position.
- PoseAug jointly optimizes augmentation and network training online, using training feedback to adapt the generated data.The framework replaces separate offline augmentation and model training with end-to-end optimization.
- The differentiable augmentor controls joint angles, body size, viewpoint, and human position to create more diverse 2D-3D pose pairs.
- A part-aware 3D discriminator and a 2D pose discriminator enforce plausible joint angles, body sizes, viewpoints, and positions.The part-aware Kinematic Chain Space provides local-wise supervision for joint-angle plausibility.
- PoseAug improves four representative estimators on both source H36M and cross-scenario 3DHP datasets.The framework is described as flexible regarding the choice of 3D pose estimator.
- 88.6% 3D PCK on 3DHP under cross-dataset evaluation improves on the previous best data-augmentation method by 9.1%.PoseAug also brings more than 13.1% average MPJPE improvement for all evaluated models on 3DHP.
2. Related Work
Prior 3D human pose work relies on annotated data and external information to address poor generalization, while augmentation methods increase diversity through image manipulation, synthesis, or pose transformations.
- Deep 3D pose models rely on well-annotated data and often generalize poorly to new camera views and subject poses.
- Existing augmentation methods stitch image patches, generate graphics-based data, or apply transformations directly to 2D-3D pose pairs.
3. Method
PoseAug jointly learns to augment 2D–3D pose pairs and train a pose estimator, using differentiable geometry operations, error feedback, and plausibility discriminators. Its part-aware KCS discriminator evaluates local joint relations to preserve realistic yet diverse augmented poses.
- Problem definition: The pose estimator maps 2D keypoint coordinates to 3D joint positions and is conventionally trained on paired source poses.The source training objective uses annotated 2D–3D poses and typically minimizes mean squared error.
- Online pose augmentation: PoseAug implements a neural augmentor that transforms source pose pairs online while jointly training the augmentor and estimator.The estimator’s loss serves as feedback, guiding the augmentor toward harder samples than the current model handles well.
- Pose augmentor: The augmentor changes posture, body size, viewpoint, and position through bone-angle, bone-length, and rigid-transformation operations.It extracts bone vectors, predicts operation parameters with an MLP, reconstructs the augmented 3D pose, and reprojects it to 2D.
- Pose augmentor: The augmented 3D pose is reconstructed from modified bone vectors and then perspective-projected to form a corresponding augmented 2D–3D pair.Rigid transformation controls viewpoint and position, while inverse hierarchical conversion maps the modified bones back to joint positions.
- Pose plausibility: A two-part discriminator constrains plausibility: the 3D discriminator evaluates joint angles, while the 2D discriminator evaluates body size, viewpoint, and position.This addresses the risk that unconstrained augmentation produces biomechanically invalid or otherwise unrealistic training poses.
- Pose plausibility: The part-aware KCS discriminator evaluates local joint-angle relations across body parts, enlarging the feasible region compared with whole-body KCS supervision.The pose is divided into torso and limb parts, whose local KCS representations provide inputs for distinguishing original and augmented poses.
4. Experiments
Experiments evaluate PoseAug across datasets, estimators, data regimes, and ablations. Results show consistent gains in generalization, diversity, plausibility, and performance with limited training data.
- Experimental setup: Experiments span H36M, 3DHP, 3DPW, MPII, and LSP, using MPJPE, PA-MPJPE, PCK, and AUC where applicable.H36M evaluates within-dataset performance; 3DHP and 3DPW assess cross-scenario generalization, while MPII and LSP support qualitative evaluation.
- Results: PoseAug outperforms prior methods on H36M, despite using nearly 3× less training data than the previous best augmentation method.The comparison is reported in terms of lower MPJPE and is attributed to PoseAug’s online augmentation scheme.
- Results: 12.6% average improvement on 3DPW is obtained across four estimators trained on H36M and evaluated in the wild.The metric is PA-MPJPE.
- Limited-data evaluation: With limited H36M data, PoseAug improves cross-scenario MPJPE from 116.4 to 90.3 using S1 and from 86.6 to 73.0 using the full dataset.Using S1+S5, PoseAug reaches 77.9 MPJPE, outperforming the full-data baseline at 86.6.
- Ablation and analysis: Ablations show that augmentor operations, error feedback, both discriminators, and PA-KCS each improve reported performance or pose guidance.RT contributes most to cross-scenario performance; PoseAug also broadens viewpoint and position diversity.
5. Conclusion
PoseAug is an auto-augmentation framework that enriches training-data diversity and improves 3D pose estimation. Its augmentor, estimator, and discriminator interact through training feedback to generate diverse, informative, and plausible data across various estimators.
- PoseAug learns to enrich training-data diversity and improve the performance of trained pose estimation models.
- The framework fully integrates an augmentor, estimator, and discriminator, with the augmentor adapting 2D-3D pose geometry using estimator training error as feedback.
- A novel part-aware KCS representation enables the discriminator to preserve the plausibility of augmented poses.
- Extensive experiments show that PoseAug generates diverse and informative data that boosts performance across various 3D pose estimators.