Source-linked AI summary
RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation
Bastian Wandt, Bodo Rosenhahn
TL;DR
RepNet addresses overfitting and missing reprojection constraints in monocular 3D human pose estimation. It uses weakly supervised adversarial training with camera estimation and reprojection, achieving state-of-the-art results across three datasets while supporting real-time inference.
Problem
Neural-network pose estimators often ignore reprojection constraints and generalize poorly to unknown motions and camera positions.
Method
RepNet combines adversarial training, camera estimation, and a reprojection layer to learn 3D poses from 2D detections without 2D-3D correspondences.
Results
RepNet achieves state-of-the-art results on Human3.6M, MPI-INF-3DHP, and LSP, and predicts a pose in less than 0.1 milliseconds on standard hardware.
Takeaways & Limitations
The method generalizes to poses and camera positions absent from training data while enabling real-time pose estimation when combined with 2D joint detectors.
Takeaways & Limitations
The camera estimation network is constrained to produce weak perspective camera matrices, with poses centered at their root joint.
Abstract
from arXiv · showhide
This paper addresses the problem of 3D human pose estimation from single images. While for a long time human skeletons were parameterized and fitted to the observation by satisfying a reprojection error, nowadays researchers directly use neural networks to infer the 3D pose from the observations. However, most of these approaches ignore the fact that a reprojection constraint has to be satisfied and are sensitive to overfitting. We tackle the overfitting problem by ignoring 2D to 3D correspondences. This efficiently avoids a simple memorization of the training data and allows for a weakly supervised training. One part of the proposed reprojection network (RepNet) learns a mapping from a distribution of 2D poses to a distribution of 3D poses using an adversarial training approach. Another part of the network estimates the camera. This allows for the definition of a network layer that performs the reprojection of the estimated 3D pose back to 2D which results in a reprojection loss function. Our experiments show that RepNet generalizes well to unknown data and outperforms state-of-the-art methods when applied to unseen data. Moreover, our implementation runs in real-time on a standard desktop PC.
1. Introduction
RepNet addresses poor generalization in neural 3D pose estimation by combining weakly supervised adversarial learning with camera estimation and reprojection. It targets robustness to unknown poses and cameras without requiring 2D–3D correspondences.
- Neural pose estimators can achieve impressive results on similar data but often generalize poorly to unknown motions and camera positions.
- A reprojection layer projects the estimated 3D pose back to 2D, while a camera-estimation network supports matching the 2D observations.
- RepNet maps distributions of detected 2D keypoints to valid 3D human poses using a discriminator trained with weak supervision.The approach relaxes the requirement that every training image have a specific 3D pose label.
- The method estimates 3D skeletal keypoints and camera pose simultaneously without 2D–3D correspondences or known cameras.
- The network uses a kinematic-chain representation encoding bone lengths and joint-angle information within the discriminator.
- RepNet generalizes to unknown human poses and cameras and achieves state-of-the-art results on Human3.6M, MPI-INF-3DHP, and LSP.It can predict a human pose in less than 0.1 milliseconds on standard hardware.
2. Related Work
Related work spans optimization-based reconstruction using reprojection error and direct neural regression from images or detected keypoints. Earlier approaches also searched pose databases or learned pose dictionaries to explain 2D observations.
- Optimization-based methods deform a 3D human body model to satisfy a reprojection error.
- Some approaches search large pose databases for 3D poses that best explain 2D observations.
- Pose-database knowledge is commonly compressed by learning overcomplete dictionaries with PCA or other dictionary-learning methods.
- Recent methods directly regress 3D pose from images or detected keypoints using deep neural networks.Examples include end-to-end image systems and networks that map 2D joint detections directly to 3D pose.
3. Method
RepNet maps distributions of 2D poses to feasible 3D poses while estimating a camera and enforcing consistency through a reprojection layer. Its critic uses kinematic structure, while camera and pose processing address projection ambiguities and missing joints.
- Pose and camera estimation: The pose-and-camera network has separate branches that regress a 3n-dimensional 3D pose vector and six weak-perspective camera parameters.The pose output is reshaped into X, while the camera branch produces the parameters used to form the projection matrix.
- Reprojection layer: The reprojection layer multiplies the estimated 3D pose X by camera K to produce a 2D reprojection, then compares it with the observed 2D pose using a Frobenius-norm loss.The layer has no trainable parameters; undetected joints are zeroed so they do not affect the loss and can be hallucinated by the pose generator.
- Critic network: The critic combines a fully connected path with a KCS-based path before outputting a critic value.KCS supplies joint-angle and bone-length information, helping the critic assess kinematic-chain structure, symmetry, and joint-angle constraints.
- Camera handling: The weak-perspective camera is constrained through KKT = s2I2, with the projection scale computed from the largest singular value of K.The associated camera loss uses the Frobenius norm, and the quadratic scale requires one matrix multiplication to compute.
- Camera handling: 3D poses are aligned to a template with Procrustes-derived shoulder and hip rotation and scale, removing pose rotation and scale components that would otherwise be ambiguous with the camera.The resulting transformation is applied to all joints before camera estimation.
4. Experiments
RepNet is evaluated across standard and unseen datasets, with analyses of accuracy, pose plausibility, detector noise, unusual viewpoints, and runtime. The experiments show strong generalization, competitive weakly supervised performance, plausible reconstructions, and real-time feasibility.
- Datasets and protocols: RepNet is evaluated on Human3.6M, MPI-INF-3DHP, and LSP, using 2D joint detections from stacked hourglass networks in most experiments.Human3.6M supplies training data; MPI-INF-3DHP measures performance on unseen data, while LSP probes unusual poses and camera angles.
- Human3.6M: About 30% better performance than the best weakly supervised approach is reported on Human3.6M under protocol-II.Protocol-II applies rigid alignment before calculating MPJPE; RepNet remains comparable to supervised state-of-the-art methods despite ignoring 2D–3D correspondences.
- MPI-INF-3DHP: RepNet outperforms every other method on MPI-INF-3DHP without training on that dataset, including approaches trained on its training set.Training directly on MPI-INF-3DHP produces only minor changes in 3DPCK, AUC, and MPJPE, suggesting similar feasible-pose distributions across training sets.
- Noisy observations: 2D detector noise has a major impact: reconstruction error scales linearly with the noise standard deviation, although KCS preserves plausible poses even at σ = 20px.The input person size is approximately 200px from head to toe, making σ = 20px extremely large.
- Qualitative evaluation: RepNet reconstructs many stretched poses and uncommon camera angles despite training only on Human3.6M camera angles.The network adds 0.05ms per frame to a larger image-to-3D pipeline and therefore has nearly no runtime impact.
5. Conclusion
The paper presents RepNet as a weakly supervised 3D pose estimator that combines adversarial pose learning with camera estimation and reprojection. It avoids 2D–3D correspondences, generalizes to unseen data, and reconstructs uncommon poses and perspectives effectively.
- Contribution: RepNet infers 3D human poses from single-image 2D joint detections using weakly supervised training without 2D–3D correspondences.The method includes a camera estimation network and a reprojection layer that projects estimated 3D poses back to 2D.
- Training: Adversarial training with improved Wasserstein GANs and kinematic chain spaces avoids overfitting to limited training data.The approach uses these components to learn feasible 3D pose structure during weakly supervised training.
- Results: RepNet achieves state-of-the-art performance on Human3.6M and improves over all other methods when applied to unseen MPI-INF-3DHP data.Subjective evaluation on LSP also reports good reconstructions for uncommon poses and perspectives.