Source-linked AI summary
Can 3D Pose be Learned from 2D Projections Alone?
Dylan Drover, Rohith MV, Ching-Hang Chen, Amit Agrawal, Ambrish Tyagi, Cong Phuoc Huynh
TL;DR
Lifting a single 2D pose to 3D is ill-posed, and 3D capture data for learning pose priors are limited. This paper learns 3D pose priors through adversarial training on random 2D projections, and reports better Human3.6M results than many supervised and weakly supervised approaches.
Problem
A single 2D observation admits infinitely many 3D skeletons, while learning realistic 3D priors from ground-truth capture data is limited by the complexity of collecting that data.
Source: §1 · p. 2Method
Results
On Human3.6M, the method outperforms many supervised and weakly supervised approaches; under Protocol 1 with ground-truth 2D test points, it reduces MPJPE by 40% (34.2mm vs. 57.5mm).
Source: §1 · p. 3, §4.2 · p. 8Takeaways & Limitations
The paper demonstrates that an effective prior on 3D structure and pose can be learned from random 2D projections, in a setting where acquiring 3D MoCap data is expensive and challenging.
Source: §5 · p. 14The authors identify robustness to noisy or imperfect 2D pose inputs as an area for future work.
Source: §5 · p. 14
Abstract
from arXiv · showhide
3D pose estimation from a single image is a challenging task in computer vision. We present a weakly supervised approach to estimate 3D pose points, given only 2D pose landmarks. Our method does not require correspondences between 2D and 3D points to build explicit 3D priors. We utilize an adversarial framework to impose a prior on the 3D structure, learned solely from their random 2D projections. Given a set of 2D pose landmarks, the generator network hypothesizes their depths to obtain a 3D skeleton. We propose a novel Random Projection layer, which randomly projects the generated 3D skeleton and sends the resulting 2D pose to the discriminator. The discriminator improves by discriminating between the generated poses and pose samples from a real distribution of 2D poses. Training does not require correspondence between the 2D inputs to either the generator or the discriminator. We apply our approach to the task of 3D human pose estimation. Results on Human3.6M dataset demonstrates that our approach outperforms many previous supervised and weakly supervised approaches.
1 Introduction
The introduction frames lifting 2D landmarks to 3D as an ill-posed problem and asks whether abundant 2D pose data can support 3D pose learning. It proposes learning 3D priors through adversarial training on random 2D projections and reports comparisons on Human3.6M.
1 Introduction
- A single 2D landmark observation admits infinitely many 3D skeletons, while 3D priors are often learned from limited ground-truth capture data.The introduction asks whether abundant 2D image and video data could instead support 3D joint estimation.Source: §1 · pp. 1–2
- The proposed Random Projection layer and adversarial training aim to learn a latent 3D pose distribution solely from 2D poses, without regression from 3D data.The approach does not explicitly learn its 3D prior from 3D data or use explicit 2D-3D correspondence.Source: §1 · p. 2
- The generator hypothesizes joint depths, and a discriminator compares random 2D projections of generated skeletons with real 2D poses without requiring correspondence between their inputs.The random-projection prior favors 3D estimates whose random-view projections fit the real 2D pose distribution.Source: §1 · p. 3
- On Human3.6M, the method outperforms other weakly supervised methods using 2D pose input and a number of supervised methods using explicit 2D-3D correspondences.The comparison is reported in the introduction; no numerical margins are given there.Source: §1 · p. 3
2 Related Work
The related-work section organizes 2D-to-3D lifting approaches by their supervision and contrasts their use of 3D data and adversarial priors with this paper’s random-projection approach.
2 Related Work
- Lifting methods estimate 3D pose from 2D landmark positions, and can be paired with application-specific 2D pose estimators without retraining the lifting algorithm.The section focuses its benchmarks on methods that lift 2D landmarks to 3D.Source: §2 · p. 4
- Fully supervised methods learn from paired 2D-3D data; their dataset-specific modeling can predict metric 3D joint positions but requires difficult-to-obtain correspondences.Examples include regression from 2D joints to 3D joints and regression between 2D and 3D distance matrices.Source: §2 · p. 4
- Weakly supervised approaches commonly learn priors from unpaired 3D data, while this paper operates directly on lower-dimensional vectors of 2D landmark locations rather than image representations.The related work contrasts these approaches with methods using 3D bases, pose dictionaries, and learned 3D prior distributions.Source: §2 · pp. 4–5
- Earlier adversarial pose methods distinguish real and fake 2D or 3D poses but still require 3D data or a 3D pose model, unlike this paper’s adversarial loss on random projections.The cited SMPL-based approach uses a large database of 3D human meshes for its skeleton prior.Source: §2 · p. 5
3 Weakly supervised Lifting of 2D Pose to 3D Skeleton
The method lifts normalized 2D landmarks into a 3D skeleton by predicting joint depths, then uses randomly projected poses and adversarial training to learn an implicit 3D prior without paired 2D–3D correspondences. Its training setup constrains depth for stability and uses residual-block generator and discriminator networks.
3 Weakly supervised Lifting of 2D Pose to 3D Skeleton
- The generator predicts a depth offset for each 2D landmark, which is used to construct a 3D skeleton from the input pose.The method uses a camera-centered coordinate system, places the root joint at the origin, and targets an approximately one-unit head-to-root distance.Source: §3 · p. 5
- Back projection uses exact perspective projection to construct 3D points, while the random projection layer rotates the skeleton and reprojects it into fake 2D poses for the discriminator.The rotated points are also constrained to remain in front of the camera.Source: §3 · p. 6
- Predicting depth offsets relative to a constant depth removes the scale–distance ambiguity of perspective projection and is intended to stabilize training.The depth d is arbitrary provided d > 1; the implementation uses d = 10 units.Source: §3 · p. 6
- Random projections of generated 3D skeletons are judged against real 2D poses by a discriminator, allowing the model to learn implicit priors on limb-length ratios and joint angles without requiring corresponding real poses.The real 2D pose used by the discriminator need not match the generator's input or its multi-view correspondences.Source: §3 · p. 6
3.1 Training
- Training uses a four-residual-block generator and a three-residual-block discriminator, with 28-dimensional pose inputs and 14 predicted depth offsets.The residual blocks use paired fully connected layers with batch normalization and ReLU; training uses batch size 32,768 and Adam with a starting learning rate of 0.0002.Source: §3.1 · p. 7, Figure 3 · p. 7
4 Experimental Results
On Human3.6M, the method is competitive with supervised systems and improves on prior weakly supervised methods across both evaluation protocols. Qualitative results show reconstructions on Human3.6M and generalization to MPII and LSP, alongside limitations for occluded or incomplete poses.
4.1 Dataset and Evaluation Metrics
- Evaluation uses Human3.6M, 14-joint poses, and two protocols reporting MPJPE after scale and rigid alignment; the model estimates pose only up to scale.Training uses subjects S1, S5, S6, S7, and S8; synthetic projections augment the four camera views using eight additional camera positions.Source: §4 · p. 7, §4.1 · p. 8
4.2 Quantitative Results: Protocol 1
- For Protocol 1, errors fall 40% (34.2mm vs. 57.5mm) with ground-truth 2D points and 25% (62.3mm vs. 82.7mm) with detected points.With detected points, the method outperforms supervised approaches in all activities except Walking.Source: §4.2 · p. 8, Table 1 · p. 9, Table 2 · p. 9
4.3 Quantitative Results: Protocol 2
- For Protocol 2, error drops by more than 50% (38.2mm vs. 79.0mm) with ground-truth 2D points and 33% (64.6mm vs. 97.2mm) with detected points.With detected points, the method outperforms other weakly supervised approaches in all activity classes.Source: §4.3 · pp. 9–10, Table 3 · p. 9, Table 4 · p. 10
4.4 Qualitative Results
- The method reconstructs Human3.6M poses but can fail on severe occlusions or plausible mirror-flip alternatives, and cannot lift poses with fewer than 14 joints.On MPII and LSP, it produces qualitative reconstructions despite not being trained on poses from those datasets; they lack ground-truth 3D skeletons.Source: §4.4 · pp. 10–11, Figure 4 · p. 12, Figure 5 · p. 13
4.3 Quantitative Results: Protocol 2
- 36.3mm versus 37.1mm: an ensemble of 11 models surpasses the best fully supervised result, while the single model is within 1.1mm.The comparison uses Protocol 2 with ground-truth 2D inputs.Source: §4.3 · p. 10, Table 6 · p. 11
4.5 Discussion
- SitDown has the worst errors across methods, with leg or ankle marker occlusion cited as one contributor; results are also reported as stable per frame on video without temporal information.The authors conclude that the results support adversarial learning of 3D priors weakly supervised from 2D poses.Source: §4.5 · p. 12
5 Conclusions
The paper presents adversarial learning of a 3D pose prior from random 2D projections, without requiring 3D annotations. It also identifies broader applications and several directions for future work.
5 Conclusions
- Figure 7 shows reconstructed 3D skeletons alongside 2D pose points on Leeds, a dataset without ground-truth 3D skeletons.Source: Figure 7 · p. 14
- An adversarial framework learns an effective prior on 3D structure and pose from random 2D projections, addressing the expense of acquiring 3D motion-capture data.The paper notes that high-quality computer-vision annotations are difficult to obtain.Source: §5 · pp. 13–14
- The authors envision applying projective-geometry learning beyond human pose, including sparse 3D reconstruction and rendering models from line drawings.They describe single-image edges or keypoints for reconstruction and a small neural network for interactive graphics.Source: §5 · p. 14
- Future work includes training from images in the wild, testing difficult poses and additional datasets, and improving robustness to noisy 2D inputs.The authors also identify predicting joint-location distributions and combining with fully supervised approaches as opportunities.Source: §5 · p. 14