Source-linked AI summary
Monocular 3D Human Pose Estimation by Generation and Ordinal Ranking
Saurabh Sharma, Pavan Teja Varigonda, Prashast Bindal, Abhishek Sharma, Arjun Jain
TL;DR
Monocular 2D-to-3D lifting is ambiguous because the problem is ill-posed and existing datasets incompletely cover real-world variation. The paper generates diverse CVAE-based 3D candidates, scores them with ordinal depth relations or evaluates an Oracle-selected candidate, and reports competitive or state-of-the-art benchmark results, including without paired image-to-3D annotations.
Problem
Monocular 2D-to-3D human-pose lifting is inherently ill-posed, and existing annotated datasets inadequately cover real-world variation.
Method
A conditional variational autoencoder generates diverse 3D-pose samples from estimated 2D poses, while ordinal depth relations score and weight-average candidates.
Results
The method achieves close to state-of-the-art results with OrdinalScore and state-of-the-art results with Oracle supervision on two benchmark datasets.
Takeaways & Limitations
The generative sample set addresses 2D-to-3D ambiguity and remains competitive without paired image-to-3D annotations.
Abstract
from arXiv · showhide
Monocular 3D human-pose estimation from static images is a challenging problem, due to the curse of dimensionality and the ill-posed nature of lifting 2D-to-3D. In this paper, we propose a Deep Conditional Variational Autoencoder based model that synthesizes diverse anatomically plausible 3D-pose samples conditioned on the estimated 2D-pose. We show that CVAE-based 3D-pose sample set is consistent with the 2D-pose and helps tackling the inherent ambiguity in 2D-to-3D lifting. We propose two strategies for obtaining the final 3D pose- (a) depth-ordering/ordinal relations to score and weight-average the candidate 3D-poses, referred to as OrdinalScore, and (b) with supervision from an Oracle. We report close to state of-the-art results on two benchmark datasets using OrdinalScore, and state-of-the-art results using the Oracle. We also show that our pipeline yields competitive results without paired image-to-3D annotations. The training and evaluation code is available at https://github.com/ssfootball04/generative_pose.
1. Introduction
The paper addresses ambiguous 2D-to-3D lifting by generating diverse 3D-pose candidates from estimated 2D poses, then selecting or combining them using ordinal relations or Oracle supervision.
- Motivation: Monocular 3D human-pose estimation is challenging because real-world variation and limited dataset coverage make the 2D-to-3D problem inherently ill-posed.Regression from estimated 2D pose can ignore this ambiguity.
- Results: The modular pipeline remains competitive when its 2D-to-3D module is trained on separate motion-capture data without paired image-to-3D annotations.This setting uses data with no intersection between the training motion-capture data and evaluation image-to-3D dataset.
- Method: OrdinalScore ranks candidate poses using predicted pairwise joint-depth relations and computes their probability-weighted expectation.Ordinal relations are converted into scores, normalized with Softmax, and used to obtain the final pose.
- Results: Oracle supervision selects the generated sample closest to the ground-truth 3D pose and achieves state-of-the-art results on two benchmark datasets.The paper presents this as an upper-bound evaluation of the generative model.
- Method: The CVAE generates diverse 3D-pose samples conditioned on the estimated 2D pose.Multiple latent codes produce a sample set consistent with the estimated 2D pose.
2. Related Work
Prior work uses pose priors, anatomical constraints, reprojection objectives, and heuristic sampling to lift 2D poses into 3D. This paper instead emphasizes probabilistic generation of diverse 3D poses and ordinal scoring to address lifting ambiguity.
- 2. Related Work: Earlier lifting methods use shape bases, reprojection, anatomical constraints, or learned pose priors to estimate 3D pose from 2D pose.These approaches regularize pose estimates through sparse representations, limb lengths, joint-angle limits, or learned priors.
- 2. Related Work: Some methods combine image data with unpaired motion-capture data, while another baseline predicts a single 3D pose from each 2D pose.The paper distinguishes its approach by generating a diverse set of 3D poses.
- 2. Related Work: Previous hypothesis-generation methods obtain multiple poses through nearest neighbors, kinematic flipping, or latent-variable generative models.The paper argues that an assumption linking 2D and 3D poses can overlook inherent lifting ambiguity.
- 2. Related Work: The proposed approach uses a CVAE for pose generation and geometry-inspired ordinal scoring to merge multiple 3D-pose samples.This combines probabilistic candidate generation with a structured scoring mechanism.
3. Proposed Approach
The proposed pipeline generates diverse 3D poses conditioned on estimated 2D pose, then uses ordinal depth relations to aggregate candidates or an Oracle to select the closest sample.
- Pipeline: The pipeline combines 2DPoseNet, MultiPoseNet, OrdinalNet, OrdinalScore, and Oracle-based selection for monocular 3D-pose estimation.2DPoseNet estimates 2D pose; MultiPoseNet generates candidates; OrdinalNet predicts joint-depth relations; the final pose is obtained by scoring or Oracle selection.
- MultiPoseNet: MultiPoseNet extends a baseline regressor into a CVAE that models multimodal 2D-to-3D mappings and generates diverse candidate poses.At inference, latent codes sampled from a Gaussian prior are combined with the estimated 2D pose and passed through the decoder.
- OrdinalNet: OrdinalNet predicts lesser-than, greater-than, and equal-depth relations between joints and converts them into a 16 × 16 ordinal relation matrix.The relations are obtained from ordinal maps using non-maximal suppression and nearest-joint association.
- OrdinalScore: OrdinalScore assigns each generated pose a score from agreement between its induced ordinal matrix and the predicted matrix, then aggregates candidates with temperature-based Softmax weights.The final pose is computed as the expectation over the candidate distribution; temperature controls the contribution of high- and low-scoring samples.
- Oracle: The Oracle selects the generated pose closest to ground truth, providing an upper-bound accuracy estimate for the generative model.The closest sample is selected from the candidate set using access to the ground-truth 3D pose.
4. Experiments
The experiments evaluate the proposed approach on benchmark datasets, compare it with state-of-the-art methods, and use ablations to analyze the generative model.
- Experiments: The empirical evaluation covers benchmark comparisons, implementation details, quantitative results, and ablation studies of the generative model.The section first describes the benchmarks and implementation before presenting comparisons and ablations.
4.1. Datasets
The experiments use CMU Mocap for training pipeline modules and HumanEva-I as a standard benchmark for 3D-pose estimation.
- Training data: CMU Mocap contains diverse 3D poses from 144 subjects performing different actions, with 2D projections generated from virtual cameras.The resulting 2D-to-3D data train MultiPoseNet and the baseline in the unpaired setting.
- Benchmark datasets: HumanEva-I contains 3 subjects, 3 camera views, and fewer actions than Human3.6M, and is used as a standard 3D-pose benchmark.The dataset includes subjects S1, S2, and S3.
4.2. Implementation Details
Implementation preprocesses cropped RGB images and normalized 2D and 3D poses, with joint counts differing between Human3.6M and HumanEva-I.
- Preprocessing: Inputs use tight 224 × 224 crops around people obtained from ground-truth bounding boxes.The 3D poses are represented in camera coordinates before normalization.
- Normalization: The pipeline subtracts means, divides by standard deviations, and zero-centers 3D poses around the hip joint.The 2D and 3D pose inputs and outputs receive standard normalization.
- Pose representation: Human3.6M uses N=16 2D joints and N=17 3D joints, whereas HumanEva-I uses N=16 joints for both representations.These counts specify the pose dimensionality used by the implementation.
4.3. Quantitative Evaluation
The evaluation compares predicted, ground-truth, and Oracle-based pose selection across Human3.6M and HumanEva-I, using paired and unpaired training settings. Results show competitive or state-of-the-art performance, including when image-to-3D annotations are unavailable.
- Human3.6M: The evaluation reports predicted-ordinal, ground-truth-ordinal, and Oracle variants, with Oracle described as producing the best results.Protocol 1 uses MPJPE without rigid alignment, while Protocol 2 uses PA MPJPE after rigid alignment.
- Human3.6M: In paired Human3.6M experiments, PRED Ordinals achieve competitive results, GT Ordinals approach state of the art, and Oracle supervision outperforms existing methods.The remaining gap is attributed to methods using additional ordinal data, temporal information, or soft-argmax, which are compatible with this approach.
- Human3.6M: Without image-to-3D annotations, PRED Ordinals outperform the Baseline regression model, while GT Ordinals and Oracle further improve performance.The lifting module is trained on a separate motion-capture library, while 2D-pose and ordinal modules use Human3.6M.
- HumanEva-I: On HumanEva-I, Oracle achieves state-of-the-art results, while PRED Ordinals and GT Ordinals are close to state of the art.All modules are trained using HumanEva-I, and error is reported after rigid transformation.
- Ablations: The ablation evaluates how increasing sample count affects Oracle, OrdinalScore, and MEAN estimates, and compares MultiPoseNet with Baseline sampling.The supplied figure captions identify these as the two ablation comparisons.
4.4. OrdinalNet Accuracy
OrdinalNet accuracy is evaluated by comparing predicted and ground-truth ordinal relations on Human3.6M and HumanEva-I.
- OrdinalNet Accuracy: OrdinalNet reaches 86.8% accuracy on Human3.6M and 81% on HumanEva-I validation sets.Accuracy compares the ground-truth ordinals M with the predicted ordinals M-hat.
4.5. Ablation Studies
Increasing the sample set improves ordinally scored estimates, while CVAE sampling outperforms independent Gaussian baseline sampling and produces candidates more likely near the ground truth.
- Effect of Increasing Sample Set Size: The MEAN estimate improves with more samples but quickly saturates, whereas Oracle, PRED Ordinals, and GT Ordinals continue improving.The authors interpret the continued improvement of ordinal estimates as evidence that ordinal scoring effectively weighted-averages generated samples.
- Sampling Baseline: MultiPoseNet produces more diverse samples that are more likely to lie near the ground-truth pose than baseline sampling with variance 100.The samples are visualized after mapping them to Euclidean space with ISOMAP.
- HumanEva-I Evaluation: HumanEva-I results are reported as mean reconstruction error in millimeters after rigid transformation.The table compares the model with previous work on the HumanEva-I benchmark.
- Sampling Baseline: MultiPoseNet improves its estimate by close to 20mm as sample count increases, while baseline sampling performs poorly and worsens at 400mm variance.Baseline samples each joint independently from Gaussian distributions centered on the baseline regression output.
4.6. Sample Diversity
The generated pose set exhibits meaningful, body-part-dependent variation rather than collapsing around one prediction, with especially high variance at difficult wrist and elbow joints.
- Qualitative Analysis: Generated candidates show meaningful variation across body parts and poses, with relatively higher variance around the difficult-to-predict wrist and elbow joints.The qualitative analysis examines mean poses, per-joint standard deviations, and candidate 3D poses for two test images.
5. Conclusion and Future Work
The paper presents a CVAE-based framework that generates diverse 3D-pose candidates, scores them with ordinal relations, and achieves strong benchmark performance. It also remains competitive without paired image-to-3D annotations.
- Conclusion and Future Work: The framework samples 3D-pose candidates with a CVAE, then scores and weighted-averages them using ordinal relations predicted by a deep CNN.The generative model is intended to represent ambiguity in lifting 2D poses into 3D.
- Conclusion and Future Work: The method achieves close-to-state-of-the-art results with OrdinalScore and state-of-the-art results with an Oracle on two benchmark datasets.The Oracle has access to the ground-truth 3D pose when selecting the closest generated sample.
- Conclusion and Future Work: The pipeline remains competitive when trained without paired image-to-3D annotations.This supports using separately collected 2D-pose annotations and indoor motion-capture data.