Source-linked AI summary
Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flows
Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, Bastian Wandt
TL;DR
Monocular 3D pose estimation is ill-posed because depth ambiguities and occlusions permit multiple feasible poses, while many methods estimate only one. The paper uses a conditional normalizing flow to model the full posterior, exploiting the deterministic 3D-to-2D mapping and detector uncertainty. On Human3.6M and MPI-INF-3DHP, it outperforms comparable methods in most metrics, especially on highly uncertain examples.
Problem
Monocular 3D pose estimation is ill-posed because depth ambiguities and occlusions can make multiple 3D poses consistent with one image.
Method
A conditional normalizing flow maps 3D poses to 2D poses plus latent variables, using heatmap-derived uncertainty to generate diverse posterior hypotheses.
Results
The method outperforms comparable methods in most metrics on Human3.6M and MPI-INF-3DHP, with a large margin on highly uncertain Human3.6M samples.
Takeaways & Limitations
Generated hypotheses reflect detector uncertainty and ambiguous joint depth while remaining geometrically feasible through a learned 3D pose prior.
Abstract
from arXiv · showhide
3D human pose estimation from monocular images is a highly ill-posed problem due to depth ambiguities and occlusions. Nonetheless, most existing works ignore these ambiguities and only estimate a single solution. In contrast, we generate a diverse set of hypotheses that represents the full posterior distribution of feasible 3D poses. To this end, we propose a normalizing flow based method that exploits the deterministic 3D-to-2D mapping to solve the ambiguous inverse 2D-to-3D problem. Additionally, uncertain detections and occlusions are effectively modeled by incorporating uncertainty information of the 2D detector as condition. Further keys to success are a learned 3D pose prior and a generalization of the best-of-M loss. We evaluate our approach on the two benchmark datasets Human3.6M and MPI-INF-3DHP, outperforming all comparable methods in most metrics. The implementation is available on GitHub.
1. Introduction
Monocular 3D pose recovery is ambiguous because depth and occlusion can leave multiple poses consistent with one image. The paper models this as a probabilistic inverse problem, using normalizing flows and detector uncertainty to generate diverse hypotheses and reports strong benchmark performance.
- Problem: Monocular 3D pose reconstruction is ill-posed because depth ambiguities and occluded body parts can support multiple plausible poses.Existing approaches commonly estimate a single solution, despite these ambiguities.
- Probabilistic formulation: The method models the full posterior of plausible 3D poses conditioned on a monocular image rather than assuming a single solution.It treats 3D-to-2D projection as deterministic while allowing multiple inverse mappings.
- Uncertainty modeling: The approach explicitly incorporates uncertainty information from 2D detector heatmaps to model occlusions and uncertain detections.This preserves information that methods using only 2D coordinates discard.
- Method: A conditional normalizing flow links 3D poses with 2D poses and latent variables, enabling repeated latent sampling to produce arbitrarily many hypotheses.The latent vector encodes depth information lost during projection, while fitted heatmap Gaussians provide detector-uncertainty conditions.
- Evaluation: The model is evaluated on Human3.6M and MPI-INF-3DHP and outperforms comparable methods in most metrics, with a large margin on highly uncertain Human3.6M samples.The contributions also include a normalizing-flow posterior model and a generalized best-of-M loss.
2. Related Work
Prior work spans two-stage 2D-to-3D lifting, multi-hypothesis generation, and normalizing flows. The paper distinguishes its approach by modeling image-conditioned pose posteriors with flexible hypothesis counts and explicit detector uncertainty.
- 2D-to-3D lifting: Two-stage methods first detect 2D keypoints and then lift them to 3D, allowing 2D detectors trained across varied scenes to reduce indoor-scene bias.These methods decouple 2D localization from the harder problem of 3D depth estimation.
- Multi-hypothesis estimation: Earlier multi-hypothesis methods sample poses heuristically, use occupancy matrices, mixture density networks, or conditional variational autoencoders.These approaches differ in how they represent and rank multiple 3D pose candidates.
- Multi-hypothesis estimation: Unlike methods with fixed hypothesis counts or rejection sampling, the normalizing-flow approach can generate arbitrarily many hypotheses without specifying their number in advance.The paper presents this as a flexibility advantage for modeling posterior distributions.
- Normalizing flows: Normalizing flows are bijective transformations that support both sampling from a modeled distribution and exact density estimation.Their bidirectional structure motivates using the known 3D-to-2D projection during training.
- Positioning: The proposed method uses a normalizing flow to model the image-conditioned posterior directly, rather than using flows only as a 3D pose prior.It also fits Gaussians to heatmaps to capture detector uncertainty beyond maximum heatmap values.
3. Method
The method learns a conditional normalizing flow that maps 3D poses to 2D observations plus latent variables, enabling diverse inverse hypotheses while modeling detector uncertainty. Training combines supervised forward projection, distribution matching, geometric plausibility, detector-uncertainty matching, and a generalized best-of-M objective.
- Heatmap condition: The model conditions a normalizing flow on 2D detector heatmap uncertainty rather than only 2D joint coordinates.Each predicted heatmap is fitted with a 2D Gaussian whose parameters are encoded as the conditioning input.
- Conditional normalizing flow: A bijection maps a 3D pose to a 2D pose concatenated with a latent vector that captures depth information lost during projection.At inference, repeatedly sampling the latent vector produces arbitrarily many 3D hypotheses through the inverse path.
- Conditional normalizing flow: The Real-NVP architecture uses affine coupling blocks, permutations, and soft clamping to preserve invertibility and prevent unstable exponential scaling.The final flow output is split into the 2D pose and latent vector.
- Heatmap condition: The uncertainty-matching loss is masked when the detector is sufficiently certain and otherwise constrains covariance estimates relative to ground-truth variances.The mask uses a standard-deviation threshold, while the loss penalizes diagonal covariance entries below corresponding ground-truth values.
- Optimization: Training supervises the forward 3D-to-2D path with L1 loss and matches latent-output distributions using maximum mean discrepancy.Gradients from the MMD term are blocked with respect to the predicted 2D pose.
- Optimization: The inverse path adds a discriminator, detector-uncertainty matching, and a generalized best-of-M loss to improve pose plausibility and convergence.The best-of-M variant selects the k hypotheses with lowest MPJPE and minimizes the L1 distance to their mean.
4. Experiments
Experiments evaluate the method on Human3.6M, its highly ambiguous H36MA subset, and MPI-INF-3DHP, including uncertainty-focused, diversity, ablation, and generalization analyses.
- Datasets and metrics: Human3.6M evaluation reports MPJPE under Protocol 1 and Protocol 2, together with the CPS metric for set-valued pose predictions.Protocol 1 uses direct joint-coordinate error, Protocol 2 applies rigid alignment, and CPS requires all joints to remain below a threshold.
- Human3.6M: 4.1% and 10.7% improvements over previous best scores are achieved under Protocol 1 and Protocol 2, respectively.The comparison is reported for the Human3.6M evaluation against previous best scores.
- Ambiguous examples: On H36MA, the method significantly outperforms both competitors, with a larger margin than on the full test set.H36MA contains samples with at least one fitted Gaussian standard deviation above 5 px, representing 6.4% of the test split.
- Sample diversity: Generated hypotheses reflect detector uncertainty and exhibit greatest average variance in depth: 42.4 mm versus 18.3 mm in x and 17.3 mm in y.Ankle, elbow, and wrist joints account for the highest variance, and the depth diversity is meaningful despite not being explicitly optimized.
- Sample diversity: As the number of H36MA hypotheses increases, best-hypothesis MPJPE continues improving while a fitted-Gaussian noise baseline saturates earlier at higher error.The baseline adds fitted-Gaussian noise to z0 predictions and assumes a constant Gaussian for depth.
- Ablation studies: Removing each proposed component degrades performance, while omitting the heatmap condition causes a large CPS drop and omitting the discriminator worsens worst-hypothesis Protocol 2 error from 86.8 mm to 284.1 mm.The discriminator supports geometrical feasibility, and the heatmap condition supplies uncertainty information needed for some individual joints.
5. Conclusion
The paper presents a normalizing-flow method for ambiguous monocular 3D pose estimation that preserves detector uncertainty and generates diverse, feasible hypotheses. It outperforms prior multi-hypothesis approaches in most metrics, especially on highly ambiguous Human3.6M examples.
- Contributions: Normalizing flows use known 3D-to-2D projection, heatmap uncertainty, a 3D pose discriminator, and a generalized best-of-M loss to model feasible pose distributions.The method produces meaningful diversity along ambiguous joint depth while retaining uncertainty information from the detector.
- Results: The method outperforms previous multi-hypothesis approaches in most metrics, especially on Human3.6M samples with highly ambiguous detections.The conclusion emphasizes both broad benchmark performance and the challenging ambiguity-focused subset.
A.1. Condition Influence
The condition influence analysis shows that heatmap conditioning and its uncertainty-matching loss progressively increase meaningful diversity in generated pose hypotheses.
- Condition influence: Heatmap conditioning alone produces meaningful x- and y-direction diversity, while optimizing LHM further increases diversity that reflects detector uncertainty and depth ambiguity.The qualitative comparison is presented in Fig. 7.
A.2. Competitor Comparison
Qualitative comparisons show that the proposed method generates substantially greater diversity for occluded joints than competing methods.
A.3. High Confidence Detections
When 2D detections are highly confident, generated 3D hypotheses remain tightly clustered in image directions while retaining meaningful diversity along ambiguous depth directions. Heatmap-fitted Gaussian variances correlate with 2D error, supporting their use as a detector-uncertainty surrogate.
- High-confidence detections produce hypotheses with low x- and y-direction variance but meaningful diversity along ambiguous joint depths.The hypotheses remain consistent with the input image while varying in depth.
- Gaussian variances correlate with 2D error across Human3.6M joints, making them a good surrogate for 2D detector uncertainty.The comparison uses the mean fitted-Gaussian standard deviations for each joint in the test split.
C. Performance Lower Number of Samples
The method remains competitive when evaluated with fewer generated hypotheses on Human3.6M, while the authors emphasize that larger sample counts are needed to model the full posterior distribution.
- The evaluation compares lower hypothesis counts on Human3.6M under Protocol 1 (MPJPE) and Protocol 2 (PMPJPE), including sampled comparisons with Li et al. [28].Rows marked with * use samples drawn from the compared models.
- The authors state that modeling the full posterior distribution requires a larger number of samples.
- Our method outperforms Li et al. [28] in almost every evaluated setting and metric.The comparison includes the authors' M = 200 generated hypotheses and additional sampling results from Li et al. [28].
D. Inference Time
Inference is dominated by 2D detection and Gaussian fitting, while batch processing makes generating many hypotheses add little runtime overhead. The qualitative figures visualize model outputs and comparisons using multiple hypotheses.
- The 2D detector takes 32 ms and Gaussian fitting takes 70 ms, dominating the reported inference time.Measurements use PyTorch 1.7.1 on an NVIDIA GeForce RTX 3090 with CUDA 11.4.
- Generating 1000 samples takes 5.1 ms versus 4.6 ms for one sample, indicating nearly no batch-processing overhead.
- Qualitative results compare the full model with variants without LHM or conditioning, showing more than three hypotheses only for the most ambiguous joint.
- A competing-method visualization compares against methods [28, 40], while Li et al. [28] can generate only five pose hypotheses.
- Qualitative examples from Human3.6M and MPI-INF-3DHP show 50 generated pose hypotheses from two perspectives for images with low 2D detector uncertainty.