Source-linked AI summary
Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network
Chen Li, Gim Hee Lee
TL;DR
Monocular 3D pose estimation is ill-posed because depth ambiguity and occlusions allow multiple feasible poses for the same 2D input. The paper uses a multimodal mixture-density network to generate such hypotheses, achieving state-of-the-art Human3.6M results and demonstrating generalization to MPII and MPI-INF-3DHP.
Problem
Monocular 3D pose estimation is ambiguous because multiple feasible 3D poses can satisfy the same 2D joints, especially with nonrigidity and occlusions.
Method
The method uses an MDN whose multimodal Gaussian components generate multiple 3D pose hypotheses, with means representing poses and coefficients and variances representing uncertainty.
Results
5.5% improvement over the previous best 55.8 mm result on Human3.6M, while the method also achieves state-of-the-art best-hypothesis and multi-view results and generalizes to MPII and MPI-INF-3DHP.
Takeaways & Limitations
Consistent 2D reprojections of generated hypotheses support modeling depth ambiguity with multiple solutions rather than a single pose estimate.
Takeaways & Limitations
The optimization assumes a uniform prior over Gaussian means and variances and a Dirichlet conjugate prior over mixture coefficients.
Abstract
from arXiv · showhide
3D human pose estimation from a monocular image or 2D joints is an ill-posed problem because of depth ambiguity and occluded joints. We argue that 3D human pose estimation from a monocular input is an inverse problem where multiple feasible solutions can exist. In this paper, we propose a novel approach to generate multiple feasible hypotheses of the 3D pose from 2D joints.In contrast to existing deep learning approaches which minimize a mean square error based on an unimodal Gaussian distribution, our method is able to generate multiple feasible hypotheses of 3D pose based on a multimodal mixture density networks. Our experiments show that the 3D poses estimated by our approach from an input of 2D joints are consistent in 2D reprojections, which supports our argument that multiple solutions exist for the 2D-to-3D inverse problem. Furthermore, we show state-of-the-art performance on the Human3.6M dataset in both best hypothesis and multi-view settings, and we demonstrate the generalization capacity of our model by testing on the MPII and MPI-INF-3DHP datasets. Our code is available at the project website.
1. Introduction
3D pose recovery from monocular inputs is ambiguous because multiple geometrically feasible poses can match the same 2D joints. The paper addresses this with a mixture-density network that generates multiple hypotheses and reports strong benchmark performance.
- Monocular 3D pose estimation is an inverse problem in which depth ambiguity, nonrigidity, and occlusions permit multiple poses consistent with one 2D pose.
- The paper argues that generating multiple geometrically feasible hypotheses is more appropriate than fitting a single estimate to an inherently ambiguous inverse problem.
- The proposed MDN models a multimodal mixture of Gaussians, using component means as pose hypotheses and mixing coefficients and variances as uncertainty measures.
- The network combines a feature extractor that lifts 2D joints into a feature space with a hypotheses generator that produces multiple 3D poses.
- The paper presents multiple-hypothesis generation as an underexplored direction and identifies its MDN formulation as a contribution to 3D human pose estimation.
- The method reports state-of-the-art results on Human3.6M for best-hypothesis and multi-view evaluation, including cases with one or two missing limb joints.
2. Related Work
Prior work either estimates 3D poses directly from images or decouples image processing into 2D detection and 3D reconstruction from joints. The paper builds on multiple-hypothesis generation while replacing a traditional GMM with an MDN.
- Existing approaches use either end-to-end image-to-3D CNNs or a two-stage pipeline that detects 2D joints before estimating 3D pose.
- The decoupled approach supports training 2D estimators across indoor and in-the-wild domains, but 2D joints contain less information and therefore create more ambiguities.
- Jahangiri and Yuille addressed the ill-posed 2D-to-3D problem by sampling diverse poses from a learned 3D Gaussian mixture model subject to reprojection-error constraints.
- The proposed network follows the two-stage design, using detected 2D joints as input to a feature extractor and hypotheses generator.
- This paper replaces the traditional GMM approach with an MDN that combines a neural network with a mixture-density model to represent conditional distributions.
3. Our Mixture Density Network
The model maps detected 2D joints to a conditional multimodal Gaussian mixture whose component means represent alternative 3D pose hypotheses. It uses a two-stage network and regularized training so hypotheses can remain diverse while collapsing toward a single distribution for less ambiguous poses.
- Model Representation: The network maps 2D joint detections x ∈R2N to mixture parameters Θ = {µ, σ, α} for conditional 3D pose distributions.The parameters are produced by a deep function f, with means, variances, and mixing coefficients depending on the input 2D pose.
- Model Representation: Unlike unimodal-Gaussian training, the MDN minimizes the negative log-likelihood of a multimodal mixture-of-Gaussians to estimate multiple feasible poses.Its outputs include Gaussian means and variances together with mixing coefficients.
- Model Representation: Each Gaussian-kernel mean µi represents one 3D pose hypothesis, while M determines the number of generated hypotheses.The mixture coefficients weight the kernels, and the conditional 3D-pose density is their linear combination.
- Model Representation: When component means and variances are similar, the mixture degenerates to a single Gaussian, especially for simple 2D poses without occlusions or missing joints.This allows the model to represent fewer distinct solutions when the input is less ambiguous.
- Network Architecture: The architecture uses a linear projection to 1024 features followed by two residual blocks with linear layers, batch normalization, dropout, and ReLU activations.Residual connections link each block's input and output.
4. Experiments
Experiments evaluate the method on Human3.6M, MPII, and MPI-INF-3DHP, including multi-view, missing-joint, kernel-count, and reprojection analyses. The results report strong Human3.6M performance, generalization to unseen scenes, and multiple hypotheses that remain consistent with the 2D inputs.
- Human3.6M results: 5.5% improvement over the previous best 55.8 mm and 16.2% over the baseline architecture are reported on Human3.6M.The method also outperforms a prior multiple-hypothesis approach by 22.5% and consistently outperforms other methods under Protocol #2.
- Human3.6M results: Multi-view evaluation uses calibrated cameras to disambiguate hypotheses, with the approach achieving the best performance among spatial- and temporal-constraint methods.Poses are transformed into global coordinates, selected for cross-view consistency, and averaged.
- Ablation Study: The MDN performs better than a single-Gaussian baseline, while performance improves little beyond five kernels, leading to the choice M=5.The kernel count balances accuracy against computational complexity.
- Ablation Study: All five hypotheses have nearly identical 2D reprojections, while challenging poses produce more diverse 3D hypotheses than simple standing poses.The paper attributes greater diversity to less input information, increased ambiguity, and occlusions; adding a reprojection-consistency constraint was unnecessary.
5. Conclusion
The paper uses a mixture density network to generate multiple feasible 3D pose hypotheses from 2D inputs. These hypotheses remain consistent in 2D reprojections, while results across datasets demonstrate state-of-the-art performance and generalization.
- The mixture density network generates multiple feasible 3D pose hypotheses for the inverse problem from 2D inputs.
- Consistent 2D reprojections suggest that the generated hypotheses model ambiguity along the joints’ depth.
- State-of-the-art results in best-hypothesis and multi-view settings, plus MPII and MPI-INF-3DHP evaluations, demonstrate performance and generalization capacity.