Source-linked AI summary
Probabilistic Modeling for Human Mesh Recovery
Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, Kostas Daniilidis
TL;DR
3D human reconstruction from 2D evidence is ambiguous, but most recent methods output a single estimate. The paper models a conditional distribution of plausible 3D poses with Conditional Normalizing Flows, achieving state-of-the-art performance across supported settings while enabling downstream inference with additional evidence.
Problem
3D human reconstruction from 2D evidence is fundamentally ambiguous, whereas recent approaches typically use deterministic single-point estimates.
Method
The paper uses Conditional Normalizing Flows to predict a distribution of plausible 3D poses conditioned on the input.
Results
The model is on par with the state of the art for conventional single-estimate evaluation and improves downstream body-model fitting and multi-view refinement.
Takeaways & Limitations
A single trained probabilistic model can provide diverse samples, likelihoods, and a mode for conventional and downstream test-time estimation.
Takeaways & Limitations
The main experiments focus on single-image human mesh recovery, with only brief investigation of other settings; the pose uncertainty model is limited to SMPL pose parameters.
Abstract
from arXiv · showhide
This paper focuses on the problem of 3D human reconstruction from 2D evidence. Although this is an inherently ambiguous problem, the majority of recent works avoid the uncertainty modeling and typically regress a single estimate for a given input. In contrast to that, in this work, we propose to embrace the reconstruction ambiguity and we recast the problem as learning a mapping from the input to a distribution of plausible 3D poses. Our approach is based on the normalizing flows model and offers a series of advantages. For conventional applications, where a single 3D estimate is required, our formulation allows for efficient mode computation. Using the mode leads to performance that is comparable with the state of the art among deterministic unimodal regression models. Simultaneously, since we have access to the likelihood of each sample, we demonstrate that our model is useful in a series of downstream tasks, where we leverage the probabilistic nature of the prediction as a tool for more accurate estimation. These tasks include reconstruction from multiple uncalibrated views, as well as human model fitting, where our model acts as a powerful image-based prior for mesh recovery. Our results validate the importance of probabilistic modeling, and indicate state-of-the-art performance across a variety of settings. Code and models are available at: https://www.seas.upenn.edu/~nkolot/projects/prohmr.
1. Introduction
3D human reconstruction is inherently ambiguous, yet conventional methods typically return one deterministic estimate. This paper instead predicts a conditional distribution of plausible poses, supporting both standard single-estimate evaluation and downstream inference with additional evidence.
- 3D human reconstruction from 2D observations is fundamentally ambiguous, but recent approaches commonly return a single deterministic estimate.
- At test time, the same trained model supports image-based body-model fitting and multi-view information consolidation without task-specific retraining.These applications use the probabilistic prediction to combine 2D evidence, image evidence, and cross-view consistency.
- The probabilistic model supports diverse sampling, likelihood computation, and fast closed-form mode estimation.These properties allow the mode to serve applications requiring a single 3D estimate.
- The model performs on par with state-of-the-art single-estimate methods in conventional evaluations.This indicates that replacing point regression with density estimation need not significantly reduce performance.
2. Related work
Related work spans regression, multiple-hypothesis prediction, and normalizing-flow approaches for ambiguous 3D human reconstruction. This work differs from generic pose-prior methods by conditioning the learned pose distribution on 2D image evidence.
- Regression: Single-image mesh recovery commonly regresses parameters of a parametric human model from a deep network, following the HMR paradigm.The paper’s regression network follows HMR principles but replaces a single 3D pose estimate with a distributional prediction.
- Multiple hypotheses: Multiple-hypothesis methods address ambiguities from occlusions, truncations, and depth by generating several poses consistent with 2D keypoint evidence.Examples include compositional models with anatomical constraints and mixture-density networks producing a fixed number of proposals.
- Normalizing flows: Normalizing flows represent complex distributions through invertible transformations of a simple base distribution.The passage situates this family within probabilistic modeling and names MADE, NICE, MAF, RealNVP, and Glow as examples.
- Conditioned pose priors: Prior flow-based human-pose methods learn generic plausible-pose priors from unpaired MoCap data, whereas this work learns a prior conditioned on 2D image evidence.The paper mainly studies human mesh recovery from a single image and briefly considers 3D pose estimation from 2D keypoints.
3. Method
The method models plausible SMPL pose distributions conditioned on 2D evidence using conditional normalizing flows, while predicting shape and camera parameters separately. Its invertible design supports sampling, likelihood computation, mode-based prediction, and downstream optimization with mixed supervision.
- Model design: ProHMR maps an input image to a distribution of plausible SMPL pose parameters while producing point estimates for shape and camera.The image encoder produces context c, which conditions the pose distribution and is also decoded to β and π.
- Model design: Conditional normalizing flows represent the pose distribution through a mapping bijective in latent variable z and pose θ.The model uses θ = f(z; c) and z = f^-1(θ; c), with c obtained from the input image.
- Model design: Normalizing flows are selected over MDNs and VAEs for distributional expressiveness and accessible likelihood computation.The likelihood of a sample is described as crucial for downstream tasks.
- Model design: The flow architecture enables fast sampling and likelihood computation, while its z-independent Jacobian determinant permits efficient mode computation.The mode provides a single prediction when required, and the likelihood supports downstream use of the probabilistic output.
- Training objective: Training combines negative log-likelihood with expectation-based 2D and adversarial losses and explicit mode supervision when annotations are available.The expectation is rewritten for differentiability, while L3D uses available 3D joints or SMPL-parameter annotations.
- Downstream applications: The learned conditional distribution supports test-time downstream applications, including image-based fitting with an image-conditioned pose prior.The same trained model is used without task-specific retraining, and the learned prior improves fitting results qualitatively and quantitatively.
4. Experimental evaluation
The evaluation shows that ProHMR matches state-of-the-art regression accuracy while providing stronger probabilistic capabilities for multiple hypotheses, fitting, and multi-view refinement. Its learned image-conditioned prior improves fitting and prevents implausible pose drift when 2D evidence is weak.
- Human mesh recovery: ProHMR achieves accuracy comparable with the state of the art when its distribution mode is used as a regressor.This demonstrates that recasting prediction as density estimation need not substantially reduce conventional regression performance.
- Multiple hypotheses: The evaluation measures multiple-hypothesis quality using errors for small sample counts and the minimum error among samples drawn from each distribution.Comparisons include Biggs et al., Mixture Density Networks, and Conditional VAE variants.
- Model fitting: ProHMR’s learned image-conditioned prior improves model-fitting accuracy beyond regression and EFT under the reported settings.In 3DPW, ProHMR improves relative error by 4.7mm versus 2.6mm for EFT; with ground-truth 2D keypoints on Human3.6M, the improvements are 6.3mm versus 3.1mm.
- Multi-view refinement: The experiments evaluate multi-view refinement, including comparisons with per-view predictions and rotation averaging on Human3.6M and Mannequin Challenge.The refinement uses the learned image-conditioned prior in uncalibrated multi-view scenarios.
- Ablations and additional evaluations: Explicitly supervising the distribution mode is crucial for competitive performance in conventional regression tasks.The ablation compares ProHMR trained with and without Lmode and also evaluates skeleton-based 2D pose lifting.
- Model fitting: Unlike SMPLify, the image-based prior keeps poses from deviating far from image evidence when keypoint detections are missing or have very low confidence.The qualitative comparison reports more realistic reconstructions in these cases.
5. Summary
The paper presents ProHMR, a probabilistic model that maps 2D evidence to a distribution of plausible 3D human poses using Conditional Normalizing Flows. It supports diverse sampling, likelihood computation, and fast mode estimation, with effectiveness demonstrated across several benchmarks and settings.
- Summary: ProHMR models 3D human mesh recovery from 2D evidence as a conditional distribution rather than a single point estimate.The approach uses Conditional Normalizing Flows to represent plausible poses.
- Summary: The model provides diverse sampling, efficient likelihood computation, and a fast closed-form mode solution.These capabilities support both probabilistic prediction and applications requiring a single estimate.
- Summary: Empirical results across several benchmarks demonstrate the effectiveness of the probabilistic model.The paper also identifies extending the approach to other articulated or non-articulated objects and modeling depth-size ambiguity as future work.