Source-linked AI summary
End-to-end Recovery of Human Shape and Pose
Angjoo Kanazawa, Michael J. Black, David W. Jacobs, Jitendra Malik
TL;DR
Single-image 3D human reconstruction is underconstrained by sparse 2D evidence and ambiguous 2D-to-3D mappings. HMR directly regresses full mesh parameters from image pixels, combining reprojection loss with an adversarial human-body prior. It produces reasonable reconstructions even without paired 2D-to-3D supervision and outperforms prior mesh methods on 3D joint error and runtime.
Problem
Single-view reconstruction lacks large-scale in-the-wild 3D annotations, while reprojection permits multiple anthropometrically implausible 3D bodies.
Method
HMR directly predicts SMPL pose, shape, and camera parameters from image features using iterative regression, joint reprojection, and an adversarial prior over real human bodies.
Results
HMR produces reasonable 3D reconstructions without paired 2D-to-3D supervision and outperforms previous mesh-output approaches in 3D joint error and runtime.
Takeaways & Limitations
The results suggest that increasingly large collections of images with 2D labels can support continued improvement without requiring equally difficult in-the-wild 3D ground truth.
Takeaways & Limitations
MPJPE does not consistently correlate with visual quality, so joint-location error alone may not fully evaluate reconstructed meshes.
Abstract
from arXiv · showhide
We describe Human Mesh Recovery (HMR), an end-to-end framework for reconstructing a full 3D mesh of a human body from a single RGB image. In contrast to most current methods that compute 2D or 3D joint locations, we produce a richer and more useful mesh representation that is parameterized by shape and 3D joint angles. The main objective is to minimize the reprojection loss of keypoints, which allow our model to be trained using images in-the-wild that only have ground truth 2D annotations. However, the reprojection loss alone leaves the model highly under constrained. In this work we address this problem by introducing an adversary trained to tell whether a human body parameter is real or not using a large database of 3D human meshes. We show that HMR can be trained with and without using any paired 2D-to-3D supervision. We do not rely on intermediate 2D keypoint detections and infer 3D pose and shape parameters directly from image pixels. Our model runs in real-time given a bounding box containing the person. We demonstrate our approach on various images in-the-wild and out-perform previous optimization based methods that output 3D meshes and show competitive results on tasks such as 3D joint location estimation and part segmentation.
1. Introduction
HMR reconstructs a full 3D human mesh directly from a single RGB image, addressing the limitations of sparse joint outputs, limited in-the-wild 3D supervision, and ambiguous 2D-to-3D mapping. It combines reprojection-based fitting with an adversarial prior and demonstrates end-to-end, mesh-based recovery with and without paired 2D-to-3D data.
- HMR represents the human body as a full 3D mesh parameterized by 3D joint angles and a low-dimensional linear shape space.
- The recovered mesh supports applications including segmentation, animation, measurement, manipulation, and retargeting, while representing the full body under occlusion and truncation.
- Unlike joint-only approaches, the model predicts joint rotations and mesh shape, capturing limb and head orientation while preserving valid limb lengths and symmetry.
- The model combines 2D joint reprojection with an adversarial discriminator trained on real human shapes and poses to regularize ambiguous or implausible reconstructions.
- The framework maps image pixels directly to 3D mesh parameters instead of using an intermediate 2D keypoint-detection stage.
- HMR supports training with or without paired 2D-to-3D supervision and outperforms previous mesh-output methods on 3D joint error and runtime.
2. Related Work
Prior work largely estimates 3D joints through two-stage pipelines, model fitting, or direct prediction, while related SMPL methods often produce partial or restricted representations. HMR instead recovers complete SMPL parameters and camera information directly from images.
- 3D Pose Estimation: Most 3D pose methods estimate major 3D joint locations, with approaches divided into two-stage pipelines and direct estimation.
- Two-stage methods: Two-stage methods predict 2D joints first and then infer 3D joints or model parameters using regression, fitting, dictionaries, or pose priors.
- Direct estimation: Supervised video datasets such as HumanEva and Human3.6M support deep-learning methods that directly estimate 3D joints from images, often without solving for the camera.
- SMPL-based methods: Optimization-based SMPL approaches can require 20-60 seconds per image, while related regression methods trade runtime against accuracy.
- Related SMPL outputs: Other methods predict depth, segmentation, or dense correspondences as 2.5D projections, whereas HMR recovers all SMPL parameters and the camera.
- Related SMPL outputs: Silhouette-based SMPL inference is limited to frontal, unoccluded people, while HMR does not depend on silhouettes.
3. Model
The model reconstructs a full 3D human mesh and camera directly from image features, using reprojection, optional 3D supervision, and a factorized adversarial prior. Iterative error feedback progressively refines the parameters while the adversarial component regularizes them toward plausible human bodies.
- Overview: The framework infers 3D body parameters and camera from image features so projected 3D joints match annotated 2D joints.It can use ground-truth 2D joints for all images and optionally include paired 3D supervision.
- 3.1. 3D Body Representation: SMPL represents body shape and pose, producing a triangulated mesh from shape coefficients and joint rotations.The reconstruction also includes global rotation, translation, and scale under a weak-perspective camera model.
- 3.2. Iterative 3D Regression with Feedback: Iterative error feedback updates the current parameter estimate with recurrently predicted residuals rather than regressing all parameters in one step.The module concatenates image features with the current latent estimate and initializes the loop from the mean parameter vector.
- 3.2. Iterative 3D Regression with Feedback: Reprojection and direct 3D losses apply to the final estimate, while adversarial loss is applied at every iteration to encourage corrective updates.The 3D loss can be omitted when paired ground-truth 3D annotations are unavailable.
- 3.3. Factorized Adversarial Prior: A discriminator regularizes predicted SMPL parameters because reprojection alone can admit anthropometrically implausible bodies or gross self-intersections.Separate shape and joint-rotation discriminators exploit SMPL’s factorization and learn joint-angle limits from unpaired 3D meshes.
- 3.4. Training Data: The system is trained end-to-end on in-the-wild 2D-keypoint datasets and separate 3D human-body datasets, with experiments using ground-truth person bounding boxes.The datasets include LSP, LSP-extended, MPII, MS COCO, Human3.6M, and MPI-INF-3DHP.
4. Experimental Results
HMR is evaluated on 3D joint estimation, segmentation, qualitative robustness, and training without paired 3D supervision. It is competitive with specialized methods, while MPJPE may not reflect visual quality.
- Evaluation setup: HMR is evaluated quantitatively on 3D joint estimation because existing datasets lack ground-truth meshes for in-the-wild images.The evaluation also includes human body part segmentation and qualitative results on challenging images.
- Human3.6M: HMR out-performs previous SMPL-reconstruction approaches on Human3.6M Protocol 2 and remains competitive with methods predicting only 3D joint locations.The comparisons include methods using 2D keypoints and a method that predicts 3D joint angles in a kinematic tree.
- Qualitative analysis: MPJPE does not appear to correlate well with visual quality, since many results at high error percentiles remain semantically reasonable.Figure 3 samples results at the 15th, 30th, 60th, 90th, and 95th error percentiles across 3D and 2D datasets.
- MPI-INF-3DHP: HMR is competitive with 3D-joint methods on MPI-INF-3DHP and improves upon VNect after rigid alignment.The reported metrics include MPJPE, PCK at 150mm, and AUC over PCK thresholds; alignment increases PCK and AUC while decreasing MPJPE.
- Human body segmentation: HMR achieves segmentation results comparable to the SMPLify oracle and out-performs Decision Forests while operating in real time given a bounding box.The LSP evaluation reports average accuracy and average F1 score for six body parts plus background, as well as foreground-background segmentation.
- Without paired 3D supervision: Without paired 3D supervision, HMR produces reasonable reconstructions close to those from the supervised model, whereas removing both supervision and the adversarial loss produces extreme shapes and poses.The adversarial prior is described as essential for training without paired 3D data.
5. Conclusion
HMR reconstructs a full 3D human mesh from a single RGB image using joint angles and a low-dimensional shape space. Results without paired 3D data suggest that 2D-labeled images can support continued improvement.
- 5. Conclusion: HMR presents an end-to-end framework for recovering a full 3D human body mesh from a single RGB image.The mesh is parameterized by 3D joint angles and a low-dimensional linear shape space.
- 5. Conclusion: The model represents the body through 3D joint angles and a low-dimensional linear shape space.This representation supports practical uses of a full 3D mesh model.
- 5. Conclusion: Results without paired 3D data are promising because additional images with 2D labels are easier to acquire than ground-truth 3D data in natural settings.The authors suggest this could support continued model improvement using more 2D-labeled images.