Source-linked AI summary
Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, Michael J. Black
TL;DR
Estimating 3D pose and shape from one unconstrained image is difficult because 3D-from-2D inference is ambiguous and human appearance is complex. SMPLify detects 2D joints with DeepCut, fits the SMPL body model to them, and uses shape and interpenetration constraints. It produces full 3D meshes and outperforms prior methods on the reported evaluations, while remaining vulnerable to detector errors and depth ambiguities.
Problem
3D pose and shape must be inferred from a single unconstrained image despite complex human appearance and inherent 3D-from-2D ambiguity.
Method
SMPLify detects 2D joints with DeepCut, then fits the SMPL 3D body model by matching projected model joints while using shape, pose, and interpenetration constraints.
Results
SMPLify estimates full 3D meshes and is reported as more accurate than previous methods across the evaluated datasets.
Takeaways & Limitations
Fitting a population-statistical body model to 2D joints can provide plausible 3D pose and approximate shape from very little data.
Takeaways & Limitations
Results can fail when CNN detections are wrong or when depth ambiguities remain challenging.
Abstract
from arXiv · showhide
We describe the first method to automatically estimate the 3D pose of the human body as well as its 3D shape from a single unconstrained image. We estimate a full 3D mesh and show that 2D joints alone carry a surprising amount of information about body shape. The problem is challenging because of the complexity of the human body, articulation, occlusion, clothing, lighting, and the inherent ambiguity in inferring 3D from 2D. To solve this, we first use a recently published CNN-based method, DeepCut, to predict (bottom-up) the 2D body joint locations. We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints. We do so by minimizing an objective function that penalizes the error between the projected 3D model joints and detected 2D joints. Because SMPL captures correlations in human shape across the population, we are able to robustly fit it to very little data. We further leverage the 3D model to prevent solutions that cause interpenetration. We evaluate our method, SMPLify, on the Leeds Sports, HumanEva, and Human3.6M datasets, showing superior pose accuracy with respect to the state of the art.
1 Introduction
SMPLify automatically estimates 3D human pose and shape from a single image by combining CNN-based 2D joint detection with top-down fitting of a statistical 3D body model. Its shape model, pose prior, and differentiable interpenetration term constrain the ambiguous reconstruction problem.
- Method: SMPLify combines bottom-up DeepCut 2D joint detection with top-down fitting of the SMPL 3D generative body model.The method optimizes the model using 2D joints detected by the CNN.
- Method: SMPL captures population-level shape and pose-deformation statistics, allowing the model to fit plausible body shape from very little data.The model is trained from thousands of 3D scans and captures correlations in human shape across people.
- Method: The objective directly minimizes the discrepancy between projected 3D model joints and CNN-estimated 2D joints, while 2D locations retain substantial 3D shape information.Synthetic-data evaluation supports the claim that 2D joint locations contain more shape information than expected.
- Method: The differentiable interpenetration term uses shape-dependent capsules to efficiently discourage body parts from intersecting in 3D.Capsule dimensions are linearly regressed from model shape parameters, enabling efficient computation.
- Method: SMPLify uses gender-neutral fitting when gender is unknown and gender-specific models when it is known.The gender-specific option is intended to provide better results when the relevant gender is available.
- Evaluation: On HumanEva-I and Human3.6M, SMPLify is reported as more accurate than previous methods, with qualitative robustness shown on LSP.The comparison uses the exact same 2D joints as input for the recently published methods.
2 Related Work
Prior work largely reconstructs 3D skeletons or body shape using anthropometric priors, silhouettes, manual correspondences, or additional image cues. SMPLify instead fits a strong statistical body model directly to automatically detected 2D joints, enabling automatic pose-and-shape estimation from a single image.
- 3D pose from 2D joints: Most earlier 3D pose methods reconstruct skeletons from 2D joints and impose assumptions about limb-length variation or pose statistics.These assumptions range from known limb lengths and fixed ratios to learned dictionaries of limb lengths and poses.
- 3D pose from 2D joints: Earlier approaches generally use weak or nonexistent human-shape models, whereas SMPL learns anthropometric constraints from thousands of people and models interpenetration.The stronger body model is presented as a way to reduce ambiguity and avoid impossible poses.
- 3D pose and shape: Single-image pose-and-shape methods often require silhouettes, known camera or body parameters, manual correspondences, segmentation, or substantial intervention.The cited approaches use combinations of silhouettes, image edges, shading, clicked points, GrabCut, known focal length, lighting initialization, or known height.
- Novelty: To the authors’ knowledge, no previous method estimates 3D body shape and pose directly from only 2D joints.The paper identifies this as a gap addressed by its statistical-model-based approach.
- Novelty: SMPL connects explicit 3D joint locations to the body surface, so projected joints can constrain shape, although identical limb lengths can correspond to different weights.The paper distinguishes this surface-based shape representation from anatomical joints.
- Novelty: SMPLify provides a complete automatic system that detects 2D joints and fits both 3D pose and body shape from a single image.The related-work discussion states that prior automated methods did not estimate 3D body shape.
3 Method
SMPLify estimates a posed 3D human mesh from one image by detecting 2D joints and fitting a statistical body model to them. Its objective combines joint alignment with pose, shape, and interpenetration constraints while handling camera and initialization uncertainty.
- System overview: SMPLify first uses DeepCut to predict 2D body joints, then fits SMPL to produce a posed 3D mesh.The system is fully automatic and estimates both body pose and shape from a single image.
- SMPL body model: SMPL represents a 6890-vertex triangulated surface whose shape is controlled by low-dimensional parameters learned from thousands of registered scans.The model includes male, female, and a newly trained gender-neutral shape model.
- SMPL body model: SMPL joints are functions of surface vertices and shape coefficients, coupling joint locations and body shape during fitting.The model has 23 skeleton joints, with pose represented by relative axis-angle rotations and global rigid transformation.
- Objective function: The objective sums a weighted 2D joint error, three pose priors, and a shape prior, with joint contributions weighted by CNN confidence and robustly penalized.Low-confidence occluded joints contribute less, allowing pose priors to influence the estimate.
- Interpenetration handling: A capsule-based interpenetration term provides a differentiable penalty for incompatible body-part intersections and helps prevent incorrect poses.Capsule dimensions are regressed from shape parameters; the term penalizes but does not strictly eliminate interpenetration and is excluded from shape optimization to avoid thin-body bias.
- Optimization and assumptions: The fitting procedure assumes a known or roughly estimated focal length, stages optimization to reduce local minima, and tests a 180-degree alternative for ambiguous side views.Single-image optimization takes less than 1 minute on a common desktop machine.
4 Evaluation
SMPLify is evaluated on synthetic shape-recovery data and real pose benchmarks, including controlled laboratory datasets and challenging sports images. It achieves strong quantitative accuracy, while experiments also expose benefits from pose priors and interpenetration handling alongside CNN detection failures and depth ambiguities.
- Synthetic data: Shape estimates outperform guessing the average male or female shape when 2D joints receive 1–5 pixels of Gaussian noise.Mean vertex-to-vertex Euclidean error is measured in a canonical pose using gender-specific models.
- Synthetic data: Using more joints improves shape estimation: the experiment compares 23 SMPL joints, 12 torso-and-limb joints, and 4 torso joints.Pose is assumed known, and error is measured between estimated and true shapes in a canonical pose.
- Real data: On HumanEva-I, SMPLify achieves the lowest 3D joint errors on all evaluated sequences using the same DeepCut 2D joints as competing methods.The evaluation uses Walking and Box sequences for subjects 1, 2, and 3, with gender-specific models.
- Ablation and qualitative analysis: The interpenetration term has little significant effect on HumanEva-I joint error but improves qualitative results in complex datasets with varied poses and viewing angles.LSP examples also show failures from misplaced or cross-person limb matches and depth ambiguities.
- Real data: On Human3.6M, SMPLify again achieves the lowest average 3D error across the evaluated sequences.Evaluation uses five frontal-camera action sequences from subjects S9 and S11, with known focal length and distortion coefficients.
5 Conclusions
SMPLify estimates approximate 3D body shape and pose from 2D joints, producing meshes that can be posed and animated. The authors report strong performance while identifying additional cues, views, frames, and detectors as future extensions.
- SMPLify combines CNN-estimated 2D joints with a fitted SMPL body model to estimate 3D pose and approximate shape from single images.The method optimizes pose and shape by minimizing the discrepancy between projected model joints and estimated 2D joints.
- The resulting 3D meshes are in correspondence with images and can be immediately posed and animated.
- Future work includes incorporating silhouettes, multiple camera views, multiple frames, facial pose, automatic gender detection, richer CNN outputs, and multiple-person scenes.The paper also notes that 3D meshes could support occlusion reasoning and training for features related directly to 3D shape.