Source-linked AI summary
Learning to Estimate 3D Human Pose and Shape from a Single Color Image
Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, Kostas Daniilidis
TL;DR
The paper tackles single-image 3D human pose and shape estimation, where optimization is costly and ConvNets face limited 3D training data. It embeds SMPL in an end-to-end ConvNet framework that predicts from 2D evidence, generates and renders meshes differentiably, and achieves state-of-the-art results while running faster than previous approaches.
Problem
Single-image 3D pose and shape estimation is challenging because optimization methods are costly and ConvNets lack sufficient images with 3D ground truth.
Method
The framework predicts SMPL pose and shape parameters from 2D keypoints and silhouettes, then uses differentiable mesh generation and rendering for surface and projection-based refinement.
Results
The approach achieves state-of-the-art results on relevant benchmarks and outperforms previous direct-prediction and optimization-based solutions.
Takeaways & Limitations
The framework provides an efficient direct-prediction solution that can be trained without images containing 3D shape ground truth.
Takeaways & Limitations
The approach relies on geometric 3D data, including motion capture and body scans, while relaxing the need for images with 3D shape ground truth.
Abstract
from arXiv · showhide
This work addresses the problem of estimating the full body 3D human pose and shape from a single color image. This is a task where iterative optimization-based solutions have typically prevailed, while Convolutional Networks (ConvNets) have suffered because of the lack of training data and their low resolution 3D predictions. Our work aims to bridge this gap and proposes an efficient and effective direct prediction method based on ConvNets. Central part to our approach is the incorporation of a parametric statistical body shape model (SMPL) within our end-to-end framework. This allows us to get very detailed 3D mesh results, while requiring estimation only of a small number of parameters, making it friendly for direct network prediction. Interestingly, we demonstrate that these parameters can be predicted reliably only from 2D keypoints and masks. These are typical outputs of generic 2D human analysis ConvNets, allowing us to relax the massive requirement that images with 3D shape ground truth are available for training. Simultaneously, by maintaining differentiability, at training time we generate the 3D mesh from the estimated parameters and optimize explicitly for the surface using a 3D per-vertex loss. Finally, a differentiable renderer is employed to project the 3D mesh to the image, which enables further refinement of the network, by optimizing for the consistency of the projection with 2D annotations (i.e., 2D keypoints or masks). The proposed approach outperforms previous baselines on this task and offers an attractive solution for direct prediction of 3D shape from a single color image.
1. Introduction
The paper proposes an end-to-end ConvNet framework for estimating 3D human pose and shape from a single color image despite limited 3D training data. It combines 2D predictions, SMPL parameter estimation, differentiable mesh generation, and rendering for efficient direct prediction.
- Single-image 3D pose and shape estimation remains difficult because optimization methods are slow and vulnerable to local minima, while ConvNets lack sufficient training data and offer low-resolution predictions.
- SMPL enables detailed 3D meshes with 6890 vertices from only 72 pose and 10 shape parameters, making direct network prediction feasible.The low-dimensional representation also supports direct optimization with a 3D per-vertex surface loss.
- The framework predicts SMPL parameters from ConvNet-estimated 2D keypoints and silhouettes, reducing reliance on natural images with 3D shape ground truth.2D annotations train image-to-2D inference, while parametric-model instances train 2D-to-3D shape inference.
- A differentiable renderer projects the generated mesh into the image, enabling end-to-end refinement against annotated 2D keypoints and masks.
- The approach achieves superior performance over previous methods for 3D human pose and shape estimation at significantly faster running time.
2. Related work
Prior work includes end-to-end ConvNets, iterative fitting, and direct prediction for 3D human pose and shape. The paper distinguishes its approach by using abundant 2D joints and masks with a unified end-to-end framework.
- Many ConvNet pose-estimation methods require images with 3D pose ground truth, limiting their available training data.
- Iterative optimization has been widely used to fit 3D models from silhouettes and keypoints, but direct prediction instead estimates pose and shape discriminatively without optimizing an objective during inference.
- Compared with prior landmark-based prediction, this work uses the smaller annotation set of 2D joints and masks, which is available for in-the-wild images.
- Unlike an encoder-decoder baseline that predicts silhouettes, the paper analytically generates and differentiably projects the body mesh from SMPL parameters, avoiding roughly half a million extra learnable weights.
3. Human body shape models
Statistical body shape models provide a low-dimensional representation that supports direct prediction while preserving detailed mesh quality. SMPL represents pose and shape through parameters that generate a full body mesh.
- Statistical body shape models use a low-dimensional parameter space suitable for direct prediction without sacrificing the quality of generated 3D meshes.
- SMPL defines a function M(β, θ; Φ) that outputs a body mesh P with 6890 vertices, using β for shape, θ for pose, and Φ for fixed model parameters.
- SMPL represents shape as a linear combination of learned principal body shapes and pose through a 23-joint skeleton with 72 pose parameters.
4. Technical approach
The framework uses abundant 2D annotations and geometric 3D data to train a modular ConvNet pipeline that predicts SMPL pose and shape parameters, synthesizes meshes, and refines them through differentiable projection.
- Keypoints and silhouette prediction: 2D keypoints and silhouettes provide scalable supervision for the first network stage, leveraging large benchmarks with these annotations.The Human2D network predicts both outputs and combines heatmap MSE with silhouette binary cross-entropy losses.
- 3D pose and shape prediction: PosePrior estimates 72 SMPL pose coefficients from keypoint locations and detection confidences, while ShapePrior estimates 10 shape coefficients from silhouettes.The disentangled design was found empirically to produce more stable and accurate predictions and supports modular updates.
- Synthetic geometric supervision: SMPL instances generated from sampled motion-capture poses and body scans produce training pairs by projecting meshes into silhouettes and 2D keypoints.Different viewpoints can be used for augmentation, avoiding the need to generate synthetic color images.
- 3D supervision: The predicted mesh can be supervised with a per-vertex loss, or with a joint loss when the training focus is primarily 3D pose.The vertex loss treats all vertices equally and better correlates with the per-vertex evaluation error; the reported strategy initializes with parameter loss before adding geometric loss.
- Training data: The framework relaxes the need for images with 3D shape ground truth by relying on geometric 3D data and, where available, 2D annotations for further training.This is useful for poses absent from typical motion-capture data, such as some gymnastics or parkour poses.
- Differentiable renderer: A differentiable renderer projects the generated mesh into the image, enabling end-to-end refinement against available 2D keypoint and mask annotations.This additional supervision can also adapt a generic pose or shape prior to a setting with only 2D annotations.
5. Empirical evaluation
The evaluation uses UP-3D, SURREAL, and Human3.6M to assess reconstruction accuracy, component choices, qualitative behavior, and runtime. The approach outperforms baselines and also improves SMPLify as an initialization and anchor.
- Datasets: The evaluation uses UP-3D and SURREAL for 3D shape accuracy, and Human3.6M for additional 3D pose evaluation.UP-3D and SURREAL report mean per-vertex error, while Human3.6M evaluates pose because it lacks body-shape ground truth.
- Component evaluation: The UP-3D ablation compares supervision forms and Priors trained on UP-3D or CMU data using the same architecture and training duration.The study examines parameter losses, rotation-matrix supervision, and mesh-based training choices.
- Comparison with state-of-the-art: On UP-3D, our approach outperforms the direct-prediction baselines of Lassner et al. and Tan et al., including Tan et al.’s version trained with over 100k images.The paper also provides qualitative comparisons showing improved predictions over Lassner et al.
- Comparison with state-of-the-art: 151.5 mm mean per-vertex error is reported for Ours (GT shape), versus 200.5 for Lassner et al. and 177.2 for Bogo et al.These values are reported for the GT-shape comparison.
- Comparison with state-of-the-art: 155.5 mm is reported for Ours versus 202.0 for Bogo et al. in the reported shape-error comparison.The cited comparison reports the two mean per-vertex errors without additional conditions.
- Comparison with state-of-the-art: The approach is more resistant to noisy 2D detections on challenging SURREAL images, recovering coherent 3D shape in most cases while iterative optimization has hard failures.The reported causes of noisy detections include low illumination and out-of-context backgrounds.
- Boosting SMPLify: The anchored SMPLify version is three times faster on average than vanilla SMPLify and improves segmentation accuracy.The direct prediction serves as an initialization and reliable anchor for iterative optimization.
- Running time: 50ms per image is reported for the approach, making it more than three orders of magnitude faster than the cited iterative optimization approaches.SMPLify is reported at roughly 1 minute, or 3 minutes with publicly available unoptimized code, per image.
6. Summary
The paper presents a ConvNet-based approach for predicting 3D human pose and shape from a single color image, incorporating SMPL into an end-to-end framework. It reports state-of-the-art results on relevant benchmarks and potential to improve iterative optimization pipelines.
- SMPL enables prediction from 2D keypoints and silhouettes, full-mesh supervision during training, and differentiable refinement using 2D annotations.
- The approach achieves state-of-the-art results, outperforming previous direct-prediction and optimization-based solutions for 3D pose and shape estimation.
- The project page provides additional information about the work.https://www.seas.upenn.edu/˜pavlakos/ projects/humanshape
- The work acknowledges support from NSF, ARL, ONR, DARPA, and NSF/IUCRC grants.