Source-linked AI summary
Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation
Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter V. Gehler, Bernt Schiele
TL;DR
Estimating detailed 3D human body pose and shape from a single image is difficult because perspective ambiguities complicate prediction and paired 3D data are scarce. Neural Body Fitting integrates a differentiable statistical body model into a CNN, using body-part segmentation as an intermediate representation and supporting 2D and 3D supervision; it achieves competitive performance and similar performance with 3D information for only 20% of training data.
Problem
Estimating the parameters of a detailed statistical human body model from a single image remains more challenging than predicting 3D keypoints or stick figures, while 3D pose annotations are difficult to obtain.
Method
Neural Body Fitting integrates a differentiable statistical body model within a CNN, predicts model parameters from images or semantic segmentations, and projects the resulting 3D joints back into the image for 2D or 3D supervision.
Results
20% 3D information in the training data reaches similar performance as full 3D annotations, while a color-coded 12-body-part segmentation provides sufficient information for predicting shape and pose.
Takeaways & Limitations
The results support region-based body-part segmentation and weakly supervised training as practical directions for reducing reliance on expensive 3D annotations.
Takeaways & Limitations
Future work must extend the approach to settings with multiple, possibly occluded, people.
Abstract
from arXiv · showhide
Direct prediction of 3D body pose and shape remains a challenge even for highly parameterized deep learning models. Mapping from the 2D image space to the prediction space is difficult: perspective ambiguities make the loss function noisy and training data is scarce. In this paper, we propose a novel approach (Neural Body Fitting (NBF)). It integrates a statistical body model within a CNN, leveraging reliable bottom-up semantic body part segmentation and robust top-down body model constraints. NBF is fully differentiable and can be trained using 2D and 3D annotations. In detailed experiments, we analyze how the components of our model affect performance, especially the use of part segmentations as an explicit intermediate representation, and present a robust, efficiently trainable framework for 3D human pose estimation from 2D images with competitive results on standard benchmarks. Code will be made available at http://github.com/mohomran/neural_body_fitting
1. Introduction
The paper addresses single-image estimation of detailed 3D human body pose and shape by combining CNN prediction with model-based constraints and semantic body-part segmentation. NBF supports mixed 2D/3D supervision while analyzing segmentation as an intermediate representation.
- Motivation: Traditional fitting avoids paired 3D training data but needs near-solution initialization, whereas direct CNN prediction needs scarce 3D annotations.This motivates a hybrid architecture that can use 2D supervision.
- Approach: NBF integrates a statistical human body model into a CNN, combining bottom-up prediction with top-down model constraints.The model produces SMPL parameters, a 3D mesh, and projected joints within a differentiable loop.
- Intermediate representation: A color-coded 12-body-part segmentation contains sufficient information for predicting body shape and pose.The representation abstracts the image into semantic body regions before lifting to 3D.
- Findings: The segmentation-based intermediate representation achieves competitive performance and easier, more data-efficient training than similar raw-RGB methods.The study also finds segmentation quality strongly predicts fit quality.
- Contribution: NBF is designed for fine-grained analysis of hybrid architectures and their real-world applicability.The contribution includes both the framework and an in-depth component analysis.
2. Related Work
Prior work spans model-based optimization, 2D-guided fitting, CNN prediction, and hybrid networks. NBF differs by integrating the body model into the CNN and using semantic segmentation to study representation and supervision choices.
- Model-based methods: Model-based methods fit parametric body models to image evidence but typically require careful initialization to avoid poor local minima.Modern models separately represent pose and shape and are estimated from scans of real people.
- 2D-guided fitting: 2D keypoints simplify fitting, but lifting from 2D information alone remains ambiguous.SMPL can be fitted to 2D keypoints with reasonable accuracy, while depth ambiguities remain a challenge.
- Learning-based models: CNN methods predict 3D keypoints or stick figures directly, but standard 3D datasets have limited appearance, pose, background, and occlusion variation.These methods can perform strongly on standard benchmarks despite restricted training diversity.
- Learning-based models: Methods trained with separate 2D and 3D losses may produce unnatural limbs and proportions because they do not integrate a statistical body model.They generally predict stick figures rather than a full mesh.
- Hybrid methods: Concurrent hybrid methods integrate SMPL but differ in proxy representation and supervision, including RGB, keypoints, silhouettes, and varying 2D/3D losses.NBF focuses on analyzing how these design choices affect performance.
- NBF: NBF requires no optimization initialization at test time and uses semantic segmentation as a richer proxy than keypoints or silhouettes.The segmentation abstracts irrelevant image information while retaining spatial and semantic body-part structure.
3. Method
NBF combines semantic body-part segmentation, a CNN predictor, and a differentiable SMPL model to fit a 3D human mesh from a single image. Its differentiable projection supports training with multiple forms of 2D and 3D supervision.
- Architecture: NBF uses a two-stage pipeline: body-part segmentation followed by CNN prediction of statistical body-model parameters.The architecture is designed so model parts can be optimized in isolation and trained consecutively.
- Body model: SMPL converts predicted pose and shape parameters into a posed 3D mesh and is embedded directly within the deep-learning model.SMPL is differentiable with respect to pose and shape, enabling gradient-based training through the body model.
- Architecture: The segmentation CNN produces 12 semantic parts, whose probability maps are encoded to predict SMPL pose and shape parameters.This explicit intermediate representation separates body-part evidence from the subsequent 3D parameter prediction.
- Differentiable outputs: NBF integrates SMPL and a 2D projection layer to output meshes, 3D joints, or 2D joints while keeping the CNN monolithic.Projected joints provide image-plane supervision through the differentiable projection operation.
- Loss functions: The model supports latent-parameter, 3D-joint, 2D-joint, and combined 2D-plus-3D losses.Combining datasets with different annotations increases usable supervision by applying only the relevant loss to each example.
- Evaluation: Experiments compare individual losses and combinations, including the gain from cheaper weak 2D annotations over accurate 3D supervision.The study specifically evaluates how weak 2D supervision affects 3D estimation accuracy.
4. Results
Experiments analyze input representations, segmentation quality, supervision losses, and benchmark performance for NBF. Explicit body-part representations substantially improve 3D estimation, while some 3D supervision remains necessary and segmentation quality correlates with fit quality.
- Datasets and setup: NBF experiments use UP3D, HumanEva-I, and Human3.6M, with detailed analyses on UP3D and Human3.6M and state-of-the-art comparisons on HumanEva-I and Human3.6M.UP3D provides SMPL, 2D/3D joint, and fine-grained segmentation annotations; Human3.6M SMPL parameters and part labels are derived from MoCap data.
- Input representation: Explicit part segmentations and joint heatmaps outperform RGB images and plain silhouettes for 3D pose and shape estimation.On UP-3D, error drops from 98.5 mm to 27.8 mm with a 12-part segmentation.
- Segmentation quality: 3D joint error decreases as segmentation F1-score increases, indicating a strong correlation between segmentation quality and body-fit quality.Qualitative comparisons show that predicted-segmentation failures are more severe than errors using ground-truth segmentations.
- Loss ablation: With sufficient 3D data, parameter loss gives reasonable results, whereas 3D-location-only training yields similar joint positions but significantly worse quaternion error.The latter indicates accurate 3D joint locations alongside incorrect limb orientations.
- Supervision: 2D reprojection loss alone produces poor 3D performance, showing that some 3D annotations are necessary to resolve ambiguity from 2D keypoints.The analyzed loss terms include latent-parameter, 3D-location, and projected 2D-location supervision.
5. Conclusion
The paper integrates a parametric 3D body model into a CNN and studies representations, losses, and training regimes for body fitting. Its results support region-based segmentation and reduced reliance on 3D annotations, while identifying multiple occluded people as future work.
- 5. Conclusion: The study analyzes model integration, combined loss functions, and efficient training with scarce 3D data.These design dimensions are central to integrating parametric body models into deep CNN architectures.
- 5. Conclusion: A 12-body-part segmentation provides full spatial coverage while retaining enough part arrangement information for effective lifting to 3D.The segmentation is used as an intermediate representation before predicting 3D shape and pose.
- 5. Conclusion: 20% of training data with 3D information achieved similar performance to training with full 3D annotations.The combined 2D and 3D loss supported this weakly supervised training setting.
- 5. Conclusion: Future work will extend the approach to settings with multiple, possibly occluded, people.The current conclusion identifies this as a more challenging setting for subsequent investigation.
A. Further Qualitative Results
Qualitative analyses show a strong correlation between segmentation quality and the quality of the fitted 3D body, although segmentation improvements do not eliminate all errors.
- A. Further Qualitative Results: Segmentation quality is highly correlated with the quality of the output body fit.The paper compares worst validation examples using predicted segmentations with examples using ground-truth segmentations.
- A. Further Qualitative Results: Ground-truth segmentations produce noticeably less severe remaining 3D body errors, but do not correct every estimated body.This comparison isolates the practical effect of input segmentation quality on fitting errors.
B. Training Details
Training uses separate segmentation and fitting networks with specified optimization schedules, while strong augmentation addresses variation in person alignment and mirrored inputs.
- B. Training Details: The segmentation network is a RefineNet based on ResNet-101, trained for 20 epochs with Adam on 512x512 crops.The reported learning rate is 0.00002, with weight decay 0.0001 and polynomial decay.
- B. Training Details: The fitting network repurposes an ImageNet-pretrained ResNet-50 to regress 10 shape and 216 pose parameters.It is trained for 75 epochs with Adam, using learning rate 0.00004, weight decay 0.0001, and polynomial decay.
- B. Training Details: Strong augmentation, especially random jitter and scaling, is critical for both networks because imperfect centering can degrade performance.Mirroring also requires remapping part labels and inverting part rotations in the SMPL parameters.