Source-linked AI summary
Expressive Body Capture: 3D Hands, Face, and Body from a Single Image
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, Michael J. Black
TL;DR
The paper tackles the challenge of capturing expressive 3D body, hand, and facial information from a single monocular image. It introduces the unified SMPL-X model and SMPLify-X fitting method, which together produce natural expressive results and outperform related less powerful models on a curated evaluation dataset.
Problem
Existing 3D models and training data do not adequately support capturing the full body, hands, and face together from a single image.
Method
SMPL-X jointly models body, face, and hands, while SMPLify-X fits it to body, hand, foot, and facial 2D detections using pose and interpenetration priors.
Results
The method produces natural and expressive results and performs significantly better than related, less powerful models on a curated pseudo-ground-truth dataset.
Takeaways & Limitations
The work is a step toward expressive capture of bodies, hands, and faces together from a single RGB image.
Takeaways & Limitations
Collision handling ignores neighboring parts in areas with frequent self-contact because the model does not model contact-related deformations.
Abstract
from arXiv · showhide
To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.
1. Introduction
The paper addresses the need to capture body pose, hands, and facial expression together in 3D from a single image. It introduces SMPL-X and SMPLify-X to provide a more expressive model and fit it using 2D image features.
- Motivation: Major body joints or isolated 2D features are insufficient for representing expressive human behavior and interaction with the 3D world.The paper argues that full 3D surfaces of the body, hands, and face are needed.
- Results: The model captures natural and expressive poses, gestures, and facial expressions from single RGB images.Qualitative results include controlled and in-the-wild images.
- SMPL-X model: SMPL-X is a holistic body model that combines body, face, and hands using 5,586 curated 3D scans.It combines SMPL with the FLAME head model and MANO hand model while retaining SMPL's compact, differentiable representation.
- SMPLify-X method: SMPLify-X estimates body, hand, foot, and facial 2D features with OpenPose, then fits SMPL-X to those detections.This approach addresses the lack of paired images and 3D ground truth for directly regressing SMPL-X parameters.
- Evaluation: A curated evaluation dataset uses scanned bodies fitted with SMPL-X as pseudo-ground truth for quantitative evaluation.The dataset contains images of varied poses, gestures, and expressions.
- Results: The authors report significantly better performance than related, less powerful models and make the model, code, trained networks, fits, and dataset available for research.They characterize the work as a step toward expressive capture from a single RGB image.
2. Related work
Prior work largely modeled the body, hands, and face separately, while unified models traded realism or consistency. This work instead learns and fits a unified expressive model with a structured optimization pipeline.
- Bodies, Faces and Hands: Earlier 3D body models captured shape and pose without hands or face, while hand and face models were typically developed in isolation.
- Unified Models: Frank combined separate body, artist-designed hand, and face models, whereas SMPL+H unified the body with learned hands but not the face.
- Unified Models: SMPL-X fits a full model to 5586 scans and learns joint shape and pose-dependent blend shapes for a consistent parameterization.
- Unified Models: SMPL-X combines SMPL, FLAME, and MANO while retaining differentiability and compatibility with graphics software.
- Inferring the body: SMPLify-X detects body, hand, foot, and face features before fitting SMPL-X, using a VAE-based pose prior, interpenetration penalty, and annealed optimization weights.
4. Experiments
The experiments evaluate SMPL-X and SMPLify-X on curated pseudo-ground-truth data, against related models, and on in-the-wild images. Results show that richer holistic modeling improves reconstruction accuracy and expressivity, while holistic fitting is more robust to noisy detections.
- Dataset and protocol: A curated dataset provides 100 pseudo-ground-truth frames with full-body images, body scans, poses, gestures, and facial expressions.The dataset enables quantitative evaluation of bodies, hands, and faces together using vertex-to-vertex error.
- Quantitative comparison: Richer SMPL-X modeling produces lower vertex-to-vertex errors than body-only or body-and-hand variants, whereas mean body-joint error is less discriminative.Adding features for poorly modeled parts increases error, underscoring the importance of expressive model capacity.
- Ablation study: 52.9 mm mean vertex-to-vertex error is achieved by SMPLify-X with a gender-specific model; replacing VPoser raises error to 56.4 mm.A gender-neutral model is easier to use but slightly less accurate, while removing collision handling raises error to 53.5 mm and permits physically implausible poses.
- Comparison with related models: SMPL-X matches or exceeds Frank’s expressivity using one RGB image and 2D joints, whereas Frank uses 3D joints and point clouds from more than 500 cameras.SMPL-X also avoids the skinning artifacts around joints that appear in Frank’s reconstructions.
- Holistic versus part models: 19.8 mm mean 3D hand-joint error is obtained by SMPL-X versus 26.5 mm for the hands-only method of Panteleris et al.The holistic model is especially more robust when 2D detections are noisy; both approaches perform well with good detections.
- In-the-wild results: SMPL-X produces natural and expressive reconstructions of bodies, hands, and faces in in-the-wild LSP images.The system uses gender-specific models for confident gender detections and a gender-neutral model when the classifier is uncertain.
5. Conclusion
The paper presents SMPL-X for jointly modeling body, face, and hands, and SMPLify-X for fitting it from a single RGB image with 2D detections. It concludes that the approach advances expressive monocular capture and releases models, code, trained networks, fits, and evaluation data for research.
- Contributions: SMPL-X jointly captures the body, face, and hands, while SMPLify-X fits it to a single RGB image and 2D OpenPose detections.The fitting uses a body pose prior and a collision-penetration penalty to regularize ambiguities.
- Evaluation: A curated pseudo-ground-truth dataset enables quantitative evaluation showing the importance of more expressive models.The paper also reports qualitative results on in-the-wild images.
- Conclusion: The work is positioned as a step toward expressive capture of bodies, hands, and faces together from an RGB image.The authors identify direct regression of SMPL-X parameters from RGB images as future work.
- Resources: The SMPL-X model, SMPLify-X code, trained networks, model fits, and evaluation dataset are released for research purposes.The paper provides these resources through the project website.
**Supplementary Material**
The supplementary material identifies the paper’s authors and their institutional affiliations in Germany and the United States.
- Georgios Pavlakos, Vasileios Choutas, and Nima Ghorbani are among the listed authors.
- Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black are also listed as authors.
- The authors are affiliated with the MPI for Intelligent Systems and the University of Pennsylvania.The affiliations are located in Tübingen, Germany, and Pennsylvania, United States.
1. Qualitative results
Qualitative comparisons show that richer holistic models produce more expressive fits, while full-body context improves robustness under noisy detections and occlusions.
- SMPL-X shows greater expressiveness than SMPL and SMPL+H because it models the body together with hands and face.
- Holistic SMPL-X fitting is more robust than a hands-only approach when 2D detections are noisy.
- Full-body context improves head fitting under moderate or extreme occlusions and non-frontal views compared with head-only FLAME fitting.The cited comparison reports a 17% improvement in vertex-to-vertex error on the EHF dataset.
2. Collision Penalizer
The collision penalizer detects mesh penetrations with triangle-level geometric fields and penalizes bidirectional intrusion, while selectively ignoring frequent self-contact regions.
- SMPLify-X detects colliding triangles with BVH and computes local conic 3D distance fields from triangles and their normals.
- For each colliding triangle pair, vertices are penalized for intrusion into the other triangle’s distance field in both directions.
- The penalizer uses a quadratic response near the triangle and a linear response farther away, with σ controlling the cone’s field of view.The implementation uses σ = 0.0001 in meter units to penalize penetrations without over-penalizing self-contact.
- Collisions are ignored around eyes, toes, armpits, crotch, and neighboring kinematic-chain parts where self-contact is frequent.This simplification reflects that the model does not represent contact-induced deformations.
- The collision term resolves most penetrations and helps prevent physically implausible poses, although arm crossing can still produce many collisions.
3. Optimization
Optimization uses a PyTorch L-BFGS implementation with staged annealing that progressively emphasizes hand and facial evidence; weight sensitivity is limited.
- SMPLify-X uses PyTorch with L-BFGS and strong Wolfe line search, allowing up to 30 iterations per optimization stage.
- Annealing begins with high body regularization and gradually increases hand-keypoint influence to refine the arms.
- As hand and facial keypoint influence increases, larger pose changes and more collisions are expected.
- Perturbing each optimization weight by up to ±25% changes vertex-to-vertex error by less than 6%.This indicates robustness across significant weight ranges without fine-tuning sensitivity.
4. Quantitative evaluation on “Total Capture”
The evaluation uses curated datasets with pseudo-ground-truth annotations for full-body, face, and hand assessment, including selected Total Capture frames and qualitative THF examples.
- The Total Capture evaluation focuses on PtCloudDB, which provides pseudo-ground-truth for body, face, and hands.
- Sensitivity analysis reports relative vertex-to-vertex error changes below 6% when optimization weights are perturbed by up to ±25%.
- Quantitative evaluation on CMU Panoptic uses mean 3D joint errors after Procrustes alignment for ground-truth reprojections and OpenPose detections.
- The authors curate 200 THF images based on body visibility, interesting hand poses, and facial expressions.
5. Quantitative evaluation on Human3.6M
The evaluation compares SMPLify-X with SMPLify on Human3.6M using common 3D body-joint metrics. Results are reported as mean 3D joint errors after Procrustes alignment under SMPLify’s evaluation protocol.
- The authors motivate these metrics as a common quantitative comparison despite noting that 3D body-joint accuracy may not represent reconstruction detail.
- Mean 3D joint errors are reported after Procrustes alignment with the ground-truth 3D pose.
- The comparison uses Human3.6M and follows SMPLify’s evaluation protocol.
6. Qualitative evaluation on MPII
Qualitative experiments assess SMPLify-X across ordinary and challenging images, while supplementary analyses examine SMPL-X shape generalization and the VPoser pose prior. The results include cropped, occluded, and low-resolution people, alongside plausible pose samples from VPoser’s latent space.
- Qualitative evaluation on MPII: MPII results include images with low resolution, heavy occlusion, or cropped people.
- SMPL-X shape analysis: All SMPL-X shape models explain more than 95% of CAESAR shape variance with 10 principal components.
- Qualitative evaluation on MPII: The THF dataset curates 200 images with pseudo ground-truth for body, face, and hands, selected for body coverage, hand poses, and facial expressions.
- VPoser pose prior: VPoser is trained on poses from CMU, Human3.6M, and PosePrior, and its latent space represents physically plausible human body poses for efficient 2D-to-3D lifting.
- VPoser pose prior: During testing, VPoser uses a latent code instead of SMPL-X body pose parameters, then converts the decoder output to an axis-angle representation.
9. Gender lassifier
The gender classifier is trained from curated person crops and produces qualitative predictions on a test set. SMPLify-X uses gender-specific models for confident detections and a gender-neutral model when confidence or cropping criteria are insufficient.
- Training data: Training data comes from LSP, LSP-extended, MPII, MS-COCO, and LIP, with original train and test splits retained.
- Training data: Person crops are filtered for visible high-confidence joints in the head, torso, and limbs, and crops smaller than 200 × 200 pixels are rejected.
- Classifier implementation: The classifier uses pretrained ResNet18 features with fully connected layers, cross-entropy loss, and L2 weight regularization.
- Model selection: For MPII results, SMPLify-X uses gender-specific models for confident detections and a gender-neutral model when gender is uncertain or cropping fails the filtering criterion.
- Body-prior comparison: LSP comparisons replace VPoser with SMPLify’s GMM body-pose prior while holding the gender-neutral SMPL-X model fixed.