PaperlayerView exact revision on arXiv

Source-linked AI summary

Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, Michael J. Black

arXiv:1904.05866v1cs.CV

TL;DR

Capturing the full 3D body, hands, and face from one image remains challenging without suitable models and rich 3D training data. This paper introduces SMPL-X, a unified model trained from 3D scans, and SMPLify-X, which fits it to a single image using 2D detections. On a curated dataset with pseudo-ground-truth, the model and method perform significantly better than related, less powerful models, producing natural and expressive results.

  • Problem

    Existing 2D features and 3D representations with limited hand and face detail do not capture full 3D body, hand, and face surfaces needed to understand human behavior and interactions.

    Source: §1 · pp. 1–2

  • Method

    The authors train the unified SMPL-X body, hand, and face model from 3D scans and fit it to a single RGB image using 2D joint detections.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    On a curated dataset with pseudo-ground-truth, SMPL-X and SMPLify-X perform significantly better than related, less powerful models, producing natural and expressive results.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The curated pseudo-ground-truth evaluation shows the importance of more expressive models.

    Source: §5 · p. 8

    Because the model does not represent contact-induced deformations, collision handling ignores neighboring parts in regions with frequent or constant self-contact.

    Source: §2 · p. 14

Abstract

from arXiv · show

To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.

1. Introduction

The paper introduces SMPL-X, a holistic 3D model of body, hands, and face, and SMPLify-X, a method for fitting it from a single image. Together they address the limits of sparse 2D features and body models that omit expressive detail.

Motivation

  • Understanding human behavior requires more than major body joints or 2D hand and face features: those representations do not capture surfaces or interactions with the 3D world.The paper identifies the need for both a model that represents body, hands, and face, and a method to recover it from a single image.Source: §1 · pp. 1–2

Contributions

  • SMPL-X combines body, face, and hand models and learns their natural correlations from 5,586 curated 3D scans.It is based on SMPL, FLAME, and MANO.Source: §1 · p. 2
  • SMPLify-X fits SMPL-X to detected body, hand, foot, and face features, using a VAE pose prior, differentiable interpenetration penalty, gender detection, and a PyTorch implementation at least 8x faster than Chumpy.The pose prior addresses ambiguity in mapping 2D features to 3D pose; the penalty is described as more accurate and efficient than SMPLify’s approximate method.Source: §1 · p. 2

Evaluation

  • The evaluation uses a curated dataset of full-body images with scan-based pseudo-ground truth and finds that the model and method outperform related, less powerful models.The images cover a wide variety of poses, gestures, and expressions.Source: §1 · p. 3

Contributions

  • The authors release SMPL-X, SMPLify-X code, trained networks, model fits, and the evaluation dataset for research use.The paper presents the work as a step toward expressive capture of bodies, hands, and faces from a single RGB image.Source: §1 · p. 3

2. Related work · 3. Technical approach

SMPL-X unifies body, hands, and expressive face in a jointly trained model, while SMPLify-X fits it to a single RGB image through keypoint-driven optimization with improved pose, collision, gender, and implementation components.

2.1. Modeling the body

  • Earlier body models generally omitted hands and expressive faces, while many hand models ignored the body; FLAME models the whole head and neck, but prior models did not correlate face and body shape.SMPL+H adds learned hands to SMPL but has no deformable face.Source: §2.1 · p. 3

2.2. Inferring the body

  • Prior single-image mesh methods estimated body pose without hands or face, whereas expressive multi-camera capture required a complex setup; this work targets comparable detail from one RGB image.The cited Panoptic setup used 140 VGA cameras for the body, 480 VGA cameras for the feet, and 31 HD cameras for face and hand keypoints.Source: §2.2 · pp. 3–4

3.1. Unified model: SMPL-X

  • SMPL-X jointly models body, hand, and face shape with 10,475 vertices, 54 joints, and 119 parameters, including separate gendered models and a gender-neutral option.Its training combines body scans with MANO hand pose and FLAME expression spaces; the full model is fitted to 5,586 scans.Source: §2.1 · p. 3, §3.1 · p. 4

3. Technical approach / 3.2. SMPLify-X: SMPL-X from a single image

  • SMPLify-X fits SMPL-X to one RGB image by minimizing an objective with pose, shape, expression, hand, and collision terms.OpenPose supplies body, hand, face, and foot keypoints; detection confidence weights the reprojection loss, which uses a robust error function.Source: §3.2 · pp. 4–5, §3 · p. 4

3.3. Variational Human Body Pose Prior

  • VPoser replaces SMPLify’s insufficiently strong Gaussian-mixture pose prior with a VAE-trained latent pose representation.It is trained on roughly 1 million poses from CMU, Human3.6M, and PosePrior; fitting optimizes a 32-dimensional latent code with a quadratic penalty.Source: §3.3 · p. 5

3.4. Collision penalizer

  • SMPLify-X penalizes mesh penetrations using a detailed collision model, detecting colliding triangles with BVHs and computing intrusion depth through local conic distance fields.This replaces the faster but rough capsule-based approximation used by SMPLify.Source: §3.4 · pp. 5–6

3.5. Deep Gender Classifier

  • A ResNet18 gender classifier selects the male or female model above a 0.9 probability threshold and the neutral model below it.At that threshold, validation predictions were 62.38% correct and 7.54% incorrect.Source: §3.5 · p. 6

3.6. Optimization

  • The fitting optimization uses PyTorch with L-BFGS and anneals keypoint weights, gradually adding hand and face influence while increasing collision penalties.It begins with camera translation and global orientation, then optimizes shape and pose; staged weights help limit local optima from dense hand and face keypoints.Source: §3.6 · p. 6

4. Experiments

SMPLify-X is evaluated on a curated 100-image dataset with pseudo ground-truth meshes, and the experiments compare model expressivity, fitting components, hand reconstruction, and in-the-wild results. The reported findings show lower vertex-to-vertex error with richer modeling, gains from VPoser and collision handling, and robust expressive reconstructions from image-based inputs.

4.1. Evaluation datasets

  • A curated set of 100 EHF frames provides pseudo-ground-truth SMPL-X meshes, enabling stricter vertex-to-vertex evaluation than joint error alone.The dataset starts from SMPL+H images aligned to 4D scans, with an expert selecting frames based on alignment quality and interesting hand poses and facial expressions.Source: §4.1 · p. 6

4.2. Qualitative & Quantitative evaluations

  • Richer body, finger, and face modeling lowers vertex-to-vertex error, while mean 3D joint error does not capture model expressivity accurately.The comparison fits SMPL, SMPL+H, and SMPL-X on EHF; adding finger features to SMPL increases error. The table reports mean v2v and mean 3D body-joint error after Procrustes alignment.Source: Table 1 · p. 7, §4.2 · p. 7
  • SMPLify-X’s gender-specific model achieves 52.9 mm v2v error; substituting SMPLify’s GMM for VPoser raises error to 56.4 mm, and removing collision handling raises it to 53.5 mm.The gender-neutral model is easier to use because it needs no gender detection, with a small accuracy compromise; removing collision handling also permits non-physically-plausible poses.Source: §4.2 · p. 7, Table 2 · p. 7
  • Using one RGB image and 2D joints, SMPL-X shows at least similar face and hand expressivity to Frank [36], whose fittings use 3D joints and point clouds from more than 500 cameras.SMPL-X’s pose blend shapes avoid the joint skinning artifacts visible in Frank’s results, producing more natural-looking reconstructions.Source: §4.2 · p. 7, Figure 3 · p. 7
  • 19.8 mm mean 3D hand-joint error for SMPL-X versus 26.5 mm for Panteleris et al. [60] after independently aligning each hand to pseudo-ground-truth joints.With noisy 2D detections, the holistic body context also makes SMPL-X more robust than the hands-only approach; both perform well with good detections.Source: §4.2 · p. 8, Figure 5 · p. 8
  • SMPL-X produces natural, expressive reconstructions from everyday images in LSP, LSP-extended, and MPII; LSP results distinguish confident gender-specific fits from gender-neutral fits when classification is uncertain.The paper reports qualitative in-the-wild results and refers to supplementary material for further examples and failure cases.Source: §4.2 · p. 8, Figure 4 · p. 8

5. Conclusion

The paper presents SMPL-X for jointly modeling the body, face, and hands, and SMPLify-X for fitting it to a single RGB image using 2D OpenPose detections. Qualitative results and quantitative evaluation on a curated dataset support the method’s expressivity and the importance of expressive models.

5. Conclusion

  • SMPL-X jointly captures the body, face, and hands, while SMPLify-X fits the model to a single RGB image using 2D OpenPose joint detections.The fitting approach addresses ambiguities with a body pose prior and a fast, accurate method for detecting and penalizing penetrations.Source: §5 · p. 8
  • Qualitative results on in-the-wild images demonstrate SMPL-X’s expressivity and SMPLify-X’s effectiveness.The paper also introduces a curated dataset with pseudo-ground-truth for quantitative evaluation, which shows the importance of more expressive models.Source: §5 · p. 8
  • Future work will curate a dataset of in-the-wild SMPL-X fits and train a regressor to predict SMPL-X parameters directly from RGB images.Source: §5 · p. 8

1. Qualitative results

Qualitative comparisons show increasing expressiveness from SMPL to SMPL+H to SMPL-X, while holistic SMPL-X fitting improves head estimation but still fails under depth ambiguities and occlusion.

Comparison of SMPL, SMPL+H & SMPL-X

  • For a single RGB image, SMPL-X shows greater expressiveness than SMPL+H and SMPL, which model hands and body only, or body only, respectively.All three models were fit using a variation of SMPLify-X.Source: §1 · p. 12

Holistic vs part models

  • SMPL-X improves over head-only FLAME fitting by 17% in vertex-to-vertex error on EHF.The comparison fits FLAME to 2D keypoints similar to the SMPL-X method.Source: §1 · p. 12
  • Full-body context stabilizes SMPL-X head estimation under occlusions or non-frontal views compared with head-only fitting.The qualitative comparison includes the hand-only approach of on EHB and a similar comparison on data.Source: §1 · p. 12
  • Failure cases arise from depth ambiguities in the simple 2D reprojection term and occluded joints that leave body parts unconstrained.The authors plan to address these failures with a visibility term in the objective.Source: §1 · p. 12

2. Collision Penalizer

The collision penalizer detects candidate triangle collisions with BVHs and uses local conic distance fields to penalize bidirectional intrusion. It ignores frequent self-contact regions and uses an empirical field-of-view setting to balance penetration penalties against over-penalizing contact.

Collision Penalizer

  • SMPLify-X detects candidate colliding triangles with BVHs and penalizes bidirectional vertex intrusion using local conic distance fields defined by receiver triangles and their normals.Vertices of each colliding triangle are evaluated in the other triangle's field, and vice versa.Source: §2 · p. 12
  • The cone uses each receiver's circumcenter, circumcircle radius, and normal; projected intrusion is scaled by cone radius, with quadratic near-field and linear farther-field penalties.The parameter σ defines the cone's field of view; the penalizer also distinguishes points behind the triangle from those in front.Source: §2 · p. 13
  • Using meters and σ = 0.0001, the empirically chosen setting penalizes penetrations without over-penalizing self-contact, unlike 's millimeter units and σ = 0.5.The cited setting is chosen for the resolution of the authors' meshes, including cases such as an arm resting on a knee.Source: §2 · p. 13
  • SMPLify-X ignores collisions in regions with frequent self-contact, since the model does not represent contact deformations; detecting other collisions resolves most penetrations and helps prevent implausible poses.Frequently contacting regions include eyes, toes, armpits, crotch, and neighboring parts in the kinematic chain.Source: §2 · pp. 13–14, Figure A.5 · p. 14
  • A custom PyTorch wrapper for the CUDA collision kernel uses a highly parallelized BVH implementation for computational efficiency.Source: §2 · p. 14

3. Optimization

SMPLify-X uses staged L-BFGS optimization to refine global body pose, then arms, hands, and facial expression. Perturbing individual objective weights by up to ±25% produced less than 6% relative change in vertex-to-vertex error.

3. Optimization

  • The PyTorch implementation uses L-BFGS with strong Wolfe line search, learning rate 1.0, and up to 30 iterations, with regularization annealed across three fitting stages.It first emphasizes global body pose (γb = 1, γh = 0, γf = 0), then arms (γh = 0.1), and finally hands and face (γh = 2, γf = 2); λα, λβ, and λE decrease while λC increases.Source: §3 · p. 14
  • Perturbing each optimization weight by up to ±25% changed vertex-to-vertex error by less than 6%, indicating robustness across substantial weight ranges.Weights were set empirically; detailed stage parameters were slated for release with the code.Source: §3 · p. 14

4. Quantitative evaluation on “Total Capture”

The section evaluates SMPLify-X using the Total Capture dataset’s PtCloudDB pseudo-ground-truth for body, face, and hands, and reports sensitivity to optimization weights.

4. Quantitative evaluation on “Total Capture”

  • Perturbing each optimization weight by up to ±25% changed vertex-to-vertex error by less than 6%, indicating limited sensitivity to weight selection.The sensitivity analysis perturbed each weight separately.Source: Figure A.7 · p. 15
  • The THF dataset comprises 200 curated Total Capture images with pseudo-ground-truth for body, face, and hands, triangulated from multiview OpenPose detections.Images were selected for body visibility, interesting hand poses, and facial expressions; the subset focuses on PtCloudDB.Source: §4 · pp. 14–15
  • Table A.1 is cited for THF quantitative results, but its caption describes CMU Panoptic Studio mean 3D joint errors after Procrustes alignment.The caption specifies evaluations using ground-truth 2D joint reprojections and OpenPose detections, with alignments based on body-only or all body, hands, and face keypoints.Source: §4 · p. 15, Table A.1 · p. 15

5. Quantitative evaluation on Human3.6M

On Human3.6M, SMPLify-X outperforms SMPLify on mean 3D joint error after Procrustes alignment, though the authors caution that this metric does not capture reconstruction detail fully.

5. Quantitative evaluation on Human3.6M

  • The authors caution that 3D body-joint metrics do not fully represent reconstruction accuracy and detail, making this a crude measure of performance.Source: §5 · p. 15
  • On Human3.6M, SMPLify-X outperforms SMPLify in mean 3D joint error after Procrustes alignment under ’s protocol.The reported comparison uses 3D body-joint metrics.Source: Table A.2 · p. 15, §5 · p. 15
  • The comparison does not identify which of the several improvements over SMPLify contributes most; the authors direct readers to the main-text ablation study.The authors note that this experiment cannot determine the relative importance of those improvements.Source: §5 · p. 15

6. Qualitative evaluation on MPII

The paper presents a qualitative evaluation on MPII that includes low-resolution, heavily occluded, and cropped people.

6. Qualitative evaluation on MPII

  • Figure A.14 shows qualitative results on MPII, including low-resolution, heavily occluded, and cropped people.Source: §6 · p. 15

7. Model · 8. VPoser

The supplement reports SMPL-X shape-space and held-out alignment evaluation, then details VPoser's motion-data preparation, network training, and latent-space samples.

7. Model

  • More than 95% of CAESAR shape variance is explained by 10 principal components in each SMPL-X model.The cumulative relative variance is reported for the male, female, and gender-neutral models.Source: §7 · p. 15, Figure A.9 · p. 17
  • SMPL-X generalization is evaluated by vertex-to-vertex mean absolute error on 180 held-out male and female alignments, using gender-specific or neutral models as appropriate.The error is shown as a function of the number of principal components.Source: §7 · p. 15, Figure A.10 · p. 17

8. VPoser

  • VPoser is a new human-pose parametrization and prior; this supplement provides further details on its data preparation and implementation.Source: §8 · p. 15
  • The THF qualitative set comprises 200 curated total-capture images selected for body coverage, hand poses, and facial expressions.The figure compares fits using pseudo-ground-truth OpenPose keypoints with fits using estimated 2D OpenPose keypoints; model color indicates gender-specific or neutral selection.Source: Figure A.8 · p. 16

8.1. Data preparation

  • VPoser training uses SMPL pose parameters from CMU, Human3.6M, and PosePrior, with Human3.6M Subjects 9 and 11 held out for testing.Five percent of the training set is randomly selected for validation, and matrix rotations are used for the pose parameterization.Source: §8.1 · p. 17

8.2. Implementation details

  • VPoser uses a symmetric fully connected encoder–decoder with LReLU, trained in TensorFlow and later ported with its weights to PyTorch.The encoder has two 512-unit dense layers followed by separate mean and variance layers; training uses ADAM for 60 epochs at each of three learning rates: 5e−4, 1e−4, and 5e−5.Source: §8.2 · pp. 17–18
  • After training, VPoser's latent space describes a manifold of physically plausible human body poses that supports efficient 2D-to-3D lifting.The supplement shows random samples drawn from the latent space.Source: §8.2 · p. 18

9. Gender lassifier

The gender classifier is trained on curated, human-labeled person crops and implemented as a pretrained ResNet18-based network fine-tuned for classification.

9.1. Training data

  • Training uses person crops from five datasets, filtered for visible high-confidence joints and minimum size, with gender labels retained only when two annotators agree.The sources are LSP, LSP-extended, MPII, MS-COCO, and LIP; crops smaller than 200 × 200 pixels are rejected.Source: §9.1 · p. 18

9.2. Implementation details

  • The classifier uses Keras, TensorFlow, and pretrained ResNet18 features with fully connected layers, cross-entropy loss, and L2 weight regularization.After two epochs training the final layers, the entire network is fine-tuned end-to-end for two epochs with Adam across specified learning-rate schedules.Source: §9.2 · p. 18

Related papers

Loading 1904.05866v1…