Source-linked AI summary
Monocular Expressive Body Regression through Body-Driven Attention
Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, Michael J. Black
TL;DR
Holistic expressive 3D reconstruction from a single RGB image is difficult because expressive models are high-dimensional, training data are scarce, and small hands and faces lose detail after downscaling. ExPose directly regresses SMPL-X parameters, using curated in-the-wild fits and body-driven high-resolution refinement, and achieves competitive or better accuracy than optimization-based methods at much higher speed.
Problem
Single-image reconstruction of the body, face, and hands together is limited by high-dimensional expressive models, scarce in-the-wild SMPL-X data, and low-resolution face and hand evidence.
Method
ExPose directly regresses SMPL-X parameters from RGB images, using curated pseudo-ground-truth fits, body-driven attention, dedicated part refiners, and hand- and face-only supervision.
Results
ExPose is at least as accurate as existing optimization-based methods while running two orders of magnitude faster.
Takeaways & Limitations
Body-driven attention produces consistently better expressive reconstructions while retaining the speed advantages of regression.
Abstract
from arXiv · showhide
To understand how people look, interact, or perform tasks, we need to quickly and accurately capture their 3D body, face, and hands together from an RGB image. Most existing methods focus only on parts of the body. A few recent approaches reconstruct full expressive 3D humans from images using 3D body models that include the face and hands. These methods are optimization-based and thus slow, prone to local optima, and require 2D keypoints as input. We address these limitations by introducing ExPose (EXpressive POse and Shape rEgression), which directly regresses the body, face, and hands, in SMPL-X format, from an RGB image. This is a hard problem due to the high dimensionality of the body and the lack of expressive training data. Additionally, hands and faces are much smaller than the body, occupying very few image pixels. This makes hand and face estimation hard when body images are downscaled for neural networks. We make three main contributions. First, we account for the lack of training data by curating a dataset of SMPL-X fits on in-the-wild images. Second, we observe that body estimation localizes the face and hands reasonably well. We introduce body-driven attention for face and hand regions in the original image to extract higher-resolution crops that are fed to dedicated refinement modules. Third, these modules exploit part-specific knowledge from existing face- and hand-only datasets. ExPose estimates expressive 3D humans more accurately than existing optimization methods at a small fraction of the computational cost. Our data, model and code are available for research at https://expose.is.tue.mpg.de .
1 Introduction
Holistic reconstruction of the expressive 3D body, face, and hands from one RGB image remains difficult because existing full-body methods are slow and image resolution and training data limit detail. ExPose addresses these issues with direct SMPL-X regression, curated training fits, and body-driven attention.
- Motivation: Full expressive 3D reconstruction is needed to capture body, face, and hands together from a single RGB image.Prior work commonly estimates these components separately.
- Motivation: SMPL-X and related expressive models represent detailed 3D human surfaces beyond 2D joints, including shape, smiles, gestures, and object interactions.These parametric models provide a strong prior for reconstruction from ambiguous data.
- Limitations of prior work: Existing full expressive methods optimize model parameters, making them slow, sensitive to local optima, and dependent on initialization heuristics.Direct body-only regressors are faster, motivating a regression approach for expressive humans.
- Challenges: 256×256 input downscaling preserves body resolution but leaves the much smaller face and hands poorly resolved, especially under blur or occlusion.The resulting low resolution makes expressive hand and face inference difficult.
- Approach: ExPose curates in-the-wild SMPL-X fits, uses body estimates to localize face and hands, and refines them from higher-resolution original-image crops.Dedicated refinement modules also exploit hand-only and face-only training data.
- Result: ExPose is at least as accurate as existing optimization-based methods while running two orders of magnitude faster.The paper presents this as the model's principal efficiency and accuracy result.
2 Related Work
Related work progresses from separate body, face, and hand modeling toward richer surface representations and direct RGB regression. ExPose contributes a regression approach for expressive monocular capture that uses higher-resolution part processing rather than only pooled low-resolution features.
- Human Modeling: Human modeling has traditionally treated the face, hands, and body separately, while Adam and SMPL-X represent these parts jointly.Earlier body models such as SMPL used neutral faces and non-articulated hands.
- Human Pose Estimation: Pose estimation methods use 2D or 3D keypoints, whereas newer approaches represent the 3D body surface with parametric or non-parametric models.This reflects a shift from landmark-only representations toward richer shape descriptions.
- Human Pose Estimation: Image-based body reconstruction may stage intermediate joints, silhouettes, labels, or correspondences before optimization or learned 3D lifting.These pipelines use priors to address ambiguities in lifting 2D information to 3D.
- Human Pose Estimation: Direct RGB methods avoid intermediate information bottlenecks but must learn a harder mapping from pixels to 3D pose or surface representations.They may predict joints, model parameters, meshes, depth maps, voxels, or distance fields.
- Part-specific estimation: Hand and face literature includes monocular RGB methods that estimate joints, meshes, or parameters of statistical models such as MANO.These part-specific datasets and models provide knowledge that can support expressive reconstruction.
- Attention for Human Pose Estimation: Pose-estimation attention architectures use intermediate predictions to guide later processing toward relevant image pixels.ExPose differs by processing localized face and hand regions at higher resolution, not merely pooling existing features.
- Expressive Human Estimation: ExPose is presented as the first regression approach for expressive monocular capture and is reported as more accurate and significantly faster than prior optimization-based work.Its distinction is direct regression combined with high-resolution region refinement.
3 Method
ExPose represents expressive humans with SMPL-X and uses body predictions to guide high-resolution hand and face refinement. It combines curated SMPL-X fits with part-specific training data and jointly trained body, hand, and face networks.
- 3.1 3D Body Representation: SMPL-X represents body shape, limb articulation, and facial expressions using a 338-dimensional parameter vector.The model includes 10 shape coefficients, 10 expression coefficients, and pose parameters for 53 major joints, producing a mesh with 10,475 vertices.
- 3.2 Body-driven Attention: The body network predicts initial SMPL-X parameters and camera information from a body crop extracted from the full-resolution image.Projected joints from this initial estimate define hand and face regions for subsequent processing.
- 3.2 Body-driven Attention: Spatial transformers extract higher-resolution hand and face crops, which dedicated networks refine using part-specific parameter predictions.The hand network refines wrist orientation and finger articulation, while the face network refines expression and jaw pose.
- 3.2 Body-driven Attention: Body-driven attention uses full-resolution image information to address the low pixel resolution of hands and faces in body crops.The method also enables hand- and face-only datasets to supplement training of the corresponding refinement networks.
- Training: Training combines body, hand, face, and part-specific reprojection losses, after separate pre-training followed by joint fine-tuning.The body loss includes reprojection, 3D-joint, and SMPL-X terms; the curated SMPL-X-fit dataset is instrumental during joint fine-tuning.
- Architecture: The architecture uses HRNet for body features and ResNet18 for hand and face refinement, with iterative residual regression of parameters.The implementation uses ResNet18 for the part networks to limit computational cost.
4 Experiments
Experiments evaluate ExPose on holistic, body-only, hand-only, and face-only reconstruction tasks using datasets with SMPL or SMPL-X ground truth. Results show strong reconstruction quality, improvements from body-driven attention and end-to-end fine-tuning, and substantially faster inference than optimization-based baselines.
- 3DPW evaluation: ExPose outperforms HMR and is on par with SPIN for main-body reconstruction on 3DPW.The evaluation uses per-joint and per-vertex errors for the SMPL body part.
- EHF ablation: Body-driven attention and end-to-end fine-tuning improve holistic EHF reconstruction, especially for hands and the face.The ablation compares a low-resolution body network, attention-based refinement, and a jointly fine-tuned final regressor.
- EHF comparisons: ExPose outperforms SMPLify-X and MTC overall on EHF, while SMPLify-X is locally better for hands and face.ExPose is described as more robust overall, whereas SMPLify-X can achieve finer local detail but depends on optimization and keypoint detections.
- EHF comparisons: ExPose is approximately two orders of magnitude faster than SMPLify-X and MTC.The comparison uses one-person timings measured on an Intel Xeon W-2123 CPU and Quadro P5000 GPU; regression methods also require bounding-box processing.
- Part-specific evaluation: The hand and face subnetworks match or approach state-of-the-art performance on FreiHAND and Stirling/ESRC 3D.These evaluations use dedicated hand-only and face-only datasets, while the final part networks use a shallower ResNet-18 backbone than ResNet-50 methods.
- Qualitative results: Qualitative results show clearer hand and face reconstructions and more detailed multi-view outputs with ExPose.The comparison is against direct low-resolution regression without attention.
5 Conclusion
ExPose uses body-driven attention to improve holistic expressive body reconstruction, recovering hand and face details while retaining competitive quality at much higher speed than optimization-based fitting.
- 5 Conclusion: Body-driven attention improves reconstructions of fine hand and face details that naive low-resolution holistic regression misses.The method targets the different image scales of body parts, especially small hands and faces.
- 5 Conclusion: ExPose offers competitive results at more than two orders of magnitude greater speed than SMPLify-X.The conclusion contrasts the practical speed of regression with the finer details recovered by optimization.
- 5 Conclusion: The approach is positioned as a useful tool for applications requiring expressive human pose information.The paper identifies human-scene, human-object, and person-person interaction as potential application areas.
- 5 Conclusion: ExPose produces results of similar quality to SMPLify-X while running 200× faster.The comparison is illustrated in Figure 4 and reiterated in Figure 5.
Monocular Expressive Body Regression through Body-Driven Attention *Supplemental Material*
The supplemental material provides additional details and visualizations of ExPose's results, directing readers to the project website's narrated video for a method and results summary.
- Monocular Expressive Body Regression through Body-Driven Attention *Supplemental Material*: The supplemental material adds details and visualizations of the reported results.It supplements the main paper with additional presentation material.
- Monocular Expressive Body Regression through Body-Driven Attention *Supplemental Material*: A narrated video on the project website summarizes the method and results.The video is offered as a companion resource.
1 Training details
The training details describe a multi-scale body feature extractor, iterative parameter regression, and separate pretraining followed by fine-tuning for hand and face subnetworks.
- 1 Training details: HRNet extracts multi-scale body feature maps that are aligned, concatenated, processed by residual blocks, and pooled into the body feature vector.The feature extractor uses extra convolutional blocks and five residual blocks before global average pooling.
- 1 Training details: The body network predicts SMPL-X parameters by iteratively regressing offsets from mean parameters using body features.The feature extractor and iterative regressor together form the body network g.
- 1 Training details: Hand training examples perturb the starting global rotation uniformly within −90 to 90 degrees.The perturbed rotation is applied around the ground-truth axis, with blue showing the target mesh and gray the starting point.
- 1 Training details: The body, hand, and head networks are pretrained separately before the body network is frozen and the part subnetworks are fine-tuned jointly.Pretraining uses 3DPW for the body, FreiHAND for hands, and FFHQ for the head.
2 Data augmentation
Data augmentation perturbs part-network initialization and facial articulation to reduce the domain gap between part-only training images and body-conditioned inputs.
- 2 Data augmentation: Randomizing the initial mean point helps bridge the domain gap between hand- or face-only images and body-attention crops.The augmentation makes part subnetworks robust to varied starting configurations.
- 2 Data augmentation: Face training perturbs the global head rotation uniformly within −45 to 45 degrees around the ground-truth rotation axis.The blue mesh is the target and the gray mesh is the perturbed starting point.
- 2 Data augmentation: Jaw augmentation samples a random x-axis rotation from 0 to 45 degrees instead of always starting from a closed mouth.This introduces variation in initial jaw articulation.
3 Converting SMPL to SMPL-X
The paper converts SMPL meshes into SMPL-X topology using geometric correspondences, then optimizes SMPL-X pose, translation, shape, and expression to match the converted mesh.
- Correspondence construction: SMPL and SMPL-X are related through a correspondence map that supports converting existing SMPL annotations into SMPL-X training data.The models are articulated human-body models producing 3D triangle meshes.
- Correspondence construction: For each SMPL-X vertex, the method stores its nearest SMPL point, containing the corresponding triangle, vertex indices, and barycentric coordinates.Binary masks mark invalid correspondences such as the eyes and inner lip region.
- Shape and expression variation: Random vectors replace the default head and hand mean shapes or the head’s neutral expression to visualize variation in shape and expression.The figures compare mean or neutral meshes against samples drawn from the stated distributions.
- Parameter fitting: A posed SMPL mesh is transferred into SMPL-X topology, producing a mesh used to recover pose, shape, expression, and translation parameters.The transferred mesh is compared with the SMPL-X mesh generated by the current parameters.
- Parameter fitting: The fitting proceeds from pose optimization to translation optimization, followed by joint optimization of θ, β, ψ, and t using vertex-to-vertex losses.The first stage optimizes pose, while the second aligns translation before the final full-parameter fit.
- Optimization: A Trust Region Newton Conjugate Gradient optimizer searches for the minima of the transfer objectives.The implementation is made available on the project website.
4 SMPLify-X qualitative comparison
ExPose produces qualitatively similar results to SMPLify-X while substantially reducing computation time, with a reported speedup of almost 200×.
- Comparison with SMPLify-X: Almost 200× faster than SMPLify-X, ExPose provides qualitatively similar results, although its accuracy is slightly lower.ExPose can also provide a better initialization for SMPLify-X, helping address failures from initialization heuristics and keypoint detection.
5 In-the-wild qualitative results
In-the-wild comparisons show that body-driven attention improves detailed hand and facial reconstruction over naive body-crop regression. ExPose matches SPIN on overall body pose and shape while recovering richer expressive details, with predictions also visualized from multiple views.
- Comparison with optimization: ExPose recovers detailed finger articulation and facial expressions with results similar in quality to SMPLify-X while being 200 times faster.The qualitative comparison contrasts the input, SMPLify-X, naive regression, and ExPose.
- Robustness to detections: When 2D keypoints are missing or incorrect, SMPLify-X can produce implausible poses, whereas ExPose avoids these initialization problems and can provide better starting points.The comparison links optimization failures to noisy detections and failed initialization heuristics.
- Attention mechanism: Naive regression from a body crop misses detailed finger articulation and facial expressions, whereas ExPose’s attention mechanism captures them.The comparison uses the input image, naive body-crop regression, and ExPose outputs.
- Multi-view visualization: ExPose predictions are additionally visualized from multiple views, including overlays and rotations around the vertical axis.Figures 21–25 provide the multi-view visualizations.