Source-linked AI summary
Embodied Hands: Modeling and Capturing Hands and Bodies Together
Javier Romero, Dimitrios Tzionas, Michael J. Black
TL;DR
Existing body and hand methods model and track them separately, despite the importance of coordinated motion and the difficulty of recovering hands from full-body scans. The paper introduces MANO, combines it with SMPL as SMPL+H, and fits the result to challenging 4D scans, producing natural full-body motion with detailed hands.
Problem
Body and hand modeling and tracking are usually separate, while full-body scans make hands difficult to recover because of low resolution, noise, and occlusion.
Method
The paper learns MANO from detailed hand scans and attaches it to SMPL, producing the fully articulated SMPL+H body-and-hand model for joint 4D capture.
Results
SMPL+H fits complex 4D scan sequences automatically and produces full-body models with natural movement and detailed hand motion despite noisy, low-resolution, partially missing data.
Takeaways & Limitations
Jointly capturing bodies and hands enables study of correlated motion at a level of detail not previously possible, with models and datasets released for research.
Takeaways & Limitations
The method does not explicitly reason about self-contact or hand-object contact during fitting.
Abstract
from arXiv · showhide
Humans move their hands and bodies together to communicate and solve tasks. Capturing and replicating such coordinated activity is critical for virtual characters that behave realistically. Surprisingly, most methods treat the 3D modeling and tracking of bodies and hands separately. Here we formulate a model of hands and bodies interacting together and fit it to full-body 4D sequences. When scanning or capturing the full body in 3D, hands are small and often partially occluded, making their shape and pose hard to recover. To cope with low-resolution, occlusion, and noise, we develop a new model called MANO (hand Model with Articulated and Non-rigid defOrmations). MANO is learned from around 1000 high-resolution 3D scans of hands of 31 subjects in a wide variety of hand poses. The model is realistic, low-dimensional, captures non-rigid shape changes with pose, is compatible with standard graphics packages, and can fit any human hand. MANO provides a compact mapping from hand poses to pose blend shape corrections and a linear manifold of pose synergies. We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H). We illustrate SMPL+H by fitting complex, natural, activities of subjects captured with a 4D scanner. The fitting is fully automatic and results in full body models that move naturally with detailed hand motions and a realism not seen before in full body performance capture. The models and data are freely available for research purposes in our website (http://mano.is.tue.mpg.de).
1 INTRODUCTION
The paper addresses the separation of body and hand modeling by learning MANO and combining it with SMPL into SMPL+H for joint full-body capture. The combined model fits noisy, low-resolution 4D scans and produces natural motion with detailed hand articulation.
- Body and hand research has progressed separately, although their joint analysis is important for representing communication, actions, emotions, and intentions.
- A database of detailed hand scans covers 31 subjects, up to 51 poses, both hands, varied poses, and interactions with objects.
- MANO is a learned statistical hand model that separates identity- and pose-dependent geometry using pose blend shapes and SMPL-compatible components.
- MANO reduces hand-pose dimensionality through a linear embedding and encourages pose corrections to remain local despite the hand’s many restricted joints.
- Combining MANO with SMPL yields SMPL+H, which adds articulated hand joints and hand-specific components to body shape correlations.
- The fully automatic fitting procedure handles noisy, low-resolution, and partially missing hand data, producing natural full-body models with detailed hand motion.
2 RELATED WORK
Prior work uses diverse hand representations and capture strategies, but most systems model hands separately from the body. The paper positions MANO as a learned, articulated alternative that supports joint body-hand modeling.
- Hands are difficult to detect and track because they are dexterous, visually similar across fingers, fast-moving, self-occluding, and small in full-body scans.
- Marker, glove, and wearable-camera systems can simplify hand tracking but are intrusive, may restrict natural motion, and require synchronization with full-body capture.
- RGB-D and multicamera methods offer different resolution and volume trade-offs, yet existing approaches generally do not capture full bodies and hands together.
- Hand models range from shape primitives and Gaussian mixtures to meshes, subdivision surfaces, convex parts, and CNN-based learned representations.
- Low-dimensional hand-pose spaces exploit pose synergies, with prior studies reporting that a few components explain much of observed pose variance.
- MANO replaces SMPL’s insufficiently articulated hands with a learned model that can operate alone or jointly estimate hand poses and body movement.
3 MODEL
The model is built in two stages: detailed isolated hand scans are registered and used to learn MANO, then MANO is integrated with SMPL as SMPL+H. The resulting model represents identity- and pose-dependent hand geometry and supports automatic fitting to challenging scans.
- Hand model: MANO extends SMPL with identity-dependent shape blend shapes and pose-dependent corrective blend shapes that produce more natural finger bending than standard linear blend skinning.The model uses PCA to learn a low-dimensional shape basis from registered hand shapes, while pose corrections are driven by rotation-matrix elements.
- Registration: Registration bootstraps model learning by repeatedly aligning a template, manually curating good fits, and retraining an improved model.The process addresses self-similar fingers and self- or object-occlusion without relying on landmarks.
- Registration: The registration energy combines robust scan geometry, model coupling, pose-prior, and shape-prior terms.Direct vertex optimization permits faithful scan alignment, while model coupling keeps registrations conservatively close to the learned model.
- Registration: 1554 of 2018 registrations were accepted after inspection, and aligned meshes capture detailed pose and shape variation across subjects.The released training database contains the successful registrations; examples show correspondence between scans and aligned meshes.
- Hand model: MANO contains 15 joints plus global orientation and uses stronger vertex-joint-dependent regularization to localize pose-dependent deformations.This design addresses hand pose over-parameterization, and the resulting deformations are reported as more local and regularized.
4 CAPTURING BODY AND HAND MOTION
The authors fit SMPL+H to challenging full-body 4D scans, where hands are low-resolution and require specialized priors and coupled alignment. A two-stage subject-specific registration improves accuracy and robustness while reducing optimization dimensionality.
- Capture setup: 60 fps 3D scanning captures full-body motion, but hands suffer low resolution and missing data because no camera is dedicated to them.The system uses overlapping stereo-camera views for activities including arm movements, running, throwing, and jumping.
- Datasets: The evaluation includes 11 unconstrained sequences from 10 subjects, with additional sequences collected for a single subject and a 50-scan, six-subject hand test set.The hand test set covers single- and double-finger articulation and coordinated movement of all fingers.
- Subject-specific sequence alignment: The two-stage procedure first builds a subject-specific template, then tracks all frames with subject-specific registration and coupled alignment only for subsequent frames.After acquiring the template, optimization omits 20 shape parameters, making registrations more accurate, robust, and faster.
- Fitting objective: A Gaussian mixture model provides the hand-pose prior because hand deviations are non-Gaussian and Gaussian priors can trap poses near the mean when data disappear.The mixture is computed separately for left and right hands, with 10 components found empirically to work well.
- Fitting objective: The fitting objective combines scan alignment, coupling to the model, shape and hand-pose regularization, and temporal zero-velocity smoothing.The body pose prior is removed for flexibility, while the hand prior is retained because hand observations can disappear for several frames.
- Subject-specific sequence alignment: The proposed registration reduces dimensionality by 20 parameters and makes registrations more accurate and robust.Optimization uses gradient-based dogleg minimization with automatic differentiation and takes approximately 4 minutes per frame on the reported computer.
5 EXPERIMENTS
Experiments show that MANO is compact, generalizes to unseen hand poses and shapes, and benefits from mirrored training data, learned pose corrections, and parameter learning. SMPL+H improves realism and remains natural under noisy, missing, and fast-motion scans, while failures occur for unusual poses and hand-object interactions.
- MANO evaluation: 6, 10, and 15 pose components explain 81%, 90%, and 95% of the training-set variance, respectively.The authors use fewer components for full-body data because they provide stronger regularization against noise and overfitting.
- MANO evaluation: MANO generalizes to unseen poses and shapes, with scan-to-mesh and mean absolute errors decreasing monotonically as components increase.Pose generalization is evaluated on 50 unseen right-hand scans, while shape generalization uses leave-one-out evaluation across training subjects.
- MANO evaluation: 0.93 mm versus 1.05 mm full-dimensional pose-space error shows that augmenting right-hand training data with mirrored left poses improves performance.The mirrored-data model is selected for its lower error and more coherent training procedure.
- MANO evaluation: 0.93 mm with pose blend shapes versus 1.3 mm without them shows that learned corrections improve both fitting accuracy and visual realism.The comparison uses a fixed personalized template and the full pose space.
- SMPL+H evaluation: SMPL+H captures expressive hand poses and improves visual realism over SMPL, which keeps hands flat and open.The model also performs naturally under severe missing data, fast motions, and measurement noise, but failures occur for unusual poses and hand-object interaction.
6 CONCLUSIONS
The paper introduces MANO and SMPL+H for realistic, joint modeling and capture of hands and bodies, including under noisy or missing data. It also identifies limitations and future directions for broader interaction modeling and lower-quality inputs.
- MANO is a learned, low-dimensional, realistic hand model that is easy to pose and compatible with graphics engines through linear blend skinning and blend shapes.
- SMPL+H combines MANO with a body model to provide learned full hand articulation within a human body shape model.
- SMPL+H fits natural full-body movements stably despite noisy, low-resolution, and partially missing scan data, producing natural-looking avatars.
- Mirroring augments MANO training data and improves quantitative fitting, indicating that richer datasets are needed in future work.
- The low-dimensional pose space explains natural hand poses efficiently, but unusual poses can fall outside it and require full-space optimization or richer nonlinear models.
- Current limitations include unmodeled self-contact, hand-object interaction, and the challenge of estimating fingers from lower-quality RGB-D or RGB data.
MODEL/SCAN MIRRORING
The mirroring procedure converts left-hand scans and model components into a unified right-hand training representation. Geometry and shape blend shapes mirror directly, while pose-dependent components require rotation-aware transformations.
- Left-hand scans are mirrored into right-hand form to augment the training set and reduce handedness bias.
- The sagittal plane defined in SMPL serves as the mirroring plane.
- Scan points, template vertices, and shape blend shapes are mirrored by applying the mirroring transformation to their coordinates or displacements.
- Pose blend shapes require additional treatment because hand rotations depend on the global coordinate frame.
- Mirrored rotations are formed by pre- and post-multiplying the original rotation matrix by the mirroring transformation.
- The mirrored pose blend shapes are obtained by applying the rotation un-mirroring transformation to each 3 × 3 input block.
EVALUATION - MANO HAND MODEL
The evaluation examines MANO generalization across pose-space dimensionalities, training stages, and dataset constructions. Training reduces fitting error, and mirroring augmentation further lowers error while supporting a unified hand model.
- 2.90 mm before training versus 1.01 mm after training is reported for the full-space model.
- SMPL+H recovers natural motion in fast actions despite missing hand data in several scan frames.
- Figure A.3 compares models trained on right-hand scans alone with models trained on right-hand and mirrored left-hand scans.
- The augmented dataset produces lower fitting error across pose-space component counts and avoids separate handedness-specific MANO models.
- Figure A.2 compares mean scan-to-mesh error before and after training across varying numbers of pose-space components.