Source-linked AI summary
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies
Hanbyul Joo, Tomas Simon, Yaser Sheikh
TL;DR
Markerless systems had not simultaneously captured body, face, and hand motion across the scales needed for multi-person interaction. The paper builds a unified model by stitching detailed part models into Frankenstein, then learns Adam from 70 subjects for simpler total-body capture with hair and clothing. The resulting system captures facial expressions, body motion, and hand gestures from sparse 3D keypoint detections in social and person-object interaction sequences.
Problem
Existing markerless systems do not simultaneously track the human body, face, and hands, whose different scales require incompatible capture configurations.
Method
The method combines multiview 3D keypoint detection with a unified deformation model, stitching part models into Frankenstein and optimizing it on captures of 70 subjects to create Adam.
Results
The system captures total-body motion including facial expressions, coarse body motion, and hand gestures at a distance using sparse 3D keypoint detections, with compelling and realistic reconstruction results.
Takeaways & Limitations
The models support markerless capture of coordinated face, hand, and body motion in social behavior and person-object interaction sequences.
Takeaways & Limitations
Model fitting requires priors because FaceWarehouse correspondences and joint locations can be inconsistent with detector-training annotations, creating sources of fitting noise.
Abstract
from arXiv · showhide
We present a unified deformation model for the markerless capture of multiple scales of human movement, including facial expressions, body motion, and hand gestures. An initial model is generated by locally stitching together models of the individual parts of the human body, which we refer to as the "Frankenstein" model. This model enables the full expression of part movements, including face and hands by a single seamless model. Using a large-scale capture of people wearing everyday clothes, we optimize the Frankenstein model to create "Adam". Adam is a calibrated model that shares the same skeleton hierarchy as the initial model but can express hair and clothing geometry, making it directly usable for fitting people as they normally appear in everyday life. Finally, we demonstrate the use of these models for total motion tracking, simultaneously capturing the large-scale body movements and the subtle face and hand motion of a social group of people.
1. Introduction
The paper addresses the lack of markerless systems that simultaneously capture body, facial, and hand motion during multi-person interaction. It combines multiview keypoints with a unified generative deformation model to capture these movements at different scales.
- Motivation: Simultaneous capture matters because subtle facial expressions, hand gestures, and body posture convey information about attention, emotion, and intention.These signals are part of social communication and machine perception of human activity.
- Motivation: Existing markerless systems specialize in body, face, or hand capture and use incompatible sensing configurations.Body capture requires a large working volume, whereas facial and finger capture typically require close-range, high-resolution imaging.
- Challenge: Scale differences make room-scale, multi-person capture difficult because off-the-shelf cameras provide limited resolution for faces and hands.Torso and limbs need broad coverage, while smaller features need close, frontal, high-resolution views.
- Approach: The method uses multiview keypoint detection and a generative deformation model to represent motion across the principal body parts.Keypoints are designed to remain robust to multiple people and object interactions.
- Approach: The authors stitch existing part models into Frankenstein, optimize it using 70 subjects, and learn Adam to capture hair and clothing with simpler parameterization.They demonstrate total-body capture on social behavior and person-object interaction sequences.
2. Related Work
Prior work generally captures body, face, or hands separately, while this paper targets markerless total-body motion by combining detailed part models within a unified framework.
- Markerless motion capture: Marker-based systems remain widely used for high-accuracy motion capture, while markerless methods have historically treated marker-based output as ground truth or an upper bound.The paper positions improved image keypoint detection as enabling progress beyond that convention.
- Markerless motion capture: Markerless body capture commonly aligns a 3D template with multiview measurements such as color, silhouettes, point clouds, and landmarks.Related work also studies improved body models for tracking in controlled multiview systems.
- Facial and hand capture: Facial capture has advanced through multiview scanning, performance capture, and lightweight single-camera morphable face models.These approaches focus primarily on facial motion rather than simultaneous total-body tracking.
- Facial and hand capture: Hand motion capture is largely led by single-depth-sensor methods, reflecting a separate capture track from body and facial systems.The paper instead incorporates hands into a unified model for motion capture at a distance.
- Model unification: Figure 2 depicts the progression from separate body, face, and hand models to aligned components and a seamless Frankenstein model.This illustrates the paper’s model-unification strategy rather than a prior-work result.
3. Frankenstein Model
Frankenstein unifies body, face, and hand part models under one skeletal hierarchy and blends their meshes into a seamless representation for joint motion and identity fitting.
- Model construction: Frankenstein combines SMPL, FaceWarehouse, and an artist-defined hand rig because each captures detail at an appropriate scale for its body part.The choice reflects the resolution–field-of-view trade-off in 3D scanning systems.
- Model construction: The model is parameterized by motion, shape, and global translation, and outputs a seamless mesh expressing the target subject’s motion and shape.The mesh is denoted VU, with parameters θU, φU, and tU.
- Model construction: Its motion and shape parameters are unions of the body, face, left-hand, and right-hand parameters.The superscripts B, F, LH, and RH identify the corresponding part models.
- Mesh stitching: The final mesh is formed by linearly blending component vertices with a sparse matrix that mostly copies corresponding vertices and minimally interpolates seams.The unified mesh has 18,540 vertices, fewer than the sum of the component meshes because redundant body-model regions are removed.
- Skeleton hierarchy: All parts are rigidly linked through one skeletal hierarchy, replacing the SMPL hand and face branches with detailed part-model hierarchies.All model parameters are jointly optimized for motion tracking and identity fitting.
- Body model: The body component uses a modified SMPL model with a 6,890-vertex template, 10 identity coefficients, and linear blend skinning.Pose-dependent blendshapes are omitted, and the body uses 21 joints in the stated parameterization.
- Face model: The face component is a generative PCA model with identity and facial-expression blendshapes, manually aligned so body head transformations carry the face into world coordinates.The model specifies 150 identity blendshapes and 200 expression blendshapes.
- Hand model: The hand component is an artist-rigged, 16-joint mesh using linear blend skinning, bone scaling, and Euler-angle joint rotations.Hand-to-body alignment uses a transform that depends on body shape parameters.
4. Motion Capture with Frankenstein Model
The Frankenstein model captures body, face, and hand motion by fitting multiview 3D keypoints and point clouds with a staged optimization objective.
- 3D Measurements: The framework combines triangulated 3D body, face, hand, and foot keypoints with uncorrelated multiview stereo points matched through ICP.Keypoints correspond to mesh joints or surface points, while point clouds provide additional geometric measurements with normals.
- Objective Function: Anatomical keypoint costs match detected joints and surface landmarks to model correspondences represented by a sparse vertex regression matrix.Each correspondence interpolates a position from a small number of model vertices and is compared with a 3D detection.
- Objective Function: ICP establishes mesh-to-point-cloud correspondences during optimization and minimizes point-to-plane distances subject to distance and normal thresholds.The residual measures distance along the normal direction for each matched mesh vertex.
- Objective Function: Seam constraints penalize discrepancies between neighboring rings of vertices so independently parameterized body parts remain continuous at their boundaries.The constraint addresses artifacts such as mismatched widths where parts meet.
- Optimization: The method masks regions around the hands and face because depth-sensor resolution is too low to improve their estimates.These regions are excluded from the relevant fitting term rather than used as reliable depth measurements.
- Optimization: Because the nonlinear model can reach poor local minima, fitting proceeds in phases from torso alignment with strong priors to all available keypoints with relaxed priors.Model fitting is performed independently for each frame; priors also regularize underdetermined parameters and noisy measurements from hair and clothing.
5. Creating Adam
Adam is built from Frankenstein reconstructions of clothed subjects into a unified model with shared shape parameters across the face, hands, and body. Its construction fits clothing and hair geometry, learns a joint shape space, and supports simpler total-body fitting without seam constraints.
- Learning Adam: Adam is learned from Frankenstein reconstructions of 70 subjects, using five frames in different poses per person.Joint locations and linear shape blendshapes are learned from these reconstructions of clothed people.
- Tracking with Adam: Adam uses one joint hierarchy and one type of shape and pose parameter for all parts, avoiding Frankenstein’s seam term and simplifying fitting.Its parameterization follows SMPL while adding hand joints and facial expression blendshapes.
- Fitting clothes and hair: Per-vertex displacements compensate for point-cloud differences along mesh normals, while Laplace-Beltrami regularization and weighting control deformation.Weights reduce large deformations where point-cloud resolution is lower, including facial and hand details.
- Building the shape deformation space: Fitted surfaces are warped to a canonical rest pose, then analyzed with PCA to construct a joint linear shape space across the entire body.The FaceWarehouse expression basis is retained because the multiview point clouds are too low-resolution to fit facial expressions.
- Building the shape deformation space: The resulting model represents body, hand, and face shape variation while also modeling hair and clothing deformation.Its blendshapes replace the separate body, face, and hand shape parameters used by the part-based models.
6. Results
The models are evaluated on room-scale sequences spanning body, face, hand, social, dexterous, musical, and everyday motions. Frankenstein and Adam support realistic total-body reconstruction, with Adam sometimes affected by source point-cloud artifacts.
- Experimental setup: Experiments compare Frankenstein and Adam with SMPL on CMU Panoptic Studio data reconstructed from multiview cameras.The setup uses 140 VGA cameras for body keypoints, 480 VGA cameras for feet, and 31 HD cameras for faces and hands keypoints and point clouds.
- Quantitative evaluation: The evaluation measures silhouette overlap with ground truth across 5 viewpoints during a 10 second range-of-motion sequence.Ground-truth silhouettes are obtained by background subtraction with Gaussian pixel models and morphological noise removal.
- Quantitative evaluation: Frankenstein improves over SMPL primarily through its articulated hand model, while the body is almost identical by construction.Including the ICP term as an additional cue provides better accuracy.
- Quantitative evaluation: Frankenstein and Adam show almost similar performance, although Adam performs better for certain body shapes and sometimes produces artifacts such as thinner legs.The artifacts are attributed to poor 3D point-cloud reconstructions in the source data used to train Adam.
- Qualitative results: The method is tested on social interactions, dexterous furniture building, musical performances, typing, and other sequences where face and hand motions emerge with body movement.The authors report better scene realism than SMPL by capturing subtle hand and face details.
7. Discussion
The paper presents markerless total-body motion capture that includes facial expressions, coarse body motion, and hand gestures at a distance. Its models produce compelling and realistic reconstructions from sparse 3D keypoint detections.
- Discussion: The method captures facial expression, coarse torso and limb motion, and hand gestures simultaneously at a distance.It is presented as the first markerless method covering these total-body motion components together.
- Discussion: The reconstruction results remain compelling and realistic even when sparse 3D keypoint detections drive the models.This demonstrates the models' ability to express motion across the principal body parts from sparse measurements.