Source-linked AI summary
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration
Yu Rong, Takaaki Shiratori, Hanbyul Joo
TL;DR
Monocular capture has often focused on body or hand motion alone, while prior joint body-and-hand methods relied on relatively slow optimization. FrankMocap regresses body and hand poses separately and integrates them into a unified SMPL-X output. It runs at 9.5 fps in a live demo and its hand module outperforms prior hand-only methods on public benchmarks.
Problem
Monocular capture methods have often focused on body or hand motion separately, and prior joint body-and-hand approaches relied on relatively slow optimization.
Source: §1 · pp. 1–2Method
Results
FrankMocap runs at 9.5 fps in a live near-real-time demo—orders of magnitude faster than alternative methods—and its hand module outperforms prior hand-only methods on public benchmarks.
Source: §5 · pp. 10–11Takeaways & Limitations
The system demonstrates whole-body motion capture in real-world monocular videos, including a live demo at near-real-time speed.
Source: §5 · p. 10Hand pose estimation can become erroneous when two hands are too close, and bounding boxes are required to infer 3D body and hands.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Although the essential nuance of human motion is often conveyed as a combination of body movements and hand gestures, the existing monocular motion capture approaches mostly focus on either body motion capture only ignoring hand parts or hand motion capture only without considering body motion. In this paper, we present FrankMocap, a motion capture system that can estimate both 3D hand and body motion from in-the-wild monocular inputs with faster speed (9.5 fps) and better accuracy than previous work. Our method works in near real-time (9.5 fps) and produces 3D body and hand motion capture outputs as a unified parametric model structure. Our method aims to capture 3D body and hand motion simultaneously from challenging in-the-wild monocular videos. To construct FrankMocap, we build the state-of-the-art monocular 3D "hand" motion capture method by taking the hand part of the whole body parametric model (SMPL-X). Our 3D hand motion capture output can be efficiently integrated to monocular body motion capture output, producing whole body motion results in a unified parrametric model structure. We demonstrate the state-of-the-art performance of our hand motion capture system in public benchmarks, and show the high quality of our whole body motion capture result in various challenging real-world scenes, including a live demo scenario.
1. Introduction
Monocular motion capture has advanced for bodies and hands separately, but whole-body capture is difficult in the wild and existing joint methods are relatively slow. FrankMocap regresses body and hand poses separately, then integrates them into a unified SMPL-X output.
1. Introduction
- Body-pose methods often omit subtle finger gestures, while hand-pose methods are commonly demonstrated in hand-specific views rather than challenging in-the-wild scenes.In the wild, hands may be low-resolution and frequently blurred.Source: §1 · pp. 1–2
- Prior joint body-and-hand methods relied on optimization to fit parametric models to images, making them relatively slow and unsuitable for real-time applications.The hands’ small size also makes joint capture difficult, even with professional motion capture systems.Source: §1 · pp. 1–2
- FrankMocap combines separate body and hand pose regressors through compatible SMPL-X outputs, with copy-and-paste integration running at approximately 9.5 fps.An optimization framework is also presented to improve whole-body pose estimates.Source: §1 · p. 2
- The authors report that FrankMocap’s hand estimator outperforms previous approaches on public benchmarks and demonstrate the system on real-world monocular videos, including a real-time demo.They also report ablation studies and comparisons with prior hand and whole-body capture methods.Source: §1 · p. 2
2. Related Work
Related work spans parametric models and monocular body and hand estimation. Prior approaches to estimating body and hands together used optimization, with relatively slow computation.
2. Related Work
- Parametric human models represent body deformation with low-dimensional parameters rather than original mesh vertices.Examples include SMPL, MANO, SMPL+H, Adam, and SMPL-X.Source: §2 · p. 2
- Monocular body-pose methods range from predicting 3D keypoints to regressing parametric-model parameters, with some approaches fitting models to image observations through optimization.Keypoint-only outputs may lack joint angles and preserve no part lengths, limiting direct use in graphics applications.Source: §2 · p. 2
- Figure 2 depicts a pipeline that estimates 3D hands and body separately from one RGB image, then combines them into a unified whole-body output.The paper uses “whole body” to include fingers and “body” for torso and limbs excluding fingers.Source: §2 · p. 2, Figure 2 · p. 3
- Monocular hand-pose methods include approaches that regress MANO parameters and others that predict heatmaps before regressing hand-model vertices.The cited RGB methods differ in their use of OpenPose predictions, heatmaps, masks, and hand models.Source: §2 · p. 3
- Prior joint body-and-hand methods used optimization and took from 10 seconds to a few minutes; their accuracy also degrades significantly when 2D heatmap detection fails.SMPLify-X fits SMPL-X parameters to 2D keypoints with additional constraints, while Monocular Total Capture optimizes Adam parameters from 2.5D predictions.Source: §2 · p. 3
3. Method
FrankMocap estimates body and hand poses separately from monocular inputs, then combines compatible outputs in a unified SMPL-X representation. It offers direct parameter composition and an optimization-based integration strategy.
3. Method
- Separate body and hand modules produce compatible SMPL-X outputs that can be integrated directly or refined through optimization.The optimization framework adds constraints to improve accuracy.Source: §3 · p. 3
3.1. Overview of SMPL-X Model
- SMPL-X hand components share parameterization with the whole-body model, enabling efficient integration of hand pose, vertices, and joints.The hand model uses 778 vertices and shares its shape parameterization space with the whole-body model.Source: §3.1 · p. 4
3.2. 3D Hand Estimation Module
- The hand module regresses hand-model parameters from monocular images using an end-to-end network inspired by body-pose estimators.It uses an encoder-decoder architecture with ResNet-50 encoding and fully connected decoder layers.Source: §3.2 · pp. 4–5
- Training preprocesses diverse hand datasets for compatible keypoint scales and ordering, and motion-blur augmentation improves generalization to in-the-wild scenes.The preprocessing rescales keypoints using middle-finger knuckle length and aligns them to the model's skeleton hierarchy.Source: §3.2 · pp. 5–6
3.3. 3D Body Estimation Module
- The body module adapts a state-of-the-art SMPL-based estimator to SMPL-X, but wrist and arm orientation remains inaccurate when annotations are insufficient or inaccurate.The paper gives the example of only one annotated wrist keypoint.Source: §3.3 · p. 6
3.4. Whole Body Integration Module
- Integration either copies compatible body and hand parameters with wrist-orientation conversion or optimizes the whole-body model using 2D keypoints and pose and shape priors.The optimization initializes from copy-and-paste composition and uses 20 iterations in the reported experiments.Source: §3.4 · pp. 6–7
4. Experiments
Experiments compare the hand module on public benchmarks and assess whole-body integration against prior approaches. They also examine training-data and augmentation choices.
4.1. Implementation Details
- Online hand boxes are obtained by projecting the estimated body into the image; offline processing uses OpenPose detections for both bodies and hands.Optimization-based video outputs receive five-frame smoothing, while copy-and-paste outputs are processed frame by frame without post-processing.Source: §4.1 · p. 7
4.4. Integration Module Evaluation
- About 9.5 fps, copy-and-paste runs near real time; it is two orders of magnitude faster than alternatives while showing better 3D pose quality.The processing speed is reported for a single GeForce RTX 2080 GPU; the comparison is with alternative whole-body approaches.Source: Table 1 · p. 7, §4.1 · p. 7, §4.4 · p. 10
4.2. Datasets
- The hand module is trained using FreiHAND and HO-3D real-hand annotations, MTC annotations, STB joints, RHD synthetic data, and in-the-wild MPII+NZSL validation images.MPII+NZSL is used only for validation, not training; MTC samples with hands not visible or too small are filtered out.Source: §4.2 · p. 8
4.3. Hand Module Evaluation
- The hand module outperforms prior methods on RHD and MPII+NZSL and performs comparably on STB, with significantly better 2D localization on MPII+NZSL.STB and RHD use 3D AUC; MPII+NZSL uses 2D AUC.Source: Table 2 · p. 8, §4.3 · p. 9
- Including varied datasets and data augmentation improves results, while estimating shape parameters also helps accuracy.The limited-dataset variant remains comparable to prior methods on STB and RHD and has better in-the-wild 2D localization on MPII+NZSL.Source: §4.3 · pp. 9–10
- Using more datasets improves performance and in-the-wild generalization; augmentation improves generalization to blur, challenging poses, and occlusion.The dataset ablation includes qualitative comparisons on COCO; the augmentation ablation compares no augmentation, no motion-blur augmentation, and full augmentation.Source: Table 3 · p. 8, Table 4 · p. 9, §4.3 · p. 10
- Qualitative comparisons show the hand model produces more precise 3D poses in challenging in-the-wild scenes with occlusion, blur, and low resolution.Whole-body integration is also compared qualitatively with MTC and SMPLif-X.Source: §4.3 · p. 10, §4.4 · p. 10
5. Discussion
FrankMocap combines 3D hand and body capture, reaching 9.5 fps in a live demo while outperforming prior hand-only methods on public benchmarks. Its integration strategies produce faster, more convincing whole-body outputs than prior methods, but close hands and bounding-box requirements remain limitations.
5. Discussion
- 9.5 fps in a live demo, with whole-body capture orders of magnitude faster than alternatives and hand estimation outperforming prior hand-only methods on public benchmarks.Source: §5 · pp. 10–11
- Ours-CP and Ours-OP are faster and produce more convincing whole-body outputs than MTC [55] and SMPLify-X [37].Source: Figure 10 · p. 11
- Close hands can make hand-pose estimates erroneous, and the system requires bounding boxes to infer 3D body and hands.Source: §5 · p. 11