Source-linked AI summary
Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data
Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, Feng Xu
TL;DR
Monocular hand capture must handle difficult geometry and limited, heterogeneous training data while remaining fast and useful for animation. The paper combines multi-modal learning with joint detection and inverse kinematics, achieving over 100fps and reported improvements over the state of the art. Its pose augmentation relies on assumptions about finger independence and valid quaternion interpolation.
Problem
Monocular hand capture needs real-time single-camera operation and joint-angle outputs, but existing methods cannot use all available data modalities and often predict positions rather than animation-ready rotations.
Method
A DetNet/IKNet architecture jointly exploits 2D- and 3D-annotated images plus stand-alone MoCap data to predict 3D joints and directly regress joint rotations.
Results
The method shows significant quantitative and qualitative improvement over the state of the art in accuracy, robustness, and runtime across challenging benchmarks.
Takeaways & Limitations
Direct joint-rotation predictions make the output usable for animating hand meshes, while the architecture enables training with heterogeneous image and MoCap data.
Takeaways & Limitations
Pose-data augmentation assumes each finger moves independently and that quaternion interpolation from the rest pose yields valid poses.
Abstract
from arXiv · showhide
We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy. This is enabled by a new learning based architecture designed such that it can make use of all the sources of available hand training data: image data with either 2D or 3D annotations, as well as stand-alone 3D animations without corresponding image data. It features a 3D hand joint detection module and an inverse kinematics module which regresses not only 3D joint positions but also maps them to joint rotations in a single feed-forward pass. This output makes the method more directly usable for applications in computer vision and graphics compared to only regressing 3D joint positions. We demonstrate that our architectural design leads to a significant quantitative and qualitative improvement over the state of the art on several challenging benchmarks. Our model is publicly available for future research.
1. Introduction
The paper targets monocular hand capture that is real-time, uses one RGB camera, and predicts joint angles despite severe data and geometric challenges. It combines multiple training-data modalities with joint detection and inverse kinematics to improve capture performance.
- Motivation: Monocular hand capture is challenging because of depth ambiguity, self-occlusion, fast movements, and uniform skin appearance.The desired system should provide real-time feedback from a single RGB camera and predict joint angles for graphics, AR, and VR applications.
- Limitations of prior work: Existing methods cannot exploit all available training modalities because 3D annotations are difficult to collect, synthetic images have a domain gap, and MoCap data lacks paired images.Internet images often provide only 2D annotations, while MoCap datasets offer varied poses without corresponding images.
- Limitations of prior work: Positional predictions alone are insufficient for direct hand-mesh animation, while separate kinematic fitting requires hand-crafted energies and costly iterative optimization.The paper motivates direct joint-rotation prediction as a more usable representation for graphics applications.
- Proposed approach: The proposed architecture uses DetNet for 2D/3D joint detection and IKNet to map predicted joints directly to joint rotations.DetNet jointly uses 2D supervision from real images and 3D supervision from synthetic data, while the approach incorporates image and stand-alone MoCap data.
- Proposed approach: The inverse kinematics network maps 3D joint predictions to joint angles in a single feed-forward pass and supports joint positional and rotational supervision.This design is presented as a core architectural contribution for monocular hand shape and motion capture.
- Results: The method outperforms state-of-the-art methods quantitatively and qualitatively on challenging benchmarks while achieving unseen runtime performance.The supplied introduction reports the result without specifying a numerical runtime or accuracy value.
2. Related Work
Prior work increasingly uses deep learning for single-camera hand-pose estimation, but related approaches differ in input modality, generalization strategy, and whether they recover joint rotations.
- Depth-based methods: Depth-based methods estimate hand pose from depth images, often fitting generative models or using discriminative predictions for initialization and regularization.The related-work discussion describes both earlier model-fitting methods and more recent deep-learning approaches.
- Learning-based methods: A 2.5D heat-map formulation combines 2D joint locations with depth information, while depth datasets and rendered images are used to increase training diversity.CycleGAN post-processing was used in one rendered-data approach to bridge the domain gap.
- Joint-rotation recovery: Several related methods focus on joint positions and omit joint-rotation recovery, although rotation prediction is more suitable for hand-mesh animation.Some methods estimate rotations by fitting a generic model, rather than directly regressing them.
3. Method
The method combines DetNet for 2D/3D joint prediction, hand-model shape estimation, and IKNet for direct joint-rotation recovery. Its training design integrates diverse image annotations and standalone motion-capture data, while using forward-kinematic supervision to improve robustness to noisy predictions.
- DetNet: DetNet predicts 2D image-space and root-relative, scale-normalized 3D joint positions from a single RGB image.It uses a multi-task architecture with feature extraction, 2D detection, and 3D detection components.
- DetNet: The 3D detector estimates location maps and bone-orientation delta maps, using heat-map weighting and neighboring-joint relations for robust localization.Location and delta losses compare predictions with tiled ground-truth joint coordinates and bone directions, weighted by ground-truth heat maps.
- DetNet: 2D supervision trains the feature extractor and 2D detector on varied in-the-wild images, while mixed 2D- and 3D-annotated data supports the multi-task scheme.The method emphasizes 2D-labeled images because they provide greater visual variation than 3D-annotated datasets.
- Hand Model and Shape Estimation: MANO represents hand shape with β and pose with θ, where θ contains joint rotations in axis-angle form and β controls shape variation.The model deforms a mean template using shape and pose blendshapes before applying linear blend skinning.
- Hand Model and Shape Estimation: Shape estimation fits MANO parameters to predicted bone lengths, but scale-normalized inputs recover only relative shape, such as finger-to-palm ratios.A regularizer constrains the shape parameters during bone-length fitting.
- Inverse Kinematics Network IKNet: IKNet converts DetNet’s 3D joint predictions into joint rotations in one feed-forward pass, enabling direct mesh animation and learning pose priors from data.Its design incorporates MoCap supervision, avoids iterative model fitting, and can correct noisy DetNet predictions.
- Inverse Kinematics Network IKNet: IKNet is trained with paired MoCap positions and rotations, plus 3DPosData that supervises reconstructed positions through forward kinematics.The additional dataset exposes IKNet to the noise and errors present in DetNet predictions.
4. Results
The method is evaluated qualitatively and quantitatively across challenging images and four public datasets, with ablations examining its architecture, data, IKNet, supervision, and quaternion losses. Results show over-100fps runtime, strong generalization, and improvements from the proposed design and training data.
- Runtime and qualitative results: Over 100fps is achieved, with DetNet taking 8.9ms and IKNet 0.9ms per feed-forward pass on an NVIDIA GTX1080Ti.The reported runtime is measured for a single feed-forward pass.
- Runtime and qualitative results: The method handles motion blur, object and self-occlusion, complex poses, fast motion, and unconstrained viewpoints while producing joint rotations for direct hand-surface animation.Qualitative results also show plausible recovery of different hand shapes from a single image.
- Comparison to related work: Adding STB training raises its STB AUC to 0.991 but causes a huge performance drop on the other three benchmarks, so STB is excluded from final training.The authors attribute this trade-off to STB’s large frame count and limited variation, which can encourage overfitting.
- Comparison to related work: The system outperforms competing methods by a large margin on DO and ED, the fairest comparisons because none of the models are trained on those datasets.Table 1 compares state-of-the-art methods on four public datasets, with training-set use marked separately.
- Ablation study: Training DetNet on the same data as Zhang et al. improves results by around 5%, while adding IKNet and combining more data further improves performance.The ablation attributes cross-dataset IKNet gains to a MoCap-learned pose prior that corrects raw 3D joint predictions.
- Ablation study: Direct rotational supervision produces anatomically more correct poses, and 3DPosData helps IKNet handle noisy DetNet predictions; the combined quaternion losses give the highest average accuracy.Without rotational supervision, unnatural poses can occur even when 3D joint positions match the ground truth.
5. Conclusion
The paper combines image and MoCap data in a learning-based monocular hand estimator and directly regresses joint rotations. It reports improvements in accuracy, robustness, and runtime over the state of the art.
- The approach combines image data and MoCap data for monocular hand pose and shape estimation.
- A trained inverse kinematics network directly regresses joint rotations.
- The method improves accuracy, robustness, and runtime compared with the state of the art.
- The authors identify hand texture capture and two-hand capture from a single RGB image as future directions.