Source-linked AI summary
Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A. Otaduy, Dan Casas, Christian Theobalt
TL;DR
The paper addresses real-time reconstruction of two strongly interacting hands using a single depth camera while also estimating hand shape. It combines dense neural correspondences, parametric hand models, simulated training data, and GPU-optimized energy minimization, achieving real-time tracking with complex interactions and results quantitatively close to a prior solution while being two orders of magnitude faster.
Problem
Existing hand tracking methods do not jointly provide practical single-camera, real-time reconstruction of complex two-hand interactions with collision handling and user-specific hand-shape estimation.
Method
The method combines a dense correspondence network, independent parametric hand models, physically simulated two-hand training data, and GPU-based Gauss-Newton energy minimization.
Results
The approach achieves qualitatively similar and quantitatively close results to Tzionas et al. [2016] while running two orders of magnitude faster and handling more complex interactions qualitatively.
Takeaways & Limitations
The system combines marker-less tracking, a single depth camera, real-time operation, collision handling, and adaptation to user-specific hand shapes.
Takeaways & Limitations
The setup currently depends on two high-end GPUs, and very challenging situations such as extremely fast hand motions may still cause tracking difficulties.
Abstract
from arXiv · showhide
We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. In order to achieve this, we embed a recent parametric hand pose and shape model and a dense correspondence predictor based on a deep neural network into a suitable energy minimization framework. For training the correspondence prediction network, we synthesize a two-hand dataset based on physical simulations that includes both hand pose and shape annotations while at the same time avoiding inter-hand penetrations. To achieve real-time rates, we phrase the model fitting in terms of a nonlinear least-squares problem so that the energy can be optimized based on a highly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work, including tight two-hand grasps, significant inter-hand occlusions, and gesture interaction.
1 INTRODUCTION
The paper targets marker-less tracking of two interacting hands under practical constraints such as real-time operation, simple hardware, and person-specific shape adaptation. It combines a learned correspondence predictor, a parametric hand model, and physically simulated training data to address complex interactions.
- The method targets real-time, marker-less reconstruction of two interacting hands from a single depth camera while estimating user-specific hand shape.
- A learned dense surface correspondence predictor is combined with a parametric hand model in an energy minimization framework for estimating both hands’ pose and shape.
- The paper frames its contribution as combining several desirable properties that prior methods did not jointly provide.
- Physical simulation produces a two-hand training dataset with pose and dense shape annotations while avoiding inter-hand penetrations.
- The approach is designed for involved hand-hand interactions and is reported as more robust and reliable than existing methods in such settings.
2 RELATED WORK
Prior work addresses hand tracking with different sensing setups and algorithmic strategies, but jointly tracking interacting hands remains difficult because of segmentation and occlusion. The paper positions its method as combining practical single-camera tracking with collision handling and hand-shape adaptation.
- Capturing a Single Hand: Single-camera hand tracking methods include generative, discriminative, and hybrid algorithms, while multi-camera setups offer tracking advantages but are cumbersome for many applications.
- Hand Shape Models: Hand-shape models range from geometric primitives to meshes, but relatively few methods automatically estimate detailed hand shape from depth images.
- Capturing Two Hands: Two-hand reconstruction introduces segmentation and severe occlusion challenges beyond single-hand tracking.
- Capturing Two Hands: Earlier single-camera two-hand methods operated at 0.2 to 4 frames per second and therefore did not achieve real-time rates.
- Capturing Two Hands: The proposed approach combines real-time operation, marker-less tracking, a single depth camera, collision handling, and automatic adaptation to user hand shape.
3 OVERVIEW
The pipeline predicts dense correspondences and left/right segmentation from a depth image, then fits two parametric hand models through regularized energy minimization. Collision proxies and GPU-based Gauss-Newton optimization support physically plausible and real-time tracking.
- Pipeline: A neural network regresses dense model-to-image correspondences and left/right segmentation from depth images depicting possibly interacting hands.
- Pipeline: The correspondence maps initialize an energy minimization that fits a parametric model using pose, shape, temporal smoothness, and collision regularizers.
- Pipeline: The optimization is formulated as nonlinear least squares and implemented with a data-parallel GPU Gauss-Newton optimizer for real-time performance.
- Hand Model: MANO represents hand geometry with a low-dimensional parametric surface model capturing pose and shape variation from 778 mesh vertices, 10 shape parameters, and 51 pose parameters.
- Hand Model: Independent MANO models represent the left and right hands, whose combined mesh contains 2·778 vertices.
- Hand Model: Collision proxies approximate each hand with 35 3D Gaussians, enabling differentiable overlap penalties during pose optimization.
4 DENSE CORRESPONDENCE REGRESSION
The method learns dense pixel-to-color correspondences for two interacting hands and converts them into vertex-to-pixel mappings. Training combines physically simulated interaction data with real segmentation annotations in a stacked encoder-decoder network.
- Dense Correspondence Encoding: The correspondence map assigns each hand-model vertex to a pixel or marks it invisible in the depth image.The learned pixel-to-color function represents surface correspondence and left/right/non-hand segmentation across the full image.
- Dense Correspondence Encoding: The four-channel pixel encoding uses three surface-correspondence channels and one segmentation channel distinguishing left hand, right hand, and non-hand pixels.The segmentation values are 0 for the left hand, 0.5 for the right hand, and 1 for non-hand regions.
- Dense Correspondence Encoding: Vertex-to-pixel mappings are recovered by comparing predicted per-pixel colors with fixed per-vertex colors using thresholded nearest-neighbor matching.A closest predicted color with distance above η=0.04 indicates that the vertex is likely invisible.
- Data Generation: The synthetic dataset drives a physically based two-hand simulation with motion capture and outputs depth images plus dense surface annotations.The simulation extends prior work to simultaneous two-hand simulation and inter-hand collision detection while using a safety distance during motion capture.
- Data Generation: Training mixes synthetic depth data with real footage annotated for left/right hand segmentation to improve generalization beyond synthetic data.The synthetic set contains 80,000 images from 5 users, while the real set contains 19,926 images from 3 users.
- Network Architecture: CoRN uses two stacked encoder-decoder blocks, with the first producing three-class segmentation maps before subsequent correspondence prediction.The network is trained on the mixed real and synthetic dataset to learn the pixel-to-color mapping N.
5 POSE AND SHAPE ESTIMATION
Pose and shape estimation fits a parametric two-hand model to foreground depth points and normals by minimizing a nonlinear least-squares energy. The objective combines geometric data terms with regularization for plausible, smooth, and collision-free reconstructions.
- Pose and Shape Fitting: The method extracts foreground depth points and normals, then jointly optimizes hand shape and pose parameters to explain the point-cloud data.Foreground extraction assumes the hands and arms are closest to the camera and uses depth-based thresholding.
- Data Term: The data energy combines point-to-point distances for visible model vertices with point-to-plane deviations from the point-cloud surface tangent.These terms evaluate how well the hand model explains the observed depth image.
- Regularizer: The regularizer combines statistical pose and shape priors, temporal smoothness, and collision penalties for individual hands and hand-hand interactions.The collision term uses Gaussian proxies whose means and standard deviations depend on pose and shape.
- Optimization: The energy is formulated as nonlinear least squares and optimized with Gauss-Newton using analytically computed residual derivatives and GPU Jacobians.This formulation supports efficient computation of all Jacobian entries on the GPU.
- Shape Estimation: Shape optimization is performed in every frame so that a poorly chosen shape-estimation frame does not affect subsequent tracking.The authors report that hand shape remains robust and does not significantly change within a sequence.
6 EVALUATION
The evaluation examines implementation choices, ablations, comparisons with prior systems, and adaptation to user-specific hand shapes. The complete system supports complex interactions, recovers from tracking failures, and achieves real-time performance while maintaining competitive tracking quality.
- Implementation: The implementation achieves 30 fps using two GPUs, with CoRN and correspondence matching on one GPU and model optimization on the other.Experiments use an Intel RealSense SR300 depth camera.
- Ablation Study: The proposed CoRN training setting outperforms all other evaluated correspondence-regressor configurations.The study compares frontal and mixed-viewpoint data, synthetic-only training, and naïve color encoding.
- Ablation Study: The complete energy performs best, while removing pose or shape regularization worsens PCK more than removing collision or smoothness terms.The collision term has limited PCK impact but its removal severely worsens qualitative results, whereas smoothness reduces temporal jitter.
- Ablation Study: Naïve closest-point correspondences produce significantly higher errors than predicted CoRN correspondences, with or without segmentation information.Qualitative examples further support the importance of the complete energy formulation for hand-hand interaction tracking.
- More Results: The tracker recovers after severe errors caused by extremely fast motion, and it adjusts reconstructed hand geometry to different users’ hand shapes.Across four users, the estimated bone lengths have an average standard deviation of 0.6 mm, although direct quantitative shape evaluation is unavailable.
- Comparison to the State of the Art: Compared with Tzionas et al., the method has slightly higher relative 2D pixel error but achieves a 150× speed-up without requiring a user-specific hand model.The reported precision difference does not produce a noticeable visual-quality gap, and the method also handles complex interactions that challenge Leap Motion.
7 LIMITATIONS AND DISCUSSION
The method achieves compelling real-time two-hand pose and shape estimation but remains computationally demanding and can struggle under severe motion blur or occlusion. Its reconstructed hand shapes are also less detailed than high-quality laser scans.
- Computational cost: The setup currently requires two high-end GPUs, one for the regression network and one for the optimizer.Lighter neural architectures are suggested to reduce computational cost.
- Challenging conditions: Extremely fast hand motions can produce severely blurred depth images that challenge tracking.
- Challenging conditions: Mostly occluded hands can cause temporal jitter because the depth image contains insufficient information.The paper suggests stronger occlusion-aware smoothing or a temporal correspondence-prediction network as possible mitigations.
- Temporal behavior: The current temporal smoothness prior may delay tracking when inter-frame motion is large.
- Shape detail: Estimated hand shapes are less detailed than high-quality laser scans because MANO is coarse and depth images generally have lower resolution.The paper attributes this to MANO's 778 vertices per hand and lower-resolution depth imagery.
8 CONCLUSION
The paper concludes with a real-time, markerless system for reconstructing two interacting hands from one depth camera while accounting for collisions and user-specific shape. It combines learned correspondences, physics-based synthetic data, and GPU Gauss–Newton optimization, achieving results close to Tzionas et al. while running two orders of magnitude faster.
- Conclusion: The method combines markerless tracking, a single depth camera, collision handling, real-time operation, and user-specific hand-shape adaptation.
- Conclusion: Physics-based simulation generates annotated synthetic data with physically plausible interactions between two hands.
- Conclusion: Results are qualitatively similar and quantitatively close to Tzionas et al. [2016] while running two orders of magnitude faster.
- Conclusion: The method qualitatively handles more complex hand–hand interactions than recent state-of-the-art hand trackers.
A NEURAL NETWORK TRAINING DETAILS
The networks were trained with Adam on mixed synthetic and real data for 450,000 iterations. Depth preprocessing and on-the-fly geometric augmentations diversify the training inputs.
- Training setup: Training used TensorFlow and Adam with default settings for 450,000 iterations and batch size 8.Synthetic and real images were sampled with equal probability; training took approximately 25 hours on an Nvidia Tesla V100 GPU.
- Preprocessing: Depth values were converted to meters and centered by subtracting the mean of valid depth pixels.
- Augmentation: On-the-fly augmentation applied rotations from −90 to 90 degrees, image-plane translations up to 0.25 image size, and scale changes from 1.0 to 2.0.The sampled transformations vary across epochs, effectively increasing training-set diversity.
B GPU IMPLEMENTATION DETAILS
GPU CUDA kernels compute the Gauss–Newton residuals and Jacobian products, while the reduced linear system is solved on the CPU.
- GPU computation: CUDA kernels compute the nonconstant Jacobian entries J∈R8871×122 and residuals f∈R8871 for Gauss–Newton optimization.
- Linear-system solve: Shared memory is used for the matrix products JᵀJ and Jᵀf before transferring the reduced system to the CPU.The transferred quantities are JᵀJ∈R122×122 and Jᵀf∈R122.
C COLLISION ENERGY
The collision term uses differentiable 3D Gaussian proxies coupled to hand pose and shape, while avoiding direct shape derivatives that would shrink interacting hand models.
- 3D Gaussian collision proxies are coupled to the hand model, with means depending on pose and shape and standard deviations depending only on shape.The parameters are β and θ for pose and shape dependence of the mean, while σ depends only on β.
- The Gaussian-overlap collision term is differentiable with respect to the proxy means and standard deviations.
- The optimization excludes direct derivatives of the mean with respect to shape because they encourage shrinking the hand models during interaction.
- Shape is optimized through the other energy terms, while Gaussian parameters are updated from the shape estimate at every optimizer iteration.