Source-linked AI summary
Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, Christian Theobalt
TL;DR
Joint hand-object tracking is difficult because occlusions, fast motion, and similar hand appearance complicate single-camera estimation. The paper combines discriminative RGB-D part classification with 3D articulated Gaussian mixture alignment and contact/occlusion regularization. It reports accurate and robust real-time tracking from one commodity depth sensor, alongside a new annotated benchmark dataset.
Problem
Real-time simultaneous tracking of a hand and manipulated object remains challenging because of high dimensionality, occlusions, and hand-object ambiguity.
Method
The method combines multilayer RGB-D hand-part and object classification with 3D articulated Gaussian mixture alignment and contact and occlusion objectives.
Results
The approach provides accurate and robust real-time simultaneous hand-object tracking using a single commodity depth sensor and is evaluated on public and newly annotated data.
Takeaways & Limitations
The paper demonstrates that challenging hand-object interactions can be tracked in real time with a minimal single-sensor setup.
Takeaways & Limitations
Tracking robustness is constrained by very fast motions and by the assumption that occluded hand regions move consistently with the rest of the hand.
Abstract
from arXiv · showhide
Real-time simultaneous tracking of hands manipulating and interacting with external objects has many potential applications in augmented reality, tangible computing, and wearable computing. However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately. Many previous approaches resort to complex multi-camera setups to remedy the occlusion problem and often employ expensive segmentation and optimization steps which makes real-time tracking impossible. In this paper, we propose a real-time solution that uses a single commodity RGB-D camera. The core of our approach is a 3D articulated Gaussian mixture alignment strategy tailored to hand-object tracking that allows fast pose optimization. The alignment energy uses novel regularizers to address occlusions and hand-object contacts. For added robustness, we guide the optimization with discriminative part classification of the hand and segmentation of the object. We conducted extensive experiments on several existing datasets and introduce a new annotated hand-object dataset. Quantitative and qualitative results show the key advantages of our method: speed, accuracy, and robustness.
1 Introduction
The paper addresses real-time simultaneous tracking of a hand and manipulated object from a single commodity RGB-D camera, targeting challenges from occlusion, high dimensionality, and hand-object ambiguity.
- Real-time hand-object tracking remains unsolved because interactions increase dimensionality, occlusions, and hand-object ambiguity.
- The method uses a single commodity RGB-D camera to track hand articulation and the manipulated object's rigid-body motion.
- 3D articulated Gaussian mixture alignment provides the core approach for jointly tracking hand and object accurately.
- Novel contact-point and occlusion objectives address difficult hand-object interactions.
- A multilayer classification architecture segments hand and object regions and classifies hand parts in RGB-D sequences.
- The evaluation covers public datasets and a new fully annotated dataset containing diverse hand-object interactions.
2 Related Work
Prior work tracks hands or objects separately, or handles interactions with restrictive hardware and runtimes; this paper positions 3D articulated Gaussian mixture alignment as a unified alternative.
- Single-hand methods use discriminative retrieval or generative alignment, but their accuracy, temporal stability, or occlusion handling remains limited.
- Existing methods do not simultaneously track interacting hands and objects in non-trivial poses at real-time rates.
- Combining separate hand and object trackers is unsatisfactory because only a joint formulation can exploit their mutual constraints.
- Prior interaction trackers require stereo, multiple cameras, slow runtimes, or support only simple and lightly occluded interactions.
- Unlike rigid Gaussian mixture registration, the proposed representation extends Gaussian mixture alignment to articulated hand-object motion in 3D.
- The classification pipeline combines viewpoint selection, color-based object segmentation, and two-layer hand-part classification.
3 Discriminative Hand Part Classification
The preprocessing pipeline classifies RGB-D input into object, hand, and hand-part labels, using viewpoint selection, color segmentation, and a two-layer random forest.
- Input color and depth frames are first segmented by color to remove the object from the depth map.
- The final per-pixel output is a part-probability histogram, while object pixels receive object probability 1.
- Training uses real hand motions retargeted to a virtual model, multiple viewpoints, and inserted virtual objects to simulate occlusions.
- The system selects a viewpoint-specific forest using the hand orientation computed in the previous frame.
- A two-layer random forest first classifies hand, arm, and background pixels, then subdivides hand pixels into distinct parts.
- The forests use depth-difference features and are trained from diverse hand-object poses, including N = 100k hand-arm images and N = 38k images per viewpoint.
4 Gaussian Mixture Model Representation
The method represents both the observed depth and the hand-object scene as 3D Gaussian mixture models, enabling fast analytical pose optimization without costly correspondence search.
- Gaussian mixture alignment is chosen for accurate, outlier-robust tracking and fast pose optimization.
- The 3D formulation extends rigid point-set Gaussian mixture alignment to articulated hand-object motion.
5 Unified Density Representation
The method represents both RGB-D input and the hand-object scene as 3D Gaussian mixture probability densities, enabling unified and fast pose optimization.
- The hand uses a 26-DOF kinematic skeleton, while the rigid manipulated object uses 6 DOFs.Twenty joint angles describe non-rigid hand motion, and six DOFs describe the hand’s global rigid transform.
- Expressing depth and scene as GMMs makes the model density depend analytically on pose parameters and supports fast pose optimization.For the hand, each Gaussian is rigidly rigged to one bone.
- The hand and object are modeled as separate probability density distributions using attached or fitted Gaussian functions.Thirty Gaussians are manually attached to the hand’s kinematic chain, while a predefined number is fitted to the object’s spatial extent.
- Input depth is converted into a density representation by hierarchical quadtree clustering of adjacent pixels with similar depth.Clustering reaches depth four, covers at most 64 pixels, and stops when depth variance is below 30 mm.
- Each depth-cluster Gaussian stores a 3D center, probability, and best associated semantic label derived from per-pixel histograms.The center is obtained by backprojecting the quad’s image center using its average depth.
6 Multiple Proposal Optimization
The tracker generates separate alignment- and label-driven pose proposals, optimizes each efficiently, and selects the proposal that best explains the input while remaining anatomically plausible.
- Two proposals are computed by minimizing distinct hand-object tracking energies: one using depth alignment and one using discriminative classification.
- The separate optimization avoids failures from bad classification and supports fast pose recovery.The method uses analytical gradient descent with 10 iterations per proposal and adaptive step length.
- Initialization uses the previous frame’s solution, and the label proposal is slightly favored with λ = 1.003.
- The evaluation energy combines spatial alignment with anatomical joint-angle limits to select the proposal that best explains the input.
7 Hand-Object Tracking Objectives
The tracking objectives combine 3D spatial and semantic alignment with anatomical, temporal, contact, and occlusion regularization for hand-object tracking.
- The proposed objective operates in 3D, incorporates classification results, and adds contact-point and occlusion regularizers for real-time hand-object tracking.The formulation is designed for fast analytic pose optimization on the CPU.
- The combined energy includes spatial and semantic alignment, anatomical plausibility, temporal smoothness, contact points, and object-hand occlusions.The fixed weights are empirically determined and remain unchanged across experiments.
- Spatial Alignment: The alignment energy measures correspondence between input and scene density distributions throughout 3D space.The 3D formulation is reported to produce higher accuracy than a 2.5D formulation.
- Semantic Alignment: Semantic alignment uses classification labels, class probabilities, and a 30 cm cutoff to attract compatible model and image Gaussians.There are eight labels: six hand parts, object, and background.
- Contact Points: Contact constraints couple fingertip and object Gaussians at detected touch distances and remain active until a release threshold is exceeded.A new constraint is added when fingertip-object distance falls below the sum of their standard deviations, typically with δR > td.
- Occlusion Handling: The occlusion prior stabilizes hidden hand regions by constraining their motion according to the DOFs influenced by each occluded Gaussian.It assumes that occluded regions move consistently with the rest of the hand and excludes global rotation and translation.
8 Experiments and Results
Experiments across public datasets and a new benchmark evaluate accuracy, robustness, runtime, and component contributions for real-time hand-object tracking. The method tracks under occlusion and varied object configurations while operating at 30 Hz, though prolonged severe occlusion and very fast motion remain challenging.
- Experimental setup: The evaluation spans more than 15 sequences across three public datasets recorded with three RGB-D cameras, plus a new annotated hand-object dataset.The system uses commodity sensors including Creative Senz3D, Intel RealSense F200, and Primesense Carmine, with capture at 30 Hz.
- Accuracy and speed: 8.6px fingertip pixel error is comparable to the slower method of, while this approach runs over 60 times faster.The reported difference is only 2px and is described as within manual-annotation and sensor-noise uncertainty.
- Experimental setup: The new benchmark contains 6 sequences and 3014 annotated frames with ground truth for five fingertip positions and three cuboid corners.It supports evaluation of both fingertip positions and object pose, which existing datasets lacked.
- Accuracy and speed: 17.2 mm average fingertip error outperforms 's 19.6 mm on the Dexter dataset, with a maximum improvement of 5 mm on two sequences.The ablation attributes this improvement to continuous articulated 3D Gaussian mixture alignment using 3D spatial proximity.
- Robustness and ablation: Removing viewpoint selection, semantic alignment, occlusion handling, or contact terms worsens average error, while the hand-object terms improve robustness and recovery from failures.The method also tracks under heavy occlusion where is reported to fail, and handles varied object sizes, colors, shapes, and hand dimensions.
- Accuracy and speed: 4 ms preprocessing, 4 ms part classification, 2 ms depth clustering, and 20–30 ms pose optimization support real-time operation.The timing was measured on an Intel Xeon E5-1620 CPU with an NVIDIA GTX 980 Ti, using two proposals for pose optimization.
- Limitations: Situations with a high fraction of the hand occluded for a long period remain challenging, especially when occluded parts do not move rigidly.The classification strategy can enable recovery once previously occluded regions become visible again; very fast motions and increasing object complexity also affect robustness or runtime.
9 Conclusion
The paper presents a real-time, accurate, and robust approach for simultaneous hand-object tracking from a single commodity depth sensor, combining classification, pose optimization, and interaction constraints.
- The approach combines discriminative classification with generative pose optimization for simultaneous hand-object tracking.
- 3D articulated Gaussian mixture alignment, tailored interaction constraints, and multiple proposals support tracking during challenging hand-object interactions.
- The method is presented as the first real-time approach using a single commodity depth sensor.
- The authors report accurate and robust qualitative and quantitative performance and publicly provide a benchmark dataset with hand and object annotations.