Source-linked AI summary
Dynamics Based 3D Skeletal Hand Tracking
Stan Melax, Leonid Keselman, Sterling Orsten
TL;DR
Full 3D skeletal hand tracking is difficult because existing approaches may require hardware, pose restrictions, or substantial computation. The paper augments rigid-body simulation with depth-derived constraints and multiple candidate simulations, achieving real-time tracking across depth cameras on one x86 CPU core. Its scope is bounded by temporal-coherence assumptions and a convex-bone model whose geometric fidelity remains limited.
Problem
Full skeletal hand tracking matters for natural interfaces but existing techniques may require wearable hardware, pose restrictions, or substantial computation.
Method
The paper augments rigid-body dynamics with depth-surface, joint, contact, temporal, and pose constraints, using capped impulses and multiple simulations to fit articulated hand models.
Results
The system provides markerless, fully articulated 3D hand tracking from depth data, runs comfortably on one CPU core, and supports different cameras.
Takeaways & Limitations
The approach offers a real-time tracking framework that can incorporate object information and improve fidelity with higher resolution, model accuracy, and camera frame rate.
Takeaways & Limitations
The method assumes a correct initial fit with continuous temporal updates and currently uses convex collision bones rather than a geometrically accurate soft hand model.
Abstract
from arXiv · showhide
Tracking the full skeletal pose of the hands and fingers is a challenging problem that has a plethora of applications for user interaction. Existing techniques either require wearable hardware, add restrictions to user pose, or require significant computation resources. This research explores a new approach to tracking hands, or any articulated model, by using an augmented rigid body simulation. This allows us to phrase 3D object tracking as a linear complementarity problem with a well-defined solution. Based on a depth sensor's samples, the system generates constraints that limit motion orthogonal to the rigid body model's surface. These constraints, along with prior motion, collision/contact constraints, and joint mechanics, are resolved with a projected Gauss-Seidel solver. Due to camera noise properties and attachment errors, the numerous surface constraints are impulse capped to avoid overpowering mechanical constraints. To improve tracking accuracy, multiple simulations are spawned at each frame and fed a variety of heuristics, constraints and poses. A 3D error metric selects the best-fit simulation, helping the system handle challenging hand motions. Such an approach enables real-time, robust, and accurate 3D skeletal tracking of a user's hand on a variety of depth cameras, while only utilizing a single x86 CPU core for processing.
1 INTRODUCTION
The paper targets richer hand-based interfaces by extending depth-sensor tracking from body limbs to complete 3D hand poses. It proposes integrating physical dynamics with sensor fitting and hand mechanics in one tracking framework.
- Depth sensors already support natural interaction through body motions and gestures, motivating extension to individual-finger tracking.
- Complete 3D hand poses could support richer interactions such as grasping, pointing, and subtle manipulation.
- Dynamics systems incorporate momentum, forces, contacts, joints, and external motion constraints, while remaining extensible to new data or requirements.
- The proposed approach fits hand point-cloud data while simultaneously enforcing nonpenetration, joint ranges, temporal coherence, and momentum through added physics constraints.
2 RELATED WORK
Prior hand-tracking methods use gloves, markers, multiple cameras, classifiers, inverse kinematics, or computationally intensive search. The paper instead emphasizes camera-independent physical tracking that preserves articulated motion and uses limited simulation trials.
- Earlier systems range from datagloves and colored gloves to wrist-mounted cameras, whereas the long-term goal is apparatus-free hand tracking.
- Gesture-recognition systems often identify blob features or fingertips, but finger identity can be difficult to distinguish before fitting a complete hand pose.
- The approach avoids specific camera placement, multiple viewpoints, and camera-specific training, while scaling with better resolution, depth accuracy, or additional cameras.
- Unlike methods that restrict poses, the paper combines joint range limits and collision detection to allow more subtle and complete hand motions.
- The system runs over 60 Hz on one x86 processor core and uses a few simulations rather than hundreds of rendered pose candidates.
3 TRACKING ALGORITHM
The tracking algorithm represents the hand with convex rigid bodies and formulates sensor fitting, joints, contacts, and surface attachments as unified physical constraints. It uses temporal priors, subsampled depth data, capped impulses, and articulated geometry to obtain stable fitting.
- 3 TRACKING ALGORITHM: The method updates a previous pose using temporally coherent video and uses a convex rigid-body prior for efficient numerical computation.
- 3 TRACKING ALGORITHM: Pose tracking is formulated as a linear complementarity problem that jointly handles joint, surface, and angular constraints with a real-time solver.
- 3.1 Explanation of Rigid Body Dynamics: Rigid-body simulation handles collisions and joint configurations through pairwise constraints on relative velocities.
- 3.4 Impulse Cap: Impulse caps limit depth-derived surface constraints so noisy or incorrectly attached points do not overpower uncapped joint and finger-motion constraints.
- 3.3 Point Cloud Samples as Constraints: Voxel-grid subsampling replaces points within each occupied voxel with their center-of-mass point to remove camera noise and produce a sparser dataset.
- 3.3 Point Cloud Samples as Constraints: Each depth sample becomes a one-dimensional constraint attaching the camera-facing model surface to the point through its closest feature.
- 3.6 The Hand Model: The hand is approximated by convex rigid bodies, with interpenetrating neighboring bone geometry and disabled neighboring collisions to avoid gaps and permit surface-constraint slipping.
- 3.3 Point Cloud Samples as Constraints: Higher-level pose information can constrain samples to known model regions rather than merely attaching them to the closest feature.
4 MULTIPLE SIMULATIONS AND ERROR
The system improves hand tracking by running multiple physics-based simulations with different motion and pose hypotheses, then selecting the candidate that best fits the depth data. This strategy addresses fast motion, grasping, local minima, uncertain feature detections, and occlusions through specialized simulations and a 3D error metric.
- Multiple simulations: Multiple simulations explore different motion assumptions and return the pose with the least 3D error against the depth data.Each simulation can incorporate different subsets of heuristics, constraints, and poses.
- 4.1 Gross Motion Bias: A frozen-hand simulation improves large-scale motion tracking when rapid movement causes fingers to shift substantially between frames.It temporarily disallows relative motion between bones, helping align the whole hand and update occluded parts coherently.
- 4.2 Grasping Bias: A grasping-biased simulation quickly aligns fingers during transitions from a closed fist to an open hand, although its output is typically rejected for pointing poses.The simulation adds orientation constraints that keep neighboring finger bones nearly parallel, within 10 degrees.
- 4.3 Searching Nearby States: A finger-flipping simulation tests nearby candidate states and adopts configurations with better depth-data fits when rapid finger retractions trap the current pose.It can successively correct different fingers, such as the ring and middle fingers, through prospective candidate poses.
- 4.4 Feature Detection: Because pose classifiers can produce false positives, labeled appendages are incorporated only when the resulting simulation achieves a better fit.This prevents uncertain high-level detections from automatically overriding the physics-based tracking result.
- 4.5 Error Metric: The 3D error metric sums per-body fitting errors and background-occlusion penalties to select a candidate without directly adjusting its pose.Its fitting term uses the furthest relevant point-cloud distance for each rigid body, while occluded centroids receive a constant penalty.
5 RESULTS AND DISCUSSION
The system tracks hands across several depth-camera setups, tolerating missing and noisy samples but remaining sensitive to motion speed, occlusion, and sensor limits.
- Camera evaluations: The system demonstrated hand tracking on multiple camera setups, including stereoscopic, structured-light, time-of-flight, and projected-texture cameras.The paper reports successes and challenges across different motions and camera configurations.
- Stereoscopic Camera: Reasonable tracking persisted despite missing pixels and noisy samples from the Bumblebee XB3, although only slow, careful motions were trackable in the best circumstances.The implementation was not sufficient for application use because of missing pixels, incorrect samples, and tracking-quality degradation.
- Subsampling: Higher-resolution voxel subsampling adjusts solver performance by changing the number of constraints while preserving tracking when voxels remain smaller than tracked features.Across the tested voxel-size range, adult hands produced acceptable results, and smaller voxels minimized sensor-noise contribution.
- Structured Light: Structured-light tracking tolerated several lost frames and worked for adult hands near the camera’s limits, but motion was not smooth and tracking sometimes required a recovery pose.The ASUS Xtion PRO has a minimum working distance of around 0.6m and often misses small features such as camera-facing fingertips or child-sized fingers.
- Challenging motions: The system’s surface constraints can pull an initially occluded digit toward the correct configuration during a knife-hand gesture.Figure 11 illustrates recovery of the correct digit as fingers bend from an initially occluded state.
- Time-of-flight Camera: The time-of-flight camera supported short-range tracking with continuous fingertip updates, but very fast motion and rolling a clenched fist could still cause failures.The Creative camera begins at an effective range of 20cm; smoothing and filtering improved reliability for noisy data.
6 APPLICATION
The tracked skeletal hand supports physically enabled interaction with virtual objects, while application-specific hand and object geometry improves manipulation reliability.
- Interaction model: Skeletal hand pose can animate a 3D hand model whose bones use collision detection to interact with virtual objects.The paper describes grasping and object interaction as direct applications of tracked pose information.
- Grasping: A virtual fist can wrap around an object in front of the palm and apply forces that maintain the grasp.This behavior is described for a 3D hand model without real-world or haptic feedback.
- Hand geometry: Application-specific virtual hand geometry improves grasping, with deeper or concave palms easier to use than the tracking model’s enlarged convex palm.The recommendation follows preliminary testing of rigid-body manipulation.
- Object geometry: Objects designed to interact with the virtual hand, such as capped barbells, are less likely to slip from the grasp than convex polyhedra.Larger geometry at both ends of a barbell helps prevent slipping.
- Object mechanics: Partially constrained objects such as throw switches, levers, and dials are manipulated more successfully than fully free convex bodies with 6 degrees of freedom.The paper identifies these examples as interacting reliably.
- Tracking boundary: Tracking is lost when the hand leaves the depth sensor’s field of view, though translating or rotating the application view can keep the hand within usable space.The workaround also lets the user navigate the scene.
7 CONCLUSION
The paper presents markerless, fully articulated 3D hand tracking from depth data, with fidelity improving alongside sensor, model, and computational resources.
- Conclusion: The system fully tracks an articulated human hand from markerless depth-sensor data on a single CPU core.The conclusion states that additional resources could support more pose simulations and improve robustness further.
- Conclusion: Tracking fidelity improves with increasing resolution, better model accuracy, and faster camera frame rate.The conclusion identifies these factors as direct avenues for improving fidelity.
8 FUTURE WORK
The paper identifies several directions for improving the tracking system, including richer hand geometry, additional sensing cues, broader model support, and new interaction applications.
- The current convex-bone hand model could be replaced or enhanced with softbody support for more geometrically accurate tracking.The authors anticipate greater accuracy as depth-sensor resolution and accuracy improve.
- RGB-based natural markers or color features could provide additional tracking information beyond the depth data currently explored.Early fingertip experiments using primary colors produced flawless tracking, but RGB assistance was otherwise not investigated.
- Multiple sensors could improve coverage if infrared interference does not prevent their use.
- Extending the system from a single hand to models such as a full-body skeleton would require further work on performance and robustness.A multiresolution approach may help combine large torso sections with small finger bones.
- A real-time 3D hand tracker could theoretically replace existing 2D multitouch interfaces except for haptic feedback, supporting applications such as world-in-miniature interactions and virtual instruments.The paper notes that these application examples were not explored in the work.