PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Hand Pose Tracking and Estimation Using Stereo Matching

Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, Qingxiong Yang

arXiv:1610.07214v1cs.CV

TL;DR

The paper addresses whether passive stereo can support 3D hand-pose tracking without the interference and power limitations of active depth sensors. It introduces a benchmark and a hand-specific segmentation approach, finding comparable performance to active depth cameras under challenging scenarios when segmentation is effective.

  • Problem

    Active depth sensors used by existing hand-pose systems are vulnerable to interference and relatively high power consumption, while passive stereo is noisy and unstable in textureless scenes.

  • Method

    The paper builds a stereo-and-depth benchmark with manually labeled 3D joints and proposes stereo matching plus hand segmentation that combines skin-color modeling with passive stereo.

  • Results

    Passive stereo produces hand-pose tracking and estimation performance comparable to active depth cameras across challenging scenarios when hand segmentation is correct or provided by the proposed method.

  • Takeaways & Limitations

    Passive stereo is suitable for 3D hand-pose tracking and estimation under the evaluated challenging indoor scenarios.

Abstract

from arXiv · show

3D hand pose tracking/estimation will be very important in the next generation of human-computer interaction. Most of the currently available algorithms rely on low-cost active depth sensors. However, these sensors can be easily interfered by other active sources and require relatively high power consumption. As a result, they are currently not suitable for outdoor environments and mobile devices. This paper aims at tracking/estimating hand poses using passive stereo which avoids these limitations. A benchmark with 18,000 stereo image pairs and 18,000 depth images captured from different scenarios and the ground-truth 3D positions of palm and finger joints (obtained from the manual label) is thus proposed. This paper demonstrates that the performance of the state-of-the art tracking/estimation algorithms can be maintained with most stereo matching algorithms on the proposed benchmark, as long as the hand segmentation is correct. As a result, a novel stereo-based hand segmentation algorithm specially designed for hand tracking/estimation is proposed. The quantitative evaluation demonstrates that the proposed algorithm is suitable for the state-of-the-art hand pose tracking/estimation algorithms and the tracking quality is comparable to the use of active depth sensors under different challenging scenarios.

I. INTRODUCTION

Passive stereo could avoid active depth sensors’ interference and power limitations for mobile and outdoor hand-pose tracking, but its noisy, unstable depth estimates make segmentation difficult. The paper addresses this gap with a benchmark, evaluation, and stereo-based hand segmentation method.

  • Motivation: Active depth sensors support hand-pose tracking but can be interfered with by other active sources and consume relatively high power.These limitations make them unsuitable for mobile devices and outdoor environments.
  • Motivation: Passive stereo avoids active sensing limitations but produces noisy, unstable depth estimates, especially in textureless scenes.Textureless indoor regions are identified as a major obstacle to adopting passive stereo for 3D hand-pose tracking and estimation.
  • Paper scope: Existing hand-pose datasets use active depth sensors, motivating a benchmark that evaluates passive stereo with stereo and depth data plus manually labeled 3D joints.The benchmark includes palm and finger-joint positions and varied backgrounds designed to test segmentation and disparity-estimation difficulty.
  • Paper scope: The paper evaluates passive stereo with state-of-the-art model-based tracking and discriminative pose-estimation methods.The tracking evaluation integrates two state-of-the-art algorithms, while CHPR is adopted for discriminative hand-pose estimation.
  • Contribution: The main technical contribution is a stereo-based hand segmentation method that combines a skin-color model with passive stereo information.Unlike conventional sensor fusion, it is designed to extract a good hand segmentation for 3D hand tracking.
  • Contribution: The paper contributes a benchmark, evidence about passive stereo performance, evidence of algorithm robustness, and a hand-tracking-specific segmentation method.The benchmark contains 18,000 stereo image pairs, 18,000 depth images, and ground-truth 3D palm and finger-joint positions.

II. STEREO-BASED HAND TRACKING BENCHMARK

The benchmark captures passive-stereo hand-pose challenges across varied backgrounds and pose dynamics. It pairs stereo and active-depth imagery with manually labeled joint positions for quantitative evaluation.

  • Backgrounds: The benchmark uses six backgrounds representing different real-world difficulties for hand segmentation and disparity estimation.The backgrounds vary in texture, motion, distance, highlights, and shadows.
  • Backgrounds: Dynamic backgrounds include a PowerPoint presentation, video playback, and people walking.These sequences complement static backgrounds and test tracking under scene motion.
  • Backgrounds: Each background includes a training sequence for online skin-color model training.This adapts the skin-color model to the different benchmark environments.
  • Pose sequences: Two pose types are captured for every background: simple counting poses and difficult random poses with severe self-occlusion and global rotation.The difficult sequences use randomly moving fingers, while the simple sequences use slowly moving fingers.
  • Data collection: Stereo and active-depth images are captured simultaneously, with camera calibration and manually labeled finger-joint and palm-center positions.Each sequence contains 1,500 frames, supporting long-term tracking evaluation.

III. TRACKING WITH MANUAL HAND SEGMENTATION

With manually segmented hands, most tested stereo pipelines preserve hand-pose tracking performance comparable to active depth. Matching-cost choice matters more than most aggregation or optimization choices.

  • Overall result: Correct hand segmentation allows state-of-the-art hand-pose tracking performance to be maintained across most tested stereo matching algorithms.The experiment guarantees correct segmentation through automatic initialization followed by manual refinement.
  • Stereo evaluation: The evaluation tests 32 stereo methods formed by combining four matching costs with local aggregation or global disparity optimization.The methods include six local filters and two global methods, BP and DP.
  • Matching costs: Census transform achieves the best average performance among the tested matching costs, while color difference performs worst.Color difference is less robust to inconsistent responses from the two sensors; Census transform is invariant to illumination changes.
  • Stereo evaluation: Most cost-aggregation and disparity-optimization methods show similar tracking performance when manually segmented hand masks are used.The hand-region reconstruction accuracy of most stereo algorithms is comparable to the active depth camera under correct segmentation.
  • Tracking algorithms: ICPPSO generally outperforms PSO, but ICPPSO performs worse with DP because ICP is not robust to depth noise.The comparison reflects the complementary convergence properties of PSO and ICP and the sensitivity of ICP to noisy depth.
  • Overall result: Passive stereo achieves tracking performance comparable to the active depth sensor when the hand is segmented correctly.This conclusion is drawn from the dark curves representing active depth in the tracking evaluation.

IV. STEREO-BASED HAND POSE TRACKING AND ESTIMATION

The proposed framework combines online skin modeling, passive stereo depth estimation, and hand-region segmentation for 3D hand pose tracking and estimation.

  • IV. STEREO-BASED HAND POSE TRACKING AND ESTIMATION: The framework captures a training sequence for online adaptive skin-color modeling before tracking begins.It then estimates depth from passive stereo and segments the hand region using the depth estimates and trained skin model.
  • IV. STEREO-BASED HAND POSE TRACKING AND ESTIMATION: A novel constrained stereo matching algorithm is introduced specifically for 3D hand pose tracking and estimation.

A. Training based hand modeling

The hand model is trained online from a short sequence so skin-color detection adapts to the current environment rather than relying on a universal color model.

  • A. Training based hand modeling: Color-based segmentation is difficult because background colors may resemble skin and skin appearance varies with lighting.The paper therefore states that a generic model cannot suit all scenarios.
  • A. Training based hand modeling: An online training sequence uses adaptive GMM foreground modeling to isolate the assumed hand before estimating its skin-color distribution.The hand must wave for a few seconds during training to be detected as the foreground.
  • A. Training based hand modeling: The method computes hand and image color histograms and applies a likelihood-ratio threshold to obtain an initial hand segmentation.The resulting skin probability is defined for each color candidate c.
  • A. Training based hand modeling: The proposed model better separates skin from other objects than the generic web-trained model, especially against red backgrounds.The generic model treats more objects and colors as skin, producing greater ambiguity.

B. Constrained stereo matching

The constrained stereo matcher uses skin guidance and confidence-based handling of unreliable pixels to produce depth maps suited to hand pose tracking.

  • B. Constrained stereo matching: Traditional stereo performs well on textured scenes but can degrade sharply in real indoor environments containing large textureless regions.
  • B. Constrained stereo matching: The proposed matcher detects occluded pixels with left-right consistency and unstable pixels using matching-cost confidence.It assumes the hand skin is visible and targets accurate hand depth plus sufficient hand-background separation.
  • B. Constrained stereo matching: Pixels with confidence below 0.04 are declared unstable, and the new matching cost excludes occlusion contributions.For an occluded pixel the new cost is zero; otherwise it retains the original cost M_p(d).
  • B. Constrained stereo matching: Skin probability guides cost aggregation, stabilizing non-skin regions while preserving hand-region depth accuracy.The aggregated cost is filtered with guided image filtering, producing a final depth/disparity map with reduced background noise.
  • B. Constrained stereo matching: The method is specialized for 3D hand tracking and may not be suitable for other 3D applications.
  • B. Constrained stereo matching: Meshstereo produces clean backgrounds but is very slow and has obviously lower performance around the hand than the proposed method.

C. Hand segmentation

Hand segmentation combines skin probability with temporal depth proximity to estimate which pixels belong to the hand.

  • C. Hand segmentation: A pixel is considered part of the hand when its skin probability is high and its depth is close to the previous frame’s hand depth.
  • C. Hand segmentation: The hand probability multiplies skin probability by a Gaussian depth model whose mean is the previous frame’s average hand depth.The standard deviation is fixed at 150mm in all experiments.
  • C. Hand segmentation: Pixels with hand probability greater than 0.1 are classified as inside the hand region.
  • C. Hand segmentation: The constrained matcher yields globally smooth background disparities in low-texture scenes, reducing disparity noise that harms segmentation.Traditional stereo is sufficiently accurate on highly textured backgrounds but produces many background noises in low-texture B6.

V. EXPERIMENTS

The experiments compare passive stereo variants and active depth sensing across tracking and estimation tasks, backgrounds, and pose types. Correct hand segmentation is central: the proposed stereo system approaches active-camera performance, while weaker segmentation or stereo methods degrade results.

  • Experimental setup: Evaluation measures the percentage of joints whose maximum error is below a threshold across backgrounds, pose types, and tracking or estimation methods.Figure 6 averages results over backgrounds, while Figure 7 reports individual-background results.
  • Passive versus active sensing: The proposed stereo with proposed hand segmentation performs comparably to Intel F200 active depth sensing across the evaluated tasks.The corresponding curves are close, with the active-camera curve highest on average.
  • Stereo and segmentation: Traditional local stereo with proposed hand segmentation performs substantially below the proposed stereo because inaccurate hand masks reduce pose accuracy.This isolates the importance of the proposed stereo and segmentation combination.
  • Failure conditions: Generic skin-color segmentation fails completely for B4 and B6 and produces lower results elsewhere, particularly for PSO on random poses.The generic model also yields noisy disparities that limit CHPR except in B5.
  • Failure conditions: Meshstereo performs much worse than the proposed stereo on the remaining backgrounds and is excluded from B2 because its hand disparities are too poor for the adopted algorithms.The comparison indicates that general stereo benchmark performance does not transfer directly to hand pose tracking.
  • Stereo and segmentation: Without correct hand segmentation, passive stereo performance is low, whereas the proposed stereo method remains close to active depth cameras.The conclusion is supported across the reported tracking experiments.

VI. CONCLUSIONS

The paper introduces a passive-stereo benchmark and a hand segmentation method for 3D hand pose tracking and estimation. Results are comparable to active depth cameras in challenging indoor scenarios, but bad lighting remains a major limitation and outdoor performance is not quantitatively reported.

  • Conclusions: The proposed stereo-based hand segmentation method achieves performance comparable to active depth cameras under challenging indoor scenarios.The paper also captures outdoor sequences but does not provide quantitative outdoor results because ground-truth labeling is difficult.
  • Limitations: Passive stereo accuracy can drop dramatically under bad lighting because increased matching noise produces lower-quality disparity estimates.The reported failure case involves dark foreground objects lacking texture.
Loading 1610.07214v1…