PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, Christian Theobalt

arXiv:1704.02201v2cs.CV

TL;DR

Moving egocentric RGB-D cameras make real-time hand pose estimation difficult in cluttered scenes with self-occlusion and object occlusion. The paper combines two CNN stages, photorealistic merged-reality training data, and kinematic tracking, reporting robust quantitative and qualitative performance in these conditions.

  • Problem

    Real-time hand pose estimation from moving first-person cameras in cluttered real-world scenes with naturally occluded hand-object interactions remains unsolved.

  • Method

    Two CNNs localize the hand and regress root-relative 3D joint locations, while merged-reality synthetic data trains them and kinematic tracking refines temporal pose consistency.

  • Results

    The method is reported to be robust to self-occlusion and object occlusion, particularly from moving egocentric viewpoints, in quantitative and qualitative evaluation.

  • Takeaways & Limitations

    The paper provides a real-time approach and benchmark resources for hand tracking in cluttered egocentric scenes and natural hand-object interactions.

Abstract

from arXiv · show

We present an approach for real-time, robust and accurate hand pose estimation from moving egocentric RGB-D cameras in cluttered real environments. Existing methods typically fail for hand-object interactions in cluttered scenes imaged from egocentric viewpoints, common for virtual or augmented reality applications. Our approach uses two subsequently applied Convolutional Neural Networks (CNNs) to localize the hand and regress 3D joint locations. Hand localization is achieved by using a CNN to estimate the 2D position of the hand center in the input, even in the presence of clutter and occlusions. The localized hand position, together with the corresponding input depth value, is used to generate a normalized cropped image that is fed into a second CNN to regress relative 3D hand joint locations in real time. For added accuracy, robustness and temporal stability, we refine the pose estimates using a kinematic pose tracking energy. To train the CNNs, we introduce a new photorealistic dataset that uses a merged reality approach to capture and synthesize large amounts of annotated data of natural hand interaction in cluttered scenes. Through quantitative and qualitative evaluation, we show that our method is robust to self-occlusion and occlusions by objects, particularly in moving egocentric perspectives.

1. Introduction

Real-time hand pose estimation from moving egocentric cameras in cluttered scenes with occlusions remains difficult, despite its relevance to VR/AR and other applications. The paper addresses this with two CNNs, synthetic training data, and temporal pose refinement.

  • Motivation: Real-time hand pose estimation from moving first-person cameras in cluttered scenes with natural hand-object interactions remains unsolved.Occlusions, clutter, manipulated objects, and limited field of view make the setting particularly challenging.
  • Data and evaluation: SynthHands provides photorealistic annotated RGB-D training data covering natural hand interactions, object occlusions, and background variation.The merged-reality framework renders motions from novel egocentric views.
  • Approach: The method localizes the hand in 2D, regresses relative 3D locations for 21 joints, and uses a single commodity RGB-D camera.The two-stage design separates hand localization from 3D joint regression.
  • Approach: A pose-tracking energy estimates temporally smooth joint angles to refine CNN predictions affected by kinematic inconsistencies and temporal jitter.The tracking stage uses 3D and 2D joint-location constraints.
  • Data and evaluation: EgoDexter is a new annotated benchmark for evaluating hand tracking in egocentric cluttered scenes and object interactions.The paper reports extensive evaluation on this real benchmark dataset.

2. Related work

Prior work largely addressed free-hand tracking in uncluttered, third-person settings, while egocentric hand-object tracking in clutter remained difficult. The paper builds on synthetic-data approaches to target real-time, temporally consistent tracking.

  • Free-hand tracking: Most previous methods estimate hand pose for mid-air motion in uncluttered scenes viewed from third-person cameras.This simpler setting reduces the importance of occlusions.
  • Egocentric tracking: Egocentric tracking with natural hand-object interactions had been demonstrated in cluttered scenes, but earlier work was not designed for real-time tracking.Synthetic images were already used to address difficult egocentric views.
  • Contribution: The overview combines HALNet hand localization, JORNet joint regression, SynthHands training, and kinematic pose tracking.The pipeline proceeds from an RGB-D frame to a tracked kinematic skeleton.
  • Contribution: The paper uses large amounts of synthetic training data to pursue real-time, temporally consistent hand tracking under challenging occlusions.The approach extends the use of synthetic data toward real-time tracking.

3. Overview

The system formulates hand pose estimation as temporal energy minimization over a kinematic skeleton. It localizes the hand, regresses root-relative 3D joints from a normalized crop, and tracks the resulting pose over time.

  • Overview: The system uses RGB and depth from a single 640×480 Intel RealSense SR300 captured at 30 Hz.Pose estimation is formulated as energy minimization over the joint angles of a kinematic hand skeleton.
  • Hand localization: A localization CNN predicts a root heatmap, whose 2D and 3D positions define a normalized crop around the hand.The root may lie on the hand center or on an object occluding it.
  • 3D joint regression: A second CNN regresses root-relative 3D joint locations from the cropped hand image.The predicted joints are subsequently used by the kinematic tracking framework.
  • Pose tracking: The tracking stage estimates temporally smooth joint angles using 3D joint-location constraints and a kinematic skeleton.The implementation models the hand with 26 degrees of freedom.

4. Single Frame 3D Pose Regression

The method first converts RGB-D input into a colored depth map and localizes the hand before regressing root-relative 3D joints. A photorealistic SynthHands dataset supplies varied annotated training data for cluttered hand-object interactions.

  • Input representation: The system creates a calibrated colored depth map D and downsamples it to 320×240 for real-time pose regression.The colormap maps color-image pixels onto the depth map, reducing camera-specific extrinsic variation.
  • Hand localization: HALNet estimates a 2D hand-root heatmap, whose maximum and associated depth define a depth-dependent normalized crop containing the hand.The crop side length is inversely proportional to depth, and its depth channel is centered by subtracting the root depth.
  • Postprocessing: A history-based postprocessing rule marks low-likelihood maxima more than 30 pixels from the previous maximum as uncertain and progressively downweights them with δ = 0.98.The rule is designed to prevent outlier root locations from affecting subsequent 3D joint estimates.
  • 3D joint regression: JORNet regresses root-relative 3D joint positions from the normalized crop and also predicts per-joint 2D likelihood heatmaps for later regularization.Global joint positions are recovered by adding the backprojected 3D hand-root position.
  • SynthHands dataset: SynthHands combines retargeted real hand motion, photorealistic hand models, virtual objects, natural backgrounds, and randomized appearance variation for annotated RGB-D training data.The dataset contains roughly 220,000 RGB-D images and uses plain-color foreground and background rendering for postprocessed composition.

5. Hand Pose Optimization

The method fits a kinematic skeleton to CNN-regressed 3D joints, combining 3D positions and 2D heatmaps while enforcing joint limits and temporal smoothness.

  • Kinematic fitting addresses temporal jitter and inconsistent bone lengths in single-frame 3D pose predictions.The skeleton is parameterized by joint angles Θ and fitted to regressed 3D joint positions.
  • The data term combines 3D joint-position fitting with 2D heatmap fitting.Edata(Θ, pG, H) = wp3Epos3D(Θ, pG) + wp2Epos2D(Θ, H).
  • Joint-limit and temporal-smoothness regularizers constrain the fitted pose.The joint-limit term acts as a soft prior for biomechanical pose plausibility.
  • The temporal term enforces constant velocity to prevent dramatic pose changes.The regularizer weights are wl = 0.03 and wt = 10^-3, and optimization uses 20 conditioned-gradient-descent iterations.

6. Results and Evaluation

Evaluation on EgoDexter shows that the two-step RGB-D approach remains effective under clutter, occlusion, and moving egocentric viewpoints, with kinematic refinement producing the lowest reported average fingertip error.

  • Dataset: EgoDexter contains 3190 frames of natural hand-object interactions in cluttered scenes and moving egocentric viewpoints, with 1485 manually annotated frames.Annotations mark 2D and 3D fingertip positions.
  • Hand localization: 2.2 px average root-localization error with a 1.5 px standard deviation provides reliable crops for JORNet.HALNet is evaluated using 2D Euclidean pixel error on the test set.
  • JORNet evaluation: Ground-truth crops outperform hand-localization crops for 3D JORNet error, but the difference is not substantial and does not cause catastrophic failures.Hand-localization initialization yields marginally better 2D accuracy, which the authors associate with training on noisy root positions.
  • CNN architecture: The two-step RGB-D CNN achieves the best performance on real test sequences compared with depth-only and single combined CNN variants.Directly regressing global 3D pose with a single RGB-D CNN performs worse than the two-step approach; RGB-D also outperforms depth-only input.
  • Qualitative results: The method works qualitatively with complex object interactions, strong self-occlusions, varied users, and backgrounds, but fast motion or depth-image misalignment can produce incorrect predictions.The authors also report failure cases under large occlusions.
  • Kinematic refinement: Combining 2D and 3D fitting terms yields the lowest average fingertip error, 32.6 mm, while enforcing biomechanical and temporal consistency.Using only the 2D fitting energy causes catastrophic tracking failure, whereas the 3D-only version has similar error to raw predictions.

7. Conclusion

The paper presents a real-time hand pose estimation method for difficult egocentric scenes, supported by synthetic training data and an annotated benchmark. Additional appendix material documents experiments, dataset construction, and CNN training details.

  • 7. Conclusion: The method uses two CNNs to localize the hand and estimate 3D joint locations in real time, with kinematic tracking for temporal smoothness.The pose tracking energy estimates joint angles of a kinematic skeleton.
  • 7. Conclusion: SynthHands provides photorealistic annotated RGB-D training data covering natural hand interactions, hand variation, object occlusions, and background variation.
  • 7. Conclusion: EgoDexter is an annotated benchmark dataset containing challenging cluttered egocentric scenes.
  • 7. Conclusion: Quantitative and qualitative evaluation reports low errors and consistent performance under difficult occlusions, scene clutter, and background changes.
  • 7. Conclusion: The appendix contains additional experiments, SynthHands dataset details, and CNN architecture and training information.

A. Evaluation

The evaluation examines component combinations, comparisons with prior approaches, viewpoint generalization, and failure cases. Results support the value of combining 2D and 3D predictions while identifying severe occlusion and field-of-view failures.

  • A. Evaluation: Combining 2D and 3D predictions with the tracking framework consistently produces the best results because their failure modes differ.
  • A. Evaluation: Quantitative comparison with the other existing egocentric hand dataset was unavailable because its sensor is unsupported without camera-specific adaptation.The paper instead reports a qualitative comparison using mimicked sequences.
  • A. Evaluation: The method generalizes to third-person views despite machine-learning components trained only on the egocentric SynthHands dataset.The hand localization step is also reported as robust to faces and other skin-colored parts.
  • A. Evaluation: Failure cases include extreme self-occlusion, severe hand and object occlusion, and hands outside the camera field of view.

B. SynthHands Dataset

The appendix identifies SynthHands dataset variation and places it alongside additional experiment and training documentation. It points readers to the dataset’s variation table and representative frames.

  • B. SynthHands Dataset: Table 1 summarizes the modes of variation represented in the SynthHands dataset.
  • B. SynthHands Dataset: Figure 12 provides representative frames from SynthHands.
  • B. SynthHands Dataset: The surrounding appendix also presents experiments leading to network design decisions and training details for HALNet and JORNet.

C.1. Network Design

The network design prioritizes speed for fast egocentric hand motion, while intermediate supervision and full-joint prediction improve learning and pose accuracy. The resulting architecture substantially reduces inference time with only marginal localization-accuracy loss.

  • C.1. Network Design: HALNet and JORNet use architectures designed for faster inference than ResNet50 because hand and egocentric-camera motion are rapid.
  • C.1. Network Design: Training JORNet on heatmaps and local joint positions for all joints reduces SynthHands test-set error compared with JORNet light, trained only for fingertips and the wrist.
  • C.1. Network Design: 10 ms for HALNet and 6 ms for JORNet are reported forward-pass times, compared with 18 ms and 11 ms for ResNet50 at the corresponding resolutions.
  • C.1. Network Design: 2.2 px average error for HALNet versus 2.1 px for ResNet50 indicates only a marginal accuracy drop on the SynthHands test set.
  • C.1. Network Design: 6.4% improves the 2D pixel error when HALNet regresses heatmaps for all joints instead of only the root, from 2.35 px to 2.2 px.

C.2. Training Details

Training uses specified optimization settings for the two CNNs, while the accompanying figures document dataset samples, predictions, tracking, qualitative comparisons, generalization, and failure cases.

  • C.2. Training Details: Both networks are trained in Caffe with AdaDelta, momentum 0.9, weight decay 0.005, and input batches of 16.
  • C.2. Training Details: HALNet uses 320x240 px inputs, 40x30 px output heatmaps, a base learning rate of 0.05, and 45k training iterations.The main heatmap loss has weight 1.0, while intermediate heatmap losses have weight 0.5.
  • C.2. Training Details: The figures show qualitative predictions and tracked skeletons, comparisons with prior methods, third-person generalization, and failure cases across intermediate pipeline stages.The tracking framework combines 2D and 3D predictions, while failures may arise during localization, prediction, or tracking.
Loading 1704.02201v2…