Source-linked AI summary
Training a Feedback Loop for Hand Pose Estimation
Markus Oberweger, Paul Wohlhart, Vincent Lepetit
TL;DR
The paper addresses challenging 3D hand pose estimation from depth images without fitting an explicit 3D hand model. It uses learned synthesis and iterative pose updates in a feedback loop, achieving state-of-the-art performance and over 400 fps on a single GPU.
Problem
3D hand pose estimation remains challenging because hands have many possible degrees of freedom, self-similarity, and self-occlusions.
Method
The method replaces explicit hand-model fitting with learned image synthesis and an updater network that iteratively improves an initial pose estimate using the input and synthesized depth images.
Results
The method achieves state-of-the-art performance, reducing average Euclidean joint error to 16.5 mm, and runs at over 400 fps on a single GPU.
Takeaways & Limitations
The learned feedback loop can correct predictor errors while avoiding the carefully designed fitting function and algorithm required by explicit 3D model fitting.
Takeaways & Limitations
The optimization-based formulation has many local minima and can produce physically infeasible poses whose synthesized images may cause divergence or local-minimum stagnation.
Abstract
from arXiv · showhide
We propose an entirely data-driven approach to estimating the 3D pose of a hand given a depth image. We show that we can correct the mistakes made by a Convolutional Neural Network trained to predict an estimate of the 3D pose by using a feedback loop. The components of this feedback loop are also Deep Networks, optimized using training data. They remove the need for fitting a 3D model to the input data, which requires both a carefully designed fitting function and algorithm. We show that our approach outperforms state-of-the-art methods, and is efficient as our implementation runs at over 400 fps on a single GPU.
1. Introduction
The paper targets challenging 3D hand pose estimation from depth images with a data-driven feedback loop that iteratively corrects an initial CNN estimate without fitting a 3D hand model. The approach learns its components from training data and reports superior state-of-the-art performance with over 400 fps efficiency.
- Accurate hand pose estimation supports Human Computer Interaction and Augmented Reality, but remains difficult because hands have many freedoms, self-similarity, and self-occlusions.
- Existing methods commonly refine discriminative joint predictions by fitting a 3D hand model to depth data, an approach requiring complex optimization, multiple hypotheses, and a carefully designed fit criterion.
- The method replaces the hand model with learned image generation and learns pose updates from the input depth image and the image generated for the current estimate.
- The feedback loop iteratively applies learned updates to correct mistakes in an initial estimate from a discriminative method, with all components implemented as deep networks.
- The approach reports superior performance against state-of-the-art methods and runs at over 400 fps on a single GPU.
2. Related Work
Prior hand pose methods use discriminative prediction or generative model-based optimization, with the latter requiring hand models, similarity functions, optimization algorithms, and initialization. This paper replaces the first three building blocks with learned image synthesis and pose-update prediction.
- Recent discriminative methods directly predict joint locations from RGB or RGB-D images, but reported difficulties include occluded joints, fingertip inaccuracies, and limited accuracy.
- Generative model-based methods combine a hand model, similarity function, optimization algorithm, and initial pose to search for a fitting pose.
- Hand-crafted models trade speed against accuracy, using geometric or holistic representations such as spheres, cylinders, ellipsoids, cones, Gaussians, or linear blend skinning.
- Similarity functions use modalities including depth, salient points, edges, color, or combinations tied to the chosen hand model.
- Optimization is challenging in the high-dimensional pose space because of local minima and high computation time, motivating particle-swarm, gradient-based, and dynamics-simulation approaches.
- Initialization strategies include discriminative predictions, fingertip estimates, Hough Forest candidates, predefined hand color and position, manual initialization, or the previous frame's pose.
- Unlike prior work, this approach learns realistic depth-image generation from registered depth images instead of using a deformable CAD model.
- It also learns pose updates from training data rather than designing a hand-crafted similarity function and optimization algorithm.
3. Model-based Pose Optimization
The method replaces hand-crafted model fitting with learned depth-image synthesis and iterative pose updates. A CNN initializes the pose, while learned feedback repeatedly refines it using the observed and synthesized images.
- Method overview: The pipeline combines a discriminative CNN initializer, a learned hand synthesizer, and an updater that refines pose estimates iteratively.Each component is implemented as a Deep Network and trained using annotated depth-image data.
- Learned synthesizer: The synthesizer maps a pose to a realistic depth image, replacing a deformable CAD hand model and associated rendering engineering.It is trained from registered depth images and uses a CNN architecture with fully connected, unpooling, and convolutional layers.
- Optimization alternative: Directly optimizing image similarity is vulnerable to local minima and physically infeasible poses, whose synthesized images can make optimization diverge or stall.The learned updater is introduced as an alternative to this nonlinear least-squares procedure.
- Learned updater: The updater receives the observed depth image and the synthesizer output for the current pose, then predicts a pose update.Its two shared-weight convolutional paths are concatenated before a fully connected network produces the update.
- Learned updater: Iterative updates are trained to move estimates closer to ground truth rather than solve the full correction in one step.The training set includes ground-truth poses, predictor outputs, and additional meaningful deviations; the synthesizer enables virtual pose augmentation.
4. Evaluation
The method is evaluated on the NYU Hand Pose Dataset against established baselines and image-based optimization. Its learned feedback loop improves pose estimates, avoids failures of direct image matching, and runs at over 400 fps on a single GPU.
- 4.3. Benchmark: The evaluation uses the NYU Hand Pose Dataset, following the established protocol of evaluating 14 of its 36 annotated joints.The dataset contains over 72k training frames and 8k test frames, using depth images from a structured-light RGB-D sensor.
- 4.4. Comparison with Baseline: 16.5 mm: the proposed method reduces average Euclidean joint error from 21 mm and 20 mm for the two baselines.The method achieves state-of-the-art performance on the dataset, while its simple predictor alone has a 27 mm initialization error.
- 4.5. Image-Based Hand Pose Optimization: Directly minimizing the squared difference between input and synthesized depth images often worsens the pose because optimization fits contour noise and artifacts.Particle Swarm Optimization produces similar results, supporting the conclusion that the objective function—not only the optimizer—is problematic.
- 4.4. Comparison with Baseline: The proposed feedback loop improves joint localization even when initialized by a less accurate pose estimate than both comparison methods.A more complex initialization reaches 16 mm, but the simpler and faster predictor is already sufficient for the method.
- 4.6. Qualitative results: The updater predicts pose changes toward the ground-truth location for almost all tested initializations and can make larger corrections for worse initial poses.Qualitative examples show improvements even when the initial predictor is already good, particularly for the thumb.
- 4.7. Runtime: Over 400 fps: the implementation uses 0.07 ms for initialization and 1.2 ms per updater iteration, including the 0.8 ms synthesizer.The reported runtime uses two updater iterations in practice.
5. Discussion and Conclusion
The discussion explains how learned feedback can improve predictions despite the predictor and synthesizer sharing training data, and argues that the approach is not specific to hand pose or depth cameras. The authors also relate feedback mechanisms to biological evidence while noting that this interpretation remains unproved.
- 5. Discussion and Conclusion: Feedback paths in the visual cortex reportedly contain more neurons than the forward path, but the proposed connection to this mechanism remains unproved.The paper presents its approach only as a possible explanation for biological feedback in visual processing.
- 5. Discussion and Conclusion: Training the predictor and synthesizer on the same data still permits improvement because synthesized pose samples augment training beyond observed training poses.The updater learns to move sampled poses toward ground truth, including regions that may arise on unseen images.
- 5. Discussion and Conclusion: The learned components and simple pose representation suggest applicability to other detection and tracking tasks given suitable training data.The authors state that the approach does not depend specifically on hand pose detection or depth cameras.