Source-linked AI summary
Hands Deep in Deep Learning for Hand Pose Estimation
Markus Oberweger, Paul Wohlhart, Vincent Lepetit
TL;DR
The paper addresses accurate 3D hand-pose estimation from depth maps, a difficult task because of hand articulation, self-similarity, and occlusion. It evaluates CNN architectures with a learned pose prior and context-aware refinement, achieving higher accuracy and speed than previous state-of-the-art methods on challenging benchmarks. The results support direct 3D regression with constrained pose modeling and multiscale refinement.
Problem
Accurate 3D hand-pose estimation from depth maps remains challenging because hands have many degrees of freedom, self-similarity, and self-occlusions.
Method
The paper evaluates CNN architectures that jointly regress 3D joints, optionally predict a lower-dimensional constrained pose, and refine joints independently using overlapping multiscale regions.
Results
The proposed architectures significantly outperform the state of the art on several challenging benchmarks in localization accuracy and computation time.
Takeaways & Limitations
A learned hand-pose prior and multiscale contextual refinement improve the accuracy, reliability, and efficiency of CNN-based hand-pose prediction.
Takeaways & Limitations
Evaluation on ICVL is constrained by inaccurate test annotations, whose uncertainty affects measured accuracy.
Abstract
from arXiv · showhide
We introduce and evaluate several architectures for Convolutional Neural Networks to predict the 3D joint locations of a hand given a depth map. We first show that a prior on the 3D pose can be easily introduced and significantly improves the accuracy and reliability of the predictions. We also show how to use context efficiently to deal with ambiguities between fingers. These two contributions allow us to significantly outperform the state-of-the-art on several challenging benchmarks, both in terms of accuracy and computation times.
1. Introduction
The paper investigates CNN architectures for regressing 3D hand joints from depth maps, focusing on pose priors and context-aware refinement. These contributions improve accuracy and speed over prior methods on challenging benchmarks.
- Motivation: The paper evaluates CNN architectures for predicting 3D hand joint locations from depth maps, where network layout strongly influences accuracy.The task remains difficult because hands have many degrees of freedom, self-similarity, and self-occlusions.
- Pose prior: A learned prior model of hand pose is integrated into the network through an unusual bottleneck to improve predicted pose accuracy.The bottleneck has fewer neurons than the final layer.
- Context-aware refinement: A refinement stage uses multiple input regions and scale-dependent pooling to combine precise local information with broader context.Smaller regions provide accuracy, while larger regions provide contextual information for resolving finger ambiguities.
- Results: The method significantly outperforms the state of the art on several challenging benchmarks in both accuracy and computation time.It runs at over 5000 fps on a single GPU and over 500 fps on a CPU, described as one order of magnitude faster than the state of the art.
2. Related Work
Prior hand-pose methods include generative tracking and discriminative prediction from RGB or RGB-D images. The paper positions its CNN architecture as a direct 3D-regression approach that addresses limitations in earlier methods.
- Generative approaches: Generative tracking methods use 3D hand models and synthesis-based optimization, but require careful initialization to guarantee convergence.They may rely on the pose from preceding frames or separate initialization procedures.
- Discriminative approaches: Discriminative methods directly predict joint locations or classify hand configurations from RGB or RGB-D images using forests, regression, or CNNs.Earlier approaches include multi-layered Random Forests, regression forests, and cascaded CNNs.
- Discriminative approaches: Dictionary-based forest methods classify given poses rather than predicting the actual 3D pose.This limitation distinguishes them from direct 3D regression approaches.
- CNN-based approaches: A heatmap-based CNN approach is limited by 2D joint prediction, hidden-joint depth estimation, heatmap resolution, and costly pixel-wise CNN evaluation.These constraints motivate alternatives that directly regress 3D joint locations.
- Position of this work: This paper proposes and investigates CNN architectures for hand pose estimation, reporting performance above baselines on two difficult datasets.The authors build on CNN performance while emphasizing that network structure is important.
3. Hand Pose Estimation with Deep Learning
The method uses a two-stage CNN pipeline: jointly predict all 3D joints, optionally through a low-dimensional pose prior, then independently refine each estimate using overlapping multi-scale regions.
- Overall approach: The pipeline first predicts all joint locations simultaneously, then independently refines each joint estimate from the first stage.The first stage may predict pose parameters in a lower-dimensional space before reconstruction into joint coordinates.
- Problem formulation: The task is to estimate J 3D hand-joint locations from a single depth image after extracting and normalizing a coarse hand-centered cube.The cube is resized to a 128 × 128 depth patch and normalized to [−1, 1].
- First-stage architectures: The evaluated first-stage networks include shallow, deep, and multi-scale CNN architectures, with multi-scale inputs formed from downscaled depth maps.The shallow and deep models use convolutional, pooling, and fully connected layers; the multi-scale design is intended to capture context.
- First-stage architectures: The multi-scale architecture outperforms the deep architecture, which outperforms the shallow one, while the paper’s pose prior and refinement contributions provide larger improvements.The comparison is reported as an ordering of the baseline architectures, followed by stronger gains from the proposed contributions.
- Pose prior: A constrained pose prior is implemented with a bottleneck containing fewer than 3 · J neurons, forcing a low-dimensional representation of physically constrained hand poses.The bottleneck is followed by a reconstruction layer that maps the embedding back into the 3 · J-dimensional joint space; an 8-dimensional embedding can represent a 42-dimensional pose vector.
- Refinement: The ORRef refinement architecture combines small, lightly pooled patches for accuracy with larger patches for spatial context around each predicted joint.The context is intended to help avoid confusion between fingers, and the standard single-patch architecture serves as the StdRef baseline; refinement can be iterated.
4. Evaluation
The evaluation compares architectures on two challenging hand-pose benchmarks using joint-distance metrics, showing benefits from pose priors, overlapping-patch refinement, and fast inference.
- Benchmarks: The evaluation uses NYU and ICVL hand-pose benchmarks, reporting quantitative and qualitative results across different architectures.NYU provides accurate annotations and high pose variability, while ICVL has noisy annotations and more limited pose variability.
- Evaluation metrics: The study reports average Euclidean joint error and the fraction of test samples whose joints all remain below a distance threshold.The threshold metric is stricter because one dislocated joint can deteriorate the whole predicted pose.
- Pose prior: 30 dimensions provide the best pose embedding, while only 8 dimensions already outperform direct regression in the full pose space on both datasets.The pose prior improves results on NYU and ICVL, although the improvement is less drastic on ICVL because of noisy annotations.
- Pose refinement: The ORRef refinement stage improves joint localization by combining small high-resolution regions with larger contextual patches.For inaccurate initial estimates, larger patches supply needed offset context; for accurate pose-prior estimates, small patches improve precision. Two iterations are used.
- Qualitative and annotation effects: Evaluation is sensitive to annotation and depth quality: noisy ICVL labels affect measured accuracy, while large missing-depth regions worsen refinement but preserve topology under the pose constraint.Revised ICVL annotations yielded an average error of 2.4 mm with a standard deviation of 5.2 mm on the evaluated test sequence.
5. Conclusion
The paper evaluates CNN architectures for direct 3D hand-joint regression and combines a constrained pose prior with joint-specific refinement to improve accuracy and speed.
- A constrained prior hand model improves joint localization accuracy, while joint-specific refinement uses overlapping patches with different pooling sizes to combine resolution and context.The architectures outperform previous state of the art on two datasets in both localization accuracy and speed.