Source-linked AI summary
DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation
Markus Oberweger, Vincent Lepetit
TL;DR
Accurate 3D hand pose estimation is difficult because hand geometry produces self-similarity and self-occlusion. DeepPrior++ updates a simple depth-based estimator with ResNet layers, data augmentation, and trained hand localization, achieving better or similar performance than recent methods on NYU, ICVL, and MSRA while retaining simplicity.
Problem
Accurate 3D hand pose estimation remains challenging because hands have many degrees of freedom, self-similarity, and self-occlusions.
Method
DeepPrior++ combines a Residual Network, improved training with data augmentation, and a trained refinement method for initial hand localization.
Results
DeepPrior++ performs better than the original DeepPrior and similarly or better than more recent works on the three main benchmarks while retaining the same key ideas.
Takeaways & Limitations
The results suggest that careful improvements to the steps of a simple method can substantially improve its benchmark performance without abandoning its simplicity.
Takeaways & Limitations
The approach’s accuracy depends on using a more accurate 3D hand localization method.
Abstract
from arXiv · showhide
DeepPrior is a simple approach based on Deep Learning that predicts the joint 3D locations of a hand given a depth map. Since its publication early 2015, it has been outperformed by several impressive works. Here we show that with simple improvements: adding ResNet layers, data augmentation, and better initial hand localization, we achieve better or similar performance than more sophisticated recent methods on the three main benchmarks (NYU, ICVL, MSRA) while keeping the simplicity of the original method. Our new implementation is available at https://github.com/moberweger/deep-prior-pp .
1. Introduction
DeepPrior++ updates a simple depth-based 3D hand pose estimator with a stronger architecture, improved localization, and expanded training, targeting accurate and fast benchmark performance.
- Accurate 3D hand pose estimation remains challenging because hands have many degrees of freedom, self-similarity, and self-occlusions.
- DeepPrior predicts 3D hand skeleton joint locations directly from a single depth image while integrating a PCA-based pose prior into a convolutional network.
- DeepPrior++ introduces a Residual Network to make feature extraction more powerful.
- DeepPrior++ replaces heuristic initial hand localization with a trained method.
- DeepPrior++ improves training to leverage more information from available data, including augmentation.
- The modifications are evaluated against state-of-the-art methods on public benchmark datasets.
2. Related Work
Related work explores hierarchical, generative, geometric, and alternative depth representations, often adding multiple models, optimization, or specialized requirements. DeepPrior++ emphasizes a simpler, faster approach with comparable real-time speed.
- Recent hand-pose methods exploit kinematic hierarchies through multiple predictors or specially designed CNN architectures.
- Alternative input representations include 3D volumes, surface normals, segmentation, heatmaps, and reprojections, each introducing computational or processing requirements.
- Probabilistic and cross-network methods use multiple networks or complex procedures to model pose distributions or shared mappings.
- DeepModel integrates manually defined physical constraints from a 3D hand model into a CNN.
- Generative and model-based approaches can require hand models, user-specific adjustment, or complex optimization during inference.
- DeepPrior++ is described as easier and faster to train, simpler in architecture, more accurate, and comparable in speed, namely realtime, than recent approaches.
3. Original DeepPrior
Original DeepPrior detects and normalizes a hand region from one depth image, then predicts a lower-dimensional pose representation using a PCA-based prior within the network.
- DeepPrior estimates 3D hand joint locations from a single depth image using labeled depth images for training.
- Unavailable depths or values beyond the cube’s back face are assigned depth 1, supporting invariance to hand-camera distance during learning.
- Instead of directly predicting all joint locations, DeepPrior predicts pose parameters in a lower-dimensional space to enforce hand-pose constraints.
- The network initializes its last-layer weights with major PCA components from 3D hand-pose data, then trains the full network by standard back-propagation.
4. DeepPrior++
DeepPrior++ enhances DeepPrior with stronger architecture, improved localization, and expanded training through depth-specific augmentation and a robust pose prior.
- DeepPrior++ combines improved data augmentation, better hand localization, and a more powerful network architecture.These changes are presented as enhancements to the original DeepPrior approach.
- Online rotation, scaling, and translation augmentation produces more than 10M training samples and is intended to reduce overfitting and improve robustness.
- The pose prior is computed from 1M augmented 3D poses generated using random rotation, scaling, and translation.
- A regression CNN refines the initial center-of-mass localization by predicting the middle-finger MCP joint location as a reference.For real-time use, the CNN can refine the previous frame’s hand location instead of recomputing the center of mass.
- The adapted ResNet takes a normalized 128 × 128 hand crop, removes global average pooling, and adds two fully connected layers for regression.Its architecture includes four residual modules, while the localization-refinement network is a simpler convolutional model with pooling and fully connected layers.
- Dropout with rate 0.3 is applied to both fully connected layers to regularize the high-capacity ResNet model.
5. Evaluation
The evaluation tests DeepPrior++ on three public hand-pose benchmarks using established average joint error and an all-joints threshold metric.
- DeepPrior++ is evaluated on the NYU, ICVL, and MSRA public benchmark datasets.
- All experiments use a 30-dimensional PCA prior, with training taking approximately 10 hours under the reported hardware setup.
- Average 3D joint error is used as the primary accuracy metric because it is common and straightforward to compare across methods.
- The evaluation also plots the fraction of frames whose predicted joints all lie below a specified maximum Euclidean distance from ground truth.
5.2. NYU Dataset
On NYU, DeepPrior++ is evaluated using single-camera depth data and established 14-joint metrics, outperforming reported competing methods under multiple comparisons.
- The NYU dataset contains over 72k training frames and 8k test frames, with noisy outlines, missing depth values, accurate annotations, and varied poses.
- DeepPrior++ significantly outperforms all other methods on NYU according to the reported average 3D error in millimeters.
- Compared with discriminative approaches, DeepPrior++ performs better for the majority of frames, although Supancic et al. report high accuracy for a fraction of frames.
- Using the alternate protocol evaluating the first 2400 test frames, DeepPrior++ significantly outperforms Taylor et al.’s state-of-the-art method.
- Unlike several compared methods requiring a possibly user-specific 3D hand model, DeepPrior++ uses only training data without a 3D model.
5.3. ICVL Dataset
On ICVL, DeepPrior++ achieves state-of-the-art accuracy, although its advantage over other methods is small. Similar performance curves suggest that this dataset may be approaching saturation, with annotation uncertainty contributing to the remaining error.
- 5.3. ICVL Dataset: DeepPrior++ achieves state-of-the-art accuracy on the ICVL dataset, but the gap to other methods is much smaller.The dataset is described as easier because it has smaller pose variations and evaluation annotation errors.
- 5.3. ICVL Dataset: DeepPrior++ performs similarly to Guo et al., Wan et al., and Tang et al., all of which achieve state-of-the-art accuracy on ICVL.Tang et al. performs slightly better in parts of the curve, while DeepPrior++ performs significantly better on NYU.
- 5.3. ICVL Dataset: Similar error curves across several methods may indicate saturating performance on ICVL.The passage attributes remaining error to annotation uncertainty.
5.4. MSRA Dataset
On MSRA, DeepPrior++ outperforms existing methods by a substantial margin under the average 3D error evaluation and also leads on the plotted metric. Its performance indicates that it handles hands from different users.
- 5.4. MSRA Dataset: The MSRA dataset contains about 76k depth frames from nine subjects and uses leave-one-out cross-validation.Each run trains on eight subjects and evaluates on the remaining subject.
- 5.4. MSRA Dataset: The MSRA results show that DeepPrior++ can handle different users’ hands.
5.5. Ablation Experiments
Ablations show that data augmentation, refined hand localization, and the ResNet architecture each improve accuracy, while the resulting ResNet model still runs in real time.
- 5.5.1 Training Procedure: Over 7mm: data augmentation increases accuracy on NYU.Translation augmentation accounts for hand-detection errors, while rotation augmentation enlarges the range of training poses.
- 5.5.1 Training Procedure: Computing the pose prior from augmented 3D poses is important when data augmentation is used.Using augmented training data with a prior computed from original poses performs worse than using no data augmentation.
- 5.5.2 Hand Localization: More accurate 3D hand localization remains an avenue for improvement because practical detectors do not provide perfect localization.
- 5.5.2 Hand Localization: Refined center-of-mass localization reduces 3D localization error by almost 20mm and final average 3D pose error by over 1mm.Ground-truth localization achieves the highest accuracy but is not feasible in practice.
- 5.5.3 Network Architecture: More than 3mm: improved training and localization improve the original architecture over the 19.8mm result from [18].The proposed ResNet architecture improves accuracy by another 4mm on average because of its higher model capacity.
- 5.5.3 Network Architecture: The ResNet architecture remains applicable to realtime applications at over 30fps on a single GPU.It is slower than the original implementation but maintains realtime performance.
5.6. Qualitative Evaluation
Qualitative comparisons show that DeepPrior++ produces substantially better 3D pose estimates than DeepPrior, especially for highly articulated poses.
- 5.6. Qualitative Evaluation: DeepPrior++ provides significantly better qualitative results than DeepPrior, especially on highly articulated poses.The improvement is attributed to data augmentation, better localization, and the more powerful CNN structure.
6. Discussion and Conclusion
The paper situates DeepPrior++ among pose-prior methods and concludes that careful attention to implementation steps can substantially improve a simple approach. DeepPrior++ performs significantly better than original DeepPrior and similarly or better than more recent works while retaining the same key ideas.
- Prior work replaced PCA with encoders or auto-encoders to learn more expressive pose representations for improved pose estimation accuracy.These approaches modify how pose embeddings or compact representations are learned.
- Pose-prior methods therefore differ in representation design, including linear PCA, learned encoders, higher-dimensional embeddings, and factorized priors.The cited approaches illustrate several alternatives to the original PCA formulation.
- Another method factorized the hand pose prior into canonical coordinates and relative motion, unlike the PCA-based prior used here.The distinction concerns whether the prior separates these two components.
- DeepPrior++ performs significantly better than original DeepPrior and similarly or better than more recent works while retaining the same key ideas.The authors emphasize that the approaches share key ideas despite the performance difference.