Source-linked AI summary
Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs
Liuhao Ge, Hui Liang, Junsong Yuan, Daniel Thalmann
TL;DR
Accurate 3D hand pose estimation remains difficult because hand motion is high-dimensional and direct image-to-pose regression is highly non-linear. The paper uses multi-view CNNs to predict heat-maps from three orthogonal depth projections, fuses them with learned pose constraints, and reports superior real-time performance with cross-dataset generalization.
Problem
3D hand pose estimation remains challenging because large pose variations and high-dimensional hand motion make accurate image-to-pose regression difficult.
Method
The method projects depth-derived 3D points onto three orthogonal planes, predicts joint heat-maps with separate CNNs, and fuses them using learned hand-pose constraints.
Results
The method achieves superior 3D hand pose estimation performance on public datasets, operates at over 70fps, and shows relatively high cross-dataset accuracy.
Takeaways & Limitations
Multi-view heat-map fusion better leverages the 3D information in a single depth image for accurate real-time hand-joint estimation.
Takeaways & Limitations
The fusion formulation assumes the three projections are conditionally independent given joint locations and assumes equal prior probability for joint locations.
Abstract
from arXiv · showhide
Articulated hand pose estimation plays an important role in human-computer interaction. Despite the recent progress, the accuracy of existing methods is still not satisfactory, partially due to the difficulty of embedded high-dimensional and non-linear regression problem. Different from the existing discriminative methods that regress for the hand pose with a single depth image, we propose to first project the query depth image onto three orthogonal planes and utilize these multi-view projections to regress for 2D heat-maps which estimate the joint positions on each plane. These multi-view heat-maps are then fused to produce final 3D hand pose estimation with learned pose priors. Experiments show that the proposed method largely outperforms state-of-the-art on a challenging dataset. Moreover, a cross-dataset experiment also demonstrates the good generalization ability of the proposed method.
1. Introduction
The paper addresses difficult, high-dimensional 3D hand pose regression by combining CNN heat-maps from three orthogonal projections of a depth image. Learned pose constraints and multi-view evidence improve robustness to depth errors and ambiguous hotspots, with strong benchmark and cross-dataset performance.
- 3D hand pose estimation remains challenging because large pose variations and high-dimensional hand motion make image-to-pose mapping difficult.
- Direct CNN mapping from image features to 3D locations is highly non-linear, complex to learn, and has low generalization ability.
- The method projects an input depth image’s 3D points onto three orthogonal planes and feeds each projection to a separate CNN for joint heat-map prediction.
- Fusing heat-maps from multiple views with learned hand-pose priors produces 3D joint locations and helps resolve ambiguous single-view estimations.
- Multi-view CNNs reduce sensitivity to single-view depth errors by estimating front, side, and top-view evidence simultaneously.
- Unlike approaches relying on predefined hand models, the method embeds hand-pose constraints learned from training samples implicitly.
- The method outperforms state-of-the-art methods on public datasets, runs at over 70fps, and achieves relatively high cross-dataset accuracy.
2. Literature Review
Prior hand pose estimation methods include model-driven fitting and data-driven regression, with CNNs offering effectiveness but facing training and generalization challenges for direct coordinate mapping.
- Hand pose estimation methods are commonly divided into model-driven approaches that fit deformable 3D models and data-driven approaches that learn from training data.
- Model-driven methods can be effective with depth cameras but require explicit anatomical constraints and hand-model fitting.
- Data-driven methods encode hand size and motion constraints in training data instead of explicitly specifying them.
- Random forests and related methods have demonstrated reasonable accuracy and speed for direct hand-pose regression from depth images.
- CNNs have been effective for articulated pose estimation, but prior direct-mapping approaches can require long training times and face high nonlinearity, complexity, and limited generalization.
3. Methodology
The method projects a depth image onto three orthogonal planes, uses view-specific CNN heat-maps to represent joint-position probabilities, and fuses them under a learned low-dimensional hand-pose prior to estimate 3D joints.
- Problem formulation: The task estimates K 3D hand joint locations from a cropped depth image containing a human hand.The method uses K = 21 joints in this work.
- Pose-prior fusion: The optimized pose is constrained to a PCA-learned low-dimensional subspace and reconstructed by back-projecting the optimal coefficients into the original joint space.The subspace uses principal components and the empirical mean learned from training hand configurations, with M ≪ 3 × K.
- Multi-view learning: Each view uses the same multi-resolution CNN architecture to generate 21 heat-maps representing joint-location confidence on that projection plane.The network combines three resolution banks and produces heat-maps of size 18×18 pixels.
- Multi-view projection: The depth image is converted into 3D points and projected onto the x-y, y-z, and z-x planes of an OBB-based projection coordinate system.The OBB is fitted by PCA, centered on the 3D-point bounding box, with axes aligned to its principal components.
- Multi-view fusion: The three views’ heat-maps provide pairwise coordinate probabilities that are combined into a product distribution for each joint.The product Q is evaluated from heat-map intensities, using uniformly sampled 3D points in the bounding box.
- Multi-view fusion: The product distribution is approximated by a 3D Gaussian, whose parameters are estimated from sampled data before pose optimization.The Gaussian mean and covariance summarize each joint’s fused multi-view evidence.
4. Experiments
Experiments evaluate the proposed multi-view regression through self-comparisons, state-of-the-art comparisons, runtime measurement, and cross-dataset testing. Multi-view fine fusion generally improves accuracy, viewpoint robustness, and real-time performance, while retaining limitations at very small error tolerances and against methods using additional information.
- Dataset and evaluation: The evaluation uses eight subjects for training and one for testing on the dataset in, repeated across all nine subjects, with an additional cross-dataset evaluation.The cross-dataset evaluation trains on and tests on.
- Self-comparisons: Multi-view regression significantly outperforms single-view regression, while fine fusion improves mean error over coarse fusion to about 13 mm on.Fine fusion is overall preferred because its stronger performance occurs at smaller error tolerances, despite being worse than coarse fusion at some large tolerances.
- Self-comparisons: Fine fusion resolves ambiguous fingertip assignments more accurately than single-view regression by combining heat-maps from three views.In the illustrated case, single-view regression confuses the index and little fingertips, while coarse fusion underuses the available heat-map information.
- Self-comparisons: Using 35 principal components for joint constraints is selected as reasonable based on worst case accuracy.This setting is used in the remaining experiments.
- Comparison with state-of-the-art: The proposed method significantly outperforms in worst case accuracy using 14 corresponding joints for fair comparison.The comparison accounts for estimating 14 joints and the proposed method estimating 21.
- Comparison with state-of-the-art: At 20 mm and 30 mm error tolerances, good-sample proportions are about 10% and 15% higher than, respectively, while performance is slightly lower below 15 mm.Average errors are smaller across all yaw and pitch angles, and pitch-angle standard deviation is 0.64 mm versus 0.79 mm for.
- Runtime and generalization: The full pipeline takes 14.1 ms, corresponding to real-time operation above 70 fps.Projection takes 2.6 ms, CNN forward propagation 6.8 ms, and fusion 4.7 ms.
- Runtime and generalization: Cross-dataset testing shows generalization without calibrated hand models, ground-truth initialization, or temporal information, although methods using those resources perform better.The method still outperforms FORTH, PSO, and ICP, but is worse than ICP-PSO and ICP-PSO* in the more challenging cross-dataset setting.
5. Conclusion
The paper presents a multi-view CNN method that fuses heat-maps from multiple views to estimate 3D hand-joint locations. Experiments show accurate real-time 3D hand pose estimation.
- The proposed method uses multi-view CNNs to generate and fuse heat-maps for estimating 3D hand-joint locations.
- The multi-view approach better leverages 3D information from a single depth image for accurate joint-location estimation.
- Experimental results show superior performance for 3D hand pose estimation in real time.