Source-linked AI summary
Dense 3D Regression for Hand Pose Estimation
Chengde Wan, Thomas Probst, Luc Van Gool, Angela Yao
TL;DR
The paper targets limitations of holistic 3D hand-pose regression and volumetric detection from single depth frames. It introduces dense pixel-wise 2D/3D estimation with consensus-based aggregation, and reports superior performance to prior methods across three public datasets.
Problem
Holistic regression struggles with local finger-pose combinations, translation variation, self-occlusion ambiguity, and the mismatch between 2.5D depth data and conventional CNN representations.
Method
The method re-parameterizes pose as per-pixel 2D/3D heat maps and 3D unit vector fields, then aggregates local estimates with a non-parametric mean-shift variant.
Results
The method outperforms previous state-of-the-art approaches on NYU, ICVL, and MSRA, with large-margin gains over holistic-regression baselines.
Takeaways & Limitations
Dense pixel-wise fusion provides a more accurate alternative to holistic 3D regression while retaining 2D and 3D consistency through post-processing.
Abstract
from arXiv · showhide
We present a simple and effective method for 3D hand pose estimation from a single depth frame. As opposed to previous state-of-the-art methods based on holistic 3D regression, our method works on dense pixel-wise estimation. This is achieved by careful design choices in pose parameterization, which leverages both 2D and 3D properties of depth map. Specifically, we decompose the pose parameters into a set of per-pixel estimations, i.e., 2D heat maps, 3D heat maps and unit 3D directional vector fields. The 2D/3D joint heat maps and 3D joint offsets are estimated via multi-task network cascades, which is trained end-to-end. The pixel-wise estimations can be directly translated into a vote casting scheme. A variant of mean shift is then used to aggregate local votes while enforcing consensus between the the estimated 3D pose and the pixel-wise 2D and 3D estimations by design. Our method is efficient and highly accurate. On MSRA and NYU hand dataset, our method outperforms all previous state-of-the-art approaches by a large margin. On the ICVL hand dataset, our method achieves similar accuracy compared to the currently proposed nearly saturated result and outperforms various other proposed methods. Code is available $\href{"https://github.com/melonwan/denseReg"}{\text{online}}$.
1. Introduction
The paper addresses limitations of holistic 3D regression and volumetric detection by combining dense pixel-wise estimation with 2D and 3D geometric information from depth maps. Its method uses carefully parameterized local predictions and consensus-based aggregation, achieving strong results on three public datasets.
- Holistic regression struggles with disparate image-to-pose mapping, local finger-pose combinations, translational variance, and bounding-box sensitivity.
- 3D CNN approaches are computationally heavy at useful resolutions and struggle with self-occlusion ambiguities in 2.5D depth inputs.
- The method formulates hand pose estimation as dense regression that leverages both 2D surface geometry and 3D coordinates.
- Pose parameters are represented as 2D and 3D joint heat maps plus 3D unit vector fields, estimated through pixel-wise network predictions.
- The method outperforms previous state-of-the-art approaches on NYU, ICVL, and MSRA datasets, with large-margin advantages over several holistic-regression baselines.
- A non-parametric post-processing method aggregates pixel-wise estimates while enforcing consensus between 2D and 3D estimates.
2. Related Works
Related work spans 2D–3D pose fusion, alternative pose parameterizations, deep-learning hand-pose systems, and offset regression. The paper distinguishes its approach through dense pixel-wise 3D estimation and lightweight consensus-based post-processing.
- Coupling 2D joint detection with 3D estimation: Prior 2D–3D fusion methods combine 2D detections and 3D estimation, often using multi-task setups or additional optimization procedures.
- Coupling 2D joint detection with 3D estimation: Unlike holistic 3D regression, the proposed fusion performs pixel-wise 3D estimation, providing translation invariance and better generalization across finger-gesture combinations.
- Pose Parameterization: Existing pose representations include latent spaces and hierarchical skeleton parameterizations designed to model correlations and dependencies among joints.
- Hand Pose Estimation: Deep-learning hand-pose methods include two-stage pipelines with 2D detection and model-based optimization, as well as single-stage holistic regression.
- Offset Regression and Hough Voting: Previous offset-regression methods offer translation invariance and bottom-up compatibility; this work extends the idea to end-to-end learned dense 3D offset regression.
3. Method
The method combines pixel-wise 2D and 3D estimations with dense offset regression, then aggregates these local evidences into holistic 3D joint coordinates using mean shift and 2D–3D consensus.
- Dense pixel-wise estimation: The network represents each joint with 2D and 3D heat maps plus a 3D unit-vector field estimated from depth-map pixels.These estimates capture both the depth map’s 2D surface structure and its interpretation as a set of 3D points.
- Pose parameterization: 3D offset regression uses translation-invariant offsets, decomposed into a 3D heat map and directional unit vector rather than directly predicting joint coordinates.The 3D heat map identifies candidate support regions around joints, while the vector field provides directions from depth points toward joints.
- Pose parameterization: A separate 2D projected heat map provides over-complete evidence that adds robustness to local 3D joint estimates.The projected heat map is combined with 3D estimates through non-parametric post-processing.
- Network architecture: Cascaded hourglass modules estimate the heat maps and vector fields in a multi-task network with intermediate supervision and L2 losses across joints and stacks.Each module receives previous estimates, and the corresponding ground-truth and predicted maps and vector fields contribute to the loss.
- Inference: During inference, mean shift aggregates pixel-wise estimates into holistic 3D joint coordinates while enforcing consensus between 2D detections and 3D estimates.The procedure selects points using estimated 3D distances, projects them into 2D, retrieves 2D heat-map weights, and iteratively estimates each joint.
- Inference: The implementation uses an unweighted mean-shift approximation as an alternative to the 2D-projection strategy, with nearly identical results in Table 1.In practice, the method selects K = 5 nearest points and uses a 40mm kernel width based on ablative analysis.
4. Experiments
Experiments evaluate the method on three public datasets, using NYU for ablations and baseline comparisons. Results show that dense pixel-wise estimation and the proposed fusion outperform holistic regression baselines, with strong gains on NYU and MSRA and near-saturated performance on ICVL.
- Experimental setup: Experiments use NYU, MSRA, and ICVL, with NYU selected for ablations because it covers more hand poses.Evaluation uses mean joint error and the percentage of frames whose joints fall below a threshold.
- Baseline methods: Adding 2D detection to direct 3D coordinate regression improves mean joint error by only 0.16mm on NYU.The two approaches perform similarly above a 25mm error threshold, indicating that this coupling does not resolve holistic regression’s limitations.
- Baseline methods: Detection plus depth regression outperforms detection plus coordinate regression across average joint error and thresholds from 20 to 50mm, but remains far below the proposed method.The result supports explicitly enforcing consensus between 2D detections and 3D estimates.
- Baseline methods: Direct dense offset regression with distance-masked loss fails to train reliably, whereas the proposed pixel-wise dense estimation outperforms holistic regression.The proposed reparameterization decomposes offsets into a 3D heat map and directional unit vector, while applying loss to outlier points rather than excluding them.
- Comparison to state of the art: The method improves NYU success rates by 8% at 20mm and 9.2% at 30mm, while MSRA reaches over 81% and 91% below those thresholds.On ICVL, accuracy is similar to the strongest nearly saturated result and better than the other compared methods.
5. Conclusion and discussion
The method estimates 3D hand pose from a single depth map by combining dense pixel-wise representations with consensus-based vote aggregation.
- The method decomposes pose parameters into 2D and 3D joint heat maps plus 3D unit vector fields of offset directions.
- A non-parametric mean shift variant aggregates local estimations while enforcing consistency between 3D joint coordinates and local 2D/3D predictions.
- The approach provides a better fusion of 2D detection and 3D regression than previous state-of-the-art methods and baselines.
- The authors propose extending the method to RGB-based 3D pose estimation and hands grasping objects.