Source-linked AI summary
V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map
Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee
TL;DR
Existing single-depth-map pose estimators commonly treat depth as a 2D image and directly regress 3D coordinates, despite perspective distortion and nonlinear mapping. V2V-PoseNet instead predicts per-keypoint likelihoods over a 3D voxelized grid, achieving state-of-the-art performance on almost all public datasets in real time and winning the HANDS 2017 challenge.
Problem
Existing methods use 2D depth images for direct 3D coordinate regression, which introduces perspective distortion and a highly nonlinear mapping.
Method
V2V-PoseNet converts depth maps into 3D voxelized grids and estimates per-voxel likelihoods for each keypoint.
Results
The method achieves state-of-the-art performance on almost all public 3D hand and human pose datasets, runs in real time, and wins the HANDS 2017 challenge.
Takeaways & Limitations
Voxel-to-voxel prediction provides a new approach to accurate 3D pose estimation from a single depth map.
Takeaways & Limitations
Keypoint localization requires a cubic box containing the target object, and depth-thresholding reference points can fail in cluttered scenes.
Abstract
from arXiv · showhide
Most of the existing deep learning-based methods for 3D hand and human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of keypoints, such as hand or human body joints, via 2D convolutional neural networks (CNNs). The first weakness of this approach is the presence of perspective distortion in the 2D depth map. While the depth map is intrinsically 3D data, many previous methods treat depth maps as 2D images that can distort the shape of the actual object through projection from 3D to 2D space. This compels the network to perform perspective distortion-invariant estimation. The second weakness of the conventional approach is that directly regressing 3D coordinates from a 2D image is a highly non-linear mapping, which causes difficulty in the learning procedure. To overcome these weaknesses, we firstly cast the 3D hand and human pose estimation problem from a single depth map into a voxel-to-voxel prediction that uses a 3D voxelized grid and estimates the per-voxel likelihood for each keypoint. We design our model as a 3D CNN that provides accurate estimates while running in real-time. Our system outperforms previous methods in almost all publicly available 3D hand and human pose estimation datasets and placed first in the HANDS 2017 frame-based 3D hand pose estimation challenge. The code is available in https://github.com/mks0601/V2V-PoseNet_RELEASE.
1. Introduction
V2V-PoseNet addresses perspective distortion and nonlinear coordinate regression by predicting keypoint likelihoods on a 3D voxelized representation. Experiments report state-of-the-art performance across nearly all public datasets and real-time operation.
- Motivation: Perspective distortion creates a many-to-one relation in 2D depth images, forcing distortion-invariant estimation.The 3D point cloud maintains a one-to-one relation with the 3D pose.
- Motivation: Prior methods commonly use 2D depth images to directly regress 3D keypoint coordinates.This treats intrinsically 3D depth data as a projected 2D image.
- Motivation: Directly mapping a 2D depth image to 3D coordinates is highly nonlinear and hampers learning.The paper identifies this as the second serious drawback of the conventional framework.
- Proposed approach: V2V-PoseNet converts depth maps to a 3D voxelized grid and estimates per-voxel likelihoods for each keypoint.This casts pose estimation as voxel-to-voxel prediction.
- Results: The method achieves state-of-the-art performance on almost all public datasets while running in real time.The evaluation covers three 3D hand and one 3D human pose datasets.
2. Related works
Related work spans generative, discriminative, hybrid, and volumetric approaches to depth-based pose estimation. V2V-PoseNet differs by combining voxelized input with voxelized keypoint-likelihood output through a 3D fully convolutional network.
- Existing approaches: Depth-based hand and human pose methods include generative models, discriminative models, and hybrids.Generative approaches fit predefined models, while discriminative approaches directly estimate joint positions.
- Existing approaches: CNN-based discriminative methods replaced many hand-crafted-feature approaches in depth-based hand pose estimation.Earlier CNN work estimated 2D heatmaps for hand joints.
- Volumetric representations: Volumetric representations encode depth information in 3D voxel form for classification, retrieval, and detection.Earlier work modeled voxels as binary random variables or used volumetric input representations.
- Input and output representations: Most prior 3D pose methods combine 2D depth input with direct coordinate regression, while some use volumetric input with coordinate regression.The paper organizes these alternatives as four input-output combinations.
- Input and output representations: V2V-PoseNet uses voxelized input and per-voxel likelihood output through a 3D fully convolutional network.This corresponds to the voxel-to-voxel configuration distinguished from prior alternatives.
3. Overview of the proposed model
V2V-PoseNet reprojects a depth image into 3D, discretizes the space into voxels, and predicts a likelihood volume for every keypoint. The highest-response voxel is converted back to real-world coordinates.
- Input construction: The model first reprojects depth pixels into 3D space and discretizes the continuous space into voxels.The resulting voxelized data is the network input.
- Voxel-to-voxel prediction: V2V-PoseNet estimates a per-voxel likelihood volume for each hand or human-body keypoint.The network processes the voxelized data to localize all keypoints.
- Output decoding: The highest-likelihood response for each keypoint is warped into real-world coordinates as the final prediction.This converts volumetric responses into 3D keypoint locations.
4. Refining target object localization
The method requires a cubic box containing the target object, centered on a reference point. Because ground-truth positions are unavailable and thresholding can fail in clutter, the paper addresses reference-point errors.
- Localization prerequisite: A cubic box containing the hand or human body in 3D space is required for keypoint localization.The box is usually placed around a reference point.
- Reference point: Reference points may come from ground-truth joints or the center of mass obtained by depth thresholding.Ground-truth joint positions are infeasible in real-world applications.
- Practical limitation: Depth-thresholding center-of-mass estimates can fail in clutter because nearby objects receive the same threshold treatment.Such errors can prevent the target object from being correctly contained in the cubic box.
5. Generating input of the proposed system
The system converts a 2D depth map into a 3D voxelized representation by reprojection, discretization, and target-object extraction.
- Each depth-map pixel is reprojected into 3D space before the continuous space is discretized according to a predefined voxel size.
- The target object is extracted by drawing a cubic box around a reference point obtained earlier in the method.
6. V2V-PoseNet
V2V-PoseNet uses a 3D hourglass CNN with volumetric building blocks to predict per-keypoint voxel likelihoods, supervised by Gaussian 3D heatmaps and mean square error.
- Building block design: The network combines volumetric convolutional, residual, downsampling, and upsampling blocks with 3D kernels and stride-2 spatial changes.
- Network design: V2V-PoseNet treats the Z-axis as an additional spatial axis and performs voxel-to-voxel prediction with a modified 3D hourglass architecture.
- Network design: Encoder downsampling reduces feature-map size while residual blocks increase channels; decoder upsampling enlarges spatial size and compresses channels.
- Network training: Each keypoint is supervised with a 3D heatmap whose Gaussian peak is centered at its ground-truth joint location.
- Network training: The training loss is mean square error between the ground-truth and estimated heatmaps across all keypoints and voxels.
7. Implementation details
The model is trained end-to-end from scratch using RMSProp, an 88×88×88 input, and augmentation over rotation and scale.
- The network is trained end-to-end from scratch with zero-mean Gaussian weight initialization having σ = 0.001.
- RMSProp optimization uses mini-batches of 8 and a learning rate of 2.5×10^-4.
- The proposed system receives an 88×88×88 input and applies rotation and 3D scaling augmentation.
8. Experiment
Experiments evaluate V2V-PoseNet across three 3D hand datasets and one 3D human dataset, including representation ablations, localization refinement, comparisons, and runtime. The method achieves state-of-the-art results across the evaluated benchmarks while supporting real-time inference.
- Datasets and metrics: The evaluation covers ICVL, NYU, MSRA, HANDS 2017, and ITOP datasets, using 3D distance error, success-frame percentage, and human-pose mAP metrics.ITOP is evaluated from front and top views; mAP follows the 10 cm rule.
- Ablation study: Voxelized 3D input substantially improves performance over 2D depth input, while per-voxel likelihood output improves performance over direct coordinate regression.The voxel-to-voxel combination performs best among the four tested input-output combinations and uses the fewest parameters.
- Ablation study: Refined reference points significantly boost model accuracy, demonstrating the importance of the localization refinement procedure.The effect is measured using average 3D distance error on NYU.
- Comparison with state-of-the-art methods: V2V-PoseNet outperforms existing methods on all three evaluated 3D hand pose datasets in standard metrics and performs better across almost all MSRA yaw and pitch angles.The performance gap is largest on the challenging, unsaturated NYU dataset.
- Comparison with state-of-the-art methods: The method places first in the HANDS 2017 frame-based 3D hand pose estimation challenge and outperforms the other participants.The comparison reports the top five challenge results.
- Comparison with state-of-the-art methods: On ITOP, V2V-PoseNet outperforms existing methods by a large margin in both front-view and top-view human-pose evaluations.The result supports applying the model to 3D human pose estimation as well as hand pose estimation.
- Computational complexity: Testing runs at 3.5 fps with ten epoch-ensemble models and can reach 35 fps in a multi-GPU environment.Without epoch ensemble, the method runs in real time on a single GPU while retaining large-margin improvements on the evaluated datasets.
9. Conclusion
The conclusion presents V2V-PoseNet as a voxel-to-voxel framework for single-depth-map 3D hand and human pose estimation. It reports large-margin improvements across the evaluated datasets and first place in a 3D hand pose challenge.
- 9. Conclusion: V2V-PoseNet converts a 2D depth map into a 3D voxel representation and processes it with a 3D CNN.The conversion treats depth data volumetrically rather than as a conventional 2D image.
- 9. Conclusion: Instead of directly regressing keypoint coordinates, the network estimates per-voxel likelihoods for each keypoint.This is the paper’s voxel-to-voxel prediction formulation.
- 9. Conclusion: The two representation conversions significantly improve performance over previous approaches on three 3D hand and one 3D human pose datasets.The conclusion describes the improvements as large-margin results.
- 9. Conclusion: The system wins the 3D hand pose estimation challenge, where voxel-to-voxel prediction is introduced for single-depth-map hand and human pose estimation.The paper positions the approach as a new way to perform accurate 3D pose estimation.