Source-linked AI summary
BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis
Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, Tae-Kyun Kim
TL;DR
Existing hand-pose benchmarks are limited in scale, annotation accuracy, articulation, hand-shape variation, and viewpoint coverage. BigHand2.2M automatically annotates depth maps using six 6D magnetic sensors and inverse kinematics, producing broad natural-motion coverage. It achieves high cross-benchmark performance and improves egocentric estimation when used to train a CNN.
Problem
Existing hand-pose benchmarks are limited in scale, annotation accuracy, articulation, hand-shape variation, and viewpoint coverage.
Method
BigHand2.2M automatically annotates depth maps using six 6D magnetic sensors, inverse kinematics, and a constrained hand model.
Results
BigHand2.2M achieves high cross-benchmark performance and significantly improves egocentric hand pose estimation when used to train a CNN.
Takeaways & Limitations
The dataset provides broad hand-pose variation, accurate annotations, and 290K egocentric frames for hand pose estimation research.
Takeaways & Limitations
Performance is poorer on the MSRC test set because synthetic and real data differ in appearance, hand shape, and pose plausibility.
Abstract
from arXiv · showhide
In this paper we introduce a large-scale hand pose dataset, collected using a novel capture method. Existing datasets are either generated synthetically or captured using depth sensors: synthetic datasets exhibit a certain level of appearance difference from real depth images, and real datasets are limited in quantity and coverage, mainly due to the difficulty to annotate them. We propose a tracking system with six 6D magnetic sensors and inverse kinematics to automatically obtain 21-joints hand pose annotations of depth maps captured with minimal restriction on the range of motion. The capture protocol aims to fully cover the natural hand pose space. As shown in embedding plots, the new dataset exhibits a significantly wider and denser range of hand poses compared to existing benchmarks. Current state-of-the-art methods are evaluated on the dataset, and we demonstrate significant improvements in cross-benchmark performance. We also show significant improvements in egocentric hand pose estimation with a CNN trained on the new dataset.
1. Introduction
Existing hand-pose benchmarks are limited in scale, annotation quality, articulation, shape, and viewpoint, while manual annotation restricts dataset growth. BigHand2.2M addresses these limitations with large-scale, accurately annotated depth data and broad motion coverage.
- Existing benchmarks restrict the number of annotated images, annotation accuracy, articulation coverage, hand-shape variation, and viewpoint variation.
- Deep neural networks scale with training-dataset size, making large-scale, accurately annotated data important for advancing hand pose estimation.
- Manual annotation is labor-intensive and can produce inaccurate position labels, creating a bottleneck for large-scale benchmark construction.
- BigHand2.2M contains 2.2 million accurately annotated depth maps covering hand motions adoptable without external forces.
2. Existing benchmarks
Earlier benchmarks are constrained by dataset size, annotation quality, articulation, viewpoint, and hand-shape coverage. Their manual, semi-automatic, sensor-based, or synthetic construction methods introduce practical limits on scale, free articulation, or realism.
- Existing benchmarks range from a few hundred to tens of thousands of frames and remain limited in annotation accuracy, articulation, viewpoint, and hand shape.
- Manual annotation is labor-intensive and inaccurate, leaving datasets such as MSRA14 and Dexter 1 with only 2,400 and 2,137 frames.
- Model-based tracking can drift to incorrect poses and require manual correction, as reported for the NYU dataset.
- Some sensor-based capture methods restrict free articulation because devices such as data-gloves influence captured hand images and hand movement.
- Synthetic data can be generated without limit but differs from real data in hand characteristics, sensor noise, and pose plausibility.
3. Full hand pose annotation
The annotation system combines six 6D magnetic sensors, a 21-joint hand model, inverse kinematics, synchronized tracking, and depth capture. Physical constraints and calibration convert sensor measurements into full-joint annotations while supporting broad viewpoints and egocentric data collection.
- Hand model: The hand model represents 21 joints with 31 degrees of freedom, including global pose and finger joint angles.Each finger uses five angles, and bone lengths are measured manually for each subject.
- Annotation by inverse kinematics: Six magnetic sensors provide measurements used with inverse kinematics to infer the locations of all 21 hand joints.Five sensors attach to finger tips and one attaches to the back of the palm.
- Capture setup: The capture setup combines synchronized electromagnetic tracking units, six 6D sensors, a transmitter, and an Intel RealSense SR300 depth camera.The sensors capture location and orientation at 720fps with minimally affected depth images and hand movements.
- Annotation by inverse kinematics: The inference enforces fixed wrist-MCP relations, constant bone lengths, and coplanarity of each finger’s MCP, PIP, DIP, and TIP joints.
- Synchronization and calibration: The tracking and depth-camera coordinate systems are aligned using a perspective-n-point calibration procedure.
4. BigHand2.2M benchmark
BigHand2.2M combines broad viewpoint, articulation, and hand-shape coverage through schemed, random, and egocentric pose capture. Its embedding comparisons show wider and denser pose variation than existing benchmarks.
- Viewpoint space: Viewpoint coverage varies sensor height, subject position, arm orientation, and egocentric viewpoint across divided viewpoint regions.The third-person viewpoint hemisphere is divided into 16 regions, with random viewpoint changes within each region.
- Hand articulation space: Articulation coverage uses 32 extremal poses and captures natural transitions across 496 possible pose pairs.Each extremal pose places every finger in a maximally bent or extended position.
- Dataset composition: The benchmark comprises 1.534 million schemed frames, 375K random-pose frames, and 290K egocentric-pose frames.Egocentric recordings combine the 32 extremal poses with random movements.
- Hand articulation space: BigHand2.2M spans a wider and denser articulation and combined articulation-viewpoint space than ICVL and NYU.Figure 5 compares global viewpoint, 25D articulation, and combined coverage using t-SNE embeddings.
- Hand shape space: BigHand2.2M includes 10 hand shapes plus an additional testing shape, whereas ICVL has one, NYU two, and MSRC one synthetic shape.Figure 6 visualizes hand-shape variation using PCA of shape parameters.
5. Analysis of the state of the art
The analysis evaluates BigHand2.2M CNNs against cross-benchmark, within-dataset, and egocentric baselines. The results show strong real-dataset generalization, improved performance with more training data, and competitive egocentric estimation, with weaker transfer to synthetic MSRC data.
- Cross-benchmark performance: BigHand2.2M-trained CNNs achieve high performance across real ICVL and NYU datasets, whereas CNNs trained on existing benchmarks generalize poorly.Cross-testing shows comparable or better performance than models trained on the corresponding real benchmarks.
- Cross-benchmark performance: BigHand2.2M-trained models perform worse on synthetic MSRC than on real datasets because synthetic images differ in appearance and contain implausible poses.The paper identifies hand-shape differences, including an oversized synthetic thumb, as another source of mismatch.
- Cross-benchmark performance: Increasing BigHand2.2M training fractions from 1/16 to the full dataset improves cross-benchmark test results.The trend holds on ICVL, NYU, MSRC, and BigHand2.2M test sequences.
- State-of-the-art comparison: On NYU, the BigHand2.2M-trained CNN outperforms DeepPrior and FeedLoop and matches Hier despite never seeing NYU data.The comparison uses a common subset of 11 joint locations because the annotation schemes differ.
- State-of-the-art comparison: On ICVL, the model outperforms most compared methods, with a mean estimation error of 14mm despite annotation discrepancies.Differences in annotation accuracy and hand measurement schemes affect the reported error.
- Baselines on BigHand2.2M: On the 37K-frame new-subject sequence, the BigHand2.2M CNN outperforms FORTH and Intel generative tracking methods.On validation data, 90% of joints fall within a 5mm error bound.
- Egocentric dataset: BigHand2.2M’s 290K-frame egocentric subset supports CNN training that achieves performance competitive with third-view hand pose estimation under 10-fold cross-validation.Each fold trains on nine subjects and tests on the remaining subject.
6. Discussion and conclusion
The paper addresses the lagging development of hand-pose datasets by introducing BigHand2.2M, a real-depth benchmark with automatic magnetic tracking and inverse-kinematics annotation. It reports improved cross-benchmark performance and includes an egocentric subset for further hand-pose research.
- Discussion and conclusion: BigHand2.2M contains 2.2 million real hand depth images with automatically annotated 21-joint poses from six 6D magnetic sensors and inverse kinematics.A movement protocol was designed to capture natural hand poses, and approximately 290K frames use an egocentric view.
- Discussion and conclusion: The benchmark demonstrates significant improvements in cross-benchmark evaluations and is intended to support further hand-pose estimation research.The paper evaluates current state-of-the-art methods using the new benchmark.