FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images

Christian Zimmermann; Duygu Ceylan; Jimei Yang; Bryan Russell; Max Argus; Thomas Brox

Source-linked AI summary

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images

Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, Thomas Brox

arXiv:1909.04349v3Sep 10, 2019cs.CVcs.LGcs.RO

TL;DR

Single-view 3D hand estimation suffers from dataset bias because existing training data poorly represents varied real-world hands and poses. FreiHAND introduces a large-scale real multi-view dataset with pose and shape labels, annotated through iterative human-in-the-loop fitting and verification. Training on it improves cross-dataset generalization and supports monocular full hand-shape estimation.

Problem
Single-view 3D hand pose and shape estimation is ambiguous, while existing synthetic and controlled datasets provide limited real-world variation and generalize poorly across datasets.
Method
FreiHAND combines varied real-world multi-view capture with sparse annotations, deformable hand-model fitting, network bootstrapping, and iterative human verification.
Results
Training on FreiHAND improves cross-dataset generalization and enables a network to estimate full 3D hand shape from a single RGB image.
Takeaways & Limitations
FreiHAND provides a training and benchmarking dataset for single-image 3D hand pose and articulated hand-shape estimation.
Takeaways & Limitations
Synthetic hand datasets remain constrained by difficulty modeling real-image characteristics and by limited evidence for feasible, realistic hand-pose distributions.

Abstract

from arXiv · show

Estimating 3D hand pose from single RGB images is a highly ambiguous problem that relies on an unbiased training dataset. In this paper, we analyze cross-dataset generalization when training on existing datasets. We find that approaches perform well on the datasets they are trained on, but do not generalize to other datasets or in-the-wild scenarios. As a consequence, we introduce the first large-scale, multi-view hand dataset that is accompanied by both 3D hand pose and shape annotations. For annotating this real-world dataset, we propose an iterative, semi-automated `human-in-the-loop' approach, which includes hand fitting optimization to infer both the 3D pose and shape for each sample. We show that methods trained on our dataset consistently perform well when tested on other datasets. Moreover, the dataset allows us to train a network that predicts the full articulated hand shape from a single RGB image. The evaluation set can serve as a benchmark for articulated hand shape estimation.

1. Introduction

FreiHAND addresses dataset bias in single-view 3D hand estimation by combining varied real-world data with semi-automated multi-view annotation. The resulting dataset supports stronger cross-dataset generalization and monocular hand-shape estimation.

Motivation: Single-image 3D hand pose and shape estimation depends on training data, but existing synthetic and controlled real datasets have important limitations.Synthetic data struggles to model real-image characteristics and realistic hand-pose distributions, while real-data annotation is difficult and expensive.
Motivation: Existing datasets perform well on their own evaluation splits but generalize poorly to other datasets and in-the-wild scenarios, revealing dataset bias.The paper analyzes this cross-dataset failure as a central motivation for a new benchmark dataset.
Dataset and annotation: FreiHAND uses synchronized multi-view recordings, sparse 2D annotations, hand-model fitting, network bootstrapping, and iterative human verification to obtain 3D pose and shape labels.The procedure acquires 11592 annotations with moderate manual effort.
Dataset and annotation: The dataset includes 32 people, articulated hand shapes, varied poses, object interactions, compositable green-screen training images, and indoor/outdoor evaluation recordings.These design choices increase variation between samples and expose evaluation to diverse environments.
Results and impact: Training on FreiHAND improves cross-dataset generalization and enables full 3D hand-shape estimation from a single RGB image.The dataset is intended to support both training and benchmarking for articulated hand-shape estimation.

2. Related Work

Prior hand datasets use synthetic rendering, controlled capture, markers, manual annotation, or multi-view reconstruction, each with limitations. FreiHAND differs by combining real-image variation with pose and shape annotations for cross-dataset evaluation.

Dataset acquisition: Hand pose and shape estimation has motivated substantial effort to acquire datasets because data quality is crucial for task success.Existing acquisition strategies span synthetic and real-data approaches.
Real datasets: Controlled real datasets use depth or multi-view stereo, while in-the-wild model-fitting methods require semi-automated keypoint or segmentation annotations.These approaches differ in sensing and annotation requirements.
Real datasets: Markers simplify pose estimation but alter hand appearance, reducing the data’s value for training discriminative methods.Examples include infrared markers, color-coded gloves, and electrical sensing equipment.
Real datasets: Manual image annotation covers only visible regions, restricting complex self-occluded poses or the number of annotatable joints.Multi-view recording can help avoid occlusions and scale annotation.
Evaluation: Table 1 measures cross-dataset generalization using AUC of percentage of correct keypoints, with rows as training sets, columns as evaluation sets, and a final average-rank column.The HO-3D evaluation set was unavailable at submission, leaving one entry missing and other entries in that column computed on training data.
Synthetic datasets: Synthetic datasets vary rendered poses but struggle to reproduce real-image characteristics and realistic hand-pose distributions.They may therefore limit pose variety or include unrealistic poses.

3. Analysis of Existing Datasets

The analysis evaluates single-view 3D hand pose networks across existing datasets and finds strong within-dataset performance but substantial cross-dataset degradation. These results motivate FreiHAND, a real multi-view dataset designed for broader variation and cross-dataset generalization.

Evaluation setup: The study tests a state-of-the-art RGB-to-3D-keypoint network trained on each dataset and evaluated on the others.Standard dataset splits are used when available; otherwise, the data are divided 80%/20%.
Dataset comparison: Existing datasets differ in scale, subjects, capture modality, hand-object interaction, shape annotation, and label-generation method.The comparison includes both real and rendered datasets, with varied acquisition and annotation strategies.
Cross-dataset results: Each network achieves its best results on its own evaluation split, but performance drops substantially on other datasets.This pattern indicates a classical dataset bias in single-view hand pose estimation.
Cross-dataset results: GAN and FPA generalize especially poorly, with FPA affected by marker-induced appearance cues absent from other evaluation datasets.The altered appearance gives the FPA-trained network task-specific cues that do not transfer across datasets.
Dataset motivation: FreiHAND uses real images with viewpoint, pose, and object-interaction variation, achieving a substantial improvement in cross-dataset ranking.Its multi-view, human-in-the-loop annotation strategy supports 3D pose and shape labels with high fidelity.

4. FreiHAND Dataset

FreiHAND combines portable multi-view capture with sparse-annotation hand fitting and iterative human verification to produce 3D pose and shape labels at scale. Its pipeline bootstraps learned multi-view predictions while retaining manual quality control.

Dataset Capture: The dataset captures hand poses from 32 subjects performing actions with and without everyday objects using a portable multi-view setup.The setup supports indoor and outdoor capture and records hand-object interactions through varied grasping techniques.
Annotation Challenge: Manual annotation of all 21 keypoints across multiple views is prohibitively expensive and does not provide shape information by itself.Annotating one multi-view set takes about 15 minutes, while challenging self- and object-occluded poses also degrade existing 2D estimators.
Sparse-Annotation Fitting: The pipeline uses green-screen extraction, corrected segmentation masks, and six manually annotated 2D keypoints to initialize deformable hand-model fitting.The sparse annotations comprise finger tips and wrist, and segmentation-mask correction averages 12 seconds.
Sparse-Annotation Fitting: Fitting optimizes MANO parameters with 2D and 3D keypoint, segmentation, shape-prior, and pose-prior terms.The MANO parameterization includes 10 shape, 45 articulation, and 6 global translation-orientation parameters; the 2D keypoint term compares projected model keypoints with observations across views.
Multiview Prediction: MVNet aggregates features from all eight camera images in a common 3D volume and predicts keypoint locations through voxel-wise regression.A shared 2D CNN processes each view, calibration-based unprojection aligns features, and a 3D CNN localizes keypoints; confidence guides later annotation.

5. Experiments

The experiments evaluate FreiHAND for cross-dataset pose generalization, supervised monocular shape estimation, and iterative annotation quality. Results show strong cross-dataset pose accuracy, improved shape estimation over baselines, and steadily expanding data through bootstrapping.

Cross-dataset generalization: FreiHAND training uses green-screen recordings from 24 subjects, while evaluation uses data from 11 subjects across indoor and outdoor environments.Only three subjects are shared between the splits.
Cross-dataset generalization: The pose network trained on FreiHAND achieves strong accuracy across all tested datasets and ranks first for cross-dataset generalization.
3D shape estimation: Direct CNN prediction of MANO parameters outperforms constant mean-shape and keypoint-fitting baselines across all shape-estimation measures.The evaluation uses mesh error and F@5mm and F@15mm after Procrustes alignment.
Iterative labeling: Cross-dataset AUC improves monotonically across bootstrapping iterations as the dataset grows.The convergence evaluation uses RHD and PAN.
Iterative labeling: The iterative labeling process expands the dataset from 302 accepted samples to 993, 1449, 2609, and finally 4565 samples.Samples are accepted through heuristic filtering and manual verification before retraining the networks.

6. Conclusion

The paper presents FreiHAND as a large real-image dataset with hand pose and shape labels, captured using an iterative procedure. It supports improved single-image pose generalization and supervised monocular hand-shape estimation, with planned expansion toward in-the-wild benchmarking.

FreiHAND is presented as the largest available RGB dataset of real images with hand pose and shape labels.
The dataset improves generalization for single-image 3D hand pose estimation and supports supervised learning of monocular hand shape estimation.
The authors plan to extend FreiHAND into a challenging benchmark for evaluation under realistic in-the-wild conditions.

7. Cross-dataset generalization

This section details the single-view pose-estimation architecture and pretrained-network variant used to assess cross-dataset generalization. Pretraining leaves average ranks mostly unchanged, with lower in-domain and better out-of-dataset performance tending to emerge.

The single-view pose network uses an encoder-decoder architecture with skip connections and blocks defined in the supplementary architecture tables.
The network is trained for 300 k iterations with batch size 16 using Momentum optimization, lowering the learning rate after 150 k iterations.
Pretrained network: The pretrained variant uses an ImageNet-pretrained ResNet50 backbone to map RGB images directly to normalized 3D pose.
Pretrained network: Average ranks are mostly unchanged with pretraining, while performance tends to decrease on the respective evaluation set and improve on other datasets.

8. MVNet

MVNet is analyzed through alternative training losses and sparsification curves that assess whether prediction scores reflect expected error. Scorevolume loss produces more meaningful uncertainty scores than Softargmax for the labeling procedure.

Training loss: The experiments compare Scorevolume loss with Softargmax loss for training MVNet.
Training loss: Softargmax achieves better keypoint-estimation results, but its prediction scores are less meaningful as indicators of expected error.
Uncertainty analysis: Sparsification curves remove increasingly uncertain predictions and compare the resulting error with an oracle based on ground-truth error.
Uncertainty analysis: Scorevolume scores stay closer to the oracle line than Softargmax scores, motivating their use in the labeling procedure.
MVNet architecture: MVNet aggregates multi-view voxel-grid features with a 3D U-Net-like encoder-decoder and trains using Scorevolume loss.The voxel grid is formed by averaging unprojected features across camera views.

9. Extended Evaluation of iterative procedure

The iterative labeling procedure progressively improves MVNet’s 3D keypoint accuracy, whereas training only on Panoptic is insufficient. Joint training with FreiHAND produces an initial large gain, followed by further gains in later iterations.

Training MVNet on Panoptic alone provides insufficient performance for generalization to FreiHAND data.
Joint training on Panoptic and FreiHAND yields a large performance gain in the first iteration.
Each subsequent iteration of the procedure further improves MVNet’s 3D keypoint estimation accuracy.
Figure 8 reports performance using 3D PCK, with 3D AUC values shown in brackets.

10. Image Compositing

The study evaluates background compositing and image post-processing methods for improving generalization of composite images. Post-processing is most useful for networks trained from scratch, while pretrained networks gain less from elaborate processing.

The evaluated pipeline combines green-screen foregrounds with randomly sampled backgrounds using segmentation masks, then optionally applies harmonization or colorization.Colorization either hallucinates colors automatically or uses colors sampled from 20 foreground and background points.
For networks trained from scratch, every processing option provides roughly the same performance gain, while using all options jointly performs best.
For pretrained networks, green-screen training already gives good results, and elaborate post-processing adds only a minor gain beyond Cut&Paste.
The compositing algorithms require foreground segmentation masks, which are available for RHD but not generally for other datasets.RHD is therefore the reference dataset used for applying the same post-processing methods outside FreiHAND.

11. FreiHAND details

FreiHAND combines multi-view recordings, varied subjects, poses, objects, environments, and compositing-oriented training data. Its extended materials describe dataset composition, capture hardware, qualitative examples, and post-processing comparisons.

Dataset composition: The dataset records 32 people performing articulated hand actions, including American Sign Language signs, counting, and movements toward kinematic limits.
Image processing: Training networks from scratch benefits from random background insertion into green-screen recordings and from image post-processing.
Image processing: With robust pretrained weights, background removal and recombination are less important but still substantially improve results.
Image processing: Applying the same post-processing methods to RHD yields no clear improvement over training on its original unaltered images.RHD is used because it is the only referenced dataset providing the foreground segmentation masks required by these methods.
Dataset composition: Object interactions cover workshop tools, kitchen supplies, and other objects selected to span major power and precision grasp types.
Capture and examples: Each FreiHAND sample contains eight synchronized views, while recordings span indoor and outdoor evaluation environments and green-screen training captures.The capture setup uses eight color cameras arranged approximately as a one-meter cube around the hand.