PaperlayerView exact revision on arXiv

Source-linked AI summary

InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image

Gyeongsik Moon, Shoou-i Yu, He Wen, Takaaki Shiratori, Kyoung Mu Lee

arXiv:2008.09309v1cs.CV

TL;DR

3D hand pose estimation has focused mainly on isolated hands, leaving interacting-hand poses insufficiently covered. The paper introduces the real RGB InterHand2.6M dataset and InterNet, which estimates single and interacting hand poses from one RGB image; experiments show interacting-hand data is essential for accurate interacting-hand estimation.

  • Problem

    Most prior 3D hand pose estimation methods address single hands, although realistic human hand postures often involve interactions with other hands, people, or objects.

  • Method

    The paper constructs InterHand2.6M and proposes InterNet to estimate handedness, 2.5D hand pose, and relative depth for single or interacting hands from one RGB image.

  • Results

    Interacting-hand training data produces significantly lower interacting-hand pose error than single-hand training data, and InterNet provides a baseline on InterHand2.6M.

  • Takeaways & Limitations

    Accurate 3D interacting-hand pose estimation requires data containing interacting-hand sequences, not only isolated single-hand examples.

Abstract

from arXiv · show

Analysis of hand-hand interactions is a crucial step towards better understanding human behavior. However, most researches in 3D hand pose estimation have focused on the isolated single hand case. Therefore, we firstly propose (1) a large-scale dataset, InterHand2.6M, and (2) a baseline network, InterNet, for 3D interacting hand pose estimation from a single RGB image. The proposed InterHand2.6M consists of \textbf{2.6M labeled single and interacting hand frames} under various poses from multiple subjects. Our InterNet simultaneously performs 3D single and interacting hand pose estimation. In our experiments, we demonstrate big gains in 3D interacting hand pose estimation accuracy when leveraging the interacting hand data in InterHand2.6M. We also report the accuracy of InterNet on InterHand2.6M, which serves as a strong baseline for this new dataset. Finally, we show 3D interacting hand pose estimation results from general images. Our code and dataset are available at https://mks0601.github.io/InterHand2.6M/.

1 Introduction

3D hand pose estimation has largely targeted isolated hands, motivating InterHand2.6M and InterNet for single-RGB-image estimation of both single and interacting hands. The dataset provides large-scale real multi-view sequences, while experiments show interacting-hand data is important for accurate interacting-hand pose estimation.

  • 3D hand pose estimation localizes hand joints in 3D space for human behavior understanding and human-computer interaction.
  • Most prior methods target cropped single-hand images, limiting coverage of realistic postures involving hand-hand or hand-object interactions.
  • InterHand2.6M is a large-scale real RGB dataset containing single and interacting hand sequences across varied poses and multiple subjects.It uses calibrated multi-view capture and semi-automatic annotation combining manual and machine labeling.
  • InterNet simultaneously estimates 3D single and interacting hand pose from one RGB image by predicting handedness, 2.5D poses, and relative depth.
  • Models trained with interacting-hand data achieve significantly lower interacting-hand pose error than models trained only on single-hand data.The comparison supports the conclusion that single-hand data alone is insufficient for accurate interacting-hand pose estimation.
  • Existing alternatives often rely on synthetic images, depth maps, multi-view inputs, limited scale, or less stable annotations, whereas InterHand2.6M uses real RGB images with broader sequence variety.

3 InterHand2.6M

InterHand2.6M is a large-scale, real RGB dataset of single and interacting hand sequences captured from multiple subjects, with multi-view calibration and semi-automatic 3D annotations.

  • Data Capture: The capture studio uses 80–140 cameras, 350–450 directional LED lights, and image resolution 4096 × 2668.The cameras operate at 30–90 fps, and calibration achieves 0.42–0.48 pixel root mean square error.
  • Data Capture: InterHand2.6M contains single and interacting hand sequences captured from 26 unique subjects across 36 recordings.The dataset includes peak-pose and range-of-motion sequences covering predefined poses and conversational gestures.
  • Annotation: Each hand is annotated with 21 keypoints, yielding 42 points for both hands, including fingertips, joint rotation centers, and wrist rotation centers.Rotation-center annotation is difficult under skin and finger occlusions or oblique viewpoints.
  • Annotation: The dataset uses a two-stage annotation procedure combining manual multi-view labeling with automatic keypoint detection and triangulation.Human annotators labeled 94,914 2D images from 9,036 time instants, producing 698,922 labeled images before machine annotation expanded coverage.
  • Release Configuration: The released data provide downsized 512×334 RGB sequences at 5 fps and 30 fps, with annotations including camera parameters, bounding boxes, handedness, and 3D joints.Reported frame numbers and experimental results use the 5 fps configuration.

4 InterNet

InterNet estimates single- and interacting-hand 3D pose from one cropped RGB image by combining handedness, 2.5D pose, and relative-depth predictions into final 3D coordinates.

  • Network Overview: InterNet takes a cropped, resized RGB image and predicts handedness, 2.5D right and left hand pose, and right-hand-relative left-hand depth.The network uses a ResNet image feature and does not normalize hand scale for 2.5D pose estimation.
  • Handedness Estimation: Handedness probabilities identify whether the right and left hands exist, allowing the network to exclude absent-hand poses during testing.The handedness branch estimates right- and left-hand existence probabilities from the image feature.
  • 2.5D Pose Estimation: The pose branches output 3D Gaussian heatmaps whose x and y coordinates are image-space and whose z coordinate is wrist-relative depth.Each heatmap voxel represents the likelihood that a joint occupies that position.
  • Relative Depth Estimation: InterNet estimates right-hand-relative left-hand depth with a 1D heatmap followed by soft-argmax rather than direct regression.The depth of each hand is defined by its root joint, and the authors report greater accuracy for this heatmap-based design.
  • Final 3D Pose: The final 3D poses are obtained by back-projecting 2.5D poses after adding absolute root depths, with an additional relative-depth offset for the left hand when both hands are present.Π denotes camera back-projection and T−1 denotes inverse affine transformation.
  • Loss Functions: Training combines handedness binary cross-entropy, 2.5D pose L2, and relative-depth L1 losses into a single objective.Losses for absent hands are set to zero where specified.

5 Implementation details

The implementation uses a ResNet-50 backbone, fixed crop and heatmap resolutions, data augmentation, and training on dataset splits distinguished by annotation source and hand type.

  • Optimization: The backbone is initialized from ImageNet-pretrained ResNet-50, while remaining weights use Gaussian initialization and Adam optimization.Training uses mini-batches of 64.
  • Dataset Splits: InterHand2.6M training, validation, and test splits distinguish human versus machine annotations and single versus interacting hands.The table uses H and M for annotation sources and SH and IH for hand configuration.
  • Input and Augmentation: Hand crops are resized to 256×256, producing 64×64 spatial heatmaps with depth discretization size D = 64.Training augmentations include translation, scaling, rotation, horizontal flipping, and color jittering.
  • Optimization: The learning rate starts at 10^-4, drops by 10× at epochs 15 and 17, and training runs for 20 epochs on four NVIDIA TitanV GPUs.

6 Experiment

The experiments evaluate hand-pose accuracy with complementary metrics, compare training-data choices, and assess InterNet on standard and interacting-hand settings. Results show interacting-hand data improves interacting-hand estimation, while the task remains substantially harder than single-hand estimation.

  • 6.1 Dataset and evaluation metric: EPE measures mean root-aligned Euclidean 3D hand-pose error in millimeters on STB and RHP.
  • 6.1 Dataset and evaluation metric: MPJPE measures root-aligned 3D joint error separately for each hand, while MRRPE measures relative root-position error.
  • 6.2 Ablation study: Interacting-hand training data produces significantly lower interacting-hand pose error than single-hand training data.The comparison uses models trained on single-hand, interacting-hand, or combined data.
  • 6.2 Ablation study: As input views increase, annotation error and standard deviation decrease, reaching 2.78 mm with all 90 views.The authors attribute this accuracy to 2D keypoint detection and multi-view triangulation.
  • 6.3 Comparison with state-of-the-art methods: InterNet outperforms prior methods on STB and RHP without using ground-truth information during inference.The network predicts per-voxel 3D heatmap likelihoods rather than directly regressing joint coordinates.
  • 6.4 Evaluation on InterHand2.6M: Interacting-hand sequences remain harder: InterNet has 16.02 mm error, compared with 8–9 mm on large-scale depth-map datasets.On Test (H+M), InterNet achieves 99.09 APh and 32.57 MRRPE.
  • 6.5 3D interacting hand pose estimation from general images: InterNet produces interacting-hand pose results on general images despite receiving 3D supervision only from InterHand2.6M.Training combines InterHand2.6M with the Tzionas et al. dataset, whose 3D supervision is unavailable.

7 Conclusion

The paper introduces InterHand2.6M and InterNet for single-image 3D interacting-hand pose estimation. It concludes that interacting-hand data is essential, while identifying interacting-hand modeling and richer 3D hand representation as ongoing challenges.

  • 7 Conclusion: InterHand2.6M is presented as the first large-scale 3D hand-pose dataset containing varied single and interacting sequences from multiple subjects.
  • 7 Conclusion: InterNet is proposed as a baseline for 3D interacting-hand pose estimation from a single RGB image.
  • 7 Conclusion: Because InterHand2.6M provides only 3D joint coordinates, fitting a 3D hand model for interacting-hand rotations and meshes is identified as future work.

A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image”

The supplementary material adds experimental results omitted from the main manuscript because of space limitations.

  • The supplementary material presents additional experimental results excluded from the main manuscript because of limited space.
  • Its purpose is to provide further evidence beyond the main paper’s reported experiments.
  • The supplement complements the main manuscript rather than introducing a new dataset or method.

8 Comparison of human and machine annotation

The paper compares human and machine-generated hand annotations by embedding 20-dimensional pose vectors with t-SNE. The representation uses predefined finger-joint degrees of freedom expressed as pitch and yaw angles.

  • 8 Comparison of human and machine annotation: Each t-SNE vector represents a 20-dimensional hand-pose vector.
  • 8 Comparison of human and machine annotation: The representation assigns two degrees of freedom to each finger root, using pitch and yaw angles.
  • 8 Comparison of human and machine annotation: Additional finger joints are represented with one pitch-angle degree of freedom each.

9 Effect of number of the available views

The study evaluates how the number of available training views affects InterNet’s 3D hand pose estimation accuracy. Using all views significantly outperforms training with only four selected views.

  • Effect of number of the available views: The experiment compares training with four views—top, frontal, right, and left—against training with all available views.The comparison reports APh, MRRPE, and MPJPE on Test (H+M).
  • Effect of number of the available views: The four-view setting uses more iterations to match the total training iterations of the all-view setting.The same RootNet trained on all views is used in both settings for fair comparison.
  • Effect of number of the available views: More available views significantly improve 3D hand pose estimation performance.The authors conclude that the large number of views in InterHand2.6M is beneficial.

11 InterHand2.6M human annotation procedure

InterHand2.6M uses a human annotation procedure that combines annotations across views with automatic geometric processing. Annotators label the same hand joints in two views, after which the tool triangulates and propagates the labels.

  • Human annotation procedure: An annotator clicks hand joint positions at the easiest camera view.The selected view is indicated by a red circle in the visualization.
  • Human annotation procedure: The annotator then clicks the corresponding hand joints in another camera view.This supplies the second-view observations used for 3D reconstruction.
  • Human annotation procedure: The annotation tool triangulates the two-view annotations in 3D and projects the resulting point to the remaining views.Projected points are shown as green circles in the figure.

12 InterHand2.6M capture studio environment

The capture setup uses a constructed multi-view studio, while the accompanying experiments evaluate InterNet across multiple training and testing splits. Reported metrics include APh, MRRPE, and MPJPE.

  • Capture studio environment: InterHand2.6M data are captured in a constructed multi-view studio.Figure 14 provides a rendering of the studio environment.
  • Capture studio environment: The experiments train InterNet on Train (H), Train (M), and Train (H+M) and evaluate it on several corresponding test sets.The reported test sets include Val (M), Test (H), Test (M), and Test (H+M).
  • Capture studio environment: The evaluation reports APh, MRRPE, and MPJPE across these training and testing configurations.Tables 7–12 organize the results by metric and split.

14 Qualitative results

Qualitative comparisons show that InterNet benefits from training on both single and interacting hand data. Severe occlusions remain a failure case for estimating 2.5D hand pose and relative depth.

  • Qualitative comparison: Training InterNet only on single-hand data produces reasonable poses for separated hands but fails on all interacting-hand sequences.The comparison is shown against training on both single and interacting hand data.
  • Qualitative comparison: InterNet trained on both single and interacting hand data is qualitatively compared with the single-hand-only model.Figure 15 presents the two training conditions side by side.
  • Failure cases: Severe occlusions cause failures in 2.5D hand pose estimation and right-hand-relative left-hand depth estimation.These failures are shown among qualitative results on Test (H+M).
Loading 2008.09309v1…