PaperlayerView exact revision on arXiv

Source-linked AI summary

H2O: Two Hands Manipulating Objects for First Person Interaction Recognition

Taein Kwon, Bugra Tekin, Jan Stuhmer, Federica Bogo, Marc Pollefeys

arXiv:2104.11181v2cs.CV

TL;DR

Egocentric interaction recognition lacks rich 3D evidence for two hands manipulating objects. The paper introduces H2O and a unified RGB-based framework that jointly estimates hand–object poses and recognizes interactions, achieving better overall performance than state-of-the-art methods. The framework also establishes a strong baseline for joint pose estimation.

  • Problem

    First-person interaction datasets largely lack markerless 3D annotations for both hands and manipulated objects, although 3D hand–object motion is central to understanding egocentric interactions.

  • Method

    The paper constructs H2O and uses an RGB-based topology-aware graph convolutional framework to jointly estimate two-hand 3D poses, object 6D poses, and interactions.

  • Results

    The framework establishes a strong baseline for joint hand–object pose estimation and achieves better overall interaction-recognition performance than state-of-the-art methods.

  • Takeaways & Limitations

    H2O enables comprehensive egocentric scene interpretations using coordinated annotations of hand–object poses, shapes, actions, cameras, and scene structure.

  • Takeaways & Limitations

    Camera-pose smoothing assumes that the head moves with uniform speed.

Abstract

from arXiv · show

We present a comprehensive framework for egocentric interaction recognition using markerless 3D annotations of two hands manipulating objects. To this end, we propose a method to create a unified dataset for egocentric 3D interaction recognition. Our method produces annotations of the 3D pose of two hands and the 6D pose of the manipulated objects, along with their interaction labels for each frame. Our dataset, called H2O (2 Hands and Objects), provides synchronized multi-view RGB-D images, interaction labels, object classes, ground-truth 3D poses for left & right hands, 6D object poses, ground-truth camera poses, object meshes and scene point clouds. To the best of our knowledge, this is the first benchmark that enables the study of first-person actions with the use of the pose of both left and right hands manipulating objects and presents an unprecedented level of detail for egocentric 3D interaction recognition. We further propose the method to predict interaction classes by estimating the 3D pose of two hands and the 6D pose of the manipulated objects, jointly from RGB images. Our method models both inter- and intra-dependencies between both hands and objects by learning the topology of a graph convolutional network that predicts interactions. We show that our method facilitated by this dataset establishes a strong baseline for joint hand-object pose estimation and achieves state-of-the-art accuracy for first person interaction recognition.

1. Introduction

First-person interaction recognition remains limited by scarce 3D, two-hand datasets, despite the importance of hand–object motion. H2O addresses this gap with richly annotated egocentric data and unified recognition methods.

  • First-person action recognition trails third-person recognition partly because large, diverse egocentric datasets are scarce.
  • The dataset includes synchronized RGB-D images, camera poses, object meshes, scene point clouds, and action labels for comprehensive egocentric scene interpretation.
  • Existing datasets often provide only 2D annotations or focus on single-hand manipulation, limiting 3D reasoning about two-hand interactions and mutual occlusions.
  • The proposed framework jointly predicts two-hand and object poses from RGB images and recognizes interactions using a topology-aware graph convolutional network.
  • H2O provides markerless 3D poses for both hands and 6D poses for manipulated objects, together with interaction labels and rich scene annotations.
  • The method models interdependencies among hands and objects and reports better overall interaction-recognition performance than state-of-the-art methods.

2. Related Work

Prior egocentric datasets largely provide 2D or single-hand information, whereas H2O targets real multi-view RGB-D interaction recognition with two-hand and object 3D annotations.

  • Existing egocentric datasets commonly provide 2D features and limited multi-view data without hand or object poses.
  • Prior hand-pose datasets generally omit object interactions, while interaction datasets often lack corresponding images or use synthetic data.
  • Many existing interaction datasets focus on single-hand manipulation, despite the frequent involvement of two hands in hand-object interactions.
  • H2O is the first dataset combining real multi-view RGB-D data with accurate 3D poses of two hands, object pose, and action labels.
  • 2D annotations are insufficient for the 3D reasoning needed to resolve intricate hand-object interactions and mutual occlusions.
  • First-person interaction recognition remains challenging because of large occlusions, fast camera motion, background clutter, and limited data.

3. Annotation Method

The annotation pipeline combines synchronized multi-view RGB-D capture, camera calibration, object reconstruction and tracking, and semi-automated two-hand pose fitting followed by correction and smoothing.

  • Five synchronized RGB-D cameras capture egocentric and complementary static views for semi-automated hand and object pose annotation.
  • Camera extrinsics are estimated from nine IR reflective spheres whose 3D locations are solved with PnP and tracked across frames.
  • Object meshes are reconstructed from handheld RGB-D scans using BADSLAM, then used with Mask R-CNN and DenseFusion for object pose estimation.
  • Hand tracking fits the MANO model using 2D and 3D joint errors, silhouette and surface terms, pose regularization, and physical constraints.
  • Subject-specific hand shape is optimized on one frame, after which fixed shape parameters and pose parameters are optimized across both-hand sequences.
  • The pipeline removes inaccurate poses and applies Kalman filtering to smooth and interpolate the final annotations.

4. The H2O Dataset

H2O is an indoor egocentric dataset of two-handed object interactions, combining synchronized multi-camera RGB-D frames with diverse action and pose annotations.

  • H2O contains 571,645 RGB-D frames from four participants performing 36 action classes with eight objects across three environments.
  • The dataset annotates left- and right-hand pose, 6D object pose, camera pose, and action labels, while also providing MANO hand data.
  • Five Azure Kinect cameras acquire synchronized RGB and depth images, using four static views alongside one head-mounted egocentric view.
  • The capture setup includes action-instance and frame-count distributions, complementary static views, and a scene point cloud from multi-view data.
  • The subject-based split assigns 344,645 frames to training, 73,380 to validation, and 153,620 to testing.

5. Recognizing 3D Hand-Object Interactions

The framework jointly estimates two-hand 3D poses and manipulated-object 6D poses, then recognizes interactions using a topology-aware graph convolutional network.

  • Pose Prediction: Each RGB frame predicts left-hand, right-hand, and object poses together with confidence values.Hand poses use 3D joint coordinates, while object pose uses 3D bounding-box control points aligned by a rigid transformation.
  • Interaction Recognition: The interaction recognizer uses a spatiotemporal graph to encode spatial and temporal information across hands and objects.The graph represents both intra-graph dependencies and inter-graph dependencies between the two hands and the object.
  • Interaction Recognition: Parameterized intra- and inter-graph adjacency matrices learn interaction-specific graph topology and edge weights from training data.This design adaptively models dependencies within each hand or object and between left hand, right hand, and object.
  • Interaction Recognition: The model replaces fixed attention masking with additive learned adjacency terms so interaction-relevant edges can contribute even without fixed connections.The learned matrices can attribute greater importance to edges between hands and objects involved in interaction.
  • Interaction Recognition: Stacking ten layers produces the topology-aware graph convolutional network used for 3D interaction recognition.The predicted two-hand and object poses provide the initial feature map for modeling hand-object interactions.

6. Evaluation

The evaluation verifies annotation precision, demonstrates joint hand-object pose estimation, and shows that pose-aware interaction recognition benefits from complementary modalities and learned interdependencies.

  • 6.1. Dataset Analysis: Annotation errors for both hands and the object are approximately within 1 cm, despite mutual occlusions from two-hand manipulation.The verification compares the provided annotations with manually created 3D annotations from 500 images across five camera views.
  • 6.1. Dataset Analysis: H2O’s precise poses and meshes support hand-object contact maps that identify contact hotspots on both hand and object surfaces.For each hand-mesh vertex, nearby object vertices within a threshold such as 2 cm are counted and normalized.
  • 6.2. Experimental Results: The method reliably predicts both hand poses and manipulated-object poses, providing a strong baseline for joint pose estimation from a single RGB image.Hand accuracy is evaluated with 3D PCK, while object accuracy uses 2D reprojection and ADD metrics.
  • 6.2. Experimental Results: Combining left- and right-hand poses, or hand and object poses, significantly improves interaction-recognition scores.The experiments describe these modalities as complementary inputs for action recognition.
  • 6.2. Experimental Results: Modeling interdependencies between both hands and objects boosts interaction-recognition accuracy, while combining all graph terms gives the best results among tested baselines.The learned connections emphasize interaction-relevant relationships in the graph.

7. Conclusion

The paper introduces H2O and a joint 3D hand-object interaction framework for first-person action recognition, reporting state-of-the-art accuracy.

  • Conclusion: H2O provides action labels, object classes, two-hand 3D poses, 6D object poses, camera poses, scene point clouds, and object meshes.These annotations support comprehensive egocentric scene interpretations.
  • Conclusion: The proposed method jointly recognizes two-hand and manipulated-object 3D poses and uses a topology-aware graph convolutional network for interaction recognition.The framework models hand-object interactions in 3D from first-person views.
  • Conclusion: The framework yields state-of-the-art accuracy for first-person interaction recognition.The conclusion positions the dataset and experiments as relevant to 3D pose estimation, hand-object interaction, robotics, and action recognition.

Supplementary Material: H2O: Two Hands Manipulating Objects for First Person Interaction Recognition

The supplementary material adds analyses of annotation and model choices, implementation details, evaluation procedures, and qualitative results.

  • Supplementary Material: The supplement analyzes the annotation method and evaluates different error and regularization terms.It also describes preparation of training images for object pose estimation.
  • Supplementary Material: It provides implementation details, evaluation metrics, and further analyses for joint pose estimation and interaction recognition.The supplement also presents additional qualitative results.

S.1. Analysis of the Annotation Method

The annotation pipeline combines image-derived hand and object cues with geometric and physical regularization, then corrects optimized trajectories to produce accurate hand–object annotations.

  • Loss design: The joint loss combines silhouette, 2D and 3D joint, physical-constraint, and joint-angle terms to optimize hand and object poses.The physical terms discourage hand–object interpenetration and encourage surface contact, while angle limits penalize unrealistic poses.
  • Image cues: Hand silhouettes are initialized from OpenPose joints using GrabCut and merged with object masks for multi-camera silhouette supervision.The silhouette error compares projected hand-mesh vertices with the merged hand–object mask for each camera.
  • Pose priors: Hand pose regularization models MANO pose distributions as multivariate Gaussians and penalizes Mahalanobis distance from those distributions.This pose prior is applied to both left and right hand poses.
  • Trajectory correction: After optimization, manually selected keyframes are used to interpolate small hand- and object-pose errors and create smoother trajectories.The number of selected and interpolated keyframes is reported in Table S3.
  • Hand representation: The method maps MANO joints to the OpenPose skeleton by reordering joints and adding wrist and fingertip locations from the MANO mesh.This provides the hand-joint representation used by the annotation pipeline.
  • Object initialization: Object pose annotations are bootstrapped with DenseFusion from multi-view RGB-D images and segmentation masks, then refined using ICP.Synthetic images with known 6D object poses expand training variation before per-view pose selection and refinement.

S.2. Analysis of Pose and Interaction Recognition

The recognition system predicts hand and object poses from RGB inputs and feeds their structured keypoints into a topology-aware graph network for interaction classification. Qualitative and confusion-matrix results indicate accurate joint estimation and class discrimination, while motion blur, reflection, occlusion, and cross-dataset differences remain visible challenges.

  • Pose prediction: The pose-prediction network uses a YOLOv2 backbone with a 3D 13×13×5 output grid and image augmentation during training.The grid uses 32×32-pixel image cells and 15 cm depth cells.
  • Interaction recognition: Data-dependent adjacency matrices are formed from two 1×1 convolution outputs and normalized with softmax to model interaction structure.This design complements the graph network’s modeling of hand–object dependencies.
  • Interaction recognition: The TA-GCN represents two hands and objects with 51 keypoints and processes clips as 3 × 200 × 51 feature tensors.The representation contains 21 hand keypoints and 9 object keypoints, with sampling or looping used to obtain 200 frames.
  • Limitations: Failure cases arise from motion blur, reflection, and occlusion, while cross-dataset evaluation is affected by camera differences and magnetic-sensor artifacts.The latter artifacts particularly harm right-hand-pose generalization on FPHA.
  • Interaction recognition: The model stacks 10 TA-GCN blocks followed by average pooling and a fully connected action-classification layer.Training uses SGD with momentum, dropout, and scheduled learning-rate reductions.
  • Evaluation: The evaluation uses 3D PCK for hand poses and projection or ADD-based criteria for 6D object poses.Hand correctness is based on mean joint distance without rigid alignment, while object correctness uses projection error or average model-vertex distance.
  • Recognition results: A strong confusion-matrix diagonal indicates that the model distinguishes interaction classes with high accuracy.The supplied qualitative figures also show joint predictions for two hands, objects, actions, and object classes.
  • Pose results: Average keypoint errors are reported for 21 hand joints and 21 object bounding-box points, supporting the method as a strong joint pose-estimation baseline.Additional qualitative results visualize the predicted hand and object poses.
Loading 2104.11181v2…