PaperlayerView exact revision on arXiv

Source-linked AI summary

HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, Li Yi

arXiv:2203.01577v4cs.CV

TL;DR

Existing HOI datasets provide limited support for category-level interaction with diverse rigid and articulated objects from egocentric 4D inputs. HOI4D addresses this gap with a large, richly annotated RGB-D dataset and three benchmarks. Its analyses show substantial challenges for current methods, while fine-grained action segmentation and realistic interaction data define important research opportunities.

  • Problem

    Existing egocentric HOI datasets focus largely on known object instances, omit articulated objects, or provide mostly 2D features instead of comprehensive 3D interaction annotations.

  • Method

    HOI4D constructs a large-scale egocentric RGB-D dataset with reconstructed geometry, frame-wise interaction annotations, and benchmarks for three category-level HOI tasks.

  • Results

    HOI4D poses great challenges to existing methods across dynamic scene segmentation, category-level pose tracking, and fine-grained action segmentation.

  • Takeaways & Limitations

    HOI4D provides real-world data and benchmarks for studying category-level HOI under occlusion, diverse motion, sensor noise, and varied interaction targets.

  • Takeaways & Limitations

    HOI4D does not cover human manipulation tasks using two hands, which remain challenging for current research.

Abstract

from arXiv · show

We present HOI4D, a large-scale 4D egocentric dataset with rich annotations, to catalyze the research of category-level human-object interaction. HOI4D consists of 2.4M RGB-D egocentric video frames over 4000 sequences collected by 4 participants interacting with 800 different object instances from 16 categories over 610 different indoor rooms. Frame-wise annotations for panoptic segmentation, motion segmentation, 3D hand pose, category-level object pose and hand action have also been provided, together with reconstructed object meshes and scene point clouds. With HOI4D, we establish three benchmarking tasks to promote category-level HOI from 4D visual signals including semantic segmentation of 4D dynamic point cloud sequences, category-level object pose tracking, and egocentric action segmentation with diverse interaction targets. In-depth analysis shows HOI4D poses great challenges to existing methods and produces great research opportunities.

1. Introduction

HOI4D addresses the need for annotation-rich 4D egocentric datasets that capture category-level interactions with diverse rigid and articulated objects. It contributes a large dataset and benchmarks spanning dynamic scene understanding, pose tracking, and fine-grained action segmentation.

  • Motivation: Existing egocentric HOI datasets mainly use a small pool of known object instances and often omit articulated objects.These limitations restrict coverage of everyday object diversity and more complex interaction patterns.
  • Dataset contribution: HOI4D contains 2.4M RGB-D frames across 4000 sequences, 800 object instances, 16 categories, and 610 indoor scenes.The dataset includes interactions by 9 participants and covers both rigid and articulated objects without participant-worn markers.
  • Dataset contribution: HOI4D provides frame-wise panoptic, motion, 3D hand-pose, object-pose, and action annotations, plus reconstructed object meshes and scene point clouds.These resources support category-level human-object interaction analysis from 4D visual signals.
  • Benchmarks: The paper benchmarks 4D dynamic point-cloud segmentation, category-level object pose tracking, and egocentric hand action segmentation.The authors analyze existing methods and identify challenges posed by the dataset.

2. Related Work

Prior work provides egocentric interaction datasets, 4D scene-understanding methods, and category-level pose representations, but HOI4D combines broader category-level coverage with richer annotations. The related methods also leave challenges involving occlusion, articulated objects, indoor dynamics, and joint hand-object tracking.

  • Egocentric HOI datasets: Many egocentric HOI datasets emphasize daily-activity recognition and mostly 2D features, omitting 3D hand and object poses.The omitted poses are described as important for comprehensive interaction understanding.
  • Egocentric HOI datasets: HOI4D is presented as the first large-scale 4D egocentric category-level HOI dataset covering rigid and articulated objects with rich annotations.This distinguishes its scope from prior egocentric datasets summarized in the comparison.
  • 4D dynamic scene understanding: Indoor interactive 4D scenes introduce smaller objects, more diverse movement, and object deformation beyond challenges in outdoor point-cloud datasets.These properties create new difficulties for existing 4D dynamic scene-understanding methods.
  • Category-level pose estimation: NOCS defines a category-specific canonical reference frame by projecting object pixels into normalized 3D coordinates, while ANCSH extends this idea to articulated parts.These representations support pose definition for novel objects and parts.
  • Category-level pose estimation: Existing pose-tracking methods generally do not jointly track hand and object pose, while egocentric views intensify object occlusion.Prior datasets such as NOCS also use synthetic data and simpler scenes without hand occlusion.

3. Constructing HOI4D

HOI4D combines synchronized head-mounted RGB-D capture with a multi-stage annotation pipeline for dynamic scenes, hand pose, category-level object pose, and fine-grained actions.

  • Hardware Setup and Data Collection: The head-mounted suite combines a bicycle helmet, Kinect v2 RGB-D sensor, and Intel RealSense D455 RGB-D sensor.The two sensors are precalibrated and synchronized before capture.
  • Panoptic Annotation: 4D panoptic labeling separates moving and static content, reconstructs the static scene, and merges both segmentations into dynamic scene labels.Annotators label frame-wise motion masks, reconstruct static content with SLAM, and manually annotate the reconstructed scene.
  • Hand Pose Annotation: Hand pose annotation uses 2D keypoints, MANO-based optimization, temporal propagation, refinement, and manual correction of failure frames.The process has four stages: annotation, initialization, propagation, and refinement.
  • Category-Level Pose Annotation: Category-level object pose annotation comprises measurement and annotation, model scanning and reconstruction, and label propagation with pose optimization.Rigid objects receive 9D poses, while articulated-object parts are annotated separately and optimized using rendered-mask, depth, point-cloud, and temporal losses.
  • Action Annotation: Fine-grained action labels are directly annotated on the original videos to support temporal action segmentation in long untrimmed 4D sequences.Each frame receives an action category in the interactive scene.

4. Dataset Statistics

HOI4D spans 16 object categories and 54 interaction tasks, with diverse reconstructed instances and RGB-D recordings designed to reflect everyday functionality.

  • Category Overview: 16 common object categories comprise 7 rigid and 9 articulated categories, with 50 unique object instances per category.Each instance has a CAD model reconstructed from high-resolution RGB images.
  • Category Overview: Reconstructed meshes connect HOI4D categories to ShapeNet and Sapien Assets and can support sim-to-real transfer studies.Human hand trajectories and reconstructed meshes can potentially be used in simulation environments for robot learning.
  • Data Scale: 2.4M frames are captured at 15fps in 20-second videos using both RealSense D455 and KinectV2 sensors.Using both sensors supports studying knowledge transfer across different depth sensors.

5. Cross-Dataset Evaluations

Cross-dataset evaluations compare HOI4D with H2O and NOCS-related settings across hand pose, object pose tracking, and action segmentation, revealing substantial domain-transfer asymmetry.

  • Evaluation Settings: The evaluation covers 3D hand pose estimation, category-level object pose tracking, and action segmentation using Mesh Graphormer, CAPTRA, and ASFormer backbones.Object tracking is evaluated on the bottle category, while action segmentation uses five shared action classes.
  • 3D Hand Pose Estimation: 2.2× error increase occurs when a HOI4D-trained hand-pose model is evaluated on H2O, rising from 22.3 to 48.9.The paper attributes this increase to domain gaps.
  • 3D Hand Pose Estimation: 3.5× error increase occurs when an H2O-trained model is evaluated on HOI4D, indicating worse generalization from H2O to HOI4D.The reported errors are root-relative MPJPE in millimeters, where lower is better.
  • Cross-Dataset Findings: Training on combined HOI4D and H2O reduces error relative to training only on H2O, while increasing error on HOI4D relative to HOI4D-only training.The authors conclude that HOI4D complements H2O better than the opposite.

6. Tasks and Benchmarks

HOI4D benchmarks category-level pose tracking, 4D point-cloud semantic segmentation, and egocentric fine-grained action segmentation. Across these tasks, existing methods struggle with interaction-specific occlusion, motion, clutter, and fine-grained action distinctions.

  • Benchmark Tasks: Three benchmarks cover category-level object and part pose tracking, 4D point-cloud semantic segmentation, and egocentric hand-action segmentation.The sequences are randomly split 7:3 into training and test sets, using task-specific HOI4D annotations.
  • Category-Level Pose Tracking: BundleTrack evaluates category-level pose tracking without instance or class-level 3D models, alongside an ICP baseline.Evaluation uses perturbed ground-truth object poses for initialization and follows prior-work protocols.
  • Category-Level Pose Tracking: 19.3 is BundleTrack’s bottle accuracy at 5°5cm on HOI4D, versus 86.5 on NOCS, highlighting the difficulty of interactive pose tracking.HOI4D includes heavy hand occlusions absent from the cited NOCS setting.
  • 4D Point-Cloud Segmentation: Indoor 4D point-cloud segmentation is challenged by clutter, egocentric occlusion, 6DOF ego-motion, and sensor noise differing from LiDAR.The benchmark selects 376 videos spanning 14 semantic categories from 38 total indoor categories.
  • 4D Point-Cloud Segmentation: Existing outdoor segmentation methods perform better on background than object categories, whose smaller size, flexible motion, and severe occlusion increase difficulty.The authors identify simultaneous modeling of object geometry and background as an interesting direction.
  • Egocentric Action Segmentation: 46.8 is Asformer’s accuracy on HOI4D, compared with 85.6 on 50Salads, while failure analysis suggests models learn action order better than action identity.All three evaluated action-segmentation methods drop substantially from coarse to fine-grained labels.

7. Limitations and Future Work

HOI4D does not cover two-handed human manipulation tasks, because single-hand manipulation remains challenging and coordinated hand behavior presents additional research challenges.

  • Limitations: HOI4D excludes two-handed manipulation tasks, leaving hand cooperation as a future research direction.The authors state that single-hand manipulation remains challenging for current research.

Supplementary Material

HOI4D reconstructs realistic CAD models from multi-view, high-resolution color images captured at varied camera depression angles. The process includes object decoration to enrich texture and reduce specular regions.

  • CAD Model Reconstruction: Multi-view high-resolution images with varied depression angles provide inputs for reconstructing corresponding CAD models.The examples include both the color-image views and reconstructed models.
  • CAD Model Reconstruction: Objects are manually decorated with stickers before off-the-shelf reconstruction software produces detailed CAD models.The decoration enriches texture and hides highly specular areas; reconstruction covers inner and outer surfaces for some categories.

A.2. Dataset Statistics

The supplementary material documents HOI4D’s scene organization, task coverage, hand-pose supervision, CAD examples, contact modeling, and temporal optimization setup.

  • Dataset Statistics: Simple scenes avoid surrounding-object occlusion and limit camera-view variety, whereas complex scenes impose neither restriction.Tables report instance counts and object-point proportions for both scenario types.
  • Dataset Statistics: HOI4D defines 76 tasks across 16 categories, including universal pick-and-place and functionality-based mobility tasks.The functionality-based tasks target object mobility and functionality in interactive scenarios.
  • Hand-Pose Supervision: Hand-pose training combines joint-angle, 2D joint, mask, depth, point-cloud, contact, and temporal-consistency losses.The point-cloud term uses Chamfer distance to supervise all MANO hand vertices, while contact terms penalize attraction without contact and interpenetration.
  • CAD Models: CAD examples span all 16 HOI4D object categories.The supplementary figure presents representative reconstructed models across the category set.
  • Hand-Pose Supervision: The contact weighting coefficient λR balances attraction and repulsion penalties, with λR = 1 when action labels indicate no object interaction.Contact-mode weights are manually tuned, and temporal optimization uses batches of 6 to 11 consecutive frames.

B. Qualitative analysis of Category-Level Pose Tracking

HOI4D’s category-level interaction setting exposes pose-tracking failures during hand-object contact, while its annotations and demonstrations support broader interaction analysis and robot learning.

  • Pose-tracking analysis: Hand-object interaction is the main failure source for existing pose-tracking methods.Hand occlusion and rapid object movement increase difficulty, whereas camera motion without interaction is handled more successfully.
  • Action segmentation: HOI4D defines finer-grained action categories within interactive scenes than existing action-segmentation datasets.
  • Robot learning: HOI4D supports category-level dexterous manipulation by providing human-object trajectories and hand-object pose states as imitation-learning demonstrations.
  • Environment and learning setup: The simulated Pick Up task uses point-cloud observations, 30-joint motor commands, velocity control, and a three-stage reward design.
  • Robot learning: The manipulation pipeline retargets 51-DoF MANO hand poses to the 30-DoF Adroit Hand and derives smoothed actions from cubic-spline-fitted joint trajectories.

D.4. Results and Analysis

The Pick Up experiments compare reinforcement learning with imitation learning using HOI4D-derived demonstrations. Imitation learning performs better, but both approaches remain challenged by point-cloud observations.

  • Evaluation: Success rate is evaluated over three random seeds on the Pick Up task, with results reported in Table 10.
  • Comparison: The comparison presents reinforcement-learning and imitation-learning agents on the Pick Up task.
  • Results: Imitation learning outperforms reinforcement learning, although both methods achieve low success rates.
  • Results: Twelve demonstrations from different toy cars greatly improve task success and help reinforcement learning learn a better policy.
Loading 2203.01577v4…