Source-linked AI summary
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, Tae-Kyun Kim
TL;DR
The paper addresses limited evidence for recognizing dynamic first-person hand-object actions with reliable 3D hand poses, especially when hands are occluded. It introduces and evaluates a real RGB-D benchmark with 3D hand-pose annotations, and finds that hand pose is a rich cue for recognizing manipulation actions, while testing assumes annotated hand pose in part of the action-recognition experiments.
Problem
Reliable hand-pose estimators and real annotated sequences are limited for first-person hand-object actions, where objects cause severe occlusion.
Method
The paper constructs a first-person RGB-D benchmark with magnetic-sensor and inverse-kinematics hand-pose annotations, evaluates 18 action-recognition approaches, and tests hand-pose estimation under occlusion.
Results
Hand pose features are a rich source of information for recognizing manipulation actions; reducing hand-pose error by a factor of two yields a more than twofold improvement in action recognition.
Takeaways & Limitations
The benchmark and experiments support further work in action recognition, hand-pose estimation, object-pose estimation, and robotics.
Takeaways & Limitations
In the initial action-recognition experiments, hand pose is assumed given from magnetic-sensor and inverse-kinematics annotations.
Abstract
from arXiv · showhide
In this work we study the use of 3D hand poses to recognize first-person dynamic hand actions interacting with 3D objects. Towards this goal, we collected RGB-D video sequences comprised of more than 100K frames of 45 daily hand action categories, involving 26 different objects in several hand configurations. To obtain hand pose annotations, we used our own mo-cap system that automatically infers the 3D location of each of the 21 joints of a hand model via 6 magnetic sensors and inverse kinematics. Additionally, we recorded the 6D object poses and provide 3D object models for a subset of hand-object interaction sequences. To the best of our knowledge, this is the first benchmark that enables the study of first-person hand actions with the use of 3D hand poses. We present an extensive experimental evaluation of RGB-D and pose-based action recognition by 18 baselines/state-of-the-art approaches. The impact of using appearance features, poses, and their combinations are measured, and the different training/testing protocols are evaluated. Finally, we assess how ready the 3D hand pose estimation field is when hands are severely occluded by objects in egocentric views and its influence on action recognition. From the results, we see clear benefits of using hand pose as a cue for action recognition compared to other data modalities. Our dataset and experiments can be of interest to communities of 3D hand pose estimation, 6D object pose, and robotics as well as action recognition.
1. Introduction
The paper introduces a first-person benchmark for dynamic hand-object actions with RGB-D videos and 3D hand poses, addressing limited real-world hand-pose evidence under occlusion. It evaluates RGB-D, pose-based, and hand-pose-estimation approaches for action recognition.
- First-person hand actions matter for applications requiring high precision, including rehabilitation, virtual/augmented reality, teleoperation, and robot imitation learning.
- The benchmark contains more than 100,000 RGB-D frames with 3D hand-pose annotations obtained from six magnetic sensors and inverse kinematics.
- It captures 1,175 action samples across 45 categories, 26 objects, and 3 scenarios, with varied hand configurations and temporal dynamics.
- The dataset combines egocentric dynamic hand-object actions and poses in real videos with quality hand-pose labels.
- The study evaluates 18 RGB-D and pose-based action-recognition baselines and state-of-the-art approaches across different data modalities.
- It also evaluates a state-of-the-art hand-pose estimator in the occluded hand-object setting and assesses its influence on action recognition.
2. Related work
Prior work uses hands, depth, and pose-related representations for action recognition, but existing hand representations and datasets remain limited for dynamic real-world hand-object actions. This paper targets those limitations with real egocentric RGB-D sequences and 3D hand poses.
- Earlier first-person action-recognition methods use hand silhouettes or discrete grasp classifications derived from low-level image features.
- Existing hand representations based on low-level features or grasp taxonomies are limited compared with 3D hand-pose sequences in real videos.
- RGB-D action-recognition methods address noisy, textureless, and discontinuous depth images with depth-tailored features and pose-based approaches.
- Pose-based action recognition includes temporal state-space models, key-poses, hand-crafted pose features, and recurrent models, sometimes combined with other data streams.
- Accurate 3D hand-pose annotations remain difficult, motivating synthetic, manual, or semi-automatic datasets that can have unrealistic images, few samples, or inconsistent annotations.
- Progress in 3D hand-pose estimation has been stronger for object-less third-person views than for first-person views.
3. Daily hand-object actions dataset
The dataset contains diverse first-person hand-object actions with accurate 3D hand-pose annotations, realistic variation, and severe viewpoint- and object-induced occlusion. It also provides 6D object poses and meshes for a subset of interactions.
- Object annotations: 6D object poses, 3D locations, angles, and mesh models are provided for four objects spanning 10 action categories.The objects are milk bottle, salt, juice carton, and liquid soap.
- Action taxonomy: The 45 actions span diverse hand configurations and temporal dynamics, with objects linked to between one and four actions.Examples include spoon actions such as stir, sprinkle, scoop, and put sugar, while pen is linked to write.
- Data acquisition: Six magnetic sensors on five fingertips and one wrist capture hand pose, while an RGB-D camera records color and depth at 30 fps.The camera provides 1920×1080 color and 640×480 depth streams; inverse kinematics is used to obtain hand-pose annotations.
- Dataset statistics: The dataset includes 34 grasp types, averaging 26.11 sequences per action class and 45.19 sequences per object.The grasp taxonomy includes precision and power grasps for prismatic, round, flat, and deformable object attributes.
- Occlusion and viewpoints: Hand viewpoints are more prone to self-occlusion than typical third-person views, and actions average only 10 visible joints out of 21.The viewpoint is defined as the angle between camera direction and palm direction.
- Benchmark position: The benchmark is presented as the first first-person dataset combining hand-pose and action annotations in real data rather than synthetic data.Prior action-recognition datasets lacked hand-pose labels, while pose and grasp datasets lacked dynamic actions or used synthetic or rough manual annotations.
4. Evaluated algorithms and baselines
The evaluation covers representative RGB, depth, pose, and hybrid action-recognition approaches, alongside a frame-based hand-pose estimator tested under hand-object occlusion. Pose methods include recurrent, descriptor-based, temporal, and heterogeneous-feature approaches.
- Evaluation scope: The benchmark evaluates RGB-D and pose-based action-recognition methods, with two RGB methods included for comparison.The selected approaches are intended to cover representative trends in methodology and data modality.
- RGB baseline: A fine-tuned two-stream ConvNet provides the RGB baseline by learning descriptors from color and motion flow.This baseline follows successful RGB action-recognition approaches using ConvNets.
- Depth methods: Depth baselines include local HOG2 and HON4D descriptors plus a global view-invariant ConvNet approach using synthesized depth views.The local descriptors exploit gradient and surface-normal information, while the global approach learns from multiple synthesized views.
- Pose methods: Pose recognition methods include an LSTM baseline, Moving Pose, Lie-group pose representations, HBRNN, Gram Matrix, and TF.These methods encode motion, learn temporal dependencies, model hierarchical recurrent features, or represent static poses and transitions.
- Hybrid method: JOULE jointly learns heterogeneous features from color, depth, and hand pose using an iterative algorithm.It is the evaluation’s hybrid approach combining all available data channels.
- Hand-pose estimation: Hand-pose estimation uses a discriminative frame-by-frame ConvNet selected for interpretability and reported cross-benchmark performance.The estimator requires no initialization or manual recovery when tracking fails.
5. Benchmark evaluation results
The benchmark evaluates pose-based and RGB-D action recognition under multiple protocols and modalities, finding that hand pose is especially informative while estimated-pose quality affects recognition.
- Experimental setup: The experiments assume annotated hand poses for action recognition, with estimated-pose evaluation conducted separately.Pose annotations came from magnetic sensors and inverse kinematics.
- Training and testing protocols: Three sequence-level training:testing ratios—1:3, 1:1, and 3:1—and six-fold leave-one-person-out validation were evaluated.The main experiments use the 1:1 split, with 600 training and 575 testing sequences.
- Training and testing protocols: Cross-person validation performs worst, consistent with differences in hand action styles between subjects.The paper therefore uses the 1:1 setting for subsequent experiments.
- State-of-the-art evaluation: Hand-pose methods achieve the best performance overall, with Gram Matrix and Lie group performing particularly well.Two-stream performs well when combining spatial and temporal cues, while depth methods perform slightly worse.
- State-of-the-art evaluation: Gram Matrix outperforms other representative methods on most Figure 5 measures, especially top-k action retrieval.Two-stream is stronger for some large-object, low-motion actions, while object retrieval can succeed despite missed temporal dynamics.
- Data modalities: Hand-pose features are the most discriminative modality, while combining them with RGB and depth improves performance through complementary information.Object pose and hand pose are likewise reported as complementary and useful on the annotated subset.
- Hand pose estimation: Estimated hand poses generalize to unseen subjects but struggle with unseen objects, although qualitative estimates are good enough for action recognition.The results suggest that object shape and associated grasp are important for training pose estimators.
- Hand pose estimation: Reducing hand-pose error by a factor of two yields more than a twofold action-recognition improvement, while estimated poses differ from labels by 6.67% in recognition.Lie group reaches 69.22%, Gram Matrix 32.22%, and the LSTM baseline is more robust to noisy estimates.
6. Concluding remarks
The paper introduces a first-person benchmark connecting RGB-D action recognition with 3D hand pose and partial 6D object pose annotations. Its evaluations identify hand pose as a rich information source and support further work across related fields.
- Benchmark contributions: The benchmark provides temporal action labels and full 3D hand-pose labels, plus 6D object-pose labels for part of the dataset.It is presented as a first attempt to relate first-person hand action recognition and 3D hand pose.
- Conclusions: Experiments across RGB-D and pose-based first-person hand-action recognition identify hand-pose features as a rich source of information for manipulation actions.The benchmark is intended to encourage work in action, hand-pose, object-pose, and joint hand-object pose estimation.