Source-linked AI summary
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
Bugra Tekin, Federica Bogo, Marc Pollefeys
TL;DR
Egocentric understanding has largely treated humans and objects separately, limiting unified interpretation of hand-object interactions from ordinary RGB video. H+O jointly predicts 3D hand and object poses, object and action classes, and interactions from monocular sequences, achieving state-of-the-art results at real-time speed, including against methods using depth and ground-truth pose annotations.
Problem
Jointly understanding 3D hand-object poses and interactions from cluttered egocentric monocular RGB images has received limited attention despite its importance for semantically meaningful scene interpretation.
Method
H+O uses a unified single-pass architecture with shared features to predict 3D hand and object poses, object and action classes, and temporal interaction relations from monocular color sequences.
Results
H+O achieves state-of-the-art performance on challenging egocentric sequences at real-time speed, outperforming approaches that use depth data and ground-truth pose annotations.
Takeaways & Limitations
The framework demonstrates unified monocular RGB recognition of 3D hand-object poses, object and action categories, and interactions without external detections or ground-truth 3D poses.
Takeaways & Limitations
The framework is scoped to monocular RGB sequences and leaves explicit modeling of interactions involving two hands or other people for future work.
Abstract
from arXiv · showhide
We present a unified framework for understanding 3D hand and object interactions in raw image sequences from egocentric RGB cameras. Given a single RGB image, our model jointly estimates the 3D hand and object poses, models their interactions, and recognizes the object and action classes with a single feed-forward pass through a neural network. We propose a single architecture that does not rely on external detection algorithms but rather is trained end-to-end on single images. We further merge and propagate information in the temporal domain to infer interactions between hand and object trajectories and recognize actions. The complete model takes as input a sequence of frames and outputs per-frame 3D hand and object pose predictions along with the estimates of object and action categories for the entire sequence. We demonstrate state-of-the-art performance of our algorithm even in comparison to the approaches that work on depth data and ground-truth annotations.
1 Microsoft 2 ETH Z¨urich · 1. Introduction
H+O addresses the underexplored challenge of jointly understanding hands, objects, and their interactions from monocular egocentric RGB images. It unifies 3D pose estimation, object and action recognition, and temporal interaction reasoning in a single feed-forward framework.
- 1. Introduction: Jointly understanding humans and objects is crucial for semantically meaningful scene interpretation but has received comparatively little attention.
- 1. Introduction: Reasoning about contacting hands and objects from moving egocentric cameras is especially challenging because interactions are subtle and cluttered, with frequent occlusion.
- 1. Introduction: Existing approaches are limited by reliance on active depth sensors or costly calibrated multi-camera systems and by omitting recognition of the subject’s action.
- 1. Introduction: H+O simultaneously predicts 3D hand and object poses, object classes, and action categories from a single image without external region proposals or pre-computed detections.
- 1. Introduction: The unified feed-forward network operates on monocular color images and shares joint features across 3D hand pose, object pose, object recognition, and activity classification.
- 1. Introduction: A common 3D-control-point representation lets one single-shot architecture jointly estimate articulated hand and rigid object poses directly in 3D.
- 1. Introduction: The temporal model merges and propagates information across time to model hand-object interactions and infer their relations directly in 3D.
- 1. Introduction: The authors report better overall performance on targeted tasks at real-time speed, without detailed 3D hand and object models, and state-of-the-art results on challenging sequences.
2. Related Work
Prior work separately estimates hand or object pose, often using depth or third-person views, while egocentric action recognition addresses motion, appearance, and other cues amid substantial visual challenges. The paper identifies a gap: no prior approach jointly estimates 3D hand and object pose from RGB video only, while related scene-understanding methods remain 2D or omit temporal modeling.
- Hands and Objects: Prior methods commonly estimate either hand or object pose in isolation, with RGB object-pose systems using regression forests, CNNs, segmentation, bounding boxes, or PnP.The cited object methods recover 6D pose from single RGB images through multi-stage regression forests or CNN-based pipelines.
- Hands and Objects: Egocentric 3D hand-pose estimation is difficult because of self-occlusions and limited training data, prompting synthetic-data, generative-model, and interaction-aware approaches.Several methods exploit object interactions or grasp constraints, but most focus on third-person scenarios and use depth input.
- Hands and Objects: No approach in the literature jointly estimates 3D hand and object pose from RGB video only.This is the related-work gap targeted by the paper’s unified RGB framework.
- Action Recognition: First-person action recognition has recently expanded with egocentric datasets and wearable sensors, but fast camera motion, large occlusions, and background clutter create distinctive challenges.Existing approaches use motion, appearance, depth, gaze, head motion, and CNN-learned features.
- Action Recognition: Related structured methods analyze grasp types, object attributes, contextual relationships, or semantic object instances, but they either omit temporal modeling or understand scenes only in 2D.The paper instead models more complex hand and object attributes in 3D.
3. Method
The method unifies 3D hand-object pose estimation with object, action, and interaction recognition from egocentric RGB sequences. It combines single-frame grid-based predictions with recurrent temporal reasoning about hand-object interactions.
- Single-frame prediction: A fully convolutional network divides each frame into an H × W × D 3D grid, predicts separate hand and object values per cell, and produces outputs in one forward pass.Low-confidence cells, corresponding to absent hands or objects, are pruned at test time.
- Pose parameterization: Shared 3D control-point predictions define hand pose and enable 6D object pose recovery through rigid Procrustes alignment rather than a 2D-to-3D PnP correspondence step.This formulation reduces projection-induced depth ambiguities and improves object pose estimation accuracy.
- Confidence estimation: The network predicts cell confidence from spatial prediction error, extending confidence estimation beyond 2D locations to include metric depth.The final confidence averages the 2D and depth confidence values.
- Temporal reasoning: An LSTM models longer-term dependencies across frames and explicitly reasons about interactions between hand and object poses for activity recognition.The interaction mapping is implemented with an LSTM and an MLP; experiments with GRUs showed no substantial result differences.
4. Evaluation
The evaluation measures unified egocentric activity recognition and 3D hand-object pose estimation on FPHA-derived data using standard metrics and official splits. Results show that temporal pose reasoning and interaction modeling improve recognition, while the unified model achieves competitive pose accuracy and real-time inference.
- Evaluation Protocol: The framework is evaluated on FPHA across activity recognition, 3D hand pose estimation, and 6D object pose estimation using standard metrics and official train/test splits.FPHA contains annotations for hand pose, object pose, and action categories; its FPHA-HO subset covers four objects and ten action categories.
- Recognizing Interactions: Temporal reasoning over hand and object poses improves activity recognition, their combination further boosts accuracy, and explicit interaction modeling produces an additional improvement.The unified predictions outperform a ground-truth-pose-based state-of-the-art method in the reported FPHA-HO comparison.
- Recognizing Interactions: On the full FPHA dataset, the temporal model uses 3D hand poses together with predicted object classes and action categories because object poses are unavailable for all action categories.This design increases the descriptive power of the temporal model while retaining coverage of the full dataset.
- 3D Hand Pose Prediction: The model achieves competitive 3D hand pose accuracy from a single full color image without depth input or a known hand bounding box.Hand pose estimation is a subtask of the unified architecture, which simultaneously addresses multiple tasks.
- 6D Object Pose Prediction: Explicitly reasoning about 6D object pose in 3D improves object pose accuracy over a method based on 2D-to-3D correspondences.The evaluation compares against the state-of-the-art results of [66] using its publicly available code on FPHA.
- Runtime: 25 fps is achieved for simultaneous per-frame hand pose, object pose, object-class, and action-category prediction on an NVIDIA Tesla M40.Pose-only inference reaches 33 fps, while the interaction RNN adds virtually no time overhead and processes a sequence in an average of 0.003 seconds.
5. Conclusion
The paper presents a unified, real-time method for jointly estimating 3D hand and object poses from monocular color sequences while recognizing object and action categories.
- The Hand+Object model jointly predicts per-frame 3D hand poses, 6D object poses, object classes, and action categories.It is described as the first method to jointly recognize 3D hand and object poses from monocular color image sequences.
- The single-pass architecture jointly solves 3D articulated and rigid pose estimation and models hand-object interactions in 3D for action recognition.The unified framework is designed to run at real-time speeds.
Supplemental Material: H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions
The supplemental material details the unified single-pass network, training-image preparation, and additional qualitative and action-specific recognition results. It emphasizes robustness to occlusion and visual variation, while identifying action-specific confusions on FPHA.
- Training Images: Random PASCAL VOC backgrounds improve robustness to different backgrounds and are essential for proper generalization during synthetic-hand training.Synthetic objects with known 6D poses are also superimposed on training images to support joint pose estimation and robustness to object occlusions.
- Qualitative Results: Qualitative results show robustness to severe occlusions, rotational ambiguities, reflections, viewpoint changes, scene clutter, motion blur, self-occlusions, and complex articulations.The Hand + Object approach also produces temporally coherent estimates, although examples include failure cases.
- Recognition Accuracies Per Action: On FPHA, ‘sprinkle’, ‘give coin’ and ‘pour juice’ are easily identifiable, whereas ‘open letter’ and ‘light candle’ are commonly confused.The confusion is attributed to subtler hand poses that differ across trials.