PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Hands as Probes for Interactive Object Understanding

Mohit Goyal, Sahil Modi, Rishabh Goyal, Saurabh Gupta

arXiv:2112.09120v2cs.CVcs.AIcs.LGcs.RO

TL;DR

Interactive object understanding asks what can be done to objects and how, but large-scale labeled datasets are absent and manual annotation is challenging. The paper learns from hands in egocentric videos, using them to supervise state-sensitive features and masked-context predictions of object affordances. It reports that hand observations provide information for learning about objects and how to interact with them, while noting constraints from its models and evaluation scope.

  • Problem

    Interactive object understanding requires learning object states, plausible interaction sites, and suitable interactions, but large-scale labeled datasets are absent and manual annotation is challenging.

    Source: §1 · p. 1

  • Method

    The paper uses egocentric hand observations to learn state-sensitive object features and predicts interaction locations and grasps from context with the hand masked out.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The authors report that observing hands, even through relatively crude 2D boxes, provides a rich source of information for learning about objects and how to interact with them.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The learned state-sensitive features and object affordances could aid robot policy learning, including through manipulation rewards and exploration priors.

    Source: §1 · pp. 1–2

    The approach relies on off-the-shelf models and is limited by their output quality; ACP's lack of access to prediction-region pixels also reduces localization, and finer-grained evaluation data are needed.

    Source: §5 · p. 8

Abstract

from arXiv · show

Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the EPIC-KITCHENS dataset, and successfully learn state-sensitive features, and object affordances (regions of interaction and afforded grasps), purely by observing hands in egocentric videos.

1. Introduction

The paper seeks interactive object understanding: not just recognizing objects, but learning their states, interaction sites, and suitable grasps. It uses hands in egocentric videos to derive supervision for state-sensitive features and object affordances.

1. Introduction

  • Interactive object understanding requires identifying plausible interaction sites, how to interact at each site, and what may happen as a result.The paper frames this as learning state-sensitive features and locating object-afforded grasps.Source: §1 · p. 1
  • Large-scale labeled datasets for object states and affordances are absent, while manual annotation of contextual states and human interaction sites is challenging and potentially biased.An object can occupy several context-dependent states, and annotated actions may differ from how people use objects in daily life.Source: §1 · p. 1
  • Egocentric hands provide both data and supervision: their locations stabilize active objects and reveal interaction sites, while their actions indicate object state and how to interact.The approach learns from natural interactions rather than a complete state taxonomy or manually defined action classes.Source: §1 · p. 2
  • The approach learns state-sensitive features from object appearance and hand appearance and motion, encouraging objects paired with similar hand cues to be similar.The paper reports these features are more state-sensitive than alternate self-supervision and direct semantic supervision.Source: §1 · p. 2
  • For affordances, a context-prediction task masks the hand and predicts interaction locations and applicable grasps from surrounding context.The paper reports better recall for small sites such as knobs and handles, more specific site predictions, and successful learning of applicable grasps.Source: §1 · p. 2

2. Related Work

Related work spans hand understanding, using people as cues for scenes and objects, interactive object learning, and self-supervision. This paper builds on hand understanding while focusing on hands in unscripted egocentric video to learn fine-grained object affordances.

2. Related Work

  • Prior hand-understanding work covers grasp classification, pose, gestures, detection, reconstruction, and force estimation; this paper instead uses hand understanding to improve object understanding.The cited examples span RGB, RGB-D, egocentric video, and other sensors.Source: §2 · p. 2
  • Prior work uses people or hands as probes for scene affordances, grasp regions, object detection, manipulation policies, and egocentric pose prediction.The paper distinguishes its focus on hands in unscripted in-the-wild RGB egocentric video and fine-grained object affordances rather than scene affordances.Source: §2 · p. 2
  • Other interactive-object work uses varied supervision to learn grasping, state classifiers, interaction hotspots, action-site priors, articulation, rewards, and functional correspondences.This paper differs in using observations of hands interacting with objects in egocentric videos.Source: §2 · p. 2
  • The approach draws on self-supervised context prediction and contrastive learning, designing supervision sources for interactive object understanding in egocentric video.Figure 2 depicts the data preparation: detections are linked into paired object-hand tracks for state-sensitive learning.Source: §2 · p. 2, Figure 2 · p. 3

3. Approach

The approach uses hand-object observations in egocentric video to learn state-sensitive object features and object affordances. Temporal and hand consistency shape feature learning, while masked context prediction trains affordance predictions from surrounding image context.

3. Approach

  • Egocentric hand, object-of-interaction, contact-state, and grasp detections provide tracks for state-sensitive feature learning and per-frame examples for affordance learning.Grasp-type predictions come from a model trained on the 71-way GUN-71 dataset.Source: §3 · p. 3

3.1. State Sensitive Features via Temporal and Hand Consistency

  • Temporal SimCLR with Object-Hand Consistency (TSC+OHC) combines temporal object consistency with hand-conditioned contrastive learning to produce state-sensitive object features.Objects close in time are positives for the temporal loss; the hand loss uses corresponding hands as positives and other hands as negatives, indirectly linking objects associated with similar hand interactions.Source: §3.1 · p. 3
  • The hand representation combines appearance with motion encoded as hand-box location and scale relative to the object over three neighboring frames.A second object crop from the same track supplies a temporal augmentation.Source: §3.1 · p. 3

3.2. Object Affordances via Context Prediction

  • Affordances via Context Prediction (ACP) masks the hand and predicts interaction regions and applicable grasps from surrounding context, avoiding a model that trivially copies the visible hand.The task predicts hand location and grasp type from patches around the masked hand.Source: §3.2 · pp. 3–4
  • ACP trains on contact-state hand patches, using detections to supervise hand and optionally object segmentation and grasp-type prediction.At inference, predictions from densely sampled, masked context patches are mapped back to per-pixel interaction and grasp probabilities.Source: §3.2 · pp. 4–5
  • The contextual prediction framework can support richer hand representations, including fine-grained segmentation and 2D or 3D hand pose, given appropriate pretrained models.Source: §3.2 · p. 5

4. Experiments

The experiments evaluate state-sensitive object features and object affordances using EPIC-KITCHENS data, with separate evaluations for interaction regions and afforded grasps. Across these tasks, hand-derived supervision supports state-feature learning and region prediction, while grasp prediction improves on chance.

4. Experiments

  • Experiments evaluate state features, interaction regions, and grasps using EPIC-STATES, EPIC-ROI, and YCB-Affordance, with participant-disjoint EPIC-KITCHENS splits and novel test objects for YCB-Affordance.EPIC-STATES contains 14,346 boxes across 10 binary state labels; EPIC-ROI contains 103 pixel-annotated images and is used only for evaluation.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 7

4.1. State Sensitive Features for Objects

  • 84.8% mAP for TSC+OHC exceeds ImageNet pre-training’s 83.1%, with larger gains of 4.4% in low-data and 9.5% on novel categories.The advantage over ImageNet features is larger in the more challenging settings.Source: Table 1 · p. 6, §4.1 · p. 6
  • TSC and TSC+OHC outperform action-classification features by 3 to 10%, and hand motion contributes more than appearance in the ablations.Both hand appearance and motion improve on TSC alone.Source: §4.1 · p. 6

4.2. Regions of Interaction

  • ACP combined with Mask RCNN achieves the strongest region-of-interaction performance across categories, improving over the Mask RCNN-based method by 4.7%.ACP alone uses no segmentation masks and approaches Mask RCNN particularly with 1% SLACK; Mask RCNN performs less well on non-COCO objects and parts.Source: §4.2 · p. 7, Table 2 · p. 8

4.3. Hand Grasps Afforded by Objects

  • 38.1% mAP on YCB-Affordance exceeds chance performance of 30.2% but remains below the supervised result of 56.8%.The method reduces the chance-to-supervised gap by 30%; temporal adaptation of the hand-grasp classifier raises performance from 34.3% to 38.1%.Source: §4.3 · p. 8

5. Discussion and Limitations

The approach extracts object understanding from hand understanding and demonstrates that even 2D hand boxes provide useful information. Its predictions are constrained by model quality and by limitations in localization, evaluation data, and dataset scope.

5. Discussion and Limitations

  • Off-the-shelf models generate the hand data and supervision, so the approach is limited by the quality of their output.The paper notes that richer hand segmentation, pose, or 3D reconstruction could support richer object understanding.Source: §5 · p. 8
  • Because ACP does not inspect the pixels where it predicts, its predictions are not as well-localized.EPIC-ROI requires finer reasoning for large objects such as microwaves; small-object annotations are more subjective.Source: §5 · p. 8
  • EPIC-KITCHENS limits the data scope and carries biases in object presence, appearance, and usage; fine-grained small-object and large-scale in-the-wild grasp evaluations are future needs.The paper also notes that its object-understanding aspects were studied in isolation and that joint formulation could do better.Source: §5 · p. 8

S6.1. Model Details

The supplement specifies baseline comparisons and implementation choices for temporal and hand-aware contrastive models. TSC+OHC uses separate object and hand models, with temporal and hand losses trained on sampled tracks.

S6.1.1 Baseline Models

  • Six baselines cover ImageNet-only features, self-supervision, action classification, and MIT States supervision; all use ImageNet initialization.The six are ImageNet pre-training, SimCLR, TCN, SimCLR+TCN, EPIC-KITCHENS action classification, and MIT States supervision.Source: S6.1.1 Baseline Models · p. 12
  • All models use ResNet-18; downstream EPIC-STATES and EPIC-KITCHENS state classifiers use only the learned backbone features.Self-supervised models use three projection layers, while semantically supervised baselines use one linear classifier during training.Source: S6.1.1 Baseline Models · p. 12
  • Baseline pretraining uses batch size 256 and Adam at 10^-4, with 200 epochs for models other than the two supervised baselines.The final checkpoint is selected, avoiding dependence on the pretraining validation set.Source: S6.1.1 Baseline Models · p. 12

S6.1.2 TSC and TSC+OHC Models

  • TSC samples a positive object crop within one quarter of the track length; TSC+OHC adds separate object and hand models and uses independent tracks for temporal and hand losses.The TSC+OHC hand model encodes motion, while evaluation retains only the object model; tracks with fewer than four hand frames are filtered out.Source: S6.1.2 TSC and TSC+OHC Models · p. 13
  • TSC+OHC uses an object-model head for temporal similarity and another for object-hand similarity, while the hand model has one head and motion positional encodings.The positional encodings combine alternating sines and cosines over 12 frequencies; horizontal flips are omitted to preserve handedness.Source: S6.1.2 TSC and TSC+OHC Models · p. 13

S6.2. EPIC-STATES Dataset and State Classification Task

EPIC-STATES adds object-state annotations to EPIC-KITCHENS interaction tracks. The supplement describes its annotation procedure, data splits and distributions, and construction of binary state-classification tasks.

S6.2.1 Data Annotation

  • EPIC-STATES annotates up to five frames from tracks in 13 selected object categories, producing 14,346 images after ambiguous examples are rejected.Each image was annotated once and reviewed by a high-quality annotator; images were labeled separately by object category.Source: S6.2.1 Data Annotation · p. 14
  • State labels are category-specific and can include multiple applicable attributes, with mutually exclusive state pairs specified for several object groups.Examples include OPEN/CLOSE for appliances, and INHAND/OUTOFHAND, RAW/COOKED, or WHOLE/CUT for applicable foods and tools.Source: S6.2.1 Data Annotation · p. 14

S6.2.2 Dataset Statistics

S6.2.3 State Classification Task

S6.3. Detailed Results and Ablations

TSC+OHC improves state classification across categories, with hand motion and appearance both contributing. For tracking, object-of-interaction tracks perform best among the compared approaches and approach hand-annotated tracks.

S6.3.1 State-wise Performance

  • TSC+OHC improves performance across state categories, and improves over TSC in all but one category for novel categories with limited data.ResNet-18 without feature learning already performs well on several states, including OPEN, RAW, COOKED, PEELED, and CUT.Source: S6.3.1 State-wise Performance · p. 17, Table S8 · p. 17

S6.3.2 TSC+OHC Ablations

  • Hand motion and appearance each improve on TSC, with motion contributing more and both together performing best for novel categories with limited data.The ablations evaluate the components of TSC+OHC on the validation set.Source: Table S9 · p. 18, S6.3.2 TSC+OHC Ablations · p. 18

S6.3.3 Track Ablations

  • The tracking study varies both the starting patch and tracking algorithm to address object selection and tracking under egomotion.The tested starting patches include arbitrary crops, background objects, and objects-of-interaction; tracking variants include no tracking, SiamRPN++, hand-context tracking, and ground-truth tracks.Source: S6.3.3 Track Ablations · p. 18
  • Object-of-interaction tracks perform best among the compared tracking approaches and come close to hand-annotated tracks.No stabilization performs poorly; SiamRPN++ stabilization works well, while hand-context tracking enables Object-Hand Consistency and ground-truth tracks yield better learning.Source: Table S10 · p. 19, S6.3.3 Track Ablations · p. 19, Figure S10 · p. 19

S7. Object Affordance Prediction

The section defines two affordance tasks: locating regions frequently touched during everyday interaction and predicting which grasps objects afford. EPIC-ROI supplies annotated interaction regions, while YCB-Affordance supports grasp evaluation on held-out objects.

S7.1. EPIC-ROI Dataset and Task

  • EPIC-ROI evaluates regions affording TAKE, OPEN, CLOSE, PRESS, DRY, TURN, or PEEL using 500 images sampled from nine EPIC-KITCHENS participants.Annotations use polygons for larger regions and dilated strips for thin regions; participant-based splits yield 32 validation frames and 71 test images.Source: S7.1. EPIC-ROI Dataset and Task · p. 20, Table S11 · p. 20
  • EPIC-ROI regions are labeled as COCO objects, non-COCO objects, COCO parts, or non-COCO parts for detailed analysis.The labels distinguish whole objects from parts and whether the object belongs to the selected COCO categories.Source: S7.1. EPIC-ROI Dataset and Task · p. 20, Figure S11 · p. 21

S7.2. Grasps Afforded by Objects (GAO) Task

  • The GAO task predicts multiple grasps afforded by each scene object while accounting for occlusion that can make some grasps inapplicable.It uses YCB-Affordance annotations built on YCB-Videos; object masks are supplied, and train, validation, and test splits do not overlap in objects or videos.Source: S7.2. Grasps Afforded by Objects (GAO)… · p. 20, Figure S12 · p. 21

S7.3. Model Details

ACP predicts interaction regions and hand grasps from masked image context. The section also describes the training data, inference procedures, and comparison baselines for the ROI and GAO tasks.

S7.3.1 Affordances via Context Prediction (ACP) Details

  • ACP trains ROI segmentation and grasp prediction jointly from context with the hand region masked, using hand- and object-centered patches for supervision and sampling.The grasp loss uses labels from a GUN-71 classifier; grasp training uses positive patches around hands, while segmentation trains on positive and negative patches.Source: S7.3.1 Affordances via Context… · p. 22
  • At inference, ACP aggregates predictions from sampled context regions into spatial ROI and grasp maps, then averages grasp scores within object masks for YCB-Affordance.EPIC-ROI predictions are smoothed with a Gaussian kernel; combining the spatial grasp and ROI predictions yields affordances.Source: S7.3.1 Affordances via Context… · p. 23

S7.3.2 Region of Interaction (RoI) Baselines

  • ROI baselines include Mask R-CNN, interaction hotspots, DeepGaze2, and SalGAN; Mask R-CNN predictions can be combined with ACP or DeepGaze2 using validation-selected weights.The relevant-category Mask R-CNN baseline excludes microwave, oven, sink, and refrigerator because predicting whole-object segmentation on them reduces validation performance for interaction-region localization.Source: S7.3.2 Region of Interaction (RoI)… · p. 23, Table S12 · p. 23

S7.3.3 Grasps Afforded by Objects (GAO) Baselines

  • GAO baselines include chance performance and a supervised ceiling trained on object-centered patches with binary grasp classifiers.The supervised ceiling uses the training split's 15 objects and 77 videos and is validated on six held-out objects.Source: S7.3.3 Grasps Afforded by Objects… · p. 24

S7.4. Detailed Results, Ablations, and Visualizations

ACP ablations find that the full set of data-preparation and supervision choices contributes to performance, with object regions particularly important for ROI prediction. Qualitative examples show localized interaction regions and plausible object-specific grasps.

S7.4.1 ACP Ablations

  • Masking the hand, using an asymmetric context window, and predicting only for hand-contact objects each contribute to full ACP performance.Removing object regions from sampling or targets causes a large ROI drop; removing hand regions causes a small ROI drop and prevents GAO training, while hand segmentation masks improve ROI performance.Source: S7.4.1 ACP Ablations · p. 24, Table S13 · p. 24

S7.4.2 GAO Category-wise Performance

S7.4.3 Qualitative Results

  • ACP localizes interaction regions such as drawer handles, knobs, and buttons, as well as interaction-specific parts such as plate edges.The visualizations are validation predictions on EPIC-ROI.Source: S7.4.3 Qualitative Results · p. 25, Figure S13 · p. 26
  • On EPIC-ROI and the GAO validation set, top grasp detections include plausible object-grasp matches, with many GAO detections marked applicable to their highlighted objects.The visualizations convert grasp-specific heatmaps into detections and color GAO frames green for correct or red for incorrect predictions.Source: S7.4.3 Qualitative Results · p. 25, Figure S14 · p. 27, Figure S15 · p. 28

Related papers

Loading 2112.09120v2…