Source-linked AI summary
Grounded Human-Object Interaction Hotspots from Video
Tushar Nagarajan, Christoph Feichtenhofer, Kristen Grauman
TL;DR
Existing affordance methods require heavy supervision or sensing and may not reflect real, dynamic object use. This paper learns interaction hotspots from human-object videos by recognizing active actions and anticipating their correspondence to inactive objects. The resulting weakly supervised maps are competitive with strongly supervised methods, generalize to unseen object classes, and encode functional similarity beyond appearance.
Problem
Affordance methods often rely on manual masks, narrow object vocabularies, artificial settings, or specialized sensing rather than real human-object interactions.
Method
The framework combines video action recognition, an anticipation model, and feature localization to infer interaction hotspots from RGB videos using coarse action and object labels.
Results
Weakly supervised hotspots outperform prior weakly supervised approaches by up to 25% relative improvement, compete with strongly supervised methods, and anticipate affordances for unseen object classes.
Takeaways & Limitations
Video-grounded hotspots connect inactive objects to both the actions they afford and the spatial ways people interact with them, while encoding functional similarity beyond appearance.
Takeaways & Limitations
Evaluation is constrained by limited vocabulary overlap between existing affordance-segmentation datasets and the OPRA/EPIC-Kitchens video classes.
Abstract
from arXiv · showhide
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object interaction "hotspots" directly from video. Rather than treat affordances as a manually supervised semantic segmentation task, our approach learns about interactions by watching videos of real human behavior and anticipating afforded actions. Given a novel image or video, our model infers a spatial hotspot map indicating how an object would be manipulated in a potential interaction-- even if the object is currently at rest. Through results with both first and third person video, we show the value of grounding affordances in real human-object interactions. Not only are our weakly supervised hotspots competitive with strongly supervised affordance methods, but they can also anticipate object interaction for novel object categories.
1. Introduction
Existing affordance methods rely on manual annotations, artificial settings, or specialized sensing, limiting their grounding in real object use. The proposed approach learns interaction hotspots directly from natural human-object videos, anticipates interactions for objects at rest, and performs competitively with supervised methods while generalizing to unseen object classes.
- Motivation: Visual recognition systems identify object appearance but do not explain how objects function or can be used by embodied agents.Affordances describe objects’ action possibilities, such as sitting on a chair or cleaning with a broom.
- Motivation: Object affordances require spatially specific interaction knowledge, including where and how an object should be grasped or manipulated.Knowing that a broom affords sweeping does not specify the interaction points or manipulation needed to achieve a goal.
- Limitations of prior work: Traditional affordance segmentation is detached from dynamic, multimodal interactions and depends on costly manual supervision, often in artificial tabletop environments.Such labeling captures annotators’ expectations and restricts scalability and object coverage.
- Approach: The method learns spatial interaction hotspots from natural human-object videos without keypoint, mask, pose, or 3D reconstruction supervision.Coarse action labels guide a framework combining action recognition, anticipation, and feature localization.
- Approach: An action model first recognizes afforded actions in active videos, while an anticipation model transfers that information to inactive-object images.This enables predictions such as identifying a carrot as cuttable even when no hand or knife is visible.
- Results: On OPRA and EPIC-Kitchens, weakly supervised hotspots outperform prior weakly supervised approaches by up to 25% relative improvement and anticipate affordances for unseen object classes.The datasets include hundreds of object and action categories from first- and third-person viewpoints; learned representations also encode functional similarity beyond appearance.
3. Approach
The approach learns affordances from human-object interaction videos, maps inactive object images into hypothesized active states, and derives spatial interaction hotspots without region supervision. Its action-agnostic design supports hotspot generalization across object categories, while localized activation mapping improves on conventional Grad-CAM.
- Overview: The method learns interaction hotspots from video rather than manually annotated affordance segmentations, linking inactive objects to both afforded actions and how they are manipulated.It uses real human behavior and avoids keypoint or mask supervision.
- 3.1. Learning Afforded Actions from Video: A video action classifier recognizes afforded actions from frame features aggregated over time, then predicts the action with a linear classifier trained by cross-entropy loss.Each frame is encoded by a convolutional backbone, spatially pooled, and passed through a temporal aggregation module such as an LSTM.
- Generalization: The classifier is object-category-agnostic, recognizing actions across objects, which enables hotspot prediction for unfamiliar object classes and unseen action-object combinations.For example, knowledge about opening microwaves can support anticipating how to open a fridge.
- 3.2. Anticipation for Inactive Object Affordances: The anticipation module transforms an inactive object embedding into a hypothesized active-state representation, using classifier guidance and feature matching to connect static images with interactions.Training can use inactive frames before interaction or catalog photos, while auxiliary classification encourages the anticipated representation to predict the afforded action.
- 3.3. Interaction Hotspot Activation Mapping: At inference, the model predicts affordances for an inactive image and produces one hotspot heatmap per afforded action by propagating gradients through the anticipation module to the original image.The hotspot maps summarize spatial regions associated with potential object interactions rather than the hypothesized active representation itself.
- 3.3. Interaction Hotspot Activation Mapping: Reduced stride, L2 pooling, and direct gradient weighting produce more localized affordance activations than traditional action recognition with Grad-CAM.The paper attributes the richer hotspots to both anticipation distillation jointly trained with recognition and the specialized activation mapping strategy.
4. Experiments
The experiments evaluate interaction hotspots across OPRA and EPIC, comparing weakly and strongly supervised methods, testing unfamiliar objects, and examining functional representations. The model achieves stronger weakly supervised performance while using video action labels rather than manual heatmap supervision.
- Datasets: OPRA pairs product-review videos with static product images, action labels, and manually created heatmaps, while EPIC-Kitchens provides egocentric clips with action, object, and bounding-box labels.EPIC spans approximately 40k training instances, 352 objects, and 125 actions; OPRA has approximately 16k instances spanning 7 actions.
- Evaluation: The evaluation tests affordance prediction on inactive images and generalization to possible interactions for unfamiliar objects.The experiments also assess whether learned representations encode functional similarities between object classes.
- Grounded affordance prediction: Our model outperforms all other weakly supervised methods across metrics on both datasets and captures interaction-specific cues beyond generic saliency.Saliency baselines produce one action-independent importance map, whereas the proposed model predicts distinct affordances for different actions.
- Grounded affordance prediction: 25% relative improvement in KLD on OPRA versus the strongest baseline accompanies matches to a strongly supervised method on two metrics; EPIC improvements reach 7% KLD.The EPIC setting has a larger, more granular action vocabulary and less spatially distinct hotspots, which benefits redundant baseline heatmaps.
- Model analysis: The anticipation model is crucial for bridging the gap between active training videos and inactive test images, while the full design also depends on class activation mapping and gradient propagation.Simply applying Grad-CAM to an action-recognition model is insufficient, and the ablation identifies the anticipation and gradient-based components as critical.
- Generalization to novel objects: The model infers hotspots for object categories unseen during training and reorganizes representations by functional similarity rather than appearance alone.Examples include anticipating opening for cupboards and cutting for squashes, while active embeddings group objects such as fridges, cupboards, and microwaves by interaction style.
5. Conclusion
The paper concludes that interaction hotspot maps can be learned directly from videos of natural human-object interactions. These maps explain affordances, generalize to unseen objects, and encode functional rather than purely visual similarity.
- Conclusion: Interaction hotspots are characteristic object regions that anticipate and explain human-object interactions.The method learns these maps by watching people naturally interact with objects.
- Conclusion: The learned representation goes beyond appearance similarity to encode functional similarity between objects.The authors identify action anticipation and robot-object policy learning as future applications.
Supplementary Material
The supplementary material provides demonstrations, ablations, dataset annotations, implementation and architecture details, evaluation protocols, and additional hotspot examples.
- Supplementary contents: The supplement includes a video demonstration of the method on human-object interaction clips.
- Supplementary contents: It reports ablations of model components and provides details on EPIC-Kitchens annotations, anticipation loss, implementation, architecture, and evaluation.
- Supplementary contents: Additional hotspot prediction examples on OPRA and EPIC supplement Figure 4, while clustering visualizations accompany the results in Section 4.2.
S1. Interaction hotspots generated on video clips
On EPIC video clips, the model generates frame-level hotspot maps before actions occur, including for frequent actions such as cut, mix, adjust, open, and wash.
- Interaction hotspots generated on video clips: The model computes hotspots for each frame and derives maps on inactive objects before the interaction takes place.The demonstration shows five frequent actions for clarity, although the model is trained on all actions.
S2. Ablation study
The ablation study shows that higher feature resolution, L2-pooling, and the anticipation module progressively improve hotspot localization, with the full design reaching the strongest reported AUC-J.
- AUC-J rises from 0.707 to 0.766 when backbone feature resolution increases to N=28.
- The three enhancements improve performance and are essential for well-localized hotspot maps.
- AUC-J increases to 0.770 with L2-pooling, which makes spatial gradients depend on activation magnitudes.
- AUC-J further reaches 0.806 after adding the anticipation model.
- Deriving hotspots at the hypothesized output image yields 0.723 AUC-J, whereas propagating gradients through anticipation produces correctly aligned gradients and 0.806 AUC-J.
S3. Data collection setup for EPIC-Kitchens annotations
EPIC-Kitchens interaction annotations are collected from short videos by placing points where object interactions occur, then converting those points into evaluation heatmaps.
- EPIC annotations are collected for quantitative evaluation and are not used to train the model.
- The task covers the 20 most frequent verbs and 31 nouns that afford those interactions.
- Annotators watch 2–5-second object-interaction videos and place 1–2 points where the interaction is performed.
- Five unique users annotate each image; the task collects 19,800 responses from 613 workers.
- The annotation interface asks users to mark interaction keypoints on the object shown in a video.
- Each point is converted into a centered Gaussian heatmap used as ground truth, yielding localized annotations grounded in human demonstrations.
S4. Anticipation loss for EPIC-Kitchens
For EPIC-Kitchens, the anticipation loss handles appearance differences between inactive and active objects by using normalized features and a fixed-margin triplet objective.
- EPIC inactive images are cropped from frames using bounding boxes and matched to the video object by class label.
- Because inactive and active objects may look different, the method replaces direct L2 loss with an anticipation loss based on correct and incorrect object classes.
- The triplet loss uses inactive features from the correct and incorrect classes, Euclidean distance, and margin M.
- After input normalization, the margin is fixed at 0.5.
- The loss makes correct-class inactive objects anticipate active features better than incorrect classes and is less sensitive to appearance mismatches.
S5. Implementation details
The implementation uses an ImageNet-pretrained, high-resolution ResNet-50 backbone, with modified strides and dilated filters to preserve spatial scale.
- All experiments use an ImageNet-pretrained ResNet-50 modified for higher output resolution.
- The backbone output dimension increases from n = 7 to n = 28 by setting res4 and res5 strides to 1 instead of 2.
- Dilation rates of 2 in res4 and 4 in res5 preserve the original feature scale.
- Fant uses two sets of convolution-batch-normalization-ReLU blocks with 3x3 convolutions.
S6. Architecture details for supervised baselines
IMG2HEATMAP uses a fully convolutional encoder-decoder to predict affordance heatmaps, with an ImageNet-pretrained VGG16 encoder and a mirrored upsampling decoder. The model outputs one channel per action and is trained with sigmoid outputs and binary cross-entropy reconstruction loss.
- Architecture: IMG2HEATMAP is a fully convolutional encoder-decoder that predicts an affordance heatmap from an image.Its encoder uses an ImageNet-pretrained VGG16 backbone through conv5.
- Architecture: The encoder produces a 512-channel feature with spatial extent 7 before decoding.
- Architecture: The decoder mirrors the backbone, replacing max-pooling with bilinear upsampling to restore the input dimensions.
- Training objective: The output has as many channels as actions, passes through a sigmoid, and is optimized with binary cross-entropy reconstruction loss.
S7. Evaluation protocol for grounded affordance prediction
The evaluation compares predicted hotspot maps with annotated affordance heatmaps on OPRA and EPIC. Supplementary examples show multiple affordances without heatmap annotations during training, while learned representations cluster objects by function and interaction modality, with failures on small or unfamiliar objects.
- Evaluation protocol: Predicted and baseline heatmaps are evaluated against manually annotated ground-truth heatmaps from OPRA and EPIC.For each action, heatmaps from multiple instances are united into one target affordance heatmap.
- Qualitative results: The supplementary OPRA and EPIC examples show the model highlighting multiple distinct affordances without heatmap annotations during training.OPRA examples include holdable, rotatable, and pushable regions; EPIC examples include cuttable, mixable, adjustable, and openable regions.
- Qualitative results: The last four examples in each supplementary figure contain failures involving small or unfamiliar objects.
- Representation analysis: Object representations are formed by averaging instance embeddings within each class and clustering the resulting class vectors with agglomerative hierarchical clustering.L2 distance in the interaction-hotspot space measures average similarity between object representations.
- Representation analysis: Interaction-hotspot features group objects by function and interaction modality more than appearance features do.Examples include cupboards, microwaves, and fridges grouped by swinging open, and knives and scissors by cutting actions.