Source-linked AI summary
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
Chen Gao, Yuliang Zou, Jia-Bin Huang
TL;DR
HOI detection requires recognizing relationships between people and objects, not just locating instances. iCAN learns instance-conditioned attention to highlight informative image regions, achieving favorable performance on V-COCO and HICO-DET, including a 4.90-point gain over InteractNet on HICO-DET.
Problem
HOI detection addresses the gap between locating individual instances and recognizing how people interact with surrounding objects in a scene.
Source: Introduction · p. 2Method
iCAN uses an end-to-end instance-centric attention module that conditions dynamically highlighted image regions on each person’s or object’s appearance.
Source: Introduction · p. 2, §3.1 · p. 5Results
4.90 points absolute gain over InteractNet under HICO-DET’s full-category Default setting, with favorable performance on V-COCO and HICO-DET.
Source: §4.2 · p. 8Takeaways & Limitations
Instance-conditioned attention provides a strong approach for aggregating contextual information to recognize human-object interactions across two benchmark datasets.
Source: Introduction · p. 2, §5 · p. 10The paper identifies jointly training ResNet-50-FPN for object detection and HOI detection as an untested direction that could improve performance.
Source: §4.1 · p. 7
Abstract
from arXiv · showhide
Recent years have witnessed rapid progress in detecting and recognizing individual object instances. To understand the situation in a scene, however, computers need to recognize how humans interact with surrounding objects. In this paper, we tackle the challenging task of detecting human-object interactions (HOI). Our core idea is that the appearance of a person or an object instance contains informative cues on which relevant parts of an image to attend to for facilitating interaction prediction. To exploit these cues, we propose an instance-centric attention module that learns to dynamically highlight regions in an image conditioned on the appearance of each instance. Such an attention-based network allows us to selectively aggregate features relevant for recognizing HOIs. We validate the efficacy of the proposed network on the Verb in COCO and HICO-DET datasets and show that our approach compares favorably with the state-of-the-arts.
Introduction
The paper frames human-object interaction detection as essential for understanding what is happening in scenes, and proposes instance-conditioned attention to highlight informative regions for HOI prediction. It reports state-of-the-art results on two benchmarks alongside ablations, error analysis, and released source code.
Problem
- HOI detection extends object-instance recognition by identifying visual relationships between people and surrounding objects.The task addresses scene understanding beyond detecting individual instances.Source: Introduction · p. 2
Why HOI?
- HOI detection aims to answer “What is happening?” and provides cues for pose estimation, image captioning, and image retrieval.These applications motivate HOI as a high-level vision problem.Source: Introduction · p. 2
Why attention?
- Existing HOI methods combine person and object appearance with spatial relationships, while action recognition motivates incorporating broader contextual cues.The introduction identifies contextual feature encoding as a limitation and opportunity for HOI detection.Source: Introduction · p. 2
Our work
- The proposed end-to-end instance-centric attention module uses each human or object instance’s appearance to dynamically highlight informative image regions.For example, recognizing carrying may require attention near a person’s hands.Source: Introduction · p. 2
Our contributions
- The work claims state-of-the-art performance on two large-scale HOI benchmarks and includes ablation studies, error analysis, and source-code release.The analyses quantify component contributions and different error types.Source: Introduction · p. 3
2 Related Work
The paper situates HOI detection within object detection, visual relationship detection, and attention-based recognition, emphasizing its focus on human-object interactions. Prior HOI work includes action-specific density maps based on detected-person appearance and stronger ResNet-50 backbones.
Object detection
- The method uses off-the-shelf Faster R-CNN to localize persons and object instances before recognizing interactions between person-object pairs.Object detection is presented as an essential building block for scene understanding.Source: §2 · p. 3
Visual relationship detection
- The work addresses human-object interactions as one class of visual relationship detection problems amid broader relationship and scene-graph research.Prior methods also use language priors to address numerous subject-predicate-object triplets and limited data samples.Source: §2 · p. 3
Attention
- Earlier attention methods for action recognition and HOI often use hand-designed regions, while newer trainable approaches target image-level classification.The paper positions its approach as building upon recent attention-based methods while addressing HOI detection.Source: §2 · p. 3
Human-object interactions
- HOI detection supports finer-grained scene understanding by identifying actions, interacting objects, and their semantic roles.Associating objects with various semantic roles helps characterize the current state of activity.Source: §2 · pp. 3–4
- Gkioxari et al. extend earlier HOI detection by introducing an action-specific density map over target-object locations based on detected-person appearance.The cited work also reports significantly improved results after replacing the feature backbone with ResNet-50.Source: §2 · p. 4
3 Instance-Centric Attention Network
iCAN detects human-object interactions by conditioning learned attention on individual human or object instances, then combining appearance, contextual, and spatial relationship features. Its inference first detects instances and evaluates human-object pairs, while training treats interactions as independent multi-label actions.
3.4 Inference
- Inference first uses Faster R-CNN from Detectron to detect person and object instances, then evaluates all human-object bounding-box pairs for each possible action.HOI scores depend on individual detection confidence, human and object interaction predictions, and spatial relationship scores.Source: §3.1 · p. 5
3.1 Overview
- Training formulates HOI detection as multi-label classification, using an independent binary sigmoid classifier and cross-entropy loss for each action category.A person may perform different actions concurrently toward one or multiple target objects.Source: §3.1 · p. 5
3.2 Instance-Centric Attention Module
- iCAN dynamically learns an instance-centric attention map conditioned on a human or object’s appearance to highlight interaction-relevant image regions.Instance and convolutional features are embedded in a 512-dimensional space, compared by dot product, normalized with softmax, and used to compute contextual features.Source: §3.2 · p. 6
- The attention module concatenates instance-level appearance features with attention-weighted contextual features, and learns these features jointly with the rest of the network.This contrasts with hand-designed pose, whole-image, or secondary-region contextual features and with image-level attention modules.Source: §3.2 · p. 6
3.3 Multi-Stream Network Architecture
- The network uses human, object, and pairwise streams to compute action scores from instance appearance, contextual features, and human-object spatial relationships.The pairwise stream supplements appearance because appearance alone can produce plausible but incorrect interactions, while human appearance helps distinguish actions with similar spatial layouts.Source: §3.3 · p. 6
3.4 Inference
- Late fusion predicts human and object stream scores independently before summing them, whereas early fusion concatenates human iCAN, object iCAN, and pairwise features before classification.The paper implements both late- and early-fusion variants of iCAN.Source: §3.4 · p. 7
4 Experimental Results
Experiments on V-COCO and HICO-DET show that iCAN improves HOI detection over competing methods, while attention visualizations and ablations support its instance-centric design. Both human- and object-centric attention contribute, and early fusion offers the best V-COCO performance at higher computational cost.
Overall quantitative results
- On V-COCO and HICO-DET, iCAN achieves favorable quantitative performance compared with existing HOI detection methods.The evaluation reports AProle for V-COCO and HICO-DET, including all, rare, and non-rare interactions under two settings.Source: §4.2 · p. 8
HOI detection results
- Sample detections demonstrate that iCAN recognizes varied interactions, multiple actions, and different objects associated with the same object category.The examples span V-COCO and HICO-DET scenes and include persons performing multiple actions or interacting with different objects.Source: §4.3 · p. 8
Attention map visualization
- Human-centric attention focuses on surrounding objects, while object-centric attention highlights informative human body parts for action prediction.Visualizations show human-centric maps emphasizing interacting objects and object-centric maps highlighting body regions such as a hand.Source: §4.3 · pp. 8–9
Contextual feature
- Contextual features generally improve HOI detection, with instance-conditioned attention providing a larger boost than unconditioned contextual features.The comparison considers bottom-up attention, whole-image convolutional features, and the proposed instance-centric attention map.Source: §4.4 · pp. 9–10
Human-centric vs. object-centric
- Combining human-centric and object-centric attention maps is important for performance.The ablation study validates the contribution of leveraging both attention streams.Source: §4.4 · p. 10
mAP vs. time vs. memory
- Early fusion achieves the best V-COCO performance but requires evaluating all human-object pairs and increases training and testing time.The model variants are compared by mAP, inference time, and memory usage; early fusion has the stated computational trade-off.Source: §4.4 · p. 10
5 Conclusions
The paper proposes an instance-centric attention module for human-object interaction detection that highlights informative image regions based on person and object appearance. The approach gathers relevant contextual information and achieves a sizable performance boost on two HOI benchmark datasets.
Conclusions
- The paper proposes an instance-centric attention module for HOI detection.The work focuses on learning attention around individual instances.Source: §5 · p. 10
- The module highlights informative image regions using the appearance of a person and an object instance.This instance-conditioned attention enables the network to gather relevant contextual information for HOI detection.Source: §5 · p. 10
- The approach delivers a sizable performance boost compared to state-of-the-art methods on two HOI benchmark datasets.The paper validates the approach on two HOI benchmarks.Source: §5 · p. 10