Source-linked AI summary
VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions
Oytun Ulutan, A S M Iftekhar, B. S. Manjunath
TL;DR
HOI detection requires models to learn object interactions while analyzing individual entities, especially through relative spatial and structural cues. VSGNet combines visual features with spatial-attention refinement and graph convolutions, and it outperforms state-of-the-art solutions on V-COCO and HICO-DET.
Problem
HOI detection must capture subtle human-object relations while localizing the interacting pairs, but existing approaches do not explicitly exploit interaction information or pairwise spatial relations.
Method
VSGNet combines Visual, Spatial, and Graph branches, refining pair features with spatial configurations and using interaction proposal scores as graph edge intensities.
Results
VSGNet outperforms state-of-the-art methods by 4 mAP (8%) on V-COCO and 3 mAP (16%) on HICO-DET.
Takeaways & Limitations
Spatially refined visual features and graph convolutions together provide a model that improves HOI detection performance across V-COCO and HICO-DET.
Takeaways & Limitations
Some action classes perform poorly because object detectors miss small or occluded objects, and the existing detectors were not fine-tuned for these cases.
Abstract
from arXiv · showhide
Comprehensive visual understanding requires detection frameworks that can effectively learn and utilize object interactions while analyzing objects individually. This is the main objective in Human-Object Interaction (HOI) detection task. In particular, relative spatial reasoning and structural connections between objects are essential cues for analyzing interactions, which is addressed by the proposed Visual-Spatial-Graph Network (VSGNet) architecture. VSGNet extracts visual features from the human-object pairs, refines the features with spatial configurations of the pair, and utilizes the structural connections between the pair via graph convolutions. The performance of VSGNet is thoroughly evaluated using the Verbs in COCO (V-COCO) and HICO-DET datasets. Experimental results indicate that VSGNet outperforms state-of-the-art solutions by 8% or 4 mAP in V-COCO and 16% or 3 mAP in HICO-DET.
1. Introduction
HOI detection must reason about subtle human-object relations and spatial configurations, which existing pipelines do not explicitly exploit. VSGNet addresses this with visual, spatial-attention, and graph-convolution branches.
- HOI detection requires localizing human-object interactions despite subtle relations, setting variation, many-to-one interactions, and visually similar relations.
- Existing methods pair human and object features but do not explicitly use interaction information or spatial relations between pairs.
- VSGNet’s spatial-attention branch uses pair configurations to refine visual features, amplifying spatially relevant pairs.
- Interaction proposal scores from spatially refined features define edge intensities between human and object nodes for graph convolutions.
- 4 mAP (8%) and 3 mAP (16%) improvements over state-of-the-art methods are reported on V-COCO and HICO-DET, respectively.
- The complete VSGNet pipeline combines Visual, Spatial, and Graph branches and achieves state-of-the-art HOI detection results on V-COCO and HICO-DET.
2. Related Work
Related work frames HOI detection as an extension of activity recognition that requires detecting humans, objects, and their interactions. Prior methods use object, human-centric, attention, pose, or graph cues but leave gaps in spatial or relational modeling.
- VSGNet uses a pre-trained Faster-RCNN detector and extends its region-proposal idea to predict whether human-object pairs interact.
- V-COCO and HICO-DET require explicit detection of humans, objects, and their interactions, extending activity recognition beyond single actions in short videos.
- Human-centric methods exploit human appearance but may miss interactions where the object is far from the human.
- A prior graph-based network detects adjacency among human and object nodes but does not use spatial relation cues between pairs.
- Attention-based approaches derive maps from human and object features, while concatenating spatial and visual features does not enforce configurations as strongly as attention.
- Pose-integrated methods predict pairwise interaction probabilities but do not explicitly use them to learn relational structure between human-object pairs.
3. Proposed Method
VSGNet detects human-object interactions by combining visual, spatial-attention, and graph-convolutional branches for each human-object pair. It uses pair spatial configurations and interaction-based graph structure to refine features and aggregate predictions.
- Architecture: VSGNet uses visual, spatial-attention, and graph-convolutional branches to analyze each human-object pair.The visual branch extracts human, object, and context features; the other branches model pair geometry and structural interactions.
- Visual Branch: The visual branch extracts human and object features with RoI pooling, residual blocks, and global average pooling, while incorporating whole-image context.Human, object, and context vectors are concatenated and projected into a combined pair representation.
- Spatial Attention Branch: Spatial attention convolutions encode binary human-object box configurations and use them to refine visual features for spatially correlated pairs.The resulting attention vector represents the pair’s spatial configuration and is used mainly as an attention mechanism.
- Graph Convolutional Branch: The graph branch represents humans and objects as nodes, connects human-object pairs, and uses interaction proposal scores rather than visual similarity for adjacency.These graph convolutions model structural relations and generate features for interaction classification.
- Prediction Aggregation: The network predicts interaction proposal scores and action-class probabilities, then multiplies branch predictions and proposal scores to form the final prediction.The action class probability vector has size A, and the final prediction vector also has size A.
4. Experiments
VSGNet is evaluated on V-COCO and HICO-DET using mAP under established settings, with experiments covering implementation, comparisons, per-class performance, and branch ablations. It achieves improvements over prior methods, while detector failures remain a limitation for some action classes.
- Implementation details: The implementation uses a pretrained Faster-RCNN detector, experimentally selected confidence thresholds, and a fixed ResNet-152 backbone that is not fine-tuned during training.Human and object boxes are filtered at 0.6 and 0.3 confidence, respectively.
- Datasets and evaluation: VSGNet is evaluated on V-COCO and HICO-DET using reported mAP settings and comparisons with five recent state-of-the-art methods.The evaluation includes quantitative and qualitative analyses of the model and its improvements over existing methods.
- State-of-the-art comparison: 4 mAP improvement is reported for V-COCO Scenario 1, while Scenario 2 also outperforms all available reported methods.Table 1 compares both scenarios; the caption states an 8% advantage over the closest method.
- State-of-the-art comparison: VSGNet achieves the best results among previous works on HICO-DET and performs better in the majority of reported V-COCO per-class comparisons.The HICO-DET table reports a 16% advantage over the closest method.
- Limitations: Some action classes perform poorly because object detectors fail on small or occluded objects, and the existing detectors were not fine-tuned for these cases.The paper gives “eat instruments” as an example of an affected class.
5. Discussions
VSGNet differs from prior HOI methods by using spatial relations to alter visual features and interaction proposal scores to define graph edges. These choices support graph-based modeling of relevant human-object connections and are associated with improved performance over state-of-the-art methods.
- Spatial attention: Spatial relations alter visual features through element-wise multiplication, rather than being used only directly for classification or concatenated with visual features.This attention mechanism links spatial configuration to visual feature channels and amplifies spatially relevant pairs.
- Spatial attention: Unlike iCAN, VSGNet applies pairwise spatial configuration directly to visual features instead of modeling human or object regions against the whole scene individually.The resulting features amplify connected pairs and dampen irrelevant ones.
- Graph convolutions: Interaction proposal scores define graph edge intensities between human and object nodes, replacing visual similarity as the adjacency signal.Graph convolutions traverse relevant object nodes for humans and relevant human nodes for objects.
- Summary: VSGNet combines visual, spatial, and graph branches to model interactions between human-object pairs.Spatially altered visual features generate the interaction proposal scores used by the graph branch.