Source-linked AI summary
Spatially Conditioned Graphs for Detecting Human-Object Interactions
Frederic Z. Zhang, Dylan Campbell, Stephen Gould
TL;DR
HOI detection requires fine-grained identification of interacting human–object pairs, where appearance alone can confuse correspondences. The paper proposes spatially conditioned graph message passing with multi-branch fusion across adjacency, messaging, and feature refinement, and reports improved state-of-the-art performance, especially with fine-tuned detections.
Problem
HOI recognition must distinguish specific human–object interactions and correspondences, including cases with multiple similar pairs or weak visual cues.
Method
The method uses spatially conditioned, anisotropic graph message passing with multi-branch fusion applied to adjacency, messages, and refined graph features.
Results
The model achieves state-of-the-art performance on HICO-DET and V-COCO, with a larger improvement margin as detection quality increases.
Takeaways & Limitations
Spatial information becomes relatively more significant than coarse appearance features as detection quality improves, allowing the model to better exploit higher-quality detections.
Takeaways & Limitations
Qualitative results show that spatial proximity and visual similarity can still cause false high scores for non-interactive human–object pairs.
Abstract
from arXiv · showhide
We address the problem of detecting human-object interactions in images using graphical neural networks. Unlike conventional methods, where nodes send scaled but otherwise identical messages to each of their neighbours, we propose to condition messages between pairs of nodes on their spatial relationships, resulting in different messages going to neighbours of the same node. To this end, we explore various ways of applying spatial conditioning under a multi-branch structure. Through extensive experimentation we demonstrate the advantages of spatial conditioning for the computation of the adjacency structure, messages and the refined graph features. In particular, we empirically show that as the quality of the bounding boxes increases, their coarse appearance features contribute relatively less to the disambiguation of interactions compared to the spatial information. Our method achieves an mAP of 31.33% on HICO-DET and 54.2% on V-COCO, significantly outperforming state-of-the-art on fine-tuned detections.
1. Introduction
HOI detection requires identifying interacting human–object pairs and their interaction types, but appearance-only graph reasoning can confuse correspondences among many candidate pairs. The paper addresses this with spatially conditioned message passing and consistent spatial conditioning across graph computation stages.
- Task and motivation: HOI detection localises and describes interacting human–object pairs as predicate–object triplets with humans as subjects.Examples include riding a horse and sitting on a bench.
- Task and motivation: Appearance-only reasoning struggles when multiple human–object pairs share an interaction type or interactions lack strong visual cues.The task requires fine-grained reasoning about instance relationships and context, including abstract interactions such as buying an apple.
- Task and motivation: Conventional graph messages are identical across neighbours apart from adjacency scaling, allowing salient objects to dominate node features and confuse pair classification.The paper argues that spatial information should regulate message passing to distinguish correct human–object correspondences.
- Proposed direction: The method uses a spatially conditioned message passing algorithm whose outgoing messages depend on receiving nodes.It also extends spatial conditioning to adjacency computation and graph-feature refinement through a multi-branch fusion module.
2. Related Work
Prior HOI work constructs human–object candidates from object detections, combines appearance and spatial modalities, and increasingly uses graph structures to propagate contextual information. The paper is positioned within this progression toward structured reasoning over heterogeneous human and object nodes.
- Detection and feature pipelines: Many HOI pipelines first generate object detections and then construct candidate human–object pairs for interaction classification.This proposal-based formulation commonly uses Faster R-CNN detections followed by exhaustive pairing after filtering.
- Graph-based methods: The proposed graph is illustrated with directed edges and anisotropic message passing, including adjacency, messages, and class-logit computation.The figure omits update functions and refers readers to the corresponding equations.
- Detection and feature pipelines: Earlier methods combine human, object, and pairwise spatial information through multiple streams, while some alternatives detect interactions directly as keypoints.Keypoint-based detection represents a human–object pair by the centre of its minimum covering rectangle.
- Feature representations: HOI representations have evolved from binary spatial masks toward explicit box geometry, with human pose also explored as an additional modality.This paper focuses on how structured architectures exploit appearance and spatial information rather than using pose.
- Graph-based methods: Graph-based HOI methods model detected humans and objects as nodes and update their appearance features through message passing.Later work distinguishes messages between heterogeneous human and object nodes from messages between homogeneous nodes.
3. Spatially Conditioned Graphs
The model represents detected humans and objects as a bipartite graph, then applies spatial conditioning to adjacency, messages, and pairwise features through multi-branch fusion. Spatial and appearance information are combined throughout graph reasoning and classification.
- Bipartite Graph Structure: A Faster R-CNN detector supplies filtered detections, whose appearance features initialise graph nodes and whose spatial relationships provide edge features.Each detection contains box coordinates, confidence, and class; pairwise spatial features include geometry, overlap, relative area, and direction.
- Bipartite Graph Structure: The graph connects every human node to object nodes and performs bidirectional message passing with direction-specific message functions.LayerNorm and ReLU are used in the updates, while separate parametrisations allow messages to depend on the receiving-node direction.
- Spatial Conditioning: Spatial conditioning makes messages depend on pairwise relationships rather than sending identical appearance-based messages to all neighbours.The sender node encoding and edge encoding are fused by elementwise product, allowing messages to express relative location.
- Spatial Conditioning: The model applies spatial conditioning to adjacency, messages, and final human–object features, corresponding to early, mid, and late modality fusion.Spatially conditioned adjacency is intended to identify interactive human–object pairs, while conditioned graph features support pair classification.
- Multi-Branch Fusion: Multi-branch fusion maps appearance and spatial modalities into reduced subspaces, fuses each branch, and aggregates branch outputs.The module is called MBF, and its parameter count is designed to remain independent of the number of branches through dimensionality reduction.
- Training and Inference: Training and inference combine action scores with human and object detection scores, while focal loss downweights well-classified examples and normalises by positive logits.At inference, detection scores are raised to λ before multiplication with action scores; the loss restricts outputs to meaningful interactions.
4. Experiments
Experiments on HICO-DET and V-COCO evaluate the model across detector qualities, modality contributions, spatial-conditioning stages, fusion choices, and qualitative cases. The model benefits most from higher-quality detections, with spatial information becoming more influential as detection quality improves.
- Datasets and setup: The model is evaluated on HICO-DET and V-COCO using pre-trained, fine-tuned, and oracle detection settings.HICO-DET has 37,633 training and 9,546 test images; V-COCO has 2,533 training, 2,867 validation, and 4,946 test images.
- Comparison with state of the art: A 20% relative improvement over state of the art is achieved on HICO-DET with fine-tuned detections.The model exceeds IDN by 5 mAP with fine-tuned detections and exceeds the next best method by 7.5 mAP with oracle detections.
- Contribution of different modalities: As detection quality improves, spatial information contributes relatively more to interaction disambiguation than coarse appearance features.With pre-trained detections, appearance and spatial corruption have approximately equal effects; with fine-tuned detections, spatial corruption has a much larger effect.
- Ablation studies: Ablation studies examine spatial conditioning in adjacency computation, message passing, global features, and refined graph features.Additional studies vary multi-branch fusion operations, cardinality, and the number of message-passing iterations.
- Qualitative results: Qualitative results include successful suppression of a non-interactive human–bench pair but confusion between a correct and nearby visually similar bike pair.The bike failure is attributed to spatial proximity and visual similarity between bike instances.
5. Conclusion
The paper concludes that spatially conditioned graph neural networks improve human–object interaction detection by modulating appearance features with pairwise spatial configurations. Its gains are especially pronounced with fine-tuned detections, indicating stronger exploitation of improved detector outputs.
- Conclusion: The proposed multi-branch fusion mechanism modulates appearance features with human–object spatial configurations.Spatial conditioning is applied to adjacency computation, messages, and refined graph features.
- Conclusion: The model outperforms state of the art by a considerable margin with fine-tuned detections.The margin of improvement increases with detection quality, allowing the model to exploit advances in object detector research.
A. Known Object Setting for HICO-DET
The known object setting evaluates HOI detection when ground-truth interaction object types are known, removing predictions involving other object types. The paper provides comparisons under this less frequently reported protocol.
- Known object setting: The known object setting treats interaction object types as known and removes predicted pairs involving other object types.This is an additional evaluation protocol for HICO-DET beyond the default setting.
- Known object setting: Performance comparisons for the known object setting are provided in Table 10.The table identifies the most competitive and second-best methods in each category.
B. Additional Qualitative Results
The qualitative results show that spatially conditioned graph reasoning is most competitive in crowded scenes with many candidate pairs, but offers less benefit when images contain few distractions.
- Crowded scenes: The model correctly ranks all four interactive human–horse pairs above 16 non-interactive pairs in a crowded scene.The example contains 20 combinatorial human–horse pairs, only four of which are interactive.
- Crowded scenes: The model identifies several correct human–suitcase pairs, but a positive pair receives a low score when bounding boxes are small and object detection confidence is lower.The model also assigns a high carrying score to a non-target pair with a close relative location and plausible gesture.
- Clean scenes: The model does not consistently improve over previous methods on clean images containing only a small number of human and object instances.In a single-pair image, limited message passing provides little advantage; when both zebra pairs are interactive, appearance-only scoring also succeeds.
- Overall pattern: Overall, spatial conditioning is particularly competitive when many human and object instances create multiple same-interaction candidate pairs.Its advantage is reduced on clean images with very few distractions.
C. Additional Ablations
Additional training choices improve the model's performance, including larger batches, AdamW optimization, and backbone fine-tuning.
- Training techniques: Increasing batch size from 4 to 32 yields roughly 0.8 mAP improvement by stabilizing focal-loss training.The focal loss is normalized by the number of positive logits, an unstable statistic.
- Training techniques: Using AdamW instead of SGD contributes about 1 mAP to the model's performance.The authors attribute this improvement to similarities between graphical models and transformers.
- Training techniques: Fine-tuning the backbone produces a further 1 mAP improvement.