Source-linked AI summary
DRG: Dual Relation Graph for Human-Object Interaction Detection
Chen Gao, Jiarui Xu, Yuliang Zou, Jia-Bin Huang
TL;DR
HOI detection requires recognizing interactions between persons and objects despite isolated pairwise predictions and complex appearance-based relation modeling. The paper introduces spatial-semantic pair representations and human- and object-centric dual relation graphs to aggregate contextual cues. The model achieves favorable results on two large-scale benchmarks, while its object detector and action predictions retain stated limitations.
Problem
Existing HOI methods often predict human-object pairs in isolation or use complex appearance-based relation features, limiting their use of contextual and spatial cues.
Method
DRG represents each human-object pair spatial-semantically and propagates contextual information through human-centric and object-centric HOI subgraphs.
Results
The model achieves competitive or favorable performance on V-COCO and HICO-DET, and spatial-semantic features alone yield competitive accuracy without appearance features.
Takeaways & Limitations
Abstract spatial-semantic representations with dual contextual graphs provide a promising path for HOI activity understanding through visual abstraction.
Takeaways & Limitations
The pipeline relies on an off-the-shelf object detector, whose errors are not corrected by the contextual method.
Abstract
from arXiv · showhide
We tackle the challenging problem of human-object interaction (HOI) detection. Existing methods either recognize the interaction of each human-object pair in isolation or perform joint inference based on complex appearance-based features. In this paper, we leverage an abstract spatial-semantic representation to describe each human-object pair and aggregate the contextual information of the scene via a dual relation graph (one human-centric and one object-centric). Our proposed dual relation graph effectively captures discriminative cues from the scene to resolve ambiguity from local predictions. Our model is conceptually simple and leads to favorable results compared to the state-of-the-art HOI detection algorithms on two large-scale benchmark datasets.
1 Introduction
HOI detection must recognize interactions between detected persons and objects while using contextual cues beyond isolated pairwise predictions. DRG combines spatial-semantic pair representations with human- and object-centric graphs, achieving competitive results on V-COCO and HICO-DET.
- HOI detection localizes persons and objects and recognizes interaction relationships between each person-object pair.
- Existing methods often predict each human-object interaction in isolation, ignoring scene context and relying on complex appearance-based relation features.
- Spatial-semantic representations encode relative spatial layout and object-category word embeddings, providing appearance-invariant cues and enabling knowledge transfer among object classes.
- DRG aggregates contextual information through human-centric and object-centric HOI subgraphs with attentional graph convolution.
- 47.1 mAP is achieved by the spatial-semantic stream alone on V-COCO without visual features.
- Competitive results are reported against state-of-the-art methods on both V-COCO and HICO-DET.
2 Related Work
Prior work uses appearance and spatial cues, message passing, language priors, and graph structures to model visual relationships and contextual information. DRG differs by applying abstract spatial-semantic representations within human- and object-centric subgraphs rather than a single dense graph.
- Human-object interaction detection: HOI methods generally combine detected human/object appearance features with relative spatial layout to predict interactions.
- Human-object interaction detection: Relation proposal and interactiveness networks improve performance by filtering candidate pairs with no likely relation, whereas DRG does not add a pruning network.
- Human-object interaction detection: Long-tailed HOI methods transfer knowledge, augment data using semantically similar objects, or leverage external knowledge graphs; DRG does not explicitly target rare classes.
- Human-object interaction detection: A small rare-versus-non-rare performance gap is attributed to DRG's abstract spatial-semantic representation.
- Scene graph: Scene-graph approaches use message passing over dense object-pair graphs, while DRG uses explicit relative spatial layout and human- and object-centric subgraphs.
- Visual abstraction: DRG applies contextual modeling to abstract spatial-semantic representations, extending related contextual approaches in sequence prediction, object detection, and action recognition.
3 Method
The method combines spatial-semantic representations of human-object pairs with human-centric and object-centric relation graphs to aggregate scene context for HOI prediction. It fuses appearance, object-detection, and graph-based scores, while supporting object-free actions separately.
- Network overview: The network decomposes HOI detection into object detection followed by HOI prediction over detected human and object instances.It uses an off-the-shelf Faster R-CNN detector before constructing pairwise HOI predictions.
- Spatial-semantic representation: Each human-object pair is represented by relative spatial layout and the object category’s 300-dimensional fastText embedding.The spatial-semantic feature is constructed by concatenating the spatial feature with the category embedding into xij ∈R5708.
- Dual Relation Graph: The Dual Relation Graph inserts HOI nodes between paired human and object detections, then forms human-centric and object-centric HOI subgraphs.The two subgraphs connect HOIs involving the same person or the same object, respectively.
- Contextual aggregation: Attentional graph convolution iteratively aggregates contextual features across each subgraph while retaining the original pair representation through residual normalization.Learned attention weights combine neighboring features, and LayerNorm produces the updated feature.
- Inference: Final HOI scores combine human and object detection confidence, human and object appearance scores, and the aggregated spatial-semantic prediction.Actions without associated objects use only the human-stream score.
- Training: Training treats HOI detection as multi-label classification and sums the cross-entropy losses from the individual streams.This supports simultaneous actions involving different objects.
4 Experimental Results
Experiments on V-COCO and HICO-DET evaluate DRG using role mAP, state-of-the-art comparisons, visualizations, and ablations. Results show competitive performance, benefits from spatial-semantic representation and dual contextual graphs, and limitations from detector errors and ambiguous spatial configurations.
- Experimental setup: V-COCO and HICO-DET experiments use role mAP to evaluate correctly localized human-verb-object triplets.A true positive requires human and object IoUs ≥0.5 and correct action prediction.
- Quantitative evaluation: DRG achieves competitive state-of-the-art performance on both V-COCO and HICO-DET, including the best performance on rare HICO-DET categories.On V-COCO, the spatial-semantic stream alone achieves 47.1 mAP, while the reported comparison excludes only PMFNet with additional human-pose information.
- Ablation study: More feature-aggregation iterations improve performance for either subgraph, and aggregating on both subgraphs further improves final performance.The human-centric and object-centric graphs propagate contextual information through the spatial-semantic stream to refine HOI predictions.
- Ablation study: 51.37 mAP is achieved using the spatial-semantic representation in the dual relation graph, outperforming alternative word2vec and appearance-based features.The representation is designed to encode relative spatial layout and object-category semantics, supporting invariance to appearance variation and knowledge transfer among object classes.
- Limitations: The method remains limited by off-the-shelf object detection and confusion from plausible spatial configurations that produce incorrect actions.The authors suggest joint end-to-end training and modeling co-occurring actions as possible ways to reduce these errors.
5 Conclusions
DRG uses human-centric and object-centric subgraphs to propagate contextual relations among HOIs, exploiting global object layout for HOI detection. On two large-scale benchmarks, it achieves a sizable performance boost, while spatial-semantic features alone remain competitive.
- DRG exploits global object layout through human-centric and object-centric subgraphs to propagate and integrate relations among individual HOIs.The method aggregates contextual information from both the same person's and the same object's associated HOIs.
- The approach achieves a sizable performance boost over state-of-the-art algorithms on two large-scale HOI benchmark datasets.
- Abstract spatial-semantic representation alone yields competitive accuracy without appearance features extracted from a deep CNN.This result demonstrates a promising path for activity understanding through visual abstraction.