PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Contextual Attention for Human-Object Interaction Detection

Tiancai Wang, Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao, Jorma Laaksonen

arXiv:1910.07721v1cs.CV

TL;DR

HOI detection must localize people and objects while recognizing their interactions, but existing approaches largely overlook context. The paper adds context-aware appearance features and instance-centric attention, and reports state-of-the-art performance across three benchmarks.

  • Problem

    HOI detection involves identifying ⟨human, action, object⟩ triplets, but existing approaches mainly use human and object appearance and underuse context that can help distinguish interactions.

    Source: §1 · p. 1

  • Method

    The framework enriches human and object appearance with context and uses instance-conditioned attention to select relevant context and highlight likely interaction regions.

    Source: §1 · p. 2

  • Results

    The approach outperforms state-of-the-art methods on all three benchmarks; on HICO-DET it reports a 9.4% relative gain in mAP over the best published method.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported results support contextual attention as a competitive approach to HOI detection across V-COCO, HICO-DET, and HCVRD.

    Source: §5 · p. 8

Abstract

from arXiv · show

Human-object interaction detection is an important and relatively new class of visual relationship detection tasks, essential for deeper scene understanding. Most existing approaches decompose the problem into object localization and interaction recognition. Despite showing progress, these approaches only rely on the appearances of humans and objects and overlook the available context information, crucial for capturing subtle interactions between them. We propose a contextual attention framework for human-object interaction detection. Our approach leverages context by learning contextually-aware appearance features for human and object instances. The proposed attention module then adaptively selects relevant instance-centric context information to highlight image regions likely to contain human-object interactions. Experiments are performed on three benchmarks: V-COCO, HICO-DET and HCVRD. Our approach outperforms the state-of-the-art on all datasets. On the V-COCO dataset, our method achieves a relative gain of 4.4% in terms of role mean average precision ($mAP_{role}$), compared to the existing best approach.

1. Introduction

HOI detection requires localizing people and objects while recognizing their interactions, including fine-grained and co-occurring actions. The paper adds instance-centric context and contextual attention, and reports state-of-the-art results on three benchmarks.

  • HOI detection requires localizing a human and an object and recognizing their interaction as a ⟨human, action, object⟩ triplet.The task includes fine-grained actions and multiple co-occurring actions.Source: §1 · p. 1
  • Existing approaches emphasize human and object appearance, while context at multiple image granularities remains underused in HOI detection.Global context can distinguish driving a boat from driving a car, while nearby context can distinguish interactions involving the same object.Source: §1 · p. 1
  • The framework enriches human and object appearance features with context, then uses instance-conditioned attention to suppress background noise and highlight likely interaction regions.The attention maps modulate global features to emphasize regions likely to contain human-object interactions.Source: §1 · p. 2
  • A 9.4% relative gain in mAP over the best published method is reported on HICO-DET, and the approach sets state-of-the-art results across three benchmarks.The benchmarks are V-COCO, HICO-DET, and HCVRD.Source: §1 · p. 2

2. Related Work

Prior HOI methods combine object localization with interaction recognition, using human, object, and pairwise streams or other structural and human-intention cues. Contextual information has aided other vision tasks but remains underexplored for HOI detection.

  • HOI detection research includes human-centric, graph-based, gaze-and-pose, and human-object region-based approaches, with HICO-DET introduced as a benchmark.Source: §2 · p. 2
  • Prior HOI systems typically localize people and objects with off-the-shelf two-stage detectors, while object-detection work also includes single-stage methods.The paper's framework uses an FPN detector for both human and object instances.Source: §2 · p. 2
  • The proposed framework separates localization from interaction prediction and fuses human, object, and pairwise stream scores.Context-aware appearance and contextual attention modules are placed in the human and object streams.Source: Figure 2 · p. 3
  • Context has supported object detection, action recognition, and semantic segmentation, but its impact on HOI detection remains incompletely explored.Source: §2 · p. 3

3. Overall Framework

The framework detects human–action–object triplets through localization and fused human, object, and pairwise streams; its human/object streams add context-aware features and attention-based refinement.

3. Overall Framework

  • FPN localizes human and object instances, then human, object, and pairwise streams fuse their scores to predict human–action–object triplets.Human and object scores are added, and the sum is multiplied by the pairwise-stream score.Source: §3 · p. 3

3.1. Proposed Human/Object Stream

  • Both human and object streams combine contextual appearance features with refined, attention-modulated features to predict actions.The same architecture is used for both streams; only their human or object bounding-box inputs differ.Source: §3.1 · p. 3

3.1.1 Contextually-Aware Appearance Features

  • A factorized large-kernel block expands context while preserving spatial information, and PSRoIAlign-based local encoding preserves relative position within each ROI.The local encoding uses max-pooling on the contextualized feature map to produce contextual appearance features.Source: §3.1.1 · p. 4

3.1.2 Contextual Attention

  • Contextual appearance features generate bounding-box attention that modulates global features, which spatial and channelwise refinement further reweight before action prediction.The refined representation is globally pooled and combined with contextual appearance features for the stream’s action predictions.Source: §3.1.2 · p. 5

4. Experiments

The experiments evaluate the approach on V-COCO, HICO-DET, and HCVRD using the datasets’ original protocols. The reported results include benchmark comparisons, component and backbone analyses, and qualitative detection examples.

4.3. Results on V-COCO Dataset

  • On V-COCO, Table 1 reports a 6.3% relative mAProle gain over the same-backbone baseline, with both context-aware appearance and contextual attention contributing.The text reports mAProle rising from 44.5 to 46.0 with contextual appearance, then improving by a further 1.3% with contextual attention.Source: Table 1 · p. 6, §4.3 · p. 6

4.1. Dataset and Evaluation Protocol

  • The evaluation uses V-COCO, HICO-DET, and HCVRD with their original protocols; V-COCO uses mAProle, HICO-DET mAP, and HCVRD top-1/top-3 results at recall 50 and 100.V-COCO detection requires correct human and object boxes at IoU greater than 0.5 and the correct action label.Source: §4.1 · pp. 5–6

4.2. Implementation Details

  • Interaction recognition takes 130 ms per 480 × 640 image on a Titan X GPU, compared with 111 ms for the baseline.The reported training schedules are 300k iterations on V-COCO and 1800k on HICO-DET and HCVRD.Source: §4.2 · p. 6

4.3. Results on V-COCO Dataset

  • On V-COCO, the method reaches mAProle 47.3, an absolute 2.0% gain over the best existing method.Table 4 reports the state-of-the-art comparison.Source: Table 4 · p. 6, §4.3 · p. 7
  • On V-COCO, the approach is superior to the baseline in classification-focused comparisons, and its gains persist across VGG-16, ResNet-50, and ResNet-101 backbones.The classification comparison uses different IoU thresholds; lower thresholds focus more on classification capability.Source: §4.3 · pp. 6–7, Table 3 · p. 6
  • On V-COCO examples, contextual attention maps focus on interaction-relevant regions and are more anchored than iCAN’s for single and multiple interactions.The examples also depict multiple humans sharing interactions and one human performing multiple interactions.Source: Figure 5 · p. 7, §4.3 · p. 8

4.4. Results on HICO-DET and HCVRD datasets

  • On HCVRD, the method exceeds iCAN at R@50 with top-1/top-3 accuracies of 37.1/51.3 versus 33.8/48.9, and at R@100 with top-3 accuracy of 51.9 versus 49.4.Source: §4.4 · p. 8
  • On HICO-DET, Default-setting relative gains are 9.4%, 6.7%, and 9.9% on the full, rare, and non-rare sets, respectively.The reported comparison covers both Default and Known Object settings across all three sets.Source: Table 5 · p. 8

5. Conclusion

The framework learns contextually aware appearance features and uses adaptive attention to select instance-centric context; it outperforms state-of-the-art methods on three HOI benchmarks.

  • The framework outperforms state-of-the-art methods on V-COCO, HICO-DET, and HCVRD.Source: §5 · p. 8
  • The approach learns contextually aware appearance features for human and object instances.Source: §5 · p. 8
  • Its attention module adaptively selects relevant instance-centric context to suppress background noise and capture human-object interactions.Source: §5 · p. 8
Loading 1910.07721v1…