Source-linked AI summary
QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information
Masato Tamura, Hiroki Ohashi, Tomoaki Yoshinaga
TL;DR
HOI detection requires context beyond human and object regions, while conventional CNN-based methods can miss relevant cues, rely on heuristic locations, and mix nearby instances. QPIC extends DETR with transformer attention for image-wide aggregation and pairwise queries, achieving significantly better performance than existing methods on two benchmarks.
Problem
Conventional CNN-based HOI detectors are limited in using image-wide context, choosing feature locations, and separating closely located HOI instances.
Method
QPIC extends DETR with transformer attention for image-wide contextual aggregation and queries designed to capture at most one human-object pair.
Results
QPIC achieves significantly better performance than state-of-the-art methods on two challenging HOI detection benchmarks, including gains of 5.37 mAP on HICO-DET and 5.7 mAP on V-COCO.
Takeaways & Limitations
Attention and query-based detection play a crucial role in extracting contextual HOI features while enabling simple, intuitive detection heads.
Takeaways & Limitations
The evaluation excludes a pre-training practice that inappropriately uses COCO train2017 images contained in the V-COCO test set.
Abstract
from arXiv · showhide
We propose a simple, intuitive yet powerful method for human-object interaction (HOI) detection. HOIs are so diverse in spatial distribution in an image that existing CNN-based methods face the following three major drawbacks; they cannot leverage image-wide features due to CNN's locality, they rely on a manually defined location-of-interest for the feature aggregation, which sometimes does not cover contextually important regions, and they cannot help but mix up the features for multiple HOI instances if they are located closely. To overcome these drawbacks, we propose a transformer-based feature extractor, in which an attention mechanism and query-based detection play key roles. The attention mechanism is effective in aggregating contextually important information image-wide, while the queries, which we design in such a way that each query captures at most one human-object pair, can avoid mixing up the features from multiple instances. This transformer-based feature extractor produces so effective embeddings that the subsequent detection heads may be fairly simple and intuitive. The extensive analysis reveals that the proposed method successfully extracts contextually important features, and thus outperforms existing methods by large margins (5.37 mAP on HICO-DET, and 5.7 mAP on V-COCO). The source codes are available at $\href{https://github.com/hitachi-rd-cv/qpic}{\text{this https URL}}$.
1. Introduction
HOI detection must localize humans and objects while identifying their interactions, but conventional CNN-based approaches struggle with image-wide context, heuristic feature locations, and closely overlapping instances. QPIC addresses these issues with transformer attention and pairwise queries, reporting better performance on two challenging benchmarks.
- HOI detection identifies human-object interactions as human and object boxes, an object class, and an action class.
- Conventional methods use either individually localized regions with auxiliary context or heuristically chosen integrated features such as human-object midpoints.
- CNN locality can exclude important cues outside target boxes, contaminate union features with irrelevant content, and mix features from overlapping HOI instances.
- QPIC uses transformer attention to aggregate contextually important information across the image and queries that capture at most one human-object pair.
- QPIC reports significantly better performance than state-of-the-art methods on two challenging HOI detection benchmarks.
2. Related Work
Related HOI detectors either crop features after separate human and object localization or collect integrated features at heuristic locations. QPIC instead extends DETR to aggregate image-wide contextual features pairwise with a transformer.
- Two-stage methods localize humans and objects with off-the-shelf detectors before cropping and processing their regional features.
- Single-stage methods use integrated human-object features, often collected at a midpoint or within a union box, but remain vulnerable to locality and heuristic location choices.
- QPIC differs by extending DETR with a transformer that aggregates image-wide contextual features in a pairwise manner.
3. Proposed Method
QPIC uses a transformer encoder-decoder with learnable pairwise queries to aggregate image-wide context and produce separate embeddings for human-object pairs. Simple feed-forward heads then predict each interaction’s boxes, object class, and action classes, trained with Hungarian matching and composite losses.
- Overall Architecture: QPIC projects backbone features into a lower-dimensional map, then uses a transformer encoder with positional encoding to enrich them through self-attention.The encoder outputs a contextual feature map from the projected backbone representation.
- Overall Architecture: Learnable decoder queries attend to the encoded feature map, with each query designed to capture at most one human-object pair and its interactions.The query count is set larger than the number of actual human-object pairs so unmatched queries can represent no pair.
- Interaction Detection Heads: Each decoded embedding feeds four dedicated FFNs that predict normalized human and object boxes, object class, and multi-label action classes.The action vector can contain multiple positive actions for one human-object pair, while the object classifier includes a no-pair class.
- Loss Calculation: Training first applies Hungarian bipartite matching between predictions and padded ground truths, then computes losses for the matched assignments.The matching cost combines box-regression, intersection-over-union, object-class, and action-class costs.
- Loss Calculation: The total training loss combines box, intersection-over-union, object-class, and action-class components with adjustable hyper-parameter weights.The action matching cost balances positive and negative action classes using inverse nonzero-count weighting because positive actions are typically fewer.
- Inference: At inference, each query-action combination yields a human box, object box, predicted object class, and action label when its score exceeds a threshold.The score multiplies the predicted action probability by the maximum object-class probability.
4. Experiments
QPIC is evaluated on HICO-DET and V-COCO against existing methods, with analyses of spatial distribution, detection heads, feature extractors, and qualitative attention behavior. Results show strong benchmark performance and support image-wide contextual aggregation with pairwise query-based detection.
- Benchmark comparison: QPIC achieves a 5.37 mAP gain over DRG and an 8.17 mAP gain over PPDM on HICO-DET in the default full setting.These correspond to relative gains of 21.9% and 37.6%, respectively, using the ResNet-101 backbone.
- Benchmark comparison: QPIC achieves a 5.7 mAP gain over FCMNet on V-COCO, demonstrating state-of-the-art performance among the reported baselines.The gain is 10.7% with the ResNet-50 backbone.
- Spatial distribution: As human-object distance grows, QPIC’s relative performance advantage over DRG and PPDM becomes more evident.The analysis suggests that distant HOI instances are especially difficult for CNN-based methods, while QPIC handles them relatively better.
- Detection heads: QPIC’s simple detection heads provide sufficient accuracy, whereas simple heads paired with a CNN-based feature extractor perform far worse than QPIC.The comparison indicates that rich transformer-derived features, rather than complicated manually designed heads, support the simple prediction design.
- Feature extractor: The decoder improves HOI detection more than object detection, with relative gains of 45.8% versus 23.9% when the encoder is included.The corresponding gains without the encoder are 41.6% for HOI detection and 11.8% for object detection.
- Qualitative analysis: Qualitative examples show QPIC attending to contextual cues outside target boxes and selectively avoiding irrelevant information in overlapping HOI instances.These behaviors address failures caused by missing context, contaminated region features, and heuristic detection points.
5. Conclusion
QPIC selectively aggregates image-wide contextual information for HOI detection through attention, enriching HOI features so simple detection heads can be used. Evaluation and analysis show superiority over existing methods and identify attention and query-based detection as crucial.
- QPIC selectively aggregates image-wide contextual information for detecting a wide variety of human-object interactions.Its attention mechanism enriches HOI features.
- QPIC’s enriched HOI features enable simple and intuitive detection heads.
- QPIC showed significant superiority over existing methods on two benchmark datasets.
- Analysis identifies the attention mechanism and query-based detection as crucial for HOI detection.
A. Supplementary V-COCO Settings
The supplementary V-COCO settings combine training and validation data for training QPIC and exclude five action classes when calculating mAP.
- QPIC is trained on the combined V-COCO training and validation sets, following previous work.
- Five of V-COCO’s 29 action classes are excluded from mAP evaluation.The excluded classes are run, smile, stand, walk, and point.
B. Supplementary Implementation Note
The supplementary implementation note describes training augmentation, auxiliary losses for transformer decoder layers, and an evaluation rule for object-class predictions.
- Training uses random horizontal flipping, scale, random crop, and color augmentation to alleviate over-fitting.
- QPIC calculates auxiliary losses for each transformer decoder layer using shared FFNs added to the layer outputs.
- At evaluation, the second-highest object-class score can generate a detection when the highest score is the “no pair” class.This technique is used to optimize mAPs.
C. Additional List of Comparison
Additional comparisons report that QPIC significantly outperforms conventional two- and single-stage HOI detection methods on HICO-DET and V-COCO.
- QPIC is compared with six additional methods across the HICO-DET and V-COCO comparison tables.The methods are PMFNet, Wang et al., In-GraphNet, VSGNet, PD-Net, and DJ-RM.
- QPIC significantly outperforms conventional two- and single-stage methods on both datasets.
D. Computational Efficiency Comparison
QPIC is compared with PPDM for inference efficiency, with QPIC showing a shorter inference time under the reported test setup. The comparison tables identify benchmark evaluations on HICO-DET and V-COCO, while Table 7 covers efficiency.
- Benchmark comparisons: Table 5 compares state-of-the-art methods on HICO-DET using mAP, grouped into two-stage, single-stage, and QPIC method blocks.
- Benchmark comparisons: Table 6 compares state-of-the-art methods on V-COCO using the same two-stage, single-stage, and QPIC block structure as Table 5.
- Efficiency evaluation: Table 7 presents the comparison of computational efficiency.
- Inference-time comparison: 5.4 ms: QPIC spends less time organizing network outputs than PPDM's 17 ms.The authors attribute this difference mainly to QPIC's simple detection heads and inference procedures.
E. Additional Qualitative Analysis
Additional qualitative cases show QPIC succeeding where conventional detectors fail, particularly when contextual regions matter or multiple HOI instances overlap. Its attention selectively aggregates relevant image regions for each interaction.
- Contextual information: QPIC uses the basketball goal as contextual information to correctly detect an interaction missed by DRG.The attention map indicates aggregation from the basketball-goal region.
- Overlapping instances: QPIC separately aggregates features for overlapping HOI instances, avoiding the contamination that degrades DRG's detection.The example contrasts QPIC's attention map with a DRG track box that includes another human.
- Additional failure cases: QPIC successfully detects HOIs in the additional failure cases shown for DRG and PPDM.Figure 6a–6b concern DRG, while Figure 6c–6d concern PPDM.