Source-linked AI summary
Reformulating HOI Detection as Adaptive Set Prediction
Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, Chen Qian
TL;DR
HOI detectors struggle to select effective interaction features when relying on detected pairs or predefined locations. The paper reformulates detection as adaptive set prediction and introduces AS-Net with parallel instance and interaction branches. AS-Net outperforms prior methods across three datasets, including a 31% relative improvement over the previous best one-stage method on HICO-DET.
Problem
Existing HOI detectors focus on detected human-object pairs or predefined interaction locations, limiting adaptive selection of interaction-relevant features.
Method
AS-Net maps learnable interaction queries to predictions, aggregates global features through co-attention, matches predictions adaptively to ground truth, and transfers instance features through instance-aware attention.
Results
AS-Net outperforms prior methods on HICO-DET, V-COCO, and HOI-A, with a 31% relative improvement over the previous best one-stage method on HICO-DET.
Takeaways & Limitations
Adaptive set prediction lets the method concentrate on suitable interaction-relevant features without extra human pose or language features.
Abstract
from arXiv · showhide
Determining which image regions to concentrate on is critical for Human-Object Interaction (HOI) detection. Conventional HOI detectors focus on either detected human and object pairs or pre-defined interaction locations, which limits learning of the effective features. In this paper, we reformulate HOI detection as an adaptive set prediction problem, with this novel formulation, we propose an Adaptive Set-based one-stage framework (AS-Net) with parallel instances and interaction branches. To attain this, we map a trainable interaction query set to an interaction prediction set with a transformer. Each query adaptively aggregates the interaction-relevant features from global contexts through multi-head co-attention. Besides, the training process is supervised adaptively by matching each ground truth with the interaction prediction. Furthermore, we design an effective instance-aware attention module to introduce instructive features from the instance branch into the interaction branch. Our method outperforms previous state-of-the-art methods without any extra human pose and language features on three challenging HOI detection datasets. Especially, we achieve over $31\%$ relative improvement on a large-scale HICO-DET dataset. Code is available at https://github.com/yoyomimi/AS-Net.
1. Introduction
HOI detection must select interaction-relevant regions beyond fixed instance pairs or locations. AS-Net reformulates it as adaptive set prediction with parallel branches and achieves strong benchmark results.
- Motivation: HOI detection identifies <human, verb, object> triplets, but conventional methods rely on detected instances or predefined interaction locations.Two-stage methods use detected human-object pairs, while one-stage methods use union boxes or interaction midpoints.
- Adaptive set prediction: AS-Net maps learnable interaction queries to predictions, aggregates global interaction-relevant features through co-attention, and adaptively matches predictions with ground truth.This set-based formulation removes the location limitation of conventional one-stage methods.
- Framework: AS-Net uses parallel instance and interaction branches, with instance-aware attention transferring instance-branch information into interaction prediction.The instance branch predicts instance locations and categories, while the interaction branch predicts interaction vectors and categories.
- Results: 31% relative improvement over the previous state-of-the-art one-stage method is reported on HICO-DET without extra features.The method is evaluated on HICO-DET, V-COCO, and HOI-A.
2. Related Work
Prior HOI detectors mainly use two-stage instance-centric pipelines or one-stage location-based predictions. These approaches differ in efficiency and interaction localization strategy.
- Two-stage methods: Two-stage methods first detect human and object instances, then classify interactions using cropped, spatial, union-box, or context features.Some methods additionally use human pose and language features to obtain interaction-relevant information.
- One-stage methods: One-stage methods predict HOI triplets directly and divide into anchor-based and point-based approaches according to interaction localization.Anchor-based methods use human-object union boxes, whereas point-based methods use interaction key points such as pair midpoints.
3. Methods
AS-Net reformulates HOI detection as parallel set prediction with separate instance and interaction branches. Transformer decoding, adaptive matching, branch aggregation, and semantic embeddings support interaction prediction and human-object matching.
- Backbone and parallel branches: AS-Net uses a CNN-transformer backbone to extract image features with global contexts for parallel instance and interaction set prediction.The backbone reduces CNN feature channels, reshapes them into a sequence, and refines them with transformer self-attention and feed-forward layers.
- Instance branch: The instance branch predicts bounding boxes, categories, and semantic embeddings through transformer decoding and feed-forward heads.Its set-based training matches predictions to ground-truth instances using bounding-box and category costs, with Hungarian assignment.
- Interaction branch: The interaction branch predicts interaction vectors and multi-label interaction categories from learnable query sets attending to shared image features.Each vector points from a human center toward an object center, while matching uses predicted vectors and interaction-category scores.
- Interaction branch: Hungarian matching adaptively assigns ground-truth interactions to predictions, and focal classification plus L1 regression train interaction categories and vectors.The matching cost combines interaction-vector distance with scores for ground-truth interaction categories.
- Instance-aware attention: An instance-aware attention module transfers relevant instance features into the interaction branch through attention weights between predicted interactions and detected instances.The module computes an attention matrix, aggregates instance features, and adds the resulting features to interaction representations.
- Semantic embedding: Semantic embeddings separate different instances and pull embeddings referring to the same human or object together for more accurate matching.Embeddings from the instance and interaction branches are compared, with a thresholded push objective for distinct instances and a pull objective for corresponding instances.
4. Experiments
Experiments evaluate AS-Net on three HOI detection benchmarks and through ablations of matching, semantic embeddings, branch structure, and attention. AS-Net achieves strong benchmark performance, while the ablations support the contributions of its components.
- Datasets and Metrics: AS-Net is evaluated on HICO-DET, V-COCO, and HOI-A using mean average precision as the evaluation metric.HICO-DET contains 600 HOI triplet classes, while V-COCO and HOI-A provide additional benchmark settings.
- Comparing to State-of-the-art: 31% relative improvement over the previous state-of-the-art one-stage method is achieved on HICO-DET with a ResNet-50 backbone.With frozen MS-COCO-pretrained instance parameters, AS-Net* reaches 24.40% mAP and exceeds existing two-stage methods by more than 3% mAP.
- Comparing to State-of-the-art: 53.9% mAP is achieved on V-COCO, outperforming previous works when instance-detection parameters pretrained on MS-COCO are frozen.The authors also evaluate matching without semantic embeddings because V-COCO’s smaller scale may limit their representation capability.
- Comparing to State-of-the-art: 72.19% mAP is achieved on HOI-A, exceeding previous methods including one using the heavier Hourglass-104 backbone.The comparison concerns the HOI-A test set.
- Ablation Study: Using only vector or semantic-embedding matching compromises effectiveness, while embedding dimension changes from 4 to 32 alter mAP by only 0.66 point.The best embedding-loss weight is λemb = 0.1; increasing or decreasing it impairs effectiveness.
- Ablation Study: The single-branch variant reaches 25.91% mAP, 2.96% below AS-Net, while its Rare mAP is 17.88%, 6.37% below AS-Net.The authors attribute this behavior to detection and interaction relying on different features.
- Ablation Study: The basic two-branch model reaches 27.52% mAP, and instance-aware attention improves mAP by 0.44 point without embeddings and 1.12 points with embeddings.The results support the value of instance-aware features for interaction prediction and indicate that semantic embeddings contribute less than the attention module.
- Qualitative Results: Qualitative visualizations show that branch aggregation shifts attention from scattered or irrelevant regions toward the umbrella and nearby body parts relevant to the interaction.The instance-aware attention separately focuses on the girl’s body and head.