Source-linked AI summary
HOTR: End-to-End Human-Object Interaction Detection with Transformers
Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, Hyunwoo J. Kim
TL;DR
HOI detection must localize interacting humans and objects and classify their interactions, but prior methods infer object pairs indirectly and rely on costly post-processing. HOTR directly predicts triplet sets with a transformer encoder-decoder, achieving state-of-the-art results on two benchmarks with under-1-ms inference after object detection. Its rare-category performance remains below some baselines, while external prior information may improve that setting.
Problem
Existing HOI detectors indirectly infer interactions from detected human-object pairs, requiring post-processing and leaving high-level interaction dependencies insufficiently explored.
Method
HOTR uses transformer-based direct set prediction with parallel instance and interaction decoders, HO Pointers, recomposition, and end-to-end set-based matching.
Results
HOTR achieves state-of-the-art performance on V-COCO and HICO-DET with inference time under 1 ms after object detection.
Takeaways & Limitations
Direct set prediction eliminates hand-crafted post-processing while modeling correlations between interactions in HOI detection.
Takeaways & Limitations
HOTR underperforms baseline methods in HICO-DET’s Rare setting, where categories have fewer than 10 training instances; external prior information may improve performance.
Abstract
from arXiv · showhide
Human-Object Interaction (HOI) detection is a task of identifying "a set of interactions" in an image, which involves the i) localization of the subject (i.e., humans) and target (i.e., objects) of interaction, and ii) the classification of the interaction labels. Most existing methods have indirectly addressed this task by detecting human and object instances and individually inferring every pair of the detected instances. In this paper, we present a novel framework, referred to by HOTR, which directly predicts a set of <human, object, interaction> triplets from an image based on a transformer encoder-decoder architecture. Through the set prediction, our method effectively exploits the inherent semantic relationships in an image and does not require time-consuming post-processing which is the main bottleneck of existing methods. Our proposed algorithm achieves the state-of-the-art performance in two HOI detection benchmarks with an inference time under 1 ms after object detection.
1. Introduction
HOTR formulates HOI detection as direct set prediction of human-object-interaction triplets, replacing indirect pairwise inference and hand-crafted post-processing with a transformer encoder-decoder. It achieves state-of-the-art performance on two benchmarks with inference under 1 ms.
- Previous HOI detectors first detect objects, then associate human-object pairs through separate post-processing, making sequential approaches time-consuming and computationally expensive.
- Parallel detectors reduce inference cost using heuristic matching, but still require duplicate suppression, thresholding, and leave high-level interaction dependencies underexplored.
- HOTR directly predicts a fixed set of human-object-interaction triplets with a transformer encoder-decoder and end-to-end set loss matching predictions to ground truth.
- HOTR uses HO Pointers, recomposition, and a new loss function to associate parallel instance and interaction decoder outputs into final triplets.
- HOTR achieves state-of-the-art performance on V-COCO and HICO-DET with inference under 1 ms, compared with 5∼9 ms for previous parallel detectors.
2. Related Work
Prior HOI work separates sequential pairwise interaction inference from faster parallel heuristic matching, while transformer set prediction offers a framework for fixed-size structured outputs. HOTR’s pipeline applies parallel instance and interaction decoders with shared encoding.
- Sequential HOI methods detect objects first and use a separate neural network to infer interactions for every detected object pair.
- Parallel HOI detectors localize interactions directly and associate them with detected objects using distance or IoU heuristics for faster inference.
- DETR predicts a fixed-size set of N objects in one decoder pass and trains with optimal bipartite matching between predictions and ground truth.
- HOTR’s pipeline shares an encoder between parallel Instance and Interaction Decoders, then associates their representations to produce fixed-set HOI triplets.
3. Method
HOTR formulates HOI detection as direct set prediction with a shared transformer encoder and parallel instance and interaction decoders. HO Pointers associate interaction representations with detected instances, while recomposition and Hungarian matching produce and train the final HOI set.
- HOI Detection as Set Prediction: HOTR extends transformer set prediction to HOI detection by predicting human localization, object localization, and multi-label interaction types for each set element.The straightforward extension would redundantly predict the same object localization across multiple interaction queries.
- Transformer Encoder-Decoder architecture: The architecture uses a shared encoder with parallel instance and interaction decoders, whose outputs are associated to generate final HOI triplets.Instance queries produce object-detection representations, while interaction queries produce interaction representations.
- HO Pointers: HO Pointers localize each interaction’s human and object by selecting indices of instance representations rather than directly regressing bounding boxes.Human and object pointer vectors are compared with normalized instance representations using similarity scores, and the highest-scoring indices are selected.
- Recomposition for HOI Set Prediction: Recomposition applies bounding-box regression and action-classification networks to interaction representations to form a set of K final HOI triplets.The triplets combine predicted human and object boxes, pointer-selected instances, and interaction labels.
- Complexity & Inference time: HOTR associates K interactions with N instances in O(KN) time and removes interaction-region NMS and triplet-matching post-processing.The method is designed to reduce inference cost after object detection by replacing prior post-processing stages with pointer-based association.
- Training HOTR: Training uses Hungarian matching between predicted and ground-truth HOI triplets, with ground-truth boxes converted to instance indices for pointer-based costs.The matching operates on K predictions and ground-truth triplets padded with no-interaction entries.
- Defining No-Interaction with HOTR: HOTR adds an explicit No-Interaction class to suppress redundant predictions for human-object pairs without an interaction.This addresses the absence of a suppressive no-object class in multi-label action classification.
4. Experiments
HOTR is evaluated on V-COCO and HICO-DET using fixed evaluation protocols, achieving state-of-the-art results while reducing interaction-prediction inference time. Ablations examine its pointers, shared encoder, and interactiveness suppression, while rare-category performance remains a limitation.
- 4.1. Datasets: HOTR is evaluated on the V-COCO and HICO-DET benchmark datasets using official evaluation code and fixed or fine-tuned object detectors.V-COCO results use a fixed detector, while HICO-DET results are reported with both frozen MS-COCO weights and a detector fine-tuned on HICO-DET.
- 4.2. Quantitative Analysis: HOTR achieves state-of-the-art performance on both V-COCO and HICO-DET and is the fastest parallel detector.It exceeds the state-of-the-art parallel detector by 4.2mAP on V-COCO Scenario1, and by 4.1mAP and 4mAP under the two HICO-DET detector conditions.
- 4.2. Quantitative Analysis: HOTR outperforms the state-of-the-art sequential detector by 1.9 mAP and 4.1 mAP in V-COCO Scenario1 and Scenario2, respectively.On HICO-DET Default(Full), HOTR shows comparable performance with a margin of 0.1∼0.52 mAP despite using only visual information.
- 4.2. Quantitative Analysis: HOTR underperforms baseline methods in the HICO-DET Rare setting, where action categories have fewer than 10 training instances.The authors note that high Rare-setting performance commonly uses external spatial, pose, linguistic, or coherence features, whereas HOTR is vision-based.
- 4.2. Quantitative Analysis: 0.9ms is the average time for recomposition and final HOI triplet inference, while HOTR runs ×5 ∼×9 faster than state-of-the-art parallel HOI detectors.The reported analysis excludes object-detection time and attributes the speedup to replacing explicit post-processing with an O(KN) HO Pointer search.
- 4.3. Ablation Study: Removing HO Pointers changes performance from 55.2 to 39.3 in V-COCO and from 23.5 to 17.2 in HICO-DET.The ablation compares pointer-based localization with directly regressing bounding boxes for each interaction.
- 4.3. Ablation Study: Sharing the encoder outperforms separate encoders by 21.3mAP in V-COCO and 9.0mAP in HICO-DET.The authors suppose that a shared encoder helps the decoders learn common visual patterns and share overall context through HO Pointers.
- 4.3. Ablation Study: An explicit interactiveness class contributes 3mAP to the final performance by suppressing redundant low-probability pairs.This mechanism is used because action classification is multi-label binary classification rather than softmax classification.
5. Conclusion
HOTR achieves state-of-the-art HOI detection on V-COCO and HICO-DET while eliminating hand-crafted post-processing and maintaining inference under 1 ms.
- HOTR achieves state-of-the-art performance on both V-COCO and HICO-DET, with a significant margin over previous parallel HOI detectors.
- The set prediction approach eliminates hand-crafted post-processing steps used by previous HOI detectors.
- HOTR models correlations between interactions through transformer-based set prediction.
- The framework combines HOI decomposition with parallel decoders, similarity-based recomposition during inference, and interactiveness suppression.
- Under 1 ms inference time makes HOTR significantly faster than previous parallel HOI detectors operating at 5∼9 ms.