Source-linked AI summary
End-to-End Human Object Interaction Detection with HOI Transformer
Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, Jian Sun
TL;DR
HOI detection struggles to model long-range human–object dependencies, while existing two-stage pipelines are complex and inefficient. HOI Transformer directly predicts HOI instances with a transformer encoder-decoder and unified quintuple matching loss, achieving considerable gains over state-of-the-art methods on two benchmarks.
Problem
HOI detection requires capturing especially long-range human–object dependencies, but existing approaches use complex two-stage or surrogate-proposal strategies.
Method
HOI Transformer uses a transformer encoder-decoder to directly predict HOI instances end to end, supervised by a unified quintuple matching loss.
Results
4.88% point gain over one-stage methods on HICO-DET Full categories, with best performance against two-stage methods without additional human pose features or language priors.
Takeaways & Limitations
The method provides a simpler HOI pipeline that abandons additional features and complex post-processing while dynamically identifying discerning features for different HOI queries.
Takeaways & Limitations
Two-stage alternatives remain vulnerable to sub-optimal separated optimization, low-quality proposals, and the computational burden of processing all human–object pairs.
Abstract
from arXiv · showhide
We propose HOI Transformer to tackle human object interaction (HOI) detection in an end-to-end manner. Current approaches either decouple HOI task into separated stages of object detection and interaction classification or introduce surrogate interaction problem. In contrast, our method, named HOI Transformer, streamlines the HOI pipeline by eliminating the need for many hand-designed components. HOI Transformer reasons about the relations of objects and humans from global image context and directly predicts HOI instances in parallel. A quintuple matching loss is introduced to force HOI predictions in a unified way. Our method is conceptually much simpler and demonstrates improved accuracy. Without bells and whistles, HOI Transformer achieves $26.61\% $ $ AP $ on HICO-DET and $52.9\%$ $AP_{role}$ on V-COCO, surpassing previous methods with the advantage of being much simpler. We hope our approach will serve as a simple and effective alternative for HOI tasks. Code is available at https://github.com/bbepoch/HoiTransformer .
1. Introduction
HOI detection seeks to localize humans and objects while recognizing their interactions, but prior pipelines rely on separated stages or surrogate proposals. HOI Transformer instead uses transformer-based global reasoning to predict HOI instances directly and supervise them with unified matching.
- HOI detection localizes humans and objects while recognizing the interaction between them.
- Two-stage methods separately detect human-object proposals and classify interactions, risking sub-optimal optimization, low-quality proposals, and redundant pairwise computation.
- One-stage methods detect interaction and object proposals simultaneously but still assign results to human-object pairs through predefined matching strategies.
- Capturing especially long-range human-object dependencies in image space is identified as the main HOI detection problem.
- HOI Transformer combines a transformer encoder-decoder with quintuple HOI matching loss to directly predict HOI instances end-to-end.
2. Related work
Related HOI detectors use staged pipelines, auxiliary features, or surrogate interaction proposals, while end-to-end transformer methods provide a simpler parallel prediction paradigm. HOI Transformer applies this paradigm specifically to direct HOI relation prediction without post-processing.
- Two-Stage HOI Detection: Two-stage methods detect humans and objects first, then use multi-stream classifiers to predict interactions for each human-object pair.
- Two-Stage HOI Detection: Auxiliary pose, body-part, language, and graph models can improve two-stage HOI detection but contribute to heavy complexity and low efficiency.
- One-Stage HOI Detection: One-stage methods simplify detection and improve deployment efficiency, yet still require complex post-processing to group object and interaction predictions.
- HOI Transformer: HOI Transformer extends end-to-end parallel prediction to HOI relations, outputting them directly from an image without post-processing.
- End-to-End Detection: End-to-end detection methods replace autoregressive sequence prediction with transformer-based parallel decoding and use Hungarian matching for predictions and ground truths.
3. Method
The method treats HOI detection as direct set prediction, using a transformer encoder-decoder and a unified quintuple matching loss to produce human, object, and interaction predictions end to end.
- Overview: The system directly predicts HOI relations from an image without post-processing, using an end-to-end transformer encoder-decoder and quintuple HOI matching loss.Its architecture includes a CNN backbone, transformer encoder-decoder, and MLP for HOI prediction.
- Network Architecture: The CNN backbone converts image features into a flattened sequence for transformer processing, reducing ResNet conv-5 channels from 2048 to 256.The feature map is reshaped to [H × W, d] so the transformer encoder can process spatial features as a sequence.
- Network Architecture: The transformer encoder summarizes global image information, while the decoder transforms learned HOI queries into output embeddings for HOI instance prediction.The decoder uses global memory, HOI queries, and positional encoding, with cross-attention linking queries to encoded image features.
- HOI Instance Representation: Each HOI instance contains human, interaction, and object confidence values together with human and object bounding boxes.This unified representation contrasts with two-stage detectors that first detect human-object proposals and then classify interactions for enumerated pairs.
- HOI Instance Matching: The model treats HOI detection as bipartite set matching, padding ground-truth sets and assigning fixed-size predictions through an injective one-to-one mapping.Hungarian matching selects the optimal assignment before losses are computed for matched prediction-ground-truth pairs.
- HOI Instance Matching: The proposed matching loss jointly accounts for classification and localization, producing human and object boxes simultaneously with their interactions.This differs from conventional methods that separately optimize an object detector and an interaction classifier.
4. Experiments
Experiments evaluate HOI Transformer on HICO-DET and V-COCO, compare it with prior methods, and study matching, loss weighting, augmentation, generalization, and decoder attention.
- Experimental Setting: The evaluation uses HICO-DET and V-COCO, with role mean average precision as the performance metric.HICO-DET contains 600 HOI categories; V-COCO annotates 29 action categories, and a true positive requires accurate human and object localization plus correct interaction prediction.
- Comparisons with State-of-the-Art methods: 4.88% point gain over one-stage methods on HICO-DET Full categories and 5.37% point gain on Rare categories.The method achieves the best Full-category performance against two-stage methods without additional human pose or language priors.
- Comparisons with State-of-the-Art methods: 1.9% point gain over the previous one-stage method on V-COCO without external pose, HICO-DET, or language-prior data.The reported comparison describes the method as competitive with state-of-the-art methods under this setting.
- Ablation Study: Classification receives greater matching weight than localization, with the best result at β1 = 2.0 and β2 = 1.0.The ablation frames matching as a choice between prioritizing interaction category correctness and human-object box localization.
- Ablation Study: 4.29% point gain comes from multi-scale training and 5.08% from random cropping on HICO-DET Full categories.Their combination performs better still, attributed to easier learning of scale-invariant and shift-invariant features on a small dataset.
- Discussion: The model generalizes to a synthetic image containing 6 ‘drink with bottle’ HOIs although training images contain at most 3.The experiment treats the six-interaction image as out of distribution and reports good HOI recognition generalization.
5. Conclusion
HOI Transformer directly predicts HOI instances end to end using a transformer encoder-decoder and unified quintuple matching loss. The method reports improved performance while removing additional features and complex post-processing.
- HOI Transformer directly predicts HOI instances in an end-to-end manner.
- Its core architecture combines a transformer encoder-decoder with a quintuple matching loss for unified HOI supervision.
- The method achieves a considerable performance boost over state-of-the-art results on two challenging HOI benchmarks.
- The approach abandons additional features and complex post-processing.
- Decoder attention maps indicate that different HOI queries can dynamically attain discerning features.
Appendix A: Illustration of Inference Process
The model outputs multiple HOI instances in parallel, while its inference can be understood autoregressively through global memory and previously detected instances. Decoding continues until a confidence threshold or maximum output limit is reached.
- Parallel decoding simultaneously outputs a group of N HOI instances.
- During training, outputs are matched one-to-one with ground truth before matched-pair losses optimize the network.
- During inference, the network directly outputs N HOI instances without one-to-one matching.
- Each decoder step considers global memory and previously detected HOI instances to produce an undetected instance.
- For a person with horse interactions, decoding can produce “jump horse” first and “ride horse” next after accounting for the previous output.
- Inference stops when confidence falls below a predefined threshold or the maximum number of instances is reached.
- Figure 6 illustrates the decoder inference process, with each step producing a new HOI instance from global memory and prior detections.