Source-linked AI summary
Mining the Benefits of Two-stage and One-stage HOI Detection
Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, Xiaobo Li
TL;DR
The paper examines limitations of two-stage and one-stage HOI detectors and proposes CDN, a one-stage cascade that disentangles human-object detection from interaction classification. CDN outperforms previous methods, including a 25.35% gain on rare HICO-Det classes.
Problem
Two-stage methods struggle to locate positive interactive human-object pairs, while one-stage methods struggle to balance object detection and interaction classification.
Method
CDN uses a cascade framework that separates human-object detection from interaction classification within a one-stage HOI detector.
Results
25.35% performance gain on rare classes of HICO-Det compared with previous state-of-the-art methods.
Takeaways & Limitations
The cascade disentangling paradigm combines direct interactive-pair localization with separated detection and interaction classification.
Takeaways & Limitations
The technique could be used for military purposes or other malicious human-activity detection, potentially negatively impacting society.
Abstract
from arXiv · showhide
Two-stage methods have dominated Human-Object Interaction (HOI) detection for several years. Recently, one-stage HOI detection methods have become popular. In this paper, we aim to explore the essential pros and cons of two-stage and one-stage methods. With this as the goal, we find that conventional two-stage methods mainly suffer from positioning positive interactive human-object pairs, while one-stage methods are challenging to make an appropriate trade-off on multi-task learning, i.e., object detection, and interaction classification. Therefore, a core problem is how to take the essence and discard the dregs from the conventional two types of methods. To this end, we propose a novel one-stage framework with disentangling human-object detection and interaction classification in a cascade manner. In detail, we first design a human-object pair generator based on a state-of-the-art one-stage HOI detector by removing the interaction classification module or head and then design a relatively isolated interaction classifier to classify each human-object pair. Two cascade decoders in our proposed framework can focus on one specific task, detection or interaction classification. In terms of the specific implementation, we adopt a transformer-based HOI detector as our base model. The newly introduced disentangling paradigm outperforms existing methods by a large margin, with a significant relative mAP gain of 9.32% on HICO-Det. The source codes are available at https://github.com/YueLiao/CDN.
1 Introduction
The paper contrasts two-stage and one-stage HOI detectors, identifying distinct weaknesses in pair localization and multi-task learning. It proposes CDN, a cascade framework that disentangles detection from interaction classification.
- HOI detection must locate human-object pairs and classify their corresponding actions from static images.
- Conventional limitations: Two-stage methods detect humans and objects first, then classify matched pairs, but struggle to locate interactive pairs amid many negative pairs.
- Conventional limitations: One-stage methods directly detect HOI triplets end-to-end, but must balance human-object detection and interaction classification within one model.
- Cascade Disentangling Network: CDN uses a cascade disentangling head to separate human-object pair detection from interaction classification while retaining a one-stage framework.
- Results: 25.35% performance gain is achieved on rare classes of HICO-Det over previous state-of-the-art methods.
2 Analysis of Two-stage and One-stage HOI detectors
The analysis explains that two-stage detectors generate and classify many candidate pairs, creating computational and representation problems, while one-stage detectors reduce these issues but couple incompatible tasks. CDN addresses this trade-off with cascade decoders for pair prediction and action classification.
- Problem formulation: The HOI task predicts triplets containing human and object bounding boxes together with an action category.
- Two-stage HOI detector: Two-stage detectors separately perform instance detection and interaction classification, generating M × N human-object pairs before classification.
- Two-stage HOI detector: Two-stage classification requires O(M × N) inference, is exposed to positive-negative imbalance, and relies on region features optimized for box regression.
- One-stage HOI detector: One-stage detectors directly predict HOI triplets with O(K′) complexity and reduce negative-pair interference, but coupling detection with classification limits unified feature learning.
- Cascade Disentangling Network: CDN decomposes one-stage optimization into cascade decoders that first predict human-object pairs and then classify actions with an isolated decoder.
3 Method
CDN uses a CNN-transformer feature extractor followed by cascade decoders that disentangle human-object pair detection from interaction classification. It additionally applies decoupled dynamic re-weighting to address long-tailed object and action classes during training.
- Visual Feature Extractor: CDN extracts sequenced visual features with a CNN followed by a transformer encoder before HOI decoding.The visual feature extractor combines convolutional features with transformer-based global contextualization.
- Human-Object Pair Decoder: The Human-Object Pair Decoder predicts human and object boxes, object classes, and an interactive-pair score from learnable queries and visual features.Its heads estimate the two bounding boxes, object category, and binary interaction status for each query.
- Interaction Decoder: The Interaction Decoder initializes interaction queries from HO-PD outputs and assigns one or more action categories to each corresponding human-object query.The one-to-one query correspondence transfers pair information into the isolated action-classification stage.
- Cascade Disentangling HOI Decoder: The cascade disentangling decoder separates human-object pair detection from interaction classification so each step can aggregate task-relevant features.The two transformer-based decoders have independent weights and focus on their respective tasks.
- Decoupling Dynamic Re-weighting: CDN mitigates long-tailed distributions by freezing the visual extractor and training the decoders with dynamic re-weighted losses after regular training.The final weights blend static and dynamically computed coefficients using a smooth transition factor.
- Inference: During inference, one-to-one corresponding instance and interaction queries combine box, object, interaction, and action outputs into HOI triplets.The cascade outputs are composited across homologous query dimensions to form final predictions.
4 Experiments
Experiments evaluate CDN on HICO-Det and V-COCO, compare two-stage and one-stage designs, visualize feature learning, and test proposed strategies. CDN consistently improves HOI detection, including rare classes and settings with similar computational cost.
- Experimental Setup: Experiments use HICO-Det and V-COCO with standard mAP-based evaluation and benchmark-specific test protocols.HICO-Det contains 600 HOI classes; evaluation also reports performance on 138 rare classes.
- Two-stage and One-stage Analysis: CDN-S base reaches mAP 30.96, a 6.50% relative gain over QPIC, with a 23.66% improvement on rare classes.The rare-class result is used as a partial indicator of interaction-classification accuracy.
- Feature Learning: CDN separates pair detection from interaction classification: HO-PD attends to human-object boundaries, while the interaction decoder attends to pose and action-relevant context.For “hold cake,” HO-PD focuses on cake boundaries while the interaction decoder emphasizes the hands holding the cake; analogous separation appears for “ride horse.”
- Comparison to State-of-the-Art: CDN-B improves HICO-Det mAP from 20.41 to 31.78 over FCMNet, a 55.71% relative gain, and exceeds QPIC by a 9.32% point relative gain.It also reports 62.29 AProle on V-COCO Scenario 1 and 64.42 AProle on Scenario 2, with relative gains of 5.94% and 5.61% over QPIC.
- Efficiency Analysis: CDN-S achieves mAP 31.44 on HICO-Det, 8.15% higher than QPIC, with almost the same parameter count and FLOPs.This comparison supports the efficiency analysis of the small CDN variant.
- Ablation Study: On HICO-Det, the base model reaches mAP 31.06, re-weighting raises it to 31.38, and PNMS further improves it to 31.78.The base model raises rare-class mAP from QPIC’s 21.85 to 26.68; dynamic re-weighting and PNMS provide subsequent gains.
5 Related Work
Related work divides HOI detection into two-stage methods that detect instances before classifying matched pairs and one-stage methods that detect HOI triplets directly.
- Two-stage Methods: Two-stage methods first detect human and object instances, generate matched pairs, and feed them into an interaction classifier.
- One-stage Methods: One-stage methods detect HOI triplets directly, including point-based, anchor-based, and set-based approaches.
6 Conclusion
The conclusion presents CDN as a cascade framework that disentangles detection from interaction classification while retaining direct interactive-pair localization. It reports improved performance but limits the current implementation to one specific version and identifies broader extensions as future work.
- CDN combines one-stage interactive-pair localization with two-stage disentangling of human-object detection and interaction classification.
- CDN outperforms previous methods by margins on the reported HOI detection evaluations.
- The study implements only one specific version of the proposed approach.
- Future work will apply the idea to more general one-stage methods and introduce additional two-stage advantages into the one-stage framework.
Potential Negative Societal Impacts
The authors state that CDN itself is harmless but could be used for military purposes or malicious human-activity detection, potentially negatively affecting society.
- CDN could be repurposed for military purposes or malicious human-activity detection with potentially negative societal effects.