Source-linked AI summary
HOI Analysis: Integrating and Decomposing Human-Object Interaction
Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, Cewu Lu
TL;DR
HOI detection must represent human, object, and an implicit verb without relying only on direct pixel-to-semantics mapping. The paper introduces HOI Analysis and IDN, which decompose and reintegrate HOI while learning verbs in transformation-function space, achieving state-of-the-art benchmark performance.
Problem
HOI contains an implicit verb that is difficult to represent directly in image space, motivating alternatives to direct pixel-to-HOI semantic mapping.
Method
HOI Analysis decomposes coherent HOI into isolated human and object, reintegrates them, and uses IDN to learn verb representations in transformation function space.
Results
IDN achieves state-of-the-art HOI detection performance, including 23.36 mAP on HICO-DET Default Full with a COCO detector.
Takeaways & Limitations
Integration and decomposition provide a framework for learning interaction representations and transforming human-object pairs with the same HOI.
Takeaways & Limitations
The inter-pair transformation scope excludes transformations between different HOIs and leaves more sophisticated transformations for future work.
Abstract
from arXiv · showhide
Human-Object Interaction (HOI) consists of human, object and implicit interaction/verb. Different from previous methods that directly map pixels to HOI semantics, we propose a novel perspective for HOI learning in an analytical manner. In analogy to Harmonic Analysis, whose goal is to study how to represent the signals with the superposition of basic waves, we propose the HOI Analysis. We argue that coherent HOI can be decomposed into isolated human and object. Meanwhile, isolated human and object can also be integrated into coherent HOI again. Moreover, transformations between human-object pairs with the same HOI can also be easier approached with integration and decomposition. As a result, the implicit verb will be represented in the transformation function space. In light of this, we propose an Integration-Decomposition Network (IDN) to implement the above transformations and achieve state-of-the-art performance on widely-used HOI detection benchmarks. Code is available at https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/IDN-(Integrating-Decomposing-Network).
1 Introduction
The paper reframes HOI as a structured composition of human, object, and implicit verb, asking how isolated elements form interactions and how same-HOI pairs relate. Inspired by Harmonic Analysis, it proposes HOI Analysis and IDN to model these transformations.
- Motivation: HOI detection must jointly locate human and object while classifying their implicit verb.The paper represents HOI as <human, verb, object> and identifies verb localization and semantic mapping as central challenges.
- Motivation: The paper studies how isolated humans and objects compose coherent HOI and how human-object pairs with the same HOI relate.These are presented as the two central structural questions of HOI Analysis.
- HOI Analysis: HOI Analysis decomposes coherent HOI into isolated human and object, then integrates them back while modeling verbs in transformation function space.The framework is motivated by Harmonic Analysis and uses transformation functions rather than directly mapping pixels to HOI semantics.
- Contributions: IDN implements HOI Analysis through integration and decomposition transformations and achieves state-of-the-art HOI detection performance.The stated contributions include the HOI Analysis paradigm, the Integration-Decomposition Network, and its benchmark performance.
2 Related Work
Prior HOI work uses learned visual features, interactiveness estimation, multimodal representations, 3D cues, and knowledge bases. This paper connects HOI learning to compositional representation and human-perception findings while emphasizing that verbs are implicit and difficult to localize.
- Prior HOI methods: Existing HOI methods span end-to-end learned features, interactiveness filtering, visual-language alignment, 3D representations, and knowledge-base assistance.The related-work landscape includes methods addressing pair selection, unseen HOIs, spatial structure, and fine-grained action semantics.
- Positioning: HOI Analysis is presented as a new representation-learning perspective that studies HOI structure and same-HOI pair relationships.Figure 2 describes decomposition, integration, and inter-pair transformation as the framework's core operations.
- Human perception: Human-perception research reports higher-level HOI representations that integrate isolated human and object information rather than simply summing them.The paper cites pSTS as a brain region associated with coherent HOI modeling.
- Compositional learning: Attribute-object learning offers compositional analogies, but HOI verbs differ because they are implicit and difficult to localize in images.The paper contrasts verbs with attributes expressed on object appearance.
3 Method
IDN learns HOI through latent-space integration and decomposition, representing implicit verbs with transformation functions and exchanging instances between pairs sharing the same HOI.
- 3.1 Overview: The method prepares coherent union and isolated human/object features from detector boxes, ResNet-50 RoI features, location encodings, and auto-encoder compression.The auto-encoder compresses the coherent and isolated representations to 1024-dimensional features before transformation learning.
- 3.2 Integration and Decomposition: For each verb, paired MLPs integrate isolated human/object features into verb-specific union features and decompose union features back into isolated representations.The transformations are trained so the matching verb-specific output is close while outputs for other verbs are separated.
- 3.3 Inter-Pair Transformation: Inter-pair transformation exchanges humans or objects between pairs with the same HOI, using pose-similar persons and same-category, similarly sized objects as candidates.The implementation uses simple feature replacement rather than strict motion transfer, while transformations across different HOIs remain outside the paper’s scope.
- 3.3 Inter-Pair Transformation: The method does not cover richer transformations such as changing 2D posture, recovering 3D HOI, adjusting 3D pose, or altering interacted-object classes.These extensions are left for future work.
- 3.1 Overview: IDN uses integration and decomposition transformations to model interaction changes and dynamically infer whether an interaction exists within human-object pairs.The network constructs a loop between coherent union features and isolated human/object features, using interactiveness to monitor semantic changes.
4 Experiment
IDN is evaluated on HICO-DET and V-COCO, with comparisons covering detection settings, efficiency, scalability, visualizations, and ablations. The results show strong benchmark performance, effective integration and inter-pair transformation, and complementary contributions from the model components.
- Comparison: 23.36 mAP on HICO-DET Default Full with the COCO detector, while IDN exceeds 20 mAP on all three Default sets without additional information.With the HICO-DET-finetuned detector, performance exceeds 26 mAP; with ground-truth boxes, IDN improves HOI recognition by more than 9 mAP.
- Comparison: 53.3 mAP on V-COCO S1 and 60.3 mAP on S2, both significantly outperforming previous methods.Applying integration and decomposition to iCAN improves HICO-DET Full performance from 14.84 mAP to 18.98 mAP.
- Efficiency and Scalability: 10.04 FPS inference speed keeps IDN runnable on a single GPU, although it is slower than PPDM at 14.08 FPS.All transformations operate in parallel; a single conditioned MLP for all verbs scales to new verbs but reaches only 20.86 mAP versus 23.36 mAP for the reported version.
- Visualization: Integrated T^v_I(f_h⊕f_o) is closer to the real union feature f_u than the simple combination f_h⊕f_o, which cannot represent interaction information.The visualization evaluates f_u, f_h⊕f_o, and transformed features using t-SNE.
- Ablation Study: Removing any validity objective degrades performance, while TI, TD, and AE achieve 21.26, 21.05, and 17.27 mAP respectively and show complementary effects.Removing IPT reduces performance to 22.63 mAP; removing the AE verb-classification loss hurts especially on the Rare set, while reconstruction remains an important auxiliary loss.
5 Conclusion
The paper proposes HOI Analysis and IDN to learn interaction representations through integration and decomposition, achieving state-of-the-art HOI detection performance.
- IDN achieves state-of-the-art HOI detection performance with significant improvements.
Broader Impact
The proposed HOI detection paradigm is intended to promote human activity understanding and support applications such as intelligent-hospital healthcare systems.
- The paradigm could support vision applications such as healthcare systems in intelligent hospitals.The paper notes that current activity-understanding systems can require substantial computational and financial resources.
A Visualized Results
Figure 6 visualizes IDN's HOI detection results on HICO-DET across diverse scenes.
- IDN accurately detects and both decomposes and integrates varied HOIs in diverse HICO-DET scenes.
B Result Analysis
On HICO-DET's Rare set, IDN outperforms Peyre et al. and DJ-RN across various rare HOIs.
- IDN outperforms Peyre et al. and DJ-RN on various rare HOIs in HICO-DET's Rare set.The authors attribute this effectiveness to dynamically learned interaction representations that alleviate rare-HOI data deficiency.
C Code
The paper provides source code for the IDN implementation and includes a performance comparison on the Rare set of HICO-DET.
- C Code: Source code for the IDN implementation is provided through the HAKE-Action-Torch project repository.The repository includes an IDN-specific tree.
- C Code: Figure 7 compares the method with Peyre et al. and DJ-RN on the Rare set of HICO-DET.
- C Code: The reported comparison focuses on performance across three methods evaluated on the Rare HICO-DET subset.