Source-linked AI summary
Pose-aware Multi-level Feature Network for Human Object Interaction Detection
Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, Xuming He
TL;DR
HOI detection is challenged by varied human-object configurations, co-occurring relations, and subtle differences between interaction categories. The paper proposes PMFNet, a pose-guided multi-level network, and reports state-of-the-art performance on V-COCO and HICO-DET with interpretable attention outputs.
Problem
HOI detection must localize and classify human-object-relation triplets despite large configuration variation, co-occurring relations, and subtle differences between similar relations.
Method
PMFNet uses human pose for global spatial configuration and semantic-part attention, reasoning over interaction, visual-object, and human-part representations with a multi-branch network.
Results
The method achieves state-of-the-art performance on both V-COCO and HICO-DET; on V-COCO, it reaches 52.0 mAP, improving its baseline by 3.4 mAP.
Takeaways & Limitations
Pose-guided multi-level reasoning supports robust, fine-grained HOI predictions with interpretable relation affinity and part-attention outputs.
Takeaways & Limitations
The object detector Faster R-CNN and pose estimator CPN are external modules and do not participate in the learning process.
Abstract
from arXiv · showhide
Reasoning human object interactions is a core problem in human-centric scene understanding and detecting such relations poses a unique challenge to vision systems due to large variations in human-object configurations, multiple co-occurring relation instances and subtle visual difference between relation categories. To address those challenges, we propose a multi-level relation detection strategy that utilizes human pose cues to capture global spatial configurations of relations and as an attention mechanism to dynamically zoom into relevant regions at human part level. Specifically, we develop a multi-branch deep network to learn a pose-augmented relation representation at three semantic levels, incorporating interaction context, object features and detailed semantic part cues. As a result, our approach is capable of generating robust predictions on fine-grained human object interactions with interpretable outputs. Extensive experimental evaluations on public benchmarks show that our model outperforms prior methods by a considerable margin, demonstrating its efficacy in handling complex scenes.
1. Introduction
HOI detection must handle varied human-object configurations, co-occurring relations, and subtle differences between similar interactions. PMFNet addresses these challenges with pose-guided reasoning across interaction, object, and human-part levels, achieving state-of-the-art benchmark performance.
- HOI detection localizes and classifies human-object-relation triplets, but remains difficult because configurations vary, relations co-occur, and categories can be visually similar.
- Object-level reasoning can miss context, confuse related human-object pairs, and fail to distinguish fine-grained actions such as ‘hold’ and ‘catch’.
- PMFNet uses human pose to capture global relation configuration and guide local feature extraction at the semantic-part level.
- The model reasons at interaction, visual-object, and human-part levels, combining context, appearance, relative position, part attention, and fine-level spatial configuration.
- The approach achieves state-of-the-art performance on both V-COCO and HICO-DET and produces interpretable outputs through relation affinity and part attention.
2. Related Work
Related work covers visual relationship detection, human-object interaction datasets and methods, and attention mechanisms. The paper distinguishes its approach by using pose-guided fine-grained features to capture subtle differences between similar interactions.
- Visual relationship detection jointly detects objects and describes their interactions, whereas this work focuses specifically on human-centric human-object interactions.
- HOI research uses datasets such as V-COCO and HICO-DET and combines human, object, spatial, and pose information through multi-stream approaches.
- Prior pose-based methods use pose mainly as a spatial constraint between human parts and objects rather than to extract zoom-in features.
- The proposed fine-grained feature use captures subtle differences between similar interactions.
- Attention mechanisms highlight informative regions or parts while suppressing irrelevant global information across several vision tasks.
3. Method
PMFNet detects human-object interactions by combining proposal-level context, pose-augmented object features, and attention-weighted human-part cues. Its multi-branch design supports fine-grained relation scoring from holistic and local representations.
- Problem setup: PMFNet first generates human-object proposals, spatial configurations, and human pose inputs before classifying relations for each proposal pair.The pipeline uses an object detector for proposals and an off-the-shelf pose estimator for each human proposal.
- Fusion module: The fusion module combines object-level and part-level representations to determine fine-grained relation scores for the human-object proposal.The architecture contains backbone, holistic, zoom-in, and fusion modules, with the backbone producing image features and proposals.
- Holistic module: The holistic module combines human, object, union, and spatial branches to encode object-level appearance, relation context, pose, and relative location.Human, object, and union features are RoI-aligned from the convolutional feature map, while the spatial branch uses the spatial configuration map.
- Zoom-in module: The zoom-in module extracts fine-grained human-part and object features, aligns their spatial information, and models local interaction patterns.Part regions are cropped around pose joints, combined with object features, and enriched with relative coordinate maps.
- Zoom-in module: Semantic attention assigns each human part a normalized weight, enhances interaction-relevant features, and treats the object as uniformly important with attention value 1.The attention network uses two fully connected layers with ReLU and Sigmoid activations, then weights the part features element-wise.
4. Experiments
Experiments on V-COCO and HICO-DET evaluate the pose-aware multi-level framework through benchmark comparisons, ablations, and qualitative visualizations. The model achieves state-of-the-art performance, with improvements linked to pose configuration, part-level features, spatial alignment, semantic attention, and interaction affinity.
- Experimental Setting: The framework is evaluated on V-COCO and HICO-DET using mean average precision, with true positives requiring both human and object box IoUs above 0.5 and correct HOI classification.V-COCO contains 26 action categories, while HICO-DET contains 600 HOI categories over 80 object categories.
- Quantitative Results: 52.0 mAP on V-COCO, outperforming current state-of-the-art methods and improving the baseline by 3.4 mAP.The baseline achieves 48.6 mAP and includes human, object, and union branches from the holistic module.
- Quantitative Results: 17.46 mAP and 20.34 mAP on HICO-DET Default and Know Object categories, respectively, outperforming existing methods.The full model further improves its baseline by 2.54 mAP on Default and 1.51 mAP on Know Object.
- Quantitative Results: The model improves more on Interactiveness than No-interaction HOIs, indicating stronger gains for fine-grained interaction classification.The Interactiveness group contains 520 non-trivial classes, whereas No-interaction contains 80 background classes.
- Ablation Study: Ablations show that SCM improves the baseline by 0.7 mAP, part-crop raises mAP from 49.9 to 51.0, and spatial alignment raises it from 51.0 to 52.4.Semantic attention and interaction affinity each contribute an additional 0.3 mAP, while independent-component removal confirms each component contributes to final performance.
- Qualitative Visualization Results: Qualitative results show more confident predictions for difficult HOIs with very small target objects, while semantic attention highlights interaction-relevant human parts.Attention focuses on full-body joints for chair interactions and hands for instrument interactions, producing interpretable outcomes.
5. Conclusion
The proposed multi-level reasoning approach combines interaction-, object-, and human-part-level features under human-pose guidance for HOI detection. It recognizes subtle relation differences, achieves state-of-the-art performance on V-COCO and HICO-DET, and outperforms other approaches by a large margin on V-COCO.
- The method combines interaction-level, visual object-level, and human-part-level features using human pose information.A multi-branch deep neural network instantiates this multi-level reasoning approach.
- A semantic part-based attention mechanism automatically extracts relevant human parts for each interaction instance.Its attention maps provide interpretable outputs for human-object relation detection.
- The approach recognizes visual relations with subtle differences.
- The method achieves state-of-the-art performance on both V-COCO and HICO-DET benchmarks.
- The approach outperforms other methods by a large margin on the V-COCO dataset.