Source-linked AI summary
Learning to Detect Human-Object Interactions
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, Jia Deng
TL;DR
HOI detection asks whether interactions occur and where, by predicting linked human and object boxes with an interaction label. The paper introduces HICO-DET and HO-RCNN, using Interaction Patterns to encode spatial relations. Experiments show improved HOI detection performance over baselines, while proposal recall and quadratic growth constrain the approach.
Problem
HOI recognition lacked broad, localized evaluation: prior datasets had limited categories, while HICO classification did not provide annotations for grounding interactions to image regions.
Method
The paper augments HICO with instance annotations to create HICO-DET and uses HO-RCNN, a multi-stream detector whose Interaction Pattern encodes human-object spatial relations.
Results
HO-RCNN significantly improves HOI detection performance over baseline approaches, with Interaction Patterns achieving 7.30% mAP in Default and 10.37% mAP in Known Object on the full dataset.
Takeaways & Limitations
HICO-DET provides a large benchmark for localized HOI detection, and spatial relations are useful cues for distinguishing interactions involving the same object category.
Takeaways & Limitations
Proposal recall can be low, while increasing human and object detections raises human-object proposals quadratically and can make evaluating all proposals infeasible.
Abstract
from arXiv · showhide
We study the problem of detecting human-object interactions (HOI) in static images, defined as predicting a human and an object bounding box with an interaction class label that connects them. HOI detection is a fundamental problem in computer vision as it provides semantic information about the interactions among the detected objects. We introduce HICO-DET, a new large benchmark for HOI detection, by augmenting the current HICO classification benchmark with instance annotations. To solve the task, we propose Human-Object Region-based Convolutional Neural Networks (HO-RCNN). At the core of our HO-RCNN is the Interaction Pattern, a novel DNN input that characterizes the spatial relations between two bounding boxes. Experiments on HICO-DET demonstrate that our HO-RCNN, by exploiting human-object spatial relations through Interaction Patterns, significantly improves the performance of HOI detection over baseline approaches.
1. Introduction
HOI recognition aims to capture relationships between people and objects, but existing evaluations lacked broad interaction coverage and precise localization. This paper defines HOI detection, introduces HICO-DET, and proposes HO-RCNN with Interaction Patterns.
- Motivation: HOI recognition identifies relationships between people and objects, providing deeper scene semantics than object recognition or detection alone.Without HOI recognition, systems can identify objects and proximity but not actions such as swinging a baseball bat.
- Limitations of prior benchmarks: Existing approaches were evaluated on small datasets with limited HOI categories and interaction diversity, making object-only recognition difficult to rule out.PASCAL VOC had 10 categories, while Stanford 40 Actions had 40 categories and only one car-related HOI category.
- From classification to detection: HICO expanded HOI classification to 600 categories over 80 common objects, but image-level labels could not localize interactions in images.The paper therefore proposes detecting human-object interactions in static images.
- HOI detection: HOI detection predicts a person bounding box, an object bounding box, and the interaction class connecting them.Unlike conventional object detection, it estimates both interaction presence and location.
- Contributions: The paper introduces HICO-DET and HO-RCNN, whose Interaction Pattern input characterizes spatial relations between two bounding boxes.Experiments report that exploiting human-object spatial relations through Interaction Patterns significantly improves HOI detection over baseline approaches.
2. Related Work
Prior work addressed HOI-related recognition, object detection, and grounding, but did not directly evaluate HOI detection. HICO-DET adds instance annotations and enables precision-sensitive evaluation for this task.
- HOI recognition: Earlier HOI studies were evaluated on action classification, object detection, or human pose estimation rather than directly on HOI detection.HICO provided a large image dataset for HOI classification, but lacked ground-truth annotations for evaluating HOI detection.
- HOI detection: HICO-DET augments HICO with annotations that support evaluating localized human-object interactions.The benchmark addresses the absence of ground-truth instance annotations for HOI detection.
- Evaluation: Exhaustive labeling of instances for each positive HOI label enables evaluation with mean Average Precision rather than recall alone.Other datasets discussed in the paper allowed detection evaluation with recall but not precision.
- Object detection: Standard object detectors produce class-specific boxes for individual objects, whereas HOI detection localizes the individual objects in each interacting pair.The paper distinguishes HOI detection from visual-phrase detection, which treats an interacting object pair as one unit.
- Grounding text descriptions to images: HOI detection also grounds interactions, in addition to people and objects, to image regions.This connects HOI detection to work grounding textual entities such as nouns and pronouns in images.
3. HO-RCNN
HO-RCNN detects human-object interactions by pairing human and object proposals, then combining visual and spatial evidence through a multi-stream network. Its pairwise stream uses Interaction Patterns to represent relative bounding-box configurations, with translation normalization and multi-label training.
- Human-Object Proposals: HO-RCNN first generates human-object proposals by pairing detected humans with detected objects relevant to each HOI category.This avoids the quadratic explosion from pairing all class-agnostic candidate boxes.
- Multi-stream Architecture: The network extracts complementary features from detected humans, objects, and their pairwise spatial relations.Human and object streams use cropped bounding-box regions, while the pairwise stream encodes spatial configuration.
- Pairwise Stream: Interaction Patterns encode a human-object proposal as a two-channel binary image whose channels mark the human and object bounding boxes.This representation lets the DNN learn 2D filters for recurring spatial configurations without using pixel values.
- Pairwise Stream: The attention window removes pixels outside the tightest window enclosing both boxes, making pairwise representations invariant to joint translations.The pattern is then normalized while preserving the attention window’s aspect ratio.
- Training: HO-RCNN treats HOI recognition as multi-label classification because a person can perform multiple actions toward the same object concurrently.It applies a binary classifier for each HOI class and sums stream scores separately for each class.
4. Constructing HICO-DET
HICO-DET augments image-level HICO labels with instance-level human-object pair annotations for HOI detection. Its annotation procedure identifies participating people and objects, links their interactions, and records edge cases such as invisible entities and shared participants.
- Annotation Procedure: Annotators use a three-step procedure: draw involved people, draw involved objects, and link each interacting person-object pair.The procedure explicitly excludes people and objects not involved in the described interaction.
- Annotation Procedure: Complex one-versus-many and many-versus-one interactions are decomposed into multiple person-object interaction pairs.The stated detection goal is to recover all decomposed pairs in such cases.
- Annotation Issues: 1.33% of 90,641 annotation tasks were marked invisible because an involved person or object could not be seen despite an inferable HOI.Two of the 600 HOI classes consequently had no training instances under the retained split.
- Dataset Statistics: HICO-DET contains 151,274 instances, averages 1.67 instances per positive HOI label, and includes 256,672 bounding boxes because instances can share people or objects.Two additional images were added to ensure at least one training instance for every HOI class.
5. Experiments
Experiments evaluate HOI detection under Known Object and Default settings, compare pairwise-stream variants and baselines, and analyze qualitative behavior and proposal recall. Interaction Patterns with a convolutional pairwise stream provide the strongest reported performance, while proposal recall limits remain a bottleneck.
- Evaluation Setup: Known Object evaluates images containing the target object, whereas Default evaluates the full test set including images without it.
- Training Setup: HO-RCNN training pairs the top 10 human detections with the top 10 detections for each object category, producing 100 proposals per object category per image.
- Ablation Study: 7.30% Default mAP and 10.37% Known Object mAP are achieved by HO+IP1 (conv) on the full dataset.
- Ablation Study: Interaction Patterns with the convolutional architecture significantly outperform both the human-object baseline and the 2D vector baseline across paired category-level tests.
- Qualitative Analysis: Average Interaction Patterns distinguish interactions involving the same object category, such as sitting on versus carrying a chair.
- Error Analysis: 46.75% mean recall with 10 human and object detections helps explain low mAP, while adding detections increases proposals quadratically and can make evaluation infeasible.
6. Conclusion
The paper addresses HOI detection in static images by introducing a benchmark with instance annotations and a region-based framework centered on spatial relations. Experiments report improved HOI detection performance over baseline approaches.
- HICO-DET augments HICO with instance annotations for human-object pairs across 600 HOI categories.
- HO-RCNN extends region-based detection to human-object interaction detection using Interaction Patterns to characterize spatial relations between bounding boxes.
- Experiments show that HO-RCNN significantly improves HOI detection performance over baseline approaches.
A.1. Pairwise Stream
The pairwise stream compares fully connected and convolutional architectures that consume Interaction Patterns and output scores for K HOI classes. The comparison controls parameter count and training scheme.
- Pairwise Stream: The pairwise stream has fully connected and convolutional variants.
- Pairwise Stream: Both variants take an Interaction Pattern as input and output a vector of classification scores on K HOI classes.
- Pairwise Stream: The variants use approximately the same number of parameters and identical training schemes for fair comparison.