Source-linked AI summary
Detecting and Recognizing Human-Object Interactions
Georgia Gkioxari, Ross Girshick, Piotr Dollár, Kaiming He
TL;DR
The paper addresses detecting fine-grained human-object interaction triplets in everyday images, where object recognition alone is insufficient. InteractNet uses person appearance to predict action-specific target locations and jointly trains detection and interaction branches. It reports a 26% relative V-COCO role AP improvement and a 27% relative HICO-DET improvement, while failures include erroneous interaction inferences and confusing actions.
Problem
Recognizing human-object interactions requires detecting ⟨human, verb, object⟩ triplets because understanding images involves relationships beyond individual object recognition.
Method
InteractNet predicts action-specific target-object location densities from human appearance and jointly trains object detection, human-centric, and interaction branches.
Results
26% relative improvement raises V-COCO role AP from 31.8 to 40.0, with a further 27% relative improvement on HICO-DET.
Takeaways & Limitations
Human appearance provides a useful cue for inferring target-object locations and enables efficient human-object interaction detection.
Takeaways & Limitations
Failures include false interaction inferences, assigning another person’s or irrelevant objects, and confusing actions.
Abstract
from arXiv · showhide
To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting <human, verb, object> triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with. To exploit this cue, our model learns to predict an action-specific density over target object locations based on the appearance of a detected person. Our model also jointly learns to detect people and objects, and by fusing these predictions it efficiently infers interaction triplets in a clean, jointly trained end-to-end system we call InteractNet. We validate our approach on the recently introduced Verbs in COCO (V-COCO) and HICO-DET datasets, where we show quantitatively compelling results.
1. Introduction
The paper frames human-object interaction detection as recognizing ⟨human, verb, object⟩ triplets, then proposes InteractNet, a human-centric model that uses person appearance to localize action targets and achieves strong benchmark results.
- Human-object interaction detection requires recognizing relationships between individual instances, not only detecting objects.
- The task is challenging because human actions are fine-grained and interact with a wide range of object types.
- InteractNet predicts action-specific target-object location densities from features extracted from a detected person’s region.
- The model combines a human-centric recognition branch with standard object detection and pairwise interaction components.
- 26% relative improvement raises V-COCO role AP from 31.8 to 40.0, while HICO-DET improves by 27%.The system runs at about 135ms per image.
2. Related Work
Related work spans object detection, human action and pose recognition, visual relationship modeling, and human-object interaction detection. InteractNet builds on shared region features while targeting precise, human-centered interactions.
- Object Detection: Fast and Faster R-CNN use region proposals and shared feature maps to support efficient instance detection and higher-order interaction recognition.
- Human Action & Pose Recognition: Human action and pose methods provide cues about interactions but generally focus on humans without predicting their object interactions.
- Visual Relationships: Visual relationship models cover broad relation vocabularies, whereas this work targets direct human-object interactions evaluated with high precision.
- Human-Object Interactions: Human-object interactions involve finer-grained actions and simultaneous multiple actions, creating challenges beyond general visual relationships.
- The architecture includes object detection, human-centric, and optional interaction branches, sharing person features between the latter two.
3. Method
InteractNet extends Fast/Faster R-CNN with a human-centric branch that predicts actions and action-specific target-object locations, then combines these with object detections to score interaction triplets. The branches are trained jointly, while cascaded inference efficiently selects compatible objects.
- InteractNet detects ⟨human, verb, object⟩ triplets by localizing human and object boxes and identifying the performed action.
- Human-centric branch: The human-centric branch reuses Fast R-CNN features to classify actions and estimate a target-object density conditioned on human appearance and action.Its Gaussian mean represents the target object’s four-dimensional location relative to the person.
- Target localization: The target-localization term measures compatibility between a detected object box and the predicted action-specific target location.This lets the model use actual object detections to refine a location predicted from the person alone.
- Target localization: Human appearance provides action- and instance-specific cues for predicting objects outside the person box, including carrying, throwing, and sitting configurations.The density is modeled as unimodal despite the diversity of target locations.
- Multi-task training and inference: All three branches are trained jointly with object-detection, human-centric, and interaction-branch losses.During inference, the cascaded procedure has dominant complexity O(n) and the full system takes approximately 135ms per image on an Nvidia M40 GPU.
4. Datasets and Metrics
The evaluation uses V-COCO as the primary testbed and HICO-DET as a larger supplementary benchmark for grounded human-object interaction detection. Performance is measured mainly with role AP, which requires correctly localized human and target boxes plus the correct action.
- HICO-DET: HICO-DET contains approximately 48k images and 600 interaction types, providing an additional benchmark for the approach.
- V-COCO: V-COCO contains approximately 5k trainval and 5k test images, with 26 common action classes and about 8k person instances in trainval.
- Metrics: The AP metrics measure both precision and recall, unlike Recall@N metrics that ignore precision.
- Metrics: Role AP is the central metric and counts a triplet as correct when human and object IoUs are at least 0.5 and the predicted action matches.The metric evaluates target-box localization but does not require the target object category to be correct.
- Metrics: Agent AP evaluates ⟨human, verb⟩ pairs for actions without objects and is secondary because it does not require target localization.
5. Experiments
InteractNet is evaluated on V-COCO through quantitative comparisons, qualitative detections, efficiency measurements, and ablations. Results show strong overall performance, with target localization providing the main gain, while failures remain in interaction reasoning.
- Quantitative Results: InteractNet achieves 40.0 AProle on all V-COCO test actions, an 8.2-point absolute and 26% relative improvement over the 31.8 baseline.The baseline uses a reimplementation of Gupta and Malik’s model with a ResNet-50-FPN backbone.
- Qualitative Results: The method detects multiple persons, actions, and target objects in one image, including objects outside a person’s bounding box.These multi-instance, multi-action, and multi-target detections are produced in one forward pass.
- Efficiency: InteractNet runs at about 135ms per image while detecting multiple interaction instances in a single forward pass.The reported qualitative examples include multi-instance, multi-action, and multi-target results.
- Ablation Studies: Removing target localization decreases AProle by 5.6 points, from 37.5 to 31.9, whereas removing the interaction branch decreases it by only 0.7 point.The ablations identify target localization as the main contributor to system performance.
- Per-action Accuracy: Target localization improves actions with high spatial variance, such as hold, look, carry, and cut, but yields little or no improvement for ride, kick, and read.The per-action comparison reports leading AProle performance consistently for InteractNet.
- Failure Cases: False positives arise from incorrect interaction inferences, assigning another person’s target, irrelevant targets, or confusing actions such as ski and surf.Some errors are attributed to failures of reasoning and identified as an open problem for future research.
Appendix A: Mixture Density Networks
The appendix tests a Mixture Density Network as an alternative target-localization module, allowing multimodal conditional distributions, but finds no performance gain over the simpler model.
- The MDN replaces target localization with a multimodal conditional distribution over interaction-object locations.It predicts means, variances, and mixing coefficients for multiple Gaussian components conditioned on human appearance.
- The MDN is trained by minimizing the negative log likelihood of the predicted target-location density for ground-truth interaction objects.The localization term evaluates the conditional density for an object hypothesis given a human hypothesis.
- The MDN parameterizes each component with action-specific 4-D means and variances, assuming diagonal covariance.Mixing coefficients are predicted with a softmax, while variances use softplus and a lower bound of 0.3.
- MDN with M = 1 performs similarly to the original model, while M = 2 provides no gain on V-COCO validation.The appendix attributes the lack of improvement possibly to strong human-appearance cues and few objects per action type.
- The authors choose the simpler target-localization model because the MDN adds complexity without improving performance.They nevertheless show that MDN can be trained within the InteractNet framework.
Appendix B: HICO-DET Dataset
HICO-DET provides a large, densely specified interaction-detection benchmark, but its incomplete object annotations require frozen external object detections during training.
- HICO-DET contains approximately 48k images annotated with 600 interaction types, 80 object types, and 117 verbs.Its annotations include human and interaction-object bounding boxes, with object categories matching COCO.
- Because HICO-DET does not exhaustively annotate objects, InteractNet uses frozen ResNet50-FPN detections trained on COCO.The object-detection loss is assigned zero weight during training, while human-centric and interaction branches use overlap-based labels.
- InteractNet is trained for 80k iterations with a 10x learning-rate decay after 60k iterations and evaluated using released test-code.Dropout with ratio 0.5 is used in the interaction branch.
- InteractNet achieves a 27% relative gain over published HICO-DET results and a 9% relative gain over the authors’ baseline.These results are reported in Table 6, with example predictions shown in Figure 9.