Source-linked AI summary
Learning Human-Object Interaction Detection using Interaction Points
Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, Jian Sun
TL;DR
HOI detection must localize humans and objects while recognizing complex interactions, but existing instance-centric approaches rely on pairing all possible human-object combinations and appearance-based features. This paper instead detects interaction points and vectors and groups them with object detections, achieving state-of-the-art results on V-COCO and HICO-DET.
Problem
HOI detection requires localizing humans and objects and inferring complex relationships, while existing approaches pair all human-object instances and predominantly rely on appearance features.
Method
The point-based framework treats HOI detection as keypoint detection and grouping, generating interaction points and vectors and pairing them with human and object boxes.
Results
The points-based approach outperforms state-of-the-art methods on both V-COCO and HICO-DET benchmarks.
Takeaways & Limitations
Interaction points and vectors provide a direct representation for grouping interactions with detected human and object instances into HOI triplets.
Takeaways & Limitations
Multiple HOI pairs cannot share the same interaction point, although such cases are rare in practice.
Abstract
from arXiv · showhide
Understanding interactions between humans and objects is one of the fundamental problems in visual classification and an essential step towards detailed scene understanding. Human-object interaction (HOI) detection strives to localize both the human and an object as well as the identification of complex interactions between them. Most existing HOI detection approaches are instance-centric where interactions between all possible human-object pairs are predicted based on appearance features and coarse spatial information. We argue that appearance features alone are insufficient to capture complex human-object interactions. In this paper, we therefore propose a novel fully-convolutional approach that directly detects the interactions between human-object pairs. Our network predicts interaction points, which directly localize and classify the inter-action. Paired with the densely predicted interaction vectors, the interactions are associated with human and object detections to obtain final predictions. To the best of our knowledge, we are the first to propose an approach where HOI detection is posed as a keypoint detection and grouping problem. Experiments are performed on two popular benchmarks: V-COCO and HICO-DET. Our approach sets a new state-of-the-art on both datasets. Code is available at https://github.com/vaesl/IP-Net.
1. Introduction
HOI detection seeks to localize humans and objects while identifying their relationships, but complex interaction scenarios challenge existing instance-centric approaches. Prior methods pair all detected humans and objects and fuse appearance and spatial-stream scores, increasing computational cost.
- HOI detection localizes humans and objects while inferring relationships such as eating an apple or driving a car.
- Multi-stream approaches fuse human, object, and pairwise scores after extracting RoI features from detected human-object boxes.
- Multiple humans, multiple objects, shared interactions, and fine-grained actions create complex and diverse detection scenarios.
- Most existing methods decompose HOI detection into object detection and interaction recognition for human-action-object triplets.
- Pairing every human with every object makes inference scale quadratically with scene instances and increases computational expense.
- These approaches predominantly rely on appearance features and a simple union-box representation for pairwise spatial information.
2. Related Work
Related work contrasts two-stage and single-stage object detectors, then situates HOI detection among keypoint-based and interaction-specific approaches. Single-stage methods emphasize speed, while two-stage methods are generally more accurate.
- Two-stage detectors generate object proposals before classification and regression, whereas single-stage detectors directly classify and regress default anchors.
- Two-stage object detectors are generally more accurate, while single-stage methods primarily offer speed.
- Anchor-free single-stage detectors eliminate anchor boxes by treating object detection as keypoint estimation, including corner- and center-based formulations.
3. Method
The method reframes HOI detection as interaction-point estimation and groups predicted interactions with detected human and object boxes. Its fully convolutional design combines point and vector predictions with geometric constraints to produce HOI triplets.
- Overall Architecture: The architecture uses FPN for human and object bounding-box detection, then applies feature extraction, interaction generation, and interaction grouping.Hourglass extracts features, and the generation module has separate branches for interaction points and vectors.
- Motivation: Existing multi-stream HOI methods fuse human, object, and pairwise appearance scores, but their instance-centric inference evaluates all human-object pairs.This makes inference time scale quadratically with the number of scene instances.
- Interaction Representation: The method defines each human-object interaction as an interaction point and predicts an interaction vector relative to the human and object centers.The point serves as the interaction center, while the vector encodes geometric relations to the paired detections.
- Interaction Generation: Interaction-point heatmaps use multiple Gaussian peaks and separate channels can represent multiple interaction categories at one location.This handles cases where one human interacts with the same object through multiple actions, such as holding and hitting a tennis racket.
- Interaction Generation: The vector branch predicts unsigned horizontal and vertical lengths, from which four possible human-center locations and an interaction box are constructed.At inference, these predictions are extracted at top-k interaction-point locations.
- Interaction Grouping: Interaction grouping combines detected boxes, interaction points, and vectors, retaining pairs whose interaction and reference boxes satisfy soft geometric constraints.The constraints filter many negative human-object pairs before forming final HOI predictions.
4. Experiments
Experiments on V-COCO and HICO-DET evaluate the approach using role mean Average Precision under standard protocols. The method achieves state-of-the-art benchmark results, while ablations and qualitative examples examine its components and interaction-handling capabilities.
- Datasets and Metrics: Experiments use V-COCO and HICO-DET, with performance reported using role mean Average Precision under standard evaluation protocols.A detected HOI triplet is correct only when both human and object boxes exceed 0.5 IoU with ground truth and the interaction class is correct.
- State-of-the-art Comparison: 51.0 mAProle establishes a new V-COCO state of the art, improving by 3.2% over TIK; HICO-DET pre-training and V-COCO fine-tuning further reach 52.3.The 52.3 result uses pre-training on HICO-DET followed by fine-tuning on V-COCO.
- State-of-the-art Comparison: 19.56 mAProle on HICO-DET’s Default full set accompanies superior performance over existing methods in both Default and Known Object settings.The method also obtains 12.79 and 21.58 mAProle on the Default rare and non-rare sets, respectively, and gains 2.9% over TIK on the Known Object full set.
- Ablation Study: 11.4% absolute improvement over the baseline is reported for the overall architecture in V-COCO ablations.The ablation evaluates interaction points, filtering, interaction boxes, corner distance, and center-pool components using the same Hourglass-104 backbone.
- Ablation Study: Interaction grouping reaches 46.2 mAProle, interaction boxes raise performance from 46.2 to 48.2, and corner-distance filtering further improves it to 50.5.The corner-distance constraint filters some negative human-object pairs during inference.
- Qualitative Visualization Results: Qualitative results show single and multiple human-object interactions, humans performing multiple interactions, fine-grained interactions, and heatmaps handling multiple HOI pairs and shared objects.The paper notes that long-tailed classes remain particularly challenging and that multiple HOI pairs cannot share the same interaction point, although such cases are rare in practice.
5. Conclusion
The paper proposes a point-based HOI detection framework that treats interaction detection as keypoint detection and grouping. It outperforms state-of-the-art methods on two HOI detection benchmarks.
- The framework formulates HOI detection as a keypoint detection and grouping problem.
- A keypoint network generates interaction points and corresponding interaction vectors.
- The method pairs interaction points with human and object bounding boxes through an interaction grouping scheme.
- The point-based approach outperforms state-of-the-art methods on both evaluated HOI detection benchmarks.