Source-linked AI summary
PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection
Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, Jiashi Feng
TL;DR
Conventional HOI detection relies on sequential proposal generation and classification, limiting proposal quality and efficiency. PPDM reformulates HOI detection as parallel point detection and matching, while introducing the application-oriented HOI-A dataset. The paper reports real-time, state-of-the-art performance and presents HOI-A as a supplement to existing datasets.
Problem
Sequential proposal-based HOI methods independently generate human-object proposals and then scan proposal pairs, limiting proposal quality and efficiency.
Method
PPDM uses parallel point detection and matching, representing an HOI with human, interaction, and object points and regressing displacements for matching.
Results
PPDM is reported as real-time and outperforming state-of-the-art methods on both HICO-Det and HOI-A benchmarks.
Takeaways & Limitations
HOI-A provides an application-oriented supplement to existing datasets, while PPDM directly predicts HOI triplets through a one-stage framework.
Takeaways & Limitations
Future work includes using human context in the framework and enriching HOI-A’s action categories.
Abstract
from arXiv · showhide
We propose a single-stage Human-Object Interaction (HOI) detection method that has outperformed all existing methods on HICO-DET dataset at 37 fps on a single Titan XP GPU. It is the first real-time HOI detection method. Conventional HOI detection methods are composed of two stages, i.e., human-object proposals generation, and proposals classification. Their effectiveness and efficiency are limited by the sequential and separate architecture. In this paper, we propose a Parallel Point Detection and Matching (PPDM) HOI detection framework. In PPDM, an HOI is defined as a point triplet < human point, interaction point, object point>. Human and object points are the center of the detection boxes, and the interaction point is the midpoint of the human and object points. PPDM contains two parallel branches, namely point detection branch and point matching branch. The point detection branch predicts three points. Simultaneously, the point matching branch predicts two displacements from the interaction point to its corresponding human and object points. The human point and the object point originated from the same interaction point are considered as matched pairs. In our novel parallel architecture, the interaction points implicitly provide context and regularization for human and object detection. The isolated detection boxes are unlikely to form meaning HOI triplets are suppressed, which increases the precision of HOI detection. Moreover, the matching between human and object detection boxes is only applied around limited numbers of filtered candidate interaction points, which saves much computational cost. Additionally, we build a new application-oriented database named HOI-A, which severs as a good supplement to the existing datasets. The source code and the dataset will be made publicly available to facilitate the development of HOI detection.
1. Introduction
PPDM reformulates HOI detection as a parallel point detection and matching problem to address the effectiveness and efficiency limits of sequential proposal-based methods. It also introduces an application-oriented dataset, HOI-A, alongside reported real-time and state-of-the-art performance.
- Results: 37 fps (0.027s) is reported for PPDM-DLA on HICO-Det, described as the first real-time HOI detection method.PPDM-Hourglass is also reported to improve mAP by 4.27% over state-of-the-art methods at faster speed.
- Motivation: Conventional HOI detectors generate human-object proposals and then classify interactions, using sequential and separate stages.Human and object proposals are independently generated before pairwise proposal classification.
- Motivation: Proposal quality and exhaustive pair scanning limit conventional methods because many generated human-object pairs are not meaningful HOI triplets.The framework argues for a non-sequential, highly coupled design to reduce these limitations.
- PPDM framework: PPDM represents each HOI as human, interaction, and object points, with the interaction point defined as the midpoint of the human and object points.Its parallel branches detect points and regress displacements that match each interaction point to its human and object points.
- Dataset: HOI-A contains 10 application-relevant HOI categories with substantial variation across scenes, lighting conditions, and human poses.The dataset is intended as an application-driven supplement to general existing datasets.
- Contributions: The paper reports PPDM as real-time and outperforming state-of-the-art methods on both HICO-Det and HOI-A benchmarks.The authors also describe HOI-A as a large-scale, application-oriented dataset supplement.
2. Related Work
Prior HOI work primarily uses proposal-based two-stage pipelines, whose performance and computational cost are constrained by proposal quality and feature processing. PPDM instead offers a one-stage, proposal-free point-based framework, while HOI-A targets practical action categories underrepresented in general datasets.
- HOI Detection Methods: Existing HOI methods generally localize humans and objects first, then classify interactions for paired detections.Recent work emphasizes context and human structural information in the classification stage.
- Proposed Framework: Figure 3 depicts PPDM’s feature extraction, parallel point detection, displacement regression, and point-triplet matching pipeline.The point detection branch predicts centers, sizes, and offsets; the matching branch links interaction points to human and object points.
- HOI Detection Methods: Proposal-based methods are limited by proposal quality and incur computational cost during proposal generation and feature extraction.The paper motivates a one-stage, proposal-free alternative to these drawbacks.
- HOI Detection Datasets: HICO-Det and VCOCO are common HOI benchmarks, while HCVRD addresses broader human-centric relationships beyond actions.HICO-Det is described as large-scale and generic, whereas VCOCO is comparatively small.
- HOI Detection Datasets: HOI-A contains about 38K images annotated with limited typical action categories selected for practical significance.It supplements datasets that mostly concentrate on common and general actions.
3. Parallel Point Detection and Matching
PPDM reformulates HOI detection as parallel point detection and matching, representing each interaction with human, interaction, and object points. Its branches detect these points and match them through displacements from the interaction point.
- Point Detection: The point-detection branch estimates human, object, and interaction points, together with box sizes and local offsets.Gaussian heatmaps encode point locations, while convolutional blocks estimate sizes and offsets.
- Overview: PPDM represents an HOI triplet using human and object center points plus an interaction point at their midpoint.Boxes are represented by center points and corresponding widths and heights.
- Point Matching: The point-matching branch predicts displacements from each interaction point to its corresponding human and object points.The interaction point serves as the anchor, and the two resulting points form a matched pair.
- Triplet Matching: Triplet matching ranks candidate human and object points by proximity to displacement-derived coarse points and by confidence scores.During inference, top-K points are selected from the heatmaps before matching and final box recovery.
- Training Objective: The training objective combines point-location, displacement, box-size, and offset losses into a weighted total loss.The displacement terms are weighted by λ, set to 0.1.
4. HOI-A Dataset
HOI-A is an application-oriented dataset focused on practically significant HOI categories rather than broad general actions. It provides large scale, substantial intra-class variation, and positive and negative examples for selected interactions.
- Motivation and Categories: HOI-A associates each selected verb with an application scenario, such as dangerous-driving detection for talking on a phone in a car.The categories are chosen according to practical application needs.
- Image Collection: The dataset includes positive images collected through camera shooting and Internet crawling, along with negative samples where the object appears without the action or a similar action occurs.Figure 4 illustrates intra-class variation and two negative-sample types for smoking-related interactions.
- Dataset Properties: HOI-A comprises 38,668 images, 11 object kinds, 10 action categories, 43,820 human instances, 60,438 object instances, and 96,160 interaction instances.Each person performs 2.2 interactions on average.
- Dataset Properties: Images vary across indoor, outdoor, and in-car scenes; dark, natural, and intense lighting; human poses; viewing angles; and RGB or IR cameras.These collection settings are intended to enlarge intra-class variation.
5. Experiments
Experiments evaluate PPDM on HICO-Det and HOI-A using mAP, qualitative comparisons, speed measurements, and component analyses. PPDM outperforms prior methods while achieving substantially faster inference, and its midpoint interaction-point design improves performance.
- Quantitative Analysis: PPDM-Hourglass achieves a 24.5% performance gain over PMFNet on HICO-Det, while PPDM-DLA and PPDM-Hourglass both outperform previous state-of-the-art methods.PPDM uses appearance features without human pose, although it performs slightly worse than PMFNet on the rare subset.
- Quantitative Analysis: PPDM outperforms the compared methods on the application-oriented HOI-A dataset, including C-HOI, which uses a very strong detector.The selected practically significant interaction types also achieve high performance.
- Qualitative Analysis: Qualitative comparisons show PPDM correctly predicts HOI triplets in cases where iCAN favors high-scoring humans or objects without interactions, predicts no-interaction, or misses small people.The comparison uses top-3 confidence predictions on HICO-Det.
- Efficiency Analysis: 27ms inference makes PPDM-DLA the only real-time method among the compared approaches on a single Titan Xp GPU.PPDM-DLA is faster than the proposal-generation or classifier stages of the compared two-stage methods.
- Component Analysis: Adding feature fusion and global reasoning improves the basic DLA framework by 0.35% mAP, while using the union-box center as the interaction point lowers mAP by 1.64 points.The results support larger receptive fields, global context, and midpoint interaction points for interaction prediction.
6. Conclusion
The paper introduces a one-stage PPDM framework and the application-oriented HOI-A dataset for HOI detection. It reports improved performance with faster inference, while identifying human-context utilization and expanded HOI-A action categories as future work.
- Contributions: PPDM is a one-stage framework that directly predicts HOI through a parallel architecture, and HOI-A is an application-oriented dataset for practical HOI detection.The paper presents both as contributions to HOI detection.
- Contributions: The proposed method outperforms existing methods while operating at significantly faster speed.The conclusion characterizes PPDM as breaking the limits of traditional two-stage methods.
- Future Work: Future work will explore how to use human context in PPDM and enrich the action categories in HOI-A.These directions address the framework's human-context use and the dataset's category coverage.