Source-linked AI summary
Learning Human-Object Interactions by Graph Parsing Neural Networks
Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, Song-Chun Zhu
TL;DR
HOI recognition requires reasoning over relationships among humans and objects beyond individual instance recognition. GPNN combines graph-based structural knowledge with neural message passing to infer parse graphs end-to-end, and it substantially improves performance across image and spatial-temporal benchmarks.
Problem
HOI recognition requires reasoning beyond individual human, object, and action recognition because it integrates complex relationships among scene elements.
Method
GPNN jointly infers HOI graph structures and node labels through iterative graph parsing and message passing in an end-to-end framework.
Results
GPNN substantially outperforms comparative methods across HICO-DET, V-COCO, and CAD-120, including 31.89% improvement over the second-best method on one HICO-DET category set.
Takeaways & Limitations
GPNN provides a scalable, generic HOI representation applicable to both static images and spatial-temporal settings.
Abstract
from arXiv · showhide
This paper addresses the task of detecting and recognizing human-object interactions (HOI) in images and videos. We introduce the Graph Parsing Neural Network (GPNN), a framework that incorporates structural knowledge while being differentiable end-to-end. For a given scene, GPNN infers a parse graph that includes i) the HOI graph structure represented by an adjacency matrix, and ii) the node labels. Within a message passing inference framework, GPNN iteratively computes the adjacency matrices and node labels. We extensively evaluate our model on three HOI detection benchmarks on images and videos: HICO-DET, V-COCO, and CAD-120 datasets. Our approach significantly outperforms state-of-art methods, verifying that GPNN is scalable to large datasets and applies to spatial-temporal settings. The code is available at https://github.com/SiyuanQi/gpnn.
1 Introduction
HOI recognition requires reasoning over humans, objects, and their relationships beyond individual instance recognition. GPNN combines graph representations with neural networks to learn HOI structures end-to-end across images and videos.
- HOI understanding infers relationships between humans and objects, requiring reasoning beyond individual pose, action, and object recognition.
- GPNN explicitly represents HOI structures with graphs and automatically parses optimal graph structures in an end-to-end framework.
- Graph structure learning explicitly utilizes rich relations and integrates information from individual elements through message passing.
- GPNN addresses uncertain human-object relations and more complex temporal dependencies without relying on pre-fixed graph structures.
- GPNN was evaluated on HICO-DET, V-COCO, and CAD-120 for image-based detection and spatial-temporal recognition or anticipation.
- The paper reports that GPNN jointly performs graph structure inference and message passing in a scalable representation applicable to static and dynamic settings.
2 Related Work
Prior HOI methods achieved success in specific domains but lacked explicit structural representations and a generic formulation extending naturally to videos. GPNN addresses these gaps with learnable graph structures and unified neural message passing.
- HOI reasoning concerns human actions with objects rather than recognizing individual actions or object instances separately.
- Existing deep HOI models lacked powerful tools for explicitly representing HOI structures and a complete representation extending to video recognition.
- GPNN extends graphical neural networks with learnable graph structures that automatically infer relations and enhance information propagation in spatial and spatial-temporal settings.
3 Graph Parsing Neural Network for HOI
GPNN parses a complete human-object interaction graph by jointly inferring meaningful connections and node labels through iterative message passing. Its modular, differentiable architecture repeatedly learns graph structure, propagates information, updates node states, and reads out labels.
- Graph formulation: GPNN represents humans and objects as nodes and their relationships as edges, then extracts a labeled parse subgraph from the complete HOI graph.The parse graph retains meaningful edges and labels its nodes using node and edge features.
- Graph formulation: The link function uses node and edge features to produce a soft adjacency matrix that approximates the learned parse-graph structure.Adjacency values weight information flow between connected nodes.
- Message passing: Message functions aggregate incoming information, update functions revise hidden node states, and readout functions convert final states into node labels.The message-passing phase runs for S steps, with initial hidden states initialized from node features.
- Iterative parsing: GPNN iteratively recomputes graph structure and message passing using updated node information and edge features at each step.The authors report that this joint iterative strategy improves performance in practice.
- Network architecture: The four modules are implemented with neural networks, making the full system differentiable and trainable end-to-end with gradient-based optimization.The implementation includes shared 1 × 1 convolutional link layers, sigmoid-normalized adjacency values, convLSTMs for temporal structure, GRU updates, and task-dependent activations.
4 Experiments
Experiments evaluate GPNN for image-based HOI detection, video-based recognition and anticipation, and component effectiveness across three datasets. Results report strong performance across static and spatio-temporal settings, with ablations supporting joint graph parsing and message passing.
- Experimental Setup: GPNN is evaluated on HICO-DET and V-COCO for image HOI detection, and CAD-120 for video recognition and anticipation.The image benchmarks cover large-scale detection, while CAD-120 evaluates spatial-temporal activity understanding.
- Image HOI Detection: GPNN consistently outperforms comparative methods on V-COCO and is reported to scale to large datasets and more than 600 HOI classes.The V-COCO benchmark includes separate one-object and two-object action sets.
- Video HOI Recognition and Anticipation: On CAD-120, GPNN improves over ATCRF and S-RNN on detection and activity anticipation, especially on anticipation, using class-averaged F1-scores.ATCRF is a graphical-model baseline, whereas S-RNN uses a pre-defined spatial-temporal graph.
- Ablation Study: Ablations show that jointly learning graph structures with message passing improves performance, iterative learning generally helps, and excessive iterations can slightly reduce it.The baseline uses three message-passing iterations; supervision on the graph is also described as important.
5 Conclusion
GPNN infers HOI parse graphs end-to-end through iterative graph inference and message passing. The framework applies across spatial and spatial-temporal domains and achieves substantial gains on three HOI datasets.
- GPNN infers a parse graph end-to-end using link, message, update, and readout functions for iterative graph inference and message passing.
- The framework provides a generic HOI representation applicable to both spatial and spatial-temporal domains.
- GPNN demonstrates substantial performance gains across three HOI datasets, supporting the framework’s effectiveness.