Source-linked AI summary
Scene Graph Generation by Iterative Message Passing
Danfei Xu, Yuke Zhu, Christopher B. Choy, Li Fei-Fei
TL;DR
The paper addresses the challenge of generating visually grounded scene graphs rather than recognizing objects in isolation. It proposes end-to-end RNN-based iterative message passing for joint graph inference, and reports improved relationship prediction across Visual Genome and NYU Depth v2 support-relation tasks.
Problem
Scene understanding requires generating image-grounded objects and relationships, but prior methods often made local relationship predictions or depended on annotations, synthetic images, or text.
Method
An end-to-end model passes contextual messages between scene-graph subgraphs and iteratively refines object and relationship predictions using RNNs.
Results
The model improves scene-graph relationship prediction, including an 18% gain on predicate classification with R@100 over the baseline, and achieves state-of-the-art support-relation results on NYU Depth v2.
Takeaways & Limitations
Joint contextual inference improves graph generation across semantic scene graphs and indoor support-relation prediction.
Abstract
from arXiv · showhide
Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel end-to-end model that generates such structured scene representation from an input image. The model solves the scene graph inference problem using standard RNNs and learns to iteratively improves its predictions via message passing. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods for generating scene graphs using Visual Genome dataset and inferring support relations with NYU Depth v2 dataset.
1. Introduction
Scene graph generation addresses the limits of object-only perception by representing objects together with their semantic relationships. The proposed end-to-end model uses contextual message passing and iterative RNN refinement to generate these graphs from images.
- Motivation: Object detectors can miss semantic differences between scenes when they recognize objects without modeling their relationships.The paper contrasts images of a man feeding a horse and a man standing by a horse.
- Motivation: Scene graphs provide visually grounded representations whose nodes are object instances and whose edges encode pairwise relationships.They support tasks including image retrieval, 3D scene synthesis, and visual question answering, but prior systems often relied on annotations, synthetic images, or text.
- Problem: Scene graph generation predicts object categories and bounding boxes together with directed relationship predicates between object pairs.The central challenge is reasoning about relationships rather than making isolated local predictions.
- Approach: The proposed model passes contextual messages between bipartite scene-graph subgraphs and iteratively refines predictions with RNNs.It jointly infers scene-graph components instead of predicting each component in isolation.
- Evaluation: The model is evaluated on Visual Genome scene graphs and NYU Depth v2 support relations.The Visual Genome dataset contains 108,077 images averaging 25 objects and 22 pairwise relationships per image; the paper reports improved relationship prediction and state-of-the-art support-relation results.
2. Related Work
Prior work represented scenes through contextual models, text, structured inference, or independently predicted relationships. The paper positions its approach as visually grounded joint graph inference with iterative refinement rather than one-time or isolated prediction.
- Scene understanding: Earlier visual relationship methods used co-occurrence statistics, spatial layout, or contextual pixels and regions, but some lacked detailed visual grounding.The related work also distinguishes physical support and stability relations from general relationship prediction.
- Model pipeline: The architecture overview begins with RPN object proposals and extracted region features before graph inference produces localized objects, categories, and relationships.This passage describes the model pipeline used for the paper’s scene-graph output.
- Relationship prediction: Prior relationship models predicted each relationship independently, while this work uses joint inference to exploit contextual information.The paper reports that its model outperforms the independently predicting method discussed in related work.
- Visual scene representation: Text descriptions support scene classification and retrieval, whereas scene graphs explicitly ground visual concepts and avoid referential uncertainty.Scene graphs have also been used for image retrieval, 3D scene synthesis, and visual question answering.
- Graph inference: RNN-based graph inference connects this work to CRFasRNN, Graph-LSTM, and Structural RNN, but the proposed model refines predictions through message passing.Structural RNN is described as making one-time temporal predictions without refining past predictions.
3. Scene Graph Generation
The model jointly infers scene-graph objects and relationships by iteratively passing contextual messages between node and edge representations. It uses recurrent inference, adaptive message pooling, and a primal-dual bipartite structure to refine predictions efficiently.
- 3. Scene Graph Generation: Unlike independent pairwise predictions, the framework propagates contextual information through the scene-graph topology for joint inference.Context can improve object and relationship predictions because node and edge variables are inferred together rather than in isolation.
- 3.3. Primal Dual Update and Message Pooling: The scene-graph topology yields primal and dual bipartite message-passing sub-graphs, allowing iterative exchange between edge GRUs and node GRUs instead of dense graph inference.Messages pass from edge GRUs to node GRUs in the primal graph and from node GRUs to edge GRUs in the dual graph.
- 3.1. Problem Formulation: Scene graph generation predicts object classes, bounding-box offsets, and relationship predicates from an image and object-box proposals.The proposals can come from ground-truth annotations or an RPN; each relationship is predicted between a pair of proposals.
- 3.2. Inference using Recurrent Neural Network: Mean-field inference is approximated with shared generic GRUs whose hidden states represent the current states of nodes and edges.The first iteration uses visual node and edge features; later iterations use aggregated messages from other GRU units.
- 3.3. Primal Dual Update and Message Pooling: Learned adaptive weights pool incoming messages so each GRU can modulate their influence and retain relevant information.The primal-dual update rules use learnable parameters and sigmoid-based weighting rather than fixed average- or max-pooling.
- 3. Scene Graph Generation: Final prediction layers use softmax scores for object classes and predicates, bounding-box regression, cross-entropy classification losses, and L1 box-offset loss.The output design follows the faster R-CNN setup, with a separate fully connected regressor for each object class.
4. Experiments
The experiments evaluate scene graph generation on Visual Genome and support relation prediction on NYU Depth v2. Iterative contextual message passing improves predicate classification, especially for less frequent predicates, while two training iterations perform best among tested settings.
- Experimental setup: The evaluation covers Visual Genome scene graph generation and NYU Depth v2 support graph generation, including sparse and dense relationship annotations.Visual Genome is used for predicate classification, scene graph classification, and scene graph generation; NYU Depth v2 evaluates support relations and structure classes.
- Experimental setup: The Visual Genome tasks predict predicates, object categories, or detected objects and predicates under PREDCLS, SGCLS, and SGGEN setups.R@50 and R@100 measure the fraction of ground-truth relationship triplets recovered among the top k predictions.
- Visual Genome results: The final model with learnt weighted-sum message pooling achieves an 18% gain on predicate classification with R@100 over the baseline and outperforms the pooling variants.The baseline makes isolated predictions, whereas the final model aggregates contextual hidden states through learnt weighting.
- Visual Genome results: The final model peaks after two training iterations, while average- and max-pooling variants barely improve after the first iteration.Performance gradually degrades after two iterations, which the authors hypothesize results from noisy messages permeating the graph.
- Visual Genome results: Contextual information produces a larger advantage over the baseline for less frequent predicates, whose predictions suffer more from the skewed annotation distribution.Both models perform well on frequent predicates, but the performance gap expands for less frequent ones.
- Qualitative results: Qualitative results show that two message-passing iterations resolve some subject-object ambiguities and cyclic relationships, producing semantically more accurate predicates in examples.The two-iteration model predicts examples such as umbrella-behind-man and vase-with-flower-in-vase.
- Support relation prediction: On NYU Depth v2, the model achieves new state-of-the-art support-relation performance using only RGB images and generalizes to support relations without architecture changes.The task predicts support relation types and one of four structure classes for each object; ambiguous supports and weak geometric features remain failure cases.
5. Conclusions
The paper presents iterative message passing between primal and dual scene-graph subgraphs to improve node and edge predictions using contextual cues. It demonstrates the framework on Visual Genome scene graphs and indoor support relations, while identifying broader structured prediction as future work.
- Iterative message passing between primal and dual subgraphs incorporates contextual cues to improve node and edge predictions.Messages follow the scene graph’s topological structure.
- The framework is demonstrated for predicting Visual Genome scene graphs and support relations in indoor scenes.
- The authors propose exploring the framework in other structured prediction problems and domains.