Source-linked AI summary
Graph R-CNN for Scene Graph Generation
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, Devi Parikh
TL;DR
Scene graph generation must capture relationships beyond objects while handling a quadratic number of possible connections and evaluating complete graphs fairly. Graph R-CNN combines trainable relationship pruning with attentional graph convolution and introduces SGGen+; it outperforms existing methods under existing and proposed metrics.
Problem
Scene graph generation requires efficiently and accurately predicting objects, attributes, and relationships, but evaluating complete graph predictions remains under-examined and exact graph similarity is computationally difficult.
Method
Graph R-CNN uses a Relation Proposal Network to prune unlikely object-pair edges, an attentional graph convolution network to propagate context, and SGGen+ to evaluate objects, attributes, and relationships holistically.
Results
Graph R-CNN outperforms existing scene graph generation methods under both existing metrics and the proposed SGGen+ metric.
Takeaways & Limitations
The model provides a trainable alternative to heuristic relationship pruning and a more fine-grained, realistic measure of scene graph quality.
Takeaways & Limitations
The paper distinguishes the first and last object in each relationship as subject and object, respectively.
Abstract
from arXiv · showhide
We propose a novel scene graph generation model called Graph R-CNN, that is both effective and efficient at detecting objects and their relations in images. Our model contains a Relation Proposal Network (RePN) that efficiently deals with the quadratic number of potential relations between objects in an image. We also propose an attentional Graph Convolutional Network (aGCN) that effectively captures contextual information between objects and relations. Finally, we introduce a new evaluation metric that is more holistic and realistic than existing metrics. We report state-of-the-art performance on scene graph generation as evaluated using both existing and our proposed metrics.
1 Introduction
Graph R-CNN addresses the challenge of efficiently and accurately generating scene graphs by pruning unlikely object relationships, propagating graph context, and introducing a holistic evaluation metric.
- Motivation: Scene graphs represent objects, attributes, and relationships, supporting higher-level tasks but remaining difficult to extract efficiently and accurately.Object-centric representations alone miss relationships essential for scene understanding.
- Graph R-CNN: Graph R-CNN factorizes scene graph generation into object extraction, relationship edge pruning, and graph context integration.It uses standard object detection, a Relation Proposal Network, and an attentional graph convolution network.
- Graph R-CNN: The Relation Proposal Network learns relatedness scores to prune unlikely object-pair connections instead of relying on random pruning.This produces a sparse candidate graph for subsequent contextual reasoning.
- Results: On Visual Genome, Graph R-CNN achieves an absolute gain of 5.0 on Recall@50 for scene graph generation and receives extensive ablation analysis.The authors also report that the approach outperforms existing methods under standard metrics and SGGen+.
- Evaluation: SGGen+ evaluates singleton entities, object-attribute pairs, and subject-predicate-object triplets to measure scene graph quality more holistically.The metric addresses the mismatch between triplet-based metrics and metrics that assume ground-truth object regions.
2 Related Work
Prior scene graph methods use contextual reasoning and exploit object-relationship regularities, but often rely on heuristics for the quadratic relationship space. Graph R-CNN introduces trainable relationship pruning and attentional graph convolution in this setting.
- Contextual Reasoning and Scene Graphs: Scene graphs extend object detection by representing relationships and attributes, while contextual reasoning has a long history in computer vision.The representation was introduced to support richer scene understanding than object detection alone.
- Scene Graph Generation: Prior methods address the quadratic number of possible relationships with heuristics such as random sampling.Graph R-CNN instead introduces a trainable RePN to prune unlikely relationship edges.
- Context Propagation: Existing approaches propagate context through object and relationship sub-graphs, message passing, conditional random fields, or associative graph embeddings.These methods differ in whether they use object detectors and how they exchange information across graph components.
- Relationship Regularities: Earlier work exploits semantic priors, region captions, and recurring graph motifs to improve relationship prediction.These regularities motivate Graph R-CNN’s use of object-relationship structure.
- Related Architectures: Graph R-CNN’s RePN is related to region proposal networks and Rel-PN, while its aGCN uses attention to update graph representations.The cited approaches differ in how they generate relationship proposals and propagate contextual information.
3 Approach
Graph R-CNN factorizes scene graph generation into object extraction, relation-edge pruning, and graph-context integration. It uses RePN to sparsify likely object relations and aGCN to propagate learned contextual information across object and relationship nodes.
- Framework overview: Graph R-CNN models scene graphs as localized object regions, relationships, and their labels, with generation factorized into proposal, relationship, and graph-labeling stages.The factorization separates graph construction from graph labeling and uses object detection for initial region proposals.
- Framework overview: The pipeline first extracts object regions, then prunes candidate connections with RePN, and finally integrates graph context using an attentional GCN.The resulting scene graph is produced after contextual reasoning over the constructed graph.
- Relation Proposal Network: RePN learns object-pair relatedness from object class distributions to prune unlikely edges from the O(n^2) directional pair candidates.An asymmetric kernel uses separate subject and object projections, allowing the score matrix to be computed through two projection processes and matrix multiplication.
- Relation Proposal Network: Top-scoring object pairs are selected and filtered with non-maximal suppression, while each retained relation receives visual features from the union box of its object pair.The overlap computation accounts for the ordered subject-object pair.
- Attentional GCN: aGCN learns attention weights from concatenated node features with a two-layer MLP and softmax instead of using preset GCN coefficients.Attention is recomputed as node features change across iterations.
- Attentional GCN: The scene-graph aGCN connects relationship nodes to their associated objects, adds object-to-object skip connections, and uses distinct transformations for connection types and directions.It updates object and relationship representations by propagating contextual information through the graph.
4 Evaluating Scene Graph Generation
Prior triplet-recall evaluation can score graphs identically despite substantial differences in object and relationship accuracy. SGGen+ addresses this by adding singleton-level credit to triplet matching for a more comprehensive similarity measure.
- SGGen represents each ground-truth graph as object–relationship–subject triplets and counts a match only when labels and both node localizations are correct.Node localization requires bounding box IoU > 0.5.
- Exact triplet matching can assign the same recall of 0 to graphs with starkly different node and relationship accuracy.A graph that merely mislabels one object and a graph that fails to predict any node or relationship may both miss every complete triplet.
- SGGen+ augments SGGen by evaluating singleton objects and predicates in addition to graph triplets.This adds partial credit for correctly localized and recognized components that do not form complete matched triplets.
- SGGen+ counts correctly localized and recognized objects, correctly recognized predicates with correctly localized endpoints, and complete triplets, normalized by ground-truth graph entries.Its recall combines C(O), C(P), and C(T) over the total number of objects, predicates, and relationships.
5 Experiments
The experiments evaluate Graph R-CNN on Visual Genome using established and proposed metrics, fair reimplementations, and component ablations. Results show gains from RePN and aGCN, with improved effectiveness and efficiency.
- Experimental Setup: The experiments follow a clarified Visual Genome protocol, using the common split, two-stage training, and four scene graph generation metrics.The dataset contains 75,651 training images and 32,422 test images, with 150 object classes and 50 relation classes.
- Experimental Setup: SGGen+ evaluates singleton entities, object-attribute pairs, and subject-predicate-object triplets, complementing existing triplet- and localization-based metrics.The evaluation ranks predictions by multiplied subject, object, and relationship scores before computing Recall@50 and Recall@100.
- Metric Analysis: SGGen remains insensitive to perturbations of objects without relationships, whereas SGGen+ remains non-zero even when object labels are fully perturbed but localizations and relationships stay correct.At 20% perturbation of objects with relationships, SGGen reports 54.1, while 50.1% of dataset objects lack relationships and are omitted by SGGen.
- Quantitative Comparison: Graph R-CNN outperforms the reimplemented IMP and MSDN baselines, and it beats the frequency baseline on all reported metrics.The reimplementations use the same codebase and control for differences in pretraining, data splits, and extra supervision.
- Quantitative Comparison: Graph R-CNN is faster than the compared baselines: IMP and MSDN require 2.15× and 1.86× its training cost, and are 3.27× and 3.80× slower during inference.The paper attributes the efficiency mainly to its simplified architecture, especially the aGCN context propagation design.
- Ablation Study: RePN significantly improves SGGen, SGGen+, and object detection, with gains concentrated across nearly all categories and especially smaller objects.RePN has less effect on PhrCls and PredCls because those settings provide ground-truth object locations and therefore involve fewer relation pairs.
- Ablation Study: Adding aGCNs further improves performance, while attention on the GCNs produces higher results by controlling contextual information flow across graph edges.Qualitative results show correct relationship predictions in generated scene graphs with RePN and aGCN.
6 Conclusion
The paper introduces Graph R-CNN for scene graph generation, combining relation pruning, attentional context propagation, and a holistic evaluation metric. It reports that the approach outperforms existing methods under both existing and proposed metrics.
- Conclusion: Graph R-CNN combines a Relation Proposal Network, attentional graph convolutional networks, and the SGGen+ evaluation metric for scene graph generation.RePN prunes unlikely object pairs, aGCN propagates contextual information, and SGGen+ evaluates objects, attributes, and relationships.