Source-linked AI summary
Bridging Knowledge Graphs to Generate Scene Graphs
Alireza Zareian, Svebor Karaman, Shih-Fu Chang
TL;DR
Scene graph generation needs better ways to use commonsense knowledge and its rich graphical structure for complex visual understanding. GB-Net reframes SGG as bridging scene instances to commonsense classes with iterative graph message passing, and it outperforms the state of the art across metrics.
Problem
Scene graph generation methods have limited performance and often use commonsense heuristically or fail to exploit its graphical structure.
Method
GB-Net jointly classifies scene entities and predicates by iteratively propagating information within and between scene and commonsense graphs while refining bridge edges.
Results
GB-Net outperforms the state of the art across multiple metrics, while GB-Net-β raises mean recall by +2.7% on average with only −0.2% recall.
Takeaways & Limitations
The unified formulation turns SGG into dynamic graph bridging, allowing visual instances and commonsense classes to be jointly connected and refined.
Takeaways & Limitations
The formulation assumes every scene graph node label exists in the commonsense graph, adding special background nodes for exceptions.
Abstract
from arXiv · showhide
Scene graphs are powerful representations that parse images into their abstract semantic elements, i.e., objects and their interactions, which facilitates visual comprehension and explainable reasoning. On the other hand, commonsense knowledge graphs are rich repositories that encode how the world is structured, and how general concepts interact. In this paper, we present a unified formulation of these two constructs, where a scene graph is seen as an image-conditioned instantiation of a commonsense knowledge graph. Based on this new perspective, we re-formulate scene graph generation as the inference of a bridge between the scene and commonsense graphs, where each entity or predicate instance in the scene graph has to be linked to its corresponding entity or predicate class in the commonsense graph. To this end, we propose a novel graph-based neural network that iteratively propagates information between the two graphs, as well as within each of them, while gradually refining their bridge in each iteration. Our Graph Bridging Network, GB-Net, successively infers edges and nodes, allowing to simultaneously exploit and refine the rich, heterogeneous structure of the interconnected scene and commonsense graphs. Through extensive experimentation, we showcase the superior accuracy of GB-Net compared to the most recent methods, resulting in a new state of the art. We publicly release the source code of our method.
1 Introduction
Scene Graph Generation extracts symbolic objects and interactions from images, supporting visual reasoning and downstream tasks. GB-Net reframes SGG as bridging image-specific scene graphs with commonsense graphs and reports state-of-the-art performance.
- Motivation: Scene Graph Generation extracts graphical representations of objects and their relationships from images.These representations support visual question answering, image captioning, retrieval, and synthesis.
- Motivation: 16% mean recall was achieved by a recent method when matching the top 100 predicted triples against ground truth.The result indicates substantial difficulty in current SGG methods.
- Motivation: Recent SGG methods use external commonsense, but often integrate limited knowledge heuristically or fail to exploit its graphical structure.The paper targets a general framework for incorporating graphical knowledge into scene understanding.
- Contribution: GB-Net classifies scene entity and predicate instances by bridging them to corresponding classes in a commonsense graph.It iteratively propagates messages within and between the graphs while refining the bridge and interconnected heterogeneous structure.
- Results: Extensive Visual Genome experiments show that GB-Net consistently outperforms the state of the art across performance metrics.The authors also report ablations and publicly release software for reproducing the results.
2 Related work
Prior SGG methods propagate context over detected objects and relations, while external-knowledge approaches use statistical or ontological information with limited integration. GB-Net extends graph reasoning to dynamically refine connections between scene and commonsense graphs.
- Scene graph generation: Conventional SGG methods detect regions, propagate information among object and relation nodes, and classify objects and relations.Their propagation modules operate on scene-level visual representations.
- External knowledge: External knowledge has been used through triplet frequencies, graph edge-weight initialization, and ontological information.These approaches encode commonsense priors rather than specific instances.
- Graph-based neural networks: Graph-based neural networks iteratively update node representations using learnable messages from neighboring nodes.GCNs and GGNNs are examples of this model family.
- Graph-based neural networks: GB-Net dynamically updates edges while propagating messages and is presented as the first method to refine scene-to-external-knowledge graph edges.This differs from methods operating on a static graph.
- Comparison with ontology-assisted recognition: Unlike prior ontology-assisted object recognition, GB-Net explicitly infers instance-to-class connections and includes predicate nodes in the more complex graph structure.The method also iteratively refines the bridge between scene and commonsense graphs.
3 Problem Formulation
The paper formalizes scene and commonsense graphs as related knowledge graphs and recasts SGG as inferring bridge edges between their instance and class nodes. GB-Net initializes proposals, propagates messages, and jointly refines entity and predicate bridges.
- 3.1 Knowledge graphs: A knowledge graph contains entity and predicate nodes with semantic labels and directed, weighted edges drawn from predefined types.Edge sets are indexed by source and destination node types and the relation between them.
- 3.1 Knowledge graphs: Commonsense graphs represent general entity and predicate concepts, with each semantic class appearing in exactly one node.Edges encode relational facts such as Hand-partOf-Person and Cup-usedFor-Drinking.
- 3.1 Knowledge graphs: Scene graphs associate entity nodes with image bounding boxes and predicate nodes with ordered subject-object pairs.Subject and object connections are represented as two directed edges in opposite directions.
- 3.1 Knowledge graphs: The formulation assumes every scene node label exists in the commonsense graph, adding background entity and predicate nodes for otherwise unrepresented classes.This assumption covers objects and predicates that may belong to background classes.
- 3.2 Bridging knowledge graphs: Bridge edges replace class labels on scene nodes and connect each scene entity or predicate instance to its corresponding commonsense class.classifiedTo edges point to classes, while hasInstance edges provide reverse connectivity.
- 3.2 Bridging knowledge graphs: GB-Net initializes scene graphs and entity bridges with Faster R-CNN, then repeatedly propagates messages and updates entity and predicate bridges.After T iterations, the final bridge determines each node’s output label.
- 3.2 Bridging knowledge graphs: GB-Net decomposes SGG into extracting implicit scene entity and predicate proposals and inferring their bridge edges from image and graph structure.The model avoids factorizing entity classification followed by predicate classification, enabling joint classification.
4 Method
GB-Net constructs a heterogeneous graph linking image-derived scene entities and predicates to commonsense classes, then iteratively updates node representations and bridge edges. This framework uses bidirectional information flow and learned pairwise matching to classify scene-graph instances.
- Graph initialization: The model detects objects as scene entity nodes, creates predicate nodes for object pairs, and connects predicates to subjects and objects with four directed edge types.These directions distinguish how predicate information updates entity nodes and how entity information updates predicates.
- Graph initialization: Commonsense entity and predicate nodes are initialized from projected word embeddings, while the method can use commonsense graphs with arbitrary edge types.The commonsense graph includes relations such as UsedFor and PartOf, but the model does not depend on a specific edge vocabulary.
- Graph initialization: Bridge edges connect scene instances to commonsense classes, using detector probabilities for entity candidates and learned matching for predicate candidates.Each bridge has a forward classifiedTo direction and a reverse hasInstance direction, enabling information flow between scene and commonsense graphs.
- Successive message passing and bridging: Message passing computes outgoing messages, propagates them along weighted edges, aggregates incoming messages by edge type, and updates nodes with a GRU rule.The receive network operates on concatenated representations across node and edge types, extending GGNN-style propagation to heterogeneous graphs.
- Successive message passing and bridging: After node updates, GB-Net recomputes pairwise scene-to-commonsense similarities, retains the top Kbridge bridges, and repeats propagation for T iterations.The final bridge weights provide the model outputs for classifying scene entities and predicates; retaining only top-Kbridge links avoids performance degradation from fully connected bridges.
- Training: Training aligns predicted entities and predicates with ground truth and minimizes node-level cross-entropy, with optional class-balanced weighting for imbalanced predicates.The reweighting parameter β trades mean recall against recall, and an unweighted variant is also reported as outperforming other methods.
5 Experiments
Experiments on Visual Genome evaluate GB-Net across three scene-graph tasks and 24 recall metrics, including comparisons, ablations, and implementation settings. GB-Net improves state-of-the-art performance, particularly mean recall, while class balancing further benefits rare predicates.
- Evaluation setup: Evaluation uses Visual Genome with 150 entity classes and 50 predicate classes across SGGen, SGCls, and PredCls.The benchmark contains 108,077 images and follows established train/test splits.
- Evaluation setup: The study reports mean and overall triplet recall at K=50 and K=100, with and without graph constraint, yielding 24 metrics across the three tasks.Both recall types are included for comprehensive evaluation.
- Main results: GB-Net improves both overall recall and mean recall over prior methods, while GB-Net-β raises mean recall by 2.7% on average with only a 0.2% decrease in overall recall.KERN encodes statistics more implicitly than SMN, reducing bias toward frequent classes.
- Main results: Mean recall is emphasized because overall recall can remain high while models focus on frequent predicates and neglect rare ones.The authors identify object detection as a bottleneck in the saturated SGGen setting.
- Comparison scope: Some recent methods are excluded because their evaluation settings differ or their code is unavailable, limiting direct comparison.One cited method reports only 8 of the paper’s 24 metrics, so the authors do not treat the comparison as fair.
- Ablation study: Removing the commonsense graph harms every metric, while fewer than three message-passing steps significantly reduce performance and additional steps provide little benefit.The ablation supports the contribution of commonsense bridging and iterative refinement, with performance saturating beyond three steps.
6 Conclusion
The conclusion presents GB-Net as a graphical neural framework that unifies scene and commonsense knowledge graphs through dynamic message passing and bridging. Experiments show that the method outperforms the state of the art across multiple metrics.
- Conclusion: GB-Net incorporates external commonsense knowledge into scene graph generation through a graphical neural framework.The method treats scene and commonsense graphs as two knowledge-graph types.
- Conclusion: The model fuses the graphs by iteratively updating nodes, comparing them to refine bridge edges, and repeating the process until they are connected.This describes the paper’s dynamic message-passing and bridging algorithm.
- Conclusion: Extensive experiments show that GB-Net outperforms the state of the art on various metrics.
Supplementary Material
The supplementary material extends the main evaluation with quantitative, computational, and qualitative analyses. It specifically examines predicate-frequency robustness and deployment speed.
- Supplementary analyses: The supplement analyzes whether GB-Net addresses shortcomings in modeling the tail of the predicate distribution.It includes additional quantitative analysis beyond the main manuscript.
- Supplementary analyses: The supplement also provides qualitative examples and further implementation details.
A Per-class performance
Per-class analysis compares GB-Net with KERN across predicates ordered by training frequency. GB-Net improves performance on rare predicates without sacrificing performance on frequent ones.
- Per-class performance: Figure 1 orders predicates by decreasing training frequency and compares per-class recall between GB-Net and KERN.
- Per-class performance: GB-Net substantially improves recall on the distribution tail while retaining performance on frequent predicates.The authors describe this as producing more reliable and consistent overall performance.
B Computational cost
GB-Net is evaluated for deployment and training cost against state-of-the-art methods on identical hardware. It is faster while achieving higher accuracy, with only a modest parameter increase over KERN.
- Table 1 summarizes the method's time and memory costs compared with the state of the art.
- More than twice faster training than the state of the art is reported per Visual Genome training epoch.The comparison averages training time over 10 epochs on 56,224 training images.
- GB-Net has 10% more parameters than KERN while being 52% faster during training and 34% faster during testing.The Faster R-CNN detector accounts for 139.8 million fixed parameters in all methods.
C Commonsense graph construction
The commonsense graph combines affordance, ontological, and statistical knowledge to support scene graph generation. It is built from multiple sources and uses typed, weighted edges over the task's entity and predicate classes.
- Commonsense knowledge supports scene graph generation by helping disambiguate visual classes through higher-level semantic meanings and relationships.Examples include bike affordances and hierarchical relations such as Man and Woman being subtypes of Person.
- The graph contains 150 entities and 50 predicates, matching the classes conventionally used in the scene graph generation task.The method is independent of commonsense-graph content when the nodes match target classes and facts use directed, typed, weighted edges.
- ConceptNet contributes 104 manually cleaned and pruned edges of five types among the 200 WordNet-grounded nodes.The edge types include partOf, relatedTo, isA, mannerOf, and usedFor, with reverse edges added for asymmetric relations.
- WordNet supplies ontological similarity using path, LCH, and WUP metrics with manually tuned thresholds.
- Visual Genome training data supplies conditional probabilities and correlations that encode statistical interactions between entity and predicate classes.These statistics are represented as pairwise matrices that are sparsified.
- The three knowledge sources produce a graph with 19 edge types and require manual effort during graph generation.The authors describe the resulting graph as more sophisticated and complete than graphs used in recent knowledge-aware vision systems.
D Qualitative results
Qualitative comparisons show GB-Net producing more commonsensical scene graphs under visual ambiguity. Its joint use of entity and predicate semantics improves robustness to misclassification and loose bounding boxes.
- GB-Net predicts semantically consistent entities and predicates where KERN often produces inappropriate triplets.The comparisons cover visual ambiguity, clutter, loose bounding boxes, and inappropriate predicate choices.
- Predicate semantics help GB-Net correct entity classifications that are visually ambiguous or affected by clutter.KERN and similar methods classify entities before predicates, preventing predicates from enhancing entity classification.
- GB-Net is more robust to loose bounding boxes that include substantial background and can otherwise cause entity misclassification.
- GB-Net usually selects more appropriate predicates when entities are classified correctly but KERN chooses unsuitable relations.
E Software package
The paper provides software intended to reproduce reported results, while qualitative examples illustrate GB-Net's behavior against KERN. These examples show errors involving clutter, occlusion, and visually ambiguous objects.
- The software package is designed to reproduce every reported number from Tables 1 and 2.It includes experiment-specific IPython Notebooks, a README with step-by-step instructions, and a mapping from notebooks to table cells.
- GB-Net predicts table rather than room when KERN's loose box covers the scene, avoiding triplets such as laptop on room.
- GB-Net produces clock-face relations using prior knowledge that clocks can have faces and faces can be on clocks.KERN instead predicts the nonsensical triplet clock has clock.
- Without retraining, readers can evaluate supplied parameter checkpoints using the evaluation notebooks.
- GB-Net avoids errors where KERN labels a boy as fence, yielding meaningless relations such as fence has ear and fence has nose.
- GB-Net correctly classifies a fruit-covered table, avoiding the nonsensical relation banana in fruit.
- GB-Net favors a commonsensical graph by predicting kite rather than the visually similar class tail.
- GB-Net correctly detects two pieces of curtain on window instead of KERN's curtain on curtain prediction.