Source-linked AI summary
Knowledge-Embedded Routing Network for Scene Graph Generation
Tianshui Chen, Weihao Yu, Riquan Chen, Liang Lin
TL;DR
Scene graph generation must infer object relationships despite severely uneven relationship frequencies, which particularly harms less frequent relationships. The paper develops KERN, explicitly encoding object–relationship statistical correlations as a knowledge graph and learning message propagation through it. On Visual Genome, KERN outperforms leading methods and substantially improves mean recall, including mR@50 from 15.4% to 19.8% and mR@100 from 20.6% to 26.2% on scene graph classification.
Problem
Severely uneven relationship frequencies make existing scene graph generation methods perform poorly on less frequent relationships.
Method
KERN explicitly represents object–relationship statistical correlations in a structured knowledge graph and learns routing-based message propagation to capture their interplay.
Results
KERN consistently outperforms existing methods across the reported tasks and settings, with mR@50 and mR@100 on scene graph classification increasing from 15.4% and 20.6% to 19.8% and 26.2%.
Takeaways & Limitations
Explicit statistical knowledge regularizes relationship prediction and helps address the uneven distribution of real-world relationships.
Abstract
from arXiv · showhide
To understand a scene in depth not only involves locating/recognizing individual objects, but also requires to infer the relationships and interactions among them. However, since the distribution of real-world relationships is seriously unbalanced, existing methods perform quite poorly for the less frequent relationships. In this work, we find that the statistical correlations between object pairs and their relationships can effectively regularize semantic space and make prediction less ambiguous, and thus well address the unbalanced distribution issue. To achieve this, we incorporate these statistical correlations into deep neural networks to facilitate scene graph generation by developing a Knowledge-Embedded Routing Network. More specifically, we show that the statistical correlations between objects appearing in images and their relationships, can be explicitly represented by a structured knowledge graph, and a routing mechanism is learned to propagate messages through the graph to explore their interactions. Extensive experiments on the large-scale Visual Genome dataset demonstrate the superiority of the proposed method over current state-of-the-art competitors.
1. Introduction
Scene graph generation represents objects and their pairwise relationships, but uneven relationship frequencies make rare relationships difficult to predict. KERN addresses this by explicitly embedding statistical object–relationship knowledge into graph propagation and reports stronger performance, especially on mean recall.
- Scene graphs encode object semantics, spatial information, and relationships between object pairs.
- Uneven real-world relationship frequencies cause existing methods to perform poorly on relationships with limited training samples.
- Statistical correlations between object pairs and relationships can regularize the prediction space and address uneven relationship distributions.
- KERN explicitly represents object–relationship correlations in a structured knowledge graph and propagates messages to capture their interplay.
- Mean recall@K averages recall across relationships, providing more comprehensive evaluation than recall@K under severely uneven distributions.
- 19.8% and 26.2% are KERN's mR@50 and mR@100 on scene graph classification, rising from 15.4% and 20.6%, respectively.The reported relative improvements are 28.6% and 27.2%.
2. Related Work
Prior visual relationship detection methods address broad relationship types and long-tailed distributions using visual, language, contextual, or implicitly propagated information. KERN differs by explicitly representing object–relationship co-occurrences in a structured graph and integrating them into graph propagation.
- Visual relationship detection identifies scene objects and infers relationships between subject–object pairs across spatial, action, and affiliation types.
- Earlier methods modeled subjects, relationships, and objects separately, used language priors, or refined predictions through recurrent message passing.
- Existing approaches noticed object–relationship co-occurrences but implicitly mined them through message passing.
- Knowledge-graph methods have also used structured prior knowledge to enhance image representations for recognition and zero-shot learning.
- Figure 2 depicts region correlations in one graph and correlations between an object pair and all relationships in another.
3. Proposed Model
The model represents scene graphs as object regions, labels, and pairwise relationships, then uses two knowledge-guided graphs to infer them. Statistical object co-occurrences guide contextual object recognition, while pair-specific object–relationship correlations guide relationship prediction.
- Scene graph representation: A scene graph consists of region candidates, object class labels, and relationship triplets, including a no-relationship label.The graph is represented as G = {B, O, R}.
- Inference pipeline: The model decomposes scene-graph prediction into bounding-box generation, object classification, and relationship inference.Faster RCNN generates candidate regions; subsequent components predict object labels and relationships.
- Object inference: Object-region graphs are built from training-set statistical co-occurrence probabilities and processed with message propagation to predict contextualized region labels.Each region is associated with category-specific nodes, and gated recurrent updates propagate information through the graph.
- Relationship inference: Predicted object labels define a second graph containing the object pair and all possible relationship nodes, connected by pair-specific co-occurrence statistics.The relationship graph uses K relationship nodes and correlations between each object and each candidate relationship.
- Relationship inference: A gated graph neural network propagates messages between object and relationship nodes, aggregates their final features, and predicts the relationship for each object pair.Relationship nodes receive messages from object nodes, while object nodes receive messages from relationship nodes before relationship classification.
4. Experiments
Experiments on Visual Genome evaluate KERN across three scene-graph tasks using recall and mean recall metrics, with comparisons to established methods and ablations. KERN improves especially on less frequent relationships, and removing statistical correlations substantially reduces performance.
- Experiment setting: Visual Genome experiments cover predicate classification, scene graph classification, and scene graph generation under constrained and unconstrained evaluation settings.Predicate classification uses given object labels and boxes; scene graph classification uses ground-truth boxes; scene graph generation detects objects and predicts relationships.
- Evaluation metrics: mR@K evaluates relationships more evenly than R@K because severe relationship imbalance causes R@K to be dominated by frequent relationships.The top 10 relationships account for almost 90% of Visual Genome samples, while the remaining 40 account for about 10%.
- Comparison with state-of-the-art methods: 11.7% and 26.5% mean mR under constrained and unconstrained settings improve over SMN by 30.0% and 28.6%, respectively.KERN consistently outperforms existing methods across all three tasks and both settings; these statistics are mean mR values.
- Comparison with state-of-the-art methods: 44.1% and 55.4% mean R under constrained and unconstrained settings improve over SMN by 0.4% and 0.7%, respectively.The gains on mean recall are more pronounced than those on recall because KERN performs better on less frequent relationships.
- Relationship-wise analysis: KERN improves R@50 for 47 of 50 relationships, with larger gains for relationships having fewer samples.Figure 5 relates per-relationship R@50 improvement to sample proportion in predicate classification without constraint.
- Ablative study: Replacing statistical probabilities with a uniform distribution lowers mean mR from 11.7% to 7.9% and mean R from 44.1% to 40.6%.Removing contextual message propagation also causes 0.3% drops in both mean mR and mean R.
5. Conclusion
Statistical correlations between object pairs and relationships regularize relationship prediction through a knowledge graph and learned message propagation, addressing uneven relationship frequencies. Experiments on Visual Genome demonstrate the proposed method’s superiority.
- Statistical correlations between object pairs and relationships can regularize the semantic space of relationship prediction and address uneven relationship distributions.
- KERN represents these correlations in a knowledge graph and learns a routing mechanism to propagate node messages under structured-knowledge guidance.
- Experiments on the Visual Genome benchmark demonstrate the superiority of the proposed method.