Source-linked AI summary
Scene Graph Generation with External Knowledge and Image Reconstruction
Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, Mingyang Ling
TL;DR
Existing scene graph datasets provide incomplete relationship coverage and noisy or missing object annotations, making reliable prediction difficult. The paper combines external commonsense knowledge for feature refinement with an auxiliary image-reconstruction regularizer, and reports state-of-the-art performance on VRD and VG.
Problem
Scene graph datasets have biased relationship distributions and noisy or missing object annotations, limiting reliable scene graph prediction.
Method
The model refines object and phrase features using ConceptNet and multihop reasoning, while an auxiliary image-reconstruction path regularizes scene graph generation during training.
Results
The framework outperforms state-of-the-art scene graph generation methods on the VRD and VG benchmark datasets.
Takeaways & Limitations
Combining commonsense knowledge with image-level supervision improves high-level image understanding through scene graphs.
Abstract
from arXiv · showhide
Scene graph generation has received growing attention with the advancements in image understanding tasks such as object detection, attributes and relationship prediction,~\etc. However, existing datasets are biased in terms of object and relationship labels, or often come with noisy and missing annotations, which makes the development of a reliable scene graph prediction model very challenging. In this paper, we propose a novel scene graph generation algorithm with external knowledge and image reconstruction loss to overcome these dataset issues. In particular, we extract commonsense knowledge from the external knowledge base to refine object and phrase features for improving generalizability in scene graph generation. To address the bias of noisy object annotations, we introduce an auxiliary image reconstruction path to regularize the scene graph generation network. Extensive experiments show that our framework can generate better scene graphs, achieving the state-of-the-art performance on two benchmark datasets: Visual Relationship Detection and Visual Genome datasets.
1. Introduction
Scene graph generation is difficult because current datasets incompletely cover relationships and contain noisy or missing object annotations. The paper addresses these issues with commonsense knowledge refinement and image-level reconstruction supervision.
- Scene graphs represent objects, their locations, and ⟨subject-predicate-object⟩ relationships for high-level visual understanding tasks.
- Limited relationship coverage and long-tail distributions bias models toward frequently occurring relationships.
- Missing bounding-box annotations and meaningless proposals make Visual Genome object labels noisy, hindering predicate prediction.
- Commonsense knowledge from ConceptNet is used to refine object and phrase features through multihop reasoning before scene graph generation.
- An auxiliary image-reconstruction branch regularizes training by reconstructing images from detected objects, penalizing errors that undermine scene completeness.
- Experiments on VRD and VG demonstrate that the approach improves state-of-the-art scene graph generation performance.
2. Related Works
Prior work incorporates external knowledge and message passing into visual relationship detection, while this approach combines commonsense cues with image reconstruction for scene graph generation. Its framework generates graphs, refines features with external knowledge, and uses an auxiliary reconstruction path during training.
- External knowledge bases have been used to improve data-driven neural models in natural language processing and computer vision.
- Visual relationship detection methods commonly detect objects and then recognize predicates between object pairs, with later work adding message passing for context propagation.
- Unlike efficient-generation and linguistic-regularization approaches, this method uses ConceptNet commonsense cues to address long-tail relationships through feature refinement.
- The auxiliary image-reconstruction branch is discarded after training.
3. Methodology
The framework generates scene graphs by proposing objects and subgraphs, refining their features with ConceptNet-based commonsense reasoning, and predicting object and predicate labels. An auxiliary object-to-image path reconstructs the input image, while scene-graph losses supervise detection, classification, and localization.
- Proposal Generation: The framework proposes object regions and subgraphs, then combines refined object and subgraph features to generate scene graphs.Object proposals come from an RPN; subgraphs reduce the number of potential relations through clustering and non-maximum suppression.
- Proposal Generation: For N object proposals, N(N −1) directed object-pair relations are possible, motivating subgraph proposals to reduce computational cost.Subgraph proposals are formed from union boxes of object pairs and filtered with non-maximum suppression.
- Feature Refinement with External Knowledge: ConceptNet supplies weighted commonsense relationships, which are encoded as facts and selected through attention-based episodic memory reasoning.The dynamic memory process uses multiple passes to retain useful knowledge for relationship prediction.
- Feature Refinement with External Knowledge: The knowledge module refines object features and jointly refines object and subgraph features before object and predicate recognition.The object and subgraph refinement is applied iteratively after external-knowledge refinement.
- Scene Graph Generation: Scene-graph training uses predicate classification, object classification, and bounding-box regression losses.Predicate and object outputs use softmax losses, while bounding-box regression uses smooth L1 loss.
4. Experiments
Experiments on VRD and cleansed VG-MSDN evaluate the knowledge-based and image-reconstruction components across scene-graph tasks. The full KB-GAN model improves relationship recall, object detection, robustness to missing instances, and comparison with existing methods.
- Datasets and Metrics: The framework is evaluated on VRD and cleansed VG-MSDN using PhrDet, SGGen, and Rec@50/Rec@100.Rec@K counts labeled relationships hit among the top K predictions, using detected objects rather than ground-truth locations.
- Compared Models: KB-GAN combines ConceptNet-based feature refinement with an auxiliary GAN image-reconstruction branch.KB incorporates external knowledge through DMN reasoning, while GAN reconstructs images from objects; the full model is initialized from both components and fine-tuned jointly.
- Component Analysis: KB significantly boosts PhrDet and SGGen, while adding image-level supervision further improves performance and recall.The results attribute the gains to commonsense refinement and meaningful context captured across objects.
- Object Detection: KB-GAN outperforms Faster R-CNN and ViP-CNN on object detection measured by mAP, with the knowledge component providing a substantial contribution.Table 3 reports improved object recognition for the full model and highlights the gain associated with commonsense knowledge.
- Overall Comparison and Qualitative Results: The method outperforms existing methods in recall on both datasets, while qualitative examples show high-quality relationship predictions and reasonable reconstructed images.The full model learns using both external knowledge and an auxiliary image-level regularizer.
5. Conclusion
The paper introduces a scene graph generation model that combines knowledge-base feature refinement with image-level supervision, outperforming state-of-the-art methods on VRD and VG datasets.
- The model propagates contextual information through a knowledge-base feature refinement network and regularizes generation using image-level supervision.
- The framework outperforms state-of-the-art scene graph generation methods on the VRD and VG datasets.
- Experiments indicate that incorporating commonsense knowledge and image-level supervision is fruitful for scene graph generation.
- The work presents a promising route for improving high-level image understanding through scene graphs.