Source-linked AI summary
A Comprehensive Survey of Scene Graphs: Generation and Application
Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, Alex Hauptmann
TL;DR
Scene-graph research needed a comprehensive, systematic survey because the field is cross-modal, complex, and rapidly developing. This paper defines scene graphs, reviews SGG methods with and without prior knowledge, surveys applications and datasets, and discusses evaluation and future directions. It concludes that scene graphs remain valuable for high-level visual understanding while facing challenges including long-tailed relationships, prior-knowledge limitations, and insufficient dynamic-scene research.
Problem
Scene-graph research lacked a comprehensive, systematic survey despite rapidly increasing work and the need for higher-level visual understanding and reasoning.
Method
The paper systematically reviews scene-graph definitions, SGG methods, prior-knowledge use, applications, datasets, evaluation methods, and future research directions.
Results
The survey classifies existing SGG methods and prior-knowledge approaches, covers applications and datasets, compares evaluation and model performance, and identifies future research directions.
Takeaways & Limitations
Scene graphs provide an explicit representation of objects and relationships that serves high-level scene understanding and reasoning across visual tasks.
Takeaways & Limitations
Current research remains constrained by incomplete external knowledge, long-tailed relationship distributions, and limited attention to dynamic video-scene relationship reasoning.
Abstract
from arXiv · showhide
Scene graph is a structured representation of a scene that can clearly express the objects, attributes, and relationships between objects in the scene. As computer vision technology continues to develop, people are no longer satisfied with simply detecting and recognizing objects in images; instead, people look forward to a higher level of understanding and reasoning about visual scenes. For example, given an image, we want to not only detect and recognize objects in the image, but also know the relationship between objects (visual relationship detection), and generate a text description (image captioning) based on the image content. Alternatively, we might want the machine to tell us what the little girl in the image is doing (Visual Question Answering (VQA)), or even remove the dog from the image and find similar images (image editing and retrieval), etc. These tasks require a higher level of understanding and reasoning for image vision tasks. The scene graph is just such a powerful tool for scene understanding. Therefore, scene graphs have attracted the attention of a large number of researchers, and related research is often cross-modal, complex, and rapidly developing. However, no relatively systematic survey of scene graphs exists at present. To this end, this survey conducts a comprehensive investigation of the current scene graph research. More specifically, we first summarized the general definition of the scene graph, then conducted a comprehensive and systematic discussion on the generation method of the scene graph (SGG) and the SGG with the aid of prior knowledge. We then investigated the main applications of scene graphs and summarized the most commonly used datasets. Finally, we provide some insights into the future development of scene graphs. We believe this will be a very helpful foundation for future research on scene graphs.
1 INTRODUCTION
Scene graphs represent objects, attributes, and inter-object relationships to support higher-level visual understanding and reasoning. This survey organizes SGG methods, construction processes, applications, datasets, evaluation, and future challenges.
- Survey Scope: The survey addresses a gap in comprehensive SGG synthesis by reviewing methods, prior knowledge, applications, datasets, evaluation, and future directions.It also identifies long-tailed visual relationships as a major challenge for scene-graph research.
- Construction Process: Given an image, SGG outputs localized object instances and relationships between object pairs.The graph may include people, places, things, and object parts, with attributes describing shape, color, or pose.
- Definition: Scene graphs encode detected objects, their attributes, and relationships as a directed graph of object nodes and relational edges.Each relationship is commonly represented as a subject–predicate–object triplet.
- Construction Process: SGG pipelines commonly perform feature extraction, contextualization, and graph construction or inference.These stages encode objects or pairs, associate entities through context, and predict predicates to complete the graph.
- Construction Process: Current SGG frameworks are broadly bottom-up or top-down, differing in whether object detection and relationship recognition are staged or joint.Bottom-up methods separately predict boxes, objects, and relations, whereas top-down methods jointly infer objects and relationships from region proposals.
2 SCENE GRAPH GENERATION
Scene graph generation methods model objects and their relationships using statistical, embedding-based, convolutional, recurrent, and graph-based approaches. The survey emphasizes challenges including relational context consistency, unseen relationships, and graph structure.
- Method taxonomy: SGG methods are organized into CRF-based, TransE-based, CNN-based, RNN/LSTM-based, and GNN-based categories.These categories structure the survey’s review of scene graph generation methods.
- CRF-based SGG: CRF-based SGG models statistical correlations while SG-CRF additionally learns semantic compatibility and subject-object order.These additions target inconsistent node semantics and reversed relationship predictions.
- TransE-based SGG: TransE-based SGG maps objects and predicates into a low-dimensional space, interpreting predicates as translations between subject and object embeddings.The basic formulation is s + p ≈ o.
- TransE-based SGG: UVTransE adds a subject-object union feature and context-augmented translation embeddings to improve generalization to rare or unseen relationships.It combines vision, language, and object-detection scores to rank relationship triples.
- RNN/LSTM-based SGG: VCTree replaces chain or fully connected structures with dynamic trees that represent hierarchical and parallel relationships more efficiently.SIG similarly uses a human-mimetic hierarchy reflecting the order in which people describe subjects and key relationships.
3 SGG WITH PRIOR KNOWLEDGE
Prior knowledge assists scene graph generation by supplying semantic, statistical, visual, contextual, and commonsense information for relationship prediction. Surveyed methods use language models, statistical correlations, knowledge graphs, and learned visual commonsense, while the completeness and utilization of such priors remain challenging.
- Overview: Prior knowledge helps SGG learn visual relationship representations when relationships are numerous and training examples are limited.The survey identifies language, visual, knowledge, and contextual information as relevant forms of prior knowledge.
- Language Prior: Language priors combine visual appearance with semantic embeddings to infer relationships involving semantically related objects.Examples include transferring from “person riding a horse” to “person riding an elephant” through a tighter semantic embedding space.
- Statistical Prior: Statistical priors model regularities and co-occurrence patterns among objects and predicates to constrain relationship prediction.KERN represents statistical knowledge as a structured graph, while LKD supplements training annotations with language knowledge from Wikipedia.
- Statistical Prior: Statistical priors improve performance in corresponding scene analysis tasks.The survey reports this outcome for complex indoor scene analysis.
- Knowledge Graphs: Knowledge graphs encode commonsense structure and can be unified with scene graphs by treating the latter as image-conditioned instantiations of the former.GB-Net formulates scene graph generation as a bridge mapping between scene graphs and commonsense graphs.
- Discussion: Prior-knowledge approaches remain limited by incomplete external knowledge, heuristic co-occurrence rules, and unresolved challenges in acquiring and exploiting visual commonsense.Recent work learns visual commonsense directly from datasets to improve robustness, but the survey identifies prior acquisition and utilization as open problems.
4 LONG-TAILED DISTRIBUTION ON SGG
Long-tail distribution is a central SGG challenge because common predicates dominate relationship expressions while rarer predicates remain difficult to recognize. Existing approaches use sampling strategies, knowledge transfer, and energy-based learning, but the problem remains ongoing.
- Challenge: Long-tail distribution makes common predicates such as “on”, “have”, and “in” dominate visual relationship expressions, limiting relationship specificity.The survey contrasts these general predicates with more accurate expressions describing distinct actions and contexts.
- Challenge: Relationship complexity is O(N^2R), while separately learning objects and predicates can reduce it to O(N + R), but predicate appearance variation remains challenging.Object-predicate combinations such as ⟨person−ride−bike⟩ and ⟨person−ride−horse⟩ can differ substantially.
- Existing approaches: Zero-shot, one-shot, and few-shot learning methods address recognition across the long tail of relationship categories.These methods target the effects of uneven relationship distributions in scene graph datasets.
- Existing approaches: Knowledge transfer methods mitigate long-tail effects by transferring information from frequently observed head relationships to sparsely observed tail relationships.The survey notes that traditional debiasing methods may not distinguish desirable from undesirable prediction deviations.
- Existing approaches: Energy-based learning incorporates output-structure information into the loss and achieves good results under zero-shot and few-shot settings.The survey presents this as another direction for alleviating long-tail effects.
- Open issue: The long-tail problem remains a continuing hot issue in scene graph research.Despite extensive efforts, the survey identifies it as an unresolved research concern.
5 APPLICATIONS OF SCENE GRAPH
Scene graphs provide a representation of objects and their relationships that supports scene-understanding tasks. The survey reviews applications spanning visual-textual modeling, retrieval, reasoning, generation, and interaction recognition.
- Applications: Scene graphs represent objects and relationships explicitly, providing a structured basis for visual and textual scene-understanding applications.The survey introduces application scenarios including visual-textual transformers, retrieval, question answering, reasoning, 3D scene graphs, and interaction recognition.
5.1 Visual-Textual Transformer
Scene graphs support visual-textual applications by encoding object semantics and pairwise relationships. Research uses them for image generation, captioning, and related layout- and detail-control problems.
- Overview: Scene graphs encode objects, attributes, and pairwise relationships, supplying semantic information for image generation and image or video captioning.Their structured representation is used as reasoning information for vision-text tasks.
- Image Generation: Scene-graph-based image generation is better suited than text-only generation to complex scenes containing multiple objects and desired layouts.The survey motivates scene graphs as a way to represent relationships and layouts in complex descriptions.
- Image Generation: Layout-based generation improves controllability and flexibility but remains a difficult one-to-many problem with limited information in bounding boxes and labels.Methods must generate realistic images while conforming to scene-graph relationships.
- Layout Generation: Generating a layout from a scene graph is an important step, motivating canonical graph representations, structured layouts, and triplet-aware embeddings.These methods target improved scene-layout prediction and image-generation quality.
- Image Details: PasteGAN uses scene graphs and object crops to improve visual details and the quality of generated images.Its design includes a Crop Refining Network and an Object-Image Fuser with attention mechanisms.
- Application Scope: Scene-graph applications span visual-textual transformers, image-text retrieval, visual question answering, reasoning, 3D scene graphs, and human-object or human-human interaction recognition.These categories summarize the application scenarios illustrated in the survey.
- Image Captioning: Scene graphs address image-captioning inaccuracies by capturing semantic relationships that conventional object-feature and language-model pipelines do not fully use.Captioning methods embed, decompose, or attend over scene-graph structure to produce more accurate, diverse, or controllable descriptions.
- Image Captioning: Unpaired scene-graph-based captioning methods target the high cost and difficulty of obtaining large-scale paired image-text data.The approach captures rich semantic information from scenes without relying exclusively on image-caption pairs.
5.2 Cross-Modal Retrieval
Scene graphs provide structured semantic representations for cross-modal retrieval, including image-text retrieval and 2D-3D or 3D-3D matching. The survey describes them as interpretable alternatives or intermediate representations for multimodal search.
- Image-Text Retrieval: Scene graphs are used in cross-modal retrieval because they can provide a comprehensive and unified representation of multimodal data.Image-text retrieval is presented as a classic application in this area.
- Image-Text Retrieval: Replacing textual descriptions with scene graphs enables image retrieval using more structured and interpretable image semantics.Early scene-graph retrieval methods reported better retrieval results in their experiments.
- Cross-Modal Alignment: Semantic graphs can bridge scene graphs and text by measuring semantic correlation between different modalities.This addresses difficulties faced by text-based retrieval systems when searching large-scale image collections.
- 3D Matching: Scene graphs also serve as intermediate representations for 2D-3D and 3D-3D matching.The survey notes that related work remains limited but identifies broader cross-modal retrieval potential.
5.3 Visual Question Answering
Scene graphs support VQA by representing essential image information as graph structures and enabling graph-based visual feature learning and reasoning over questions.
- Scene graphs capture essential image information in graph structures, helping scene graph-based VQA methods outperform traditional algorithms.
- Graph networks can encode scene graphs derived from images and perform reasoning according to the questions provided.
5.4 Image Understanding and Reasoning
Image understanding and reasoning require combining detected visual components with higher-level events, interactions, and prior knowledge. Scene-graph-based approaches organize these components and priors for downstream reasoning.
- Scene graphs encode subject-relation-object triplets that are essential to image understanding and reasoning.
- A perception system detects objects and scenes, while a Bayesian Network builds commonsense knowledge and an SDG predicts object interactions.
- Interaction priors feed image reasoning models, while XNMS assigns specific functions such as object location and attention transformation to neural modules.
5.5 3D Scene Understanding
3D scene graphs extend scene-graph representations to 3D environments, providing numerically accurate object relationships and supporting understanding of complex indoor spaces.
- 3D scene graphs abstract objects and their relationships in 3D space as graphs for understanding indoor complex environments.
- Constructing a 3D scene graph requires locating objects, identifying attributes and relationships in 2D images, and integrating that information into a 3D scene.
- A building-scale 3D scene graph can contain four layers: building, rooms, objects, and cameras, with node attributes and relational edges.
5.6 Human-Object / Human-Human Interaction
Scene-graph research addresses human-object and human-human interactions through detection, recognition, and human-centric modeling. Long-tailed categories, ambiguous annotations, and complex pair construction remain central challenges.
- Human-Object / Human-Human Interaction: HOI and HHI research focuses on detecting and recognizing human-object and human-human interactions, with long-tailed relationships remaining problematic.
- Human-Object / Human-Human Interaction: HOI benchmarks include HICO-DET and HCVRD, whose natural long-tail distributions make one-shot and zero-shot detection difficult.
- Human-Object / Human-Human Interaction: Zero-shot learning is used to address scaling HOI recognition to long-tail categories, while object detection and human-object pair construction add complexity.
- Human-Object / Human-Human Interaction: Human-centric approaches first identify people and then select relevant human-object pairs, supporting multi-stream recognition and human-verb-object detection.
- Human-Object / Human-Human Interaction: Social relationship recognition uses contextual cues, attention, ambiguous-annotation handling, global features, mid-level details, and GCNs.
6 DATASETS AND PERFORMANCE EVALUATION
This section surveys commonly used scene graph datasets and evaluation protocols, emphasizing dataset diversity, relationship-retrieval tasks, graph constraints, and metrics addressing long-tailed predicates.
- Datasets: The survey covers 14 datasets: 10 static-image, 2 video, and 2 3D scene graph datasets.
- Datasets: Datasets span visual relationship prediction, unusual relations, object and relationship detection, image retrieval, video action reasoning, and 3D scene understanding.
- Datasets: VRD and HCVRD highlight long-tailed relationships, whereas VrR-VG suppresses this distribution by selecting visually relevant relationships.
- Evaluation methods: Existing relationship-detection metrics overlook long-tail distributions and graph-level coherence, motivating SGG diagnosis and broader evaluation of zero-shot retrieval.
- Evaluation methods: Evaluation includes predicate classification, scene graph classification, scene graph detection, zero-shot relationship retrieval, and sentence-to-graph retrieval.
- Evaluation methods: Mean Recall@K averages predicate-specific recalls and is generally fairer than Recall@K on datasets with pronounced long-tailed distributions.
7 FUTURE RESEARCH
The survey identifies unresolved challenges in scene graph research, including long-tailed relations, distant-object interactions, dynamic scenes, social relationships, and limited reasoning capability.
- Long-tailed distribution: Long-tailed visual relationships create an unavoidable tension between balanced training data and naturally imbalanced real-world distributions.
- Distant-object relationships: Current relationship prediction depends heavily on spatial distance, limiting detection of relationships between distant objects.
- Dynamic images: Most scene graphs are generated from static images, while dynamic object behavior and spatio-temporal relationship reasoning remain underexplored.
- Social relationships: Social relationship detection is proposed as an extension of human-object and human-human interaction detection using scene graphs.
- Reasoning methods: Online learning, reinforcement learning, and active learning are identified as possible directions because current datasets and relationship models limit continual prediction improvements.
- Overall direction: The field has broad application prospects but remains insufficiently mature and requires further exploration.
8 CONCLUSION
The survey systematically reviews scene graph generation, prior knowledge, applications, datasets, evaluation, and future directions. It is intended to provide a foundation for understanding the field and its development.
- The survey addresses the lack of a comprehensive review by classifying SGG methods, including methods that introduce additional prior knowledge.
- It also surveys scene graph applications and summarizes datasets to support readers’ selection of approaches.
- The paper discusses future development directions and presents the survey as a foundation for expanding understanding of scene graph research.