Source-linked AI summary
Scene Graph Generation: A Comprehensive Survey
Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, Mohammed Bennamoun
TL;DR
Scene graph generation seeks accurate, structured representations of visual scenes, but the field faces challenges in relationship definitions, evaluation, and multimodal representation. This survey reviews 138 works across input modalities, organizes image-based methods by feature extraction and fusion, compares datasets and performance, and discusses open problems and future directions. It synthesizes current methods and findings while identifying boundaries imposed by data-dependent priors and fragmented 3D representations.
Problem
Scene graph generation requires accurate object and relationship labeling, yet relationship definitions, evaluation metrics, and scene representations remain challenging.
Method
The survey reviews 138 representative works, organizing image-based SGG methods by feature extraction and fusion and covering 2D, spatiotemporal, and 3D settings.
Results
The survey synthesizes methods, datasets, performance comparisons, challenges, and opportunities across scene graph generation research.
Takeaways & Limitations
The survey provides a systematic account of current SGG research intended to support deeper understanding of its methods, status, and future directions.
Takeaways & Limitations
Prior-information methods are data-dependent and do not provide a universal SGG network, while 3D scene-graph representations remain fragmented across visual modalities.
Abstract
from arXiv · showhide
Deep learning techniques have led to remarkable breakthroughs in the field of generic object detection and have spawned a lot of scene-understanding tasks in recent years. Scene graph has been the focus of research because of its powerful semantic representation and applications to scene understanding. Scene Graph Generation (SGG) refers to the task of automatically mapping an image into a semantic structural scene graph, which requires the correct labeling of detected objects and their relationships. Although this is a challenging task, the community has proposed a lot of SGG approaches and achieved good results. In this paper, we provide a comprehensive survey of recent achievements in this field brought about by deep learning techniques. We review 138 representative works that cover different input modalities, and systematically summarize existing methods of image-based SGG from the perspective of feature extraction and fusion. We attempt to connect and systematize the existing visual relationship detection methods, to summarize, and interpret the mechanisms and the strategies of SGG in a comprehensive way. Finally, we finish this survey with deep discussions about current existing problems and future research directions. This survey will help readers to develop a better understanding of the current research status and ideas.
1 INTRODUCTION
Scene graphs provide structured semantic representations that support scene understanding and multiple vision-language applications. This survey organizes 138 SGG works, emphasizing feature representation and refinement, datasets, evaluation, and open research issues.
- Applications: Scene graph generation maps images into visually grounded graphs whose objects and relationships support captioning, VQA, retrieval, image generation, and referring expression comprehension.The graph can also support more complex visual reasoning.
- Motivation: Scene graphs represent visual scenes structurally and provide contextual cues for recognition and higher-level vision tasks.They act as a bridge between visual scenes and textual descriptions.
- Survey scope: The survey reviews 138 scene graph generation papers and systematically analyzes 2D methods through feature representation and refinement.It connects methods addressing long-tail distributions and intra-class diversity through prior knowledge, commonsense knowledge, message passing, attention, and visual translation embedding.
- Survey scope: The paper reviews datasets for 2D, spatio-temporal, and 3D scene graph generation and analyzes corresponding method performance.A curated list of scene graph generation methods is also provided.
- Open problems: The survey concludes by discussing open problems and future research directions in scene graph generation.Its organization includes methods, datasets, performance comparisons, challenges, opportunities, and conclusions.
2 SCENE GRAPH
A scene graph is a directed, visually grounded representation of objects, attributes, and relations. The survey extends this representation from 2D images to 3D and spatiotemporal settings while noting viewpoint and representation challenges.
- Definition: Scene graphs explicitly model objects, object attributes, and relations between paired objects, with objects grounded by bounding boxes.Examples include object categories, colors or states, and relations such as a person jumping over a fire hydrant.
- Formalization: The survey treats a relationship or triplet as a subject-relation-object tuple and defines a scene graph as a set of visual triplets over objects, attributes, and relations.The formal representation includes an “is” relation when only one object is involved.
- Structure: For semantic expression, objects with their attributes are treated as nodes and relations as directed edges in a graph.Subgraphs, triplets, objects, and attributes can correspond to regions or substructures in the scene.
- Visual relationships: Visual relationships differ from relations in social networks and knowledge bases because they arise incidentally from image content rather than intentional construction.This distinction motivates treating scene graphs as visual, scene-grounded representations.
- Modalities: 2D projections can leave 3D spatial relationships incomplete or ambiguous because viewpoint and dimensionality reduction affect the representation.The survey therefore includes 3D and spatiotemporal scene graphs in addition to 2D graphs.
3 SCENE GRAPH GENERATION
Scene graph generation parses images or image sequences into structured representations by detecting objects, attributes, and visual relationships. The survey reviews more than one hundred deep learning methods, prioritizing 2D SGG before 3D and spatiotemporal approaches.
- Task definition: The goal of SGG is to generate a structured representation from an image or image sequence that bridges visual and semantic perception.The task is generally bottom-up: entities are grouped into triplets, which are connected into a scene graph.
- Task definition: Visual relationship detection is central to SGG because the task must identify subject-relation-object triplets.Visual Relationship Detection and scene graph generation are often treated as the same task.
- Probabilistic formulation: The scene graph probability can be decomposed into components for candidate regions, object labels, attributes, and relations.When attribute detection and relationship prediction are independent, the distribution includes separate attribute and relation factors.
- Pipeline: The bounding-box component proposes candidate regions, the object component predicts their classes, and the attribute and relation components infer object properties and pairwise relationships.The first two steps correspond to two-stage object detection and can use Faster R-CNN.
- Survey organization: The survey reviews more than one hundred deep learning methods, covering 2D SGG comprehensively before reviewing 3D and spatiotemporal methods.The ordering reflects the larger volume of published 2D SGG work.
3.1 2D Scene Graph Generation
2D scene graph generation methods first detect objects or region proposals, then represent and refine multimodal features to predict visual relationships. The survey organizes these methods around feature representation and refinement, including priors, commonsense knowledge, message passing, attention, and visual translation embeddings.
- Framework: 2D SGG commonly detects objects first and classifies relationships between object pairs, while alternative approaches jointly infer objects and relationships from region proposals.Both approaches use object or proposal detection and pair objects using union-region relation features.
- Feature Representation: Multimodal feature methods combine appearance, semantic, spatial, and contextual information because each captures different cues for visual relationship detection.The surveyed feature combinations include appearance, spatial, depth, mask, label, and broader scene context.
- Framework: The framework uses detected subjects, objects, and predicate regions before feature representation, feature refinement, and final relation prediction.The survey focuses on feature representation and refinement rather than object detection itself.
- Prior Information: Prior information uses visual-phrase frequencies and language semantics to regularize relationship prediction, but training-data statistics remain data-dependent and exclude knowledge beyond object and predicate categories.The long-tailed relationship distribution makes balanced training data costly, motivating commonsense knowledge beyond statistical and language priors.
- Feature Refinement: Message passing refines object and relationship features through local triplet propagation or global propagation across scene elements, including graph, chain, and tree structures.VCTREE uses a dynamic tree and bidirectional TreeLSTM to encode visual context, offering hierarchy, dynamicity, and efficiency over chains and fully connected graphs.
3.2 Spatio-Temporal Scene Graph Generation
Spatio-temporal scene graph generation extends visual relationship detection to videos by modeling spatial and temporal interactions among tracked objects. Its performance depends on video object detection, relation prediction, and association, while blur, camera motion, and occlusion remain major obstacles.
- Motivation: Videos enable visual relationship models to capture dynamic interactions and relations requiring temporal observations.Examples include distinguishing “man, lift up, box” from “man, put down, box.”
- Problem formulation: ST-SGG represents video relationships as a spatio-temporal graph because relations correlate across spatial and temporal dimensions.
- Challenges: Video object detection remains a bottleneck because blur, camera motion, and occlusion reduce localization accuracy and amplify errors in downstream relationship detection.
- Methods: The common VidVRD pipeline separates multi-object tracking, relation prediction, and relation-instance association over fixed-duration video segments.Relations are first predicted between co-occurring short-term object tracklets before association.
- Methods: Relation prediction extracts motion and low-level visual features, relative object characteristics, and additional pairwise information from object tracklets.
- Methods: GSTEG models entities and their dependencies in a spatial-temporal fully connected graph with an adaptively parameterized energy function for diverse relations.The survey reports state-of-the-art performance for this approach.
3.3 3D Scene Graph Generation
3D scene graph generation extends scene understanding into three-dimensional environments, where object interactions are more complex than in 2D images. The field lacks a unified definition, while proposed representations encode objects, relations, attributes, and hierarchical spatial concepts.
- Motivation: 3D scene graph generation has gained attention alongside advances in 3D object detection and semantic scene segmentation.
- Motivation: 3D scene understanding is generally more complicated than 2D scene graph generation because it represents object interactions in three-dimensional space.
- Existing approaches: Early 3D scene graph work extended 2D graph structures across multiple views, while other studies modeled cardinal-direction and support relations from RGB-D images.
- Open issues: There is no unified definition or representation of 3D scene graphs.
- Representations: Proposed 3D representations encode object categories, pairwise relations, physical attributes, and hierarchical concepts such as cameras, objects, rooms, buildings, places, and agents.Dynamic variants represent spatio-temporal relations in layered directed graphs.
- Open issues: The survey identifies long-tail distributions and large intra-class diversity as continuing challenges for generating useful scene graphs.
4 DATASETS
The survey organizes scene graph datasets into 2D images, videos, and 3D representations. These datasets vary in scale, annotation type, relation coverage, temporal structure, and strategies for handling rare or noisy labels.
- Dataset overview: Scene graph datasets are grouped into three categories: 2D images, videos, and 3D representations.
- 2D image datasets: 2D image datasets dominate visual relationship detection research and include benchmarks for visual phrases, scene graphs, spatial relations, rare relationships, and general relationship detection.
- 2D image datasets: Visual Relationship Detection uses 4,000 training images and 1,000 test images to benchmark long-tailed relationship prediction across action, verbal, spatial, preposition, and comparative categories.
- 2D image datasets: Visual Genome provides diverse relation triplets and object categories, but crowd-sourced annotations can contain poor quality, overlapping boxes, and ambiguous object names.
- 2D image datasets: VG150 refines Visual Genome annotations and evaluates 150 object categories with 50 predicates, producing about 11.5 objects and 6.2 relationships per image.
- 2D image datasets: Open Images contains 9M images and 16M bounding boxes across 600 object classes, with annotations including object locations, segmentation masks, visual relationships, and localized narratives.
- Video datasets: Video datasets support spatio-temporal relation understanding through object trajectories and temporally bounded relationships, including ImageNet-VidVRD and VidOR.ImageNet-VidVRD contains 1,000 videos, while VidOR contains 10,000 user-generated videos with dense object and predicate annotations.
- 3D datasets: 3D datasets provide scene graph annotations over scanned or reconstructed environments using representations such as meshes, point clouds, voxels, object attributes, class hierarchies, and semantic relations.
5 PERFORMANCE EVALUATION
The survey defines standard evaluation settings and metrics for 2D, video, and zero-shot scene graph generation, then compares representative methods across common datasets. Results highlight dataset effects, knowledge-based approaches, feature-refinement strategies, and the added complexity of spatio-temporal evaluation.
- Evaluation Tasks: SGG evaluation covers Phrase Detection, Predicate Classification, Scene Graph Classification, Scene Graph Generation, and video Relation Detection and Relation Tagging.These settings differ in whether objects, categories, predicates, localization, or temporal tracks are given or predicted.
- Evaluation Metrics: Recall@K measures whether correct relationships appear among the top K predictions, while meanRecall averages recall across predicates to represent performance beyond frequent relations.Weighted triplet Recall downweights frequent triplets, and zero-shot Recall evaluates detection of unseen relationships under long-tailed distributions.
- Evaluation Metrics: Unconstrained recall permits multiple predictions per object pair, addressing synonymy and annotation noise that make perfect top-1 triplet matching unreasonable.Recallk@K retains the top-k predictions per subject-object pair before selecting the K most confident predictions overall.
- Dataset Comparisons: VG150 generally yields better method performance than VRD because its data were cleaned and enhanced, making it a more representative comparison benchmark.Only a few methods were evaluated on both datasets, so cross-dataset comparisons remain limited.
- Representative Results: SABRA and HET achieve state-of-the-art PhrDet and RelDet results on VRD, respectively, using negative-sample subdivision and object-salience weighting.On VG150, RiFa benefits from semantic information in objects and relations, while Gb-net connects scene graphs with knowledge graphs to use commonsense knowledge.
- Spatio-Temporal Evaluation: Spatio-temporal SGG is more complex than 2D SGG because it requires object tracking, temporal segmentation, and merging relationships across segments.Performance is reported on ImageNet-VidVRD and VidOR using Relation Detection and Relation Tagging.
6 CHALLENGES & FUTURE RESEARCH DIREC-
The survey identifies unresolved challenges in scene graph generation, including inefficient relation prediction, long-tailed relationships, visual variation, ambiguous definitions, imperfect evaluation, and limited 3D comparability. It proposes richer data, scenario-specific modeling, ensemble methods, and unified 3D graph structures as future directions.
- Challenges: SGG remains difficult because real scenes contain many objects and relation categories, while meaningful relationships are relatively scarce.Predicting all object pairs and classifying them does not scale efficiently to real-world images.
- Challenges: Long-tailed relationship distributions make rare relationships and triplets difficult to recognize accurately.Frequent coarse predicates can dominate more precise but less common relations during testing.
- Challenges: Visual appearances of the same relation vary substantially across scenes, making discriminative feature extraction challenging.The survey emphasizes extracting more discriminative visual features rather than relying only on semantic features.
- Challenges: Ambiguous relationship definitions can assign different reasonable predicates to the same relation, making SGG datasets noisy and the task ill-posed.The survey identifies a well-defined relationship set as a key challenge.
- Challenges: Recall@K and meanRecall@K remain imperfect because they provide relative performance statistics rather than fully measuring scene-graph quality.Even Recall@50=100 would not necessarily imply a perfect generated scene graph.
7 CONCLUSION
This paper surveys deep-learning developments in scene graph generation across 2D images, videos, and 3D representations. It reviews datasets, compares methods, and discusses challenges and opportunities to support deeper research on SGG.
- The survey covers representative 2D, spatio-temporal, and 3D scene-graph generation methods.
- It summarizes widely used visual-relationship and scene-graph datasets grouped into 2D images, video, and 3D representations.
- The paper compares the performance of different approaches across datasets and discusses challenges, problems, and research opportunities.