Source-linked AI summary
Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions
Johanna Wald, Helisa Dhamo, Nassir Navab, Federico Tombari
TL;DR
The paper addresses the need to represent object relationships alongside object identities for 3D scene understanding. It introduces 3DSSG and a PointNet-GCN-based graph predictor, and applies the resulting graphs to cross-domain retrieval in changing indoor environments. The approach uses semantic graphs to bridge 2D and 3D domains.
Problem
3D scene understanding needs representations that capture object semantics together with contextual relationships, while semantic scene graphs remain underexplored for real-world 3D scenes.
Method
The paper introduces 3DSSG and a learned PointNet- and GCN-based method that jointly predicts semantic scene graphs from class-agnostic instance-segmented point clouds.
Results
The method is applied to cross-domain 2D-3D scene retrieval in changing indoor environments, where semantic graphs serve as an intermediate representation.
Takeaways & Limitations
Semantic scene graphs can bridge 2D and 3D domains and support retrieval using object semantics together with scene context and meaningful relationships.
Takeaways & Limitations
The graph prediction network currently takes class-agnostic instance segmentation from dense 3D ground-truth segmentation, although other geometric segmentation methods could serve as input in theory.
Abstract
from arXiv · showhide
Scene understanding has been of high interest in computer vision. It encompasses not only identifying objects in a scene, but also their relationships within the given context. With this goal, a recent line of works tackles 3D semantic segmentation and scene layout prediction. In our work we focus on scene graphs, a data structure that organizes the entities of a scene in a graph, where objects are nodes and their relationships modeled as edges. We leverage inference on scene graphs as a way to carry out 3D scene understanding, mapping objects and their relationships. In particular, we propose a learned method that regresses a scene graph from the point cloud of a scene. Our novel architecture is based on PointNet and Graph Convolutional Networks (GCN). In addition, we introduce 3DSSG, a semi-automatically generated dataset, that contains semantically rich scene graphs of 3D scenes. We show the application of our method in a domain-agnostic retrieval task, where graphs serve as an intermediate representation for 3D-3D and 2D-3D matching.
1. Introduction
The paper frames semantic 3D scene graphs as compact representations of objects, relationships, and attributes, then introduces a learned predictor and dataset for cross-domain retrieval.
- 3D scene understanding must capture object semantics, spatial context, and relationships, not only recognize and localize objects.
- Semantic 3D scene graphs represent labeled instances as nodes, meaningful relationships as edges, and attributes such as color, shape, or affordances.
- The proposed method uses PointNet and Graph Convolutional Networks to jointly infer a semantic graph from class-agnostic instance-segmented point clouds.
- 3DSSG extends 3RScan with relationships, attributes, and class hierarchies, and its 2D scene graphs can be obtained by rendering the 3D graphs.
- The paper applies semantic scene graphs to 2D-3D retrieval in changing indoor environments, using them as an intermediate representation across domains.
2. Related Work
Prior work established scene graphs for image understanding and several forms of 3D structure modeling, but semantic inter-instance relationships in real-world 3D scenes remained comparatively underexplored.
- Image scene graphs model objects as nodes, interactions as edges, and object properties as node attributes.
- Image-based graph prediction methods use strategies including message passing, graph convolutions, permutation invariance, and attention, often after object detection.
- Most 3D understanding methods emphasize semantic segmentation, detection, classification, or layout rather than explicit semantic relationships between objects.
- Object-level graph methods address parts or shape hierarchies, but generally lack semantic relationships beyond generic edges or do not model entire scenes.
- Earlier hierarchical 3D mappings organize cameras, objects, rooms, and buildings but omit semantically meaningful inter-instance relations such as support.
- Existing 2D-3D retrieval work includes CAD retrieval, single-image shape modeling, simulated occlusion, and diverse-scene benchmarks.
3. 3D Semantic Scene Graphs
3DSSG represents indoor scenes as semantically rich graphs whose nodes encode hierarchical object classes and attributes, while edges encode spatial, support, and comparative relationships.
- 3DSSG extends 3RScan with semantic scene graphs over 1482 reconstructions from 478 naturally changing indoor environments.
- 3.1. Nodes: Each graph node represents a 3D object instance assigned to a scene and described by a variable-depth class hierarchy rather than one category.
- 3.1. Nodes: WordNet hypernyms expand annotated labels into hierarchical classes; 534 class labels receive selected lexical descriptions and hierarchies.
- 3.2. Attributes: Attributes capture static visual and physical properties, dynamic states, and state-conditioned affordances across 93 attributes on approximately 21k instances.
- 3.3. Relationships: Relationships comprise spatial or proximity relations, support relations, and comparisons derived from attributes.
4. Graph Prediction
The Scene Graph Prediction Network extracts object and pairwise point-cloud features with PointNet, processes them as graph triples using a GCN, and predicts object and predicate labels jointly.
- Given a point cloud and class-agnostic instance segmentation, SGPN predicts a graph of scene objects and their relationships.
- ObjPointNet processes each instance point set, while RelPointNet processes the union of each node pair’s bounding-box point sets with an orientation mask.
- The extracted features are arranged as subject-predicate-object triples and processed by a GCN with message passing and averaged neighborhood aggregation.
- Residual connections help preserve node features against potential Laplacian smoothing, while GCN depth determines the relation order the model can capture.
- Two MLPs predict node and predicate classes, and end-to-end training optimizes object and predicate classification losses.
- Predicate loss uses per-class binary cross entropy to allow multiple valid relationships per object pair, with focal loss addressing class imbalance.
5. Scene Retrieval
The paper introduces image-based 3D scene retrieval for changing indoor environments, using semantic scene graphs as a shared representation between 2D and 3D domains. Graph matching combines object semantics, generic edges, and semantic relationships to retrieve corresponding scenes.
- Scene graphs are used in cross-domain retrieval to close the domain gap among 2D images, 3D scenes, and other modalities.
- Image-based 3D scene retrieval identifies a 3D scene from scans using a single 2D image despite global and local scene changes.The task spans 2D images and 3D models and addresses changes such as moved objects and altered illumination.
- Scene graphs provide a shared domain between 2D and 3D and are described as more stable toward scene changes.The approach also supports retrieval from other input domains that can generate scene graphs.
- Graph similarity compares multisets of node classes and semantic edge tuples rather than solving graph edit distance directly.The paper motivates this transformation because direct graph matching via graph edit distance is NP-complete.
- The retrieval method combines similarities for object semantics, generic node edges, and semantic relationships.The augmented graph includes nodes, binary edges, and semantic relationships.
- Jaccard and Szymkiewicz-Simpson coefficients are evaluated for comparing the resulting graph multisets.The Szymkiewicz-Simpson coefficient is motivated for cases where the compared sets have substantially different sizes, as in 2D-3D matching.
6. Evaluation
Evaluation covers 3D semantic scene graph prediction and 3D-3D and 2D-3D retrieval. The method outperforms the relationship-prediction baseline on graph metrics while remaining on par in object classification, and semantic relationships improve retrieval.
- Semantic Scene Graph Prediction: The 3DSSG evaluation compares the proposed graph predictor with an adapted relationship-prediction baseline using the original train and test splits.The baseline passes PointNet-translated node and edge features directly to object and predicate classifiers.
- Semantic Scene Graph Prediction: Scene graph performance is measured separately for predicate and object prediction and jointly for triplet classification using top-n recall.Joint scores are formed by multiplying independently predicted relationship and object-category scores.
- Semantic Scene Graph Prediction: The proposed method outperforms the baseline on graph-related metrics while remaining on par in object classification.The evaluation also reports triples prediction, object classification, and predicate prediction accuracy.
- Semantic Scene Graph Prediction: The multiple-predicate model achieves higher predicate accuracy, attributed to ambiguity when several outputs are plausible in a single-classification problem.
- Scene Retrieval: Retrieval evaluates matching 2D images or 3D rescans to corresponding 3D reference scans using top-n placement of true matches.Both ground-truth and predicted query and source graphs are evaluated.
- Scene Retrieval: Szymkiewicz-Simpson performs better for 2D-3D matching, Jaccard is on par for 3D-3D matching, and semantic relationships improve retrieval.
7. Conclusion
The paper presents 3DSSG and a learned network for predicting semantic scene graphs from 3D scenes, then demonstrates their use in cross-domain retrieval. It positions semantic graphs as a representation for bridging 2D and 3D scene understanding.
- The proposed graph prediction network estimates object semantics and relationships between objects in 3D scenes.
- Semantic scene graphs are applied to image-based 3D scene retrieval in changing indoor environments.
- The conclusion identifies semantic scene graphs as a way to bridge the 2D-3D domain gap and support applications such as text-3D retrieval or VQA.
8. Supplementary Material
The supplementary material expands 3DSSG statistics, annotation and representation details, implementation choices, and qualitative evaluation. It also illustrates dynamic attributes, rendered 2D graphs, and retrieval-based semantic change detection.
- 3DSSG Dataset: 3DSSG scene graphs are semantically rich and dense, containing object relationships, attributes, predicates, affordances, and common semantic connections.The supplementary statistics cover relationship and attribute distributions across scenes and object instances.
- 3DSSG Dataset: WordNet hierarchy labels augment annotated object classes with broader categories, such as chair, seat, furniture, and entity.For a chair instance, the hierarchical label is represented as c = {chair, seat, furniture, ..., entity}.
- 3DSSG Dataset: Rendered 2D scene graphs include RGB images, depth, dense semantic instance masks, textured-model masks, and graph representations rather than only bounding boxes.These rendered modalities support exploration of alternative graph-prediction and related tasks.
- 3DSSG Dataset: Dynamic attributes such as open/closed and on/off can change over time and may indicate activities that occurred in a space.The examples contrast object states across time, including toilet-seat, desk-and-floor, and bed conditions.
- Implementation and Annotation: The annotation workflow uses interfaces for binary support, semantic relationships, and attributes, with semantic support annotations following binary support annotation.The supplementary material also describes filtering dense graphs into 4–9-node subgraphs, retaining 160 object classes and 26 relationships for experiments.
- Results and Retrieval: Qualitative predictions visualize correct, partially correct, false-positive, misclassified, and missing node or edge predictions against ground truth.The supplementary retrieval examples show unmatched nodes and edges, enabling semantic change detection between scenes.