Source-linked AI summary
Generation of High-Level Concepts in 3D Scene Graphs via Autoregressive Diffusion
Jose Andres Millan-Romera, Samuel Cognolato, Holger Voos, Jose Luis Sanchez-Lopez, Luciano Serafini
TL;DR
Complete multi-level 3DSG generation is difficult because prior approaches use concept-specific rules or separate models for structure and spatial features. The paper introduces a unified autoregressive diffusion model that generates hierarchies bottom-up from observed planes, outperforming learning-based and random baselines across diverse datasets and surpassing an oracle-size one-shot model on the largest hierarchy and real single-floor data.
Problem
Prior methods rely on concept-specific rules or separate models for graph structure and spatial node features, limiting complete multi-level 3DSG generation.
Method
The model autoregressively generates complete 3DSGs bottom-up from observed plane graphs while jointly modeling structure, semantic labels, and centroids.
Results
The model outperforms learning-based and random baselines across diverse 3DSG datasets and surpasses an oracle-size one-shot model on M-C and R-F.
Takeaways & Limitations
The approach supports multi-level 3DSG generation across synthetic scenes, architectural floor plans, and real robotic sensing data.
Takeaways & Limitations
Autoregressive generation can accumulate errors when mistakes at lower hierarchy levels propagate upward.
Abstract
from arXiv · showhide
Indoor 3D Scene Graphs (3DSGs) represent environments as multi-layer hierarchies that connect observed geometric primitives (e.g., planes) to higher-level metric-semantic concepts (e.g., rooms, floors, buildings), enabling incremental spatial reasoning for robotic perception and SLAM. However, classical high-level concept generation approaches rely on hand-crafted rules for specific concept classes, while learning-based methods require separate models for graph structure and spatial node features (e.g., centroids), which limits scalability to novel classes and more complex hierarchies. We propose a unified autoregressive diffusion-based graph generative model that jointly learns structure and features, constructing complete 3DSGs bottom-up from observed vertical planes across arbitrary hierarchy depths. Our method consistently surpasses all learning-based and random baselines across 3DSG datasets spanning synthetic scenes, real architectural floor plans, and robotic sensor data, with varying layout complexity and hierarchy depth, and surpasses a one-shot model with oracle access to the target graph size on the largest hierarchy and on real single-floor data. Finally, we propose an adaptation of the Fused Gromov--Wasserstein distance for principled graph-level evaluation of generated 3DSGs against ground truth.
1 INTRODUCTION
3DSGs organize robot-observed planes into semantic and geometric hierarchies for multi-scale spatial reasoning, but complete high-level generation remains difficult. The paper proposes unified bottom-up generation and holistic graph evaluation.
- 3DSGs connect observed plane entities to increasingly higher-level spatial concepts for representing indoor environments.
- Existing methods use concept-specific heuristics or separate learned models, often covering only one or two hierarchy levels.
- The proposed model jointly generates nodes, edges, semantic labels, and centroids while progressively extending an observed plane graph.
- An adapted Fused Gromov-Wasserstein metric evaluates semantic, metric, and relational fidelity together.
- The work evaluates diverse scenarios and reports improved precision and broader multi-level coverage.
2 RELATED WORK
Prior 3DSG work progressed from handcrafted concept-specific pipelines to learning-based hierarchical aggregation and relation prediction. These approaches generally do not provide unified generation of complete multi-level graphs.
- Classical pipelines infer rooms or floors using occupancy maps, free-space clustering, or geometric cues tailored to specific concepts.
- The related work motivates a unified generative treatment beyond concept-specific heuristics and relation-only prediction.
- Neural Trees performs hierarchical node classification by message passing over an auxiliary tree whose nodes represent subgraphs.
- Other learning-based studies focus on relations between observed objects rather than complete high-level graph generation.
3 PRELIMINARIES
The paper defines indoor environments from observed vertical planes and organizes their concepts bottom-up into a hierarchical 3DSG. Its target graph connects low-level observations to progressively higher-level entities.
- The target graph encodes part-of relations between planes, walls, rooms, floors, buildings, and city-level nodes.
- A plane is represented by a 3D normal orientation, non-negative width, and 3D centroid.
- The hierarchy proceeds from planes to walls, rooms, floors, buildings, and cities.
- Each high-level concept depends directly on children in the layer below, while walls connect to planes without a parent node.
4 GENERATING 3DSGS
The model grows a 3DSG from plane observations through repeated node insertion, feature generation, and halting decisions. Proximity edges provide initial relational context for the generative process.
- Input planes are converted into a starting graph with initial proximity edges.
- Each iteration introduces nodes, samples their labels, connectivity, and positions, then predicts whether generation should halt.
- Autoregressive generation adds new nodes and edges from the existing graph and supports larger structures with lower memory use than one-shot models.
- A Graph Neural Network exploits the relational context supplied by proximity-based edges.
- The Fill model represents new labels, edges, and centroids and generates them from the sparse conditioning graph.
4.2 FLAGG FOR GRAPH EXTENSION
FLAGG extends an observed plane graph autoregressively by adding node blocks, filling their graph content, and deciding when generation stops. Its modular insertion, fill, and halting components define the complete hierarchy generation process.
- Autoregressive graph extension: At each step, FLAGG samples the size of the next node block before generating its contents.The block size determines how many new nodes are jointly added to the current graph.
- Autoregressive graph extension: The Fill model samples new-node connectivity and features conditioned on the current graph, producing the next graph state.Generated features can include labels and centroids.
- Stopping criterion: A Halting model signals when generation ends, with the positive halting signal occurring only at the final insertion step.The resulting insertion sequence contains intermediate graphs from the plane graph to the completed hierarchy.
- Modular implementation: FLAGG implements insertion and halting with GINE networks, while the Fill module can use an adapted one-shot graph generator.Training reconstructs nodes gradually removed from a data graph until reaching the observed plane graph.
4.3 GENERATING EDGES, LABELS AND CENTROIDS
The Fill module generates edges, labels, and centroids for each newly added block using adapted diffusion models. D4 separates topology, distances, and coordinate reconstruction, whereas MiDi jointly denoises categorical graph variables and coordinates.
- Generated graph content: The Fill module generates new high-level nodes’ labels, new-new and new-old edges, and centroids from the sparse conditioning graph.Plane features remain fixed because they belong to observed inputs rather than generated concepts.
- D4-based generation: D4 represents new nodes with categorical labels, edges, and centroid coordinates, then samples them from an encoded conditioning graph.Its distance-based variant generates positions through pairwise distances.
- D4-based generation: D4 factorizes generation into topology sampling, distance sampling on generated edges, and deterministic coordinate recovery with MDS.The sampled topology and recovered centroids are merged with the preceding graph state.
- MiDi-based generation: MiDi jointly denoises categorical node labels, edges, and continuous coordinates using an equivariant Graph Transformer.The conditioning graph is centered before generation, and its centroid is added back after denoising.
- Noise augmentation: Training perturbs non-protected generated nodes with label, position, and edge noise to reduce exposure bias from predicted conditioning graphs.Observed plane nodes remain protected, while existing edges may be removed and missing edges inserted.
4.4 ORDERINGS FOR 3DSGS
FLAGG’s insertion ordering determines the sequence in which nodes are generated. The proposed hierarchical ordering follows bottom-up concept layers, while hierarchical proximity additionally orders nodes spatially within each layer.
- Ordering design: FLAGG learns generation backward from an ordering that defines the node removal sequence, interpreted here as insertion order.Different orderings can influence the probability assigned to a graph.
- Hierarchical ordering: Hierarchical ordering groups nodes by concept layer and generates layers bottom-up because higher-level concepts depend on lower-level children.A concept ordering first assigns indices to node classes, which defines the ordered layers.
- Hierarchical proximity ordering: Hierarchical proximity ordering replaces uniform within-layer ordering with proximity ordering from a randomly selected starting node.Nodes equidistant from the start receive a random tie-breaking order.
4.5 ONE-SHOT GENERATION OF 3DSGS
One-shot graph generators provide a comparison point by sampling an entire graph in one operation after receiving its final size. In 3DSG generation, conditioning is complicated because only a new graph portion must be generated from an existing input graph.
- One-shot graph generation: One-shot models receive the final graph size and jointly sample connectivity, labels, and features for the whole graph.The target size is usually sampled from the dataset’s empirical size distribution.
- 3DSG conditioning: Conditioning one-shot generation for 3DSGs is difficult because the model must generate only a new graph portion from an existing input graph.
5 HIERARCHICAL 3DSG GENERATION BENCHMARK
The benchmark combines nine hierarchical 3DSG subsets with graph-matching and distribution-based evaluation to assess structural, geometric, semantic, and topological fidelity.
- Datasets: Nine subsets span synthetic scenes, MSD architectural floor plans, and real LiDAR-recorded environments, with progressively deeper hierarchy levels.Real-world data are limited to testing, while synthetic and MSD sources support training, validation, and testing.
- Datasets: Deeper hierarchies produce larger and more complex graphs, while average node degree remains approximately constant because of their shared tree-like structure.MSD and real sources contain substantially larger and more complex graphs than the synthetic source.
- Distribution evaluation: Distribution evaluation uses MMD with an RBF kernel over degree, clustering, spectrum, and randomly initialized GIN embedding features.These feature-specific metrics compare generated graphs with the original graph distribution.
- Graph matching evaluation: FGW aligns generated and ground-truth hierarchies despite differing node counts by combining feature-based Wasserstein and topology-based Gromov-Wasserstein distances.The method pads the smaller graph with isolated dummy nodes so real-to-dummy matches are penalized.
- Graph matching evaluation: The Wasserstein term matches nodes using centroids and labels, whereas the Gromov-Wasserstein term compares shortest-path structure.Together, these terms evaluate metric-semantic attributes and relational topology.
6 EXPERIMENTS
Experiments compare the proposed methods with learning-based, random, and oracle-assisted one-shot baselines across synthetic, MSD, and robot-recorded environments. The proposed variants consistently outperform non-oracle baselines, with strong gains across hierarchy sizes and evaluation metrics.
- Evaluation protocol: Experiments use 60% training, 20% validation, and 20% test splits, selecting checkpoints by validation FGW and reporting test metrics.Most experiments use three random seeds, while MIDI+OR and OURS-MIDI use one seed because of computational constraints.
- Baselines: The evaluation compares EC baselines, RAND, MIDI+OR, and two proposed variants across datasets with different hierarchy coverage.EC methods generate only walls and rooms, while RAND, MIDI+OR, and the proposed methods generate full floor, building, and city hierarchies.
- Overall results: OURS-FD4 and OURS-MIDI consistently outperform EC and RAND in FGW, GW, and W without privileged information.MIDI+OR achieves lower errors on several datasets with oracle graph-size access, but OURS-FD4 surpasses it on the largest hierarchy, M-C, and real single-floor data, R-F.
- Synthetic and MSD results: 50% FGW reductions occur for both proposed variants against EC-L on rectangular synthetic layouts, while OURS-MIDI reduces FGW by 65% on M-F.For larger hierarchies, OURS-MIDI reduces RAND FGW by 93% on M-B, and OURS-FD4 reduces it by 57% on S-C and 73% on M-C.
- Additional metrics: 87% Degree and 96% Spec. reductions versus EC-L occur for OURS-MIDI on M-F, while OURS-FD4 achieves 59% and 84% reductions versus RAND on M-C.At least one proposed variant has the lowest non-oracle GIN score on every dataset, with values no higher than 0.29.
- Real environments: 41% FGW reduction versus EC-L occurs for OURS-FD4 on real single-floor data, while reductions versus RAND reach 80% on R-B and 74% on R-C.OURS-MIDI remains close to OURS-FD4 on R-F but more than doubles its error on R-B, where OURS-FD4 is more robust under the training-to-robot environment shift.
7 DISCUSSION AND LIMITATIONS
The model improves structural, semantic, and metric generation across increasingly complex 3DSGs, but performance degrades as layouts and hierarchies become more complex. Its autoregressive design avoids graph-size oracles, while introducing error accumulation and possible spurious edges.
- The model improves structural, semantic, and metric generation over non-oracle baselines across increasing layout complexity and hierarchy depth.
- FGW error approximately doubles from synthetic to MSD single-floor layouts as real plans add layout variability, rooms, and non-rectangular geometries.
- OURS-FD4 surpasses MIDI+OR on the largest hierarchy M-C (0.20 vs. 0.28) and real single-floor R-F (0.13 vs. 0.32), despite lacking oracle graph size at deployment.
- OURS-MIDI performs better on MSD floor and building datasets, whereas OURS-FD4 performs better on deepest hierarchies and all robot-recorded environments.
- Learning adjacency increases flexibility and richer hierarchy representation but also raises the likelihood of spurious edges and clustering-related error.
- GW trends consistently with degree-, spectral-, clustering-, and GIN-based scores, while FGW combines structural and node-feature discrepancies.
8 CONCLUSION
The paper presents an autoregressive diffusion-based model that generates complete 3DSG hierarchies from observed plane graphs and evaluates them across synthetic, architectural, and robotic settings. It outperforms learning-based baselines, remains competitive with an oracle one-shot model, and identifies dynamic scenes, richer observations, larger environments, and online updates as future directions.
- The model jointly generates structural, semantic, and metric information across complete hierarchies from walls through city-level nodes.
- Across synthetic scenes, architectural floor plans, and robotic sensing data, the model outperforms learning-based baselines and extends to multi-floor and multi-building scenarios.
- The model remains competitive with a one-shot model given oracle graph size, surpassing it on the largest hierarchy and real single-floor data.
- Future work includes dynamic scenes, object-level observations beyond planes, larger urban environments, and online hierarchy updates.