PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systems

Nathan Hughes, Yun Chang, Siyi Hu, Rajat Talak, Rumaisa Abdulhai, Jared Strader, Luca Carlone

arXiv:2305.07154v1cs.RO

TL;DR

Flat metric-semantic maps scale poorly with environment size and semantic vocabulary, limiting compact representations for spatial perception. The paper develops hierarchical 3D scene graphs and algorithms for their incremental construction, and reports that Hydra runs in real time on a multicore CPU.

  • Problem

    Flat metric-semantic maps scale poorly with environment size and the vocabulary of semantic labels, while sensor data is costly to store for long-term symbol grounding.

    Source: §I · p. 2, A. Symbols and Symbol Grounding · p. 3

  • Method

    The paper uses hierarchical representations and combines geometric processing, topology, and geometric deep learning to construct indoor 3D scene graphs incrementally.

    Source: §I · p. 2, §VIII · p. 28

  • Results

    Hydra runs in real time on a multicore CPU, with only 2D semantic segmentation relying on GPU computing.

    Source: §V · p. 17

  • Takeaways & Limitations

    Hierarchical representations compress semantic information and support efficient inference through layered graphs with small treewidth.

    Source: §I · p. 2

    The places sub-graph omits terrain type and steepness, which the authors identify as important for outdoor extensions, particularly for ground-robot traversability.

    Source: §VIII · p. 29

Abstract

from arXiv · show

3D spatial perception is the problem of building and maintaining an actionable and persistent representation of the environment in real-time using sensor data and prior knowledge. Despite the fast-paced progress in robot perception, most existing methods either build purely geometric maps (as in traditional SLAM) or flat metric-semantic maps that do not scale to large environments or large dictionaries of semantic labels. The first part of this paper is concerned with representations: we show that scalable representations for spatial perception need to be hierarchical in nature. Hierarchical representations are efficient to store, and lead to layered graphs with small treewidth, which enable provably efficient inference. We then introduce an example of hierarchical representation for indoor environments, namely a 3D scene graph, and discuss its structure and properties. The second part of the paper focuses on algorithms to incrementally construct a 3D scene graph as the robot explores the environment. Our algorithms combine 3D geometry, topology (to cluster the places into rooms), and geometric deep learning (e.g., to classify the type of rooms the robot is moving across). The third part of the paper focuses on algorithms to maintain and correct 3D scene graphs during long-term operation. We propose hierarchical descriptors for loop closure detection and describe how to correct a scene graph in response to loop closures, by solving a 3D scene graph optimization problem. We conclude the paper by combining the proposed perception algorithms into Hydra, a real-time spatial perception system that builds a 3D scene graph from visual-inertial data in real-time. We showcase Hydra's performance in photo-realistic simulations and real data collected by a Clearpath Jackal robots and a Unitree A1 robot. We release an open-source implementation of Hydra at https://github.com/MIT-SPARK/Hydra.

I. INTRODUCTION

The introduction argues that scalable spatial perception requires hierarchical representations and presents algorithms and Hydra for building, maintaining, and evaluating indoor 3D scene graphs in real time. It connects symbolic concepts to geometric maps and combines geometry, topology, and geometric deep learning.

I. INTRODUCTION

  • Flat metric-semantic maps scale poorly, while prior hierarchical scene-graph methods either operate offline or omit upper layers to run in real time.Prior LIDAR-based situational graphs focus on geometric features and lack the object and room-label semantics considered here.Source: §I · p. 2
  • Hierarchical representations reduce memory through lossless semantic and lossy geometric compression, while small treewidth supports efficient inference.For indoor scene graphs, treewidth depends on the maximum number of objects in a room rather than the total number of graph nodes.Source: §I · p. 2
  • Incremental construction combines geometric processing for meshes and places, persistent-homology room clustering, and neural trees for room labels.Incremental tree-decomposition and bounded scene-graph treewidth support real-time neural-tree inference on an embedded GPU.Source: §I · pp. 2–3
  • Hierarchical loop-closure detection uses scene-graph descriptors to find candidate matches, then geometric verification estimates their poses.Embedded deformation graphs correct the mesh, objects, places, and robot trajectory together in response to loop closures.Source: §I · p. 3
  • Online algorithms achieve accuracy comparable to batch offline methods, build richer representations than competing approaches, and detect more loop closures with higher quality than bag-of-words and visual-feature matching.Experiments include challenging simulated and real datasets across an apartment complex, an office building, and two student residences.Source: §I · p. 3
  • Compared with prior work, the paper formalizes why hierarchy supports scalable scene understanding and adds persistent-homology segmentation and learning-based place-recognition descriptors.It also extends neural trees across rooms and their relations and evaluates on real robots, including Clearpath Jackal and Unitree A1 platforms.Source: §I · p. 3

A. Symbols and Symbol Grounding

  • Symbols must be grounded in geometric representations for robot action, and the symbols a map contains depend on the robot's task.The paper focuses on indoor navigation to objects and rooms, representing free space, objects, rooms, and buildings.Source: A. Symbols and Symbol Grounding · pp. 3–4

B. Hierarchical Representations Enable Large-Scale Mapping

Hierarchical representations separate semantic symbols from dense geometric data, reducing storage by avoiding repeated labels for every voxel. Their layered graph structure also supports tree decompositions whose treewidth is bounded by the complexity of individual layers rather than the size of the full graph.

B. Hierarchical Representations Enable Large-Scale Mapping

  • Flat metric-semantic storage scales with both the voxel count V/δ^3 and the number of labels L, making it unsuitable for large-scale spatial perception.The paper illustrates the scale with 10^10 voxels for a 10 km × 10 km area at 10 cm resolution, even without the vertical dimension.Source: B. Hierarchical Representations Enable… · p. 4
  • For a 336-voxel example with L = 5, hierarchy reduces storage from 1,680 symbols to 355 symbols and 354 edges without information loss.Only the lowest-level symbols are directly grounded in voxels; higher layers organize them hierarchically.Source: B. Hierarchical Representations Enable… · pp. 4–5
  • Compressing the sub-symbolic layer can further reduce memory, but may approximate geometry and degrade task performance, such as by producing longer motion-planning paths.The paper describes compression as adjustable to the robot’s available memory.Source: B. Hierarchical Representations Enable… · p. 5

C. Hierarchical Representations Enable Efficient Inference

  • A tree decomposition of a hierarchical graph can be constructed by concatenating decompositions of its top layer and the child subgraphs at each level.The procedure adds each parent node to its children’s decomposition bags before joining the decompositions.Source: C. Hierarchical Representations Enable… · p. 6
  • The resulting treewidth is bounded by the treewidths of the layer subgraphs, rather than growing with the total number of graph nodes.The paper identifies small treewidth as a condition for polynomial-time graphical-model inference and for applying neural trees.Source: C. Hierarchical Representations Enable… · pp. 5, 7

D. 3D Scene Graphs and Indoor Environments

The paper specializes hierarchical graphs to indoor 3D scene graphs, whose layers represent geometry, objects and agents, places, rooms, and a building. It analyzes how the room and object layers constrain treewidth and reports empirical upper bounds from Matterport3D scenes.

D. 3D Scene Graphs and Indoor Environments

  • An indoor 3D scene graph layers a metric-semantic mesh, objects and agents, topological places, rooms, and a building, with within-layer and cross-layer relations.The setup assumes one building; objects, rooms, and buildings are grounded by centroids and bounding boxes, while places encode obstacle-free locations and traversability.Source: D. 3D Scene Graphs and Indoor… · p. 8
  • Room-graph treewidth is at most 2 when each room connects to at most two other rooms that have multiple entryways; the 90 Matterport3D scenes also have observed upper bounds at most 2.The condition permits a room to connect to arbitrarily many rooms accessible through a single-entry corridor.Source: D. 3D Scene Graphs and Indoor… · p. 8
  • Object-subgraph treewidth is bounded by the maximum number of objects in a room, and its observed upper bounds remain below 20 in all tested scenes.The object-subgraph bounds tend to exceed those of the room subgraphs because objects in different rooms are not connected.Source: D. 3D Scene Graphs and Indoor… · p. 8
  • Under the assumption that room-graph treewidth is lower than object-graph treewidth, object-room-building graph treewidth does not grow with scene size but depends on room clutter.The clutter term is the largest number of objects in a room.Source: D. 3D Scene Graphs and Indoor… · pp. 8–9

3D SCENE GRAPH LAYERS CONSTRUCTION

The construction pipeline estimates an odometric 3D scene graph from visual-inertial data as the robot explores, with loop-closure correction handled subsequently. It builds geometry, objects, places, and rooms in sequence, then adds a building node.

3D SCENE GRAPH LAYERS CONSTRUCTION

  • Loop-closure correction of the scene graph is addressed after the odometric graph construction stage.The paper assigns scene-graph correction in response to loop closures to the subsequent section.Source: 3D SCENE GRAPH LAYERS CONSTRUCTION · p. 9
  • From visual-inertial data, the system first reconstructs a metric-semantic mesh and extracts objects from it while the robot explores.These populate the mesh and object layers of the scene graph.Source: 3D SCENE GRAPH LAYERS CONSTRUCTION · p. 9
  • The pipeline derives places by sparsifying a generalized Voronoi diagram, segments rooms using persistent homology, and classifies room labels with a neural tree.It assumes a single-building scene and connects one building node to the estimated rooms.Source: 3D SCENE GRAPH LAYERS CONSTRUCTION · p. 9

A. Layers 1: Mesh

The scene graph combines an active-window metric-semantic mesh with object, agent, and place layers. Places are derived incrementally from the GVD and sparsified into a graph.

A. Layers 1: Mesh

  • An active-window TSDF is incrementally converted into a semantic mesh and integrated into a lighter-weight full mesh, which remains uncorrected for odometric drift.The active window bounds voxel-map memory; loop-closure correction is addressed separately.Source: A. Layers 1: Mesh · p. 9

B. Layers 2: Objects and Agents

  • Object nodes store semantic labels, centroids, and bounding boxes, while agents are represented by the robot's pose graph.Objects are clustered by semantic label and merged with existing same-class nodes when their bounding boxes overlap.Source: B. Layers 2: Objects and Agents · p. 9

C. Layers 3: Places

  • The incremental GVD and places graph are produced during 3D mesh construction, avoiding prior approaches that relied on computationally expensive, offline processing of a monolithic ESDF.The GVD is sparsified because it can contain over 10,000 voxels in the Office dataset and prior sparsification could fail to preserve free-space connectivity.Source: C. Layers 3: Places · pp. 10–11
  • A user-specified spatial resolution clusters connected GVD voxels into place nodes and proposes edges according to their connectivity.The clustering preserves connected components in the voxel graph while allowing the user to choose the compression resolution.Source: C. Layers 3: Places · p. 11

D. Layer 4: Rooms

Rooms are formed by clustering the sparse place graph across dilation scales, then assigning room semantics from the object-room graph. A neural tree uses that hierarchy for room classification.

D. Layer 4: Rooms

  • Dilation removes place-graph nodes and edges closer to obstacles than δ; the connected components of the resulting graphs correspond to candidate rooms.The method counts 0-homology across the filtration and ignores components with fewer than 15 vertices in its tests.Source: 1) Layer 4: Room Clustering via… · p. 12
  • Room-count candidates come from persistent Betti-curve intervals, with α setting the persistence threshold and the largest qualifying component count selected.The method chooses the largest component count among intervals longer than α times the longest interval; α balances over- and under-segmentation.Source: 1) Layer 4: Room Clustering via… · p. 12
  • Flood-fill assigns remaining places to rooms by expanding edges farthest from obstacles first, provided each original place-graph component contains a room.Source: 1) Layer 4: Room Clustering via… · p. 12
  • The approach clusters sparse places in arbitrary 3D environments without full voxel-map processing, but may miss rooms lacking a clear geometric footprint, such as open floor-plans.It also computes the dilation distance automatically and is described as computationally and memory efficient in practice.Source: 1) Layer 4: Room Clustering via… · pp. 12–13
  • Room labels are inferred from object semantics and room connections using a neural tree on the heterogeneous object-room graph.The neural tree exploits the graph hierarchy for efficient and accurate inference; its message passing treats the different node and subgraph types as heterogeneous.Source: 2) Layer 4: Room Classification via… · pp. 13–14
  • Building classification is not trained in the experiments because the available dataset is too small and mostly residential.The dataset contains 90 buildings.Source: 2) Layer 4: Room Classification via… · p. 13

IV. PERSISTENT REPRESENTATIONS: DETECTING AND ENFORCING LOOP CLOSURES IN 3D SCENE GRAPHS

This section turns the odometric 3D scene graph into a backend graph by detecting loop closures and correcting the map. It covers both joint optimization of graph layers and post-processing to remove redundant subgraphs.

IV. PERSISTENT REPRESENTATIONS: DETECTING AND ENFORCING LOOP CLOSURES IN 3D SCENE GRAPHS

  • The section uses the constructed odometric 3D scene graph as the frontend representation, which drifts as the robot follows its odometric trajectory.Source: §IV · p. 14
  • Loop closures are detected and used to correct the scene graph during backend processing.Source: §IV · p. 14
  • The backend aims to optimize all graph layers consistently and remove redundant subgraphs from revisited locations.Source: §IV · p. 14

A. Loop Closure Detection and Geometric Verification

The paper augments visual loop-closure detection and geometric verification with information from multiple 3D scene-graph layers. Those layers contribute visual features as well as semantic, geometric, and topological information.

A. Loop Closure Detection and Geometric Verification

1) Top-Down Loop Closure Detection:

Loop closures are proposed by comparing hierarchical descriptors from places down to objects and appearance, then verified by attempting registration from lower layers upward. The backend optimizes a sparse deformation graph and interpolates the remaining scene graph before reconciling overlaps.

1) Top-Down Loop Closure Detection:

  • For each agent node, hierarchical descriptors summarize its surroundings from places and objects to appearance; matching proceeds down that hierarchy and triggers geometric verification when a threshold is met.Source: 1) Top-Down Loop Closure Detection · p. 15
  • Hand-crafted descriptors use object-label and place-distance histograms, while learned GNN descriptors encode object and place subgraphs and are intended to capture their spatial structure.The learned descriptors use edge distances rather than absolute positions, making the embedding pose-invariant.Source: 1) Top-Down Loop Closure Detection · p. 15

2) Bottom-up Geometric Verification:

  • Bottom-up verification first registers visual features with RANSAC, then attempts object registration with TEASER++ if visual registration fails.Object registration can recover matches that fail appearance-based verification under viewpoint or illumination changes.Source: 2) Bottom-up Geometric Verification · p. 15

B. 3D Scene Graph Optimization

  • The backend jointly optimizes agent poses, mesh control points, and places in a sparse deformation graph, then reconstructs the remaining graph by interpolation.The place layer is represented by a minimum spanning tree for sparsity, and the optimization minimizes mismatch to relative measurements.Source: B. 3D Scene Graph Optimization · pp. 15–16
  • After optimization, the backend updates agent and place positions, interpolates the full mesh, merges overlapping nodes, and recomputes rooms from the place graph.Place nodes within 0.4 m are merged in the implementation.Source: B. 3D Scene Graph Optimization · pp. 16–17, Figure 9 · p. 16

V. THINKING FAST AND SLOW: THE HYDRA ARCHITECTURE

Hydra organizes spatial perception into fast, mid-level, and slower high-level processes to build a persistent 3D scene graph. The experiments describe its datasets, learning setups, and implementation.

V. THINKING FAST AND SLOW: THE HYDRA ARCHITECTURE

  • Hydra schedules sensor-rate, sub-second, and slower processes so scene graph optimization does not obstruct faster perception.Its modules span feature tracking and odometry, mesh and place reconstruction, object bounding boxes, and global graph optimization.Source: §V · p. 17
  • Hydra runs in real time on a multi-core CPU, with 2D semantic segmentation as its only GPU-dependent module.The neural tree and GNN-based loop closure detection can optionally use a GPU but also run on CPU.Source: §V · p. 17

VI. EXPERIMENTS

  • The experiments compare Hydra with SceneGraphFusion and offline Kimera, test learned-component design choices, and analyze runtime.The paper reports its implementation as open source.Source: §VI · p. 17

A. Datasets

  • Experiments use simulated Matterport3D and uHumans2 data, real SidPac and Simmons recordings, and Stanford3D scene graphs for neural-tree design comparisons.SidPac includes a two-floor traversal of around 400 m; the Simmons Jackal recording covers around 500 m, with similar student-room layouts posing a challenge.Source: A. Datasets · pp. 17–18

B. Experimental Setup

  • For real-world semantic segmentation, the setup compares HRNet, MobileNetV2, and OneFormer, while visual-inertial odometry is fused with external odometry for SidPac and Jackal Simmons.Hydra's remaining blocks are implemented in C++, and timing is also reported on an Nvidia Xavier NX embedded computer.Source: B. Experimental Setup · pp. 18–19
  • GNN loop-closure descriptors are trained on MP3D object and place subgraphs using triplet loss and bounding-box overlap to select positive and negative examples.Subgraphs use a minimum 3 m radius, grow to at most 5 m, and use a 40% overlap threshold; descriptors have dimension 64.Source: B. Experimental Setup · p. 19
  • The neural tree is evaluated for semi-supervised node classification on Stanford3D and room classification on MP3D, with graph features and connectivity varied.Stanford3D comparisons average test accuracy over 100 runs; MP3D room labels are filtered using a 60% footprint-IoU threshold.Source: B. Experimental Setup · pp. 19–20

C. Results and Ablation Study

Hydra produces better object maps than SceneGraphFusion while adding place and room layers. Its accuracy and loop-closure results vary with scene scale, semantic inputs, and model choices.

1) Comparison against SceneGraphFusion:

  • Hydra largely outperforms SceneGraphFusion on Percent Correct and Percent Found, while adding place and room layers that SceneGraphFusion does not estimate.The visual comparison shows SceneGraphFusion over-segmenting large objects and Hydra under-segmenting some nearby objects.Source: 1) Comparison against SceneGraphFusion · p. 20, Figure 13 · p. 21

2) Accuracy Evaluation and Ablation:

  • In larger scenes, scene-graph loop closures outperform vision-based or absent loop closures for object accuracy; semantic-segmentation quality also affects object metrics, especially % Found.In small-to-medium scenes, configuration differences are within standard deviation; the object metrics are coupled, and thresholds vary by scene.Source: 2) Accuracy Evaluation and Ablation · pp. 21–22
  • Place-position errors are similar across Hydra configurations in low-drift uHumans2 scenes, while loop-closure configurations generally improve over VIO and vary on larger scenes.For SidPac, VIO+GNN-LC performs worse than VIO+SG-LC, while Simmons Jackal shows little advantage over VIO+V-LC.Source: 2) Accuracy Evaluation and Ablation · p. 22, Figure 14 · p. 22
  • On Stanford3D, neural trees outperform standard GNNs by 1.63% to 11.72%; relative-position edge features improve neural-tree accuracy by 7.45% on homogeneous and 2.86% on heterogeneous graphs.Heterogeneous graphs degrade neural-tree performance on Stanford3D, but not significantly on MP3D.Source: 2) Accuracy Evaluation and Ablation · p. 23, Table II · p. 23, Table III · p. 23
  • For MP3D room classification, object semantic labels improve both approaches by 14% to 17%, and room connectivity further improves the neural tree; the best model uses both.Hydra’s room-label accuracy is lower for out-of-distribution scenes; on uHumans2, ground-truth semantics yield 27% versus 40% for HRNet and 45% for OneFormer.Source: 2) Accuracy Evaluation and Ablation · pp. 23–24, Table IV · p. 23, Table V · p. 24
  • Hydra generally outperforms Kimera on room clustering, including in multi-floor scenes, and mostly maintains precision and recall across loop-closure configurations despite VIO drift.Precision and recall are computed over free-space voxels; low precision corresponds to under-segmentation and low recall to over-segmentation.Source: 2) Accuracy Evaluation and Ablation · p. 24, Figure 15 · p. 24
  • Scene-graph loop closure yields about twice as many matches within 10 cm translation and 1° rotation error as permissive vision-based detection; SG-GNN outperforms both vision baselines and SG-LC.Across descriptor tests, one-hot encoding exceeds word2vec by 5% on MP3D and 10% on uHumans2; handcrafted descriptors remain competitive, while learned descriptors appear better suited to uniform scenes.Source: 2) Accuracy Evaluation and Ablation · pp. 24–25, Figure 16 · p. 25, Table VI · p. 25
  • In the Unitree A1 experiment, Hydra produced a scene graph with four estimated rooms, but over-segmented the scene and room labels were inconsistent over time.The experiment used a 1 Hz nominal Hydra update rate and dense reconstruction at 5 Hz.Source: 2) Accuracy Evaluation and Ablation · p. 25, 3) Onboard Operation on a Robot · p. 25, Figure 17 · p. 25

3) Onboard Operation on a Robot:

The supplied passages identify a runtime plot comparing incremental Hydra with batch Kimera and a timing-breakdown table. The onboard graph’s room labels were not consistently estimated.

3) Onboard Operation on a Robot:

  • Figure 18 compares scene-graph construction runtime over time for incremental Hydra and batch Kimera on SidPac Floors 1–3.The caption directs readers to [2] for Hydra’s low-level process timing because Hydra also relies on Kimera-VIO.Source: Figure 18 · p. 26
  • Table VII presents a timing breakdown for Hydra’s incremental layer creation across scenes.Source: Table VII · p. 26
  • In the onboard building 31 run, Hydra’s estimated room categories were inconsistent over time, likely reflecting poor 2D semantics and few useful object categories.Source: 3) Onboard Operation on a Robot · p. 26

4) Runtime Evaluation:

Hydra’s incremental construction has a fixed-cost frontend and much lower runtime than batch processing as the scene grows, while its active-window memory use is smaller. Layer timings are usually below the target rate, with occasional object-extraction overruns.

4) Runtime Evaluation:

  • Batch scene-graph construction exceeds five minutes and grows with the ESDF, while Hydra’s mid-level frontend has fixed computation cost and its high-level cost remains much lower.High-level runtime spikes occur when new loop closures trigger scene-graph optimization.Source: 4) Runtime Evaluation · p. 26
  • Hydra uses 74.1 MiB for its active window on SidPac Floors 1–3, versus Kimera’s 422 MiB for the entire environment; Kimera’s memory grows with scene size.Source: 4) Runtime Evaluation · p. 26
  • Although layer means and standard deviations are below the 200 ms keyframe limit, object extraction sometimes exceeds it in larger scenes; Hydra restores real-time operation shortly afterward.Source: 4) Runtime Evaluation · p. 26
  • On the Xavier NX, Hydra processes objects in 83.9 ± 65 ms, places in 114.8 ± 103 ms, and rooms in 34.7 ± 37.6 ms during the onboard experiment.The 1 Hz target leaves Xavier resources available for other Hydra modules and external processes.Source: 4) Runtime Evaluation · p. 26

VII. RELATED WORK

  • The related-work review covers abstractions and symbolic representations, metric-semantic and hierarchical maps, and loop-closure detection and optimization.Source: §VII · p. 26

A. Need for Abstractions and Symbolic Representations

The section reviews hierarchical and symbolic representations, metric-semantic mapping, and persistent-map maintenance before concluding that hierarchical scene graphs support real-time spatial perception. Hydra builds accurate 3D scene graphs in real time across photo-realistic simulations and real datasets.

A. Need for Abstractions and Symbolic Representations

  • Prior work on state and action abstractions, ontologies, compositional models, and scene graphs provides several routes to symbolic or hierarchical representations.Compositional models organize constituents recursively, while stochastic grammars use hierarchical tree structures to model objects, images, and scenes.Source: A. Need for Abstractions and Symbolic… · pp. 26–27

B. Metric-semantic and Hierarchical Representations for Scene Understanding and Mapping

  • Existing 2D scene graphs are viewpoint-dependent and image-grounded, while flat metric-semantic maps generally lack higher-level semantics such as rooms.Prior 3D scene graphs include geometry, objects, rooms, and places, but the cited hierarchical approaches operate offline.Source: B. Metric-semantic and Hierarchical… · pp. 27–28

C. Maintaining Persistent Representations

  • Prior loop-closure methods use visual or semantic cues, but the reviewed approaches are not hierarchical and existing corrections do not jointly address multiple representation layers.Dense-map correction is non-trivial in real time, and some earlier scene-graph approaches would require rebuilding after loop closures.Source: C. Maintaining Persistent… · p. 28

VIII. CONCLUSIONS

  • The paper argues hierarchical representations scale better in memory and support more efficient inference, and combines geometry, topology, and geometric deep learning to construct indoor scene graphs incrementally.Its framework also introduces hierarchical loop-closure descriptors and unified scene-graph optimization; the paper identifies open-floor-plan room segmentation and ground-robot traversability as limitations.Source: §VIII · pp. 28–29
  • Hydra builds accurate 3D scene graphs in real time across photo-realistic simulations and real datasets.The conclusion also identifies limitations in traversability for ground robots, room segmentation in open floor plans, and downward propagation of high-level semantics.Source: §VIII · pp. 28–29
  • The authors identify richer relations and affordances, mixed indoor-outdoor environments, and scene-graph implications for planning and decision-making as future work.They also note that the detectable object vocabulary is limited by pre-trained 2D segmentation networks.Source: §VIII · p. 29

APPENDIX

The appendix defines tree decompositions and treewidth, then uses tree-decomposition constructions to establish bounds for hierarchical graphs and a room-graph lemma.

A. Preliminaries on Tree Decomposition and Treewidth

  • Treewidth measures the size of the largest bag minus one, minimized over a graph’s valid tree decompositions.Finding a minimum-treewidth decomposition is NP-hard, though algorithms can produce decompositions with small treewidth.Source: A. Preliminaries on Tree Decomposition… · p. 34
  • A junction tree is constructed by triangulating the graph, forming maximal-clique bags, and extracting a maximum-weight spanning tree of their intersection graph.The resulting tree with those bags satisfies the connectedness and covering properties.Source: A. Preliminaries on Tree Decomposition… · p. 34

B. Proof of Proposition 3

  • The hierarchical-graph treewidth bound follows by reusing decompositions of the top layer or child graphs and increasing child-graph bag sizes by one.The appendix relates this construction to the bound stated in equation (5).Source: B. Proof of Proposition 3 · p. 34

C. Proof of Lemma 4

  • For the room-graph lemma, removing degree-one rooms leaves a graph of treewidth at most two under the stated degree assumptions, and leaf rooms can be reattached with size-two bags.The proof uses the bound on the reduced graph and appends bags representing each removed degree-one room and its connection.Source: C. Proof of Lemma 4 · p. 34

D. Object and Room Labels

The appendix specifies object and room label spaces and describes graph preprocessing, model tuning, and runtimes for room-classification experiments on Stanford3d and MP3D.

D. Object and Room Labels

  • Object and room labels are drawn from mpcat40 and MP3D, with mappings and grouped categories tailored to the datasets and indoor focus.Outdoor MP3D room labels are grouped with unknown and excluded from loss and accuracy; rooms not meeting inference criteria are also labeled unknown.Source: D. Object and Room Labels · pp. 34–35

E. Graph Neural Network Training

  • Neural-tree training took 1.5–2 times as long as standard GNNs on homogeneous graphs and around 3.5 times as long on heterogeneous graphs.The authors attribute the longer runtimes to more message-passing iterations and larger H-trees, with more node and edge types in heterogeneous inputs.Source: E. Graph Neural Network Training · p. 35
  • For Stanford3d, tuned hyperparameters were consistent with, and changes within the search space improved accuracy by no more than 0.5 standard deviations.A dropout ratio of 0.25 was optimal in all reported cases.Source: E. Graph Neural Network Training · p. 36
  • For MP3D, three message-passing iterations, hidden dimension 64 for the first two iterations, and L2 regularization 0.001 worked best across setups.The experiments tuned separate hyperparameters with and without word2vec object-semantic features and reused them without room edges.Source: E. Graph Neural Network Training · p. 36

F. Additional Experiments on Semi-supervised Node Classification

The additional experiments find that neural trees retain an accuracy advantage over standard GNNs despite implementation changes, while scene-graph loop closures span more varied time and viewpoint differences than vision-only detections. SG-GNN also detects some loop closures over a longer horizon than SG-LC.

F. Additional Experiments on Semi-supervised Node Classification

  • Neural trees maintain a clear accuracy advantage over standard GNNs across message-passing architectures, PyTorch Geometric versions, and H-tree decomposition methods.The reported version change reduced neural-tree accuracy by about 4% with GraphSAGE and GIN and 8.3% with GAT; decomposition changes varied by architecture.Source: F. Additional Experiments on… · pp. 36–37

G. Loop Closure Ablation Parameters

  • SG-GNN’s model encodes node and edge inputs with MLPs, applies message passing, then aggregates and transforms the resulting node embeddings.The appendix denotes an MLP as M(i, h1, h2, . . . , hn, o), with intermediate layers connected by ReLU.Source: G. Loop Closure Ablation Parameters · p. 37

H. Additional Loop Closure Experiments

  • Scene-graph loop closures have wider time-difference and rotation-angle distributions than vision-based configurations, suggesting improved viewpoint invariance.The figure plots time differences in seconds and rotation angles in degrees; vision-based detections cluster around revisits with similar viewpoints.Source: H. Additional Loop Closure Experiments · pp. 37–38, Figure 19 · p. 37
  • SG-LC and SG-GNN outperform vision-only loop closures in Hydra’s overall system performance, although some registrations add no information to optimization.When registered subgraphs overlap, TEASER may return an identity pose that agrees with the current relative-pose estimate, while still constraining local scene deformation.Source: H. Additional Loop Closure Experiments · p. 38
  • SG-GNN detects loop closures over a slightly longer horizon than SG-LC, especially in the 150–250 s range.Many detections from both configurations come from overlapping scene-graph regions.Source: H. Additional Loop Closure Experiments · p. 38
Loading 2305.07154v1…