Source-linked AI summary
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization
Nathan Hughes, Yun Chang, Luca Carlone
TL;DR
Hydra addresses the challenge of building rich hierarchical 3D scene graphs in real time for persistent robotic autonomy. It combines incremental scene-graph construction, hierarchical loop-closure detection and optimization, and a highly parallelized architecture; evaluations show real-time reconstruction with accuracy comparable to batch offline methods.
Problem
Building rich hierarchical 3D scene graphs from sensor data in real time remains largely unresolved, despite their value for high-level robot representations and persistent autonomy.
Method
Hydra incrementally builds scene-graph layers from local ESDFs, extracts GVD-based place graphs, segments rooms, and uses hierarchical loop closures with embedded deformation graphs to correct all layers.
Results
Hydra reconstructs large real environments in real time with accuracy comparable to batch offline methods and detects more and higher-quality loop closures than standard visual approaches.
Takeaways & Limitations
The system provides a persistent hierarchical scene-graph representation that can be corrected as new evidence arrives during long-term robotic operation.
Abstract
from arXiv · showhide
3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory. This paper describes a real-time Spatial Perception System, a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed Spatial Perception System into a architecture named Hydra, that combines fast early and mid-level perception processes with slower high-level perception. We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods despite running online.
I. INTRODUCTION
Hydra addresses the lack of real-time methods for constructing rich hierarchical 3D scene graphs by combining incremental perception, loop-closure optimization, and a parallel architecture. Experiments report real-time reconstruction with accuracy comparable to batch methods.
- 3D scene graphs represent environments hierarchically, from geometry to objects, places, rooms, and buildings, with relations across concepts.
- Existing real-time methods largely build flat object-focused graphs, while hierarchical approaches operate offline and may require minutes.
- Hydra incrementally reconstructs scene-graph layers from sensor data using local ESDFs, metric-semantic meshes, GVD-derived places, and millisecond-scale room segmentation.
- Hydra detects loop closures with hierarchical descriptors and geometrically verifies matches, then uses embedded deformation graphs to correct mesh, places, objects, and rooms.
- Hydra combines fast early and mid-level perception with slower high-level optimization in a highly parallelized real-time architecture.
- Experiments in heterogeneous simulated and real environments show real-time reconstruction, accuracy comparable to batch offline methods, and stronger loop-closure detection than standard visual approaches.
II. RELATED WORK
Related work spans metric-semantic mapping, hierarchical map models, building-layout parsing, and loop-closure detection and optimization. Hydra targets the unresolved combination of hierarchical representation, real-time operation, and online correction.
- Metric-Semantic and Hierarchical Mapping: Metric-semantic mapping includes object-based, volumetric, point-cloud, mesh, and combined representations, but these approaches are not concerned with hierarchical scene graphs.
- Metric-Semantic and Hierarchical Mapping: Hierarchical mapping research connects metric and topological representations, with recent 3D scene graphs modeling geometry, objects, rooms, and camera locations.
- Building Layout Parsing: Building-layout methods range from rule-based and learning-based 2D parsing to Voronoi graphs, multi-view stereo, dense stereo, and 3D region growing.
- Loop Closures Detection and Optimization: Loop-closure research uses visual retrieval, image sequences, deep learning, and semantic object- or room-based descriptors, often addressing appearance and viewpoint changes.
- Loop Closures Detection and Optimization: Correcting dense structured maps after loop closure remains non-trivial, motivating surfel optimization and volumetric reintegration alternatives.
III. REAL-TIME INCREMENTAL
Hydra incrementally constructs indoor scene-graph layers around the robot using a spatially bounded volumetric model, mesh and object processing, and GVD-based place extraction. The resulting place graph supports room-level organization.
- Incremental Construction: Hydra constructs Layers 1–3 incrementally and segments places into rooms while processing an odometric robot trajectory.
- Scene Graph Model: The adopted indoor scene graph contains mesh, objects and agents, places, rooms, and a building node, with within-layer and cross-layer relations.
- Layers 1-3: Mesh, Objects, and Places: The mesh layer integrates semantically labeled point clouds into TSDF and ESDF representations, extending Kimera with real-time modifications.
- Layers 1-3: Mesh, Objects, and Places: An 8m active window bounds ESDF memory, while extracted mesh and places are transferred to the scene-graph frontend as they leave the window.
- Layers 1-3: Mesh, Objects, and Places: Objects are formed by semantic-class-specific Euclidean clustering, then represented with centroids and bounding boxes and merged with overlapping existing objects.
- Layers 1-3: Mesh, Objects, and Places: A Generalized Voronoi Diagram is built during ESDF integration and incrementally sparsified into a graph whose nodes are obstacle-free places connected by traversability edges.
B. Layer 4: Room Detection
Hydra detects rooms directly from a sparse place subgraph by combining obstacle dilation, connected-component analysis, and seeded community detection. The topology-based method is scalable and adaptable but cannot distinguish semantically different rooms in open floor plans.
- Rooms are segmented directly from the sparse place subgraph Gp rather than from a full volumetric representation.
- Increasing dilation distances closes small apertures, while pruning places below each distance exposes candidate room components.
- The method selects the largest pruned graph with the median component count, then assigns remaining nodes using seeded greedy modularity-based community detection.
- Topology-only reasoning reduces clutter sensitivity and supports varied environments, but fails to segment semantically distinct rooms in open floor plans.
- The implementation avoids explicitly constructing dilated subgraphs by excluding below-threshold nodes and edges during breadth-first search.
IV. PERSISTENT REPRESENTATIONS: LOOP CLOSURE DETECTION AND 3D SCENE GRAPH OPTIMIZATION
Hydra performs hierarchical loop closure detection by descending scene-graph descriptors from places toward appearance, then verifies matches geometrically by ascending through available layers. The approach augments visual matching with object and place information.
- Top-down Loop Closure Detection: Loop closure detection searches for a past agent node that observed the same scene portion as the current node.
- Top-down Loop Closure Detection: Each agent node stores descriptors spanning low-level appearance, object semantics, and place geometry.
- Top-down Loop Closure Detection: Descriptor comparison proceeds from places to objects to appearance, advancing only when the preceding distance satisfies a threshold.
- Bottom-up Geometric Verification: Putative matches undergo bottom-up geometric verification, registering frames at whichever scene-graph layer produced the match.
B. 3D Scene Graph Optimization
Hydra corrects a scene graph after loop closure by optimizing a connected deformation-graph subgraph, interpolating the remaining geometry, and merging overlapping nodes. The design preserves sparsity through a minimum spanning tree of places.
- The backend optimizes a deformation graph and removes redundant subgraphs created by repeated visits to the same location.
- The frontend assembles windowed mesh, places, objects, and agent poses, adds inter-layer edges, and creates mesh control points by octree clustering.
- The deformation graph contains agent poses, mesh control points, and the places-layer minimum spanning tree, forming a connected subgraph.
- Embedded deformation optimization adjusts local node frames to minimize edge deformations, including loop-closure constraints.
- After optimization, Hydra interpolates the full mesh, recomputes object geometry, and merges nearby place or object nodes.
V. THINKING FAST AND SLOW: THE HYDRA ARCHITECTURE
Hydra organizes perception into parallel processes operating at sensor, sub-second, and slower rates. This architecture combines fast perception and incremental scene-graph construction with slower global optimization while maintaining real-time operation.
- Hydra combines sensor-rate, sub-second, and slower perception processes whose workloads depend on map size.
- The architecture separates early perception, mid-level incremental layer construction, and high-level loop closure, optimization, and room detection.
- Hydra runs in real-time on a multi-core CPU, using a GPU only for 2D semantic segmentation.
- Hydra builds 3D scene graphs in real-time with accuracy comparable to batch offline methods.
A. Experimental Setup
Hydra is evaluated on simulated and real indoor datasets using heterogeneous sensing and perception configurations. Its architecture separates fast low- and mid-level processing from slower high-level operations.
- Datasets: Experiments use uHumans2 simulated scenes and SidPac real recordings collected with visual-inertial and depth-sensing devices.uHumans2 includes an apartment, office, and subway station; SidPac covers multiple floors of graduate housing.
- System Architecture: Room detection operates over the entire graph, making it more suitable as a slow high-level perception process.
- System Architecture: Hydra organizes low-level, mid-level, and high-level perception into functional groups with increasing latency and separate logical threads.The functional-block organization is illustrated in Fig. 5.
- Datasets: The SidPac recordings cover multi-floor residential environments including common, music, recreation, lounge, kitchen, and apartment areas.
- Perception Configuration: Hydra uses Kinect depth and HRNet semantic segmentation on real data, provided depth and segmentation on simulated data, and Kimera-VIO for odometry.
B. Results and Ablation Study
Hydra is evaluated for scene-graph accuracy, room detection, runtime, and loop-closure quality against batch and vision-based baselines. Results show comparable or stronger performance while retaining incremental and real-time operation.
- Objects and Places: Hydra’s real-time scene graphs achieve 80–100% found and correct objects and sub-25cm place position error with ground-truth trajectories.These results are reported relative to the batch ground-truth scene graph.
- Objects and Places: In larger scenes, VIO+SG-LC substantially outperforms VIO+V-LC in object accuracy and typically reduces result variance.Place-position errors remain similar between the two loop-closure configurations.
- Rooms: Hydra’s room segmentation is especially stronger than Kimera in multi-floor environments, including SidPac Floors 3–4, where Kimera reaches 0.88 precision but 0.06 recall.Hydra generally maintains consistent precision and recall despite slight over-segmentation.
- Runtime: The batch runtime grows over time and exceeds 40 seconds for moderate scenes, whereas Hydra’s mid-level frontend has fixed computation cost and its high-level runtime remains much lower.Hydra’s high-level runtime shows a slight upward trend from room detection and scene-graph optimization.
- Runtime: On an Nvidia Xavier NX, Hydra processes objects in 75 ± 35 ms, places in 33 ± 6 ms, and rooms in 55 ± 41 ms at a 5 Hz target rate.These measurements support real-time operation on embedded robotics hardware.
- Loop Closure Ablation: SG-LC produces approximately twice as many loop closures within 10cm and 1 degree of error as permissive vision-based detection.The comparison includes nominal and permissive vision-based configurations.
VII. CONCLUSIONS
Hydra combines online algorithms and parallelized perception to build persistent 3D scene graphs from sensor data in real time. The conclusion identifies richer labeling, relations, optimization efficiency, and downstream use as open directions.
- Contributions: Hydra builds 3D scene graphs from sensor data in real time through novel online algorithms and a highly parallelized perception architecture.
- Contributions: Hydra supports persistent environmental representations through an approach for 3D scene-graph optimization.
- Future Work: Some reconstructed nodes remain unlabeled, including rooms that are detected but not classified as kitchens or bedrooms.The paper proposes connecting Hydra with learning-based methods for node labeling.
- Future Work: Future work includes richer node and edge relations, more efficient optimization through pose-graph sparsification, and studying scene graphs for prediction, planning, and decision-making.
DISCLAIMER
The paper reports Air Force sponsorship and states that its views and conclusions do not represent official United States Air Force or government policies.
- Disclaimer: The research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator.
- Disclaimer: The authors state that the paper’s views and conclusions should not be interpreted as official United States Air Force or U.S. Government policy.
APPENDIX
The appendix documents parameters for visual and scene-graph loop-closure ablations, including descriptor radius and NSS usage for SG-LC.
- Visual loop closure: Table II lists key parameters for the “V-LC (Permissive)” and “V-LC (Nominal)” visual loop-closure ablations.Parameter meanings are referenced through the open-source Kimera implementation.
- Scene-graph loop closure: Table III lists key parameters for the “SG-LC” scene-graph loop-closure ablation.The scene-graph loop-closure configuration is separately documented from the visual loop-closure settings.
- Scene-graph loop closure: SG-LC computes all scene-graph descriptors with a 13-meter radius and does not use NSS to filter matches.NSS denotes Normalized Similarity Scoring.