Source-linked AI summary
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans
Antoni Rosinol, Arjun Gupta, Marcus Abate, Jingnan Shi, Luca Carlone
TL;DR
The paper addresses the lack of a unified representation combining metric-semantic understanding, hierarchical abstractions, and dynamic entities. It introduces DSGs and the fully automatic SPIN pipeline, demonstrating robust and expressive operation in simulation and reporting 99.89% average precision and 99.84% average recall for place-to-room classification on uH_01.
Problem
Existing spatial representations do not simultaneously provide metric-semantic understanding, actionable hierarchical abstractions, and modeling of dynamic entities needed for high-level task execution.
Method
The paper introduces layered 3D Dynamic Scene Graphs and a fully automatic SPIN pipeline that builds them from visual-inertial data while integrating object, human, and indoor-environment parsing.
Results
99.89% average precision and 99.84% average recall were obtained for classifying places into rooms on uH_01.
Takeaways & Limitations
DSGs support actionable queries for planning, collision checking, human-robot interaction, data compression, and scene prediction.
Abstract
from arXiv · showhide
We present a unified representation for actionable spatial perception: 3D Dynamic Scene Graphs. Scene graphs are directed graphs where nodes represent entities in the scene (e.g. objects, walls, rooms), and edges represent relations (e.g. inclusion, adjacency) among nodes. Dynamic scene graphs (DSGs) extend this notion to represent dynamic scenes with moving agents (e.g. humans, robots), and to include actionable information that supports planning and decision-making (e.g. spatio-temporal relations, topology at different levels of abstraction). Our second contribution is to provide the first fully automatic Spatial PerceptIon eNgine(SPIN) to build a DSG from visual-inertial data. We integrate state-of-the-art techniques for object and human detection and pose estimation, and we describe how to robustly infer object, robot, and human nodes in crowded scenes. To the best of our knowledge, this is the first paper that reconciles visual-inertial SLAM and dense human mesh tracking. Moreover, we provide algorithms to obtain hierarchical representations of indoor environments (e.g. places, structures, rooms) and their relations. Our third contribution is to demonstrate the proposed spatial perception engine in a photo-realistic Unity-based simulator, where we assess its robustness and expressiveness. Finally, we discuss the implications of our proposal on modern robotics applications. 3D Dynamic Scene Graphs can have a profound impact on planning and decision-making, human-robot interaction, long-term autonomy, and scene prediction. A video abstract is available at https://youtu.be/SWbofjhyPzI
I. INTRODUCTION
The paper identifies a gap between metric-semantic understanding, hierarchical actionable representations, and dynamic-entity modeling, then proposes 3D Dynamic Scene Graphs and an automatic SPIN pipeline to address it.
- Existing approaches do not simultaneously provide metric-semantic understanding, actionable hierarchical abstractions, and modeling of dynamic entities.
- 3D Dynamic Scene Graphs unify spatial concepts, hierarchical abstraction, places and connectivity, and spatio-temporal relations for actionable perception.
- SPIN builds a DSG fully automatically from visual-inertial data without human supervision, integrating object and human perception with indoor place, structure, and room partitioning.
- The paper demonstrates SPIN in a photo-realistic Unity simulator and assesses robustness, expressiveness, and handling of known- and unknown-shape objects.
- The authors discuss DSG queries for hierarchical planning, collision checking, human-robot interaction, long-term autonomy, and scene prediction.
II. RELATED WORK
Related work spans scene graphs, hierarchical robotic representations, metric-semantic reconstruction, topological parsing, dynamic-environment SLAM, and human pose estimation, but prior systems generally cover only subsets of the target capabilities.
- Scene Graphs: Computer-vision scene graphs primarily abstract objects and relations in 2D images for retrieval, captioning, understanding, question answering, and action detection.
- Scene Graphs: Prior 3D scene-graph work models static scenes semi-automatically, while another robotics model includes objects but omits multiple abstraction levels.
- Representations and Abstractions in Robotics: Earlier robotic map representations introduced hierarchical spatial and semantic abstractions but focused largely on 2D occupancy maps and pre-deep-learning semantic capabilities.
- Metric-Semantic Scene Reconstruction: Metric-semantic reconstruction estimates semantic maps from sensor data, increasingly in real time, but is typically non-hierarchical.
- Metric-to-Topological Scene Parsing: Metric-to-topological parsing partitions metric maps into semantically meaningful places such as rooms and hallways using geometric, probabilistic, and object-based methods.
- SLAM and VIO in Dynamic Environments: Dynamic-environment research addresses moving targets and robustness, but this paper claims the first integration of visual-inertial SLAM, dense object models, known-object poses, and dense human SMPL meshes.
III. 3D DYNAMIC SCENE GRAPHS
A DSG is a spatially grounded, layered directed graph that organizes geometry, semantics, entities, and relations across abstraction levels, including places, rooms, and buildings.
- A DSG represents 3D geometry and semantics at multiple abstraction levels while modeling objects, places, structures, agents, and their relations.
- Its nodes are spatially grounded with coordinates and shape or bounding-box attributes, and its layers correspond to different levels of scene abstraction.
- The single-story indoor DSG contains five layers: metric-semantic mesh; objects and agents; places and structures; rooms; and building.
- Layer 1: Metric-Semantic Mesh: The metric-semantic mesh stores static 3D points with position, normal, color, and panoptic semantic labels, while mesh faces encode environmental topology.
B. Layer 2: Objects and Agents
Layer 2 separates static objects from dynamic agents and represents both with semantic, geometric, and relational information. This layer also connects to places and structures that organize the environment.
- Objects are static, non-structural entities represented by nodes with 3D poses, bounding boxes, and semantic classes.
- Object edges encode relations including co-visibility, relative size, distance, and contact.
- Agents are dynamic entities, focused here on humans and robots, with time-varying pose graphs, mesh models, and semantic classes.
- Places model free-space positions and traversability, forming a topological map for path planning.
- Structures represent elements such as walls, floors, ceilings, and pillars, with geometric attributes and relations to rooms and objects.
D. Layer 4: Rooms
Layer 4 represents rooms, corridors, and halls as an indoor spatial level. Room connectivity captures adjacency, containment, and the building-wide organization of rooms.
- Room nodes encode 3D pose, bounding box, and semantic class, including categories such as kitchen, dining room, and corridor.
- Edges connect adjacent rooms and link each room to the places it contains.
- A single building node represents the building and connects to all rooms within it.
F. Composition and Queries
The DSG is task-dependent and designed to support planning queries through semantic, geometric, and relational information. Its layered structure is compositional and can be extended across additional spatial abstractions.
- Composition and Queries: The choice of DSG nodes and edges is task-dependent and motivated by intended planning queries.
- Composition and Queries: Semantic attributes support high-level specifications, while geometric attributes and edges support motion planning.
- Composition and Queries: Places provide a topological graph for path planning, while bounding boxes support motion-planning operations.
- Composition and Queries: DSGs can add layers below, above, or between existing layers, such as levels between buildings and rooms or neighborhoods above buildings.
- Composition and Queries: A SPIN pipeline populates DSG nodes and edges from stereo-camera and IMU streams, producing a 3D DSG.
- Composition and Queries: The metric-semantic mesh and agent nodes are built incrementally in real time, while objects, places, structures, and rooms are built automatically at run end.
A. From Visual-Inertial data to Mesh and Agents
SPIN builds mesh, robot, human, and object representations from visual-inertial data. It combines semantic reconstruction, robust tracking, dynamic masking, clustering, and CAD-model registration.
- From Visual-Inertial data to Mesh and Agents: Kimera reconstructs a semantically annotated 3D mesh from visual-inertial data using odometry, pose-graph optimization, meshing, and semantic labeling modules.
- From Visual-Inertial data to Mesh and Agents: Robot tracking uses an IMU-aware optical-flow method and 2-point RANSAC for operation in crowded environments.
- From Visual-Inertial data to Mesh and Agents: Human nodes represent dense, time-varying meshes estimated from images and tracked over time with pose graphs.
- From Visual-Inertial data to Mesh and Agents: Human tracking rejects small or boundary-near detections and accepts measurements only when joint motion is physically plausible.
- From Visual-Inertial data to Mesh and Agents: Dynamic masking prevents human pixels from contributing occupied geometry during depth ray casting.
- From Visual-Inertial data to Mesh and Agents: Unknown-shape objects are segmented by semantic class and Euclidean clustering, then represented using centroids, canonical orientations, and bounding boxes.
- From Visual-Inertial data to Mesh and Agents: Known-shape objects are aligned to CAD models by keypoint matching and TEASER++ registration under extreme outliers.
C. From Mesh to Places, Structures, and Rooms
SPIN parses Kimera’s 3D mesh into hierarchical places, structures, and rooms, using topology, semantic labels, and a gravity-aware ESDF procedure.
- Places: SPIN extracts places and their topology directly from an ESDF-derived topological graph, then associates object and agent poses with their nearest place.Graph nodes sparsely sample free space, while edges represent straight-line traversability.
- Structures: Structural layers are obtained from semantic-mesh labels for walls, ground floor, and ceiling, with each type represented by a centroid, canonical orientation, and axis-aligned bounding box.
- Rooms: Rooms are segmented by horizontally slicing the 3D ESDF 0.3m below the detected ceiling, where clutter has limited effect on the room-layout signature.The section is truncated to distances above 0.2m to remove small openings caused by error accumulation.
- Rooms: The resulting disconnected 2D ESDF regions define rooms, which label interior places; unlabeled places are assigned rooms by neighborhood majority voting.
V. EXPERIMENTS IN PHOTO-REALISTIC SIMULATOR
Experiments in a photo-realistic Unity simulator evaluate SPIN in a crowded 65m×65m office with datasets containing 12, 24, and 60 humans. The results report robust VIO and mesh reconstruction, including benefits from dynamic masking.
- Testing Setup: SPIN is evaluated in a 65m×65m simulated office using three uHumans visual-inertial datasets containing 12, 24, and 60 humans.Ground-truth human and object poses are used only for benchmarking.
- Enhanced VIO: IMU-aware feature tracking and 2-point RANSAC enhance VIO robustness, with the proposed approach dominating Kimera-VIO on datasets containing dynamic entities.On static EuRoC data, the approach performs on-par with the state of the art, while 2-point RANSAC already boosts performance.
- Dynamic Masking: Dynamic masking removes artifacts caused by moving humans and produces clean mesh reconstructions compared with reconstruction without masking.The comparison reports RMSE mesh error for dynamic masking and no masking under both ground-truth and VIO poses.
- Dynamic Masking: Ground-truth-pose results show that dynamic entities hinder mesh accuracy even with perfect localization, while dynamic masking preserves its advantage when VIO poses are used.
B. Parsing Humans and Objects
SPIN evaluates human tracking and object localization on uHumans, finding that pose-graph tracking improves human localization and CAD models further improve object accuracy.
- Human Nodes: Human localization is evaluated using average torso-position error against ground truth across the uHumans datasets.The comparison includes single-image detections, filtered detections, and the proposed tracking model.
- Human Nodes: The proposed pose-graph model provides accurate human tracking despite occlusions that cause incorrect or significantly erroneous single-image estimates.
- Object Nodes: SPIN correctly localizes objects of both known and unknown shape, while CAD models further boost accuracy for known-shape objects.Object error is measured as the distance between estimated and ground-truth centroids; known-shape objects use fitted CAD-model centroids.
C. Parsing Places and Rooms
The DSG supports actionable queries across planning, human-robot interaction, long-term autonomy, and scene prediction. Its hierarchical structure connects spatial abstractions to practical navigation, collision checking, information retention, and dynamic-scene reasoning.
- Obstacle Avoidance and Planning: Hierarchical bounding boxes form a Bounding Volume Hierarchy that can speed collision checking for obstacle avoidance and motion planning.Higher-layer boxes contain lower-layer boxes, such as a room containing its objects.
- Obstacle Avoidance and Planning: The connected subgraph of places and objects lets a robot identify the closest place needed for an object-search task and plan a feasible path there.
- Obstacle Avoidance and Planning: Multiple abstraction levels support hierarchical and multi-resolution planning that can save computational resources.
- Human-Robot Interaction: DSGs enable time-aware human-robot interaction queries involving human trajectories, dense poses, locations, and object-picking events.Examples include asking where a person was at time t or which object they picked in a room.
- Long-term Autonomy: Pruning higher-level branches lets robots forget infrequently observed environment portions while selectively retaining objects or discarding cumbersome mesh models.Shared CAD references for repeated known objects can further compress stored information.
- Prediction: A dense metric-semantic mesh combined with agent descriptions supports short-term scene-dynamics prediction and queries about possible future outcomes.The mesh can be supplied to a physics simulator to roll out potential high-level human actions.
VII. CONCLUSION
The paper introduces 3D Dynamic Scene Graphs and a fully automatic SPIN that builds them from sensor data. It demonstrates the system in simulation, discusses applications, and identifies substantial directions for extending the representation and perception engine.
- The paper introduces DSGs as a unified representation for actionable spatial perception and SPIN as the first fully automatic system building DSGs from sensor data.
- SPIN is showcased in a photo-realistic simulator, while DSG applications are discussed for planning, human-robot interaction, data compression, and scene prediction.
- The authors identify future work on richer node attributes, new node types for environments such as outdoors, and alternative, incremental, real-time, or distributed SPIN designs.