Source-linked AI summary
SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, Niko Suenderhauf
TL;DR
Robotics LLMs struggle to ground long-horizon plans in large, multi-room and multi-floor environments while staying within token limits and avoiding infeasible actions. SayPlan uses hierarchical 3D scene graphs for semantic subgraph search, a classical path planner for navigation, and simulator-guided iterative replanning. Across 90 tasks and two expansive environments, it reduced scene representations by up to 82.1% and achieved near-perfect executability rates, while remaining limited by LLM reasoning weaknesses and static pre-built maps.
Problem
Grounding LLM task plans in large multi-room and multi-floor environments requires representing expansive scenes within token limits and mitigating hallucinated or erroneous long-horizon actions.
Method
SayPlan uses hierarchical 3D scene graphs for semantic search of task-relevant subgraphs, a classical path planner for navigation, and simulator feedback for iterative replanning.
Results
82.1% reduction in large-scale scene representations for LLM parsing and near-perfect executability rates were reported across 90 tasks in two expansive environments.
Takeaways & Limitations
SayPlan grounds long-horizon natural-language plans for mobile-manipulator execution across environments spanning multiple rooms and floors.
Takeaways & Limitations
SayPlan requires a pre-built 3D scene graph, assumes objects remain static after mapping, and inherits LLM failures in distance-, count-, and negation-based graph reasoning.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics. We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a 'semantic search' for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an 'iterative replanning' pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and natural language instruction for a mobile manipulator robot to execute. We provide real robot video demonstrations on our project page https://sayplan.github.io.
1 Introduction
SayPlan addresses the difficulty of grounding LLM plans in expansive, multi-room and multi-floor environments by combining hierarchical 3D scene graphs with semantic search, classical navigation, and iterative replanning. It evaluates this approach across large environments and reports reduced representations and near-perfect executability.
- Grounding robot plans requires respecting physical affordances, predicates, action effects, locations, and environmental topology.
- Pre-encoding all information from expansive scenes into an LLM context becomes increasingly infeasible.
- SayPlan uses 3D scene graphs to represent large environments with hierarchical topology, object states, predicates, affordances, and attributes in natural language.
- Its innovations are semantic subgraph search, classical path planning for navigation, and iterative replanning with simulator feedback to correct unexecutable actions.
- 82.1% reduction in large-scale scene representations for LLM parsing was achieved, alongside near-perfect executability rates across 90 tasks.
2 Related Work
Prior robotics planning methods struggle to combine task generality with scalability, while LLM-based approaches remain difficult to ground beyond single-room settings. SayPlan uses hierarchical 3D scene graphs to address multi-room and multi-floor planning.
- Conventional symbolic and learning-based planning methods face limitations in task generality, data demands, or scalability to large environments.
- Prior LLM-based robotics planners use detectors, PDDL representations, or value functions, but predominantly remain constrained to single-room environments.
- SayPlan explores hierarchical and semantic 3D scene graphs to generate grounded plans over large-scale multi-room and multi-floor scenes.
3 SayPlan
SayPlan represents environments as hierarchical 3D scene graphs and uses an LLM to search for task-relevant subgraphs before planning. A path planner supplies navigation, while simulator feedback supports iterative correction toward executable plans.
- Problem Formulation: The problem is to plan long-range navigation and manipulation from abstract or ambiguous natural-language instructions in multi-room, multi-floor environments.
- Scene Representation: A 3D scene graph hierarchically abstracts spatial semantics and object relationships while representing entity states, affordances, and predicates.
- Scene Representation: SayPlan operates on a pre-constructed, JSON-serialized 3D scene graph that can be parsed by a pretrained LLM.
- Scene Graph Simulator: The simulator collapses, expands, contracts, and verifies graph states, with verification forward-simulating actions against predicates, states, and affordances.
- Semantic Search: Semantic search identifies a smaller task-specific subgraph from the full graph by exploiting hierarchical structure and LLM reasoning.
- Iterative Replanning: The path planner connects high-level locations, while simulator feedback iteratively corrects plans that violate environmental constraints.
4 Experimental Setup
The experiments evaluate SayPlan’s 3D scene-graph reasoning and high-level planning for a mobile manipulator across multi-room and multi-floor environments. They separately test semantic search and causal planning against human and ablated baselines.
- Experimental Environments: Experiments use a mobile manipulator with a 7-degree-of-freedom arm, two-fingered gripper, and mobile base in two large-scale environments.
- Semantic Search: Semantic search tasks test whether an LLM can use collapsed scene-graph room and floor information to locate relevant assets and objects.
- Causal Planning: Causal planning is evaluated through correctness and executability, measuring goal alignment and compliance with scene-graph constraints for mobile-manipulator execution.
- Baselines: LLM-As-Planner generates complete navigation and manipulation sequences, whereas LLM+P adds only a path planner and omits iterative replanning.
5 Results
SayPlan combines semantic search over collapsed 3D scene graphs with path planning and iterative replanning to ground task plans in large environments. GPT-4 found relevant subgraphs, while replanning improved long-horizon executability and the hierarchical representation reduced input-token demands.
- Semantic Search: 86.7% and 73.3% success were achieved by SayPlan (GPT-4) on simple and complex semantic-search tasks, respectively.
- Semantic Search: SayPlan (GPT-4) explored node sequences remarkably similar to a human baseline, using meaningful names and attributes to guide searches.
- Scalability Analysis: Node contraction maintained near-constant token representation during semantic search over the 3D scene graph.
- Semantic Search: Negation, distance-based reasoning, and count-based reasoning remained failure cases for the LLM during semantic search.
- Scalability Analysis: 82.1% and 60.4% reductions in initial input tokens were measured for the Office and Home environments, respectively.
- Causal Planning: 93% correctness was achieved by all three methods on simple planning tasks, while path planning and iterative replanning improved correctness on long-horizon tasks.
- Causal Planning: Near-perfect executability was achieved by SayPlan through iterative replanning, although 6.67% of tasks retained uncorrected hallucinated-node errors.
6 Limitations
SayPlan remains constrained by limitations of current LLMs and by assumptions about its scene-graph representation and environment dynamics.
- SayPlan is limited by LLM biases, inaccuracies, and weak graph-based reasoning for distance, node-count, and node-negation tasks.
- The framework requires a pre-built 3D scene graph and assumes objects remain static after map generation, restricting adaptability to dynamic environments.
7 Conclusion
SayPlan combines hierarchical 3D scene graphs, LLM planning, and simulator-based iterative replanning to generate feasible plans for large-scale environments. The framework is demonstrated with a real mobile manipulator across multiple rooms, assets, and objects.
- SayPlan integrates hierarchical 3D scene graphs and LLMs to plan across large-scale, multi-floor, multi-room environments.
- The framework reduces initial scene-representation tokens through high-level exploration, then uses iterative replanning to make generated plans feasible and grounded.
- SayPlan surpasses existing techniques in producing correct, executable plans that a robot can follow.
- Validated plans were translated to a real-world mobile manipulator operating across multiple rooms, assets, and objects in a large office environment.
A Implementation Details
The implementation uses GPT-4 with a static, task- and environment-agnostic prompt, while updating scene, memory, or simulator-feedback components during execution.
- GPT-4 serves as the underlying LLM agent unless otherwise stated.
- The static prompt defines the agent role, scene-graph details, output structure, and in-context examples, requiring approximately 3900 input tokens.
- During semantic search, the 3D Scene Graph and Memory components update at each step.
- During iterative replanning, only the Feedback component updates with information from the scene graph simulator.
- The LLM outputs a JSON object containing arguments for calling the provided API functions.
B Environments
SayPlan is evaluated in two multi-room, multi-floor environments represented with detailed 3D scene graphs. Real-world execution additionally requires aligning scene-graph nodes with a 2D metric map and locating visible objects.
- The evaluation uses two large-scale environments spanning multiple rooms and floors.
- Office: The Office environment spans 37 rooms and contains 151 interactive assets and objects on a real-world office floor.
- Home: The Home environment spans 28 rooms across 3 floors and contains 112 interactive assets and objects.
- Real-world execution: Real-world execution requires a corresponding 2D metric map to align the posed nodes captured by the 3D scene graph.
- Real-world execution: The mobile manipulator combines a Franka Panda 7-DoF arm with an LD-60 Omron mobile base, LiDAR, RGB-D vision, and pick, place, open, and close skills.
C Tasks
SayPlan is tested on 90 instructions spanning semantic search and increasingly complex planning tasks across large-scale scene graphs. The scene-graph representation is evaluated in both fully expanded and contracted forms to support scalable reasoning.
- The evaluation contains 90 instructions grouped to test different planning capabilities across large-scale scene graphs.
- Four instruction sets evaluate simple search, complex search, simple planning, and long-horizon planning capabilities.
- Search tasks: Simple search queries reference scene-graph information directly, whereas complex search queries require the LLM to infer information not readily available in the graph.
- Scene-graph representations: The fully expanded graph exposes all available rooms, assets, and objects, while the contracted graph exposes only room nodes.
- Scene-graph representations: 82.1% reduction in representation tokens is achieved by contracting the office scene graph before semantic search.
F Semantic Search Evaluation Results
The semantic search evaluation reports explored-node sequences for simple and complex instructions in both office and home environments. Examples span object, room, attribute, negation, and multi-constraint queries.
- Office semantic search: Office queries include locating objects or rooms using containment, attributes, negation, and multi-step relational constraints.Examples ask for objects containing screwdrivers, offices without cabinets, and offices satisfying cabinet-content conditions.
- Reported search behavior: The reported outputs are ordered sequences of room or region nodes explored for each instruction.Examples include office sequences such as peters_office michaels_office nikos_office tobis_office and home sequences such as bedroom2 bedroom3 closet0.
- Evaluation organization: The evaluation is organized into simple and complex semantic-search instruction sets for both office and home environments.Tables 13–16 list the explored-node sequences for these four evaluation settings.
- Home semantic search: Home queries include finding objects and rooms using object identity, room type, proximity, counts, contents, and comparative properties.Examples include finding wine, thorny plants, bathrooms without toilets, bedrooms near a home office, and rooms with specified contents.
G Causal Planning Evaluation Results
The causal-planning evaluation examines correctness, executability, and iterative-replanning counts across simple and long-horizon instructions. Detailed sequences show navigation, access, pickup, release, and device-operation actions.
- Evaluation measures: Tables 17 and 18 report correctness, executability, and the number of iterative replanning steps for simple and long-horizon instructions.A maximum of 5 replanning steps was used when no executable plan was identified.
- Failure inspection: The full plan listings identify the precise action where a plan failed by highlighting the corresponding regions in red.This presentation covers SayPlan and all baseline methods for the evaluated instructions.
- Instruction execution: The evaluated plans include navigation and manipulation tasks such as refrigerating an orange, cleaning a plate, and placing a document on Will’s desk.The listed sequences combine goto, access, pickup, release, opening, closing, and turning-on actions.
- Long-horizon plans: Long-horizon sequences distribute multiple pickups and deliveries across supplies_station and meeting_room3.One sequence collects bottled water, orange juice, and biscuits before releasing them in the meeting room.
H Scalability Ablation Study
The scalability study tests semantic search and iterative replanning as the collapsed 3DSG grows. Failures were attributed to input-token limits, while larger graphs could require more exploration steps.
- Study design: The study evaluates how increasing nodes in the collapsed 3DSG affects semantic search and iterative replanning.The analysis focuses on whether the LLM can attend to task-relevant graph regions as environments grow.
- Failure boundary: All observed failures in semantic search and iterative replanning resulted from exceeding the LLM input-token limit, including GPT-4’s 8192-token limit.The passage attributes these failures to input size rather than another reported source.
- Scaling behavior: Larger environments may require more semantic-search steps because additional relevant floor or room nodes must be considered.The analysis states that increasing noisy node counts did not significantly affect the LLM’s reasoning or attention to relevant 3DSG regions.