SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

Introduction

• Grounded robotics planning must respect physical affordances, predicates, action effects, and the locations and topology of relevant regions, while pre-encoding all scene information becomes infeasible as environments grow.The introduction notes that earlier grounding approaches were primarily limited to small-scale settings, typically single rooms with pre-encoded scene information.Source: §1 · pp. 1–2
• 3D scene graphs provide a hierarchical, topological semantic representation that can encode object states, predicates, affordances, and attributes in natural language for LLM input.SayPlan uses this representation to ground task planning across environments spanning multiple rooms and floors.Source: §1 · p. 2
• Semantic search manipulates a collapsed 3D scene graph through expand and contract calls to keep a task-relevant subgraph small enough for LLM planning.The collapsed graph exposes only the full graph’s top level; the resulting subgraph helps the LLM stay within its token limit.Source: §1 · p. 2
• A classical planner such as Dijkstra connects LLM-generated high-level nodes, removing the need for the LLM to generate navigation and addressing infeasible long-horizon action sequences.The paper links longer planning horizons in complex environments with increased tendencies to hallucinate or produce infeasible sequences.Source: §1 · p. 2
• A scene graph simulator checks and refines the initial plan through iterative replanning, correcting unexecutable actions such as putting an item into a fridge that has not been opened.The pipeline aims to avoid failures from inconsistencies, hallucinations, and violations of environmental constraints and predicates.Source: §1 · p. 2
• Up to 82.1% reduction of full large-scale scene representations for LLM parsing accompanies near-perfect executability rates.SayPlan was evaluated on 90 tasks across four difficulty levels in an office floor with 37 rooms and 150 interactable assets and objects, and a three-storey house with 28 rooms and 112 objects.Source: §1 · p. 3

• Traditional PDDL [20, 21, 22]/ASP approaches lack scalability and task generality, while hierarchical and reinforcement-learning alternatives [29] [30], face data-demand and scalability challenges.Source: §2 · p. 3
• Prior LLM planners generated actionable embodied-agent plans [9] [8] [32] [7] [33], but grounding via detectors [8] [7], PDDL [9] [34], or value functions remained mostly single-room and scaled poorly with object count.SayPlan instead uses hierarchical, semantically structured 3D scene graphs to plan over large-scale scenes.Source: §2 · p. 3
• External-tool methods use APIs [35] [36] or environmental text feedback [37] [8]; CLAIRIFY [38] iterates on compiler errors, while SayPlan verifies plans with scene-graph simulator feedback for executable mobile-manipulator actions.Verification checks plans against constraints and predicates captured by the pre-constructed scene graph.Source: §2 · p. 3

3.1 Problem Formulation

• SayPlan targets natural-language task planning for mobile manipulators in large environments, addressing LLM token limits and hallucinations in long-horizon plans.The task requires interpreting abstract or ambiguous instructions and planning both navigation and manipulation across multiple floors and rooms.Source: §3.1 · pp. 3–4

3.2 Preliminaries

• SayPlan represents environments as hierarchical 3D scene graphs with floors, rooms, assets, and objects, whose nodes encode states, affordances, attributes, and poses.The simulator collapses, expands, and contracts the graph and forward-simulates plans to check actions against environment predicates, states, and affordances.Source: §3.2 · pp. 4–5

3.3 Approach

• Semantic Search collapses the scene graph, then uses LLM expand and contract calls to identify a task-specific subgraph, reducing initial input tokens by ≈80%.In-context examples and chain-of-thought guide node selection; memory of previously expanded nodes supports long search sequences without retaining the full interaction history.Source: §3.3 · p. 5
• Iterative Replanning delegates pose-level routes to a classical path planner and uses simulator feedback to revise plans until they are feasible.The simulator checks actions against graph predicates, states, and affordances; the validated plan is then passed to a low-level motion planner for execution.Source: §3.3 · pp. 5–6

Experimental scope

• The robot embodiment is a 7-degree-of-freedom arm with a two-fingered gripper attached to a mobile base.The evaluated plans must span multiple rooms and floors in two large-scale environments.Source: §4 · p. 6

Semantic Search

• Semantic search tests whether an LLM can use room and floor names and attributes in a collapsed 3D scene graph to find task-relevant assets and objects.The evaluation compares the LLM with a human baseline and a SayPlan variant using GPT-3.5.Source: §4 · p. 6

Causal Planning

• Causal planning evaluates plans for both correctness against the task goal and human-like solution, and executability under scene-graph constraints for a mobile manipulator.Correctness and executability are independent: a plan can satisfy one without satisfying the other.Source: §4 · p. 6
• The causal-planning baselines are LLM-As-Planner, which generates a full open-loop navigation and manipulation sequence, and LLM+P, which uses path planning but no iterative replanning.LLM+P is an ablated SayPlan variant with shorter-horizon plan sequences.Source: §4 · p. 6

5.1 Semantic Search

• GPT-4 SayPlan found the desired subgraph with 86.7% success on simple search tasks and 73.3% on complex tasks, while GPT-3.5 consistently failed to reason over the graph.GPT-3.5 hallucinated nodes or repeatedly explored the same node.Source: §5.1 · p. 6, Table 1 · p. 6
• Semantic search failed on negation and on some distance-based and count-based reasoning, where the LLM tended to hallucinate or miscount connected nodes.Examples included requests for an office without a cabinet or a bathroom without a toilet.Source: §5.1 · p. 7
• Collapsed-graph inputs reduced initial token requirements by 82.1% for the Office and 60.4% for the Home, while node contraction maintained near-constant input memory during search.The reductions come from exploring the hierarchical scene graph from a collapsed initial state and contracting explored nodes deemed unsuitable for the task.Source: §5.1 · p. 7, Table 2 · p. 7, Figure 3 · p. 7

5.2 Causal Planning

• 93% simple-task correctness was reported for SayPlan, LLM-As-Planner, and LLM+P; SayPlan achieved near-perfect executability through iterative replanning.The path planner reduces the planning horizon, while iterative replanning grounds plans in environment constraints. SayPlan failed to correct some hallucinated nodes in 6.67% of tasks with replanning limited to five steps.Source: Table 3 · p. 7, §5.2 · p. 8
• LLM+P reduced navigational path-planning errors but still made manipulation errors, whereas SayPlan mitigated these errors through iterative replanning.The remaining SayPlan execution failures included hallucinated nodes that were not corrected within the five replanning steps used in the experiments.Source: Table 3 · p. 7, §5.2 · p. 8

LLM reasoning limitations

• The underlying LLM fails at distance-based reasoning, node-count reasoning, and node negation, while its biases and inaccuracies can undermine plan validity.Future work could fine-tune models for these tasks or incorporate more complex graph-reasoning tools [46].Source: §6 · p. 8

Scene graph limitations

• SayPlan requires a pre-built 3D scene graph and assumes objects remain static after mapping, restricting adaptability to dynamic real-world environments.The paper suggests integrating online scene-graph SLAM systems and adding open-vocabulary representations beyond textual node descriptions.Source: §6 · p. 8

Simulator limitations

• The scene graph simulator may not feasibly provide relevant feedback for every planning failure in complex tasks with diverse predicates and affordances.Although it works well in the cases presented, expanding feedback coverage is identified as future work.Source: §6 · p. 8

7 Conclusion

• SayPlan surpasses existing techniques in producing correct, executable plans that a robot can follow.Validated plans were also translated to a real-world mobile manipulator operating across multiple rooms, assets, and objects in a large office.Source: §7 · p. 8
• Hierarchical 3D scene graphs and LLM semantic reasoning let SayPlan explore from the hierarchy’s highest level, significantly reducing the initial tokens needed to represent larger environments.A scene graph simulator checks plan feasibility and grounding through iterative replanning.Source: §7 · p. 8
• SayPlan generates task plans for mobile manipulators across multiple floors and rooms, and its validated plans have been executed by a real-world robot in a large office.The paper describes this framework as laying groundwork for future general-purpose service robotics in homes, hospitals, and workplaces.Source: §7 · p. 8

A Implementation Details

• GPT-4 [3] is the underlying LLM agent unless otherwise stated, with a prompting structure similar to Wake et al..The static prompt defines the agent role, scene-graph environment, output structure, and input-output examples; it occupies approximately 3,900 tokens.Source: A Implementation Details · p. 13
• During semantic search, the prompt’s 3D Scene Graph and Memory components are updated at each step; during iterative replanning, only Feedback is updated.Replanning feedback comes from the scene graph simulator.Source: A Implementation Details · p. 13
• In all cases, the LLM is prompted to output a JSON object containing arguments for the provided API functions.The static prompt is task- and environment-agnostic.Source: A Implementation Details · p. 13

B Environments

• The two evaluation environments span multiple rooms and floors, as shown in Figure 4.The paper gives entity and token breakdowns for both environments.Source: B Environments · p. 13, Figure 4 · p. 13
• The office environment contains 37 rooms and 151 interactable assets and objects on a real-world office floor.A mobile manipulator is present there to embody generated plans and evaluate their feasibility; video demonstrations are available on the project site.Source: B Environments · p. 13
• The Klickitat family home from the Stanford 3D Scene Graph dataset [11] spans 28 rooms across 3 floors and contains 112 interactable assets and objects.A 3D visual is available on the 3D Scene Graph project website.Source: B Environments · p. 13

B.1 Real World Environment Plan Execution

• Real-world execution requires a 2D metric map to align the posed nodes captured by the 3DSG, with visual location of appropriate visible assets and objects assumed at each room node.Source: §B.1 · p. 13
• The demonstration robot combines a Franka Panda 7-DoF manipulator with an LD-60 Omron mobile base and uses LiDAR to localize in the real world and corresponding 3DSG.Source: §B.1 · p. 14
• Pick, place, open, and close affordances use the motion controller from [44] with RGB-D grasp detection, while a behaviour tree manages execution and failure recovery.Source: §B.1 · p. 14
• Future work could expand SayPlan’s task diversity by incorporating pre-trained skills such as whisking, flipping, and spreading through imitation learning [49] or reinforcement learning [52].Source: §B.1 · p. 14

C Tasks

• The 90 evaluation instructions test four capabilities: simple search, complex search, simple planning, and long-horizon planning.Simple search queries reference scene-graph information; complex search requires inference. Planning tasks involve graph search, causal reasoning, and environment interaction, with long-horizon tasks spanning multiple rooms.Source: Table 6 · p. 14, C Tasks · p. 15

C.1.1 Office Environment; C.1.2 Home Environment

• Simple Search instructions are listed for the office and home environments in Tables 7 and 8.Source: Table 7 · p. 15, Table 8 · p. 16

C.2.1 Office Environment; C.2.2 Home Environment

• Complex Search instructions are listed for the office and home environments in Tables 9 and 10.Source: Table 9 · p. 17, Table 10 · p. 18

C.3 Simple Planning

• Table 11 lists simple-planning instructions evaluated in the office environment.Source: Table 11 · p. 19

C.4 Long Horizon Planning

• Table 12 lists long-horizon planning instructions evaluated in the office environment.Source: Table 12 · p. 19

D Full 3D Scene Graph: Office Environment

• Figure 5 shows the fully expanded office scene graph, exposing all rooms, assets, and objects.The LLM agent does not see all this information unless it expands every possible node without contraction.Source: Figure 5 · p. 20

E Contracted 3D Scene Graph: Office Environment

• 82.1% fewer tokens are required to represent the contracted office scene graph before semantic search than the full representation.Figure 6 shows the contracted graph exposing only the highest hierarchical level: room nodes.Source: Figure 6 · p. 21

Simple Search Office Environment Evaluation

• The simple search office evaluation lists explored nodes including peters_office, mobile_robotics_lab, robot_lounge1, agriculture_lab, manipulation_lab, and robot_lounge2.Source: F Semantic Search Evaluation Results · p. 23, Table 13 · p. 24

Complex Search Office Environment Evaluation

• For the complex office evaluation, the rotten-apple query with an even-numbered cabinet explores jasons_office, peters_office, michaels_office, nikos_office, tobis_office, and other offices (Table 14).Source: F Semantic Search Evaluation Results · p. 25, Table 14 · p. 26

Simple Search Home Environment Evaluation

• In the simple home evaluation, the FooBar search lists explored nodes across bedrooms, kitchens, dining rooms, and living rooms (Table 15).Source: F Semantic Search Evaluation Results · p. 27, Table 15 · p. 28

Complex Search Home Environment Evaluation

• In the complex home evaluation, the request for a dining room with exactly five chairs lists dining_room0 and dining_room1 as explored nodes (Table 16).Source: F Semantic Search Evaluation Results · p. 29, Table 16 · p. 30

G Causal Planning Evaluation Results

• Tables 17 and 18 report correctness, executability, and replanning iterations for each simple and long-horizon instruction.Five was the maximum number of replanning steps; red values indicate that no executable plan was identified by that point.Source: G Causal Planning Evaluation Results · p. 31, Table 17 · p. 31, Table 18 · p. 32
• Table 19 lists SayPlan’s and the baselines’ full plan sequences, highlighting the precise action where a plan failed.Source: G Causal Planning Evaluation Results · p. 32

Food preparation

• For the kebab task, the trace retrieves the item from the fridge and uses the microwave to heat it.The shown actions include opening the fridge, picking up the kebab, releasing it in the microwave, and turning the microwave on and off.Source: G Causal Planning Evaluation Results · p. 36

Waste disposal

• Disposal traces address a smelly kitchen item and an item held by the agent, with actions involving bins and release.The excerpts show access to a recycling bin and release of a sweet wrapper, as well as access to a rubbish bin and release of the wrapper.Source: G Causal Planning Evaluation Results · p. 36

Food preparation

• For the noodles task, traces retrieve and microwave the food, then place it on a lunch table.One sequence ends by carrying the noodles to the cafeteria and releasing them on the lunch table.Source: G Causal Planning Evaluation Results · p. 37

Waste disposal

• For rotten fruit in Dimity’s office, the trace retrieves an apple and carries it to the kitchen compost bin.The instruction specifies the correct bin; the shown plan accesses the compost bin and releases the apple.Source: G Causal Planning Evaluation Results · p. 37

Dishwashing

• The dishwashing task trace moves tableware to the dishwasher, runs a wash cycle, then places the fork and knife in a drawer.The sequence includes opening and closing the dishwasher, turning it on and off, and opening the drawer for release.Source: G Causal Planning Evaluation Results · p. 38

Office tasks

• The document task trace files a printed document in Will’s office cabinet and places an undergraduate thesis on his desk.The actions include opening the cabinet to release the document, then retrieving the thesis and releasing it at the desk.Source: G Causal Planning Evaluation Results · p. 38
• The coffee task trace takes a mug to the coffee machine, turns it on and off, then carries the mug to Niko’s desk.The instruction asks for coffee to be made and the mug placed on his desk.Source: G Causal Planning Evaluation Results · p. 39

Waste disposal

• The wrong-bin correction trace moves milk, orange peel, apple core, and banana peel between recycling and rubbish bins.The supplied actions show opening bins, picking up items, and releasing them; they do not state whether the corrections succeeded.Source: G Causal Planning Evaluation Results · p. 39

G Causal Planning Evaluation Results

• For the soda-spill task, the action sequence disposes of the can in recycling and retrieves a paper towel from the supplies cupboard.The instruction asks the robot to throw away the can and take something to clean with.Source: G Causal Planning Evaluation Results · p. 40
• The sandwich task sequence retrieves bread, butter, cheese, and tomato from the kitchen and places them on the lunch table.Bread and butter are picked up at the kitchen bench, while cheese and tomato are retrieved from the fridge.Source: G Causal Planning Evaluation Results · p. 41
• The dishwasher task sequences unload a plate, bowl, and spoon and place them in a cabinet or drawer.The actions include opening the dishwasher, cabinet, and drawer, then closing the storage locations.Source: G Causal Planning Evaluation Results · p. 42
• At supplies_station, the plan opens cupboard2 and picks up five bottled waters, orange_juice, and biscuits; after closing it, the plan goes to meeting_room3 and releases the five waters and orange_juice.Source: G Causal Planning Evaluation Results · p. 42
• For attendees seated in meeting room 1, each limited to a single bottle, the shown sequence goes to supplies_station, retrieves bottled_water1 from cupboard2, then accesses table5 and releases it.Source: G Causal Planning Evaluation Results · p. 43
• The evaluation covers mobile-manipulator task-planning action sequences for both simple and long-horizon instruction sets.The supplied table caption gives the evaluation scope but no outcome values.Source: Table 19 · p. 45

H Scalability Ablation Study

• The scalability ablation varied collapsed-graph room-node counts to assess semantic search and causal planning as the environment scale increased.Source: H Scalability Ablation Study · p. 46, Figure 7 · p. 46, Figure 8 · p. 46
• Across semantic search and iterative replanning, failures arose only when LLM input exceeded token limits (8,192 for GPT-4), while added nodes did not significantly impair reasoning or attention.Larger environments could require more exploration steps before semantic search converged because more relevant floor or room nodes might be considered.Source: H Scalability Ablation Study · p. 46

I Real World Execution of a Generated Long Horizon Plan.

• Figure 9 depicts execution of a generated and validated long-horizon task plan on a real-world mobile manipulator.Source: Figure 9 · p. 47

J Input Prompt Structure

• The prompt defines the agent role, environment actions and state, graph APIs, and a JSON response format for exploration and planning.Static prompt components remain fixed during both semantic search and iterative replanning.Source: J Input Prompt Structure · p. 48

K Example Input-Output Interaction Between LLM and Scene Graph Simulator During Semantic Search

• In the semantic-search example, SayPlan explores rooms to locate the coffee machine, mug, and Tom’s wardrobe, then switches to planning.It contracts Jack’s room after finding nothing relevant, to save token memory.Source: K Example Input-Output Interaction… · p. 49

L Example Input-Output Interaction Between LLM and Scene Graph Environment During Iterative Replanning

• In the replanning example, simulator feedback that the mug is inaccessible prompts SayPlan to open its closed wardrobe; the revised plan is verified.Source: L Example Input-Output Interaction… · p. 50