PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation

Daniel Honerkamp, Martin Büchner, Fabien Despinoy, Tim Welschehold, Abhinav Valada

arXiv:2403.08605v4cs.RO

TL;DR

Long-horizon mobile manipulation in large unexplored environments requires reasoning over navigation and manipulation, while much prior LLM work focuses on explored settings or one task type. MoMa-LLM grounds language models in dynamically updated open-vocabulary scene graphs and evaluates them on interactive semantic search, reporting improved search performance against prior approaches.

  • Problem

    Prior LLM work primarily addresses explored environments or navigation and manipulation in isolation, while long-horizon mobile manipulation requires reasoning in large unexplored environments.

    Source: §I · p. 1

  • Method

    MoMa-LLM grounds LLMs in dynamic scene graphs and structured compact scene knowledge, interleaving these representations with object-centric actions and low-level subpolicies.

    Source: §I · p. 2, B. High-Level Action Space · p. 4

  • Results

    MoMa-LLM outperformed fully learned or co-occurrence-based methods in efficient search, showed consistent performance across real-apartment episodes, and extended to abstract tasks.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The authors demonstrate that the approach extends to abstract tasks and point toward general household tasks.

    Source: §VI · p. 8

    LLM queries for high-level reasoning take up the majority of high-level reasoning time.

    Source: D. Runtime Analysis · p. 16

Abstract

from arXiv · show

To fully leverage the capabilities of mobile manipulation robots, it is imperative that they are able to autonomously execute long-horizon tasks in large unexplored environments. While large language models (LLMs) have shown emergent reasoning skills on arbitrary tasks, existing work primarily concentrates on explored environments, typically focusing on either navigation or manipulation tasks in isolation. In this work, we propose MoMa-LLM, a novel approach that grounds language models within structured representations derived from open-vocabulary scene graphs, dynamically updated as the environment is explored. We tightly interleave these representations with an object-centric action space. Given object detections, the resulting approach is zero-shot, open-vocabulary, and readily extendable to a spectrum of mobile manipulation and household robotic tasks. We demonstrate the effectiveness of MoMa-LLM in a novel semantic interactive search task in large realistic indoor environments. In extensive experiments in both simulation and the real world, we show substantially improved search efficiency compared to conventional baselines and state-of-the-art approaches, as well as its applicability to more abstract tasks. We make the code publicly available at http://moma-llm.cs.uni-freiburg.de.

I. INTRODUCTION

MoMa-LLM grounds language-model planning in dynamically built scene graphs for interactive search in large, unexplored environments. The introduction frames this approach against prior work focused on explored scenes or isolated navigation and manipulation tasks.

I. INTRODUCTION

  • Prior LLM planning work largely targets fully observed or explored environments, while large unexplored scenes pose challenges for executable, grounded plans.The introduction also notes that interactive articulated objects introduce many potential states and failure cases.Source: §I · p. 1
  • The proposed semantic interactive search task covers large scenes with many objects and receptacles, and evaluates performance using full efficiency curves rather than one time budget.The authors also propose AUC-E to summarize those curves.Source: §I · pp. 1–2
  • MoMa-LLM is reported to outperform state-of-the-art approaches across diverse fields while supporting zero-shot, open-vocabulary reasoning and abstract search tasks.The authors describe the approach as extendable to mobile-manipulation and household-robotic tasks.Source: §I · p. 2
  • The approach combines scalable dynamic scene graphs, compact structured knowledge for LLMs, and an object-centric action space.The listed contributions include open-vocabulary room clustering and classification.Source: §I · pp. 1–2

II. RELATED WORK

  • Rather than scoring frontiers from pairwise target-object relations, the paper treats semantic search as joint planning that includes interaction with doors and receptacles.The cited approaches focus on non-interactive search or directional reasoning.Source: §II · p. 2
  • Related work leaves interactive object search in large unexplored environments unaddressed, and the paper distinguishes its open-vocabulary reasoning from SayNav’s non-interactive, room-subgraph setting.The authors also contrast their semantic task with prior random-target setups limited in objects and receptacles.Source: §II · p. 2

III. PROBLEM STATEMENT: EMBODIED REASONING

The paper formulates embodied reasoning as acting toward a language goal in a partially observed environment that must be perceived and explored. MoMa-LLM combines hierarchical scene representations with language-model commands and low-level skills.

III. PROBLEM STATEMENT: EMBODIED REASONING

  • The agent must perceive and represent an environment while reasoning about task completion through exploration and interaction.The task is framed as a POMDP with state, action, and observation spaces, including posed RGB-D observations.Source: §III · p. 2
  • Semantic interactive search requires navigating and exploring through manipulation, including doors that block paths and targets stored in drawers or cabinets.The task extends prior interactive search to more objects and receptacles while retaining realistic room-object and object-object relations.Source: §III · pp. 2–3

IV. MOMA-LLM

  • MoMa-LLM grounds language models in hierarchical 3D scene graphs and a navigation Voronoi graph, with high-level LLM actions executed by low-level skills.The system assumes ground-truth semantic masks, depth, localization, and handle detection to focus on reasoning.Source: §IV · p. 3, Figure 2 · p. 3

1) Dynamic RGB-D Mapping:

2) Voronoi Graph:

MoMa-LLM builds a navigational Voronoi graph and derives room-level structure from door locations, object positions, and open-vocabulary classification. The graph links room and object representations for planning in explored scenes.

2) Voronoi Graph:

  • The navigational Voronoi graph excludes obstacle-adjacent and out-of-map nodes, keeps the robot-centric connected component, and is sparsified to reduce navigational nodes.It is derived from the inflated BEV map and its Generalized Voronoi Diagram.Source: 2) Voronoi Graph · p. 3

3) 3D Scene Graph:

  • Door-position density removes nearby Voronoi edges and nodes to split rooms; shortest paths infer neighboring rooms, and objects are assigned to rooms to avoid erroneous wall-crossing assignments.Doors may connect to multiple rooms.Source: 3) 3D Scene Graph · p. 4

4) Room Classification:

  • An LLM classifies rooms open-vocabulary from their object categories, with classification repeated at each high-level policy step as the scene graph evolves.The prompt is based on objects and room clusters in the scene graph.Source: 4) Room Classification · p. 4, Figure 3 · p. 4

B. High-Level Action Space

MoMa-LLM uses object-centric high-level actions and structured scene descriptions to ground LLM planning. It updates representations and action history as exploration reveals new information, while replanning from limited execution feedback.

B. High-Level Action Space

  • The object-centric action space provides navigation, opening, closing, exploration, and termination calls; subpolicies execute them and update scene representations as they succeed or fail.Navigation targets the Voronoi node associated with an object and then a free point around it; multiple matching objects are resolved by choosing the closest instance.Source: B. High-Level Action Space · p. 4

C. Grounded High-Level Planning

  • Structured scene-to-language prompts aim to ground LLM plans in scene realities, keep context specific, and support zero-shot open-vocabulary reasoning.The prompt encodes extracted scene representations as natural language for the language model.Source: C. Grounded High-Level Planning · p. 4, Figure 4 · p. 5

1) Scene Structure:

  • The scene encoding lists rooms and contained objects, bins path distances into adjectives, summarizes matching nodes, filters redundant open doors, and encodes object states in names.Source: 1) Scene Structure · p. 4

2) Partial Observability:

  • Frontiers are linked to rooms and distinguished as enclosed unexplored space or routes to new areas; receptacles and object states let the LLM infer affordances, while infeasible openings can fail at the subpolicy.The approach identifies frontiers to explorable areas and uses map hole-filling to distinguish frontier types.Source: 2) Partial Observability · pp. 4–5

3) History in Dynamic Scenes:

  • The LLM receives the latest scene representation and the last h actions, whose room and object labels are realigned to current scene-graph changes using interaction positions.For example, an explore action initially labeled for a living room can be relabeled kitchen after the room is reclassified.Source: 3) History in Dynamic Scenes · p. 5

4) Re-trial and Re-planning:

  • Replanning uses success, failure, or invalid-argument feedback; unexecuted infeasible commands prompt another command, while execution failures trigger updated scene encoding and action history.More than five failures without a state change terminate the episode as unsuccessful; detailed real-world failure feedback remains an open problem.Source: 4) Re-trial and Re-planning · p. 5

V. EXPERIMENTS

In simulation, MoMa-LLM-based methods lead the search-efficiency curves across time budgets, while experiments also assess room partitioning and search in unseen apartments.

V. EXPERIMENTS

  • Evaluation compares random and greedy heuristics, ESC-Interactive, HIMOS, an unstructured-LLM baseline, and MoMa-LLM with Hydra.ESC-Interactive scores frontiers and openable objects using co-occurrence rules; ESC and HIMOS use the same scene graph and low-level policies as MoMa-LLM.Source: §V · pp. 5–6
  • Full efficiency curves compare success across arbitrary step budgets, while AUC-E integrates the curve through 5,000 low-level steps.This addresses the dependence of success rate and SPL on a fixed, arbitrarily chosen time or step budget.Source: §V · p. 6

A. Simulation Experiments

  • The simulation evaluation uses seven unseen test apartments, with 25 procedurally generated episodes per scene varying start poses, targets, and object distributions.Source: A. Simulation Experiments · p. 6
  • MoMa-LLM-based methods achieve the highest performance across time budgets; MoMa-LLM with Hydra leads at some small budgets, but not overall.The curve plots success rate against maximum time budget.Source: Figure 5 · p. 6, A. Simulation Experiments · p. 7
  • Door-based room separation achieves higher average precision and recall with lower variance than Hydra, but can merge open kitchen/living rooms; camera-view assignment reduces cross-wall object-room errors.Its separated Voronoi graphs cover fewer ground-truth rooms per predicted room than Hydra’s.Source: A. Simulation Experiments · pp. 6–7, Table II · p. 7
  • MoMa-LLM matches HIMOS’s success rate with higher SPL and AUC-E; structured prompts outperform unstructured input, and frontier, history, distance, and nearby-object encodings affect performance.Frontier encoding is especially important; removing history causes a smaller drop, and a visited-room list helps but trails the full action history.Source: A. Simulation Experiments · p. 7

B. Real-World Experiments

MoMa-LLM transfers to a real apartment and supports abstract search tasks, while the conclusion reports efficient search compared with learned and co-occurrence-based methods. The supplementary material adds experimental details and examples.

B. Real-World Experiments

  • The real-world apartment spans four rooms, and tests assume localization, accurate semantic perception, and handle detection.The setup uses a Toyota HSR with an RGB-D camera and 270° LiDAR, ROS Nav Stack navigation, and N2M2 manipulation policies.Source: B. Real-World Experiments · p. 7

C. Towards General Household Tasks

  • On fuzzy queries, the agent found suitable objects and judged task completion; for three tasks beyond its subpolicies, it stopped after finding objects it deemed sufficient.The authors leave extension to arbitrary tasks for future work.Source: C. Towards General Household Tasks · pp. 7–8

VI. CONCLUSION

  • MoMa-LLM outperformed fully learned or co-occurrence-based methods, and both it and ESC succeeded in 8/10 real-world episodes.The authors also report that MoMa-LLM moved and opened objects more target-directly and efficiently than ESC.Source: B. Real-World Experiments · p. 7, §VI · p. 8

- Supplementary Material -

  • The supplementary material provides additional environment, subpolicy, and baseline details, further experimental results, and real-world video examples.Source: - Supplementary Material - · p. 10

S.1. SIMULATION ENVIRONMENT

The simulation setup modifies indoor scenes, procedurally generates object-search tasks, assumes accurate semantic perception, and executes navigation and interactions through specified low-level policies.

A. Modifications

  • Simulation scenes close exterior doors, remove six obstructive doors, scale Fetch to 0.85, and rename four misleading object categories.Source: A. Modifications · p. 10

B. Task Generation

  • Task generation aligns room and object labels to a prior, expands inside/on-top relations, adds admissible objects, and rejects unreachable or immediately visible targets.Source: B. Task Generation · p. 10

C. Perception

  • Perception assumes ground-truth instances and segmentation, sensing within 5 m, accurate open-state detection, and detection of receptacle contents after opening.Source: C. Perception · p. 10

D. Execution

  • The low-level action set includes movement, turning, opening and closing articulated objects, and episode completion; all methods first turn in place to initialize the scene graph.Source: D. Execution · p. 10
  • Opening an object navigates to a nearby Voronoi node, approaches free space in front, then triggers a magic action; navigation replans through unexplored space when needed.Magic interactions set joint values directly and receive a time weight of 30 for efficiency calculations.Source: D. Execution · pp. 10–11

S.2. REAL-WORLD ENVIRONMENT

The real-world environment includes multiple room types and object categories; execution uses Toyota navigation and manipulation systems, with a special workaround for one doorway.

A. Map

  • The mapped environment spans five room types and 54 object categories, with test starts in the kitchen, living room, and hallway and matched targets across methods.Source: A. Map · p. 11

B. Execution

  • Navigation uses Toyota’s ROS NavStack, while N2M2 policies open articulated objects; the kitchen–living-room door instead uses inside pushing because opening it disrupted localization.The robot uses LiDAR and depth cameras for dynamic obstacle detection; opening in the opposite direction is omitted.Source: B. Execution · p. 11

S.3. HIERACHICAL SCENE GRAPH STRUCTURE

The scene-graph supplement outlines how obstacle-based Voronoi structures form a sparse navigation graph and how object distances are translated into language.

S.3. HIERACHICAL SCENE GRAPH STRUCTURE

  • Pairwise equidistant obstacle faces form F2, the initial GVG edges; 3-Voronoi sets provide joints, and removing degree-2 edges sparsifies the graph.Source: §S.3 · p. 11

S.4. LANGUAGE ENCODING

  • Object distance is binned and mapped to natural-language categories to create a consistent relative language encoding.Source: §S.4 · p. 12, Table S.2 · p. 12

S.5. BASELINES

The Unstructured LLM baseline uses a pure JSON scene-graph prompt and exhibits more invalid actions and hallucinations. The section also describes metrics for evaluating room segmentation.

S.5. BASELINES

  • The Unstructured LLM baseline receives the same instructions and notes as MoMa-LLM but uses a longer, less structured JSON prompt that leads to more invalid actions or hallucinations.Failures include attempting to open nonexistent or already-open objects and calling done() before finding the target.Source: §S.5 · p. 12, Figure S.2 · p. 13

1) Metrics:

  • Room segmentation precision and recall are averaged across exploration to reflect instability, using a 0.05 m grid and room-specific wavefronts for separated Voronoi graphs.The 0.05 m resolution is denser than the 0.075 m resolution used in other evaluations to account for thin walls in iGibson.Source: 1) Metrics · p. 12
  • Room-segmentation purity measures how well each separated Voronoi component corresponds to a single ground-truth room, penalizing components that cover multiple rooms.Purity is averaged throughout exploration.Source: 1) Metrics · pp. 12–13

2) Room Segmentation and Classification:

The door-based Voronoi separation tends to under-segment some layouts, though the downstream policy remains robust to under-segmented rooms. Room-category accuracy varies across the reported closed-set and real-world evaluations.

2) Room Segmentation and Classification:

B. Reasoning

  • Fig. S.4 shows Voronoi graphs, frontiers, scene representations, and LLM prompts and answers, with visual markers for traces, closed doors, undiscovered targets, and selected points.Table S.4 lists fuzzy-search and infeasible queries together with the model's reasoning.Source: B. Reasoning · p. 15, Figure S.4 · p. 15

C. Deployment with Full Perception Pipeline

The deployment discussion describes perception and grasping requirements, runtime characteristics, and directions for extending MoMa-LLM. The current implementation recomputes the scene graph at each time step, while its time impact is reported as reasonable.

C. Deployment with Full Perception Pipeline

  • Full deployment requires an RGB-D sensor, localization and mapping, semantic segmentation, and grasp-pose detection; open-vocabulary reasoning supports different segmentation models.The evaluation focuses on representation and decision-making with ground-truth perception, while this section outlines full-deployment requirements.Source: C. Deployment with Full Perception… · p. 15

D. Runtime Analysis

  • LLM queries account for most high-level reasoning time, while navigation subpolicy calls are more frequent and manipulation takes longer per call.The passage reports similar total time spent in navigation and manipulation; runtime averages cover three real-world fuzzy-search episodes and 175 simulation episodes.Source: D. Runtime Analysis · p. 16, C. Deployment with Full Perception… · p. 16
  • The scene graph is not speed-optimized and is fully recomputed at each time step, but its time impact is reported as reasonable because it is needed only for high-level reasoning.The implementation does not update only areas that received new observations.Source: D. Runtime Analysis · p. 16

E. Extended Future Work Discussion

  • Future work targets noisy-perception scene graphs, denser spatial and geometric information, room classification for non-standard layouts, and richer visual feedback about object states and failures.The current approach encodes distances and spatial arrangements as adjectives and room-object relations.Source: E. Extended Future Work Discussion · p. 16

Related papers

Loading 2403.08605v4…