Source-linked AI summary
Spatial and Semantic Reasoning for LLM-Driven Robot Navigation via MCP
Jungsoo Lee, Jaegyun Park, Wansoo Kim
TL;DR
LLM-driven ROS navigation lacks both representations that make raw map data usable for spatial and semantic reasoning and reusable interfaces that avoid robot-specific wrappers. The paper adds a navigation-oriented representation layer as standardized external tools, then evaluates it on mapping and spatial- and semantic-reasoning navigation. The framework supports these tasks without modifying ROS infrastructure, with both evaluated backends achieving over 97% map coverage.
Problem
ROS occupancy grids are difficult for LLMs to interpret directly as spatial or semantic context, and adding LLM capabilities often requires custom wrappers that limit reuse.
Source: §1 · p. 1Method
The framework exposes visual map representations and waypoint-level semantic observations, associated with robot poses, as standardized external tools connected to existing ROS navigation stacks.
Source: §1 · pp. 1–2Results
The framework supported autonomous mapping and spatial- and semantic-reasoning navigation without ROS infrastructure changes; mapping coverage was 97.83% for Claude Sonnet 4.6 and 99.78% for GPT-5.5.
Source: §4.4 · p. 5, §6 · p. 6Takeaways & Limitations
The results indicate the feasibility of representation-mediated LLM navigation beyond simple command generation.
Source: §6 · p. 6Task 3 demonstrates navigation to waypoint-level semantic anchors under complete annotations, not exact object-coordinate navigation; Task 2 is sensitive to ambiguous spatial structures and backend-dependent interpretation.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as natural-language interfaces for robotic systems, yet their integration with Robot Operating System (ROS)-based navigation remains limited by two gaps. First, navigation data such as occupancy grids are represented as raw geometric messages that are difficult for LLMs to use directly as spatial or semantic context. Second, adding LLM-driven capabilities often requires custom wrappers or robot-specific interfaces, limiting reuse across systems. To address these challenges, we propose a non-invasive framework that connects LLM reasoning with ROS-based navigation through a navigation-oriented representation layer, exposed through the Model Context Protocol (MCP) as standardized, reusable tools so that any MCP-compatible LLM can access them without robot-specific wrappers. The visual map modules transform occupancy grids into metric, pose-aware images for goal reasoning, while the semantic annotation modules record waypoint-level observations with robot poses. We evaluate the framework on three tasks: autonomous mapping, spatial reasoning-based navigation, and semantic reasoning-based navigation. The results show that the evaluated LLM backends use these representations to achieve over 97% map coverage and select spatial or semantic navigation targets from natural-language instructions in a simulated indoor environment. This demonstrates representation-mediated LLM navigation without modifying the existing ROS navigation stack.
1. INTRODUCTION
LLM-driven ROS navigation faces both a representation gap, because raw occupancy grids are difficult to interpret as spatial or semantic context, and an integration gap that limits reuse. The paper proposes a navigation-oriented representation layer, exposed as standardized external tools, for map-based and waypoint-level goal reasoning without changing the robot-side stack.
1. INTRODUCTION
- Raw occupancy-grid arrays are difficult for LLMs to use as spatial or semantic context, while adding LLM capabilities often requires robot-specific wrappers that limit reuse.Existing interfaces may expose robot functions without defining navigation-specific representations for map, pose, and perception data.Source: §1 · p. 1
- The framework converts map, pose, and perception data into spatial and semantic contexts for exploration and navigation goal reasoning.Its visual maps provide metric, pose-aware spatial cues; waypoint annotations associate natural-language descriptions of landmarks, features, and places with robot poses.Source: §1 · p. 1
- Standardized external tools connect the representation layer to existing ROS SLAM and navigation stacks without modifying the robot-side stack.The framework demonstrates autonomous mapping and reasoning-based navigation across two LLM backends.Source: §1 · p. 2
2. RELATED WORK
Prior work has made ROS functions accessible to LLMs and developed spatial or semantic representations for language-guided navigation. This framework instead pairs navigation-oriented representations with reusable MCP tools for existing ROS stacks.
2. RELATED WORK
- Prior ROS interfaces expose topics, services, actions, or robot behaviors, but primarily address access to robot functions rather than representing navigation data for goal reasoning.Examples include ROSA, ROS-LLM [8], RoboNeuron [9], and ROS-MCP Server.Source: §2 · p. 2
- Prior LLM- and VLM-based navigation approaches use topological graphs, semantic or frontier targets, occupancy-grid structure, text maps, or semantic-spatial memory.These methods often couple representations to task-specific components such as routing, frontier selection, or memory retrieval.Source: §2 · p. 2
- The proposed framework exposes visual maps and waypoint-centric semantic records as MCP-compatible tools while leaving the existing SLAM and navigation stack unchanged.Table 1 compares protocol standardization, ROS compatibility, navigation-oriented representation, and robot-side stack modification.Source: §2 · p. 2
3. METHODOLOGY
The framework exposes navigation-oriented map and semantic representations as MCP tools, connecting LLM reasoning to an existing ROS stack. It turns occupancy grids into metric map context and stores waypoint observations for goal selection.
3.1 System Overview
- MCP tools extend generic ROS communication with navigation representations while leaving the existing SLAM and navigation stack unchanged.Representation modules are exposed to MCP-compatible clients through tool functions and JSON schemas.Source: §3.1 · p. 2
3.2 Visual Map Representation Tools
- The visual map tools combine occupancy-grid structure, metric axes, and robot pose so the LLM can reason about unexplored regions, room sizes, and navigation goals.An adaptive metric grid is overlaid using map dimensions, resolution, and origin; the robot pose is projected as an arrow.Source: §3.2 · pp. 2–3
3.3 Semantic Data Annotation Tools
- Waypoint-level semantic records pair natural-language place descriptions with observation poses, which can serve as approximate goals for requested objects, functions, or places.A prompt-defined filtering policy guides which observations are retained; the records distinguish safety landmarks, functional landmarks, and geometric features.Source: §3.3 · p. 3
3.4 Execution Flow
- The LLM combines annotated map images and waypoint records to choose a goal, then sends it through the existing navigation stack for planning and execution.Source: §3.4 · p. 4
4. EXPERIMENTS
In simulation, two LLM backends used the framework across autonomous mapping, spatial navigation, and semantic navigation tasks. Both achieved over 97% map coverage, while navigation success varied by task.
4.1 Experimental Setup
- The NVIDIA Isaac Sim setup used a Nova Carter robot with LiDAR, ROS 2, Cartographer SLAM, and Nav2 in an indoor warehouse.The environment contained two differently sized rooms, shelves, and other objects.Source: §4.1 · p. 4, Figure 2 · p. 4
4.2 Task Design
- Task 1 generated an annotated map and semantic data, which were then used as inputs for independent Task 2 and Task 3 evaluations.Task 1 exploration iteratively selected unexplored regions and recorded camera-based observations with waypoint poses until estimated coverage reached 90%.Source: §4.2 · p. 4, §4.2.1 · p. 4
4.2.1 Task 1 – Autonomous Mapping and Semantic Data Collection
- For Task 1, each LLM used visual map representations to identify unexplored regions, choose waypoints, and record selected semantic observations with their poses.Both backends autonomously explored and recorded observations; coverage was 97.83% for Claude Sonnet 4.6 and 99.78% for GPT-5.5.Source: §4.2.1 · p. 4, §4.4 · p. 5
4.2.2 Task 2 – Spatial Reasoning-Based Navigation
- For Task 2, the LLM used map-derived spatial properties, consulting semantic data when needed, to choose a goal satisfying a spatial condition.In evaluation, the condition was reaching the smallest room.Source: §4.2.2 · p. 4, §4.3 · p. 4
4.2.3 Task 3 – Semantic Reasoning-Based Navigation
- For Task 3, the LLM matched an abstract object reference to semantic entries and used the associated waypoint pose as its navigation target.When multiple matching entries existed, it selected the nearest waypoint using the robot’s current pose.Source: §4.2.3 · p. 4, §4.4 · p. 5
4.3 Evaluation
- Task 1 was run separately for each backend, and its resulting map and semantic data were used for that backend’s later evaluations.Task 2 and Task 3 were each repeated 30 times; Task 1 was assessed for at least 90% coverage and correct semantic recording.Source: §4.3 · p. 4
4.4 Results
- 100% of Task 3 trials succeeded for both backends; Task 2 succeeded in 26/30 Claude Sonnet 4.6 trials and 28/30 GPT-5.5 trials.Task 1 coverage was 97.83% for Claude Sonnet 4.6 and 99.78% for GPT-5.5; Table 3 summarizes results by backend.Source: §4.4 · p. 5, Table 3 · p. 5
5. DISCUSSION
The discussion identifies representation quality and backend capability as dependencies of performance, with Task 2 sensitive to ambiguous spatial structures and Task 3 limited to waypoint-level anchors under complete annotations.
5. DISCUSSION
- Task 3 results reflect navigation to waypoint-level semantic anchors under complete annotations, not exact object-coordinate navigation.The authors propose improving map representation and semantic annotation, and investigating grounding that combines semantic retrieval with spatial consistency checks under incomplete or noisy observations.Source: §5 · p. 5
6. CONCLUSION
The conclusion presents a non-invasive framework using visual map representations and semantic annotations to connect ROS and LLM reasoning. The reported tasks indicate feasibility of representation-mediated navigation beyond simple command generation.
6. CONCLUSION
- The framework supports autonomous mapping and spatial and semantic navigation without modifying existing ROS infrastructure.It connects ROS, LLM reasoning, and navigation through visual map representation and semantic data annotation modules.Source: §6 · p. 6
APPENDIX : EXPLORATION PROMPT
The exploration prompt directs repeated waypoint selection, navigation, and semantic recording until the LLM assesses coverage at 90% or more.
APPENDIX : EXPLORATION PROMPT
- At each waypoint, select one target at least 5 m away toward the largest unexplored area, avoiding backtracking within 3 m of visited locations.The prompt also directs the robot to face the unexplored or most informative direction.Source: APPENDIX : EXPLORATION PROMPT · p. 6
- After sending a navigation goal and monitoring its completion, capture a front-camera image and record landmarks, geometric features, and task-relevant objects.Semantic data are saved at the current (x, y, yaw).Source: APPENDIX : EXPLORATION PROMPT · p. 6
- Repeat exploration until map coverage reaches at least 90%.The prompt defines this as the stopping condition for the exploration loop.Source: APPENDIX : EXPLORATION PROMPT · p. 6