PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Strategies, Not Actions: Vector-Quantized Geodesics as Tools for LLM-Driven Agents

Gabriel Turinici

arXiv:2610.00613v1cs.AIcs.ROcs.SYeess.SY

TL;DR

LLMs can struggle with spatial reasoning, although it is central to navigation and other tasks involving movement through space. This paper derives reusable tools from geodesic trajectories and uses an LLM to describe and select them. In a grid-world evaluation, the tool-based approach improves target-reaching success over direct LLM queries, and with collision filtering matches thinking mode at a fraction of its cost.

  • Problem

    LLMs lack a built-in notion of metric space, obstacles, or continuous motion, limiting their spatial reasoning for tasks such as navigation and robotics.

    Source: §1 · p. 1

  • Method

    The approach quantizes sampled grid-world geodesics into reusable tools, which an LLM describes offline and selects online from the current observation and goal.

    Source: §1 · p. 2

  • Results

    The tool library improves target-reaching success over direct LLM queries, and with a deterministic collision filter matches full thinking mode at a fraction of its wall-clock cost.

    Source: §1 · p. 2, §5 · p. 8, §7 · p. 11

  • Takeaways & Limitations

    The results suggest that, in this setting, LLMs are more effective orchestrating geometry-derived skills than directly planning spatially.

    Source: §7 · p. 11

    The evaluation uses a discrete, low-dimensional 2D grid with a handful of moving obstacles, leaving scaling to continuous state spaces, 3D navigation, and robotics open.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language model (LLM) based agents are often criticized for lacking spatial understanding and mainly exploiting statistical text patterns. We investigate their spatial comprehension through an architecture combining geometrical tools with a LLM serving as a high-level orchestrator in grid-world environments. The agent first collects geodesic trajectories, which are then vector-quantized to extract a representative subset. Offline, the LLM associates a natural language description of the underlying behavioral patterns to each selected trajectory, making it a tool. Online, the LLM chooses the appropriate tool conditioned on the current state and goal. Low-level control is handled by primitive actions that execute the trajectory associated with the tool. From an agentic AI perspective, this approach separates learning into two levels: tool discovery is handled through unsupervised quantization of trajectories, while reasoning and decision-making are handled by the LLM. We test the approach in a partially observable dynamic 2D grid environment with an open vision-language model (Qwen3.6-35B-A3B). Pairing the geometry-derived tool library with an agent-centered zoom tool and a collision detection tool lets a fast, non-reasoning configuration match the goal-reaching rate of a much more costly chain-of-thought version, while cutting the cost of a decision from minutes to seconds.

1 Introduction

The paper addresses LLMs’ limited spatial reasoning, which matters for tasks requiring movement through physical or simulated space. It proposes geometry-derived motion tools for an LLM to describe and select, rather than plan low-level movement directly.

1 Introduction

  • LLMs lack a built-in notion of metric space, obstacles, and continuous motion, a weakness relevant to navigation, robotics, and embodied tasks.The paper says they tend to rely on superficial statistical regularities rather than geometric reasoning.Source: §1 · p. 1
  • The approach samples obstacle-aware geodesics in a partially observable grid and compresses them into representative motion prototypes.An LLM describes the prototypes offline as callable tools, then selects among them online from the partial observation and goal.Source: §1 · p. 2
  • The low-level controller executes the selected tool’s fixed action sequence rather than asking the LLM to derive each movement.This resembles executing a memorized sequence after recognizing which pattern applies.Source: §1 · p. 2
  • The design separates unsupervised geometric skill discovery from LLM-based skill selection and reasoning.Skill discovery requires no labels or reward signal.Source: §1 · p. 2
  • Routing decisions through the geometry-grounded library improves target-reaching success over querying the LLM directly.The paper presents the grid-world evaluation as a test of this division of labor.Source: §1 · p. 2

2 Related work

The paper connects LLM tool use, vector quantization, and spatial reasoning, deriving its tools from environment geometry rather than treating them as preset APIs or asking the LLM to plan directly.

LLM agents and tool use

  • Unlike tool-use systems such as ReAct [Yao et al., 2023] and Toolformer [Schick et al., 2023], this approach derives its tool set from environmental geometry.Those systems are described as using external APIs, while these tools encode skills extracted from the environment.Source: §2 · p. 3

Quantization and skill discovery

  • Vector quantization compresses trajectories into representative prototypes, with signed-measure variants potentially allowing new skills to emerge from novel patterns.The paper notes that this potential is not evaluated in its fixed-tool experiments.Source: §2 · p. 3

Spatial reasoning and geodesics

  • Prior work reports LLM spatial-reasoning weaknesses, while complementary systems keep geometric computation in external representations or modules.This paper instead uses geodesics for offline skill discovery and reserves the LLM for describing and selecting skills.Source: §2 · p. 3

3 Grid-world environment

The paper evaluates navigation in a controlled, partially observable 2D grid with moving cars and a target. Because the environment evolves quickly relative to LLM decisions, the agent commits to five actions at a time.

3 Grid-world environment

  • The environment is a 2D grid loosely inspired by lane-crossing games, with a controlled cell navigating toward a target.Cars move upward as dynamic obstacles; cells outside the local observation window are unknown.Source: §3 · p. 4
  • The objective is to reach the goal while avoiding car collisions, ideally minimizing expected movements and elapsed time.Only locally observed cells are visible, and the available primitive actions are wait, up, down, left, and right.Source: §3 · p. 4
  • At each decision point, the agent executes a fixed five-action sequence before deciding again, regardless of intervening changes.This limits costly LLM queries in an environment that evolves quickly relative to the decision frequency.Source: §3 · p. 4

4 Method

The method collects obstacle-aware geodesics, compresses them into reusable trajectory prototypes, turns those into natural-language tools, and has an LLM select tools from partial observations and the goal.

4.1 Geodesics as shortest-path connections

  • Geodesics are shortest admissible paths through the grid that reach the goal without crossing cars, sampled across obstacle configurations and partial-information conditions.Sampling across conditions is intended to capture useful strategies beyond a route tailored to one instance.Source: §4.1 · pp. 4–5

4.2 Vector quantization of trajectories

  • Vector quantization compresses the sampled trajectory distribution into K representative trajectories, using a procedure such as K-medoids.In the grid application, 250 sampled geodesics are reduced to K = 5 prototypes; primitive one-move-then-wait tools are also added.Source: §4.2 · p. 6, Figure 4 · p. 7
  • The signed-measure formulation could add a prototype when newly observed geodesics are poorly represented by the existing set.This proposed way to let skills emerge during collection is recorded for future reference and not evaluated in the fixed-tool experiments.Source: §4.2 · p. 6

4.3 From quantized trajectories to LLM tools

  • An LLM describes each quantized sequence offline in terms of its displacement, strategic advantages, risks, and suitable circumstances, making it a callable tool.With deterministic collision checking, the description is shortened because code handles collision and timing reasoning.Source: §4.3 · pp. 6–7

4.4 Online orchestration

  • Online, the LLM selects a described tool using the local observation, goal, and optionally a filter that removes tools colliding with observed cars.The selected fixed sequence runs open-loop for five actions before the next decision; unobserved cars are not screened by the optional collision tool.Source: §4.4 · pp. 7–8
  • The design assigns unsupervised geometric skill discovery to the pipeline and asks the LLM to describe trajectories and choose among them.The LLM is not tasked with reasoning directly about coordinates, distances, or collision geometry.Source: §4.4 · p. 8

5 Results

Geodesic tools improve target-reaching success over direct raw-grid querying under matched conditions. With collision filtering, a fast non-thinking configuration matches full-thinking success at a small fraction of the cost.

5 Results

  • Without thinking mode or the collision tool, the model fails and tries to reproduce collision computations manually before getting lost.The authors attribute this difficulty to the complexity of the task.Source: §5 · p. 8
  • Without collision detection, thinking mode requires around 20k tokens per turn and takes non-negligible time.Under matched data and query budgets, the geodesic-tool pipeline reaches the target more often than direct raw-grid querying.Source: §5 · p. 8
  • With collision filtering, non-thinking geodesic tools match full-thinking success at a small fraction of cost and outperform direct raw-grid querying under matched budgets.The direct-query comparison uses matched data and query budgets.Source: §5 · pp. 8, 10, Table 1 · p. 10
  • All Table 1 rows use an agent-centered close-up, which Qwen3.6-35B-A3B requires for good results; without it, results are very poor.Figure 5 illustrates a representative episode that reaches the target at turn 10.Source: Figure 5 · p. 9, Table 1 · p. 10

6 Discussion and limitations

The evaluation uses a deliberately simple, discrete 2D grid world, so generalization to continuous, 3D, or robotic settings remains open. Other limitations include an untuned tool-library size, a single fully evaluated LLM, static tool descriptions, and occasional cell-level perception errors.

Scope of evaluation

  • The discrete, low-dimensional grid isolates the skill-discovery question but leaves continuous state spaces, 3D navigation, and robotic manipulation untested.Those settings would require continuous analogues of the discrete geodesic search and quantization.Source: §6 · p. 10

Tool-library design

  • K = 5 was chosen for tractable descriptions and orchestration rather than systematic tuning; scaling and new-skill triggering in non-stationary environments remain open questions.The authors also leave the effect of library size on success rate and description quality unresolved.Source: §6 · pp. 10–11

Model evaluation

  • The direct-query comparison uses the same LLM in both conditions, while benefits for smaller or non-vision LLMs remain untested.Only Qwen3.6-35B-A3B was fully evaluated; a larger frontier model was spot-checked but not fully evaluated for budget reasons.Source: §6 · p. 11

Tool descriptions

  • Offline tool descriptions were generated once and used verbatim, leaving iterative refinement based on orchestration failures for future work.Source: §6 · p. 11

Perception failure

  • The orchestrator sometimes misreads a nearby car by one cell, which can turn an otherwise-safe tool choice into a collision.Prompt engineering alone did not remove these perception errors.Source: §6 · p. 11

7 Conclusion

The paper separates spatial skill discovery from decision-making by turning vector-quantized geodesic trajectories into reusable tools selected by an LLM. The tool-based approach improves target-reaching success over direct querying and, with a deterministic collision filter, matches an expensive thinking-mode configuration at a fraction of its wall-clock cost.

7 Conclusion

  • The tool-based pipeline improves target-reaching success over querying the same LLM directly, and with a deterministic collision filter matches an expensive thinking-mode configuration at a fraction of its wall-clock cost.Source: §7 · p. 11
  • Geodesic trajectories from a partially observable grid are vector-quantized into reusable tools, described offline in natural language and selected online by an LLM given state and goal.Source: §7 · p. 11
  • The conclusion advocates geometry-derived skill discovery alongside LLM-based description and selection, with higher-dimensional, continuous, and robotic settings left for future testing.Source: §7 · p. 11

A.1 Example of tool description

The examples show how action sequences become tools with descriptions of their movement patterns. When collision checking is enabled, the offline prompt is shortened to net displacement and a strategic summary.

A.1 Example of tool description

  • When collision checking is enabled, the offline prompt is shortened to net displacement and a one-line strategic summary.The prompts are in the repository.Source: §A.1 · p. 14
  • WWWWW, UWWWW, DWWWW, LWWWW, and RWWWW describe waiting or moving one step in a direction before waiting four steps.The examples label these patterns as doing nothing or making a single step followed by waiting.Source: §A.1 · pp. 14–15
  • DRDRD and RRRRR describe a bottom-right staircase and a straight run to the right, respectively.Each description includes the sequence’s total displacement and actions.Source: §A.1 · p. 15
  • DDDDD, DRRRR, and DDRDD describe a straight downward run, a downward step followed by a rightward run, and a descent with one rightward step.The descriptions pair each sequence with its net displacement and action list.Source: §A.1 · p. 15

A.2 Reproducibility details

The reproducibility details specify the grid-world setup, tool library, collision-check procedure, and evaluation criteria. They also identify the repository, model, and observation zoom used.

A.2 Reproducibility details

  • The repository contains the code, prompts, tool files, and per-run logs.Source: §A.2 · p. 15
  • In a 25 × 25 grid, the agent observes only a centered local window (half-size 10), while cars move upward and wrap, hard walls absorb moves, and decisions commit five actions open-loop.Source: §A.2 · p. 16
  • The 10-tool library combines five K-medoids prototypes from 250 sampled geodesics with five one-move primitives.The sampled trajectories use randomized car configurations and partial information, are reset to a common origin, and are quantized with K = 5.Source: §A.2 · p. 16
  • A deterministic collision check filters candidate tools against cars visible in the observation window before the LLM is queried.If none survive, the agent waits; if one survives, it is selected without an LLM call; otherwise the LLM chooses among the surviving tools. Cars outside the window can make the check incomplete.Source: §A.2 · p. 16
  • The agent-centered zoom is a fixed-size crop rendered at higher resolution and supplied alongside the wide observation.It is passed whenever the observation is passed.Source: §A.2 · p. 16
  • Success means occupying the target at any step; collision or reaching 30 decisions counts as failure.Table cells report successes over episodes, with counts varying across conditions because of the compute budget; Qwen3.6-35B-A3B supplies offline descriptions and online selection.Source: §A.2 · p. 16

B More detailed trajectory plot

Figures 6–7 provide detailed views of the same Qwen3.6-35B-A3B episode shown in Figure 5. The rows include an agent-centered close-up alongside the observation.

B More detailed trajectory plot

Loading 2610.00613v1…