PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning

Seokju Cho, Ryo Hachiuma, Abhishek Badki, Hang Su, Byung-Kwan Lee, Chan Hee Song, Sifei Liu, Subhashree Radhakrishnan, Seungryong Kim, Yu-Chiang Frank Wang, Min-Hung Chen

arXiv:2606.13673v1cs.CVcs.AI

TL;DR

SpatialClaw addresses limited flexibility in existing interfaces for open-ended 3D/4D spatial reasoning by using code as a stateful, iterative action interface. Across 20 benchmarks, it achieves 59.9% average accuracy, outperforming the recent spatial agent by 11.2 points without benchmark- or model-specific adaptation.

  • Problem

    Existing spatial agents offer limited flexibility for composing tools, observing intermediate results, and adapting analyses in open-ended 3D/4D spatial reasoning.

  • Method

    SpatialClaw uses code as a stateful action interface, allowing a VLM-backed agent to iteratively compose, inspect, and revise perception outputs in a persistent Python kernel.

  • Results

    59.9% average accuracy across 20 benchmarks, outperforming the recent spatial agent by +11.2 points across diverse static and dynamic 3D/4D reasoning tasks.

  • Takeaways & Limitations

    Action-interface design is an impactful, underexplored axis for improving spatial reasoning agents across benchmarks and VLM backbones.

  • Takeaways & Limitations

    Remaining errors are attributed mainly to the perceptual quality of the backbone VLM and composed perception tools, limiting further gains from interface design.

Abstract

from arXiv · show

Spatial reasoning, the ability to determine where objects are, how they relate, and how they move in 3D, remains a fundamental challenge for vision-language models (VLMs). Tool-augmented agents attempt to address this by augmenting VLMs with specialist perception modules, yet their effectiveness is bounded by the action interface through which those tools are invoked. In this work, we study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning. Existing spatial agents either employ single-pass code execution, which commits to a full analysis strategy before any intermediate result is observed, or rely on a structured tool-call interface that often offers less flexibility for freely composing operations or tailoring the analysis to each task. Both designs offer limited flexibility for open-ended, complex 3D/4D spatial reasoning. We therefore propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface. SpatialClaw maintains a stateful Python kernel pre-loaded with input frames and a suite of perception and geometry primitives, letting a VLM-backed agent write one executable cell per step conditioned on all prior outputs, enabling the agent to flexibly compose and manipulate perception results and adapt its analysis to both intermediate text and visual observations and the demands of each problem. Evaluated across 20 spatial reasoning benchmarks spanning a broad range of static and dynamic 3D/4D spatial reasoning tasks, SpatialClaw achieves 59.9% average accuracy, outperforming the recent spatial agent by +11.2 points, with consistent gains across six VLM backbones from two model families without any benchmark- or model-specific adaptation.

1. Introduction

SpatialClaw addresses VLMs’ difficulty with structured 3D/4D spatial reasoning by using code as a stateful, compositional action interface. Across 20 benchmarks, it achieves 59.9% average accuracy and outperforms a recent spatial agent by +11.2 points.

  • Motivation: VLMs struggle to reliably compose depth, camera pose, and temporal correspondence into coherent geometric arguments from pixels alone.These challenges arise in questions about object location, motion, and spatial relations in three dimensions.
  • Action interface: Existing spatial agents use either one-shot code execution or structured tool-calls, limiting adaptive, open-ended composition for complex 3D/4D reasoning.Single-pass execution commits before intermediate results are observed, while typed tool-calls limit task-specific computation through libraries such as numpy or scipy.
  • SpatialClaw: SpatialClaw uses a training-free stateful Python kernel in which perception outputs remain available as variables for later composition, inspection, and revision.Its perception tools produce masks, depth maps, camera geometry, and trajectories, while the kernel preserves state independently across turns.
  • Evaluation: 59.9% average accuracy across all 20 benchmarks lets SpatialClaw outperform the recent spatial agent by +11.2 points.The benchmarks span metric distance, camera and object motion, multi-view geometry, temporal reasoning, and spatial planning.
  • Evaluation: SpatialClaw achieves consistent improvement over baselines across most benchmarks and transfers across backbone models without benchmark- or model-specific adaptation.The reported evaluation provides a comprehensive empirical comparison of spatial agents.

2. Action Interfaces for Spatial Reasoning Agents

The section characterizes spatial agents by the interfaces used to acquire, inspect, and transform visual evidence. It contrasts tool-free reasoning, single-pass code, and structured tool-calls with SpatialClaw’s persistent, iterative code interface conditioned on intermediate results.

  • Interface Taxonomy: Action interfaces define how spatial agents acquire, inspect, and transform visual evidence.Without tools, VLMs reason directly over input images in natural language, without external computation or intermediate evidence.
  • Single-pass Code: Single-pass code commits to a complete Python strategy before observing intermediate outputs or runtime evidence.Retry mechanisms can repair syntax or runtime failures, but intermediate visual evidence does not enter the reasoning loop.
  • Structured Tool-calls: Structured tool-calls provide typed perception inputs and outputs but make test-time compositions difficult to express within predefined commands.The passage specifically highlights question-specific chains combining depth estimates with segmentation masks.
  • SpatialClaw: SpatialClaw uses code as an action interface, executing one Python cell per step in a persistent kernel for flexible iterative composition.Each subsequent step is conditioned on resulting text, variables, errors, and visualizations.

3. SpatialClaw

SpatialClaw uses a persistent Python workspace as its action interface, allowing a VLM-backed agent to compose, inspect, and revise spatial analyses across executable cells. A five-stage control loop structures planning, code generation, execution, feedback, and answer submission while preserving intermediate evidence for later reasoning.

  • Persistent workspace: A persistent Python kernel preloads input frames and perception and scientific primitives, enabling spatial analyses to be composed and revised as executable code.The workspace retains outputs such as masks, depth maps, numeric arrays, and plots as ordinary Python variables.
  • Persistent workspace: Each code cell can create or transform spatial evidence, render intermediate results, produce summaries, or submit an answer, with later cells accessing the shared program state.The agent can inspect coarse computations and refine its analysis rather than selecting the complete strategy in advance.
  • Agentic loop: The outer loop comprises planning, code generation, code execution, feedback assembly, and answer submission around the persistent kernel.The planner runs separately before execution, while the main agent generates one Python cell per step from the question, plan, trajectory, and prior observations.
  • Reasoning discipline: A unified system prompt structures the open-ended Python interface by defining runtime objects and requiring spatial conclusions to be cross-checked against multiple evidence sources where possible.The framework applies this reasoning discipline without benchmark-specific engineering.
  • Agentic loop: Feedback includes stdout, tracebacks, variable summaries, and images shown during execution, allowing the agent to reuse intermediate results and revise analyses after inspecting evidence.The loop continues until a valid ReturnAnswer() submission or the predefined maximum step count is reached.

4. Results

Across 20 spatial reasoning benchmarks and six open-source VLM backbones, SpatialClaw consistently improves spatial reasoning performance, with especially large gains on video/4D and multi-view tasks. It also outperforms alternative action interfaces and recent spatial agents, while baseline methods do not consistently beat no-tool reasoning.

  • Evaluation setup: 20 benchmarks spanning single-image, multi-view, general, video, and 4D spatial reasoning were evaluated, using six open-source VLM backbones with evaluation capped at 1,000 samples when necessary.All benchmarks used N_max=30, with all samples used when benchmark size did not exceed the cap.
  • Benchmark results: 26B to 397B parameter backbones show consistent gains without modification, indicating performance improvements across model scales without model-specific tuning.The reported gains hold across all six evaluated backbone models.
  • Action interface comparison: 11/13 meta-categories favor SpatialClaw over both Structured tool-call and Single-Pass Code, with the largest gains in categories requiring multi-step geometric composition.The compared interfaces use the same toolset, differing in response format and the system-prompt section describing the action interface.
  • Comparison with spatial agents: +11.2%p on average is SpatialClaw's largest margin over SpaceTools, while SpatialClaw outperforms all baselines across all benchmarks using the same Gemma4-31B backbone.SpaceTools ranks highest among the baselines, consistent with structured tool-calls generally outperforming single-pass code.
  • Limitations: Baseline methods do not consistently improve over the no-tool baseline, potentially reflecting limited zero-shot generalization or optimization for narrower task categories.The passage presents these explanations as hypotheses for the baselines' limited performance across diverse benchmarks.

5. Analysis and Insights

The analysis attributes SpatialClaw’s improvements to its flexible, stateful code interface, which generalizes across tasks, adapts tool composition to question type, and benefits problems requiring chained geometric reasoning. Its largest gains occur where cross-frame and cross-viewpoint computation is needed, while perception quality limits gains on near-saturated visual recognition tasks.

  • Finding 1: SpatialClaw generalizes across diverse spatial reasoning tasks without pre-defined utility tools.Removing utility wrappers while retaining core perception tools and scientific libraries achieves performance on par with full SpatialClaw, indicating that persistent-kernel computation can compensate for missing utility logic.
  • Finding 2: The agent adapts tool composition to question type, using KD-tree search and norm operations for distance questions and dot products for direction questions.Primitive usage frequencies across meta-categories are reported in Figure 5.
  • Finding 3: 11/13 categories show a net advantage over both Structured tool-call and Single-pass Code, with the largest lifts of +6–9 pp in Camera motion.These gains concentrate where persistent-kernel cross-step composition and revision support chained geometric computation across frames and viewpoints.
  • Finding 3: Perception quality is the bottleneck where gains are smaller, because visual recognition tasks are already near-saturated by the backbone VLM.The breakdown identifies the expressive action interface, rather than model capacity or tool coverage, as the primary performance driver.

6. Conclusion

SpatialClaw is a training-free spatial reasoning agent that uses code as its action interface, enabling flexible multi-step manipulation of perception outputs in a persistent Python kernel. Across 20 benchmarks and six VLM backbones, it achieves 59.9% average accuracy and surpasses a recent spatial agent by +11.2 points without task- or model-specific adaptation.

  • Conclusion: SpatialClaw uses code as its action interface, allowing a VLM to compose, inspect, and revise perception tool outputs across steps in a persistent Python kernel.The framework is training-free.
  • Conclusion: 20 spatial reasoning benchmarks and six VLM backbones from two model families were used for evaluation.The evaluation required no model- or benchmark-specific adaptation.
  • Conclusion: 59.9% average accuracy was achieved, outperforming the recent spatial agent by +11.2 points.This result was obtained without any model- or benchmark-specific adaptation.

Supplementary Material · A. Related Works

Prior work addresses limited spatial reasoning in VLMs through spatial supervision, geometry modules, and tool-augmented agents. SpatialClaw extends code-action interfaces to spatial reasoning with domain-specific design choices beyond general-purpose code execution frameworks.

  • A. Related Works: Spatial reasoning remains a persistently limited capability in vision-language models despite broad progress.Prior responses include distilling 3D annotations into instruction data and augmenting models with explicit geometry modules.
  • A. Related Works: Spatial-supervision approaches either distill 3D annotations into instruction data or augment VLMs with explicit geometry modules.The passage cites both strategies as common responses to limited spatial reasoning.
  • A. Related Works: Tool-augmented visual agents let LLMs compose calls to specialist vision modules, extending their capabilities through external perception.The literature includes both single-pass program synthesis and structured tool-menu interfaces.
  • A. Related Works: Early tool-augmented systems synthesize a complete program in one pass, whereas later systems dispatch requests through structured tool menus.These represent two successive interface designs for invoking specialist vision modules.
  • A. Related Works: Spatial reasoning agents include GCA’s semantic-analyst/task-solver decomposition, RieMind’s explicit 3D scene graph, and SpaceTools’ supervised and interactive-reinforcement coordination of predefined tools.These systems use geometric constraints, typed geometric operations, or learned coordination of perception tools.
  • A. Related Works: CodeAct showed that executable Python code outperforms JSON- and text-formatted action spaces for general-purpose LLM agents.This establishes code emission as a relevant action-interface paradigm.
  • A. Related Works: SpatialClaw instantiates the code-action paradigm for spatial reasoning with domain-specific design choices absent from general-purpose frameworks.The passage distinguishes SpatialClaw’s spatial-reasoning contribution from CodeAct’s focus on execution infrastructure.

B. Evaluation Protocol

SpatialClaw is evaluated across 20 benchmarks spanning five categories of static, dynamic, general spatial, and video understanding tasks, using a consistent per-sample scoring protocol adapted to each benchmark’s thresholds.

  • Benchmark Coverage: 20 benchmarks span five categories, including single-image spatial, multi-image spatial, video spatial and 4D, general spatial, and general video understanding.The evaluation covers named benchmarks across these categories, with the complete benchmark lists reported in the protocol passages.
  • Scoring Protocol: Categorical questions receive 1/0 per-sample scoring, while numerical questions use mean relative accuracy with each benchmark’s original acceptance threshold.The reported benchmark score is the unweighted mean of per-sample scores.
  • Scoring Protocol: SPAR-Bench view-change inference uses VCI, defined as mean relative accuracy averaged across five movement axes.VCI is the benchmark-specific metric for this task.
  • Sampling Protocol: 1,000 samples are evaluated for benchmarks exceeding 1,000 samples, using a randomly chosen fixed-seed subset reproducible with the supplementary code.The same subset is reproduced by running the supplementary code with the same seed.

C. Additional Analysis

SpatialClaw’s gains over structured tool-call are driven primarily by code composition, while geometric reasoning errors remain the leading failure mode. Secondary failures arise from perception limitations, VLM hallucinations, inconsistent reasoning, and weak error recovery.

  • Attribution of gains: Code composition is the main driver of SpatialClaw’s gains over structured tool-call.Composition means chaining multiple tool calls into a single coherent program.
  • Attribution of gains: Over 50% of wins are attributed to code composition, 19.5% to control flow, and 28.3% are interface-neutral.Control flow includes conditional branching or iteration over intermediate results; interface-neutral wins depend on visual recognition or luck.
  • Failure modes: Perception failures stem from VLM hallucinations, visual judgments beyond tool capabilities, and localized detection or segmentation errors that propagate downstream.These limitations affect recognition of objects, attributes, and spatial relations.
  • Failure modes: A non-trivial fraction of failures reflects poor error recovery, including commitment to flawed hypotheses or oscillation without convergence.Residual failures include ambiguous annotations and occasional budget, parsing, or runtime issues.
  • Future work: Reinforcement learning is proposed to improve tool selection, geometric-operation coding, and error recovery within the agent loop.This is identified as a promising direction for future work.

D. Baseline Implementations … E.5. Per-Frame Type Contract

The section defines controlled baseline interfaces and details SpatialClaw’s stateful, scalable execution system, including bounded visual context, persistent computation, sandboxing, and frame-index validation. Together, these designs distinguish flexible code actions from single-pass or structured tool-call alternatives while supporting reliable spatial analysis.

  • D. Baseline Implementations: SpatialClaw compares code actions with single-pass code and structured tool-calls while holding the agent loop, kernel, perception tools, and planner constant.Only the per-step action format changes, with step budgets adjusted when required by that format.
  • D.1. Single-Pass Code: The single-pass baseline compresses inference into one Python cell that performs perception, computation, and answer submission without planning or another action opportunity.It receives the question, key frames, and the same main-agent system prompt as SpatialClaw.
  • D.2. Structured Tool-Calls: The structured-tool-call baseline preserves multiple steps but restricts every step to exactly one named tool invocation represented as JSON.Each call is translated into Python and executed on the same kernel, while later steps can reference stored return values but not arbitrary Python expressions.
  • E.1. System Configuration: SpatialClaw separates language-model serving from perception serving, using vLLM-backed LLM endpoints and an HTTP service for Depth Anything 3 and SAM3.Kernel-side clients convert perception results into ordinary numpy arrays, and the DA3Nested-Giant-Large variant reconstructs metric-scale geometry.
  • E.1. System Configuration: The serving decomposition allows LLM and perception nodes to scale independently, while all LLM roles share a backbone pool with prefix-cache and KV-reuse benefits.The shared pool serves the agent, planner, grounding, and thinking roles.
  • E.2. Input Preprocessing: Input images are resized to a 768-pixel maximum long edge, and the agent receives at most 32 visual-context frames selected uniformly for longer videos or truncated for multi-image inputs.The persistent kernel retains the full sampled frame sequence for later revisiting.
  • E.3. Persistent Kernel: Each sample runs in a dedicated stateful IPython kernel, preserving variables and intermediate evidence across cells so the agent can revisit and recompose results without repeating tool calls.A per-cell wall-clock timeout limits runaway code.
  • E.4. Security Sandbox: SpatialClaw protects execution by statically rejecting unsafe code patterns and enforces frame-index alignment for composed perception outputs, raising errors on mismatched frames.The sandbox checks AST patterns and regex-detectable writes, while typed containers record absolute frame indices.

E.6. Error Handling … G.7. tools.Draw

SpatialClaw uses feedback-driven code execution with layered safeguards, standardized backbone evaluation, and prompts and tools designed for composable spatial investigation. Its API spans perception, geometry, temporal conversion, plotting, mask analysis, and image annotation.

  • E.6. Error Handling: E.6. Error Handling: Runtime failures become observations, allowing the agent to diagnose and revise code within the same episode instead of terminating.Format errors, sandbox rejections, exceptions, and timeouts are surfaced through subsequent feedback, while retries and hard budgets provide additional safeguards.
  • E.6. Error Handling: E.6. Error Handling: Perception calls retry across endpoints with increasing waits, while exceeded step, failure, or tool-call budgets force a guaranteed-answer termination path.tools.Reconstruct and tools.SAM3 retry transient failures; termination uses fallback strategies after budgets are exceeded.
  • E.7. Backbones: E.7. Backbones: Six open-source backbones are evaluated under identical prompts, tools, step limits, preprocessing, and serving configuration without per-benchmark or per-backbone tuning.All six use the same vLLM-based serving role and evaluation configuration.
  • F. Prompt Details: F. Prompt Details: SpatialClaw separates main-agent coding, planning, coordinate grounding, and visual reasoning into four VLM roles with distinct prompts and sample-dependent invocation counts.The main agent runs once per step, the planner once per sample, and kernel-invoked visual sessions vary by sample.
  • F.1. Main Agent Prompt: F.1. Main Agent Prompt: The main agent must return Purpose, Reasoning, Next Goal, and Code, while using documented visual access, tools, coordinate systems, robust computation, cross-validation, and kernel contracts.The prompt documents seven tool namespaces, four reference frames, five robust-computation principles, four evidence sources, and a ReturnAnswer API.
  • F.2. Planner Prompt: F.2. Planner Prompt: The planner plans investigations without seeing frames, mapping coordinate, geometry, visual-judgment, and textual-reading questions to specialized tools or visual sessions.Its output includes task analysis, information needs, an ordered computation plan, and a verification checklist.
  • G. Tool API Reference: G. Tool API Reference: Reconstruct and SAM3 provide aligned 3D reconstruction and segmentation outputs, while Geometry, Mask, and Time supply numerical, mask-statistics, and frame-time operations.Reconstruction exposes depth, intrinsics, extrinsics, world points, and rendering; SAM3 supports image/video segmentation and object-existence checks.

H. Limitations and Broader Impact · H.1. Limitations · H.2. Broader Impact

SpatialClaw’s main remaining bottleneck is the perceptual quality of its backbone VLM and composed perception tools, while its training-free design enables stronger spatial reasoning without added parameters or fine-tuning. The same configuration transfers across backbones and benchmarks without modification, supporting use in data-constrained domains.

  • H.1. Limitations: Perceptual quality in the backbone VLM and composed perception tools is SpatialClaw’s main remaining bottleneck.The failure-mode analysis attributes the largest share of remaining errors to perception rather than the action interface.
  • H.1. Limitations: Perception, rather than the action interface, accounts for the largest share of remaining errors.This limits the expected benefit of further interface design at the current evaluation scale.
  • H.1. Limitations: Further action-interface design has diminishing returns while perceptual quality remains the dominant error source.The paper identifies perceptual quality as the principal axis for future improvement.
  • H.2. Broader Impact: SpatialClaw is training-free and adds no parameters to the backbone VLM.This allows existing models to gain stronger spatial reasoning without additional training data or fine-tuning.
  • H.2. Broader Impact: The framework is especially valuable where data collection is expensive or impractical, including robotics, embodied applications, and assistive systems.Its training-free design avoids requiring additional training data.
  • H.2. Broader Impact: The same configuration transfers across backbones and benchmarks without modification.This supports extending existing models without benchmark- or backbone-specific changes.
Loading 2606.13673v1…