PaperlayerView exact revision on arXiv

Source-linked AI summary

S-Agent 2.0: Spatial Reasoning with a General Coding Agent

Rui Gong, Ruoyu Wang, Yalun Dai, Hao Li, Sidi Yang, Qihao Zhao, Hongyu Li, Zeyang Guan, Fangzhou Hong, Zhaoxi Chen, Kim-Hui Yap, Ziwei Liu

arXiv:2610.28354v1cs.AIcs.CV

TL;DR

Spatial reasoning systems can gather correct tool evidence yet fail to turn readings into correct answers. S-Agent 2.0 equips a general coding agent with a toolkit for acquiring, remembering, and interpreting scene evidence; on MMSI-Bench and ReVSI, it improves over the same agent without tools, with gains also reported on the stronger backbone.

  • Problem

    Even when spatial tools return the right evidence, models may misread it or fail to use it in the answer.

    Source: §1 · p. 2, §2.4 · p. 6

  • Method

    S-Agent 2.0 pairs a stock general coding agent with a command-line toolkit that grounds perception in shared scene memory and returns spatially interpreted readings.

    Source: §1.1 · p. 3

  • Results

    On MMSI-Bench and ReVSI, the toolkit adds +13.9 and +20.5 points over the same agent without tools on Qwen3.8-27B, and S-Agent 2.0 remains ahead of SpatialClaw on both benchmarks with Gemini-3.8-Flash.

    Source: §1.1 · p. 3, §3.2 · p. 8

  • Takeaways & Limitations

    The toolkit works across general coding-agent harnesses, including OpenHands, Codex, and Claude Code.

    Source: §1.1 · p. 3, §2.1 · p. 4

    Remaining errors include failing to simulate a required imagined viewpoint and measuring the right quantity in the wrong sense.

    Source: §3.4 · p. 10

Abstract

from arXiv · show

Spatial reasoning asks a model to work out where objects are in 3D space, how they relate to one another, and how they move. Agentic VLMs have recently brought large improvements on such tasks by letting the model actively gather evidence with 3D perception tools instead of answering in a single pass. Even so, on many questions the tools return the right evidence and the model still does not arrive at the correct answer: the gap between model and tools, where raw readings become usable conclusions, is left unfilled. We present S-Agent 2.0, a spatial reasoning agent that equips an off-the-shelf general coding agent with a command-line perception toolkit, and we design the interface around one principle: help the VLM close the evidence loop: acquire scene evidence, remember it in a persistent scene memory, and interpret each reading in plain spatial language. This design keeps the full flexibility of a coding agent, and every reading it acts on stays grounded in the tools' geometry. On MMSI-Bench and ReVSI, the toolkit improves a strong open-source model, Qwen3.8-27B, by +13.9 and +20.5 points over the same agent without tools, and a strong closed-source model, Gemini-3.8-Flash, by +11.2 and +9.6 points. With the open backbone S-Agent 2.0 reaches 66.5 on MMSI-Bench and 82.3 on ReVSI, ahead of the specialized system SpatialClaw on the same backbone (62.6 and 74.5), and with Gemini-3.8-Flash it reaches 74.1 and 90.5 against its 67.0 and 79.7. Both tool gains shrink as the backbone grows more capable, but SpatialClaw's shrinks far more on ReVSI, from +13.2 to +2.7, while ours stays at +9.6. S-Agent 2.0 also works across different general coding agent harnesses, such as Codex and Claude Code.

1. Introduction

Spatial tools can return correct evidence while models still misread it or fail to use it in their answers. S-Agent 2.0 pairs a stock coding agent with a toolkit designed to join model and tool perception, retain evidence, and interpret readings; it improves benchmark performance and works across coding-agent harnesses.

1. Introduction

  • Spatial reasoning tools can return correct evidence while models still answer incorrectly, including by misreading a reading or leaving evidence unused.The authors identify failures such as flipping a measured rotation’s sign or relying on visual judgment instead of a tool reading.Source: §1 · p. 2

1.1. S-Agent 2.0

  • S-Agent 2.0 adds a command-line perception toolkit to a stock coding agent, with calls that acquire evidence, store it in persistent scene memory, and return spatially interpreted readings.The design also grounds the model’s own perception in the same scene.Source: §1.1 · p. 3
  • On MMSI-Bench and ReVSI, the toolkit adds +13.9 and +20.5 points over the same agent without tools on Qwen3.8-27B.S-Agent 2.0 also surpasses the specialized agentic system SpatialClaw, whose ReVSI tool gain shrinks far more on Gemini-3.8-Flash while S-Agent 2.0’s remains substantial.Source: §1.1 · p. 3

1.2. Takeaways

  • S-Agent 2.0 works across general coding-agent harnesses, including OpenHands, Codex, and Claude Code.The method keeps the harness stock and adds perception through commands.Source: §1.2 · p. 3, §1.1 · p. 3
  • The remaining gap is grounding the question into a measurement, which fails in two classes.The paper describes failures in simulating an imagined viewpoint and measuring the right quantity in the wrong sense.Source: §1.2 · p. 3

2. Method

S-Agent 2.0 uses a general coding agent as planner and executor, with a command-line perception toolkit and persistent per-episode scene memory. Its method combines model and tool perception, keeps evidence addressable, and returns readings in spatial language.

2. Method

  • The framework uses a general coding agent as planner and executor, a command-line perception toolkit, and persistent per-episode scene memory.The agent chooses what evidence a question requires, acquires it through the toolkit, checks outputs, and answers from the readings.Source: §2.1 · p. 4, §2 · p. 4

2.1. A general coding agent as the spatial agent

  • The agent’s action space combines ordinary workspace actions with perception-tool actions; full outputs and frames remain on disk, while the agent chooses what to read into context.The framework assumes only coding agents’ CLI machinery, supporting use across harnesses.Source: §2.1 · p. 4

2.2. Scene memory kept by the environment

  • Persistent scene memory holds the raw frames, fused metric point cloud, and named instances, allowing later readouts to be computed from stored geometry.Readouts cover instance properties and relations, and the agent can query them by instance name.Source: §2.2 · pp. 4–5

2.3. Dividing perception by competence

  • The interface divides perception by competence: tools provide precise measurements, while the model supplies semantic prompts and can add or remove instances in the shared scene.The model can add imagined observers or remove erroneous tool registrations before readings are taken.Source: §2.3 · pp. 5–6

2.4. The language surface: numbers interpreted on arrival

  • Because correct measurements can still be misread or ignored, each command returns numeric fields with units alongside a plain-language conclusion naming entities and reference frame.The full measurements remain available on disk for inspection when needed.Source: §2.4 · pp. 6–7

3. Experiments

Experiments evaluate S-Agent 2.0 across multi-view and video spatial reasoning benchmarks, with two backbones and multiple coding-agent harnesses. The results show broad advantages over agentic baselines, while ablations and qualitative examples examine the toolkit’s interface and remaining error modes.

3.1. Setup

  • The evaluation covers multi-view and video spatial reasoning, using Qwen3.8-27B as the primary backbone and Gemini-3.8-Flash for the capability study.OpenHands is the default harness; the perception tools use frozen off-the-shelf models.Source: §3.1 · p. 7

3.2. Main result

  • With Gemini-3.8-Flash, S-Agent 2.0 has the highest average in Table 1 and Table 2, though GPT-6 leads on several MMSI-Bench categories and absolute distance remains a ReVSI deficit.The MMSI-Bench categories are camera, object, and region types; ReVSI includes numerical and multiple-choice question types.Source: §3.2 · p. 8, Table 1 · p. 8, Table 2 · p. 9

3.3. Ablations

  • Replacing the harness with Codex or Claude Code preserves S-Agent 2.0’s operation with the same workspaces and toolkit.Each alternative harness uses its own agent loop, prompting, and context management.Source: §3.3 · p. 8, Table 4 · p. 10
  • The toolkit is worth a large gain on both benchmarks with Qwen3.8-27B, and the score with tools keeps rising as the backbone grows more capable.SpatialClaw scores below ours on every backbone and benchmark, and its tool gain shrinks far more as the backbone grows.Source: §3.3 · p. 9
  • Interface ablations show that removing the model perception interface costs the most accuracy, followed by scene memory and interpreted readouts.The respective reported costs are −8.8 and −5.9, −3.5 and −2.5, and −1.3 and −1.2.Source: §3.3 · p. 9, Table 6 · p. 10

3.4. Qualitative analysis

  • Qualitative examples show SpatialClaw can acquire correct masks and positions yet reverse a relation or overturn a correct direction, while S-Agent 2.0 returns relations in the options’ wording.The question’s premise enters the scene as an oriented instance or compass anchor; remaining errors include imagined-viewpoint grounding and measuring the wrong sense of a quantity.Source: §3.4 · pp. 9–10

3.2. Main result

  • S-Agent 2.0 is ahead of agentic baselines on all five benchmarks with Qwen3.8-27B.Table 3 compares agentic spatial reasoning baselines on the Qwen3.8-27B backbone.Source: §3.2 · p. 8, Table 3 · p. 10

4. Conclusion

S-Agent 2.0 combines a stock coding agent with a command-line perception toolkit that closes the evidence loop, and it outperforms SpatialClaw across the tested benchmarks and backbones. The conclusion also highlights qualitative examples and provides code and run trajectories.

4. Conclusion

  • S-Agent 2.0 closes the evidence loop by joining tool measurements with model perception, retaining readings in scene memory, and returning them in spatial language.It leads on MMSI-Bench and ReVSI and stays ahead of SpatialClaw on every benchmark and backbone tested.Source: §4 · p. 10
  • Figure 4 shows two cases where SpatialClaw perceives correctly but answers incorrectly while S-Agent 2.0 answers correctly, plus examples of self-correction and grounding failures.The remaining failure classes shown are a wrongly imagined viewpoint and a quantity read in the wrong sense.Source: Figure 4 · p. 11

Reproducibility statement

  • Code and trajectories for all reported runs are available online, and Appendix B documents the toolkit’s command-line interface.Source: Reproducibility statement · p. 11

A. Related Work

Prior work improves spatial reasoning through training, tool-using agents, and visual evidence construction, while benchmarks increasingly test evidence spread across views and video. S-Agent 2.0 instead uses a general coding-agent harness and focuses its interface on shared, persistent, interpreted scene evidence.

Spatial reasoning in vision-language models

  • Training-based approaches add spatial annotations, geometry-aware modeling or tuning, and reinforcement-learning signals, while inference-time S-Agent 2.0 leaves the backbone untouched and gathers evidence on demand.The cited training lines include SpatialVLM [6], SpatialRGPT, Spatial-MLLM [38], and SpaceR [26].Source: A. Related Work · p. 16

Tool-augmented visual agents and code as action

  • Visual tool agents use synthesized programs, turn-by-turn tool dispatch, or code generation, but their tools target 2D recognition rather than maintaining metric 3D scenes across views and time.CodeAct [35] found executable code performed better than JSON or free text for general-purpose agents.Source: A. Related Work · p. 16

Agentic spatial reasoning

  • Agentic spatial systems use specialized controllers and rich geometry tools, whereas S-Agent 2.0 brings model perception and tool measurements into persistent scene memory and returns interpreted readings.Prior approaches include geometric programs, scene-graph queries, learned tool coordination, and SpatialClaw’s plan-act-reflect loop.Source: A. Related Work · p. 16

General-purpose coding agents

  • General-purpose coding-agent harnesses provide shell, filesystem, code execution, context management, and tool-calling infrastructure; S-Agent 2.0 adopts a harness unchanged and adds perception as command-line tools.The same method can run under more than one harness.Source: A. Related Work · pp. 16–17

Long-video and multi-view perception

  • Long-video frame-compression and 3D-reconstruction methods provide visual evidence but leave cross-view association and metric comparison to implicit reasoning; S-Agent 2.0 lets the agent choose perception tools per question.Its scene memory combines tool measurements with what the model registers itself.Source: A. Related Work · p. 17

Spatial reasoning benchmarks

  • Spatial benchmarks have expanded from single-image probes to multi-observation and video tasks, and the paper evaluates on five benchmarks in this landscape.Examples include MMSI-Bench for evidence across images and ReVSI for video-based 3D reasoning.Source: A. Related Work · p. 17

B. Toolkit interface

The toolkit exposes perception and spatial readouts through a command-line interface backed by scene memory. Its interface uses named instances, flexible geometry inputs, and replies that pair typed measurements with spatial-language conclusions.

B. Toolkit interface

  • Every toolkit command reads and writes scene memory, returns JSON on standard output, and saves rendered images for the agent’s file viewer.The commands are grouped by function in Table 7.Source: B. Toolkit interface · p. 17, Table 7 · p. 18
  • Scene references use names: reconstruction registers cameras and the room center, segmentation registers prompt labels, and duplicate labels receive numbered instance names.Frame-specific object views and model-chosen names are also supported.Source: B. Toolkit interface · p. 17
  • An instance position can be set from a top-down-map pixel, an image pixel lifted to 3D, or a camera position.Source: B. Toolkit interface · p. 17
  • Instance orientation uses a primary axis and an orthogonal secondary direction, with an upright-facing shorthand or explicit axis flags; orientation can be updated in place.Source: B. Toolkit interface · pp. 17, 19
  • Replies include status, typed numeric fields in metres on the reconstruction’s scale, and a spatial-language conclusion for each target.The side-reply example reports forward, right, and up offsets and summarizes the target as front-left and above the observer.Source: B. Toolkit interface · p. 19
  • The executable also exposes lower-level detection, measurement, matching, egomotion, gravity, and rendering commands that the agent is instructed not to use.Source: B. Toolkit interface · p. 19

C. Toolkit usage

Across three full-benchmark runs under OpenHands, scene construction commands were used in over ninety percent of episodes, while question-specific readouts varied by benchmark. Model-perception registration and registry edits were also routine.

C. Toolkit usage

  • Across every run, reconstruction, segmentation, and bird’s-eye-view commands appeared in over ninety percent of episodes.Source: C. Toolkit usage · p. 19
  • MMSI-Bench episodes used side, motion, and compass readouts, while ReVSI episodes used distance and instance-list readouts in line with their question types.Source: C. Toolkit usage · p. 19
  • Model-perception registration occurred in a quarter to a half of episodes and dropping instances in a fifth to two fifths, while reserved lower-level commands made up under two percent of calls.Source: C. Toolkit usage · p. 19

D. SpatialClaw with our readouts

This section tests whether interpreted readouts help within SpatialClaw by adding them to its kernel while leaving its harness, prompts, and perception backends unchanged. It compares the transplanted system with SpatialClaw and S-Agent 2.0 on the same Qwen3.8-27B items.

D. SpatialClaw with our readouts

  • The readout commands were transplanted into SpatialClaw as plain functions, with its harness, prompts, and perception backends unchanged.The transplanted commands return interpreted replies in Python dictionaries with numeric fields, per-target conclusions, and units.Source: D. SpatialClaw with our readouts · p. 19
  • list_instance was excluded because SpatialClaw lacks scene memory, while bev was excluded because it already renders a top-down map.Source: D. SpatialClaw with our readouts · p. 19
  • Table 8 reports toolkit calls per episode and the share of episodes using each command at least once across 1,000 MMSI-Bench and 1,000 ReVSI items.Image views include views of benchmark frames, segmentation overlays, and top-down maps.Source: Table 8 · p. 20
  • Table 9 compares SpatialClaw, SpatialClaw with the readouts, and S-Agent 2.0 on the same items using Qwen3.8-27B.Source: D. SpatialClaw with our readouts · p. 19, Table 9 · p. 20

E. Additional comparisons with SpatialClaw

Six additional ReVSI and MMSI-Bench examples show S-Agent 2.0 answering correctly where SpatialClaw answers incorrectly; each figure presents the question, intermediate evidence, and command-line records from both agents.

E. Additional comparisons with SpatialClaw

Loading 2610.28354v1…