PaperlayerView exact revision on arXiv

Source-linked AI summary

S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

Yalun Dai, Hao Li, Shulin Tian, Runmao Yao, Yuhao Dong, Fangzhou Hong, Zhaoxi Chen, Fangfu Liu, Tao Wang, Kim-Hui Yap, Ziwei Liu

arXiv:2606.20515v3cs.CV

TL;DR

Real-world spatial reasoning concerns evolving 3D scenes, but existing tool-augmented agents largely rely on static images or isolated observations. S-AGENT uses a VLM planner, spatial tools, and temporal memory to accumulate evidence across views and frames; it consistently improves VLMs in training-free evaluation, including a 4.5% gain over GPT-5.4 on MMSI-Bench.

  • Problem

    Existing tool-augmented spatial agents largely reason from static images or isolated observations, while real-world spatial intelligence involves hidden, evolving 3D scenes observed continuously.

    Source: §1 · p. 2

  • Method

    S-AGENT formulates spatial reasoning as spatio-temporal evidence accumulation, using a VLM planner, hierarchical spatial tools, and temporal memory to integrate evidence across views and frames.

    Source: §1 · pp. 2–3

  • Results

    S-AGENT consistently improves VLMs in training-free zero-shot evaluation, including by 4.5% over GPT-5.4 on MMSI-Bench.

    Source: §1 · p. 3, §3.2 · p. 9

  • Takeaways & Limitations

    Reported gains are especially strong on motion, perspective-aware, and multi-step spatial reasoning tasks.

    Source: §4 · p. 13

Abstract

from arXiv · show

Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textbf{\textsc{S-Agent}}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, \textsc{S-Agent} reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, \textsc{S-Agent} casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (\textit{e.g.}, counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that \textsc{S-Agent} consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on \textsc{S-Agent}-generated spatial trajectories \textsc{S-300K} yields \textsc{S-Agent-8B}, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).

1. Introduction

S-AGENT addresses stateless, frame-centric spatial reasoning by treating multi-view images and videos as spatio-temporal evidence accumulation. It combines VLM planning, spatial tools and experts, and temporal memory to build persistent 3D scene knowledge.

Introduction

  • Current VLMs rely largely on passive 2D data and lossy semantic representations, while existing tool-using agents mainly reason over static images and struggle to maintain object states or integrate evidence across views and time.This creates a semantic-to-geometric gap between qualitative inference and high-fidelity geometric grounding.Source: §1 · p. 2
  • S-AGENT casts the VLM as a semantic planner and uses hierarchical tools to ground objects in 2D, add multi-view 3D cues, and derive counts, measurements, orientations, and relative positions.The 3D cues include depth, coordinates, and camera poses.Source: §1 · p. 3
  • Scene Memory tracks entities across frames to preserve identity and suppress duplicate evidence, while Agent Memory stores tool observations and reasoning traces for iterative refinement.Together, these memories support evidence accumulation across frames and reasoning iterations as the 3D scene evolves.Source: §1 · p. 3
  • S-AGENT improves over GPT-5.4 by 4.5% on MMSI-Bench; S-AGENT-8B improves Qwen3-VL-8B accuracy from 31.1% to 41.6% and performs comparably to advanced closed-source models.S-AGENT-8B is fine-tuned on S-300K, generated from zero-shot S-AGENT trajectories on training data disjoint from the evaluation benchmarks.Source: §1 · p. 3

2. Method

The method treats spatial reasoning as iterative evidence acquisition over video or multi-view observations, supported by hierarchical spatial tools and complementary memories; it also uses generated trajectories to train a compact agent.

2.1. S-Agent Framework

  • S-AGENT iteratively answers questions over video or multi-view observations by having a VLM planner request evidence and update scene and reasoning memories until evidence is sufficient.Each evidence request is conditioned on the question and evolving memory, and tool outputs remain reusable across later reasoning steps.Source: §2.1 · p. 4

2.1.1. Hierarchical Spatial Evidence

  • Its three-level evidence hierarchy selects query-relevant 2D cues, lifts them into shared 3D geometry, then aggregates them into spatial knowledge with specialized experts.The experts address queries such as counting, relative direction, object orientation, and physical size or distance.Source: §2.1.1 · p. 5

2.1.2. Temporal Memory for Stateful Reasoning

  • Scene Memory binds repeated observations to persistent entities and reusable spatial facts, while Agent Memory records tool-use history, outcomes, uncertainties, and planner decisions.Together, they let the planner reuse accumulated scene evidence and track what has been tried and what evidence remains needed.Source: §2.1.2 · p. 6

2.2. Training-Time Distillation

  • S-AGENT generates S-300K trajectories from challenging, tool-relevant SenseNova-SI-800K [5] samples and decomposes them into answer-, turn-, and tool-level supervision for fine-tuning Qwen3-VL-8B as S-AGENT-8B.The teacher is S-AGENT instantiated with GPT-5.4; the resulting trajectory forms provide supervision for final answers, iterative tool-use decisions, and expert-level reasoning.Source: §2.2 · pp. 6–7

3. Experiments

Experiments evaluate S-AGENT across multi-image and video spatial reasoning benchmarks in zero-shot and trained-agent settings. Results show strong zero-shot performance, gains from trajectory distillation and memory, and tool-grounded evidence recovery in qualitative cases.

3.2. Zero-Shot Performance.

  • S-AGENT scores 46.4% average on MMSI-Bench, exceeding Gemini 3 Pro by 1.2% and GPT-5.4 by 4.5%.It also leads on camera motion (46.0%), object motion (48.7%), and multi-step reasoning (44.4%).Source: Table 1 · p. 8, §3.2 · p. 9

3.1. Experimental Setup

  • The evaluation spans four benchmarks: MMSI-Bench and ViewSpatial-Bench for multi-image reasoning, and ReVSI and VSI-SUPER for video spatial reasoning.Their tasks cover multi-view positional, geometric, motion, and multi-step reasoning; perspective-aware localization; 3D reasoning from dynamic observations; and spatial change over time.Source: §3.1 · p. 8
  • S-300K contains 292,391 SFT samples derived from 100K sampled SenseNova-SI-800K questions, with trajectories filtered for execution validity and answer correctness.GPT-5.4-planner trajectories are decomposed into full-answer, turn-level VLM-call, and expert/tool-specific samples; training uses Qwen3-VL-8B-Instruct for one epoch with an 8192-token limit.Source: §3.1 · pp. 8–9

3.2. Zero-Shot Performance.

  • S-AGENT reaches 60.0% average on ViewSpatial-Bench, beating GPT-5.4 by 14.4%.It leads on C-OVO (55.5%) and P-RD (81.1%), and improves over GPT-5.4 by 20.5% on P-SSRD.Source: Table 2 · p. 9, §3.2 · p. 9
  • On ReVSI, S-AGENT scores 58.8, ranks second overall, and outperforms all open-source general models and spatially specialized baselines.It leads on relative direction and route planning and ranks third on relative distance.Source: Table 3 · p. 10, §3.2 · p. 10

3.3. Trajectory Distillation from S-Agent

  • S-AGENT-8B consistently improves over Qwen3-VL-8B-Instruct and same-backbone S-AGENT across three main benchmarks, while remaining competitive with GPT-5.4 and Gemini 3 Pro.The results indicate that trajectory distillation teaches reusable tool-use and evidence-integration patterns; the base 8B planner can struggle with tool selection and noisy observations.Source: §3.3 · pp. 10–11, Table 4 · p. 11

3.4. Ablation Studies

  • On ViewSpatial, Level-1 2D evidence raises the VLM-only score from 45.6% to 49.0%, Level-3 experts reach 56.7%, and both memories together reach 60.0%.Raw Level-2 3D evidence provides limited benefit; scene memory alone scores 58.2% and agent memory alone 57.6%. Experts filter noisy geometric outputs into task-oriented measurements, positions, or conclusions.Source: Table 5 · p. 11, §3.4 · p. 11

3.5. Qualitative Analysis

  • In a first-person video case, S-AGENT recovers the shelf and telephone positions despite occlusion and correctly infers that the shelf is left of and behind the telephone.After initial grounding fails, targeted detection calls recover boxes; a depth-based relative-position expert lifts them into a metric 3D layout. Broader examples show task-dependent selection of measurement, counting, 3D, and route-planning tools.Source: §3.5 · pp. 11–12, Figure 4 · p. 12, Figure 5 · p. 12

4. Conclusion

S-AGENT frames spatial reasoning over continuous multi-view images and videos as spatio-temporal evidence accumulation, using hierarchical spatial tools and scene and agent memories. Experiments show training-free gains for strong VLMs, while its trajectories can be distilled into S-AGENT-8B.

  • S-AGENT consistently improves strong VLMs in training-free zero-shot settings, especially on motion, perspective-aware, and multi-step spatial reasoning tasks.Source: §4 · p. 13
  • S-AGENT uses a VLM planner to acquire hierarchical evidence from 2D grounding through 3D geometric lifting to expert spatial knowledge, with scene and agent memories supporting stateful reasoning.Source: §4 · p. 13
  • S-AGENT-generated trajectories can be distilled into S-AGENT-8B, enabling an open-weight 8B model to learn more reliable tool use and spatial evidence integration.Source: §4 · p. 13

Appendix · A. Related Work

Related work improves spatial reasoning through specialized training, tools, and access to long-video or multi-view evidence, but often leaves scene-specific evidence acquisition and persistent reasoning incomplete.

A. Related Work

  • Spatial VLM methods scale spatial instruction data, add geometry-aware modeling, or use reinforcement learning and curriculum design, advancing performance on spatial benchmarks.Examples include Cambrian-S [30] and SenseNova-SI [5] for data, Spatial-MLLM [26] and VST [29] for modeling, and SpaceR [19], ViLaSR [27], MindCube [24], and SpatialLadder [13] for training strategies.Source: A. Related Work · p. 16
  • Most spatial VLM approaches remain training-driven and single-shot, relying on internalized knowledge rather than acquiring explicit, scene-specific evidence at inference time.Source: A. Related Work · p. 16
  • Agentic spatial methods add geometric tools or structured computation, including Python-generated 3D APIs, robotic-tool coordination, constrained geometric computation, and active 3D exploration.Examples are VADAR [17], SpaceTools [6], GCA [7], and Think3D [38], respectively.Source: A. Related Work · p. 16
  • Despite their promise, agentic spatial methods do not fully capture continuous spatial understanding through temporal integration, cross-view object-state maintenance, and evolving scene representations.Source: A. Related Work · p. 16
  • Frame-compression methods sample, retrieve, or summarize frames for long-context VLMs, improving efficiency but risking loss of question-relevant spatial evidence.The passage cites [35] [36] [34].Source: A. Related Work · p. 16
  • Reconstruction-first methods use multi-view geometry or feed-forward models for stronger geometric grounding, but can incur unnecessary computation for sparse or localized queries.The passage cites [23] [14].Source: A. Related Work · p. 16
  • Both long-video approaches typically treat selected frames or reconstructed geometry as fixed context, leaving spatial grounding, cross-view association, and metric comparison to implicit reasoning or downstream steps.They improve access to visual or geometric information without fully linking evidence acquisition, spatial computation, and persistent scene-level reasoning.Source: A. Related Work · p. 16

B. Details of Tools and Experts

The tool hierarchy turns query-relevant 2D observations into 3D geometry and structured spatial knowledge. Specialized experts use this evidence to answer measurement, counting, orientation, and relation questions.

B. Details of Tools and Experts

  • Level 1 detects or grounds query targets in images, adds local depth cues, and selects informative video frames for subsequent perception.GroundingDINO detection returns localized boxes and related evidence; multi-frame grounding selects a visible frame before detection, while image-level depth supports simple depth, occlusion, and front/back reasoning.Source: B. Details of Tools and Experts · pp. 16–17
  • Level 2 uses metric_depth3d_tool to lift image evidence into metric depth, 3D coordinates, and camera poses for downstream spatial experts.Built on Depth-Anything-3, it provides the shared geometric substrate, especially for measurement and relative-position reasoning.Source: B. Details of Tools and Experts · p. 17
  • The Metric Measurement Expert samples grounded regions, recovers 3D points, and computes requested distances or physical dimensions as structured measurements.Its measurement route can use closest-point distance, center-to-center distance, or the longest object dimension.Source: B. Details of Tools and Experts · p. 17
  • The Counting Expert deduplicates detections across frames and aggregates candidates, filtering them by relational or attribute constraints when required.It returns a count with its aggregation mode, confidence, and supporting detections.Source: B. Details of Tools and Experts · pp. 17–18
  • The Visual Orientation Expert uses appearance cues and reference context to infer an object’s intrinsic facing direction or pose.It distinguishes object orientation from geometric relations between objects and returns supporting visual evidence.Source: B. Details of Tools and Experts · p. 18
  • The Relative Position Expert lifts grounded entities into a shared 3D coordinate system and compares their positions using the question’s reference frame.Routes include object-to-object direction, egocentric or viewpoint-conditioned direction, and cardinal-anchor reasoning.Source: B. Details of Tools and Experts · p. 18
  • The Object-Centric View Expert maps labelled views of a target to its object-centered frame to determine view-conditioned relations among surrounding objects.It uses corresponding object-centric frames and returns supporting views.Source: B. Details of Tools and Experts · p. 18

C. Details of S-300K

S-300K is constructed from SenseNova-SI-800K [5], which is disjoint from the evaluation benchmarks, through trajectory generation, filtering, and decomposition.

Trajectory generation

  • Zero-shot S-Agent with GPT-5.4 as planner generates tool-use trajectories from 100K sampled questions.Each trajectory contains the question and visual inputs, planner responses, tool calls and observations, and the agent’s final answer.Source: C. Details of S-300K · pp. 18–19

Trajectory filtering

  • Filtering retains only trajectories with valid, correct final answers under answer-type-specific checks, without requiring tool use.Numeric answers must reach MRA ≥0.6; multiple-choice predictions must match the option letter, while free-form answers must match exactly or contain the normalized ground truth.Source: C. Details of S-300K · p. 19

Trajectory decomposition

  • Decomposition creates full final-answer trajectories, independent planner-call samples, and specialized expert/tool-call samples.Expert samples are included only when their input is complete, the tool response is available, and the result can be verified.Source: C. Details of S-300K · p. 19

Dataset statistics

  • 292,391 SFT samples comprise 51,596 final-answer, 154,590 turn-level, and 86,205 nontrivial tool/expert samples.The 100,000 sampled questions yield 51,596 trajectories that pass quality filtering.Source: C. Details of S-300K · p. 19, Table 6 · p. 19

D. More Experiments

S-AGENT substantially outperforms existing methods on VSR, with especially large gains on long videos, while its VSC results are mixed and treated cautiously because VSI-SUPER’s reliability as an indicator of genuine spatial perception has been questioned [22].

D. More Experiments

  • 37.2 percentage points: under the 240-minute setting, S-AGENT surpasses the strongest Cambrian-S-7B-LFP baseline on VSR, a gain attributed to the frame-selection tool.Table 7 reports substantial VSR gains, particularly in long-video settings.Source: D. More Experiments · p. 20
  • On VSC, S-AGENT does not outperform Cambrian-S-7B-LFP but performs better on average than non-LFP baselines.The paper includes this result mainly as a reference because VSI-SUPER’s reliability as an indicator of genuine spatial perception has been questioned in recent work [22].Source: D. More Experiments · p. 20

E. Additional Qualitative Visualizations

Figures 6 and 7 show how S-AGENT adapts tool-use trajectories to varied spatial questions, grounding task-relevant evidence and converting visual observations into explicit intermediate evidence before answering.

E. Additional Qualitative Visualizations

Related papers

Loading 2606.20515v3…