PaperlayerView exact revision on arXiv

Source-linked AI summary

S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

Yalun Dai, Hao Li, Shulin Tian, Runmao Yao, Yuhao Dong, Fangzhou Hong, Zhaoxi Chen, Fangfu Liu, Tao Wang, Kim-Hui Yap, Ziwei Liu

arXiv:2606.20515v3cs.CV

TL;DR

Existing spatial reasoning methods largely rely on static, isolated visual observations, limiting reasoning over evolving 3D scenes. S-Agent addresses this with planner-guided spatial tools and persistent memories that accumulate evidence across views and frames, consistently improving VLM spatial reasoning, especially for motion, perspective-aware, and multi-step tasks.

  • Problem

    Current VLMs and spatial agents largely reason from static or isolated 2D observations rather than continuously integrating evidence about evolving 3D scenes.

  • Method

    S-Agent uses a VLM semantic planner, hierarchical 2D-to-3D spatial tools, expert interpretation, and scene and agent memories for iterative spatio-temporal evidence accumulation.

  • Results

    S-Agent consistently improves open- and closed-source VLMs across multi-view and video benchmarks, achieving 60.0% on ViewSpatial-Bench and outperforming GPT-5.4 by 14.4%.

  • Takeaways & Limitations

    S-Agent provides stateful spatial reasoning that is especially effective for motion, perspective-aware, and multi-step tasks in continuous visual environments.

Abstract

from arXiv · show

Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textbf{\textsc{S-Agent}}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, \textsc{S-Agent} reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, \textsc{S-Agent} casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (\textit{e.g.}, counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that \textsc{S-Agent} consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on \textsc{S-Agent}-generated spatial trajectories \textsc{S-300K} yields \textsc{S-Agent-8B}, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).

1. Introduction

S-Agent addresses the limitations of static, stateless spatial reasoning by enabling VLMs to accumulate spatio-temporal evidence across continuous multi-view images and videos. It combines semantic planning, hierarchical spatial tools, and temporal memory to construct persistent 3D scene knowledge, with experiments validating robustness and generalizability.

  • Spatial intelligence enables VLMs to understand geometric relations in 3D environments, supporting physical-world operation and applications such as robotics, AR/VR, and autonomous driving.
  • Existing agentic VLM methods augment models with tools, executable programs, and geometric structure but largely remain focused on static spatial reasoning.The passage cites VADAR and SpaceTools as examples.
  • S-Agent reframes spatial reasoning over continuous multi-view images and videos as connecting partial, transient observations into a spatially structured understanding of the underlying 3D world.
  • The framework casts the VLM as a semantic planner while spatial tools, experts, and temporal memory provide explicit 3D awareness and accumulate evidence across frames and reasoning iterations.Its hierarchy spans low-level object grounding and depth information to high-level orientations and relationships.
  • Experiments on MMSI-Bench, ViewSpatial-Bench, ReVSI, and VSI-SUPER evaluate S-Agent with open-source and closed-source models in a training-free setting.The introduction reports that directly applying the framework consistently improves spatial reasoning ability.

2. Method

S-Agent formulates spatial reasoning as iterative updates to scene and agent states, using a VLM-driven planner, hierarchical spatial evidence, and persistent temporal memory. Its trajectories are also used to train compact agents.

  • 2.1: S-Agent iteratively updates a scene state and an agent state for spatial reasoning.The method organizes this formulation in Section 2.1.
  • Method: S-Agent acquires hierarchical spatial evidence, maintains temporal memory for stateful reasoning, and uses generated trajectories to train compact agents.These components are described across Sections 2.1.1, 2.1.2, and 2.2.
  • 2.1.1: A VLM-driven agentic planner requests targeted spatial evidence, such as distances between candidate objects and a reference object.Example requests include distances from the chair and plant to the sofa, followed by a comparison.
  • 2.1.2: Persistent spatial memory combines Scene Memory, which stores scene knowledge, with Agent Memory, which records how priors were obtained.The memory also updates objects, relations, and tool-use history.

Hierarchical Spatial Evidence

S-Agent performs spatial reasoning as iterative, stateful evidence accumulation over multi-view images and videos rather than isolated frame-level prediction. Its hierarchy progresses from 2D visual cues to 3D geometry and high-level spatial knowledge, while Scene Memory and Agent Memory preserve scene evidence and reasoning context.

  • Evidence Accumulation: S-Agent treats spatial reasoning as an iterative evidence-seeking process whose answer depends on the underlying 3D scene state rather than a single 2D projection.The framework supports videos and multi-view image sets, where observations may evolve over time or capture the same scene from different viewpoints.
  • Evidence Accumulation: A VLM planner selects evidence requests conditioned on the question and evolving memory, while spatial tools execute them and update both memory states.This separates semantic planning from spatial evidence acquisition and avoids treating tool calls as isolated actions.
  • Persistent Memory: Scene Memory consolidates reusable grounded entities and spatial facts, whereas Agent Memory records tool-use history, observations, failures, and intermediate reasoning context.Together, the memories help the planner track uncertainty, avoid redundant calls, and determine what evidence to request next.
  • Hierarchical Tools: The three-level hierarchy acquires 2D visual evidence, lifts it into 3D geometry, and aggregates it into scene-specific spatial knowledge.The levels correspond respectively to visual cues, geometric representations such as depth and metric coordinates, and expert outputs for counting, direction, orientation, and size or distance.

3. Experiments

Experiments show that S-Agent improves zero-shot spatial reasoning across diverse multi-image and video benchmarks, while trajectory distillation produces a compact agent with consistently stronger performance. Ablations further indicate that specialized 3D experts and temporal memory are important contributors to these gains.

  • Zero-shot results: S-Agent achieves the best zero-shot MMSI-Bench average, scoring 46.4% and exceeding Gemini 3 Pro by 1.2% and GPT-5.4 by 4.5%.It also leads camera motion (46.0%), object motion (48.7%), and multi-step reasoning (44.4%).
  • Zero-shot results: S-Agent scores 60.0% on ViewSpatial-Bench, outperforming GPT-5.4 by 14.4% and improving the challenging P-SSRD split by 20.5%.It achieves the best results on C-OVO (55.5%) and P-RD (81.1%).
  • Zero-shot results: S-Agent reaches 58.8 on ReVSI, ranking second overall while outperforming all open-source general and spatially specialized baselines.It performs best on relative direction and route planning and ranks third on relative distance.
  • Trajectory distillation: Trajectory distillation produces S-Agent-8B, which consistently improves over Qwen3-VL-8B-Instruct and the same-backbone S-Agent across the three main benchmarks.The distilled agent also achieves competitive performance against GPT-5.4 and Gemini 3 Pro, indicating that training transfers reusable tool-use and evidence-integration patterns.
  • Ablations: Enabling Level-3 3D experts raises the ViewSpatial score to 56.7%, while scene memory reaches 58.2%, agent memory 57.6%, and both memories yield 60.0%.The results indicate that specialized experts and memory modules provide complementary gains over weaker configurations.

4. Conclusion

S-AGENT frames spatial reasoning over continuous multi-view images and videos as spatio-temporal evidence accumulation rather than isolated, single-shot prediction. It combines active hierarchical spatial evidence acquisition with scene and agent memories.

  • Framework: S-AGENT is a spatial tool-use agentic framework for reasoning over continuous multi-view images and videos.It targets continuous visual inputs rather than isolated observations.
  • Reasoning paradigm: It formulates spatial reasoning as spatio-temporal evidence accumulation instead of single-shot prediction from isolated visual inputs.
  • Spatial evidence: A VLM planner actively acquires hierarchical spatial evidence spanning 2D grounding, 3D geometric lifting, and expert-level spatial knowledge.
  • Memory: S-AGENT maintains scene and agent memories while accumulating spatial evidence.

Appendix · A. Related Work

Prior work advances VLM spatial intelligence through larger spatial supervision, geometry-aware architectures, and spatially focused objectives, while agentic systems interleave reasoning with external tools. Long-video and multi-view methods typically compress frames or reconstruct explicit 3D representations, trading efficiency against potential loss of spatial evidence.

  • Spatial Intelligence in VLMs: Spatial-intelligence research scales supervision through large instruction datasets such as Cambrian-S and SenseNova-SI.These approaches target improved VLM spatial intelligence by increasing the availability of spatial instruction data.
  • Spatial Intelligence in VLMs: Other methods inject explicit spatial modeling or visual spatial tuning into multimodal backbones, including Spatial-MLLM and VST.These works modify multimodal architectures rather than relying only on additional supervision.
  • Agentic Spatial Reasoning: Agentic spatial reasoning extends general tool-use frameworks such as ReAct and visual systems including ViperGPT, Visual ChatGPT, and MM-ReAct.These systems interleave model reasoning with calls to external tools.
  • Agentic Spatial Reasoning: Recent spatial agents equip VLMs with explicit geometric tools or structured computation, exemplified by VADAR’s dynamically constructed 3D APIs.VADAR synthesizes Python programs over dynamically constructed 3D APIs.
  • Long-video and Multi-view Understanding: Long-video and multi-view methods commonly process continuous observations through frame compression or reconstruction-first pipelines.Frame-compression methods sample, retrieve, or summarize limited frames, whereas reconstruction-first methods build explicit 3D representations using multi-view geometry or feed-forward reconstruction.
  • Long-video and Multi-view Understanding: Frame compression improves efficiency but risks losing question-relevant spatial evidence.The cited methods compress observations before passing them to long-context VLMs.

B. Details of Tools and Experts

S-Agent organizes spatial reasoning into a hierarchy of tools and experts that localize entities in 2D, lift evidence into metric 3D geometry, and produce structured scene-specific spatial knowledge. Level 1 tools extract query-relevant visual evidence, Level 2 supplies shared geometric evidence, and Level 3 applies specialized reasoning routes.

  • Level 1 tools: Level 1 tools localize query-relevant evidence from images and videos through open-vocabulary detection, multi-frame grounding, depth cues, and informative frame selection.Detection produces boxes, confidence scores, labels, textual locations, and visualizations; multi-frame grounding selects a visible frame before detection, while keyframe search reduces redundant video input.
  • Level 2 tools: Level 2 metric_depth3d_tool lifts localized evidence into metric depth, 3D coordinates, camera poses, and visualizations across multiple images.Built on Depth-Anything-3, it provides the shared 3D geometric substrate for metric measurement and relative-position reasoning.
  • Level 3 experts: Level 3 comprises five specialized experts that integrate Level 1 evidence and, when needed, Level 2 geometry into structured spatial knowledge.The experts cover metric measurement, counting, visual orientation, relative position, and object-centric views.
  • Level 3 experts: The Metric Measurement Expert recovers metric 3D points from normalized object boxes to estimate camera-to-object distance, object-to-object distance, and physical object size.It deterministically maps planner-specified measurement requests to geometry-grounded routes.
  • Level 3 experts: The Counting, Visual Orientation, Relative Position, and Object-Centric View Experts respectively aggregate detections, infer intrinsic pose, determine 3D directional relations, and interpret surrounding objects across labelled viewpoints.Counting normalizes boxes across frames and removes duplicates; relative position lifts entities into a shared 3D coordinate system, while the other experts use relevant visual evidence.

C. Details of S-300K

S-300K is built from SenseNova-SI-800K through trajectory generation, quality filtering, and decomposition into complementary supervision formats. From 100K sampled questions, the pipeline produces 292,391 supervised fine-tuning samples.

  • Construction pipeline: S-300K is constructed from SenseNova-SI-800K, which is disjoint from all evaluation benchmarks, using generation, filtering, and decomposition stages.The pipeline begins with 100K sampled questions and zero-shot S-Agent trajectories generated with GPT-5.4 as planner.
  • Trajectory filtering: Filtering retains trajectories only when final answers are valid and correct for their answer type, excluding failures, unrecovered errors, and missing final answers.For multiple-choice questions, the predicted option letter must exactly match the ground-truth option letter and be extracted from the final answer field.
  • Trajectory decomposition: Retained trajectories are decomposed into final-answer, turn-level, and nontrivial tool/expert supervision formats.Final-answer samples preserve complete reasoning, while turn-level samples convert each planner call into an independent training sample to reduce excessively long contexts.
  • Dataset statistics: 292,391 supervised fine-tuning samples result from the three-stage pipeline applied to 100,000 raw agent traces.The dataset statistics report 51,596 trajectories passing quality filtering and one final-answer sample retained per filtered trace.

D. More Experiments

S-Agent substantially outperforms existing methods on VSR, with especially large gains in long-video settings. On VSC, it trails Cambrian-S-7B-LFP but exceeds non-LFP baselines.

  • Results on VSR: S-Agent substantially outperforms existing methods on the VSR subset, particularly in long-video settings.The results are reported in Table 7.
  • Results on VSR: 37.2 percentage points: S-Agent surpasses the strongest Cambrian-S-7B-LFP baseline under the 240-minute VSR setting.The paper attributes this gain to the introduction and strong performance of its frame-selection tool.
  • Results on VSC: On VSC, S-Agent does not outperform Cambrian-S-7B-LFP but performs better than the non-LFP baselines.

E. Additional Qualitative Visualizations

Additional qualitative examples show that S-Agent adapts its tool-use trajectory to diverse spatial questions by grounding relevant evidence, deriving explicit intermediate evidence, and producing final answers.

  • E. Additional Qualitative Visualizations: S-Agent adapts tool-use trajectories to counting, multi-step reasoning, relative-position, and route-planning questions.The examples illustrate task-specific evidence selection or grounding followed by metric or spatial expert processing.
Loading 2606.20515v3…