PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?

Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Wu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li

arXiv:2610.26659v2cs.CVcs.CL

TL;DR

Spatial QA offers limited evidence of whether models can autonomously gather evidence, update spatial representations, and use them to guide actions. SpatialScout evaluates agents that explore from an object inventory and observations, then construct target-center maps in a shared coordinate frame. The strongest evaluated model attains a mean IoU of 41.71%, with only 3.25% of targets exceeding an IoU of 0.9.

  • Problem

    Spatial QA offers limited evidence of whether models can autonomously gather evidence, update spatial representations, and use them to guide actions.

    Source: §1 · p. 1

  • Method

    SpatialScout evaluates agents that explore from an object inventory and observations, then construct target-center maps in a shared coordinate frame.

    Source: §1 · p. 2

  • Results

    The strongest evaluated model attains a mean IoU of 41.71%, with only 3.25% of targets exceeding an IoU of 0.9.

    Source: §7 · p. 9

  • Takeaways & Limitations

    SpatialScout makes active construction of continuous 3D object maps a direct evaluation objective beyond spatial QA.

    Source: §1 · p. 2

    The benchmark currently uses physics-simulated data, which may not fully capture the complexity and variability of real-world scenes.

    Source: C. Limitations · p. 14

Abstract

from arXiv · show

Spatial intelligence requires agents to actively acquire visual evidence and integrate observations into a coherent representation of the three-dimensional world. Yet spatial question answering offers limited evidence of whether models can construct accurate metric maps through autonomous exploration. We introduce SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping. SpatialScout comprises 80 diverse indoor scenes containing 3,140 target objects. Given an object inventory, RGB observations, camera calibration, and camera poses, agents control a camera to explore each scene and estimate object centers in a shared coordinate frame. The task requires coordinating viewpoint selection, cross-view association, geometric estimation, and map revision under bounded interaction budgets. Evaluation jointly measures localization accuracy and target coverage through object-scale overlap scores that penalize missing predictions. Experiments with frontier VLMs reveal substantial limitations: the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with 47.23% of targets exceeding an intersection-over-union (IoU) threshold of 0.5 and only 3.25% exceeding 0.9. These results reveal a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps. SpatialScout provides a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception. Code is available at https://github.com/JesseZZZZZ/SpatialScout.

1. Introduction

Spatial question answering provides limited evidence about whether VLMs can autonomously gather visual evidence and build metric 3D representations. SpatialScout makes active object-level mapping a direct evaluation task, and experiments show that even frontier models struggle to achieve both comprehensive coverage and precise localization.

1. Introduction

  • Existing image- and video-based spatial QA offers limited evidence of spatial understanding because correct answers can arise from guessing, elimination, or non-spatial cues.Real-world demands also include autonomously gathering evidence, updating spatial representations, and using them to guide subsequent actions.Source: §1 · p. 1
  • SpatialScout evaluates whether agents can build object maps in continuous 3D space through active interaction, recovering target positions in a common coordinate frame.The task couples active perception, cross-view association, geometric estimation, and scene modeling: estimates shape what to observe next, while viewpoints shape later evidence.Source: §1 · p. 2
  • SpatialScout requires agents to start from an observation and target inventory, explore, and progressively map target object centers in a common coordinate frame.Its evaluation measures localization accuracy and target coverage, while long-horizon performance reflects coordination across multiple interaction steps.Source: §1 · p. 2
  • The benchmark provides camera-control and geometric assistance tools, camera intrinsics and poses, and records map updates under a limited interaction budget.It contains 80 indoor scenes and 3,140 target objects.Source: §1 · p. 2
  • Frontier VLMs recover parts of object layouts through exploration, but comprehensive coverage and precise localization remain challenging.Models differ in target omissions and center accuracy, and few meet stringent scale-relative accuracy criteria.Source: §1 · p. 2

2. SpatialScout Construction

SpatialScout combines a shared agent interface with object-level evaluation of active object-center mapping. It defines target centers in a fixed task frame and constructs a varied indoor-scene catalog with controlled target selection.

2.1. Task Definition

  • Agents use a shared interface to explore from RGB observations, associate targets, and estimate object centers; evaluation measures map accuracy and completeness.The reference center is the midpoint of the tight world-axis-aligned bounding box, expressed in the task frame; for articulated objects, the box encloses rendered link geometries.Source: §2 · pp. 2–3, §2.1 · p. 3

2.2. Benchmark Construction

  • The catalog contains 80 indoor scenes spanning rooms and apartments, with 30–40 targets per scene and varied spatial extent and repeated categories.The passage reports 3,410 instances, including 1,489 targets (43.3%) in same-scene equivalence groups with multiple instances.Source: §2.2 · p. 3
  • Scene preparation prioritizes candidates with known semantic categories and proximity to the initial camera, while hiding unselected candidates and retaining selected objects’ source placements.Each catalog entry fixes the source configuration, initial pose, and target count, and preparation saves the inventory, geometric annotations, and initial view.Source: §2.2 · pp. 3–4

3. Evaluation

The protocol standardizes multi-model exploration and mapping with shared tools, calibrated views, and bounded interaction budgets, then scores localization and coverage using object-scale overlap.

3.1. Evaluation Protocol

  • Seven models each complete one episode per model–scene pair across 80 scenes, using shared tools to inspect, submit, and revise object centers.Source: §3.1 · p. 4
  • Agents acquire explicit RGB captures while moving the camera; correspondences use the original image grid and are paired with the original intrinsic matrix.Source: §3.1 · p. 4
  • Triangulation estimates a 3D point from marked correspondences in at least two calibrated frames by minimizing squared distances to viewing lines.Diagnostics flag weak parallax, large residuals, or out-of-range estimates, but low residuals alone do not establish correct object features or centers.Source: §3.1 · p. 4
  • Episodes allow 100 model calls and 100 acquired images; map snapshots and interaction traces support offline replay of center error and coverage over time.Source: §3.1 · pp. 4–5
  • Attempt records retain model configuration, prompts, tool schemas, and budgets; service and runtime failures are logged separately and excluded from aggregate scores.Source: §3.1 · p. 5

3.2. Evaluation Metrics

  • Mean center error uses one-to-one matching within equivalence groups, while scale-sensitive IoU and strict-threshold hit rates are reported over all targets and submitted targets.Omissions count as zero in all-target overlap scores.Source: §3.2 · p. 5, Table 3 · p. 5

4. Results

Claude Opus 5.5 leads the reported overlap comparison, but high-IoU hits remain rare. Qualitative examples distinguish omissions from mislocalization, and tool-use patterns vary across models.

4.1. Quantitative Results

  • Table 3 reports all-target mean IoU of 41.71% for Claude Opus 5.5 versus 33.16% for GPT-6 Astra; submitted-object means are 46.69% and 41.76%, respectively.Source: §4.1 · p. 5
  • At IoU > 0.5, recall is 47.23% versus 35.15%, but at IoU > 0.9 it is 3.25% versus 1.62% for Claude Opus 5.5 and GPT-6 Astra.High-IoU hit rates among submitted objects are also low: 3.64% and 2.09%, respectively.Source: §4.1 · p. 5

4.2. Qualitative Analysis

  • Figure 4 contrasts complete accurate placement, omissions, and mislocalization: Claude submits 35 targets with mean IoU 0.530, while DeepSeek submits 40 with mean IoU 0.034.In the omission example, GPT-6 Astra submits 65 of 80 targets across two scenes.Source: §4.2 · p. 5, Figure 4 · p. 6
  • Across interaction steps, Claude Opus 5.5 makes the most tool calls, while GPT-6 Astra achieves competitive mapping performance with fewer calls.Figure 6 also illustrates tool-call frequency across models.Source: Figure 5 · p. 6, Figure 6 · p. 6

5. Discussions

The discussion distinguishes models by how they gather and use evidence, not only by final localization accuracy. Trajectory analyses expose recurring coverage and geometric-grounding failures, while the cost analysis shows a trade-off between accuracy and inference expense.

Tool-calling Analysis

  • Claude-Opus-5.5 made the most tool calls under a 100-turn trajectory cap, and the authors report that extensive tool use appears to benefit performance by gathering and integrating more scene information.A single turn may include multiple tool calls.Source: §5 · p. 6
  • GPT-6 Astra and Claude-Opus-5.5 balanced observation, movement, and mapping, whereas DeepSeek-V4.1-Flash and Qwen3.8-Max increasingly moved without observing or mapping, adding overhead and context use.The latter models also tended to observe less as interaction progressed, potentially leaving them reliant on earlier views that lacked information about newly encountered objects.Source: §5 · pp. 6–7

Failure Analysis

  • Among 40 trajectories, six met the reconstruction-success criteria, while incomplete coverage was the most common failure, occurring in 26 unsuccessful cases.Success required at least 90% object coverage and mean center error of at most 0.2 m over matched submitted objects.Source: §5 · p. 7, Table 4 · p. 7
  • In 21 unsuccessful cases, submitted positions remained substantially inaccurate despite low triangulation residuals, suggesting that geometric consistency alone cannot resolve incorrect correspondences or object-center selection.Another overlapping 13 cases ended with insufficient coverage despite at least ten response rounds remaining.Source: §5 · p. 7

Trajectory Analysis

  • GPT-6 Astra captured fewer images than DeepSeek-V4.1-Flash but supported triangulation for more submitted objects and produced fewer geometric warnings across 80 shared tasks.The analysis suggests that more effective geometric tool use contributes to GPT-6 Astra’s stronger localization performance.Source: §5 · p. 7, Table 5 · p. 7

Cost Analysis

  • Claude Opus 5.5 achieves the highest mean IoU but costs $28.85 per task; Qwen3.8-Max and Kimi-K3 cost more than GPT-6 Sol and score lower on all IoU metrics.Only DeepSeek-V4.1-Flash among the evaluated open-source models costs less than GPT-6 Sol.Source: Figure 7 · p. 7, §5 · p. 8

6. Related Work

Related work spans scene construction, active reconstruction, spatial benchmarks, memory and geometry tools, and long-horizon agents. SpatialScout differs by evaluating agents’ evidence acquisition and explicit object-center maps in a hidden environment.

Agentic inverse graphics and scene construction

  • Scene-construction benchmarks evaluate reconstruction from supplied images or graphics editing, while SpatialScout emphasizes acquiring new evidence in a hidden environment and revising an explicit object-center map.The cited examples include IR3D-Bench, BlenderGym, BlenderAlchemy, VIGA and Blender-Bench, and SceneActBench.Source: §6 · p. 8

Active reconstruction and next-best-view planning

  • Active-reconstruction systems study view selection and scene mapping, whereas SpatialScout compares general-purpose agents under shared tools and observation and model-call budgets.Examples include SCONE, MACARONS, GenNBV, NARUTO, ActiveSplat, and ActiveGAMER, AIR-Embodied, and AREA3D.Source: §6 · p. 8

Spatial and embodied benchmarks

  • Spatial and embodied benchmarks assess supplied observations, active exploration, or spatial QA; SpatialScout instead scores explicit 3D object-center coordinates maintained over an episode.The passage contrasts it with VSI-Bench and SITE, OpenEQA, ESI-Bench, S3-Bench, and other benchmarks.Source: §6 · p. 8

Memory, geometry tools, and process diagnosis

  • Prior work builds 3D representations, scene memory, and geometric tools; SpatialScout controls image acquisition and model calls separately and supports comparisons of final localization.Examples include ConceptFusion and ConceptGraphs, 3D-Mem, SpaMEM, DUSt3R and VGGT, and NeRF and Gaussian splatting.Source: §6 · p. 8

Long-horizon agentic benchmarks

  • Long-horizon benchmarks test repeated tool use in web, desktop, coding, and other settings; SpatialScout evaluates spatial understanding and long-horizon capability in VLM agents.The cited benchmarks include AgentBench and GAIA, WebShop and WebArena, and SWE-bench and Terminal-Bench [10].Source: §6 · p. 8

7. Conclusion

SpatialScout evaluates VLM agents’ active construction of object-level 3D maps. Frontier models recover partial layouts, but the strongest evaluated model’s overlap results show that precise, complete mapping remains difficult.

7. Conclusion

  • Claude Opus 5.5 attains a mean IoU of 41.71%, while only 3.25% of targets exceed an IoU of 0.9.The conclusion characterizes frontier VLMs as able to recover partial scene layouts but still struggling with comprehensive coverage and precise localization.Source: §7 · p. 9

A. Protocol

The protocol records reproducible scene-preparation details and constrains camera movement, prompts, and interaction budgets. It also tracks attempt configuration and completion or failure status for each episode.

Scene preparation

  • Scene preparation records source identity, seed, render and backend settings, dependencies, and SHA-256 fingerprints, and reuses cached preparations only when configuration fingerprints, initial poses, and target counts match.The distribution is shown in Fig. 8 and Fig. 9.Source: A. Protocol · p. 12, Figure 9 · p. 12

Movement

  • Movement bounds world coordinates to ±15 m, limits relative translations to 1 m, clamps final pitch to [−85°, 85°], and keeps roll fixed.Yaw and pitch increments are limited to 180° per axis.Source: A. Protocol · p. 12

Prompts and budgets

  • Prompts specify model-call and capture budgets, fixed task axes, and the world-AABB center convention; actions in one response are set before visual feedback arrives.Budget reminders appear with 10 and 3 calls remaining, and rejected capture requests do not consume additional images.Source: A. Protocol · p. 12

Attempt accounting

  • Episode accounting records model and API configuration, prompts, tool schema, and actual budgets, while retaining reruns separately and reporting completed and failed episode counts.Completed episodes are labeled by explicit finish, a response without tool calls, or turn-budget exhaustion.Source: A. Protocol · p. 12

B. Reproducibility Details

Multi-view point estimation uses marked image pixels and camera geometry in the fixed task frame. Queries are accepted only after geometric validity checks, with caution flags for weak or inconsistent solutions.

B.1. Multi-view Point Estimation

  • Camera axes are mapped into the fixed task frame to form projections and normal equations for multi-view point estimation.The camera convention is forward/left/up, and the estimate uses task-frame origins and projections.Source: §B.1 · p. 13
  • Valid queries require at least two observations, camera separation of at least 5 cm, a condition number no greater than 10^6, and forward-facing contributing rays.Queries that fail any check are rejected.Source: §B.1 · p. 13
  • Accepted solutions carry a caution when the maximum ray angle is below 3°, the residual exceeds 0.1 m, or the point lies beyond r + 0.5 m.The tool reports the root-mean-square perpendicular line residual and maximum pairwise ray angle.Source: §B.1 · p. 13

B.2. Scoring and Trace Replay

Scoring combines groupwise matching, scale-relative overlap, and explicit omission penalties, while replay records how maps change across submissions and acquired images.

B.2. Scoring and Trace Replay

  • Groupwise minimum-total-Euclidean-distance assignment underlies position metrics, while overlap scoring uses threshold-specific one-to-one matching that prioritizes strict hits.The evaluator canonicalizes ordering to make tied assignments reproducible.Source: §B.2 · p. 13
  • Scene-macro summaries weight scenes equally, while object-micro summaries aggregate across submitted objects or all targets; empty maps receive zero overlap scores.Submitted-object center errors are undefined for empty maps and excluded from their averages.Source: §B.2 · p. 13
  • The full-scene localization loss decomposes into coverage-weighted mean center error plus an omission penalty, with the default penalty set to 5 m per omitted target.Coverage and center error are also reported separately, and the overlap metrics are independent of the penalty λ.Source: §B.2 · p. 13
  • At IoU > 0.9, permitted displacement is below ℓ/19, approximately 5.26% of that axis length; at the 1 cm minimum side, tolerance is approximately 0.526 mm.Center error complements scale-relative overlap by measuring displacement in meters.Source: §B.2 · p. 13
  • Inventory-only origin and uniform-ball policies yield scene-macro full-scene losses of 3.113 m and 4.701 m, respectively, across the same 80 evaluation scenes.These are RGB-free localization baselines using the public inventory and range prior.Source: §B.2 · pp. 13–14
  • Map-update replay scores the full accepted map after every successful submission and tracks center error and coverage as model calls and acquired images accumulate.Replay can vary omission penalties without further model inference; reference-based scores are not returned to the agent.Source: §B.2 · p. 14

C. Limitations

SpatialScout currently uses data generated with a physics simulation engine. The authors note that simulated environments may not fully capture real-world scene complexity and variability.

C. Limitations

  • SpatialScout relies on physics-simulation data, which may not fully capture the complexity and variability of real-world scenes.The authors plan to release a real-world-scene dataset within the next month.Source: C. Limitations · p. 14

Related papers

Loading 2610.26659v2…