PaperlayerView exact revision on arXiv

Source-linked AI summary

AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models

Shouwei Ruan, Bin Wang, Zhenyu Wu, Qihui Zhu, Yuxiang Zhang, Jingzhi Li, Yubin Wang, Xingxing Wei

arXiv:2606.08952v1cs.AI

TL;DR

Multimodal foundation models remain fragile at physical-world spatial reasoning because they lack persistent, global allocentric representations of partial observations. AlloSpatial builds such representations and guides their use through an agentic reasoning harness, improving proprietary models by approximately 5%-18% and enabling trained agents to outperform larger baselines.

  • Problem

    Multimodal foundation models lack persistent, queryable allocentric representations that transform partial egocentric observations into reliable global spatial reasoning.

  • Method

    World2Mind converts egocentric observations into allocentric maps, including Allocentric-Spatial Trees that preserve object semantics and explicit spatial structure.

  • Results

    5%-18% overall gains improve frontier commercial models training-free, while trained AlloSpatial agents surpass larger frontier models and competitive methods across multiple tasks.

  • Takeaways & Limitations

    Allocentric priors can support strong 3D spatial reasoning even without visual inputs and help compact agents outperform larger general-purpose and spatial baselines.

  • Takeaways & Limitations

    Reconstruction drift and imperfect metric calibration limit precise distance, size, and counting estimates.

Abstract

from arXiv · show

Multimodal Foundation Models (MFMs) have made substantial progress, yet remain fragile in spatial reasoning over the physical world. A key bottleneck lies in their inability to transform local egocentric observations into a global allocentric spatial representation. To address this, we propose AlloSpatial, an agentic framework for allocentric spatial cognition in foundation models. AlloSpatial introduces World2Mind, a plug-and-play cognitive mapping sandbox that converts egocentric observations into structured allocentric priors, including Allocentric-Spatial Trees and route maps that support querying object topology, geometric relations, passability, and trajectories. To utilize these priors reliably under noisy reconstruction and ambiguous visual evidence, AlloSpatial introduces a Spatial Reasoning Harness for tool-use judgment, modality-decoupled cue collection, and geometry-semantic arbitration. We further internalize this process in Qwen3-VL through cold-start reinforcement learning with a harness-gated trajectory-level reward. Experiments on VSI-Bench and MindCube show that AlloSpatial improves proprietary models by 5%-18% in a training-free setting, while ASTs alone support strong spatial reasoning even when visual inputs are removed. The trained AlloSpatial agents further outperform larger general-purpose models and competitive spatial baselines, suggesting that structured allocentric representations, active tool use, and verifiable reasoning offer a promising route toward spatially capable foundation models.

1 Introduction

AlloSpatial addresses fragile physical-world spatial reasoning by converting egocentric observations into persistent allocentric maps and regulating tool-mediated evidence use. Its training-free framework improves frontier models by approximately 5%-18%, while ASTs retain strong 3D reasoning when visual inputs are removed.

  • Motivation: MFMs remain fragile in physical-world 3D reasoning because they lack persistent, global, and queryable representations bridging partial egocentric observations.The introduction identifies the egocentric-to-allocentric transformation as a central limitation.
  • World2Mind: World2Mind transforms egocentric videos or images into queryable allocentric spatial priors using semantic-geometry alignment and complementary cognitive maps.It constructs a sparse semantic point cloud before distilling it into landmark- and object-oriented cognitive maps.
  • Allocentric-Spatial Trees: AST compresses noisy 3D reconstruction into compact structured spatial memory without discarding object identity or metric geometry.This addresses limitations of grid maps that discard object identity and semantic graphs that omit metric geometry.
  • Spatial Reasoning Harness: The Spatial Reasoning Harness regulates tool invocation, modality-decoupled evidence collection, and cross-modal arbitration when reconstruction drift or perception errors undermine spatial priors.It first judges whether cognitive mapping is needed, then coordinates evidence from visual inputs and AST-structured text.
  • Evaluation: Approximately 5%-18% overall gains are achieved on VSI-Bench and MindCube when World2Mind and the harness are added to GPT-5.2, Claude-4.6, and Gemini-3.These improvements occur in a training-free plug-in setting across egocentric videos and sparse multi-view images.
  • Evaluation: ASTs alone achieve strong 3D reasoning performance when visual inputs are removed, indicating that structured allocentric priors can support reasoning independently of raw visual evidence.This result comes from the introduction’s reported “blind” evaluation setting.

2 Related Work: Spatial Reasoning in Foundation Models

Recent benchmarks identify spatial reasoning as a persistent MFM weakness tied to missing reasoning-ready spatial representations, motivating vision-centric, geometry-centric, and tool-augmented approaches. These methods respectively use 3D-grounded supervision, explicit spatial signals, and external tools for active evidence acquisition.

  • Benchmarks: VSI-Bench and VSI-Super evaluate egocentric video-based spatial intelligence, while MindCube probes sparse multi-view cognitive mapping.Other benchmarks extend evaluation to broader text-based and multimodal scenarios, collectively indicating that the bottleneck extends beyond visual recognition.
  • Vision-centric learning: Vision-centric methods post-train MFMs with large-scale 3D-grounded supervision from real-world images, videos, or 3D annotations.SpatialReasoner and Spatial-MLLM additionally introduce intermediate 3D representations or explicit reasoning traces to structure spatial inference.
  • Geometry-centric learning: Geometry-centric methods inject explicit spatial signals, including RGB-D inputs, depth maps, multi-view observations, depth-encoded features, and native 3D grounding.SpatialBot, MM-Spatial, SD-VLM, and N3D-VLM study these signals for improving geometric grounding and 3D reasoning.
  • Tool-augmented learning: Tool-augmented methods equip MFMs with external modules for active evidence acquisition through 3D reconstruction, pose estimation, rendering, and coordinated depth or segmentation tools.Think3D, pySpatial, SpaceTools, and SpatialDreamer exemplify interactive exploration, reinforcement-learned tool coordination, and intermediate visual evidence.

3 Methodology

AlloSpatial formulates spatial reasoning as a harness-guided, tool-using agent that converts egocentric observations into structured allocentric maps. Its Spatial Reasoning Harness selectively gathers and arbitrates visual, textual, and rendered evidence before producing a verifiable answer.

  • Agent formulation: AlloSpatial models reasoning as a multi-turn trajectory where a foundation-model policy uses World2Mind and the Spatial Reasoning Harness to answer spatial questions.The agent emits reasoning steps or structured tool calls, receives allocentric observations for valid calls, and returns the final answer in a predefined tag.
  • World2Mind mapping: World2Mind provides query-conditioned allocentric knowledge from egocentric sequences using requested categories, knowledge types, footprint formats, and scene types.The mapping pipeline estimates depth and camera poses, extracts semantic masks, filters unreliable geometry, and aggregates retained points into a global semantic point cloud.
  • Allocentric representations: World2Mind represents landmarks with Allocentric-Spatial Trees that preserve object identity, metric geometry, and containment or support relations.Adaptive DBSCAN separates queried object instances, while elliptical or rectangular footprints encode coarse occupancy or nearest-boundary geometry; ASTs are serialized into YAML.
  • Allocentric representations: Route maps voxelize traversable points into top-down grids labeled traversable, occupied, or unknown, while projected camera trajectories encode observed motion history.World2Mind can also return top-down AST layouts, route maps, and semantic segmentation maps as auxiliary global observations.
  • Spatial Reasoning Harness: The Spatial Reasoning Harness cyclically decides when to invoke tools, collects modality-decoupled cues, and arbitrates conflicts before refining the query or answering.It treats cognitive-map outputs as falsifiable evidence rather than ground truth, reducing modality lock-in and over-trust in noisy reconstructions.

4 Experiments

Experiments show that AlloSpatial improves proprietary models in training-free evaluations and that trained agents achieve strong performance with compact allocentric representations. Gains are largest on relational and viewpoint-dependent tasks, while fine-grained numerical estimation remains less consistent.

  • Training-free evaluation: AlloSpatial improves GPT-5.2, Claude-4.6-Opus, and Gemini-3-Pro by +7.3, +17.7, and +5.8 overall points on VSI-Bench, respectively.On MindCube, the corresponding gains are +4.7, +14.4, and +6.5 points.
  • Training-free evaluation: AST text substantially improves blind VSI-Bench spatial reasoning over text-only baselines, especially for object size and route planning.The result supports ASTs as compact allocentric priors that enable reasoning without visual inputs.
  • Trained-agent evaluation: AlloSpatial-8B achieves the best overall VSI-Bench score, while AlloSpatial-4B ranks second and both outperform larger proprietary and open-source model baselines.AlloSpatial-4B reaches 69.1% overall accuracy on MindCube, outperforming Gemini-2.5-Pro, Think3D-4B, Spatial-MLLM-4B, and Cambrian-S-3B.
  • Task-level analysis: AlloSpatial’s gains are strongest on relational and viewpoint-dependent tasks, whereas improvements on absolute distance and object counting are less uniform.ASTs and route maps provide robust coarse allocentric structure, but reconstruction limits fine-grained metric accuracy.
  • Observation-budget analysis: With 0 frames, AlloSpatial improves over Qwen3-VL by +18.2 points and remains stronger than Qwen3-VL and GPT-5.2 with only 3 and 7 frames.This indicates the clearest advantage occurs under sparse visual observation budgets.
  • Ablations: AlloSpatial-4B reaches 53.5 on VSI-Bench and 69.1 on MindCube with only average 358 tokens, while Qwen3-VL-4B thinking reaches 45.5 and 36.1 with 1064 tokens.QA-only training improves MindCube from 28.3 to 53.9 but decreases VSI-Bench from 45.1 to 43.1.

5 Conclusion & Limitations

AlloSpatial equips multimodal foundation models with allocentric spatial reasoning through World2Mind and a spatial reasoning harness. Experiments show training-free gains and compact open-weight agents outperforming larger baselines, while numerical reasoning remains limited by reconstruction drift and imperfect metric calibration.

  • Contributions: AlloSpatial equips multimodal foundation models with allocentric spatial reasoning through the World2Mind cognitive mapping sandbox and spatial reasoning harness.These are the framework’s proposed components for enabling allocentric spatial reasoning.
  • Empirical results: AlloSpatial improves proprietary models in a training-free setting on VSI-Bench and MindCube.The passage reports this result across both named benchmarks without specifying numeric gains.
  • Empirical results: Compact open-weight agents outperform larger general-purpose and spatially specialized baselines.This result is reported for AlloSpatial agents evaluated on VSI-Bench and MindCube.
  • Limitations: Numerical reasoning remains limited because reconstruction drift and imperfect metric calibration can impair precise distance, size, and counting.World2Mind provides robust allocentric structure, but these reconstruction and calibration issues constrain numerical precision.

A Training & Evaluation Configuration

AlloSpatial training uses GSPO on cold-start checkpoints with a shared 59,981-example pool, while evaluation applies controlled decoding, frame budgets, and online World2Mind tool execution. Reported checkpoints are selected by validation performance and inference efficiency rather than a fixed step limit.

  • Training configuration: AlloSpatial-4B and AlloSpatial-8B start from three-epoch supervised cold-start checkpoints and are optimized with GSPO using ms-swift and DeepSpeed ZeRO-2.The shared training pool contains 49,981 VSI-style examples and 10,000 MindCube examples, with 1,442 held-out validation examples.
  • Training configuration: Both runs freeze vision and aligner modules, use cosine decay with 0.005 warmup, set ϵhigh = 0.28, and filter overlong completions.Training is epoch-based, and reported checkpoints are selected by validation performance and inference efficiency.
  • Evaluation configuration: Evaluation parses predefined answer tags with temperature 1.0, permits 8 reasoning turns, and caps completions at 8,192 new tokens.The main post-trained VSI-Bench evaluation uses 7 uniformly sampled frames; the frame-budget ablation uses 0, 3, 7, 15, and 24 frames.
  • Evaluation configuration: World2Mind runs a shared cognitive-mapping pipeline, and generated tool calls execute online before returned ASTs, route maps, or visualizations enter subsequent harness-guided reasoning.The pipeline includes monocular depth and pose estimation, SAM3 segmentation, confidence filtering, semantic point-cloud construction, AST serialization, and optional top-down rendering.

B World2Mind Service Parallelization

World2Mind is deployed as a multi-process HTTP service to handle concurrent rollout and evaluation requests without serializing reconstruction and mapping. Worker-level resource controls and load balancing support efficient, stable parallel execution while preserving the mapping pipeline and AST semantics.

  • Service Deployment: World2Mind runs as a multi-process HTTP service, assigning each process to an independent NPU worker with the same cognitive mapping pipeline.The pipeline includes monocular geometry estimation, SAM3 segmentation, semantic point-cloud construction, AST generation, and route-map rendering.
  • Resource Control: NPU-intensive depth estimation and segmentation use worker-level locks, while downstream mapping and map generation are governed by CPU-side concurrency limits.This prevents NPU-memory oversubscription while allowing lightweight mapping operations to proceed concurrently.
  • Request Routing: Clients load-balance cognitive_map requests across available workers, skip failed endpoints, and retry requests when possible.The design improves throughput and tail-latency stability under concurrent rollout generation while preserving World2Mind reconstruction and AST semantics.

C Spatial Reasoning Harness Prompts

The Spatial Reasoning Harness combines structured tool interfaces with a staged protocol that separates visual observations, map evidence, visualization evidence, and final cross-validation. Its tools provide allocentric landmark and route information while requiring models to treat reconstructions as supplementary evidence and verify ambiguity against observations.

  • Harness components: The harness couples a system message, machine-readable tool interface, and user-side reasoning protocol, with function-calling schemas for proprietary models and serialized <tool_call> blocks for local agents.Visual tokens are placed before the reasoning protocol so models observe frames or multi-view images first.
  • Tools: world2mind builds query-conditioned allocentric maps containing landmark coordinates, sizes, object relations, traversable regions, camera trajectories, and grid-based routes.Its outputs support landmark knowledge, route knowledge, or both, with scene-specific reconstruction settings and selectable footprint formats.
  • Reliability and answer format: Tool outputs are supplementary rather than ground truth because monocular reconstruction may produce missing objects, ghost instances, coordinate drift, or segmentation errors.The final answer integrates validated evidence and uses <Answer></Answer> tags, placing only the final value inside for single-word, numeric, or option questions.
  • User-Side Reasoning Protocol: When structured maps are ambiguous, view_image renders available visualizations such as landmark, route, and top-down point-cloud views for layout verification.The protocol then compares visual, map, and visualization evidence, identifying reconstruction drift, missing objects, false detections, and visual ambiguity before answering.
  • User-Side Reasoning Protocol: The protocol requires visual clues before tool use, invokes world2mind only for information vision cannot reliably provide, and records map clues separately before reconciliation.Targeted uses include metric distance, 3D coordinates, route layout, viewpoint transformation, and complex spatial relations.

D Case Analysis · D.1 Case 1: Metric Closest-Point Reasoning

The case demonstrates how AlloSpatial combines egocentric visual evidence with a World2Mind landmark AST to estimate metric nearest-point distance. The agent predicts 0.99 m for the towel–refrigerator separation versus a 1.1 m ground truth, with 0.11 m absolute error.

  • D.1 Case 1: Metric Closest-Point Reasoning: Eight sampled kitchen frames show the towel and refrigerator, but they are not visible together in a calibrated view.The model therefore considers direct visual inspection insufficient for precise metric distance.
  • D.1 Case 1: Metric Closest-Point Reasoning: 0.99 m is the prediction against 1.1 m ground truth, yielding 0.11 m absolute error.The final response is formatted as a concise metric answer.
  • D.1 Case 1: Metric Closest-Point Reasoning: World2Mind constructs a landmark AST in a 2D absolute grid measured in meters, representing scene objects with rectangles.The query includes towel, refrigerator, stove, and oven as landmark categories, with the floor marked traversable.
  • D.1 Case 1: Metric Closest-Point Reasoning: 1.671 m is the center-to-center towel–refrigerator distance, but the agent instead computes the nearest-boundary distance from rectangle half-extents.The horizontal and vertical gaps are 0.956 m and 0.256 m, respectively.
  • D.1 Case 1: Metric Closest-Point Reasoning: 0.990 m is the AST-derived nearest-boundary distance between the towel and refrigerator.The trajectory requests a landmark visualization to verify the geometric layout before finalizing the answer.
  • D.1 Case 1: Metric Closest-Point Reasoning: The top-down visualization places the refrigerator and stove/towel on opposite kitchen sides, with a gap consistent with the rectangle-derived estimate.The map does not visibly contradict the raw visual evidence, so the agent trusts the AST estimate.

D.2 Case 2: Viewpoint-Conditioned Navigation

In this viewpoint-conditioned navigation case, the model combines four wall-aligned chair views with an allocentric map to resolve whether turning left and moving forward approaches the grey sofa. Visual and map evidence agree, yielding the correct answer: yes.

  • Case setup: Four wall-aligned views show the same black chair from front, left, back, and right perspectives, and the query asks whether a left turn followed by forward movement approaches the grey sofa.
  • Visual clues: Image 3 visibly contains the grey sofa, while image 4 is the chair’s right-side view; the resulting view transformation is ambiguous enough to benefit from an allocentric map.
  • Map clues: The map places the sofa at (1.524, 1.234), the chair at (−0.077, 1.314), and image 4 at (−0.170, 1.443) with heading 161.8◦.
  • Cross-validation: Turning left changes the heading to approximately 71.8◦ toward increasing x, so moving forward should reduce distance to the sofa; visual and allocentric evidence agree.
  • Final answer: Prediction B matches the MindCube ground truth: B (Yes).

E Computational Costs in Training and Inference

AlloSpatial training uses substantial NPU resources, while inference cost depends mainly on input frames, World2Mind calls, and multi-turn reasoning. Parallelization enables VSI-Bench-tiny evaluation in roughly 12 minutes for 392 questions.

  • Training cost: AlloSpatial-4B and AlloSpatial-8B require approximately 60 and 40 NPU-hours, respectively, on 8 HUAWEI Ascend 910B NPUs.The models use 4 and 6 trainer processes, respectively.
  • Training cost: A GSPO update typically takes 4–6 minutes, including rollout generation, live World2Mind execution, reward computation, and policy optimization.The exact update time varies with the tool.
  • Inference cost: Inference cost is determined primarily by the number of input frames and World2Mind calls, with valid calls potentially invoking several perception and mapping operations.These operations include depth and pose estimation, segmentation, semantic alignment, AST construction, and optional route-map rendering.
  • Inference cost: Roughly 12 minutes suffices to evaluate 392 VSI-Bench-tiny questions with concurrent evaluation and 8 World2Mind workers.Each query may involve multiple reasoning turns.

NeurIPS Paper Checklist

The checklist affirms that the paper’s abstract and introduction accurately state its contributions and scope, while documenting limitations, reproducibility information, and the absence of formal theoretical results. It also includes checklist guidance on experimental statistical significance and open access to data and code.

  • Claims: The abstract and introduction accurately present AlloSpatial, World2Mind, the Spatial Reasoning Harness, and RL-based internalization into open-weight agents.These claims are supported by evaluations on VSI-Bench and MindCube, including training-free proprietary-model results, trained Qwen3-VL agents, ablations, and limitations.
  • Limitations: The paper discusses limitations including fine-grained numerical spatial-reasoning weakness, reconstruction drift, imperfect metric calibration, computational costs, and online World2Mind constraints.These limitations appear in the Conclusion, Limitations section, and appendix.
  • Theoretical analysis: The paper marks the theoretical-results question [N/A] because it presents no theoretical theorems or formal proofs.The checklist guidelines separately describe assumptions, numbering, cross-references, and proof placement for papers containing theoretical results.
  • Reproducibility: The reproducibility justification reports benchmarks, data splits, metrics, frame sampling, baselines, training stages, reward design, evaluation settings, prompts, and World2Mind configuration.Additional hyperparameters, computational costs, and case analyses are provided in the appendix.
  • Open access and statistical significance: The checklist includes questions on open access to data and code and on suitably reporting error bars, confidence intervals, or statistical significance tests.The guidance requires variability factors captured by error bars or related statistical information to be clearly stated.
Loading 2606.08952v1…