PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialASM: Internalizing Active Mentalization for Spatial Reasoning

Yuhong Liu, Shenxi Wu, Junyi Chi, Beichen Zhang, Xuanlang Dai, Jiaqi Wang, Yuhang Zang, Dahua Lin

arXiv:2610.13832v1cs.CVcs.AI

TL;DR

Current LVLMs struggle with complex spatial reasoning that calls for alternative viewpoints and imagined observations, while existing spatial QA data often lacks trajectories teaching those skills. SpatialASM uses training-time spatial simulation to teach view planning and imagination; across seven benchmarks it improves average accuracy on four backbones while retaining general-VQA performance within 0.6 points of baseline.

  • Problem

    Current LVLMs struggle with viewpoint transformation, cross-view correspondence, and unobserved regions, while spatial QA datasets often provide answers without trajectories for planning viewpoints and imagining observations.

    Source: §1 · p. 1

  • Method

    SpatialASM uses an allocentric novel-view generator during training to ground planned viewpoints, producing explicit ASM trajectories for SFT and perspective-taking tasks for RL.

    Source: §1 · p. 2

  • Results

    Across four backbones, SpatialASM improves average accuracy by +1.2 to +3.5 points over seven spatial benchmarks, while general-VQA performance remains within 0.6 points of baselines.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    SpatialASM couples viewpoint planning and imagination in question-conditioned reasoning trajectories that can be internalized through standard post-training without architectural modification.

    Source: §2 · p. 3

Abstract

from arXiv · show

When solving complex spatial reasoning tasks, ordinary humans naturally revisit the scenes from task-relevant viewpoints and mentally reconstruct the spatial layouts from those viewpoints. But can Large Vision-Language Models (LVLMs) proactively think in the same way? We term this ability Active Spatial Mentalization (ASM), an intrinsic component of visual intelligence that remains substantially underdeveloped in current LVLMs. To bridge this gap, we introduce SpatialASM, a scalable framework to internalize ASM into LVLMs through training-time spatial simulation. SpatialASM employs an allocentric novel-view generator to ground hypothetical viewpoints in visual evidence and produces two complementary forms of supervision: i) explicit ASM trajectories that teach LVLMs when and how to adaptively invoke view planning and imagination during spatial reasoning, and ii) automatically-constructed perspective-taking tasks that further strengthens viewpoint-conditioned scene imagination. On seven benchmarks spanning relatively perception-oriented spatial understanding to more challenging tasks requiring multi-image, cross-view and viewpoint-transformation reasoning, SpatialASM yields performance gain on LVLMs of four LVLM backbones in spatial reasoning while retaining their original general vision capabilities. The experimental results demonstrate that our approach improves average accuracy across the seven spatial benchmarks. And the synthesis of ASM trajectories provides a scalable pathway to alleviate the scarcity of spatial Chain-of-Thought (CoT) supervision for eliciting active viewpoint exploration and mental scene simulation.

1. Introduction

Current LVLMs struggle with complex spatial reasoning that requires viewpoint transformation, cross-view correspondence, or reasoning about unobserved regions. SpatialASM trains them to plan relevant viewpoints and imagine observations, improving spatial benchmark accuracy while keeping general-VQA performance close to baseline.

1. Introduction

  • Complex spatial tasks require viewpoint transformation, cross-view correspondence, and reasoning about unobserved regions beyond passively extracting relations from images.Existing spatial datasets often provide answer-only QA, while LVLMs can produce incorrect plans and hallucinated scenes when generating ASM trajectories from scratch.Source: §1 · p. 1
  • Active Spatial Mentalization (ASM) combines task-relevant view planning with mentally constructing the spatial layout expected from selected viewpoints.The paper identifies these as complementary skills for reasoning about evidence unavailable in the input images.Source: §1 · p. 1
  • SpatialASM uses an allocentric novel-view generator at training time to ground LVLM-planned viewpoints, then creates SFT trajectories and verifiable perspective-taking tasks for RL.Rendered views are replaced by text captions in the final SFT data, so the model learns to imagine from original images alone.Source: §1 · p. 2
  • Allocentric targets specify location and direction in one step, while GeoP and answer-consistency thresholds filter rendered views for training.This avoids multi-step egocentric rollouts and screens for geometric consistency.Source: §1 · p. 2
  • Across four backbones, SpatialASM gains +1.2 to +3.5 average points over seven benchmarks, including +8.7 on Spatial457-6d and +5.2 on MindCube, while general-VQA stays within 0.6 points.The reported gains are on average accuracy; the task-specific values are per-benchmark gains.Source: §1 · pp. 2–3

2. Related Work

Prior work explores spatial perception, data synthesis, viewpoint planning, and world-model-assisted reasoning, but SpatialASM combines question-conditioned view planning and imagination in explicit trajectories that can be internalized through post-training.

Proactive spatial intelligence

  • SpatialASM couples question-conditioned view planning and imagination in a single reasoning trajectory, operating at the viewpoint and observation level without architectural modification.Related approaches separately address imagination, planning, physical exploration, or geometry-feature retrieval.Source: §2 · p. 3

Spatial data synthesis

  • Spatial data engines scale scene-based supervision, but most provide answer-only labels; VST-R traces derive conclusions from given evidence rather than planning and imagining missing viewpoints.SpatialASM augments existing QA pairs with trajectories linking planning, viewpoint-conditioned imagination, and evidence-based reasoning.Source: §2 · p. 3

World-model-assisted spatial reasoning

  • World-model approaches use inference-time simulators or training-time renderers, while SpatialASM distills question-conditioned planning and imagination as explicit reasoning trajectories.Prior training-time methods include motion-conditioned association, encoder alignment, or latent-view generation; test-time simulators and internalization may be complementary.Source: §2 · pp. 3–4

3. SpatialASM

SpatialASM uses an allocentric novel-view generator to turn spatial QA data into training-time supervision for active viewpoint planning and imagination. GeoP filters views for geometric consistency, while SFT trajectories and RL tasks train complementary parts of the procedure.

3. SpatialASM

  • SpatialASM combines a training-time simulator, plan-imagine-reason trajectories, and viewpoint-conditioned RL tasks to convert answer-only spatial QA into two complementary forms of supervision.The trajectories supervise the full procedure, while RL tasks strengthen viewpoint-conditioned imagination.Source: §3 · p. 4

3.1. Allocentric Novel-View Generator

  • The generator renders a scene from a directly queried allocentric viewpoint, specified by an anchor image, pixel location, and viewing direction.Training pairs images from the same scene with the ground-truth view determined by their relative camera pose.Source: §3.1 · p. 4

M Trajectory Synthesis

  • GeoP compares 3D reconstructions with and without a rendered view, using their Chamfer distance to flag views that perturb inferred scene geometry.It is a reconstruction-model-dependent proxy, not a ground-truth measure, and is used to filter rendered views against thresholds.Source: M Trajectory Synthesis · p. 5

3.2. ASM Trajectory Synthesis

  • For selected tasks the LVLM plans viewpoints, the generator renders them, and trajectories are retained only when views change an incorrect answer to a correct one and pass the GeoP threshold.Rendered views are then replaced with captions describing their 2D layout and object positions, so supervision teaches imagination from original images rather than view consumption.Source: §3.2 · p. 6

3.3. View-Imagination Reinforcement Learning

  • RL uses two automatically answerable tasks: locating a generated view from a marked image, and selecting a candidate view from a textual target pose.GeoP selects geometrically consistent candidates; the tasks train inverse observation-to-viewpoint and forward pose-to-observation mappings.Source: §3.3 · p. 7

4. Experiments

SpatialASM is evaluated across seven spatial-reasoning benchmarks and four LVLM backbones, with experiments testing explicit viewpoint planning, the full method, and its training components. Results show gains across spatial tasks while preserving general-VQA accuracy close to baseline.

4.2. Main Results

  • Table 1 compares performance on seven spatial-reasoning and three general-VQA benchmarks, reporting average accuracy over all benchmarks within each category.Source: Table 1 · p. 8

4.1. Preliminary Analysis of Explicit ASM

  • External ASM generally improves accuracy over vanilla reasoning, and LVLM-based planning matches or outperforms random planning, but models do not activate this ability spontaneously.Qwen3-VL-8B drops under both planning settings, suggesting smaller models lag in robust planning and long-context multimodal reasoning.Source: §4.1 · pp. 8–9

4.2. Main Results

  • The largest gains occur on challenging spatial-mental-modeling and viewpoint-transformation benchmarks, while perception-centric benchmarks also show modest gains.The 4B model gains +5.2% on MindCube and +5.6% on Spatial457; SpatialASM-66k SFT gains +4.8% on MindCube and +3.0% on ViewSpa.Source: §4.2 · p. 9, §4.3 · p. 10
  • General-VQA accuracy remains within 0.6% of baseline across all four backbones.The authors partly attribute SpatialASM-8B's performance against two 8B spatial-model counterparts to view-imagination RL and its diverse visual observations.Source: §4.2 · p. 9
  • Combining internalized SpatialASM with external ASM yields the highest accuracy on Among (45.0%) and Around (61.5%), although external evidence or planning can degrade baseline performance.SpatialASM-8B improves under vanilla reasoning on both subtasks without noticeable increase in inference duration.Source: §4.2 · pp. 9–10

4.3. Ablation Studies

  • SpatialASM-NVG achieves the highest reasoning accuracy across both backbone LVLMs and MindCube subsets, although SVC has lower GeoP values.The authors attribute the accuracy advantage partly to allocentric viewpoint representation, which makes correspondence between original and generated observations easier to understand.Source: §4.3 · p. 10
  • GeoP-based filtering improves average performance across four spatial-reasoning benchmarks for both SFT and RL.The authors say the filter removes geometrically distorted novel views and prevents unreliable evidence from entering trajectories or RL supervision.Source: §4.3 · p. 10
  • SFT and view-imagination RL have distinct, complementary effects, and combining them achieves the best overall accuracy.SFT strengthens explicit viewpoint transformation and multi-step reasoning, while RL is especially effective on perception-oriented EmbSpa.Source: §4.3 · p. 10

5. Conclusion

SpatialASM identifies Active Spatial Mentalization as a bottleneck in LVLM spatial reasoning and reports gains across spatial benchmarks while preserving general visual capabilities. The conclusion and background passages situate this claim in the limits of current spatial reasoning and distinguish ASM from feature-level geometry retrieval.

5. Conclusion

  • SpatialASM consistently improves spatial reasoning across seven benchmarks and four model architectures while preserving general visual capabilities.The authors present it as a post-training framework that internalizes Active Spatial Mentalization through training-time spatial simulation.Source: §5 · p. 11
  • The authors identify scalable ASM-trajectory synthesis for existing spatial datasets as a way to address scarce intermediate reasoning supervision; future work includes dynamic scenes and embodied interaction.Source: §5 · p. 11

A.1. Proactive Spatial Intelligence

  • Current LVLMs show pronounced performance degradation as spatial reasoning grows more geometrically complex, and often need alternative viewpoints to reason about unobserved regions.Spatial457, MindCube, and MMSI-Bench examine increasingly complex geometry, spatial mental models, and multi-view reasoning, respectively.Source: §A.1 · p. 18
  • Active multiple-setting performance exceeds passive single-setting performance, but remains below ground-truth passive performance, indicating exploration and trajectory planning remain limited.Source: §A.1 · p. 18
  • SpatialASM targets viewpoint- and observation-level reasoning, unlike geometry-enhanced methods that actively retrieve features and add architecture-specific components.The authors describe their data-centric post-training approach as avoiding geometry modules and architectural modifications.Source: §A.1 · p. 18

A.2. Spatial data synthesis

Prior spatial-data pipelines scale supervision through annotated scenes, expert models, and newer synthesis strategies, but often provide answers without showing how to acquire missing evidence. SpatialASM is framed as complementary to tool-assisted evidence gathering and constructs novel-view training examples from public scene data.

A.2. Spatial data synthesis

  • Spatial-data pipelines use human or model-generated annotations, public datasets with geometric labels, and visual experts, while newer work adds simulated views, self-supervised tasks, reconstructed scenes, or broader QA collections.Human annotation is costly, and current model-generated QA may contain errors on challenging spatial tasks.Source: §A.2 · pp. 18–19
  • Most spatial datasets provide answers rather than explicit reasoning traces for proactively acquiring missing evidence in perspective-taking and mental-reconstruction tasks.Source: §A.2 · p. 19
  • SpatialASM synthesizes question-conditioned planning and reasoning traces from existing spatial QA data without requiring ground-truth 3D annotations, unlike VST-R's BEV-based process.VST-R provides 135K traces but requires ground-truth 3D bounding boxes, focuses on reasoning from provided evidence, and is described as not publicly released.Source: §A.2 · p. 19
  • Unlike World2VLM's predefined egocentric camera trajectories, SpatialASM starts from the spatial question and teaches viewpoint selection for missing evidence.Its generated views are temporary scaffolding for long-CoT supervision coupling view planning, imagination, and reasoning with imagined evidence.Source: §A.2 · p. 19

A.3. Tool-Assisted Spatial Reasoning

  • SpatialASM strengthens intrinsic reasoning about needed evidence, complementing tools that improve the availability and precision of external spatial evidence.The paper describes the two paradigms as combinable along complementary axes.Source: §A.3 · p. 20

B.1.1. Statistics; B.1.2. Construction Method

  • SpatialASM-edit140k uses DL3DV scenes to pair input images with a distinct target frame and allocentric viewpoint, projected into the anchor image using camera metadata.The target frame is excluded from the inputs, whose selected frame indices differ by at least 30; the target camera position must project in front of and within the image boundary.Source: §B.1.1 · p. 20, §B.1.2 · p. 20

B.1.3. Training samples

The appendix describes examples and construction protocols for SpatialASM's edit, trajectory, and reinforcement-learning data. The trajectory pipeline converts generated views into supervision, while the RL data are constructed automatically from 2D images.

B.1.3. Training samples

  • Training examples may ask the view generator to imagine plausible objects in regions not visible in the input images.Figure 5 presents single- and multiple-image examples.Source: §B.1.3 · pp. 20–21, Figure 5 · p. 21

B.2. SpatialASM-128k

  • SpatialASM constructs 128K ASM trajectories from existing answer-only spatial datasets and uses 66K for SFT, without requiring metadata such as BEV images or bounding boxes.The authors frame this as a scalable way to provide spatial CoT supervision and elicit task-conditioned planning and imagination.Source: §B.2 · p. 21

B.2.1. Statistics

  • Original spatial-QAs come from Holi-Spatial [26], VST-500K [71], and Mindcube’s training set [60]; to avoid distribution overlap with our final benchmark evaluation, SpatialASM-66k excludes samples synthesized from Mindtube-train.Source: §B.2.1 · p. 21

B.2.2. Synthesis Protocols

  • Trajectory synthesis uses one LVLM for planning and final reasoning, another for captions of generated views, and GeoP filtering alongside reasoning-correctness checks.The planning prompt selects a source image, image area, and direction—or an empty list—before producing a short rationale; the filtering threshold is 2.5.Source: §B.2.2 · pp. 22–23

B.2.3. Data Sample

  • Generated novel views serve as temporary scaffolding that is converted into captions; they do not appear in the final SpatialASM-128k sample.The illustrated example uses one auxiliary imagined view.Source: Figure 6 · p. 22, §B.2.3 · p. 23

B.3. SpatialASM-RL-58k

  • SpatialASM-RL-58k is built automatically from Holi-Spatial and DL3DV images, and its RL data include generated-view localization and pose-conditioned view selection tasks.The dataset has 58,346 samples; the construction applies different GeoP thresholds for backward versus other target directions.Source: §B.3 · pp. 23–24, Figure 7 · p. 24

C.1. Training Details

The training details specify how the novel-view generator is fine-tuned, how view-imagination RL is configured, and the response format prompted for SFT and RL.

C.1. Training Details

  • SpatialASM-NVG is derived by fine-tuning FLUX.2-klein-9B on SpatialASM-edit140k for three epochs with MSE, updating only its DiT through rank-512 LoRA.Mapanything is used as the feed-forward 3D reconstruction model for GeoP calculation.Source: §C.1 · p. 24
  • View-imagination RL uses GRPO with KL weight 1e-2, five responses per group, and a reward weighted 0.9 for answer accuracy and 0.1 for format.Both rewards are binary; format reward requires reasoning followed by the option letter within answer tags.Source: §C.1 · p. 24
  • Table 7 reports performance comparisons on three general-VQA benchmarks.The supplied caption does not provide benchmark names or results.Source: Table 7 · p. 25
  • For both SFT and RL, the system prompt asks models to reason first and place the option letter inside answer tags.Source: §C.1 · p. 25

C.2. Benchmark Overview

The seven spatial benchmarks span perception-oriented questions through viewpoint transformation, multi-view reconstruction, and reasoning about unobserved spaces; three general-VQA benchmarks assess whether spatial gains come at the expense of broad visual understanding.

C.2. Benchmark Overview

  • ESI-Bench, MindCube, MMSI-Bench, and Spatial457-6d are the most difficult benchmarks, requiring integration of incomplete evidence and reasoning about hypothetical viewpoints.Spatial457-6d uses only the benchmark’s most challenging subtask, with 1,837 questions.Source: §C.2 · p. 25
  • ViewSpatial-Bench has intermediate difficulty, combining camera-perspective questions with person-perspective questions that require basic perspective-taking.Source: §C.2 · p. 25
  • 3DSRBench and EmbSpatial-Bench are perception-oriented, while MMMUval, MMStar, and RealWorldQA test broader visual understanding across expert reasoning, visual perception, and real-world scenes.Source: §C.2 · p. 25

C.3. Evaluation Details

Evaluation uses separate prompting for spatial reasoning and general VQA, and includes representative spatial models for comparison. General-VQA results are summarized as a three-benchmark average, with per-benchmark accuracies reported separately.

C.3. Evaluation Details

  • The evaluation covers spatial reasoning and general VQA, using the training system prompt for spatial reasoning and a direct-answer prompt for general VQA.Source: §C.3 · pp. 25–26
  • Four representative spatial models are evaluated; models without official system prompts use the paper’s prompt, while SpaceR-7B and VST-RL-7B use their repository prompts.Source: §C.3 · p. 26

C.4. General VQA Results

  • Tab. 1 reports the average across three general-VQA benchmarks, while Tab. 7 gives each model’s accuracy on each benchmark.Source: §C.4 · p. 26

C.5. Ablation Details

The ablation details specify external view-generation comparisons, the GeoP summary statistic, and filtering conditions for training. Rendering examples illustrate SpatialASM-NVG’s reconstruction of unseen content and integration of multiple views.

C.5. Ablation Details

  • The external ASM paradigm plans question-conditioned viewpoints, synthesizes observations with an NVG, then reasons from those observations; experiments vary both LVLMs and rendering models.The LVLMs are Qwen3-VL-8B, Qwen3.5-35BA3B, and Kimi-k2.5; the rendering models include GPT-Image-2, SpatialEdit, SVC, and SpatialASM-NVG.Source: §C.5 · p. 26
  • Tab. 3 reports median GeoP because a small number of high-GeoP images can disproportionately raise the average and misrepresent typical geometric consistency.Source: §C.5 · p. 26
  • The final SFT+RL setting and Tab. 1 models use both training stages and filtered data; the ablations remove trajectory filtering for SFT or GeoP-based filtering for RL.Source: §C.5 · p. 26

D.1. SpatialASM-NVG Rendering

  • SpatialASM-NVG is trained from FLUX.2-klein-9B on SpatialASM-edit140k, and its rendering examples include newly revealed regions and multi-image scene integration.Source: §D.1 · p. 26
  • The rendering examples show reconstruction of content beyond the input field of view, retention of visible objects, and reproduction of lighting and shadows.Source: §D.1 · p. 26

D.2. Results of Novel-view Generators

The novel-view generator comparisons highlight a trade-off between geometric consistency and informative viewpoint changes. Qualitative examples suggest SpatialASM-8B adapts its reasoning to task difficulty rather than following a fixed ASM template.

D.2. Results of Novel-view Generators

  • In Case 1, SpatialASM-NVG has lower GeoP than the alternative generator when both target viewpoints are approximately the same, with output objects more closely resembling the inputs.Source: §D.2 · p. 27, Figure 9 · p. 28
  • In Case 2, SVC’s lower GeoP accompanies small camera motions and little new information, whereas SpatialASM-NVG’s allocentric formulation yields larger viewpoint changes and more informative observations.The authors suggest these observations could help the LVLM understand camera poses and 3D scene layout.Source: §D.2 · p. 27, Figure 10 · p. 28

D.3. Examples of LVLM Internalization

  • SpatialASM-8B adaptively uses ASM reasoning according to task type and difficulty, and its improved spatial accuracy does not depend on explicitly following ASM in most cases.The authors interpret this as evidence of internalized generalizable reasoning rather than a rigid template.Source: §D.3 · p. 27
  • Examples show correct perspective interpretation without explicit ASM on some tasks, while more challenging tasks elicit active spatial planning and imagination.Source: §D.3 · pp. 27–28, Figure 11 · p. 29, Figure 12 · p. 30
Loading 2610.13832v1…