PaperlayerView exact revision on arXiv

Source-linked AI summary

MindCube: Spatial Mental Modeling from Limited Views

Qineng Wang, Baiqiao Yin, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Jiajun Wu, Li Fei-Fei, Manling Li

arXiv:2506.21458v2cs.AIcs.CLcs.CV

TL;DR

Reasoning about unseen spatial relationships from limited views remains a challenge for VLMs. MindCube evaluates this ability and related scaffolds, finding self-generated cognitive maps the most effective approximation across the tested elicitation methods.

  • Problem

    How well VLMs infer unseen spatial relationships from limited views remains insufficiently evaluated, despite its importance for reasoning in partially observable environments.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces MINDCUBE and tests spatial-reasoning scaffolds, including augmented cognitive maps and reasoning over self-generated maps.

    Source: §1 · p. 2, §3.1 · p. 6, §6 · p. 13

  • Results

    Constructing and reasoning over self-generated cognitive maps was the most effective approximation of spatial mental models across input-output, supervised fine-tuning, and reinforcement-learning elicitation.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The findings support self-generated cognitive maps as a useful scaffold for approximating spatial mental models across the evaluated elicitation methods.

    Source: §6 · p. 13

    The datasets may not fully represent global environments, potentially introducing geographic or cultural biases into models’ spatial understanding.

    Source: Ethics Statement · p. 13

Abstract

from arXiv · show

Can Vision-Language Models (VLMs) imagine the full scene from just a few views, like humans do? Humans form spatial mental models naturally, internal representations of unseen space, to reason about layout, perspective, and motion. Our MindCube benchmark with 21,154 questions across 3,268 images exposes this critical gap, where existing VLMs exhibit near-random performance. Using MindCube, we systematically evaluate how well VLMs build robust spatial mental models through representing positions (cognitive mapping), orientations (perspective-taking), and dynamics (mental simulation for "what-if" movements). We then explore three approaches to help approximate spatial mental models in VLMs, focusing on incorporating unseen intermediate views, natural language reasoning chains, and cognitive maps. The significant improvement comes from a synergistic approach, "map-then-reason", that jointly trains the model to first generate a cognitive map and then reason upon it. By training models to reason over these internal maps, we boosted accuracy from 37.8% to 57.8% (+20.0%). Adding reinforcement learning pushed performance even further to 61.3% (+23.5%). Our key insight is that such scaffolding of spatial mental models, actively constructing and utilizing internal structured spatial representations with flexible reasoning processes, significantly improves understanding of unobservable space.

1. Introduction

The introduction presents spatial mental models as viewpoint-independent representations that let humans reason about unseen space, a capability VLMs struggle to reproduce. It introduces MINDCUBE to evaluate this gap and investigates methods for improving spatial reasoning from limited views.

1. Introduction

  • Humans use spatial mental models to infer unseen layout, while VLMs struggle to maintain spatial consistency across views and reason about objects not directly visible.These models are internal representations that support consistent spatial understanding independent of the current viewpoint.Source: §1 · pp. 1–2
  • MINDCUBE contains 21,154 questions across 3,268 images in 976 multi-view groups, testing hidden spatial relations under varied viewpoint transformations.Its questions cover partial observations, cross-view consistency, what-if mental simulation, perspective-taking, and complex relation reasoning; camera patterns include ROTATION, AROUND, and AMONG.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Across 17 state-of-the-art VLMs, both open-weight and closed-source models perform only marginally better than random guessing on MINDCUBE.Table 1 reports model performance on the benchmark.Source: §1 · p. 3, Table 1 · p. 4
  • View interpolation was unexpectedly unhelpful; free-form reasoning gained +2.7%, direct ground-truth maps changed performance by −5.81%, and actively reasoning with maps gained +3.62%.Map generation remained a bottleneck, with isomorphic rates below 10%; the authors constructed 10,000 reasoning chains and ground-truth maps to train VLMs.Source: §1 · p. 3

2. MINDCUBE Benchmark and Evaluation

MINDCUBE evaluates spatial reasoning from partial observations and changing viewpoints using multi-view orthogonal images and spatial questions. On the benchmark, models show substantial but uneven performance, with no model excelling across categories.

2.1. MINDCUBE Benchmark

  • MINDCUBE pairs multi-view orthogonal images with spatial reasoning questions to evaluate spatial mental modeling under partial observations and dynamic viewpoints.It targets object consistency across views and reasoning about occluded or invisible elements.Source: §2.1 · p. 4
  • Its Rotation, Around, and Among settings test holistic understanding across incremental views, object permanence and depth cues under occlusion, and spatial consistency around a central object.Across settings, models must share information across views to infer arrangements and relations despite limited visibility.Source: §2.1 · p. 5
  • The dataset curation pipeline selects multi-view image groups matching movement patterns and spatial criteria, annotates key spatial information, and algorithmically generates taxonomy-aligned questions with targeted distractors.Source: §2.1 · p. 5

2.2. Evaluation on MINDCUBE

  • DeepSeek-VL2-Small achieves 47.62% accuracy, while GPT-5 reaches 93.33% in Rotation and Mantis-8B (SigLip) 50.99% in Around; no model excels across categories.Proprietary models generally outperform open-source ones, while spatial fine-tuning and multi-image input do not reliably improve spatial reasoning.Source: §2.2 · p. 5

3. Which Scaffolds Best Guide Spatial Mental Modeling?

The section examines whether structured data can scaffold spatial reasoning in frozen VLMs by approximating spatial mental models, with structures targeting integration, transformation, and inference.

3.1. Data Structures as Cognitive Scaffolds for Spatial Mental Models

  • Spatial mental models are schematic, manipulable representations of relative object and viewpoint configurations that support reasoning across fragmented observations and unseen perspectives.They are not intended to be metric-precise maps.Source: §3.1 · p. 5
  • View interpolation inserts intermediate frames between sparse views to support perceptual continuity and imagined transformations such as rotation [28].The figure illustrates inserting one frame in place of the original question images.Source: Figure 3 · p. 6
  • The augmented cognitive map combines a 2D schematic layout with discrete views annotated by position and orientation, extending maps that encode only top-down object positions [29].It is designed to approach the relational consistency of spatial mental models.Source: §3.1 · p. 6
  • Open-ended, step-by-step natural-language reasoning procedurally approximates constructing and querying spatial models, particularly with ambiguous or incomplete observations.This scaffold is less rigid than map-like structures.Source: §3 · p. 6

3. Free Form Reasoning.

Frozen-VLM experiments show that explicit reasoning helps models use spatial cues, while passive structures and interpolated views offer little benefit. Prompted map generation modestly improves accuracy, although generated maps remain inaccurate.

3.2. Experiment Setup

  • Experiments compare input structures with output formats across ten possible configurations, evaluating a representative subset and excluding two map-output settings for frozen VLMs.The excluded Aug-CGMap-Out and Plain-CGMap-Out settings were omitted because models tended to conflate map generation with reasoning.Source: §3.2 · p. 6, Table 2 · p. 7
  • Evaluation measures QA accuracy and cognitive-map quality using Overall Similarity and Isomorphic Rate.Overall Similarity combines directional and facing consistency; Isomorphic Rate measures whether all pairwise object relations match ground truth under optimal alignment.Source: §3.2 · p. 6
  • The experiments use Qwen2.5-VL-3B-Instruct [15] and evaluate on MINDCUBE-TINY, a 1,050-question subset comprising AMONG, AROUND, and ROTATION questions.The subset contains 600 AMONG, 250 AROUND, and 200 ROTATION questions.Source: §3.2 · p. 7

3.3. Do Scaffolds Improve Spatial Mental Modeling Without Training?

  • Interpolated views add only 0.09% over Raw-QA, while providing an augmented cognitive map as input lowers accuracy to 32.00%.The findings suggest that passive visual or spatial structure alone offers little benefit.Source: §3.3 · pp. 7–8, Table 3 · p. 8
  • Free-form reasoning alone or with other settings achieves 41.33% accuracy, compared with 37.81% for Raw-QA.The results indicate that spatial structure alone is insufficient without reasoning to leverage it.Source: §3.3 · pp. 7–8, Table 3 · p. 8
  • Prompting cognitive-map generation raises accuracy from 40.48% with free-form reasoning alone to 41.43%, but generated maps have under 50% similarity to grounded maps.Both map formats follow the requested format well, yet their isomorphism rates are only 0.10% for augmented maps and 7.43% for plain maps; added view-level details likely increase augmented-map errors.Source: §3.3 · pp. 7–8, Table 3 · p. 8

4. Can We Train for the Emergence of Spatial Mental Models via VLMs’ Use of Scaffolds?

Prompting frozen VLMs with external spatial scaffolds yields limited gains, while training them to generate cognitive maps and reason over them improves spatial-task performance; RL adds further gains when initialized from SFT.

4.1. Designing a Robust Experimental Framework

  • Experiments compare cognitive-map (Object-only/Object + Camera) and free-form reasoning scaffolds using Qwen2.5-VL-3B-Instruct on MINDCUBE-TINY, retaining established evaluation metrics and excluding view interpolation because earlier validations showed limited gains.RL variants span two output formats and two initialization choices, yielding six configurations. Grounded maps use the annotated front image as the upward direction, and grounded reasoning chains were manually constructed from image annotations and structured question templates.Source: §4.1 · pp. 8–9

4.3. Can Reinforcement Learning Further Refine Spatial Mental Models?

  • 61.33% accuracy was achieved by RL-Plain-CGMap-FFR-Out initialized from SFT, 3.52% above the best SFT model and 8.85% above the best scratch-RL model.Even RL-FFR initialized from SFT reached 59.14%; the Plain-CGMap variant also had a 35.33% isomorphism rate versus 16.95% for the augmented-map variant.Source: Table 4 · p. 9, §4.3 · p. 11

4.2. Do the Emergence of Spatial Mental Models Truly Benefit from Explicit Training?

  • Plain-CGMap-FFR-Out gained 5.14% over Raw QA-SFT, outperforming map-only and reasoning-only training as joint reasoning encouraged functionally useful, if less structurally perfect, maps.Map-only training rapidly improved graph similarity and isomorphism but its QA accuracy plateaued; the joint model learned maps more slowly while QA accuracy continued rising.Source: Table 4 · p. 9, §4.2 · p. 10
  • Aug-CGMap-Out reached 52.48% accuracy, versus 52.67% for Raw-QA; Plain-CGMap-Out reached 54.29%, and FFR alone reached 55.43%.Although explicit map training improved graph structure quality, neither map generation nor reasoning alone matched the joint approach.Source: Table 4 · p. 9, §4.2 · p. 10

4.3. Can Reinforcement Learning Further Refine Spatial Mental Models?

  • Scratch RL-FFR achieved 49.52% accuracy, while scratch RL-Aug-CGMap-FFR-Out and RL-Plain-CGMap-FFR-Out reached 52.48% and 50.86%, respectively, with weak geometry metrics.The results indicate task-level rewards alone provide insufficient guidance for robust spatial representations, and map formats learned from scratch do not yield meaningful geometry.Source: Table 4 · p. 9, §4.3 · p. 10

4.4. Effect of Object Presentation Order in Cognitive Map Supervision

  • Plain-CGMap-FFR-Out remains the top configuration under fixed or randomized object order, while Plain-CGMap-Out’s isomorphism rate rises from 45.52% to 89.05% under fixed order.The authors use randomized order as the primary setting because it prevents predictable ordering shortcuts and better tests genuine spatial understanding; the core learning trends hold under both settings.Source: Table 5 · p. 11, §4.4 · pp. 11–12

5. Related Works

Prior VLM spatial-understanding work often neglects human-like mental-level reasoning, while multiview inputs improve spatial tasks but still struggle with consistency and 2D-to-3D ambiguity.

Spatial Cognition

  • VLM spatial-cognition benchmarks and enhancement methods often neglect the mental-level spatial reasoning central to human cognition.Prior methods include coordinate-aware prompting [37], CoT reasoning, explicit spatial-representation alignment [23], and RL [41].Source: §5 · p. 12
  • Spatial cognition includes mental rotation, spatial visualization, and object assembly, supported by spatial mental models (SMMs) for consistent understanding of space.Source: §5 · p. 12

Multi Views understanding

  • Multiview methods include reconstruction [48] [50], view synthesis [53], equivariant learning, and topological representations.Frameworks also advance open-vocabulary concept learning from multiview data using neural fields and vision-language fusion.Source: §5 · p. 12
  • Multiview-input LMMs improve geometric understanding and perspective taking [24], but struggle with fragmented reasoning and 2D-to-3D ambiguity.Source: §5 · p. 12

6. Conclusion

MINDCUBE studies how VLMs can approximate spatial mental models from limited views. The conclusion finds that constructing and reasoning over self-generated cognitive maps is the most effective approach across elicitation methods.

6. Conclusion

  • MINDCUBE studies how VLMs can approximate spatial mental models from limited views, a core ability for reasoning in partially observable environments.Source: §6 · p. 12
  • Self-generated cognitive maps that models construct and reason over most effectively approximate spatial mental models across elicitation methods.This approach outperforms view interpolation and externally provided maps across input-output configurations, supervised fine-tuning, and reinforcement learning.Source: §6 · p. 13
  • Initializing reinforcement learning from a well-trained supervised fine-tuning checkpoint further improves spatial reasoning performance.The conclusion describes this initialization as further optimizing the process.Source: §6 · p. 13

Ethics Statement … B. MINDCUBE Benchmark

The authors describe ethical safeguards in MINDCUBE data collection and annotation, while acknowledging limits and risks of spatial-reasoning research. They also disclose LLM assistance that was reviewed by the human authors.

Ethics Statement

  • MINDCUBE combines publicly available anonymized datasets—ArkitScenes, WildRGB-D, and DL3DV-10K—with self-collected imagery.Source: Ethics Statement · p. 13
  • Self-collected indoor and outdoor scenes were captured without personally identifiable information or sensitive content.Source: Ethics Statement · p. 13
  • All human annotators involved in data curation and evaluation were compensated at rates significantly exceeding their local minimum wage.Source: Ethics Statement · p. 13
  • The datasets may not represent the full range of global environments, potentially introducing geographic or cultural biases into models’ spatial understanding.Source: Ethics Statement · p. 13
  • Training, fine-tuning, and evaluation of the large-scale VLMs carry significant computational and environmental costs.Source: Ethics Statement · p. 13
  • The authors recognize that technologies enhancing machine spatial reasoning could have dual-use applications.Source: Ethics Statement · p. 13

A. The Use of Large Language Models

B.1. Details for Data Collection and Annotation … C.5. Analysis in settings

This section describes MINDCUBE’s image selection and annotation, task taxonomy, and evaluation procedures, and examines model performance across task categories and a range of general-purpose, multimodal, proprietary, and spatially fine-tuned models. It also reports that text captions reduce model performance and that human annotators outperform state-of-the-art multimodal models on spatial problems.

B.1. Details for Data Collection and Annotation

  • MINDCUBE comprises 3,268 images selected across four view dynamics, with annotations covering spatial relations, object grouping, semantic orientation, and occlusion.The image sources include publicly released datasets and self-collected images; the text specifies 2,302 publicly sourced images and 400 self-collected images.Source: §B.1 · pp. 20–21

B.2.1. Three kinds of invisibility settings

  • Rotation, around, and among settings require models to integrate partial or occluded views to reason about objects and spatial relationships unavailable from a single viewpoint.Rotation uses 2 to 4 orthogonal views; around emphasizes object permanence and cross-view transformations, while among uses consistency across views to infer unseen positions.Source: §B.2.1 · p. 23

B.2.2. Label taxonomy

  • The label taxonomy distinguishes linear from rectilinear arrangements, four translation-and-rotation dynamics, and three levels of perspective taking.Sequential translation-rotation is represented through question text rather than explicit visual transformations.Source: §B.2.2 · pp. 24–25

B.3. Examples

B.2.2. Label taxonomy

  • Object-Object tasks generally score lower across models, InternVL2-8B has the lowest sequence-task score, and Mantis(SigLip) and Qwen2.5-VL-7B-Instruct show stability in specified categories.InternVL2-8B also shows lower overall scores and greater subcategory fluctuations than the other models discussed.Source: §B.2.2 · p. 25, Table 1 · p. 26, Table 2 · p. 26

C.1. Prompt Templates for Evaluation

  • The evaluation prompt instructs models to select one correct answer from the provided options.The prompt prefix gives an example of responding with only the chosen option.Source: §C.1 · p. 26

C.2. Details in text only evaluation

  • Text-only evaluation replaces image inputs with brief or dense captions and tests answering from those descriptions without access to the images.Dense captions focus on spatial relationships between objects.Source: §C.2 · p. 27

Prompt for Brief Captioning

  • The prompt examples include a dense-caption instruction to describe images in detail with emphasis on object spatial relationships.The supplied examples also show question prompts about movement and object proximity.Source: Prompt for Brief Captioning · p. 28

C.3. Human Evaluation

  • Five annotators independently answered every Tiny Benchmark question, and humans outperformed state-of-the-art multimodal models on challenging spatial problems.The Tiny Benchmark covers all task categories.Source: §C.3 · p. 29

C.4. Evaluation Setup

  • The evaluation suite includes multi-image and interleaved image-text models, Gemma3, proprietary APIs, and models fine-tuned specifically for spatial reasoning.Examples include LLaVA-Onevision, LLaVA-Video, Mantis, GPT-5, Claude-4-Sonnet, RoboBrain, Space-Mantis, SpaceQwen, and Spatial-MLLM.Source: §C.4 · pp. 29–30

Prompt for Brief Captioning

  • Brief captions reduce RoboBrain-8B performance by 7.83% and LLaVA-OneVision-7B performance by 12.91%; dense captions also reduce performance, though less.The comparison evaluates text-only captions against original image inputs across models; captions do not capture the richness of visual data.Source: Prompt for Brief Captioning · p. 28, Table 3 · p. 30

C.5.1. Around … D.2.1. QA Accuracy

The analyses show that MLLM spatial reasoning is sensitive to occlusion and viewpoint, inconsistent across paired tests, and often based on local rather than global relations. The appendix also details cognitive scaffolds for limited-view reasoning and defines QA Accuracy as a core evaluation metric.

C.5.1. Around

  • Accuracy fell 50.7% from fully visible to fully occluded conditions (p <0.01), with a 28.7% drop from mostly visible to mostly occluded.The decline was non-linear, and heavily occluded objects often prompted proximity-based guessing rather than cross-view reasoning.Source: §C.5.1 · p. 30
  • High-angle views yielded 24.8% higher accuracy than eye-level views, especially in dense scenes where elevated perspectives revealed spatial gaps.Eye-level views had more severe occlusions, and models appeared limited in reconstructing hidden scene elements from partial visual information.Source: §C.5.1 · pp. 30–31

C.5.2. Among

  • Paired tests found SpaceQwen inconsistent in around 40% of cases, while LLaVA-Onevision was more balanced and InternVL varied across tests.The tests changed central-object attributes while preserving layout, or increased the number of central objects while retaining peripheral objects; overall, models showed attribute-invariant reasoning but struggled to use added reference objects.Source: §C.5.2 · pp. 33–34, Figure 9 · p. 33

C.6. Failure case analysis

  • The error pattern points to local relationship matching rather than inference of global spatial configurations, motivating transitive inference and view-invariant scene representations.Figure 12 illustrates failures to align spatial locations across views and to identify the subject of a “behind” relation.Source: §C.6 · p. 34, Figure 12 · p. 35

D. Data Structures as Cognitive Scaffolds, Evaluation Metrics, and Input-Output Configurations; D.1. Data Structures as Cognitive Scaffolds

  • The appendix frames its data structures as scaffolds for dynamic updating, integrated spatial layout, and inferential reasoning, and introduces their evaluation metrics and input-output prompts.These structures aim to approximate spatial mental modeling from limited visual input.Source: D. Data Structures as Cognitive… · p. 35, §D.1 · p. 36

D.1.1. View Interpolation for Dynamic Updating

  • View interpolation inserts synthetic frames between consecutive supplied views to bridge perceptual gaps and support tracking relations across viewpoint shifts.Frames may be selected heuristically from original datasets or generated with Stable Virtual Camera.Source: §D.1.1 · p. 36

D.1.2. Cognitive Maps for Integrated Spatial Layouts

  • Cognitive maps encode a schematic bird’s-eye spatial layout as structured data, with augmented maps adding camera positions and orientations absent from plain object-focused maps.The maps use a 10×10 grid and can represent object positions, facing directions, and views; JSON provides a text-based structured format for VLMs.Source: §D.1.2 · pp. 36–37

D.1.3. Free Form Reasoning; D.2. Evaluation Metrics; D.2.1. QA Accuracy

  • Free-form chain-of-thought prompts models to reason in three stages: ground observations within views, integrate views into one scene, then infer an answer from the question.QA Accuracy measures the proportion of correctly answered questions; the appendix identifies it as the core task metric.Source: §D.1.3 · pp. 37–38, §D.2 · p. 38, §D.2.1 · p. 38

D.2.2. Graph Metrics for Cognitive Maps … C. Bed sheet with a floral pattern

This appendix presents a rotation-aware graph-based evaluation for generated cognitive maps, specifies prompts for direct answering, reasoning, view interpolation, and map construction and use, and provides example prompts and qualitative outputs for spatial questions.

D.2.2. Graph Metrics for Cognitive Maps

  • Generated maps must be valid JSON with interpretable positions and at least one valid object; a separate strict metric tests graph isomorphism under allowed 3D rotations.The allowed rotations include 90° turns about the z-axis and one 90° turn about each of the x- and y-axes.Source: §D.2.2 · pp. 38–40
  • The structural comparison builds ordered object-pair relations from direction, proximity, and optional facing annotations, with proximity threshold δ = 0.5.The relation graph is used to assess whether ground-truth relationships are preserved in the generated map.Source: §D.2.2 · p. 39
  • The evaluation scores map coverage, directional relations, and object facing, combining directional and facing similarity with α = 0.7 to emphasize layout.It chooses the best-aligned 3D rotation when comparing generated maps with ground truth.Source: §D.2.2 · p. 39

D.3.1. Example for Raw-QA

  • The Raw-QA and FFR examples ask about the same four-view jar scene; Raw-QA requests one option only, while FFR requests step-by-step reasoning before the answer.The example asks what is left of the white jar from the fourth viewpoint.Source: §D.3.1 · p. 41, §D.3.2 · pp. 41–42

D.3.3. Example for VI-1 and VI-2

  • The VI-1 and VI-2 examples provide 8 and 12 views, respectively, interpolating camera viewpoints around the same scene before posing a single-answer question.The examples ask about the view from image 7 for VI-1 and image 10 for VI-2.Source: §D.3.3 · pp. 42–43

D.3.4. Example for Aug-CGMap-In

  • The Aug-CGMap-In prompt supplies a bird’s-eye 10×10 cognitive map with object positions, view locations, and facing directions, and instructs the model to use it to answer.The example map places the jar at and the floral bed sheet at.Source: §D.3.4 · pp. 43–44

D.3.7. Example for Plain-CGMap-FFR-Out

  • Plain-CGMap-FFR-Out requires the cognitive map before step-by-step reasoning and an answer, while the Aut-CGMap-FFR-Out subsection is identified as an augmented-map reasoning prompt.The supplied Aut-CGMap-FFR-Out passage provides only its prompt title.Source: §D.3.7 · pp. 47–48, §D.3.8 · p. 49

D.3.8. Example for Aug-CGMap-FFR-Out

  • Aug-CGMap-FFR-Out asks for a 10x10 bird’s-eye map of specified objects, merging views, preserving spatial relationships, and recording camera positions and directions.The prompt requires the map before step-by-step reasoning and a selected answer.Source: §D.3.8 · p. 49

D.3.9. Example for CGMap-In-FFR-Out

  • CGMap-In-FFR-Out supplies a 10x10 bird’s-eye map for reasoning, placing the jar at, bed sheet at, headboard at, rack at, and table at.It also represents four camera views around the jar, each with a position and facing direction.Source: §D.3.9 · p. 51

E. Which Scaffolds Best Guide Spatial Thinking in Unchanged VLMs?

C. Bed sheet with a floral pattern

  • For the image-4 question, FFR answers C (bed sheet), while Aug-CGMap-FFR-Out and Plain-CGMap-FFR-Out answer A (table with cups).The FFR example describes a bed-like object to the jar’s left; the two map-based outputs select the table with cups.Source: C. Bed sheet with a floral pattern · pp. 52–53

E.2. Additional Graph Metrics for Generated Graphs

Aug-CGMap-FFR-Out achieves stronger semantic similarity, while Plain-CGMap-FFR-Out produces more valid and structurally identical maps. Both methods generate maps with high validity, but exact structural replication remains challenging.

E.2. Additional Graph Metrics for Generated Graphs

  • Valid Rate is 94.95% for Plain-CGMap-FFR-Out and 92.57% for Aug-CGMap-FFR-Out, indicating both methods generate mostly valid maps.Validity requires syntactically and semantically valid maps.Source: §E.2 · p. 54
  • Isomorphism Rate is 7.43% for Plain-CGMap-FFR-Out versus 0.10% for Aug-CGMap-FFR-Out, showing exact structural replication remains challenging.Isomorphism accounts for rotation.Source: §E.2 · p. 54
  • Aug-CGMap-FFR-Out scores 51.12% Overall Similarity, 43.57% Avg. Dir. Sim., and 68.75% Avg. Facing Sim., outperforming Plain-CGMap-FFR-Out on all three similarity metrics.The comparison evaluates generated cognitive maps against ground truth.Source: §E.2 · p. 54, Table 5 · p. 54

E.3. Further Analysis on View Interpolation … F.1.2. Free-Form Reasoning Generation

The analysis finds that cognitive maps outperform even peak view interpolation, while explicit visual sketches improve over text maps. The SFT data is designed to teach VLMs to generate spatial maps and reason through grounded, step-by-step chains.

E.3. Further Analysis on View Interpolation

  • 47.41% for Plain-CGMap-FFR-Out exceeds the peak VI-1 baseline of 46.47% on Qwen2.5-VL-3B.Adding standard FFR to interpolated views instead drops performance from 45.53% to 40.77% as density increases.Source: §E.3 · p. 55, Table 6 · p. 55
  • Scaling is inconsistent across 7B/8B models: Qwen2.5-VL-7B peaks at VI-4, while Qwen3-VL-8B performs unstably.The evaluation also includes Qwen3-VL-235B and GPT-5.Source: §E.3 · p. 55, Table 7 · p. 55
  • 46.59% at one frame falls to 42.35% at seven frames for GPT-5, while Qwen3-VL-235B drops to ∼36% with interpolation.The reported large-model results negatively correlate with view density.Source: §E.3 · p. 55, Table 8 · p. 55

E.4. Explicit Reasoning with Visual-of-Thought

  • 43.13% overall accuracy for Img-CGMap surpasses the text-based Plain-CGMap’s 41.33%.For the Among relation, the visual grid scores 41.13% versus 39.67%; the configurations use externally rendered visual maps rather than JSON text maps.Source: Table 9 · p. 56, §E.4 · p. 57

F. Can We Teach VLMs to Build and Leverage Spatial Representations?

  • Prompting frozen VLMs with external scaffolds offers limited improvements, motivating supervised fine-tuning to teach internal spatial modeling.The appendix introduces further details of the SFT methodology, beginning with data curation.Source: F. Can We Teach VLMs to Build and… · p. 57

F.1. Supervised Fine-Tuning Data Curation

  • The SFT data is curated in two types: cognitive maps and free-form reasoning chains, intended to supervise spatial representation and reasoning.The maps provide structured scene representations, while the chains provide procedural reasoning examples.Source: §F.1 · p. 57, §F.1.2 · p. 58

F.1.1. Cognitive Map Generation

  • Cognitive-map targets are generated from object-arrangement annotations using setting-specific rules to place objects and camera views on a 10x10 grid, then formatted as JSON.The automated script uses scene metadata, including object orientations and camera-view setup, to create maps and add instructional prompts to each data item.Source: §F.1.1 · pp. 57–58

F.1.2. Free-Form Reasoning Generation

  • Free-form reasoning chains are manually tailored to QA templates, moving from per-view scene understanding to cross-view integration and question-driven spatial inference.They establish a unified scene using recurring objects, then verbalize hypothetical movement or perspective shifts and deduce spatial relationships.Source: §F.1.2 · pp. 58–59

F.2. Detailed Experimental Setup · F.3. VLM Response Examples After SFT for Configurations in Section D.3

The appendix details a controlled SFT setup for Qwen2.5-VL-3B-Instruct and illustrates how different configurations produce cognitive maps and answers for a shared scene. The examples include both direct responses and reasoning over multiple views.

F.2. Detailed Experimental Setup

  • SFT used Qwen2.5-VL-3B-Instruct to teach VLMs to build and leverage internal spatial representations.The experiments aimed to compare strategies for improving spatial reasoning.Source: §F.2 · pp. 59, 63
  • Across SFT runs, the dataset configuration was the primary variation, while other hyperparameters stayed constant to isolate training-signal effects.The configurations corresponded to different SFT tasks, including Aug-CGMap-Out.Source: §F.2 · p. 59
  • Training used two NVIDIA H100 GPUs and an effective batch size of 256, with DeepSpeed ZeRO Stage 3 and full-parameter fine-tuning.Per-device batch size was 4 with gradient accumulation of 32. Training ran for 3 epochs at a 1 × 10−5 learning rate with cosine scheduling and 0.03 warmup; sequence length was 8192 tokens and maximum image pixels 90,000 (minimum 784). Gradient checkpointing was enabled, with checkpoints saved every 5 steps and 12 retained.Source: §F.2 · p. 63

F.3. VLM Response Examples After SFT for Configurations in Section D.3

  • Plain-CGMap-Out’s output lists scene-object positions, including the white jar at and clothes rack at, and gives “B. Clothes rack” as its answer.Source: §F.3 · p. 64
  • FFR reasons across four views and their 90-degree clockwise rotations, then answers “B. Clothes rack.”The reasoning uses the opposite view to infer what lies behind the initial viewpoint.Source: §F.3 · p. 64
  • Both Aug-CGMap-FFR-Out and Plain-CGMap-FFR-Out produce reasoning over a cognitive map and answer “A. Table with cups on it.”Their reasoning interprets the four views as successive 90-degree clockwise rotations and uses the opposite view to infer the scene behind the initial viewpoint.Source: §F.3 · pp. 65–66

F.4. Detailed Graph Metric Results for SFT Graph-Related Experiments … G.1. Detailed Experimental Setup

The appendix reports that joint map-and-reason training yields the strongest SFT task accuracy, while language-model tuning accounts for nearly all early Raw-QA gains. It also examines branched SFT, hyperparameter choices, viewpoint removal, and RL configurations.

F.4. Detailed Graph Metric Results for SFT Graph-Related Experiments

  • 57.81% QA Accuracy was reached by Plain-CGMap-FFR-Out, outperforming map-only training despite slower initial graph-metric convergence.Aug-CGMap-FFR-Out reached about 54%, while Plain-CGMap-Out and Aug-CGMap-Out reached 54.29% and 52.48%, respectively; map-only training converged rapidly on graph quality metrics.Source: §F.4 · pp. 66–67, Figure 20 · p. 67

F.5. Which Part of VLM is the Bottleneck for Spatial Understanding?

  • 51.43% overall accuracy from tuning only the LLM nearly matched the 52.67% from end-to-end tuning, while vision-only tuning remained at the 37.81% baseline.The analysis identifies the LLM as the primary bottleneck and suggests pre-trained visual features may already suffice for extracting objects and properties; freezing the vision encoder is an effective lower-cost strategy.Source: Table 11 · p. 68, §F.5 · p. 68

F.6. Branching from Raw-QA SFT Checkpoint

  • 49.00% overall accuracy was achieved by branched Aug-CGMap-In and Aug-CGMap-FFR-Out, while branched FF Rsn reached 46.82%.Each branch began from a Raw-QA SFT checkpoint and continued training on the scaffold task while outputs remained constrained to the final answer option; all branched tasks improved over their from-scratch counterparts.Source: Table 12 · p. 69, §F.6 · p. 69

F.7. Hyperparameter Tuning Results

  • 52.67% accuracy was the highest result among the tested SFT hyperparameter settings, using a 10−5 learning rate, batch size 512, and warmup ratio 0.03.The authors report that this original configuration was effective and well-optimized for the task.Source: §F.7 · p. 69, Table 13 · p. 70

F.8. Effect of Removing Viewpoint Descriptors

  • 42.67% frozen accuracy for Plain-CGMap-FFR-Out exceeded Raw-QA’s 38.57% by 4.1 points without viewpoint descriptors; under SFT, it retained a slight edge at 54.0% versus 52.8%.The evaluation removed viewpoint descriptors while keeping other components unchanged; map-then-reason remained the top configuration in both settings.Source: Table 14 · p. 70, §F.8 · p. 70, Table 15 · p. 71

G. Can Reinforcement Learning Further Refine Spatial Thought Processes?

  • The RL section investigates whether reward-guided training can refine spatial mental models and reasoning, and introduces case studies for qualitative analysis.Its stated inquiry is whether outcome-driven feedback can improve spatial representations and reasoning capabilities.Source: G. Can Reinforcement Learning Further… · p. 71

G.1. Detailed Experimental Setup

  • RL used VAGEN with GRPO, Qwen2.5-VL-3B-Instruct, and MINDCUBE-TINY; each configuration was trained for 0.5 epoch with batch size 32, prompt length 1024, response length 512, and 8 trajectories per rollout.Previously established evaluation metrics were retained; the setup was limited in training duration in consideration of computational costs.Source: §G.1 · p. 71
  • RL compared free-form reasoning from scratch with joint augmented-map and reasoning training from scratch or from the strongest Aug-CGMap-FFR-Out SFT checkpoint.The reward assigned +1 for structurally valid output and +5 for a correct final answer, targeting both well-formed intermediate maps and end-task accuracy.Source: §G.1 · pp. 71–72

G.2. RL Reward Design Ablation · G.3. VLM Response Examples After RL for Configurations in Section D.3

The reward ablation finds that answer correctness is the key contributor to RL performance, while response examples show models answering spatial questions with and without explicit cognitive-map reasoning.

G.2. RL Reward Design Ablation

  • The reward assigns a small score to valid formatting and a larger score to correct answers; the ablation identifies the answer score as most critical.The design is inspired by the Deepseek-R1 approach[32], and the authors attribute the improvement to the +5 correct-answer score.Source: §G.2 · p. 72
  • Answer Only (+5 correct answer, 0 format) reached 50.76%, while Format Only (+5 valid format, 0 answer) reached 45.05%.The study compares three reward configurations against the frozen VLM baseline and proposed dual-component setting.Source: §G.2 · p. 72, Table 16 · p. 72
  • 49.52% accuracy with +1 for valid format and +5 for a correct answer exceeds the frozen VLM baseline’s 40.57%.These are the proposed reward setting and baseline results in the RL reward ablation.Source: Table 16 · p. 72, §G.2 · p. 72

G.3. VLM Response Examples After RL for Configurations in Section D.3

  • In the FFR example, the model reasons from image 4 that the table with cups is left of the white jar and answers “Table with cups on it.”The response describes a floral-patterned bed sheet beneath the jar and a table with cups to its left.Source: §G.3 · p. 72
  • The RL-Aug-CGMap-FFR-Out-from-scratch example places the jar at the center facing left, then answers “Table with cups on it.”Its response mentions the clothes rack to the jar’s left and floral bed sheet to its right.Source: §G.3 · p. 73
  • In the from-SFT example, the model encodes object and view positions in a cognitive map, reasons across four rotated views, and answers “Bed sheet with a floral pattern.”It uses the opposite view to infer the bed sheet’s position relative to the initial view.Source: §G.3 · pp. 73–74

Related papers

Loading 2506.21458v2…