PaperlayerView exact revision on arXiv

Source-linked AI summary

MindCube: Spatial Mental Modeling from Limited Views

Qineng Wang, Baiqiao Yin, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Jiajun Wu, Li Fei-Fei, Manling Li

arXiv:2506.21458v2cs.AIcs.CLcs.CV

TL;DR

VLMs struggle to reason about unseen spatial relationships from limited views, motivating specialized evaluation of partial observations, cross-view consistency, and mental simulation. MindCube benchmarks this gap and shows that constructing and reasoning over self-generated cognitive maps most effectively approximates spatial mental models, with reinforcement learning further improving the process.

  • Problem

    VLMs need to reason about unseen spatial relationships from limited views, including occlusion, shifting viewpoints, and hypothetical movement.

  • Method

    The paper introduces MINDCUBE and evaluates structured scaffolds including augmented cognitive maps that encode object and view positions and orientations.

  • Results

    Existing VLMs perform only marginally better than random guessing, while constructing and reasoning over self-generated cognitive maps is most effective across elicitation methods.

  • Takeaways & Limitations

    Scaffolding VLMs with internal cognitive maps improves their approximation of spatial mental models across input-output configurations, supervised fine-tuning, and reinforcement learning.

  • Takeaways & Limitations

    The datasets may not represent the full range of global environments, potentially introducing geographic or cultural biases into spatial understanding.

Abstract

from arXiv · show

Can Vision-Language Models (VLMs) imagine the full scene from just a few views, like humans do? Humans form spatial mental models naturally, internal representations of unseen space, to reason about layout, perspective, and motion. Our MindCube benchmark with 21,154 questions across 3,268 images exposes this critical gap, where existing VLMs exhibit near-random performance. Using MindCube, we systematically evaluate how well VLMs build robust spatial mental models through representing positions (cognitive mapping), orientations (perspective-taking), and dynamics (mental simulation for "what-if" movements). We then explore three approaches to help approximate spatial mental models in VLMs, focusing on incorporating unseen intermediate views, natural language reasoning chains, and cognitive maps. The significant improvement comes from a synergistic approach, "map-then-reason", that jointly trains the model to first generate a cognitive map and then reason upon it. By training models to reason over these internal maps, we boosted accuracy from 37.8% to 57.8% (+20.0%). Adding reinforcement learning pushed performance even further to 61.3% (+23.5%). Our key insight is that such scaffolding of spatial mental models, actively constructing and utilizing internal structured spatial representations with flexible reasoning processes, significantly improves understanding of unobservable space.

1. Introduction

MINDCUBE targets whether VLMs can construct consistent spatial mental models from limited, shifting views and reason about unseen space. The benchmark exposes near-random performance and motivates scaffolding through additional views, language reasoning, and cognitive maps.

  • Spatial mental models: Spatial mental models are internal representations that support viewpoint-independent inference about layouts, hidden objects, positions, and poses.Humans integrate several egocentric observations to infer relationships among objects that are not simultaneously visible.
  • Benchmark motivation: 21,154 questions across 3,268 images test reasoning about partial observations, cross-view consistency, perspective, and “what-if” spatial transformations.Questions are organized into multi-view groups with ROTATION, AMONG, and AROUND viewpoint transformations.
  • Observed gap: 17 state-of-the-art VLMs perform only marginally better than random guessing on MINDCUBE.This result motivates methods for facilitating spatial mental models under partial observations.
  • Proposed direction: The paper investigates external and internal strategies, including view scaling, cognitive-map input, fine-tuning, and cognitive-map elicitation.These approaches are inspired by visual imagery, linguistic reasoning, and explicit cognitive maps in spatial cognition.

2. MINDCUBE Benchmark and Evaluation

MINDCUBE is a benchmark for spatial reasoning under partial observations and dynamic viewpoints, designed to analyze consistency across views and inference about invisible elements. Its evaluation shows that current VLMs remain far from uniformly strong spatial reasoning.

  • Benchmark construction: MINDCUBE pairs multi-view orthogonal images with spatial questions about object consistency, occlusion, and invisible elements.The dataset was curated from selected multi-view groups, spatial annotations, and automatically generated questions with targeted distractors.
  • Benchmark settings: ROTATION, AROUND, and AMONG create distinct viewpoint challenges involving incremental visibility, occlusion, object permanence, and depth cues.These settings require models to integrate views rather than rely on a single visible arrangement.
  • Evaluation results: 47.62% accuracy is achieved by the best model, DeepSeek-VL2-Small, while no model excels across all categories.GPT-5 reaches 93.33% in ROTATION, whereas Mantis-8B (SigLip) reaches 50.99% in AROUND.

3. Which Scaffolds Best Guide Spatial Mental Modeling?

The paper tests view interpolation, cognitive maps, and free-form reasoning as scaffolds for spatial mental modeling. In frozen VLMs, reasoning is more useful than passive structure, while generated maps provide additional but limited gains.

  • Data structures: Augmented cognitive maps encode object and view positions and orientations, extending plain top-down maps toward relational consistency.Free-form reasoning provides a procedural approximation for constructing and querying spatial models under incomplete observations.
  • View interpolation: 37.81% accuracy is obtained by raw views and direct answering, while interpolated views add only 0.09%.View interpolation inserts synthetic frames between consecutive views to provide smoother perceptual transitions.
  • Frozen-model scaffolds: 32.00% accuracy results from giving an augmented cognitive map as direct input, whereas free-form reasoning reaches 41.33%.These results indicate that passive visual or spatial structure alone provides little benefit without explicit reasoning.
  • Map elicitation: Generating a cognitive map before answering improves performance from 40.48% to 41.43% over free-form reasoning alone.Both map formats follow the requested syntax well but generate maps with low similarity to grounded maps.
  • Map quality: 0.10% and 7.43% are the isomorphism rates for augmented and plain generated maps, respectively.The near-zero augmented-map rate is attributed to added view-level details increasing generation errors.

4. Can We Train for the Emergence of Spatial Mental Models via VLMs’ Use of Scaffolds?

The paper tests whether supervised fine-tuning and reinforcement learning can teach VLMs to construct and use spatial mental-model scaffolds. Jointly generating cognitive maps and free-form reasoning is most effective, while randomized object order better tests genuine spatial understanding.

  • SFT on raw QA pairs raises accuracy from 37.81% to 52.67%, providing the baseline for explicit spatial-structure methods.
  • Map generation alone improves graph quality but yields limited QA gains: Plain-CGMap-Out reaches 54.29%, while Aug-CGMap-Out reaches 52.48% versus Raw-QA at 52.67%.FFR alone reaches 55.43%, still below the joint approach.
  • Plain-CGMap-FFR-Out gains 5.14% over Raw QA-SFT, outperforming map-only and reasoning-only configurations through their combined use.Map-only training rapidly learns structure but plateaus in QA accuracy; the joint model learns maps more slowly while QA accuracy continues increasing.
  • RL from scratch provides insufficient guidance: RL-FFR reaches 49.52%, while structured RL variants reach only 52.48% or 50.86% and learn weak geometry.The results indicate that task-level rewards and structural formats alone do not reliably produce meaningful spatial representations.
  • RL initialized from SFT performs best, with RL-Plain-CGMap-FFR-Out reaching 61.33% and map-then-reason configurations outperforming FFR-only.The Plain-CGMap variant also achieves a 35.33% isomorphism rate versus 16.95%, suggesting simpler maps better preserve spatial structure during RL.
  • Randomized object order is adopted because it prevents predictable ordering shortcuts; nevertheless, configuration rankings and the joint-training advantage persist under both orderings.Fixed order raises Plain-CGMap-Out isomorphism from 45.52% to 89.05%, but randomized order is judged more faithful to genuine spatial understanding.

5. Related Works

Related work situates MindCube at the intersection of spatial cognition and multiview spatial understanding. Prior research addresses spatial abilities, 3D reconstruction, view synthesis, geometric consistency, and relational representations.

  • Spatial Cognition: Spatial cognition includes mental rotation, spatial visualization, and object assembly for manipulating spatial relationships in 2D and 3D environments.Spatial mental models are identified as internal representations supporting consistent understanding of space.
  • Multi Views understanding: Multiview understanding reconstructs 3D structures across viewpoints using reconstruction, view synthesis, multiview-equivariant learning, and topological representations.These approaches address the limitations of single-view spatial reasoning.

6. Conclusion

MindCube evaluates how VLMs approximate spatial mental models from limited views and examines ways to scaffold internal representations. The strongest results come from constructing and reasoning over self-generated cognitive maps, with SFT initialization further improving RL.

  • MINDCUBE studies VLM reasoning in partially observable environments by evaluating spatial mental-model approximation from limited views.
  • Self-generated cognitive maps combined with reasoning outperform view interpolation and externally provided maps across input-output configurations, SFT, and RL.
  • Initializing RL from a well-trained SFT checkpoint further improves spatial reasoning performance.

Ethics Statement

The paper documents dataset, computational, and dual-use considerations. Its datasets may not represent global environments fully, and large-scale VLM experimentation carries environmental and computational costs.

  • The benchmark combines public anonymized datasets with self-collected indoor and outdoor imagery captured without PII or sensitive content.
  • Human annotators involved in curation and evaluation were compensated at rates significantly exceeding their local minimum wage.
  • The datasets may underrepresent global environments, potentially introducing geographic or cultural biases into spatial understanding.
  • Training, fine-tuning, and evaluating large-scale VLMs carries significant computational and environmental cost, while enhanced machine spatial reasoning may have dual-use applications.

B.1. Details for Data Collection and Annotation

MINDCUBE constructs a spatial reasoning benchmark from curated multi-view imagery, systematic annotations, and labels covering visual patterns, perspective, and dynamic viewpoint changes.

  • Data Collection: 3,268 images combine public and self-collected data across four view dynamics for MINDCUBE’s training and evaluation pipeline.The collection includes 2,302 publicly released images and 400 self-collected images.
  • View Selection: Rotation trajectories are filtered into linear paths or approximately 90-degree rotational arcs, then converted into clear key-frame sequences.Selected frames require suitable movement alignment, visibility, and limited motion blur or semantic clutter.
  • View Selection: Among views use four cardinal viewpoints, while Around sequences designate a clear front view as the reference for subsequent views.Among selection also rejects sets with excessive duplicate semantic background information.
  • Annotation: Annotations encode spatial relationships, grouped objects, semantic orientation, and occlusion levels to support fine-grained spatial reasoning evaluation.Object annotations include front-back and left-right relations, attributes, and viewpoint-relative orientation information.
  • Label Taxonomy: The taxonomy distinguishes linear and non-linear visual patterns, Perspective Taking levels, and “What if” viewpoint transitions formed from translation and rotation.These labels describe geometric arrangement, viewpoint projection complexity, and dynamic perspective changes.
  • Evaluation: Text-based captions reduce model accuracy relative to original images, while humans outperform advanced multimodal models on the spatial problems evaluated.Brief captions produce the largest reported drops, including 7.83% for RoboBrain-8B and 12.91% for LLaVA-OneVision-7B.

C.5.1. Around

The Around analysis shows that occlusion and viewpoint elevation strongly affect spatial reasoning, while paired tests reveal inconsistency and difficulty using additional reference objects.

  • Occlusion Degree Analysis: 50.7% average accuracy separates fully visible from fully occluded conditions, with a 28.7% drop occurring between mostly visible and mostly occluded views.The non-linear decline suggests a performance threshold as occlusion becomes substantial.
  • Camera Height Impact Analysis: 24.8% higher accuracy comes from high-angle than eye-level views, especially in dense arrangements where elevated perspectives reveal spatial gaps.The reported advantage is attributed to better visibility of partially occluded object tops and broader scene context.
  • Test Design: Around questions test whether models preserve object permanence and convert lateral relations into depth cues across changing viewpoints.The section also evaluates attribute invariance and quantity sensitivity through controlled modifications of central objects.
  • Consistency Analysis: Around 40% inconsistency appears for SpaceQwen across both paired-question tests, while LLaVA-Onevision remains comparatively balanced and high-performing.The paired analysis classifies outcomes as both correct, both wrong, or inconsistent when only one version is answered correctly.
  • Interpretation: Models can achieve attribute-invariant reasoning but struggle to exploit additional reference objects, motivating stronger geometric reasoning and broader 3D benchmarks.The findings are consistent with reliance on local relationship matching rather than global spatial configuration inference.
  • Failure Case Analysis: Failure cases include assigning incorrect spatial locations to views and confusing the subject of a “behind” relation.These errors illustrate failures in cross-view grounding and relational interpretation.

D.2.2. Graph Metrics for Cognitive Maps

The evaluation validates generated cognitive maps, aligns them under allowed rotations, and compares their object coverage, spatial relations, and facing directions against ground truth.

  • Validity Check: Cognitive-map evaluation first checks JSON validity, interpretable positions, expected fields, and the presence of at least one valid object.
  • Rotation Normalization: Generated and ground-truth maps are compared across allowed 3D rotations, selecting the best-aligned orientation for similarity scoring.The rotation set includes 90° turns about the z-axis and one 90° turn about each of the x- and y-axes.
  • Structural Matching: Relation graphs encode directional and proximity-based relationships between ordered object pairs in each map.Relations are derived from relative 2-D positions and optional facing annotations, with a distance threshold δ=0.5 for proximity labels.
  • Similarity Metrics: Coverage measures retrieved ground-truth objects, while directional and facing similarity measure preserved spatial relations and object orientations.Directional similarity reflects correctly matched ground-truth relations, and the aggregate score combines directional and facing similarity.
  • Rotation-Invariant Isomorphism: Rotation-invariant isomorphism tests whether the generated relation graph is structurally identical to ground truth under any allowed 3D rotation.

E.2. Additional Graph Metrics for Generated Graphs

The generated maps are usually valid, but exact structural replication remains difficult; camera-view augmentation instead improves semantic similarity in layout and orientation.

  • Graph Metrics: 94.95% Valid Rate for Plain-CGMap-FFR-Out and 92.57% for Aug-CGMap-FFR-Out indicate strong syntactic and semantic map validity.Both methods learned the required JSON structure for cognitive maps.
  • Graph Metrics: 7.43% Isomorphism Rate for Plain-CGMap-FFR-Out versus 0.10% for Aug-CGMap-FFR-Out shows that perfect structural replication remains challenging.
  • Graph Metrics: 51.12% Overall Similarity, 43.57% Avg. Dir. Sim., and 68.75% Avg. Facing Sim. are achieved by Aug-CGMap-FFR-Out with explicit camera views.The augmented method is superior in relative directional relationships and individual-object orientation, despite lower validity and isomorphism.

E.3. Further Analysis on View Interpolation

Additional experiments show that simply increasing view density does not reliably improve spatial reasoning, whereas cognitive maps and explicit visual sketches provide structured alternatives for representing scenes.

  • Comparison with Optimal View Interpolation: 47.41% for Plain-CGMap-FFR-Out exceeds the View Interpolation baseline’s peak of 46.47% on Qwen2.5-VL-3B.This comparison indicates a structural advantage for cognitive maps over increasing visual frame density in this setting.
  • Scaling Analysis: View interpolation shows no consistent scaling law across Qwen2.5-VL-7B and Qwen3-VL-8B.Qwen2.5-VL-7B benefits from density up to VI-4, whereas Qwen3-VL-8B is unstable.
  • Scaling Analysis: GPT-5 declines from 46.59% with one frame to 42.35% with seven, while Qwen3-VL-235B drops to approximately 36% with interpolation.The results suggest that interpolation artifacts can behave as noise without structured mapping.
  • Explicit Visual Sketching via External Tools: Img-CGMap renders model-generated coordinates through an external plotting engine and feeds the resulting visual sketch back for final spatial reasoning.Img-CGMap-In instead supplies a rendered 10 × 10 grid directly as input.
  • Explicit Visual Sketching via External Tools: 43.13% overall accuracy for Img-CGMap surpasses 41.33% for Plain-CGMap, with Among accuracy reaching 41.13% versus 39.67%.Visual grids particularly benefit relative positioning, which is described as more intuitive in pixel space than coordinate space.

F.2. Detailed Experimental Setup

The detailed setup trains Qwen2.5-VL-3B-Instruct with consistent SFT procedures across spatial-reasoning configurations, using cognitive-map and reasoning-chain supervision alongside standardized prompts and outputs.

  • Experimental Setup: Qwen2.5-VL-3B-Instruct is the base model for SFT experiments designed to teach internal spatial representations.
  • Training Procedure: All SFT configurations use a consistent training script, varying primarily the dataset to isolate the effects of different training signals.
  • Prompt Configurations: The prompts define separate Raw-QA, free-form reasoning, and cognitive-map input/output configurations for Rotation, Among, and Around settings.Examples include explicit map generation, intermediate reasoning, and final answer formats.
  • Prompt Outputs: Example cognitive-map outputs encode object positions on a 2-D grid and, for augmented maps, camera-view positions with facing directions.The shown examples place the white jar at [5,5] and views around it with cardinal facing labels.

F.4. Detailed Graph Metric Results for SFT Graph-Related Experiments

SFT training dynamics show that jointly generating cognitive maps and reasoning yields the strongest QA accuracy, despite slower initial graph-metric convergence than map-only training.

  • Training dynamics: Joint training is the most effective strategy for maximizing task accuracy across the evaluated map-generation settings.The comparison covers four map-generation settings and identifies joint map generation plus reasoning as best overall.
  • Map-only training: 54.29% QA Accuracy is reached by Plain-CGMap-Out, while Aug-CGMap-Out reaches 52.48% under map-only training.Graph quality metrics converge rapidly in both map-only configurations, but final QA accuracy remains limited.
  • Joint training: 57.81% QA Accuracy is achieved by Plain-CGMap-FFR-Out, the strongest evaluated setting.Aug-CGMap-FFR-Out also improves to about 54% QA accuracy.
  • Training dynamics: Joint approaches converge more slowly on graph quality metrics but ultimately achieve higher task accuracy than map-only approaches.The result indicates a trade-off between structural map perfection and downstream QA performance.

F.5. Which Part of VLM is the Bottleneck for Spatial Understanding?

The appendix examines which VLM component limits spatial understanding and whether staged training or prompt changes improve learning. Results emphasize language-model adaptation and show that map-then-reason remains competitive without viewpoint descriptors.

  • Bottleneck analysis: LLM fine-tuning raises accuracy above 51%, whereas the no-fine-tuning baseline achieves 37.81% overall accuracy.The comparison was conducted on MINDCUBE-TINY under Raw-QA at an early training stage.
  • Bottleneck analysis: 51.43% overall accuracy from tuning only the LLM nearly matches 52.67% from end-to-end fine-tuning, while tuning only the vision encoder remains at 37.81%.These results place the principal bottleneck in the LLM rather than jointly across both modules.
  • Branching from Raw-QA: Two-stage SFT consistently improves branched tasks over counterparts trained from scratch.The procedure first fine-tunes Raw-QA, then uses that checkpoint for cognitive-scaffold tasks while constraining outputs to final answers.
  • Branching from Raw-QA: 49.00% overall accuracy is reached by both Aug-CGMap-In and Aug-CGMap-FFR-Out after branching from a Raw-QA checkpoint.FF Rsn also improves to 46.82% under the two-stage approach.
  • Hyperparameter tuning: 52.67% accuracy is obtained with the original SFT hyperparameters: learning rate 10^-5, batch size 512, and warmup ratio 0.03.The tuning experiments are presented as confirmation that this configuration is effective for the task.
  • Viewpoint-descriptor ablation: 42.67% frozen accuracy is achieved by Plain-CGMap-FFR-Out without viewpoint descriptors, compared with 38.57% for Raw-QA.Under SFT, the same configuration retains a slight edge at 54.0% versus 52.8% for Raw-QA.

G.2. RL Reward Design Ablation

The RL ablation compares rewards for valid formatting and correct answers. Rewarding answer correctness is the critical component, while the combined design substantially outperforms the frozen baseline.

  • Reward design: The reward structure encourages both structural correctness and factual accuracy through separate format and answer components.This dual-component design was inspired by the Deepseek-R1 approach.
  • Reward configurations: 49.52% accuracy is achieved by the proposed reward setting with +1 for valid format and +5 for a correct answer, versus 40.57% for the frozen VLM baseline.The reward design assigns a smaller score to structural validity and a larger score to factual correctness.
  • Reward configurations: 50.76% accuracy is reached by the Answer Only setting, compared with 45.05% for Format Only.Answer Only assigns +5 for a correct answer and 0 for format; Format Only reverses that allocation.
  • Reward interpretation: The correct-answer reward is identified as the most critical factor in the RL reward design.The appendix attributes the stronger effect to the high correct-answer score of +5.
Loading 2506.21458v2…