PaperlayerView exact revision on arXiv

Source-linked AI summary

MindJourney: Test-Time Scaling with World Models for Spatial Reasoning

Yuncong Yang, Jiageng Liu, Zheyuan Zhang, Siyuan Zhou, Reuben Tan, Jianwei Yang, Yilun Du, Chuang Gan

arXiv:2507.12508v2cs.CVcs.AIcs.RO

TL;DR

VLMs struggle to reason about 3D scenes from images, especially when anticipating the consequences of egocentric motion. MindJourney couples a VLM with a controllable video world model for interactive exploration at test time. It improves SAT performance and reports robust gains across images, task categories, VLMs, and world models.

  • Problem

    VLMs struggle to reason about the 3D world behind 2D images, including imagining the consequences of egocentric movements, a capability relevant to embodied agents.

    Source: §1 · pp. 1–2

  • Method

    MindJourney has a VLM plan exploratory camera trajectories and uses a world model to render views that the VLM then uses to answer spatial-reasoning questions.

    Source: §1 · p. 2

  • Results

    7.7 percentage points higher average top-1 accuracy on SAT-Real, with GPT-4o gaining over 10%; the conclusion reports gains across synthetic and real images, SAT task categories, and two world-model generators.

    Source: §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    MindJourney offers a training-free, model-agnostic approach that can complement and even surpass complex RL-based self-reflection pipelines for spatial reasoning.

    Source: §1 · p. 2, §5 · p. 9

    The current pipeline assumes a single reference view and does not treat additional query images as entry points into the scene.

    Source: §5 · p. 9

Abstract

from arXiv · show

Spatial reasoning in 3D space is central to human cognition and indispensable for embodied tasks such as navigation and manipulation. However, state-of-the-art vision-language models (VLMs) struggle frequently with tasks as simple as anticipating how a scene will look after an egocentric motion: they perceive 2D images but lack an internal model of 3D dynamics. We therefore propose MindJourney, a test-time scaling framework that grants a VLM with this missing capability by coupling it to a controllable world model based on video diffusion. The VLM iteratively sketches a concise camera trajectory, while the world model synthesizes the corresponding view at each step. The VLM then reasons over this multi-view evidence gathered during the interactive exploration. Without any fine-tuning, our MindJourney achieves over an average 7.7% performance boost on the representative spatial reasoning benchmark SAT, showing that pairing VLMs with world models for test-time scaling offers a simple, plug-and-play route to robust 3D reasoning. Meanwhile, our method also improves upon the test-time inference VLMs trained through reinforcement learning, which demonstrates the potential of our method that utilizes world models for test-time scaling.

1 Introduction

Current VLMs struggle to reason about the 3D world behind an image, particularly when a question requires imagining egocentric movement. MindJourney uses a world model to generate observations during interactive exploration, improving spatial-reasoning performance.

1 Introduction

  • VLMs struggle to imagine the consequences of egocentric movement, and this gap limits their usefulness for embodied agents operating in 3D.The introduction contrasts this difficulty with humans mentally simulating a short walk to visualize the next view.Source: §1 · pp. 1–2
  • MindJourney has the VLM plan short exploratory trajectories, renders corresponding egocentric video with a world model, and uses the resulting imagined rollouts to answer.The framework searches imagined 3D space at test time rather than asking the VLM to answer immediately.Source: §1 · p. 2
  • 7.7% average improvement on SAT and a largest single gain over 10% accompany MindJourney across four VLM back-ends and two world models.The introduction also reports that the method outperforms o1-like test-time scaling VLMs.Source: §1 · p. 2
  • MindJourney couples a VLM with a controllable video world model to search imagined 3D space and improve spatial reasoning without fine-tuning.The approach is described as model-agnostic, boosting four VLMs with two distinct world models.Source: §1 · p. 2

2 Related work

Related work covers video-generation world models, VLM spatial reasoning, and test-time scaling for reasoning. MindJourney uses a physically consistent world model to add geometry-aware visual evidence to spatial-reasoning search.

World Models

  • Controllable video models conditioned on actions or camera movement have been explored for simulating 3D dynamics, while earlier world-model work often focused on static cameras and simple environments [Bar et al., 2024, Parker-Holder et al., 2024, Zhou et al., 2025b, Team et al., 2025].MindJourney leverages world-model imagination to support VLM spatial reasoning.Source: §2 · p. 3

Vision-Language Models and Spatial Reasoning

  • Despite progress in VLMs, benchmarks including SpatialRGPT, SAT, COMFORT, and SPAR show that state-of-the-art models still fall short on spatial understanding and reasoning [Cheng et al., 2024, Ray et al., 2024, Zhang et al., 2025b,a].Prior work also explores visual grounding, training strategies, and test-time scaling [Li et al., 2022b, Ma et al., 2023, Wang et al., 2024c, Rasheed et al., 2024, Zhang et al., 2024].Source: §2 · p. 3

Test-Time Scaling for Reasoning

3 Approach

MindJourney enhances VLM spatial reasoning at test time by coupling the model to a world model that explores imagined 3D space. It generates synthetic egocentric views along camera-action trajectories, selects useful observations, and answers from that evidence.

3.1 Pipeline Overview

  • The pipeline uses Spatial Beam Search to generate candidate viewpoints, keep observations useful to the query, and provide the buffered views to a VLM for its final answer.The final VLM receives the original image together with the selected observations.Source: §3.1 · p. 4

3.2 World Model Formulation

  • The world model maps a reference image and a camera-pose sequence induced by primitive actions to a synthetic egocentric video rollout.Primitive actions encode forward distance and left or right rotation; each pose conditions the generated frame on the intended motion.Source: §3.2 · pp. 4–5

3.3 Spatial Beam Search for Action Space Exploration

  • Spatial Beam Search expands candidate trajectories, scores views for further exploration and immediate usefulness, and stores helpful observations for final question answering.The search alternates question-agnostic expansion with question-aware pruning and stops at its step limit or when thresholding empties the beam.Source: §3.3 · pp. 5–6

3.4 Search World Model Details

  • Search World Model (SWM) is trained for MindJourney’s primitive action space, uses camera extrinsics embedded into video latents, and combines Habitat simulation with real-world video datasets.The training mix includes Habitat, RealEstate-10K, and DL3DV-10K.Source: §3.4 · p. 6

4 Experiment

MindJourney is evaluated on SAT-Real and SAT-Synthesized with four VLM backends and two world models; results improve on both splits. Ablations show that useful search depth depends on world-model fidelity, while analysis finds world-model search can complement RL scaling.

4.1 Experiment Settings

  • SAT covers egocentric and object motion, action consequence, and perspective shifts; evaluation uses real indoor/outdoor images and synthetic AI2-THOR scenes.The benchmark includes 4,000 synthetic questions; experiments use four VLMs and two video world models.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Experiments Results

  • SAT-Real average top-1 accuracy rises 7.7 percentage points, and SAT-Synthesized mean accuracy improves by 8.0% on average with MindJourney.On SAT-Real, GPT-4o gains over 10%; on SAT-Synthesized, an augmented model leads across all five question types.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Ablation Study

  • Deeper search helps on SAT-Synthesized through step 3, while SAT-Real accuracy peaks at step 2 for thresholds 4 and 6.The authors attribute the real-split decline at step 3 to SWM struggling with outdoor views as trajectories move farther from the reference frame; lenient pruning thresholds admit low-quality views, with a stronger effect on SAT-Real.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 Analysis

  • World-model search on a plain VLM surpasses RL-fine-tuned o1 on both SAT splits, and applying search atop o1 raises accuracy further.The analysis interprets the rollouts as information largely orthogonal to RL chain-of-thought; longer rollouts remain limited by generation quality and consistency.Source: §4.4 · p. 9

5 Conclusion

The conclusion presents MindJourney as a training-free way to pair VLMs with a world model for test-time spatial exploration. It reports robust SAT gains across image types, task categories, VLMs, and world-model generators, while noting limits involving multiple reference views and query-agnostic generation.

5 Conclusion

  • MindJourney uses Spatial Beam Search to explore a latent 3D scene from one image and cache informative imagined views; the training-free method improves SAT accuracy across varied VLMs.The reported gains span synthetic and real images, all SAT categories, and two world-model generators.Source: §5 · p. 9
  • The current pipeline assumes one reference view and does not use additional query images as separate scene-entry points.The authors identify multi-source exploration and evidence fusion as a next step.Source: §5 · p. 9
  • Question-agnostic video world models can generate views irrelevant or contradictory to assumptions implicit in the query.The authors propose query-conditioned models or lightweight constraints as future directions.Source: §5 · p. 9

A.1.1 Search Configurations

  • Experiments use a three-step search, beam size 2, thresholds γexp = 8 and γhelp = 8, and trajectories of up to eight actions.Primitive actions can repeat up to three times; each forward step moves 0.25 m and each rotation turns 9°.Source: §A.1.1 · p. 14

A.1.2 Computational Resources

  • SWM experiments use A40 GPUs, SVC experiments use H100 GPUs, and InternVL3-14B with SWM also uses H100 GPUs.The appendix also provides trajectory-expansion examples for SAT-Real and SAT-Synthesized.Source: §A.1.2 · p. 14, Figure 5 · p. 14, Figure 6 · p. 14

A.2.1 Dataset

The Search World Model training set combines simulated navigation data with real indoor and outdoor videos. Its implementation uses a camera-conditioned video-diffusion backbone with limited fine-tuning.

A.2.1 Dataset

  • SWM training data combines HM3D, DL3DV-10K, and RealEstate10K.These sources provide simulated navigation, real outdoor video, and real indoor video, respectively.Source: §A.2.1 · p. 15
  • The HM3D component contains 50K simulated navigation clips, while RealEstate10K and DL3DV-10K add 10K indoor and 10K outdoor videos.Clips are normalized for frame rate and resolution; HM3D episodes sample start and goal positions and follow shortest paths for up to 500 steps.Source: §A.2.1 · p. 15

A.2.2 Implementation Details

  • The implementation adopts Wan2.2-TI2V-5B and conditions generation on target camera poses through encoded frame extrinsics.Training fine-tunes only the camera encoder and a newly introduced projector.Source: §A.2.2 · p. 15

A.2.3 Training Details

Training clips are subsampled with varied frame skips, and optimization uses Adam with warmup and gradient clipping.

A.2.3 Training Details

  • Training uses frame-skip strides from 1 to 3, Adam with linear warmup to 3e−5, bfloat16, and gradient clipping at norm 1.0.Video-diffusion models train on eight NVIDIA H100 GPUs for approximately three days.Source: §A.2.3 · p. 15

B Failure Case Analysis

The failure analysis identifies motion-control and image-generation problems that can make imagined views unreliable.

B.1.1 Case 1: Inaccurate Forward Movement

  • Conflicting training-data scale conventions may produce inaccurate forward distances, causing overshooting and erratic motion.The authors hypothesize that this arises because HM3D provides metrically accurate distances while other sources use more heterogeneous scale calibrations.Source: §B.1.1 · p. 15

B.1.2 Case 2: Unintended Roll Movement

  • Unintended roll movements can tilt the scene and distort camera orientation.Source: §B.1.2 · p. 17

B.1.3 Case 3: Unstable Egocentric Rotation

  • During egocentric rotation, unstable viewpoints can combine inconsistent frame transitions with apparent rightward translation; this occurs more often in SWM, which the authors link to RealEstate10K training clips combining translation and rotation.Source: §B.1.3 · p. 17

B.1.4 Case 4: Visual Artifacts

  • Occluded or unseen regions can contain texture, edge, or object-boundary artifacts when the model has insufficient visual context or encounters out-of-domain data.Source: §B.1.4 · p. 17

B.1.5 Case 5: Out of domain data - scene misinterpretation

The world model can misinterpret scenes outside its training distribution, particularly complex real-world environments.

B.1.5 Case 5: Out of domain data - scene misinterpretation

  • Out-of-distribution scenes with unfamiliar lighting, textures, or layouts can trigger structural misinterpretations, which the authors attribute to limited generalization and insufficient exposure to diverse scenes during training.Source: §B.1.5 · p. 17

B.1.6 Case 6: Out of domain data - human or animal

The analysis identifies limits in world-model generation and VLM evidence use, alongside a quantitative and qualitative comparison of the world models and an ablation of trajectory descriptions.

B.1.6 Case 6: Out of domain data - human or animal

  • Underrepresented humans and animals can yield implausible or semantically incorrect predictions because the model lacks exposure to articulated, deformable entities.Source: §B.1.6 · p. 17

B.2.1 Case 1: VLM Q&A

  • Even useful generated views may leave the VLM unable to answer correctly, so the base VLM’s question-answering ability remains important.Source: §B.2.1 · p. 17, Figure 8 · p. 18

B.2.2 Case 2: VLM Scoring

  • The scoring process can discard an informative view, such as the left-turn image showing the question’s garbage can, leaving the evidence buffer insufficient for a correct answer.Source: §B.2.2 · p. 19, Figure 9 · p. 19

C.1 World Models

  • The world-model comparison evaluates SWM and SVC on AI2-THOR, which is out-of-domain for both, using prediction-accuracy, image-quality, and qualitative measures.Source: §C.1 · p. 19, §C.1.1 · p. 19

C.1.1 Quantitative Comparison

C.1.2 Qualitative Comparison

C.2 Ablation on Trajectoy Description

  • Removing natural-language trajectory descriptions lowers performance on both SAT-Real and SAT-Synthesized across the tested VLMs and world models.Source: §C.2 · p. 21

C.3 Ablation on Time Consumption

The section examines inference-time trade-offs, lists the prompts used, and discusses possible broader impacts and risks.

C.3 Ablation on Time Consumption

  • Using fewer SWM generation iterations speeds inference but lowers video quality and accuracy; two search steps and 20 iterations do not significantly degrade performance and are recommended as a more cost-effective setting.Source: §C.3 · p. 21, Table 4 · p. 21

D Prompts

E Broader Impacts

  • The authors identify potential benefits in robotics, inspection, and immersive training, while noting surveillance, labor displacement, data-bias, and energy concerns.Source: E Broader Impacts · p. 23

Related papers

Loading 2507.12508v2…