PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models

Jialong Wu, Xiaoying Zhang, Hongyi Yuan, Xiangcheng Zhang, Tianhao Huang, Changjing He, Chaoyi Deng, Renrui Zhang, Youbin Wu, Mingsheng Long

arXiv:2601.19834v1cs.AI

TL;DR

Current multimodal systems remain limited on physical and spatial reasoning, while the benefits of visual generation are not yet clear. The paper formalizes reasoning as world modeling, builds VisWorld-Eval, and finds that interleaved visual-verbal reasoning helps on tasks favoring visual world models but not otherwise.

  • Problem

    Existing systems excel in language-centered domains yet lag on physical and spatial reasoning, and prior studies provide mixed evidence about when visual generation helps.

  • Method

    The paper formalizes multimodal chain-of-thought as world modeling, distinguishes reconstruction and simulation, and evaluates them with the seven-task VisWorld-Eval suite.

  • Results

    Interleaved visual-verbal chain-of-thought substantially improves tasks with verbal representational or knowledge bottlenecks but offers no clear advantage on simple-state tasks such as mazes and Sokoban.

  • Takeaways & Limitations

    Visual generation can serve as a complementary world model for reasoning, with particular value for scenarios grounded in the physical world.

  • Takeaways & Limitations

    The study primarily focuses on spatial and physical reasoning tasks, leaving other task classes for future work.

Abstract

from arXiv · show

Humans construct internal world models and reason by manipulating the concepts within these models. Recent advances in AI, particularly chain-of-thought (CoT) reasoning, approximate such human cognitive abilities, where world models are believed to be embedded within large language models. Expert-level performance in formal and abstract domains such as mathematics and programming has been achieved in current systems by relying predominantly on verbal reasoning. However, they still lag far behind humans in domains like physical and spatial intelligence, which require richer representations and prior knowledge. The emergence of unified multimodal models (UMMs) capable of both verbal and visual generation has therefore sparked interest in more human-like reasoning grounded in complementary multimodal pathways, though their benefits remain unclear. From a world-model perspective, this paper presents the first principled study of when and how visual generation benefits reasoning. Our key position is the visual superiority hypothesis: for certain tasks--particularly those grounded in the physical world--visual generation more naturally serves as world models, whereas purely verbal world models encounter bottlenecks arising from representational limitations or insufficient prior knowledge. Theoretically, we formalize internal world modeling as a core component of CoT reasoning and analyze distinctions among different forms of world models. Empirically, we identify tasks that necessitate interleaved visual-verbal CoT reasoning, constructing a new evaluation suite, VisWorld-Eval. Controlled experiments on a state-of-the-art UMM show that interleaved CoT significantly outperforms purely verbal CoT on tasks that favor visual world modeling, but offers no clear advantage otherwise. Together, this work clarifies the potential of multimodal world modeling for more powerful, human-like multimodal AI.

1 Introduction

The paper frames visual generation as a potential complement to language-centric reasoning because current systems remain limited on physical and spatial tasks, while evidence about its benefits is inconclusive. It formalizes this question through world models, introduces VisWorld-Eval, and finds that interleaved visual-verbal reasoning helps selectively.

  • Current language-centric systems perform strongly in mathematics and programming but still struggle with physical and spatial intelligence.
  • Unified multimodal models add visual generation, enabling explicit visual world modeling alongside verbal reasoning.
  • Prior studies report mixed results and lack a principled basis for determining when visual generation benefits reasoning.
  • The paper formalizes multimodal world models and analyzes visual generation as a reasoning pathway grounded in physical-world tasks.
  • VisWorld-Eval contains seven tasks spanning synthetic and real-world domains, designed to isolate and demand distinct world-model capabilities.
  • Interleaved chain-of-thought substantially improves performance when verbal modeling faces representational or knowledge bottlenecks, but offers no clear advantage on simple-state tasks such as mazes and Sokoban.

2 Related Work

Related work has explored unified multimodal models and visual generation for reasoning, but existing benchmarks and findings have not established consistent conditions under which visual generation helps.

  • World-model research lacks consensus on definitions and methodologies, especially for learning abstract representations across domains and modalities.
  • Unified multimodal models increasingly integrate vision-language understanding and visual generation, either through external modules or a shared backbone.
  • Existing visual-generation reasoning benchmarks report mixed results, while some studies find minimal or negative gains from generated visual reasoning steps.

3 A World Model Perspective on Multimodal Reasoning

The paper models multimodal reasoning as interleaved logical steps and observations over a partially observed world, distinguishing reconstruction and simulation as core world-model capabilities. Its theory decomposes answer error into reasoning and world-modeling errors, motivating the hypothesis that visual generation is especially informative for physical-world tasks.

  • The world is represented as a multi-observable Markov decision process whose hidden states produce verbal or visual views through observation functions.
  • World reconstruction infers complete structure from limited observations and supports novel-view generation, including mental rotation.
  • World simulation predicts how latent states evolve after actions, providing an internal experience of interacting with the world.
  • VisWorld-Eval deliberately curates seven question-answering tasks to isolate the contributions of different world-model capabilities.
  • The reasoning process is formalized as interleaved logical steps and observations, with generated observations produced through reconstruction or simulation.
  • Theoretical analysis decomposes answer error into reasoning and world-modeling errors, while explicit state tracking can reduce reasoning uncertainty at the cost of modeling fidelity.
  • The visual superiority hypothesis states that visual generation yields more informative and knowledge-rich representations than verbal world models for physical-world tasks.

4 Experiment Settings

The experiments evaluate multimodal reasoning across seven capability-focused tasks using controlled CoT variants and post-trained unified multimodal models. VisWorld-Eval measures answer accuracy, while zero-shot VLM results establish current baseline limitations.

  • VisWorld-Eval: Seven question-answering tasks form VisWorld-Eval, targeting specific world-model capabilities across synthetic and real-world domains.Performance is measured by concise, verifiable answer accuracy.
  • World simulation: The suite includes world-simulation tasks such as paper folding, which requires modeling unfolding, symmetry, and spatial transformations.The task asks for hole distributions after sequential folds and hole punching.
  • World reconstruction: Cube 3-view projection tests world reconstruction by requiring an unseen viewpoint to be inferred from isometric and orthographic views.Solving it involves reconstructing a full 3D cube structure and mentally rotating or projecting it.
  • Evaluation protocol: Controlled comparisons use implicit, verbal, and visual world-modeling CoT patterns in supervised fine-tuning data.These patterns enable comparative evaluation of reasoning formulations.
  • Baseline evaluation: Zero-shot advanced VLMs perform suboptimally overall, with especially difficult results on paper folding, ball tracking, cube 3-view projection, and real-world spatial reasoning.Gemini 3 Flash and Gemini 3 Pro outperform other evaluated models but remain far from satisfactory on these challenging tasks.
  • Model training: Experiments post-train BAGEL, a state-of-the-art open-source unified multimodal model, rather than relying on base-model zero-shot performance.Training optimizes verbal and visual generation using supervised fine-tuning, followed by reinforcement learning from verifiable rewards.

5 Experimental Results

Visual world modeling improves reasoning when tasks require rich spatial simulation or reconstruction, but provides little benefit when verbal or implicit representations suffice. Analyses attribute these gains to visual prior knowledge, stronger fidelity, and emergent internal world representations.

  • Visual World Simulation: Interleaved visual-verbal CoT significantly outperforms purely verbal formulations on paper folding, multi-hop manipulation, and ball tracking.These tasks involve difficult coordinate grounding, arithmetic, geometric transformations, or incomplete verbal depictions that can cause ambiguity and hallucinations.
  • Visual World Simulation: Visual world modeling achieves comparable paper-folding performance with more than 4× less SFT data than verbal world modeling.The result indicates substantially higher sample efficiency for visual reasoning on this task.
  • Visual World Reconstruction: Visual world modeling substantially benefits cube 3-view projection and selected visual-scene reconstruction subtasks, while other positional-relationship subtasks show no consistent improvement.Visual rotation benefits from pretraining exposure, whereas current models remain limited in verbal spatial understanding and visual generation quality.
  • Visual World Reconstruction: Across cube-stack difficulty levels, visual world modeling maintains an advantage and improves performance by approximately 10% on out-of-distribution stacks of size six.The advantage persists beyond the training distribution.
  • Visual World Reconstruction: Visual world-model fidelity exceeds 50%, whereas verbal world-model fidelity degrades to near zero on cube 3-view projection.The evaluation compares structural view shapes while ignoring color information; visual reasoning captures geometric transformations more effectively.
  • Tasks Where Visual Modeling Is Unhelpful: Maze and Sokoban tasks show no notable visual-generation improvement because simple state tracking can be handled verbally, with maze reasoning possibly encoded implicitly in hidden representations.Probing supports this account: pretrained models predict masked maze coordinates above random-guess accuracy, with accuracy peaking in middle layers before declining near the output layers.

6 Discussions

The paper frames visual world modeling as especially relevant for physical and spatial reasoning, while identifying scope and methodological limitations for future work.

  • The study primarily targets spatial and physical reasoning, leaving its applicability to other task families open.The authors specifically discuss visual jigsaw and STEM reasoning as future directions.
  • The visual-generation components were not trained with reinforcement learning, although tailored RL could improve world-model fidelity.
  • The analysis of emergent representations for implicit world modeling remains preliminary.
  • The authors propose comparing VLM and UMM internal representations because multimodal generation training may capture complementary world knowledge.
  • Visual world models may support embodied agents by helping them interpret environments, occlusions, and user intentions.
  • Figure 9 illustrates interleaved verbal-visual CoT examples in which generated images serve as world models.

A.1 Informativeness

The theory represents multimodal reasoning as interleaved reasoning steps and observations, then relates explicit world modeling to answer error and reasoning uncertainty.

  • The formal process interleaves logic steps r_i with observations o_i, beginning from an input observation and ending with an answer.
  • The model factorizes joint CoT-and-answer generation into reasoning and world-modeling components.
  • Theorem 3 bounds answer-distribution error by joint CoT-and-answer error, which decomposes into world-modeling errors.
  • Explicit world modeling cannot increase reasoning uncertainty: the uncertainty reduction equals I(o1:i−1; r_i | o0, r0:i−1) and is nonnegative.
  • This uncertainty reduction is bounded by information observations provide about states and information reasoning requires about those states.
  • In fully observable environments with deterministic transitions, explicit world modeling provides no additional reasoning benefit.

A.2.1 General Transfer Learning Analysis

The transfer-learning analysis connects distribution shift and model restriction to fine-tuning risk, yielding a bound that exposes a trade-off between modality complexity and shift.

  • The setup compares pre-training distribution P and fine-tuning distribution Q using population and empirical risks within a constrained parameter neighborhood.
  • Uniform loss shift is controlled by total variation TV(P, Q), linking distribution mismatch to differences in risks.
  • Local sharpness and total variation control parameter drift between pre-training and fine-tuning minimizers.
  • The fine-tuning excess-risk result combines generalization error with bias induced by restricting optimization to a local parameter set.
  • The analysis reveals a trade-off between modality complexity and distribution shift when applied to world-modeling and reasoning policies.

B.1 VisWorld-Eval and Training Data

VisWorld-Eval combines synthetic and real-world spatial tasks with task-specific training CoTs and referenced benchmark data. Its construction includes visual, verbal, and implicit world-modeling protocols.

  • Paper folding tests unfolded hole distributions after grids undergo varied folding steps and shape-specific punching.
  • Multi-hop manipulation uses Blender-rendered colored geometric objects and text instructions for spatially specified transformations.
  • Ball tracking asks which numbered top hole a constant-speed ball reaches after elastic wall reflections.
  • Sokoban and maze tasks use grid-based puzzles, with optimal-search or rule-based CoTs and visual intermediate steps for maze reasoning.
  • Cube 3-view projection asks for visible colored-cube counts from another orthogonal view despite occlusion-related ambiguity.
  • Real-world spatial reasoning adopts MMSI-Bench positional questions and constructs training CoTs with visual-CoT generation, filtering, and rewriting.
  • VisWorld-Eval’s training and test data are summarized with task statistics and corresponding original or referenced benchmarks.

B.2 Model Training

The model-training setup uses supervised fine-tuning and reinforcement learning from verifiable rewards, with training examples covering visual and verbal world modeling across several tasks.

  • Training setup: BAGEL is supervised fine-tuned using 8 GPUs and trained with reinforcement learning from verifiable rewards using 64 GPUs.Hyperparameters for both stages are reported separately in Tables 3 and 4.
  • Training data: The training data includes chain-of-thought examples for paper folding under visual and verbal world modeling.The examples show stepwise reasoning for determining unfolded cutout patterns.
  • Training data: Additional chain-of-thought data covers ball tracking, multi-hop manipulation, mazes, Sokoban, cube 3-view projection, and real-world spatial reasoning.Cube projection examples include both visual and verbal world-modeling variants.
  • Baselines: Qwen-VL baselines use LLaMA-Factory for supervised fine-tuning and verl for reinforcement learning from verifiable rewards.This provides the stated training software configuration for the baseline models.

B.3 Analytic Experiments

The analytic experiments examine sample efficiency, task difficulty and world-model fidelity, and implicit world modeling through hidden representations.

  • Sample efficiency: Sample-efficiency experiments train on randomly subsampled sets of 500 or 1000 examples and evaluate both maximum-difficulty and in-distribution settings.The hard setting uses grid size 8 and four folding steps; the Normal setting samples grid sizes 3–8 and folding steps 1–4.
  • Task difficulties and world model fidelity: Task-difficulty experiments vary cube-stack sizes from 3 to 6, with size 6 out of distribution relative to training data.World-model fidelity is assessed by comparing generated views with ground-truth views using string matching for verbal modeling and Gemini 3 Pro for visual modeling.
  • Implicit world modeling: Implicit world modeling replaces explicit point coordinates with <point>masked<point> during supervised fine-tuning, then probes hidden representations for coordinate information.A two-layer MLP predicts x and y coordinates from representations extracted at the masked-token position.
  • Implicit world modeling: Coordinate prediction for the 5 × 5 mazes is formulated as two 5-way classification tasks, with the average coordinate accuracy reported.The two classification tasks separately predict x and y.

C.1 Full Results on MMSI-Bench

This section reports the complete SFT-trained UMM results for MMSI-Bench positional relationship tasks in Table 5.

  • Full results: All scores for MMSI-Bench positional relationship tasks are reported in Table 5.The table is identified as the full results for SFT-trained UMMs.

C.2 Additional Qualitative Evaluation

Qualitative evaluations show that visual generation can support more faithful spatial and geometric world models, but current UMMs remain limited by generation and viewpoint-related errors.

  • Additional qualitative evaluation: The qualitative evaluation includes trained UMM reasoning examples and highlights failure cases across spatial reasoning and paper-folding tasks.The examples mark hallucinated or incorrect reasoning steps in red and correctly generated visual steps in green where applicable.
  • Real-world spatial reasoning: Visual generation produces more faithful world models than implicit reasoning in real-world spatial tasks, despite blurring and corrupted details.Current models still show limited understanding of positions and directions across viewpoints.
  • Paper folding: Visual world models generate correct intermediate paper-unfolding steps even when verbal geometric reasoning is erroneous.Verbal reasoning about geometric symmetry is prone to hallucinations and inaccurate verbal world modeling.
  • Cube 3-view projection: Visual world models approximately generate novel cube-stack views for an unseen stack size of 6, indicating spatial-transformation prior knowledge.Subtle shape-generation errors and inaccurate color inference nevertheless limit overall task performance.
Loading 2601.19834v1…