PaperlayerView exact revision on arXiv

Source-linked AI summary

From Generation to Simulation: How Far Are World Models from Being True Simulators?

Tong Wang, Huan Deng, Mucheng Yang, Yang He, Xiaohui Kuang, Gang Zhao

arXiv:2608.23070v1cs.AIcs.CV

TL;DR

The paper addresses the lack of a systematic assessment of how close generative world models are to replacing traditional simulators. It maps 200 works onto eight simulator capabilities and finds scenario-specific substitution in interaction and controllability, but persistent gaps in physical guarantees, structured state feedback, and reproducible long-horizon evolution.

  • Problem

    The remaining distance between generative world-model generation and rigorous simulation lacks a systematic capability-based assessment.

  • Method

    The study uses eight traditional-simulator capabilities as an external yardstick and maps a curated 200-paper corpus across latent-dynamics, video-generation, and joint-embedding routes.

  • Results

    World models show functional substitution in interaction and controllability for specific scenarios but remain short of formal physical guarantees, structured state feedback, and reproducible long-horizon evolution.

  • Takeaways & Limitations

    The comparison supports route-aware hybridization and research focused on formalized physics, state feedback, stability, unified action interfaces, and downstream utility.

  • Takeaways & Limitations

    Current evaluations often miss closed-loop cumulative error and the incremental success of deploying policies trained inside world models to real environments.

Abstract

from arXiv · show

With the rapid progress of diffusion models and large-scale video generation, generative world models are increasingly expected to replace traditional simulators, including physics engines, game engines, and reinforcement-learning environments. Yet the remaining distance from generation to simulation lacks a systematic assessment. We present a capability-based study using an external yardstick: eight capabilities of a traditional simulator, namely asset construction, physics engine, interaction, controllability, stability, state feedback, diversity, and evaluation metrics. We trace three main technical routes--latent dynamics, video generation, and joint-embedding prediction--and map exactly 200 representative works published from 2018 to June 2026 onto these capabilities. Our analysis shows that world models have achieved functional substitution in interaction and controllability for specific scenarios, but remain short of traditional simulators in formal guarantees of physical laws, structured state feedback, and reproducible long-horizon evolution. State feedback is the most neglected cross-route shortcoming: only 6 of 163 implementation papers expose a runtime interface for querying entity states or physical parameters. We identify six research directions: formalized physics, a unified action interface, first-class state feedback, long-horizon stability, downstream-utility evaluation, and cross-route hybridization. Project page: https://github.com/AtongWang/world-model-simulators

1 INTRODUCTION

The paper asks whether generative world models can replace traditional simulators and evaluates them against eight simulator capabilities across a 200-paper corpus. It finds relative strengths in interaction, controllability, and stability, but structural gaps in asset construction, physics, and state feedback.

  • Central findings: Controllability, interaction, and stability are identified as relative strengths, whereas asset construction, the physics engine, and state feedback remain structural gaps.This comparison previews the paper’s central finding across the eight capabilities.
  • Research scope: The study compares generative world models with traditional simulators using an eight-dimensional capability yardstick.The capabilities are asset construction, physics engine, interaction, controllability, stability, state feedback, diversity, and evaluation metrics.
  • Research scope: The corpus contains 200 representative works mapped capability by capability across major world-model routes.The evidence map quantifies progress and shortcomings through paper-by-paper coding.
  • Central findings: State feedback is quantified as a neglected capability through a five-level B1–B5 labelling scheme.The scheme is used to characterize the structural absence of runtime state interfaces.
  • Research agenda: The paper proposes six actionable research directions backed by nascent work as evidence of feasibility.The directions are presented as a roadmap for addressing the identified shortcomings.

2 LITERATURE COLLECTION

The survey builds a curated corpus through multi-source search, citation expansion, bibliographic verification, and explicit inclusion criteria. It then maps all 200 records and audits state interfaces across 163 implementation papers, while emphasizing that the resulting distributions measure research attention rather than capability attainment.

  • Search strategy: The literature search combines Google Scholar, arXiv, DBLP, and Crossref to balance frontier coverage with bibliographic verification.The search cut-off is 30 June 2026, with each source serving a complementary role.
  • Search strategy: Citation-network expansion supplements Boolean concept queries because keyword search cannot exhaust a rapidly evolving field.The expansion uses survey references and forward and backward citations of foundational or peripheral works.
  • Corpus construction: Every included paper is cross-verified in DBLP and Crossref, yielding 72 formally published papers among the 200 records.Formal publication requires close title matching, author agreement, and a non-preprint venue.
  • Corpus construction: The inclusion criterion requires action or conditional input followed by a forward rollout that generates future environment states usable for interaction, control, or planning.Pure generation backbones, representation learning without action conditioning, trajectory-only predictors, and nongenerative “world model” work are excluded.
  • Corpus overview: Research attention is highly uneven, with controllability, interaction, and stability accumulating far more papers than asset construction, physics, and state feedback.The corpus is also increasingly preprint-dominated, with 128 of 200 records remaining arXiv preprints or other records.
  • Analysis procedure: The analysis annotates all 200 records and audits five state-interface variables across 163 implementation papers.The audit covers ego information, sensor output, reward and termination, runtime entity or physics annotations, and closed-loop interaction.

3 BACKGROUND

The background distinguishes latent-dynamics, video-generation, and joint-embedding predictive routes by their prediction spaces and simulator interfaces. It also defines the traditional-simulator yardstick in terms of controllable assets, physical evolution, interaction, stability, state feedback, diversity, and evaluation.

  • Technical routes: Latent-dynamics models roll compact encoded states forward for decoding and control, whereas video-generation models predict future observations in pixel or token space.The two routes expose different interfaces: compact states support imagination and control, while rendered observations support direct inspection.
  • Technical routes: Joint-embedding predictive models forecast future abstract representations against target embeddings without reconstructing pixels.This route provides an interface suited to efficient downstream prediction and planning.
  • Technical routes: JEPA-based V-JEPA 2 combines large-scale video pretraining with limited robot-data fine-tuning for zero-shot Franka manipulation.Its latent-space planning takes about 16 seconds per action versus about 4 minutes for comparable pixel-generation planning, a roughly 15× speed advantage.
  • Technical routes: Route convergence since 2024 combines diffusion with autoregression, generative models with JEPA, and explicit 3D structure with implicit generation.These examples indicate increasing hybridization across prediction spaces and representations.
  • Simulator yardstick: Traditional simulators provide independently editable assets, numerically solved physical dynamics, precise interaction and control, reproducible evolution, structured state feedback, diversity, and evaluation metrics.State feedback includes ego pose, sensor data, task-evaluation signals, runtime entity or physics annotations, and closed-loop interaction.
  • Simulator yardstick: A rigorous simulator is defined as T : S × A →Π(S × F), with physically consistent, temporally consistent, statistically reproducible evolution supporting closed-loop interactive evaluation.F is a structured state-feedback space containing at least B1 through B4.

4 WORLD-MODEL–SIMULATOR COMPARISON

World models offer advantages in learned scene generation and visual realism, but remain short of rigorous simulators in formal physical guarantees, structured feedback, and reproducible evolution. Across capabilities, research attention is concentrated on controllability, interaction, and stability while foundational asset and physics functions remain underdeveloped.

  • High-level comparison: World models learn environment dynamics without manual geometric modelling or physical-parameter calibration, and diffusion systems can approach photographic visual realism.In GameNGen’s human study, 58–60% of subjects could distinguish real from generated DOOM screenshots.
  • High-level comparison: 45 of 163 implementation papers report sensor-level outputs beyond RGB, leaving world models with a sparse information interface compared with traditional simulator APIs.Traditional simulators expose pose, depth, LiDAR, semantics, collision events, rewards, and labels.
  • Research attention: 125 papers address controllability, compared with 80 each for interaction and stability, while asset construction, physics engine, and state feedback remain concentrated in the 35–45 range.These counts measure principal-contribution research attention across the eight capabilities.
  • Asset construction: 38 of 200 papers list asset construction as a principal contribution, despite its importance for creating editable maps, objects, materials, and environments.Traditional pipelines provide controllable, decomposable instances but require costly human artists and modelling experts.
  • Asset construction: Structured geometric conditions improve generation precision and controllability, but one-off inputs such as BEV layouts and point clouds cannot be queried or edited during generation.World models can also generate navigable worlds from sparse input, yet many outputs remain implicitly stored and lack instance-level decomposability or standard export formats.

4. PHYSICS COMPARATOR

World models increasingly support interactive, controllable rollouts, but their interaction depth and physical reliability remain uneven. Progress spans camera navigation, object manipulation, latent actions, and hybrid representations, while persistent state and cross-level control remain open problems.

  • Interaction: 40.0% of the 200 papers list interaction as a principal contribution dimension, making it the second most studied capability.The interaction spectrum extends from camera control to object manipulation and multi-agent coordination.
  • Camera-level interaction: About 20-FPS real-time interaction demonstrates functional neural rendering substitution in a constrained single-game domain.GameNGen uses RL-generated VizDoom data and an autoregressive diffusion model with four-step DDIM sampling on one TPU-v5.
  • Camera-level interaction: 40 FPS at 720p with minute-level memory consistency shows that camera interaction can reach industrial-scale throughput and temporal persistence.Matrix-Game 3.0 combines residual-prediction consistency training, autoregressive distillation, and decoder pruning.
  • Object-level interaction: Object-level interaction remains less frequently demonstrated because persistent object state, dense event timing, and contact consequences must be modeled beyond viewpoint changes.ActWorld addresses the data bottleneck with 100k annotated human–object interaction videos.
  • Controllability: Explicit control is precise but transfers poorly, language expresses intent ambiguously, and latent actions generalize across environments without direct interpretability.Cross-level action mapping and temporally composed actions remain open because current systems more often continue sequences open-loop than use closed-loop state feedback.

4.5 Stability

Stability is the dividing line between demonstrative generation and engineering simulation: world-model errors accumulate over long horizons, while current methods target drift through training, memory, and persistent state.

  • Stability: 80 of 200 papers (40.0%) identify stability as a principal contribution dimension, reflecting autoregressive drift as a major engineering obstacle.Each step’s small error can accumulate into a cycle of deviation and further error during long rollouts.
  • Self-Adversarial Training: Self-forcing conditions training on the model’s own imperfect outputs, teaching recovery from erroneous context instead of relying only on clean history.This directly addresses train–test mismatch and improves long-horizon geometric consistency.
  • Memory: Implicit memory retrieves view-relevant internal states but may fade with horizon length, whereas explicit 3D memory stores observations in a persistent geometric map.Explicit memory improves persistence but incurs storage and dynamic-scene update costs.
  • Evaluation: Long-horizon stability claims lack a unified protocol across architectures, scene complexities, and action-sequence lengths.Existing reports also provide little comparable evidence on seeds, deterministic kernels, tolerance bands, or repeat-run variance.
  • Causal Consistency: Causal consistency requires an object’s position, orientation, or on/off state to remain unchanged when the camera looks away and later returns.Persistent state methods prevent moved objects from snapping back to their former locations.

4.6 State Feedback

State feedback is a major simulator capability gap: world models often produce visual or estimated structure, while traditional simulators expose structured, queryable state through APIs.

  • State Feedback: 45 of 200 papers (22.5%) list state feedback as a principal contribution dimension, indicating limited attention to structured state interfaces.Traditional simulators expose pose, sensors, events, rewards, labels, and entity information through APIs.
  • Sensor-Space Feedback: Multi-task heads can decode aligned RGB, depth, semantics, occupancy, and LiDAR from a shared evolving representation.These outputs provide complementary sensor and scene views, but remain distinct from directly queried simulator state.
  • Interface Audit: The five-variable audit records confirmed presence in 65 papers for B1, 45 for B2, 20 for B3, six for B4, and 87 for B5.B4 requires runtime, queryable named entity or physics annotations rather than sensor estimates or latent vectors.
  • Structured Outputs: Structured vector outputs preserve deterministic spatial information that would otherwise be lost when recovering structure from pixels.Vector maps can directly encode quantities such as an agent’s position and velocity for driving simulation.
  • Physical and Latent State: Physics-based state provides interpretable quantities directly consumable by downstream controllers, while latent feedback is efficient but difficult for humans to audit.JEPA-style latent representations can support control without exposing readable state, creating an auditability gap.

4.7 Diversity

Diversity is a principal potential advantage of generative world models, arising from scale, compositional recombination, and parametrically controlled variation, but it trades off against precise controllability.

  • Scale-Driven Diversity: Large-scale heterogeneous data broaden a pretrained world model’s support and can yield coverage beyond scene types emphasized during training.This scale-driven route is illustrated as diverse, evolving rollouts from data across environments, conditions, and embodiments.
  • Compositional Diversity: Compositional generation recombines reusable entities, actions, and conditions into scenarios not observed as complete examples during training.Language instructions can specify primitives such as a red block and grasp, enabling novel combinations.
  • Diversity–Controllability Trade-off: Higher diversity generally means poorer control, while more precise control narrows the generated space.Parametric controls can allocate variation across texture, lighting, camera pose, trajectory, and scene layout.
  • Open Challenge: Current designs must share capacity between representing many possible worlds and specifying which world the user wants.The paper identifies structured controllable-diversity mechanisms as an open challenge.
  • Evaluation: Evaluation must move beyond visual similarity because visually strong generation can remain physically impossible or fail during closed-loop interaction.The paper distinguishes generation quality from downstream simulator utility and highlights fragmented benchmarks and open-loop bias.

5 FUTURE DIRECTIONS AND CHALLENGES

The paper identifies a structural gap between current world models and rigorous simulators, then proposes targeted directions spanning physics, interfaces, stability, evaluation, and hybrid architectures.

  • Challenges: The gap between world models and rigorous simulators is structural rather than a marginal quantitative issue.The comparative analysis motivates targeted research on the capabilities that remain incomplete.
  • 5.1 Formalized Physics: Formalized physics should fuse differentiable engines with generative pipelines as conditional regularizers and sources of structured state feedback.The central challenge is retaining open-domain generalization while enforcing step-wise verifiable conservation constraints.
  • 5.2 Unified Action Interface: A unified action interface must bridge precise but poorly generalizing controls, readable but imprecise language, and general but unreadable latent actions.Multi-scale action representations and language-informed translation are proposed as complementary directions.
  • 5.3 First-Class State Feedback: First-class state feedback should expose named entity and physical fields that remain consistent with generated observations throughout a rollout.Sensor estimates, explicit physical state, and representation translators target different levels of interface quality.
  • 5.4 Long-Horizon Stability: Long-horizon stability requires both reset-and-anchor mechanisms and standardized measurements of error growth and repeat-run reproducibility.Common protocols are needed for A/B testing, regression testing, and safety analysis.
  • 5.5 Downstream-Utility Evaluation: Evaluation should prioritize incremental success rate: the real-environment success of an RL policy trained inside the world model.This shifts assessment from generation quality toward downstream utility and absolute comparison with traditional simulators.
  • 5.6 Cross-Route Hybridization: Cross-route hybridization can combine diffusion’s vision and interaction, latent dynamics’ control, JEPA’s efficiency, and explicit 3D/4D geometric consistency.Embedding differentiable physics into diffusion is one concrete fusion direction.

6 CONCLUSION

The study finds that world models can functionally substitute for traditional simulators in interaction and controllability under restricted conditions, but remain a critical step away in the strict sense. The central gaps concern physics guarantees, structured state feedback, long-horizon stability, and downstream-utility evaluation, motivating six research directions.

  • World models demonstrate functional substitution in interaction and controllability for specific scenarios, but not as rigorous simulators in the strict sense.Their open-domain generalization and visual quality are advancing rapidly, yet the conclusion remains conditional.
  • 45 of 163 papers implement B2, while only 6 expose B4, highlighting limited coverage of structured state feedback.The implementation audit identifies B2 and B4 as specific evidence of the remaining state-feedback gap.
  • Route comparisons reveal trade-offs: autoregressive models lead interaction coverage, diffusion models lead high-fidelity visual generation, and JEPA offers roughly 15× speed.Latent dynamics supports control decisions, while explicit 3D/4D methods provide stronger geometric consistency but weaker appearance detail than pure diffusion.
  • The proposed roadmap includes formalized physics, a unified action interface, first-class state feedback, long-horizon stability, downstream-utility evaluation, and cross-route fusion.Each direction is supported by nascent work as evidence of feasibility.
  • The paper proposes its eight-capability comparison and full-corpus evidence map as a coordinate system for tracking progress toward consumable, verifiable, reproducible world state.It frames this shift as moving beyond generating good-looking video toward using a generative model as a world.

APPENDIX A COMPLETE LIST OF SEARCH QUERIES

The appendix records the complete keyword-search protocol used to assemble the study's corpus. Queries were ranked, truncated, deduplicated, manually screened, and closed on 30 June 2026.

  • Each search expression was assigned a stable identifier and scope label so coverage can be inspected without parsing an undifferentiated code block.The Boolean expressions are reproduced verbatim apart from line wrapping.
  • Results were ranked by relevance, truncated to top entries per query, deduplicated against the accumulated corpus, and manually screened.
  • The search cutoff was 30 June 2026.

A.1 Mainstream Routes and Application Settings

The mainstream-route query set spans interactive video, playable game engines, robot manipulation, latent-action and autoregressive models, driving, embodied physics, diffusion simulators, evaluation, surveys, streaming, and explorable 4D worlds.

  • Mainstream routes and application settings: M1–M9 cover interactive video, playable neural game engines, robot manipulation, latent-action models, autoregressive video, driving, embodied physics, diffusion simulators, and evaluation benchmarks.
  • Search-query design: The queries combine multiple concept terms with Boolean AND rather than relying on a single broad term.
  • Mainstream routes and application settings: M10–M12 extend coverage to general surveys, real-time generative streaming, and explorable 4D worlds.

A.2 Supplementary Representations and Capabilities

The supplementary query set targets representations and capabilities that broad world-model searches may miss, including predictive embeddings, explicit geometry, memory, multimodality, controllability, and physics-oriented evaluation.

  • Supplementary representations: S1–S3 target joint-embedding prediction, self-supervised representation learning, and language models for planning and simulation.
  • Supplementary representations: S5–S8 cover 3D Gaussian representations, neural radiance fields, differentiable physics, and neural rigid-body dynamics.
  • Supplementary capabilities: S9–S16 address long-horizon consistency, persistent spatial memory, embodied navigation, occupancy, multimodal prediction, physical plausibility, foundation models, and camera- or action-controllable video.
  • Execution protocol: The protocol also includes execution details for applying expressions across arXiv, Google Scholar, DBLP, and Crossref, with citation-chain expansion handled separately.
Loading 2608.23070v1…