PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond

Meng Chu, Xuan Billy Zhang, Kevin Qinghong Lin, Lingdong Kong, Jize Zhang, Teng Tu, Weijian Ma, Ziqi Huang, Senqiao Yang, Wei Huang, Yeying Jin, Zhefan Rao, Jinhui Ye, Xinyu Lin, Xichen Zhang, Qisheng Hu, Shuai Yang, Leyang Shen, Wei Chow, Yifei Dong, Fengyi Wu, Quanyu Long, Bin Xia, Shaozuo Yu, Mingkang Zhu, Wenhu Zhang, Jiehui Huang, Haokun Gui, Runyi Li, Chenyu Tang, Dong Huang, Xuhang Chen, Rui Liu, Chengzu Li, Shiyi Du, Xu Huang, Haoxuan Che, Long Chen, Qifeng Chen, Wenya Wang, Wenxuan Zhang, Xiaojuan Qi, Yang Deng, Yanwei Li, Mike Zheng Shou, Zhi-Qi Cheng, See-Kiong Ng, Ziwei Liu, Philip Torr, Jiaya Jia

arXiv:2604.22748v3cs.AI

TL;DR

World-model research lacks a common technical language because communities use different meanings and evaluation criteria. The paper synthesizes the field through a three-level capability hierarchy crossed with four governing-law regimes, finding a roadmap from prediction to simulation and evidence-driven model revision. It also identifies evaluation challenges and regime-specific limits that constrain current systems.

  • Problem

    World models have diverse meanings and evaluation criteria across communities, making results difficult to align and compare.

  • Method

    The paper proposes a levels × laws taxonomy with L1 Predictor, L2 Simulator, and L3 Evolver crossed with physical, digital, social, and scientific regimes.

  • Results

    The survey synthesizes a cross-domain roadmap connecting prediction, decision-usable simulation, and evidence-driven model revision across world-modeling systems.

  • Takeaways & Limitations

    Capability claims should be evaluated through decision-centric principles and regime-specific constraints, with research agendas spanning causal representations, law-consistent rollout, and safe model revision.

  • Takeaways & Limitations

    Evaluation remains constrained by benchmark saturation, evaluation gaming, costly and variable human judgment, and limited meta-evaluation.

Abstract

from arXiv · show

As AI systems move from generating text to accomplishing goals through sustained interaction, the ability to model environment dynamics becomes a central bottleneck. Agents that manipulate objects, navigate software, coordinate with others, or design experiments require predictive environment models, yet the term world model carries different meanings across research communities. We introduce a "levels x laws" taxonomy organized along two axes. The first defines three capability levels: L1 Predictor, which learns one-step local transition operators; L2 Simulator, which composes them into multi-step, action-conditioned rollouts that respect domain laws; and L3 Evolver, which autonomously revises its own model when predictions fail against new evidence. The second identifies four governing-law regimes: physical, digital, social, and scientific. These regimes determine what constraints a world model must satisfy and where it is most likely to fail. Using this framework, we synthesize over 400 works and summarize more than 100 representative systems spanning model-based reinforcement learning, video generation, web and GUI agents, multi-agent social simulation, and AI-driven scientific discovery. We analyze methods, failure modes, and evaluation practices across level-regime pairs, propose decision-centric evaluation principles and a minimal reproducible evaluation package, and outline architectural guidance, open problems, and governance challenges. The resulting roadmap connects previously isolated communities and charts a path from passive next-step prediction toward world models that can simulate, and ultimately reshape, the environments in which agents operate. Code and resources are available at: https://github.com/matrix-agent/awesome-agentic-world-modeling.

1 Introduction

World models are used across communities with different technical meanings and evaluation criteria, creating fragmentation. This survey addresses that gap with a capability-based, cross-domain framework organized by three levels and four governing-law regimes.

  • World models support prediction, control, simulation, and scientific reasoning, but their technical meaning varies across reinforcement learning, vision, language agents, and AI for science.
  • Gaps in existing surveys: Existing surveys largely organize the field by modality or domain, leaving capability progression across modalities insufficiently captured.
  • Contribution: The survey synthesizes research communities that existing surveys treat separately and provides testable boundary conditions for interpreting and comparing world-modeling systems.
  • Capability hierarchy: The survey defines L1 Predictor as one-step prediction, L2 Simulator as long-horizon action-conditioned rollout with constraint satisfaction, and L3 Evolver as evidence-driven model revision.
  • Cross-domain framework: Its taxonomy crosses capability level with physical, digital, social, and scientific governing-law regimes, clarifying both shared principles and domain-specific constraints.

2 Preliminaries

The paper frames world modeling as a three-level capability hierarchy, progressing from one-step prediction to counterfactual simulation and evidence-driven model revision. It argues that representation is foundational because latent dynamics support L1–L2, while L3 requires revisable symbolic structure.

  • Capability hierarchy: The paper organizes world modeling into L1 Predictor, L2 Simulator, and L3 Evolver, moving from pattern recognition to rollout and adaptive model evolution.The hierarchy is presented as a capability progression rather than a claim that machine-learning systems literally implement philosophical programs.
  • L1 Predictor: L1 predicts one-step latent transitions from past observations and actions, but its learned regularities are fragile under distribution shift.The paper associates this fragility with the i.i.d. assumption that the future will resemble the past.
  • L2 Simulator: L2 adds intervention and counterfactual reasoning by composing local transitions into coherent future rollouts under chosen actions or hypothetical initial conditions.Its reliability remains model-relative, so internally coherent trajectories can drift from ground-truth dynamics outside the training manifold.
  • L3 Evolver: L3 revises the model itself when systematic prediction failures reveal that parameter updates are insufficient and the model class is too narrow.The proposed design–execute–observe–reflect loop allows evidence to drive revision across the model stack.
  • Representation: Representation is treated as a foundational question: latent dynamics scaffold L1 and L2, whereas genuine revision of governing laws requires a symbolic substrate.This makes the L1→L2→L3 progression concern not only rollout depth but also how laws are discovered, composed, and revised.

3 L1 Predictor: Local Markov Prediction

L1 Predictor models local, one-step transitions in a learned internal state to support agent decision-making, using forward dynamics alongside supporting operators. Its local accuracy alone does not guarantee coherent, decision-usable behavior over long horizons.

  • Definition: L1 predicts the next state from a learned internal state and action, representing one edge in the agent–environment transition process.The internal state may summarize observations and actions and approximate a latent environmental state or belief.
  • Definition: An L1 world model supports one-step or short-horizon decision-making rather than merely predicting the next signal.Its epistemic stance extracts observed regularities without claiming causal necessity.
  • Component factorization: L1 factorizes into state inference, forward dynamics, observation decoding, and inverse dynamics, with forward dynamics as the core transition operator.The other operators make the learned latent state usable for transition modeling and control.
  • State inference: State inference compresses observations and temporal context into a compact belief-like state that preserves information needed for prediction and control.Recurrent state-space models combine deterministic memory with stochastic latent uncertainty when a single observation is insufficient.
  • Approaches: Representative state-learning approaches include contrastive prediction, masked-region latent prediction, control-oriented auxiliary objectives, and scientific latent-state inference.Examples include CPC, JEPA variants, DeepMDP, E2C, RSSM, and protein-structure prediction systems.
  • Discussion: L1 alone does not ensure coherent long-horizon behavior because one-step errors can compound and local predictive quality may not yield decision-usable composition.This motivates distinguishing L1 from multi-step simulation based on rollout accuracy and constraint adherence.

4 L2 Simulator: Decision-Usable Multi-Step Simulation

L2 Simulator systems turn one-step transition operators into coherent, action-conditioned multi-step rollouts that agents can query for planning under governing-law constraints. Decision usability requires long-horizon coherence, intervention sensitivity, and constraint consistency, while evaluation must look beyond visual plausibility toward planner-relevant structure.

  • L2 definition: L2 answers which future trajectory is likely under a candidate action sequence and task constraints, enabling agents to compare plans before acting.The simulator provides an imagination of the future without requiring real-environment interaction.
  • L2 definition: L2 rollouts represent trajectories τ conditioned on an initial state, action sequence a1:H, and optional regime constraints c.The formulation specifies τ = (z1, ..., zH) and treats intervention-structured rollouts as aligned with interventional causal reasoning.
  • Requirements for elevation: L2 requires long-horizon coherence, intervention sensitivity, and constraint consistency rather than one-step predictive quality alone.These conditions require usable multi-step trajectories, stable changes under counterfactual inputs, and respect for physical, digital, social, or scientific laws.
  • Requirements for elevation: Composing local operators can fail through compounding errors, action-insensitive rollouts, and violated domain invariants, making trajectories misleading for planning.These failure modes show why independent one-step optimization does not automatically produce a decision-usable simulator.
  • Evaluation and limitations: Visual plausibility alone is insufficient because intervention sensitivity is fragile, perceptual metrics can overstate coherence, and constraint consistency is difficult to verify from frames.The paper points to controllability, physics consistency, commonsense reasoning, and causal-coherence evaluations as complementary evaluation targets.
  • Cross-domain analysis: Across regimes, useful simulators model constraints more directly than appearance: geometry and contact in physics, state machines in software, norms in social worlds, and evidence chains in science.Making constraints explicit, loggable, replayable, and regressable can improve long-horizon stability more than increasing perceptual fidelity.

5 L3 Evolver: Evidence-Driven Model Revision

L3 Evolvers close a design–execute–observe–reflect loop that actively acquires evidence, diagnoses prediction failures, and revises world-model parameters, structures, or assumptions. The framework is most mature in scientific settings, partial in digital environments, and largely open or aspirational in social domains.

  • Formal definition: L3 extends fixed-model simulation with closed-loop, evidence-driven revision through the full design–execute–observe–reflect cycle.New evidence is actively acquired to challenge and revise the model across iterations.
  • Revision triggers and evolution policy: Persistent anomalies that resist local adjustment signal epistemic gaps and may require structural changes to the model’s architecture, inductive biases, or hypothesis space.Small anomalies can trigger online adaptation, while persistent anomalies motivate parameter updates or deeper structural revision.
  • Boundary conditions: L3 distinguishes model evolution by deployment-time plasticity, capability accumulation, and active experimentation rather than passive data consumption.Updates may modify parameters, add architecture or capacity, or expand the hypothesis space.
  • Domain maturity: L3 is most tractable in highly instrumented domains with rapid feedback and well-defined evaluation criteria, while support remains uneven across regimes.Scientific settings provide the clearest demonstrations; social, code, and embodied environments remain partly empirical and partly prospective.
  • Maturity across regimes: Scientific L3 is established but constrained by instrument access and real-data budgets, whereas digital L3 is partial because active experiment design is often absent.Scientific systems benefit from structured feedback and unambiguous anomaly signals, while digital systems commonly rely on regression testing without active information expansion.
  • Maturity across regimes: Social L3 remains aspirational because attribution is ambiguous, behavioral ground truth is noisy, and ethical constraints limit experimentation.Persistent, validated revision of social world models from deployment evidence remains largely open.

6 Evaluations

The paper argues that world-model evaluation should prioritize decision-centric properties over perceptual realism, using long-horizon coherence, intervention sensitivity, and constraint consistency across governing-law regimes. It shows that protocols—not benchmarks alone—determine whether systems are tested as L1 Predictors, L2 Simulators, or L3 Evolvers, while current evaluation remains incomplete and uneven.

  • Evaluation Principles: Decision-centric evaluation targets long-horizon coherence, intervention sensitivity, and constraint consistency across four governing-law regimes.These conditions assess whether rollouts remain usable, respond meaningfully to interventions, and respect domain laws.
  • Motivation: Standard generative metrics provide weak evidence of agentic capability because visually convincing rollouts can still fail during planning.Such failures include hallucinated object dynamics, action-insensitive transitions, and subtle physics violations.
  • Evaluation Principles: Evaluation should assess trajectory-level rollouts and their reliability for planning rather than isolated one-step predictions.Mean success rates can also hide high variance across task instances.
  • Aggregate Metrics: Action Success Rate measures whether model-based rollouts support successful real-environment decisions, while Counterfactual Outcome Deviation measures meaningful response to action interventions.ASR evaluates task achievement; COD compares outcomes under policies differing at one intervention step, using a task-relevant distance.
  • Capability Levels: The same benchmark can test L1, L2, or L3 depending on whether its protocol demands local prediction, decision-usable simulation, or evidence-driven model revision.For RoboCasa, L2 adds full-task execution under perturbations and metrics such as long-horizon success, catastrophic action fraction, and recovery rate.
  • Coverage and Gaps: Current evaluation is concentrated at L1, while L2 protocols are nonstandard and L3 infrastructure is essentially nonexistent outside autonomous science.No single benchmark covers all capabilities, and benchmark saturation, gaming, human-evaluation costs, and weak meta-evaluation remain challenges.

7 Architectural and Computational Considerations

The paper frames world-model construction as a coordinated choice of representation, dynamics, and control interface, constrained by capability level, governing-law regime, and deployment latency. It emphasizes composed systems and regime-specific trade-offs rather than interchangeable components.

  • Design Axes: World-model systems combine representation, dynamics, and control-interface choices that determine reachable capability levels and effective governing-law regimes.Each design choice carries distinct trade-offs, so components are not interchangeable.
  • Representation: Representation choices trade interpretability and hard constraints against flexibility for high-dimensional multimodal inputs.Symbolic or programmatic states support constraint enforcement but require manual engineering and cover prespecified state spaces, whereas latent continuous states handle richer inputs.
  • Dynamics: Stochastic latent dynamics express uncertainty and multimodality but may degrade or become miscalibrated over long horizons, while deterministic value-aware dynamics prioritize downstream value prediction.The latter trades generative flexibility for direct transition optimization toward task-relevant predictions.
  • Control Interface: MPC-style control enables rapid closed-loop correction, whereas tree search supports counterfactual branching but can amplify model errors and exploit benchmark loopholes.Online replanning incurs compute and latency pressure; systematic look-ahead increases branching demands.
  • Governing-Law Regimes: Regime-specific bottlenecks differ: physical systems require geometry, contact, stability, and fast correction; digital systems prioritize state-machine and branch consistency.Social systems must preserve identity and relational state while handling ToM costs that grow with modeled agents, and generative systems trade visual fidelity against action controllability.
  • Composed Systems: Mature systems often stack symbolic planning, replayable environments, and short-horizon continuous control, making the composed system the relevant unit of analysis.This layered design supports task decomposition, failure attribution, and real-time correction.
  • Computational Considerations: Deployment latency budgets constrain viable dynamics classes and control interfaces, with budgets ranging from sub-100 ms in real-time robotics to minutes in offline scientific planning.These are deployment ranges rather than measured benchmark results, and throughput depends on model, hardware, batching, simulation, and verification.

8 Trends & Open Problems

The paper identifies a transition from isolated predictors toward agent-facing simulators that must remain faithful to governing laws over extended horizons, and it catalogs open problems for adaptation, evaluation, security, and broader model revision. It further proposes meta-world modeling as an extension beyond revising a single world model.

  • Physical World Modeling: Current systems often achieve visual plausibility without physical faithfulness, with long-horizon error accumulation remaining a central weakness.PhyWorldBench reports that the best of twelve frontier models achieves a 0.262 success rate on conservation-law and object-permanence probes.
  • Video World Modeling: Metric-aware video world modeling requires temporal metric estimation, composition of short-step predictions, identity preservation, and instruction-grounded motion.These abilities must work together to extend geometry-grounded editing from image pairs to temporally coherent video.
  • Digital Worlds: Software world models lack persistent beliefs over hidden backend state and asynchronous transitions, and realistic asynchronous failures reduce task success across state-of-the-art agents.The relevant hidden state includes server sessions, database rows, in-flight requests, and background processes.
  • Social Worlds: Social simulations face mode collapse, weak multi-turn persona calibration, and continual-learning challenges when societal transition functions shift.Large simulations reproduce phenomena such as polarization and governance formation but cannot autonomously detect changed dynamics or revise without catastrophic forgetting.
  • Scientific Worlds: Scientific surrogates trained on simulation data degrade on real measurements, with prediction error plateauing without real-data calibration.The paper treats this surrogate-to-reality gap as analogous to sim-to-real transfer in robotics.
  • Cross-Regime Challenges: Across regimes, deployment shift requires online mechanisms because interfaces, physical properties, social norms, and scientific instruments can change.The paper presents deployment shift as a recurring cross-domain challenge for robust world modeling.
  • Security and Safety: World-model security failures can violate physical, digital, social, or scientific laws, making corrupted imagined rollouts a high-leverage attack surface for agent decisions.The paper calls for regime-specific threat models and benchmarks, noting that current security evaluation is nascent and relies largely on not-yet-peer-reviewed preprints.
  • Beyond L3: Meta-world modeling would reason over spaces of possible transition functions and alternative governing laws rather than only revising a model of one observed world.Possible mechanisms include program synthesis, open-ended evolution, and procedural world generation, but the capability remains an open question.

9 Conclusion

The paper concludes that world modeling is best organized by three capability levels and four governing-law regimes. It pairs this taxonomy with evaluation guidance while identifying symbolic law discovery as a central unresolved challenge.

  • The taxonomy combines three capability levels—Predictor, Simulator, and Evolver—with four governing-law regimes: physical, digital, social, and scientific.
  • L1 predicts local transitions, L2 produces law-consistent long-horizon rollouts, and L3 revises dynamics through experiments and evidence when predictions fail.
  • Governing-law regimes share the capability hierarchy but differ in valid rollouts, law violations, available evidence, and domain-specific modeling requirements.
  • The paper proposes decision-centric evaluation, a minimal reproducible evaluation package, and architectural guidance for representations, dynamics, and control.
  • A major open problem is developing world models that discover and manipulate explicit symbolic governing laws rather than only encoding invariances latently.
  • The roadmap moves beyond larger predictors toward models that simulate dynamics and evolve through active trial-and-error loops.

A Philosophical Motivations for Hierarchical World Modeling

This appendix situates hierarchical world modeling within philosophical motivations for prediction, induction, and model revision.

  • The appendix expands the philosophical motivations behind the L1/L2/L3 hierarchy introduced in Section 2.1.

A.1 L1: Inductive Priors and the No-Free-Lunch Theorem

The appendix connects L1 world modeling to inductive priors, predictive cognition, counterfactual simulation, and theories of scientific model revision. It frames world models as action-conditioned dynamics whose reliability matters for downstream planning.

  • A.1 L1: Inductive Priors and the No-Free-Lunch Theorem: Inductive priors constrain learning because no learner is universally superior without structural assumptions.
  • Predictive coding and active inference describe perception, planning, and exploration as processes coupled to internal prediction and error minimization.
  • L1 systems such as Dreamer-style models learn temporal regularities from trajectories and rely on next-step accuracy.
  • L2 counterfactual reasoning focuses on minimally intervened, coherent trajectories rather than arbitrary possible futures.
  • MuZero operationalizes this idea by using Monte Carlo Tree Search over learned dynamics to explore action sequences for decision-making.
  • Visually fluent but incomplete simulators can generate coherent trajectories that drift beyond their training manifold, motivating reliability checks for downstream use.
  • L3 revision is complicated by confirmation holism, so held-out probes and targeted ablations help diagnose which component requires structural change.
  • World models answer action-conditioned transition queries, while planners consume their predictions to choose actions; L1/L2/L3 classify query depth and reliability, not planner presence.

C.3 Social Simulation and Multi-Agent Systems

Social simulation combines structured representations, scalable sandbox architectures, and empirical tests of emergent behavior. Evidence shows that cooperation varies sharply across models, while norms and collective phenomena can emerge and be evaluated against real-world patterns.

  • C.3 Social Simulation and Multi-Agent Systems: Structured ToM methods represent per-character beliefs or perspective-taking processes to support multi-step social reasoning.
  • Project Sid scaled a modular Minecraft architecture to 1,000 agents across six towns, producing specialization, social networks, governance, and cultural transmission.
  • Only 2 of 15 LLMs achieve sustainable cooperation in commons-dilemma scenarios.
  • Norms and conventions can emerge with group-level tipping points, and Melting Pot offers more than 50 substrates for evaluating cooperation, competition, deception, and coordination.
  • S3 and SocioVerse evaluate social simulations against real-world phenomena, including elections, information propagation, breaking-news responses, and economic surveys.
  • Institutional architectures provide explicit representations of mental states, social obligations, and formal commitments that current LLM agents often lack.
  • Neural dynamical systems such as DyNeMo support forward latent-state simulation and prediction of responses to external interventions without direct experimentation.
  • Physics-informed neural operators and molecular graph models connect learned dynamics with governing PDE constraints or quantum-chemical properties.

D.2 Strategy Revision in a Service Environment

The examples illustrate evidence-driven strategy revision across social, digital, and scientific environments. Each workflow detects a mismatch, attributes its cause, revises the model or policy, and validates the change.

  • Service environment: Across 50 cancellation dialogues, discount offers retained users 20% of the time versus the model’s predicted 70%.
  • Service environment: Conversation analysis attributed the mismatch to users’ service-quality frustration rather than price sensitivity.
  • Service environment: The revised strategy distinguishes user intents, replacing discounts with acknowledgment and escalation for quality-frustrated users while retaining discounts for price-sensitive users.
  • Service environment: Validation uses an A/B test over the next 100 interactions to measure retention, satisfaction, and new failure modes.
  • Digital environment: In software, permission-denied errors trigger privilege checks, alternative installation paths, re-verification, regression testing, and cross-environment replay.
  • Scientific environment: In materials science, an unexpected mixed phase leads to temperature-aware Bayesian revision and a follow-up experiment, after which prediction and held-out calibration improve.

E Additional Details of Evaluation and Benchmarks

The evaluation framework tests whether world-model rollouts remain coherent, action-sensitive, and consistent with governing constraints. It emphasizes horizon-dependent diagnostics, counterfactual interventions, and law-specific verification.

  • L2 boundary conditions: Long-horizon coherence measures whether task-relevant performance degrades predictably as rollout horizon increases.
  • L2 boundary conditions: Fixed-horizon success rates often omit degradation curves, leaving the relationship between performance and horizon uncharacterized.
  • L2 boundary conditions: Intervention sensitivity tests whether changing an action or initial condition produces a task-relevant trajectory change.
  • L2 boundary conditions: Counterfactual divergence testing uses action sensitivity ratio and normalized counterfactual outcome divergence to quantify action-conditioned changes.
  • L2 boundary conditions: Constraint consistency evaluates whether complete rollouts respect governing laws, exposing exploitability and calibration failures under distribution shift.

E.4 World-Model-Specific Evaluation

World-model-specific benchmarks separate perceptual realism from action-conditioned usefulness and expose failures that per-step diagnostics can miss. The evaluation scope remains incomplete across the full capability-and-regime space.

  • Benchmark findings: WorldSimBench evaluates video world simulators on both perceptual quality and manipulative capability, which can diverge sharply.
  • Benchmark findings: Per-step test success does not guarantee long-horizon coherence, and video models may rely on case-based mimicry rather than generalizable physical principles.
  • Coverage boundary: No single benchmark covers all three boundary conditions and four governing-law regimes, requiring explicit evaluation mapping to avoid over-claiming generalization.

E.6 Minimal Reproducible Evaluation Package (MREP)

The Minimal Reproducible Evaluation Package addresses non-comparable agent results through reproducible environments, replayable traces, structured failure analysis, robust statistics, and explicit boundary-condition reporting. Its assets also support governed promotion of model updates.

  • Motivation: MREP responds to evaluation non-standardization and leaderboard hacking with a proposed community-standard package.
  • Reproducibility: Version locking records exact environment and task-set commit hashes, while trace logs preserve observations, actions, and receipts for replay and failure attribution.
  • Failure and uncertainty reporting: Automated failure taxonomy aligns analysis with five L2 failure categories, and tail statistics report bootstrap intervals, IQM, and performance profiles instead of point estimates.
  • Coverage reporting: Boundary-condition mapping requires explicit declaration of which evaluation conditions were tested.
  • Implementation: Existing tooling supports trace logging, version locking, and tail statistics, whereas automated failure taxonomy and boundary-condition mapping require new infrastructure.
  • Governed validation: MREP assets provide the evidence an L3 gatekeeper needs to decide whether a model update should be promoted or rolled back.

F Implementation Details and Efficient Deployment

This appendix provides extended practical implementation and efficiency details for world-model systems summarized in Section 7.

  • The appendix expands on practical implementation considerations and efficiency techniques for world-model systems.

F.1 Practical Implementation Considerations

World-model deployment requires choices about training structure, latency, simulation fidelity, error handling, and model compression. Generative systems are being accelerated toward real-time planning, while long-horizon rollouts remain constrained by memory growth and compounding errors.

  • Training paradigms: End-to-end training jointly optimizes the encoder, dynamics model, decoder, and policy, whereas modular training assigns each component its own objective.Dreamer exemplifies end-to-end training with shared gradients and policy learning on imagined latent rollouts.
  • Latency and compute: Robotics control favors lightweight latent dynamics and short rollouts under sub-100 ms latency budgets, while web agents tolerate seconds-scale latency and richer search.High-fidelity generative models may require minutes per rollout, restricting them to offline planning.
  • Latency and compute: Learned forward passes scale as O(1) with simulated-system complexity, but a large diffusion model can still be slower than a lightweight physics engine for modest N.Explicit simulation scales as O(N) or worse, while the constant factor of learned inference remains important.
  • Scaling laws: World models primarily capture transition structure rather than memorize factual knowledge, suggesting architectural inductive biases may substitute for raw parameter count more effectively than in language modeling.
  • Sim-to-real transfer: Sim-to-real transfer combines domain randomization with system identification, progressive transfer, or hybrid learned-residual and analytic-physics models.
  • Error handling: Graceful degradation limits rollouts to accurate horizons, uses ensemble disagreement to trigger replanning, and treats unexpected software error codes as assumption violations.MBPO falls back to real data beyond reliable rollout lengths.
  • Generative simulation: Diffusion and flow-matching simulators face iterative-denoising latency, motivating few-step distillation and direct mappings from noise to clean data.
  • Generative simulation: Acceleration techniques for video generation reduce sampling costs enough to make generative models viable engines for real-time planning.
Loading 2604.22748v3…