PaperlayerView exact revision on arXiv

Source-linked AI summary

Critique of World Model

Eric Xing, Mingkai Deng, Jinyu Hou

arXiv:2507.05169v5cs.LGcs.AIcs.CLcs.CVcs.RO

TL;DR

The paper asks how world models can support purposeful reasoning and planning when current systems often fall short across real-world applications. It critiques common design choices and proposes GLP, with PAN as an instantiation, for multimodal, hierarchical simulation; it presents this as a path toward long-horizon, flexible decision-making.

  • Problem

    Most surveyed world-model systems fall short of purposeful reasoning and planning in real-world applications, while the paper argues that world models should simulate possibilities for decision-making rather than focus on video generation.

    Source: §1 · p. 2, §3 · p. 7

  • Method

    The paper surveys world-model design tradeoffs and proposes GLP, a hierarchical generative latent-prediction architecture, with PAN as a concrete instantiation.

    Source: §4.3.3 · p. 14, §5.2 · p. 22

  • Results

    PAN combines multimodal data, stateful hierarchical representations, multi-level generative modeling, and observation-grounded objectives to support long-horizon simulation and flexible decision-making across physical and cyber tasks.

    Source: §6 · p. 25

  • Takeaways & Limitations

    The paper concludes that world models should simulate real-world possibilities for reasoning, while current paradigms and efforts remain primitive.

    Source: §6 · p. 25

    A lossy, non-invertible encoder may discard dynamics-relevant information, leaving a representation insufficient for predicting the modeled transitions.

    Source: §4.3.2 · p. 13

Abstract

from arXiv · show

World Model, the algorithmic simulator of the real-world environment which biological agents experience and act upon, has been an emerging topic in recent years due to the rising need to develop virtual agents with artificial (general) intelligence. There has been much discussion on what a world model really is, how to build it, how to use it, and how to evaluate it. In this essay, starting from the imagination in the famed Sci-Fi classic Dune, and drawing inspiration from the concept of ``hypothetical thinking'' in psychology literature, we argue the primary goal of a world model to be {\it simulating all actionable possibilities of the real world for purposeful reasoning and acting}. We examine the key design dimensions of world modeling: data, representation, architecture, learning objective, and usage, surveying existing approaches and analyzing their tradeoffs. Building on this examination, we propose a new Generative Latent Prediction (GLP) architecture for a general-purpose world model, based on stateful, hierarchical, multi-level, and mixed continuous/discrete representations, and a generative and self-supervised learning framework, with an outlook of a Physical, Agentic, and Nested (PAN) AGI system enabled by such a model.

1 Introduction

The paper frames a world model as a simulator for hypothetical reasoning and purposeful action, not merely a generator of visual content. It surveys key design dimensions and proposes GLP and the PAN world model as a path toward general-purpose modeling.

1 Introduction

  • Human-level general intelligence calls for purposeful reasoning across concrete and abstract abilities, motivating hypothetical thinking through a mental model of the world.The paper calls this mental model a World Model.Source: §1 · p. 2
  • A World Model simulates possible next states from a previous state and action, supporting thought experiments over alternative plans and outcomes.The paper distinguishes this simulator role from generating visual content.Source: §1 · p. 2
  • Simulating outcomes across scenarios may help agents transfer knowledge from prior experience to unfamiliar environments.The paper illustrates this with transfer between low-gravity diving and walking on the moon, and between gaming and drone operation.Source: §1 · p. 2
  • The paper examines world-model design across data, representation, architecture, learning objective, and usage, surveying existing approaches and tradeoffs.It also presents its perspective on requirements for a general-purpose world model.Source: §1 · pp. 2–3
  • GLP combines hierarchical latent prediction with discrete and continuous reasoning, while PAN previews a Physical, Agentic, and Nested world model.The proposed design uses multimodal experience and generative, observation-grounded learning; full details and results are deferred to a separate manuscript [66].Source: §1 · p. 3

2 World Model and Agent Decision-Making

The section formalizes goal-directed agent-environment interaction and motivates a world model as a surrogate for inaccessible real-world states. Such a model lets an agent simulate possible action sequences and choose among them.

2.1 Agent-Environment Model and Optimal Agent

  • The optimal agent maximizes expected future discounted reward, or value, while choosing actions toward its goal.The environment responds to each action with a next state, and rewards may be dense or sparse.Source: §2.1 · pp. 3–4

2.2 World Model and Simulative Reasoning

  • A world model predicts from an internal belief state because agents often cannot access the true state or the universe’s response directly.The belief state is derived from sensory input, and the model predicts a next belief state for a proposed action.Source: §2.2 · p. 4
  • Simulative reasoning requires a persistent, identifiable belief state that can be revisited and updated across prediction steps.The agent can simulate action-and-state sequences, then select an action using an external goal evaluator.Source: §2.2 · p. 4
  • World models support thought experiments across physical, social, and counterfactual scenarios, with simulations adjustable in depth and width.The paper also describes simulations spanning different time scales and cites autonomous driving and military planning as examples.Source: §2.2 · pp. 5–6

3 The World Model Landscape

The surveyed systems model useful aspects of the world, but the paper argues that most do not yet support purposeful reasoning and planning in real-world applications. It distinguishes visual generation from controllable simulation for decision-making.

3 The World Model Landscape

  • Gaming world models render plausible game trajectories, but their inputs, settings, interaction, and 1–2-minute horizons constrain open-ended agent-driven tasks.The paper notes limited action control and largely absent interacting agents.Source: §3 · p. 6
  • 3D scene models offer spatial realism and navigation, but the surveyed systems lack full dynamics and rich interactivity for decision-making or agent learning.Some richer simulations rely on prescribed physics engines outside the learned model.Source: §3 · p. 6
  • Physical world models show fidelity in constrained control settings, yet remain tied to domain-specific sensors, data, or control architectures.The paper says they have not addressed broader complex, multi-agent, or socially grounded worlds.Source: §3 · pp. 6–7
  • Video-generation models produce fixed trajectories without alternative-action interaction or explicit simulation control for counterfactual decision-making.The paper characterizes them as content-generation tools rather than components of decision-making systems.Source: §3 · p. 7
  • JEPA models predict future latent embeddings rather than pixels, but practical usability and generalization beyond simpler settings remain unclear.The paper notes V-JEPA 2's robotic-arm manipulation task while identifying diverse tasks and long-term dependencies as open questions.Source: §3 · p. 7
  • Most surveyed systems fall short of purposeful real-world reasoning and planning because of limits in scope, abstraction, controllability, interactability, and generalizability.The paper argues that world models should serve as simulators for reasoning and thought experiments, rather than primarily generating video.Source: §3 · p. 7

4 Critique of World Modeling

The paper critiques common world-model design choices as coherent but tradeoff-laden for general-purpose simulative reasoning. It argues that training data should capture multiple layers of experience rather than favoring sensory volume alone.

4 Critique of World Modeling

  • Common practice favors sensory data, continuous states, latent prediction, latent reconstruction losses, and model-predictive control over alternatives.The paper presents these choices as common wisdom to examine, not as universally settled requirements.Source: §4 · p. 8
  • These choices emphasize abstract predictable structure over low-level reconstruction, but involve tradeoffs for general-purpose reasoning across domains.The paper organizes its critique around data, representation, architecture, objective, and model usage.Source: §4 · pp. 7, 9

4.1 Data: Information Density, Not Just Volume

  • Raw sensory streams can be voluminous yet semantically redundant, while language encodes abstract concepts, social phenomena, and collective human knowledge.The paper notes that some mental and counterfactual phenomena are difficult or impossible to observe directly.Source: §4.1 · p. 9
  • General-purpose world modeling should use all modalities because they encode different layers of experience, from physical dynamics to abstract concepts and social norms.The paper warns that ignoring either low-level perception or high-level abstraction risks omitting information needed for intelligent behavior.Source: §4.1 · p. 9

4.2 Representation: Continuous? Discrete? Or Both?

The section weighs continuous and discrete representations, arguing that world models need stable, stateful representations for reasoning while discrete codes can retain fine distinctions when scaled appropriately.

4.2 Representation: Continuous? Discrete? Or Both?

  • Fixed-size continuous embeddings from raw sensory streams can fluctuate with noise and viewpoint, while discrete concepts can provide more stable reasoning anchors.The section notes that continuous representations support smoother gradient flow, but may be brittle for some forms of reasoning.Source: §4.2 · pp. 9–10
  • A stateful representation is an identifiable, persistent semantic estimate of the world state that can be stored, recalled, and reasoned over across time.It should remain stable under sensory nuisance variation and suffice to predict relevant dynamics; the authors argue it is necessary for extended simulative reasoning.Source: §4.2 · p. 10
  • Vocabulary-based tokens offer composable concepts at varying abstraction levels, and language-based systems can use context length as a form of long-term memory.The section also notes that sensory data such as images or audio can be converted into discrete tokens while preserving structure and semantics.Source: §4.2 · p. 10
  • A central concern is whether discrete tokens can faithfully represent high-dimensional continuous sensory data and deeper latent structure.The section asks whether discrete tokens can distinguish subtly different world states as complexity grows.Source: §4.2 · pp. 10–11
  • Discrete codes can preserve arbitrarily fine distinctions between real-valued inputs when their representational capacity is scaled appropriately.The theorem states that inputs differing by more than any chosen ϵ can be mapped to distinct language sequences.Source: §4.2 · p. 11
  • Scaling out by length is more manageable than scaling up vocabulary size: the respective growth orders are O(TD log TD) and O((TD)^D).The section presents both strategies as ways to increase discrete-code capacity, but says vocabulary growth is likely unsustainable.Source: §4.2 · p. 11

4.3 Architecture: Pure Abstraction, or Autoregressive Generation

The section compares observation-generating and latent-prediction architectures, arguing that general-purpose world models need both abstract reasoning and predictions grounded in observable data. It proposes Generative Latent Prediction (GLP), which combines hierarchical continuous and discrete prediction with generative reconstruction.

4.3.1 Two Competing Approaches

  • JEPA predicts latent states without reconstructing observations, while GLP adds decoder-based validation against observable data.The section presents GLP as combining JEPA’s abstraction with the validation associated with encoder-decoder approaches.Source: §4.3.1 · p. 12, Figure 5 · p. 13

4.3.2 Tradeoffs for General-Purpose Modeling

  • A lossy encoder can discard information needed to predict dynamics, leaving the predictor unable to recover it and making latent predictions difficult to ground in observations.A decoder can expose an overly lossy representation when it cannot reconstruct the next observation from the predicted state.Source: §4.3.2 · p. 13
  • Generative representations can develop semantic structure while preserving variation, whereas semantic encoders may discard features relevant to tasks not anticipated in training.The section argues that retained information can be abstracted later, but discarded information cannot be recovered.Source: §4.3.2 · p. 14
  • Recent video-generation approaches narrow the efficiency gap, and the authors argue that dropping the decoder’s marginal efficiency gain does not justify fundamental information loss.FastVideo is reported to generate 30 seconds of 1080P video in approximately 3 seconds; Causal Swin-DPM uses chunk-wise causal attention to dampen error propagation and signal variability over long horizons.Source: §4.3.2 · p. 14

4.3.3 Generative Latent Prediction as a Synthesis

  • GLP combines an encoder-decoder bottleneck with an enhanced LLM for discrete reasoning and a diffusion predictor for continuous dynamics across multiple levels of abstraction.The levels range from continuous perceptual variation, through discrete modality tokens, to language-token thought for planning and counterfactual reasoning.Source: §4.3.3 · pp. 14–15
  • GLP keeps detailed reconstruction alongside abstract reasoning, using the lower encoder-decoder layer to isolate low-level variability and mixing continuous embeddings with discrete tokens.The authors say this supports generative supervision and external use while enabling more expressive reasoning and generalization at higher abstraction levels.Source: §4.3.3 · p. 15
  • GLP is presented as an inclusive architecture that combines generative completeness with semantic abstraction and incorporates efficient generative techniques.The authors state that the encoder-world-model-decoder design leads to stronger supervision and more stable training dynamics than open-loop approaches, supported in §4.4.Source: §4.3.3 · p. 15

4.4 Objective: Learning in Data Space, or Latent Space?

The section contrasts latent-transition supervision with next-observation reconstruction, arguing that latent loss can collapse or fail to ensure observation consistency. It favors generative objectives as a grounding signal, while not requiring world models to operate in pixel space.

4.4 Objective: Learning in Data Space, or Latent Space?

  • Latent objectives bypass decoding and supervise encoded transitions, motivated by tractability and avoiding irrelevant details, but may miss semantically important observation-level errors.The paper describes latent loss as a loose surrogate for consistency with what the agent would observe.Source: §4.4 · pp. 15, 17
  • Latent reconstruction admits a constant encoder and invariant transition that achieve zero loss, so the model can learn nothing about true dynamics.The paper identifies this collapse as the limiting case of statelessness.Source: §4.4 · p. 16
  • When data contains different next-observation targets, generative loss permits an encoder and world model with lower loss than the constant-valued collapse configuration.The paper argues that anchoring the objective to observed data avoids the collapse suffered by latent loss.Source: §4.4 · p. 16
  • With approximately inverse encoder and decoder, latent loss is bounded by generative loss plus ε; in practice, the inequality usually holds but does not ensure observed-data consistency.The losses are equal only when the encoder and decoder perfectly invert each other and the data support lies within the decoder image.Source: §4.4 · p. 17
  • The authors argue that learning from observations—not necessarily operating in pixel space—grounds representations and provides a safeguard against prematurely discarding task-relevant information.They present generative reconstruction as a richer, more stable signal for downstream use and human interpretability.Source: §4.4 · p. 18

4.5 Usage: MPC or RL?

The section contrasts MPC, which repeatedly simulates candidate futures at decision time, with RL that learns policies from world-model experience. It presents RL as supporting longer-horizon reasoning, while noting practical limits of MPC.

4.5 Usage: MPC or RL?

  • MPC infers a latent state, simulates candidate action sequences to a decision horizon, scores the resulting states against a goal, and selects a lower-cost action sequence.The cost can be the L2 distance between a predicted state and the encoded goal state.Source: §4.5 · p. 18
  • MPC repeatedly simulates latent trajectories during inference and typically searches only a few steps ahead, constraining foresight and making long horizons difficult.The passage gives 10–20 steps as an example of a typical MPC planning horizon.Source: §4.5 · p. 19
  • In world-model RL, an agent takes actions in simulated states, receives goal-based rewards, and learns a policy from the resulting rollouts.The agent may start from an encoded observation or an imagined state, and rollouts may continue to a horizon or indefinitely.Source: §4.5 · p. 19
  • World-model RL shifts some computation to training, while simulated trajectories can support offline learning through RL, imitation, or other signals.The learned policy can be reused for faster action selection, while MPC and RL may coexist with reactive policies in an agent.Source: §4.5 · p. 20
  • RL can learn policies reflecting long-term cumulative rewards, supporting strategic reasoning over extended horizons in tasks such as robotic manipulation, multi-turn dialog, and autonomous driving.The section describes these as practical settings for the approach.Source: §4.5 · p. 20

5 The PAN World Model

The paper presents PAN as a general-purpose world model for simulating actionable possibilities, with a hierarchical GLP architecture and multimodal training strategy. It outlines how an agent could use cached simulations to choose actions deliberately.

5 The PAN World Model

  • PAN’s design principles combine all modalities, stateful mixed continuous/discrete representations, hierarchical GLP reasoning, observation-grounded generative training, and simulated experience for agent RL.The proposed PAN architecture is a brief preview; details and preliminary results are referred to elsewhere.Source: §5 · p. 20

5.1 A Motivating Usecase

  • The motivating mountaineering task spans multimodal sensing, multiple connected subtasks, and both high-level planning and low-level motor control.The passage also includes social coordination among teammates as part of the task.Source: §5.1 · p. 21

5.2 The GLP Architecture for PAN World Model

  • PAN estimates layered world states from multimodal input, predicts future states conditioned on actions, and reconstructs future observations for generative supervision and external use.Its GLP architecture combines an enhanced LLM backbone for discrete conceptual reasoning with a diffusion-based predictor for continuous embeddings.Source: §5.2 · p. 22

5.3 Training the PAN World Model

  • PAN’s training strategy pretrains modules independently with self-supervision, then aligns or integrates them using multimodal data, cascaded embedding, and gradient propagation.The paper argues that different data sources can cover different scales of capability, rather than requiring exhaustive videos of every complex task.Source: §5.3 · p. 23
  • The pretrain-and-integrate approach is intended to connect sensory information with higher-level context and anchor abstract knowledge in embodied experience, reducing the need for exhaustive task-specific data.The paper argues that capabilities such as social reasoning and travel planning can draw on language data, while embodied skills require physical data.Source: §5.3 · pp. 23–24

5.4 Towards Agentic Reasoning with PAN

  • A PAN-Agent could precompute and cache possible states, actions, and outcomes, then select actions using current beliefs and expected rewards instead of constant forward rollouts.The authors describe this as a prospective way to combine deliberate planning with flexibility and foresight.Source: §5.4 · pp. 24–25, Figure 10 · p. 24

6 Conclusion

The paper argues that world models should simulate actionable possibilities for purposeful reasoning, rather than generate visual content. It presents GLP and previews PAN as a general-purpose design, while assigning action on simulations to a separate agent model.

6 Conclusion

  • A world model should simulate real-world possibilities for simulative reasoning, not video or virtual-reality content; current efforts remain primitive.The critique frames the model as a simulator for purposeful reasoning and acting.Source: §6 · p. 25
  • The paper proposes Generative Latent Prediction (GLP) as an alternative architecture intended to advance world-model theory and implementation.The proposal follows an analytical examination of design dimensions and approaches in the field.Source: §6 · p. 25
  • PAN instantiates GLP with multimodal data, stateful hierarchical representations, multi-level generative modeling, and observation-grounded objectives for long-horizon simulation across physical and cyber tasks.The paper presents PAN as supporting flexible decision-making across those tasks.Source: §6 · p. 25
  • The paper assigns acting on world-model simulations to a separate agent model and argues that broader agent capabilities should arise internally rather than from externally prescribed procedures.The authors defer these questions to a forthcoming companion manuscript.Source: §6 · p. 25
  • Proposed future directions for PAN include multi-agent simulation, wider time scales, improved cross-modal fidelity, and learning directly from imagined experience.The paper presents these as directions for further development, not established capabilities.Source: §6 · p. 25

A Proof for Theorem 1

The proof constructs encoding supports by considering vocabulary size and code length separately. It describes modality-tokenizer and language-expression cases, with the latter mapping values to base-N digit sequences.

A Proof for Theorem 1

  • The proof seeks a support large enough for the required subintervals and studies scaling vocabulary size or code length separately to establish existence.The authors leave more efficient solutions to future work.Source: A Proof for Theorem 1 · pp. 30–31
  • In the modality-tokenizer case, N=T and each input is encoded as T vocabulary tokens whose indices use D base-Mε digits.The passage defines the encoding as a sequence of T tokens from the constructed vocabulary.Source: A Proof for Theorem 1 · p. 31
  • In the language-expression case, each x_td maps to L base-N digits, and matching encoded digits place values in the same interval.The proof uses this interval property when considering inputs with equal encodings.Source: A Proof for Theorem 1 · p. 31

B Proof for Proposition 1

The proof establishes a global minimum for latent loss by choosing constant representations and transitions, which yield zero loss.

B Proof for Proposition 1

  • A constant representation h*(o)=c and transition f*(c,a)=c give latent loss zero, the global minimum because the loss is nonnegative.The proof uses this constructed pair to establish the proposition.Source: B Proof for Proposition 1 · p. 31

C Proof for Proposition 2

The proof shows that a degenerate constant representation and transition cannot be globally optimal under its stated assumptions. It constructs an alternative that lowers generative loss on distinct-output examples while leaving other datapoints unchanged.

C Proof for Proposition 2

  • The argument assumes sufficiently expressive function families and a dataset with at least two distinct input-action-output triplets, including different outputs.The degenerate representation maps every observation to c, and its transition maps c back to c for every action.Source: C Proof for Proposition 2 · p. 32
  • For two examples with distinct outputs, the proof constructs representations and transitions that predict each output, while preserving the degenerate behavior on other datapoints.The construction relies on an expressive decoder and keeps the outputs unchanged for the remaining inputs.Source: C Proof for Proposition 2 · p. 32
  • The constructed pair has strictly lower generative loss, so the degenerate solution is not a global minimizer when the decoder is trained or expressive and the dataset contains different outputs for different inputs.At least one of the two distinct-output examples has positive loss under the fixed-output degenerate solution.Source: C Proof for Proposition 2 · p. 32

D Proof of Theorem 2

The proof expresses observation prediction loss as a scaled KL divergence, relates it to roundtrip latent prediction, and establishes equality of the latent and generative losses under exact roundtrip consistency.

D Proof of Theorem 2

  • Under isotropic Gaussian assumptions, observation prediction loss is expressed as 2D_KL between the empirical next-observation distribution and the model prediction distribution.The model predicts a Gaussian centered at g ◦ f(h(o), a).Source: D Proof of Theorem 2 · p. 33
  • Applying the encoder transition and data processing relates observation-space prediction to roundtrip latent prediction, while roundtrip consistency and the triangle inequality upper-bound latent loss by Lgen.The roundtrip latent prediction is centered at h ◦ g ◦ f(h(o), a).Source: D Proof of Theorem 2 · p. 33
  • When h ◦ g is identity on S, g is injective and h is its inverse on Im(g), so KL invariance under that transformation gives Llatent = Lgen.The relevant distributions are supported inside Im(g).Source: D Proof of Theorem 2 · pp. 33–34
Loading 2507.05169v5…