PaperlayerView exact revision on arXiv

Source-linked AI summary

PAN: A World Model for General, Actionable, and Long-Horizon World Simulation

PAN Team, Zihan Liu, Yi Gu, Mingkai Deng, Guangyi Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Yichi Yang, Kun Zhou, Jiannan Xiang, Zhiting Hu, Zhengzhong Liu, Eric P. Xing

arXiv:2511.09057v4cs.CVcs.AIcs.CLcs.LG

TL;DR

World models still struggle with general action-conditioned prediction, long-horizon consistency, and reasoning beyond visual realism. PAN addresses these gaps with GLP, combining latent state dynamics, closed-loop generative supervision, an LLM reasoning backbone, and diffusion decoding. It achieves leading open-source performance across action simulation, long-horizon forecasting, and planning, while improving VLM-agent task success.

  • Problem

    Existing world models remain limited in general open-domain action prediction, long-horizon consistency, and abstract reasoning and planning, despite advances in video generation and related simulation.

  • Method

    PAN implements GLP with stateful latent representations, an encoder–decoder reconstruction loop, an LLM-based latent dynamics backbone, and Causal Swin-DPM video decoding.

  • Results

    PAN achieves the highest overall performance among open-source world models, is competitive with commercial systems, and improves planning success when integrated with a VLM agent.

  • Takeaways & Limitations

    PAN’s results support evaluating world models by action-conditioned simulation, long-horizon forecasting, and whether their predictions improve simulative reasoning and planning.

  • Takeaways & Limitations

    PAN uses video-chunk-level natural-language actions and remains vulnerable to long-horizon drift, object inconsistency, visual artifacts, and reduced controllability.

Abstract

from arXiv · show

A world model is a cognitive simulator of the real-world environment allowing biological agents to reason about how the world evolves, whether spontaneously or in response to their actions, and accordingly to plan and strategize. In building Artificial Intelligence (AI) systems, world models represent the next frontier beyond large language models (LLMs) to enable physical and embodied intelligence in AI agents, allowing them to perform decision-making through simulative reasoning and reinforcement-learning through simulative trials. Recent advancements in world modeling have yielded impressive progress in video generation, 3-D scene evolution, robotic dynamics, and game simulation, but limitations persist in general, open-domain, action-driven prediction, long-horizon consistency, and abstract reasoning and planning. Moreover, fundamental architectural questions, whether it be state representation, information flow, or training objectives, remain unresolved. In this paper, we introduce PAN, a world model built on the Generative Latent Prediction (GLP) architecture. GLP combines stateful latent representations of world states; an encoder--decoder closed-loop information flow; an LLM/diffusion-based mixed reasoning backbone; and a non-degenerate generative reconstruction objective whose fidelity is ``dampable'' to balance fine-grained detail against semantic saliency. Compared to several existing systems, PAN demonstrates advantages beyond standard video generation in action-conditioned world simulation, long-horizon forecasting, and simulative reasoning and planning, capabilities we argue should serve as the primary criteria for evaluating world models.

1 Introduction

PAN addresses the gap between visually realistic generation and general world simulation by combining latent dynamics, generative reconstruction, and language-based reasoning. Its GLP implementation predicts action-conditioned latent states and decodes them into observable video while supporting long-horizon rollouts.

  • PAN targets world models that generalize beyond narrow environments, fixed action spaces, and visually realistic but non-actionable video generation.
  • GLP couples an encoder, predictive backbone, and decoder so latent transitions remain grounded in realizable sensory changes.
  • PAN uses an LLM-based backbone to evolve latent states under language-conditioned actions and support compositional, counterfactual, and long-horizon reasoning.
  • Causal Swin-DPM uses chunk-wise causal attention and partially noised context to balance observational fidelity with semantically persistent structure during long rollouts.
  • The architecture encodes observations into latent states, predicts next states from state–action history, and renders temporally coherent video segments through a diffusion decoder.
  • PAN trains on observation–action–observation tuples with reconstruction-based supervision, keeping predicted latent transitions tied to realizable observations.

3 Evaluation on world reasoning

WR-Arena evaluates PAN on immediate action simulation, long-horizon forecasting, and planning usefulness. PAN leads open-source systems across these dimensions, while integrating it with a VLM improves planning success over an agent-only baseline.

  • PAN achieves the highest overall performance among open-source models and remains competitive with commercial systems across WR-Arena’s three world-model dimensions.
  • Action Simulation Fidelity: PAN scores 70.3 on agent simulation, 47.1 on environment simulation, and 58.7 overall, the highest overall score among open-source models.
  • Long-Horizon Forecast: PAN reaches 53.6% on Transition Smoothness and 64.1% on Simulation Consistency, both highest among evaluated models including commercial systems.
  • Simulative Reasoning and Planning: PAN achieves 56.1% on Step-Wise Simulation, the highest score among open-source baselines.
  • Simulative Reasoning and Planning: Integrating PAN with a VLM agent increases task success by 26.7% in Open-Ended Planning and 23.4% in Structured Planning versus no-world-model agents.

4 Discussion

PAN frames world models as observation-grounded simulators for hypothetical action reasoning, separating faithful world prediction from goal-directed agent decisions. Its GLP design supports abstract, generative, long-horizon simulation, while remaining limitations include action granularity, rollout drift, physical correctness, and evaluation reliability.

  • World-model role: PAN treats a world model as a simulator for comparing hypothetical action consequences, while a separate agent selects actions toward goals.The framework separates the universe factor pµ(s′ | s, a) from the agent factor pπ(a | s, g), assigning prediction and reward-driven decision-making different training signals.
  • Architectural principles: GLP combines stateful latent dynamics with generative reconstruction, anchoring abstract predictions to observable reality and avoiding representation collapse and information loss.The architecture decodes predicted latent transitions back into the sensory domain, retaining abstract compositional state prediction while grounding it in observations.
  • Ablation evidence: Removing the VLM backbone preserves short-horizon fidelity but degrades long-horizon consistency and agent dynamics, whereas full GLP maintains coherent extended-rollout simulation.The ablation supports coupling symbolic reasoning with generative decoding rather than relying on a decoder alone.
  • Generative supervision: Generative supervision need not reproduce every pixel: its observation-space loss supplies a tractable training signal that also lower-bounds latent prediction loss.The paper presents this as a response to the objection that full-detail future-observation reconstruction is intractable and wasteful.
  • Evaluation: Open-domain evaluation remains unsettled because perceptual video metrics miss causal fidelity, action controllability, and long-horizon consistency, while current human and VLM judgments have reliability concerns.The paper calls for horizon-dependent degradation curves, causal counterfactual tests, and standardized planning protocols.
  • Limitations: PAN remains limited by language-level actions, long-horizon drift, inherited perceptual rather than physical priors, and the absence of fully simulation-specific training.These boundaries motivate multi-resolution action representations, richer stateful architectures, more diverse data, and training from scratch for simulation.

5 Methods

PAN implements GLP with a vision encoder, autoregressive multimodal backbone, and diffusion decoder, then trains and evaluates the system for causal, long-horizon simulation. Its causal denoising mechanism uses partially noised context to preserve semantic continuity while reducing incidental detail.

  • Model implementation: The 256-token backbone predicts compact next latent states from visual state, natural-language action, and query embeddings, compressing multimodal history for long-horizon reasoning.The pretrained VLM backbone supplies multimodal knowledge, language-conditioned sequence modeling, and compositional reasoning.
  • Decoder: The decoder combines projected latent-state context with action-text conditioning, integrating global state information and action-specific visual changes.The latent state enters a new cross-attention stream, while action text uses the original text cross-attention pathway.
  • Causal Swin-DPM: Causal Swin-DPM makes long-rollout reconstruction practical by using chunk-wise causal attention and partially noised context that emphasizes persistent scene structure over pixel detail.With K = 1000 denoising steps, the earlier chunk starts at K/2 and the later chunk at K; future-action information cannot leak backward.
  • Evaluation: PAN evaluates action simulation fidelity, long-horizon forecasting, and simulative reasoning and planning using protocols covering causal consequences, temporal continuity, and downstream agent planning.These dimensions are adopted from WR-Arena and are intended to measure simulator competence beyond standard video quality.
  • Evaluation: The study compares PAN with open-source systems including WAN, Cosmos, and V-JEPA 2, plus closed-source systems including KLING, MiniMax, and Gen-3.The supplied methods passage identifies the comparison set but does not report comparative scores here.
  • Inference: PAN uses autoregressive multi-step inference, augmenting predicted states with encoder outputs from reconstructed observations to improve long-term consistency.Classifier-free guidance uses scale 4; sequence parallelism and quantized attention provide inference acceleration.

Author List

The paper lists Zihan Liu, Yi Gu, Mingkai Deng, Guangyi Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Yichi Yang, Kun Zhou, Jiannan Xiang, Zhiting Hu, Zhengzhong Liu, and Eric P. Xing as authors.

  • Authors: The author list contains fourteen contributors, including Zihan Liu, Yi Gu, Mingkai Deng, and Eric P. Xing.The passage marks Zihan Liu, Yi Gu, Mingkai Deng, and Guangyi Liu with daggers.

Extended Data

The extended-data examples test multi-step interactive simulation from a single initial frame across manipulation and driving, including diverse and counterfactual environments. They illustrate action-conditioned generation over 10+ steps and domain-appropriate outcomes.

  • Interactive simulation: Interactive simulations generate consecutive video segments from a single initial frame and natural-language action sequence across kitchen manipulation and autonomous driving over 10+ steps.
  • Cross-domain behavior: The same action types produce domain-appropriate visual outcomes across visually distinct, stylized, and non-photorealistic environments.Examples include actions such as “move forward” and “turn left.”
  • Counterfactual scenarios: PAN generates uncommon or counterfactual scenarios, including unusual weather, mechanical failures, and atypical object behaviors, when prompted with corresponding actions.

1 Model Architecture Details

PAN uses the GLP architecture to encode observations into latent world states, evolve them under actions, and decode them into video observations. Causal Swin-DPM preserves action-causal context across chunks while reducing rollout discontinuities and error accumulation.

  • GLP components: PAN combines a vision encoder, autoregressive latent backbone, and video diffusion decoder for action-conditioned world simulation.The encoder represents observations, the backbone predicts next latent states from history and actions, and the decoder renders video observations.
  • Latent prediction: The backbone interleaves visual states and natural-language actions in a conversational format, producing 256-token continuous latent world states.Each simulation step corresponds to a user observation-action turn followed by an assistant latent-state response.
  • Decoder conditioning: The decoder conditions on both the predicted latent world state and action text through separate cross-attention pathways.The world-state pathway uses a zero-initialized output projection, while action text follows the standard text cross-attention route.
  • Causal Swin-DPM: Causal Swin-DPM uses a sliding temporal window and chunk-wise causal attention to preserve previous denoising context without exposing future actions.Earlier and later chunks occupy different noise levels, while the mask blocks future action-conditioned content.
  • Causal Swin-DPM: Causal Swin-DPM is designed to improve local transitions and reduce visual discontinuities and accumulated errors during multi-chunk rollouts.Allowing new chunks to attend to partially denoised previous-chunk context reduces one source of error accumulation.

2 Training Details

PAN is trained in stages, first preparing specialized modules and then jointly coupling latent prediction with observation-space reconstruction. During inference, it feeds generated observations back into the model for closed-loop long-horizon simulation.

  • Training strategy: PAN uses staged training so individual modules develop specialized capabilities before unified optimization.The decoder is adapted into Causal Swin-DPM before integration with the latent prediction system.
  • Joint objective: Joint training couples latent prediction to generative reconstruction in observation space through a flow-matching discrepancy.This objective encourages predicted transitions to correspond to realizable sensory changes and reduces susceptibility to latent representation collapse.
  • Joint training: During Stage 2, the pretrained vision-language model is frozen while query embeddings and the video diffusion decoder are trained.The decoder learns to interpret and render the backbone’s compact latent world states.
  • Inference: Inference predicts future latent states sequentially, augments them with encodings of reconstructed observations, and conditions later predictions on the generated history.This supports open-loop simulation rather than relying on ground-truth future observations.
  • Inference: The resulting closed-loop process produces rollouts intended to remain coherent across latent and perceptual domains over long horizons.Each prediction is conditioned on the model’s own generated observations and prior actions.

3 Training Data Pipeline

PAN constructs a general-purpose training corpus from diverse online videos by segmenting coherent episodes, filtering unsuitable content, and generating temporally rich captions. The pipeline emphasizes visual dynamics relevant to action-conditioned simulation.

  • Dataset construction: The dataset spans human activities, human–object and environmental interactions, natural and urban scenes, navigation, and multi-agent scenarios.Videos and text are aligned so semantic descriptions accompany fine-grained visual dynamics.
  • Segmentation: Long videos are segmented with dynamic shot-boundary detection to produce clips with consistent scene structure and temporal continuity.The method detects both abrupt cuts and gradual transitions.
  • Filtering: A multi-stage pipeline removes clips with uninformative motion, poor quality, obstructive text, or unsuitable content.It combines rule-based heuristics, pretrained detectors, and a custom VLM filter.
  • Rule-Based Filters: Rule-based filters classify clips using optical flow, edge differences, luminance differences, and scene-cut criteria.Rejection categories include static, excessively dynamic, unstable, scene-cut, and low-motion clips.
  • Captioning: Captions are designed to be factually detailed and focused on temporal dynamics rather than static scene descriptions.The captioning requirements emphasize motion, events, environmental changes, and newly appearing objects.

4 Evaluation Details

PAN is evaluated across action simulation fidelity, long-horizon forecasting, and simulative reasoning and planning. The protocol compares world models on action faithfulness, temporal continuity, consistency, and goal-directed use by agents.

  • Evaluation framework: The evaluation framework measures action simulation fidelity, long-horizon forecasting, and simulative reasoning and planning.These dimensions assess controllable rollouts, extended-horizon quality, and whether models support agent decision-making.
  • Action Simulation Fidelity: Action simulation fidelity evaluates feasible action sequences using VLM judgments of action faithfulness and precision.It includes agent simulation settings in which the model drives a controllable entity while maintaining background stability.
  • Long-Horizon Forecast: Long-horizon forecasting measures transition smoothness at chunk boundaries and simulation consistency across extended rollouts.The metrics assess motion continuity, content alignment, and style consistency.
  • Simulative Reasoning and Planning: Simulative reasoning and planning tests whether an agent can use model predictions to select actions toward goals in atomic and structured settings.The structured evaluation includes 47 tabletop cases from the Language Table dataset.
  • Baselines: The benchmark compares PAN with video generators, world foundation models, and latent predictive models spanning open and commercial systems.Listed comparators include WAN, Cosmos, V-JEPA 2, KLING, MiniMax-Hailuo, and Gen-3.

5 Related Work

Related work spans domain-specific world models, latent predictive models, video generators, hybrid LLM–diffusion systems, and emerging evaluation benchmarks. PAN combines action-conditioned rollout with structured dynamics and generative reconstruction to target general world simulation.

  • Interactive and domain-specific world models: Domain-specific world models achieve strong within-domain performance but rely on restricted actions and task-specific designs that limit generality.
  • Joint embedding predictive models: Latent predictive models avoid pixel reconstruction, whereas PAN grounds predicted transitions through generative reconstruction for observable action-conditioned video rollout.
  • Video generation models: Standalone video generators typically lack explicit action conditioning, sequential state evolution, and closed-loop rollout, which PAN adds through structured dynamics and action-conditioned generation.
  • Hybrid LLM–diffusion architectures: Hybrid LLM–diffusion architectures establish a precedent for combining autoregressive predictions with diffusion decoders in multimodal generation.
  • World model evaluation: World-model evaluation is expanding from task-specific control and causal prediction toward visual consistency, physical understanding, action simulation fidelity, and long-horizon performance.

6 Additional Analysis

Additional analyses show that PAN maintains stronger long-horizon consistency than standalone video generators, while ablations attribute gains to both causal denoising and latent dynamics. Training remains stable and unsaturated, although broader scaling studies are left for future work.

  • 6.1 Error Accumulation Over Long-Horizon Rollouts: PAN’s Simulation Consistency declines from 87.0 at round 1 to 69.9 at round 9, yet is highest among models from round 5 onward.
  • 6.2 Ablation of the Causal Swin-DPM Denoising Mechanism: Transition Smoothness rises from 11.0 to 44.6 and Simulation Consistency from 36.2 to 60.9 when Causal Swin-DPM replaces single last-frame conditioning.
  • 6.3 Ablation of the VLM-based Latent Dynamics Backbone: Adding the VLM backbone improves Agent Simulation by 4.3 points, Transition Smoothness by 9.0 points, and Simulation Consistency from 60.9 to 64.1, while Environment Simulation remains essentially unchanged.
  • 6.4 Training Dynamics and Loss Trajectory: Stage 1 loss falls from 0.116 to below 0.085 early in training, Stage 2 begins seamlessly, and its loss decreases approximately 2% over the single pass.
  • 6.4 Training Dynamics and Loss Trajectory: A full characterization of scaling across data and compute budgets remains future work, although the persistent loss decline suggests the reported model is not saturated.

7 Limitations and Future Work

PAN’s limitations arise from its language-level action interface, instability in extended rollouts, and inherited biases from a video-generation backbone. Future work targets richer controls, improved long-horizon consistency, and simulation-specific training.

  • Language-level action interface: Natural-language actions support open-domain, high-level rollouts but limit precise low-level control and complex interactions requiring force, direction, and timing.Future versions are proposed to incorporate continuous controls, robot proprioception, and structured environment feedback.
  • Long-horizon drift and failure modes: Long-horizon rollouts can develop semantic drift, object inconsistency, visual artifacts, and reduced controllability, especially for small objects, delayed effects, and rare interactions.The reported results indicate improved stability under evaluated settings rather than a complete solution to indefinite-horizon simulation.
  • Inherited limitations from the video generation backbone: Because PAN adapts a video model optimized for perceptual realism, it can generate physically incorrect interactions or scene elements unsupported by the action.The paper proposes training future world models from scratch with objectives, architectures, and data designed for simulation and interaction.
Loading 2511.09057v4…