Source-linked AI summary
A Mechanistic View on Video Generation as World Models: State and Dynamics
Luozhou Wang, Zhifei Chen, Yihua Du, Dongyu Yan, Wenhang Ge, Guibao Shen, Xinli Xu, Leyi Wu, Man Chen, Tianshuo Xu, Peiran Ren, Xin Tao, Pengfei Wan, Ying-Cong Chen
TL;DR
Video generation models show physical coherence but remain separated from classical state-centric world models by their reliance on observation sequences. This survey bridges the gap with a State-and-Dynamics taxonomy, reviews persistence and causal evaluation, and identifies memory compression and causal reasoning as key frontiers.
Problem
Contemporary video architectures model observation sequences directly, creating a gap from state-centric world models and bottlenecks for long-term reasoning and scalability.
Method
The survey organizes video world modeling around implicit or explicit state construction and causal knowledge integration or architecture reformulation for dynamics.
Results
The survey establishes evaluation stages spanning basic quality, physical persistence, and causality, and identifies a trajectory toward compact, persistent historical representations.
Takeaways & Limitations
Assessing video models as world models should move beyond perceptual quality toward functional tests of state consistency and physical dynamics.
Takeaways & Limitations
The survey focuses on open-loop video generation and leaves investigations of joint training with policy-level signals beyond its scope.
Abstract
from arXiv · showhide
Large-scale video generation models have demonstrated emergent physical coherence, positioning them as potential world models. However, a gap remains between contemporary "stateless" video architectures and classic state-centric world model theories. This work bridges this gap by proposing a novel taxonomy centered on two pillars: State Construction and Dynamics Modeling. We categorize state construction into implicit paradigms (context management) and explicit paradigms (latent compression), while dynamics modeling is analyzed through knowledge integration and architectural reformulation. Furthermore, we advocate for a transition in evaluation from visual fidelity to functional benchmarks, testing physical persistence and causal reasoning. We conclude by identifying two critical frontiers: enhancing persistence via data-driven memory and compressed fidelity, and advancing causality through latent factor decoupling and reasoning-prior integration. By addressing these challenges, the field can evolve from generating visually plausible videos to building robust, general-purpose world simulators.
1 INTRODUCTION
The survey positions video generation as a potential world-model substrate and organizes the gap between video architectures and classical world models around state and dynamics. It proposes a taxonomy, evaluation stages, and future directions for persistence and causal consistency.
- Large-scale video models have progressed from short, low-resolution clips to cinematic sequences with temporal consistency and emergent physical coherence.The cited examples include respecting gravity, collision dynamics, and object permanence.
- World models can be understood through observation, state, and dynamics, with state representing task-relevant information and dynamics governing causal transitions.Classical control theory uses transition equations to evolve states and observation equations to map states to sensory inputs.
- Unlike explicit state-space world models, video models typically use open-loop transformer architectures that model observation sequences directly.Their training relies on passive observation rather than reinforcement learning or policy coupling.
- The survey’s taxonomy separates state construction into implicit context management and explicit compressed representations, while dynamics research divides into causal knowledge integration and causal architecture reformulation.The two dynamics categories respectively combine video models with reasoning-intensive models or redesign architectures for inherent causal reasoning.
- Evaluation is classified into basic quality, physical persistence, and causality, with future work targeting persistence and causal consistency.The survey frames these stages as a shift beyond visual quality toward functional world-model assessment.
2 RELATED SURVEY
Related surveys describe increasingly systematic memory frameworks for language models, while video memory must additionally handle high-dimensional spatial-temporal dynamics and extreme redundancy. Video systems therefore use streaming or hierarchical mechanisms to compress large frame histories.
- LLM memory research evolved from in-context learning and knowledge editing toward cognitive-inspired taxonomies of memory sources, forms, and operations.The cited development includes systematic frameworks appearing during 2024–2025.
- Video memory differs from text memory by modeling high-dimensional spatial-temporal dynamics and managing extreme signal redundancy.Text memory primarily targets semantic consistency and factual accuracy across one-dimensional sequences.
3 PRELIMINARIES
World models transform observations into state representations and model their dynamics to support prediction, planning, and causal simulation. The section contrasts recurrent stateful models with stateless video transformers, whose expanding context creates long-horizon persistence challenges.
- 3.1 The Constituents of World Models: Observation, State, and Dynamics: World models comprise observations, states, and dynamics: observations expose the environment, states retain task-relevant information, and dynamics govern causal state transitions.This framework supports internal simulation of future outcomes under actions or interventions.
- 3.2 The Operations of World Models: Estimation and Prediction: State estimation compresses sequential observations into a compact representation of the environment’s current status.It approximates the latent state conditioned on observation history.
- 3.2 The Operations of World Models: Estimation and Prediction: State transitions predict future states from the current state and actions, enabling mental rollouts, forecasting, planning, and control.The transition function is learned in modern world models and reflects causal evolution under interventions.
- 3.2 The Operations of World Models: Estimation and Prediction: Recurrent state estimation integrates observations and actions over time, allowing models to encode velocity, acceleration, and object permanence unavailable from one observation.These non-instantaneous properties support persistent temporal reasoning.
- 3.3 Stateless and Stateful World Models: Stateless video transformers treat recent observation sequences as implicit state, shifting temporal reasoning to attention over a fixed window.As context expands, computation and memory grow, making long-term reasoning harder and often reducing persistence in long-horizon simulations.
- 3.3 Stateless and Stateful World Models: Standard video models primarily render fixed-duration clips with bidirectional attention rather than explicit temporal causality in visual reasoning.This motivates evaluating video models as potential world-model substrates beyond visual generation.
4 CATEGORIZATION - STATE
The paper organizes state construction around implicit memory mechanisms and explicit latent states. Both paradigms aim to distill historical context sufficiently to maintain coherent, persistent long-term simulations.
- 4 CATEGORIZATION - STATE: State construction is treated as a sufficient-statistic problem: historical context should be distilled into a representation that supports coherent long-term simulation.The framework does not require the model to expose an explicit state variable.
- 4 CATEGORIZATION - STATE: The taxonomy distinguishes Memory Mechanisms from Explicit Latent States as two paradigms for addressing long-term consistency and physical persistence.Memory mechanisms manage context, whereas explicit latent states compress history into persistent representations.
4.1 Implicit State - Memory Mechanism
Implicit state construction manages historical observations as a functional memory rather than a fixed latent variable, using compression, retrieval, and consolidation to support long-term generation.
- Definition: Implicit state treats the model’s state as a managed context of historical observations rather than an explicitly constructed compact latent variable.The state is derived from observation history through an external memory mechanism.
- Compression: Compression condenses redundant observation history into compact representations to reduce computational bottlenecks and support longer temporal contexts.Methods include token merging, summary vectors, and content-adaptive pruning.
- Retrieval: Retrieval selectively accesses historically or externally stored segments according to contextual relevance and current generation intent.Retrieval may use internal attention mechanisms, key-value lookup, sparse attention, or external reference matching.
- Consolidation: Consolidation updates memory after generation by integrating new observations and evicting or re-weighting obsolete information for continuous streaming.Approaches distill recent content, preserve global semantics, and maintain a constant computational footprint.
- Functional comparison: The three primitives differ operationally: compression digests information, retrieval filters inputs by relevance, and consolidation maintains the post-generation memory buffer.Together they manage the trade-off between computational constraints and historical fidelity.
4.2 Explicit State
Explicit state approaches distill video history into compact variables that evolve through learned transitions, either within the generator or through a structurally separate state model.
- Definition: Explicit state construction replaces a growing history buffer with a globally updating latent variable that summarizes the video’s evolution.The state may comprise activation vectors, model weights, or explicit 3D geometries.
- Coupled States: Coupled states fuse the transition function into the generative backbone, where state appears as hidden activations, recurrent buffers, or online-updated parameters.Examples include SSM hidden units, LSTM cells, linear-attention buffers, and plastic model weights.
- Decoupled States: Decoupled states maintain a standalone representation updated separately from the visual generation backbone by semantic transition models or iterative spatial fusion.These states include abstract latent descriptions and explicit 3D memories such as point clouds or Gaussians.
- Decoupled States: Semantics-oriented states preserve object identity, scene logic, and narrative progression through symbolic or latent abstractions.They support long-duration story coherence but operate in highly abstract semantic spaces.
- Decoupled States: Geometry-oriented states use evolving 3D representations to preserve spatial consistency, view invariance, and physical structure.They can struggle with rich and fluid temporal dynamics.
4.3 Summary: Implicit vs. Explicit States in World Modeling
Implicit and explicit states differ in storage, relevance logic, persistence, and scalability: history buffers preserve visual detail, while compact recurrent states favor efficient long-term reasoning.
- Mechanism and Persistence: Implicit states manage raw observations externally, whereas explicit states distill history through internal recurrence into compact representations.This is the central storage distinction between the two paradigms.
- Mechanism and Persistence: Implicit states preserve authentic visual tokens but are limited by the context window, while explicit states support longer persistence through recurrent state updates.The trade-off is between historical fidelity and persistence beyond a bounded window.
- Logic and Causality: Implicit states select information with heuristic rules, whereas explicit states learn a transition St → St+1 intended to capture underlying causal structure.The explicit formulation aligns more closely with a world-simulator definition.
- Scalability and Trade-offs: Implicit inference costs grow with history length, from O(N) to O(N^2), while explicit states maintain constant O(1) computational scaling.The paradigms therefore trade high-fidelity synthesis against efficient long-term simulation.
5 CATEGORIZATION - DYNAMICS
Video-generation research enhances dynamics through causal architecture reformulation or causal knowledge integration, targeting physically plausible and logically consistent temporal rollouts.
- Causal Architecture Reformulation: Causal architecture reformulation redesigns video denoisers with causal attention and training strategies that better match inference-time temporal dependencies.Forcing methods simulate error accumulation and exposure bias using self-generated or degraded contexts.
- Causal Architecture Reformulation: Forcing-based methods diversify noise schedules and condition generation on extended or self-generated content to mitigate training-inference discrepancy.These methods include Self-Forcing, LongLive, Self-Forcing++, Rolling Forcing, and Resampling Forcing.
- Causal Knowledge Integration: Causal knowledge integration delegates high-level dynamics and planning to large multimodal models while using video models as high-fidelity renderers.VLMs or LMMs can provide motion plans, including through chain-of-thought reasoning.
- Causal Knowledge Integration: More tightly integrated systems use textual gradients or unified multimodal streams to guide video generation toward causally faithful outputs and long-term consistency.Examples include CSVC, UniVideo, and SemanticGen.
6 EVALUATION
Evaluating video models as world models requires moving beyond short-clip perceptual quality to assess quality, persistence, and causality across increasingly functional criteria.
- Evaluation axes: World-model evaluation is organized into Quality, Persistence, and Causality, covering observation fidelity, long-horizon state consistency, and causal or physical structure.Quality includes frame-level fidelity, short-range temporal coherence, and conditioning alignment; persistence includes revisitation and memory-dependent continuity; causality includes temporal reasoning, counterfactual response, and action-consistent dynamics.
- Quality: Traditional IS, FID, and FVD primarily measure frame realism or short-clip spatiotemporal consistency, so they can miss fine-grained temporal errors.These metrics do not fully capture the requirements of long-horizon simulation or causal reasoning.
- Persistence: Persistence measures whether generated simulations maintain coherent internal states through long-horizon coherence and memory-capacity tasks.The evaluation asks whether generation remains stable as length increases and whether specific states, causal logic, and spatial layouts are retained.
- Persistence: VBench-long monitors subject-appearance change and background continuity over hundreds of frames, addressing degradation that short-window metrics fail to capture.The protocol extends VBench to longer videos and tracks trends as generation length increases.
- Persistence: Memory-specific evaluation uses World Consistency Score to assess object permanence, relation stability, and causal compliance in generated worlds.WCS is described as a holistic, no-reference measure of internal logical integrity.
6.3 Causality
Causality evaluation tests whether video models preserve temporal and physical structure, respond coherently to interventions, and support embodied planning rather than merely produce plausible images.
- Causality: Causality is evaluated through reasoning, intervention, and planning, progressively testing physical laws and logical progressions in simulated environments.The axis concerns whether models internalize and adhere to causal constraints beyond visual plausibility.
- Temporal reasoning: ChronoMagic-Bench measures long-range temporal reasoning by penalizing semantic scrambling in monotonic transformations such as aging or fabrication.Metamorphic Progression Score and Temporal Coherence Score quantify the realism of temporal trajectories.
- Physical validity: Physics-IQ compares rollouts of deterministic events such as collisions, fluid dynamics, and gravity against ground truth using complementary spatial and spatiotemporal metrics.Its metrics include Spatial IoU, Spatio-temporal IoU, Weighted Spatial IoU, and pixel-wise comparison.
- Interventions: Intervention tests generate twin sequences that diverge after altered initial conditions or actions, requiring distinct consistent outcomes without unrelated scene artifacts.Examples include pushing an object versus leaving it stationary.
- Embodied evaluation: World-in-World couples a video generator to an embodied agent and evaluates navigation and manipulation by task success rate rather than visual fidelity.The framework treats the video model as a simulator that responds to agent actions with new frames.
- Planning: Planning evaluations use Success Rate and normalized regret to measure policy performance in generated worlds against an oracle policy in a ground-truth simulator.Normalized regret quantifies performance degradation attributable to model inaccuracies.
- Controllability: Controllability measures alignment between predicted dynamics and real-world physics under identical action sequences.The score compares generated observations with ground-truth sequences using a perceptual distance scaled by λ; values approaching 1 indicate high dynamic fidelity.
- Planning: VideoWorld reports planning capabilities in Go and robotic manipulation benchmarks without explicit reinforcement learning, suggesting video pretraining can support actionable world models.The cited tasks include CALVIN and RLBench.
7 FUTURE WORK
Future work targets persistence and causality through better memory, fidelity-preserving state compression, latent causal-factor decoupling, and integration of reasoning priors.
- Research priorities: Current video models must address persistence and causality to function as world simulators.The survey identifies these as twin challenges in the evolution from video generation to world simulation.
- Persistence: Implicit-state systems should replace fixed-length context windows and heuristic memory with learned, data-driven mechanisms that dynamically select essential information.The proposed direction uses attention-based operations to maintain long-term consistency.
- Persistence: Explicit-state systems should balance computational efficiency with visual fidelity because compact fixed-size memory buffers can bottleneck fine-grained simulation quality.Hybrid strategies are proposed to retain compressed-state efficiency without sacrificing visual detail.
- Causality: Causal architecture reformulation should use sophisticated annotations and granularities to decouple latent causal factors in video data.The survey links this decoupling to both causal reasoning and precise control of video generation.
- Causality: Causal knowledge integration could incorporate reasoning priors from large-scale understanding models into generative video processes.Aligning generative and understanding components remains a significant research challenge.
- Outlook: These advances are intended to overcome persistence and causal-consistency hurdles and move video models toward general-purpose world simulators.This is the survey’s stated concluding direction.
8 CONCLUSION
The survey frames world-state modeling as the key distinction between video content generation and simulation, arguing for compact persistent states and functional evaluation.
- World state: The world state S_t is identified as the pivotal component distinguishing a simulator from a content generator.The survey examines the convergence of video generation and Model-Based Reinforcement Learning through this distinction.
- State construction: Dependence on raw observation buffers O_1:t creates bottlenecks in reasoning and scalability for current architectures.The survey observes a trajectory toward compressing historical context into compact, persistent representations.
- Evaluation: Evaluation should shift from perceptual metrics to functional benchmarks assessing state consistency and physical dynamics.The conclusion connects this shift to evaluating models as simulators rather than renderers.
- Future direction: Closed-loop interaction and causal intervention are presented as directions for moving video models from rendering pixels toward simulating governing laws.The conclusion contrasts passive open-loop prediction with interactive simulation.