PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms

Muyang He, Hanzhong Guo, Junxiong Lin, Yizhou Yu

arXiv:2603.28489v3eess.IVcs.CV

TL;DR

Video generators can model complex dynamics and long-horizon causalities, but their computational costs limit practical world simulation. This paper systematically reviews efficiency techniques through a three-dimensional taxonomy and relates them to interactive applications and future world-modeling needs.

  • Problem

    A systematic review is missing on how efficiency improvement techniques address the computational demands of video-based world models.

  • Method

    The paper comprehensively reviews efficient modeling paradigms, network architectures, and inference algorithms for video-based world models.

  • Results

    The review identifies efficiency techniques spanning step reduction, adversarial post-training, hybrid AR-diffusion modeling, efficient attention, caching, and related approaches.

  • Takeaways & Limitations

    The reviewed techniques are connected to interactive applications including autonomous driving, embodied AI, and gaming or interactive simulation.

  • Takeaways & Limitations

    Existing efficient architectures face trade-offs between computational cost and spatiotemporal or causal integrity.

Abstract

from arXiv · show

The rapid evolution of video generation has enabled models to simulate complex physical dynamics and long-horizon causalities, positioning them as potential world simulators. However, a critical gap still remains between the theoretical capacity for world simulation and the heavy computational costs of spatiotemporal modeling. To address this, we comprehensively and systematically review video generation frameworks and techniques that consider efficiency as a crucial requirement for practical world modeling. We introduce a novel taxonomy in three dimensions: efficient modeling paradigms, efficient network architectures, and efficient inference algorithms. We further show that bridging this efficiency gap directly empowers interactive applications such as autonomous driving, embodied AI, and game simulation. Finally, we identify emerging research frontiers in efficient video-based world modeling, arguing that efficiency is a fundamental prerequisite for evolving video generators into general-purpose, real-time, and robust world simulators. A curated GitHub repository of the reviewed literature can be found at https://github.com/Isaachhh/Efficient-VWM-Survey.

I. INTRODUCTION

Video generation is increasingly viewed as a route to world simulation, but practical world models face severe computational and memory bottlenecks. This review organizes efficiency techniques across modeling paradigms, architectures, and inference, and connects them to interactive applications.

  • Motivation: Video-based world models aim to represent environmental dynamics and predict future states from historical contexts and optionally actions.Their objective is to model causal mechanisms such as gravity, collision, and object permanence rather than only pixel transitions.
  • Motivation: Large-scale video training can support emergent modeling of complex interactions and fluid dynamics, while compact latent spaces reduce the cost of imagining future scenarios.The paper also identifies unified use across media production, autonomous driving, and robotic manipulation.
  • Efficiency Gap: Effective world simulators require long-term consistency, physical accuracy, and high-resolution interactivity, yet video modeling creates substantial computational and memory demands.Autoregressive models encounter KV-cache growth, diffusion models incur iterative-denoising latency, and video frames contain extensive redundancy.
  • Review Scope: The review systematically examines efficiency in modeling, architectures, and algorithms across autoregressive and diffusion-based paradigms.Its architectural scope includes VAEs, memory mechanisms, and efficient attention; inference includes parallelism, caching, pruning, and quantization.
  • Contribution: The paper presents the first systematic review dedicated to efficiency improvement techniques for video-based world models.It distinguishes this focus from prior work centered mainly on general video generation or specific diffusion techniques.
  • Applications: The review covers downstream uses including autonomous driving, embodied AI, and interactive simulation, and discusses associated challenges and future opportunities.The paper frames these applications as important settings for efficient video-based world models.

II. BACKGROUND

The background presents diffusion, flow matching, and autoregressive generation as core video-generation paradigms. Diffusion denoises progressively corrupted data, whereas flow matching learns a continuous-time velocity field for transport from a prior to data.

  • Overview: Modern video-generation models commonly use diffusion models, flow matching, and autoregressive models as their dominant generative paradigms.The paper introduces these formulations as foundations for subsequent efficiency discussions.
  • Diffusion Models: Diffusion models formulate generation as iterative denoising and often operate in a VAE latent space to reduce computational cost.Latent diffusion applies the denoising process to compressed representations rather than directly to pixels.
  • Diffusion Models: The diffusion forward process gradually adds Gaussian noise through a fixed Markov chain, allowing a noisy state at timestep t to be sampled directly from the original data.The supplied formulation describes the noise schedule and direct sampling relation.
  • Diffusion Models: The reverse diffusion process learns a parameterized transition that predicts added noise or velocity, and generation iteratively denoises Gaussian noise back to data.Training commonly uses mean squared error between actual and predicted noise.
  • Flow Matching: Flow matching models generation as a continuous-time probability path governed by ordinary differential equations and learns a vector field matching the target velocity.Conditional flow matching uses tractable paths between source noise and target data.
  • Flow Matching: In a straight-line conditional flow path, interpolation between noise z0 and data x1 yields a constant target velocity x1 − z0.This construction makes the conditional target field tractable.

3) Auto-regressive (AR) Models:

Autoregressive video models factorize sequence probabilities and predict tokens sequentially, providing a basis for long-horizon generation. Efficient video modeling extends this foundation with compression, spatiotemporal tokenization, and interactive paradigms.

  • Autoregressive Formulation: Autoregressive models decompose a sequence distribution into conditional probabilities over preceding tokens.Visual tokens may come from VQ-VAE-style tokenizers that quantize encoded patches or frames into a finite codebook.
  • Autoregressive Formulation: Autoregressive training uses parallel teacher forcing, but inference remains sequential and can become expensive for long videos with O(N) complexity.The cost arises from predicting each next token conditioned on prior context.
  • Spatiotemporal Scaling: Video generation expands image modeling into the 3D spatiotemporal domain T × H × W, creating cubic growth in computational complexity.Directly inflating 2D kernels into 3D kernels preserves spatial priors but increases parameters and computational load.
  • Spatiotemporal Scaling: Tubelet tokenization jointly captures local spatial semantics and temporal evolution through spatial and temporal downsampling.This approach requires positional embeddings such as 3D RoPE to preserve spatiotemporal geometry.
  • Efficient Pipeline: Modern video generators typically combine latent compression, a generative backbone, and additional pipeline components to manage high-dimensional video data.Three-dimensional causal VAEs reduce spatial and temporal redundancy before generation in latent space.
  • Efficient Modeling: Efficient modeling targets practical latency and memory constraints through diffusion distillation and long-horizon paradigms such as autoregressive, hybrid AR-diffusion, and streaming causal diffusion models.These directions aim to support realtime interaction and persistent world simulation.

A. Diffusion Model Distillation for Efficient Sampling

Diffusion distillation reduces video-generation sampling cost by training students to reproduce teacher behavior in far fewer steps. Consistency, distribution-matching, and adversarial objectives extend this efficiency strategy toward one-step, real-time synthesis, but do not by themselves solve persistent long-horizon generation.

  • Step-Reduction Distillation: Distillation trains a student sampler to match a K-step teacher using K′ steps, where K′ ≪ K.The student learns the teacher’s multi-step rollout in a single macro-step; progressive variants reduce steps iteratively.
  • Step-Reduction Distillation: Wan’s sampling steps decrease from 48 to 6 with GPD while maintaining competitive quality.GPD progressively guides the student toward larger step sizes.
  • Consistency Distillation: Consistency training enforces agreement between predictions from two points on the same trajectory, enabling one-step generation.VideoLCM and AnimateLCM extend this approach to latent video models, while TurboDiffusion adds reward-guided distillation.
  • Distribution-Level Distillation: Distribution-level distillation matches student and teacher distributions using statistical, score-based, adversarial, or hybrid discrepancies.DMD uses distribution matching, while DMD2 combines it with GAN supervision on real data.
  • Adversarial Distillation: Seaweed-APT combines adversarial post-training with approximated R1 regularization to enable real-time one-step video generation.The method applies adversarial post-training against real data after diffusion pretraining.
  • Limitations: Distillation primarily improves step efficiency and wall-clock latency, but usually lacks mechanisms for causal inference, memory retention, and long-horizon error control.These missing mechanisms constrain persistent, long-horizon generation.

B. Auto-Regressive and Hybrid Approaches

Autoregressive, hybrid AR-diffusion, and streaming causal diffusion approaches target persistent, interactive, long-horizon video generation. They improve temporal rollout and context handling, yet extended timelines still exhibit structural, spatial, and interaction failures requiring stronger memory and error-control mechanisms.

  • Overview: Autoregressive and hybrid approaches combine temporal rollout with efficient synthesis to pursue persistent, interactive, and infinite-length generation.They seek to combine AR scalability with diffusion fidelity for real-time interaction.
  • Autoregressive Modeling: VideoGPT models discrete spatiotemporal VQ-VAE tokens autoregressively with a GPT-like transformer, establishing a reproducible transformer-based video-generation baseline.The formulation treats video generation as discrete token prediction.
  • Autoregressive Modeling: Genie enables frame-by-frame controllable environment simulation through a video tokenizer, autoregressive dynamics model, and latent action model.iVideoGPT extends world modeling to next-token prediction over visual observations, actions, and rewards.
  • Hybrid AR-Diffusion Modeling: Hybrid AR-diffusion models autoregressively propagate temporal context while using diffusion to generate the current frame or chunk.Progressive Autoregressive Video Diffusion improves information propagation, while FramePack compresses historical contexts by frame-wise importance and adds drift prevention.
  • Streaming Causal Diffusion Modeling: Streaming causal diffusion causalizes diffusion through temporal or block-causal attention so frames or chunks can be generated incrementally without future context.This complements token-level autoregressive and hybrid AR-diffusion pipelines.
  • Streaming Causal Diffusion Modeling: Causal attention alone remains insufficient for stable long-horizon rollout because train-test mismatch and error accumulation persist.Diffusion Forcing uses independent noise levels, while Self Forcing trains on self-generated histories and Rolling Forcing jointly denoises future frames.
  • Discussion: Over 10 minutes, current methods struggle with object permanence, structural consistency, complex human-object interactions, and coherent 3D spatial positioning.Figure 2 reports method-specific failures including structural collapse, hallucinations, duplicated characters, body distortions, and incorrect spatial relationships.
  • Discussion: Long-term interaction requires algorithms that mitigate cumulative errors and memory mechanisms that preserve spatial, logical, and physical consistency.These requirements follow from the observed degradation over extended timelines.

IV. EFFICIENT ARCHITECTURE

Efficient architectures reduce the spatiotemporal complexity of video-based world modeling through hierarchical representations, staged refinement, and compact latent states.

  • Architecture overview: Efficient architecture addresses spatiotemporal redundancy and quadratic attention costs through hierarchical representations, memory mechanisms, efficient attention, and extrapolation.These four paradigms target persistent, high-fidelity world models from short video clips.
  • Hierarchical and pyramidal generation: Coarse-to-fine hierarchies let a base module establish a general world model before specialized modules refine details.This reduces redundant detail computation during early semantic planning.
  • Hierarchical and pyramidal generation: Cascaded systems such as Waver and FlashVideo upsample low-resolution DiT outputs to 1080p through multi-stage refinement.Pyramidal Flow Matching, TPDiff, SUPERGEN, and PatchVSR apply related staged designs across temporal, spatial, or tile dimensions.
  • Hierarchical and pyramidal generation: SRDiffusion switches from a large model during high-noise steps to a small model during low-noise steps, trading quality and efficiency across diffusion stages.The large model handles structure and motion, while the small model generates finer details.
  • Hierarchical and pyramidal generation: Persistent world modeling compresses world state into manageable latent representations, including chunk-causal autoencoders with up to 64× spatial and 4× temporal compression.REGEN further permits plausible rather than exact reconstruction, allowing ultra-compact semantic tokens.
  • Long context and memory: External or implicit memory augments the generative backbone with persistent storage of the simulated world for long-horizon consistency.The supplied passage introduces memory as a general architectural requirement but does not specify a single implementation.

1) Visual Memory:

Visual memory mechanisms preserve long-horizon context by selectively retaining, compressing, retrieving, or geometrically representing historical information while efficient attention limits computation.

  • Visual Memory: Visual memory retains raw or semicompressed keyframes as reference points, compressing historical frames according to relative importance within fixed context limits.FramePack and WorldPack combine frame compression, trajectory packing, and selective retrieval to retain long-term history efficiently.
  • Visual Memory: Spatial memory uses point clouds, meshes, panoramas, or surfel-indexed maps so generation can render from persistent geometric representations.These designs shift the model’s role from pixel generation toward rendering from a consistent memory.
  • Visual Memory: Latent memory methods dynamically replay errors, retrieve relevant historical frames, or compress evicted local-window tokens into recurrent global state.SVI, MemFlow, and VideoSSM use different mechanisms to preserve relevant history and address accumulation artifacts.
  • Visual Memory: Implicit model memory embeds historical context into weights through online self-supervised updates of a small adaptive sub-network.TTT layers capture global narrative dependencies, while LaCT updates massive token blocks to support contexts exceeding 50k tokens.
  • Efficient Attention: Full attention becomes a dominant runtime cost as video resolution and frame count grow because its complexity is quadratic in context length.Efficient alternatives approximate it with sparse, windowed, or linear mechanisms.
  • Efficient Attention: Sparse attention restricts computation to relevant or local token pairs by exploiting structural sparsity in video attention patterns.SVG identifies temporal and spatial sparsity, while SVG2 uses semantics-aware permutation to improve efficiency.
  • Efficient Attention: Windowed attention limits computation to local neighborhoods and can combine short windows with global anchors, compressed tokens, or routing mechanisms.LongLive permanently caches initial frames as global anchors for long-range coherence during infinite-length streaming.
  • Efficient Attention: Linear attention replaces explicit N × N attention with kernel feature maps and associative multiplication, reducing complexity from quadratic to O(N).SANA-Video, LinVideo, and SLA apply linear attention at global, layer, or token levels.

3) Linear Attention:

Linear-complexity sequence mechanisms and positional extrapolation extend efficient video generation, but current architectures retain trade-offs between computational cost and causal integrity.

  • 3) Linear Attention: SANA-Video replaces vanilla attention with ReLU-based linear attention for efficient block-wise autoregressive generation.LinVideo selectively substitutes quadratic layers, while SLA routes decomposed attention weights at the token level.
  • 3) Linear Attention: Mamba-based state space models provide O(N) sequence processing through recurrent state transitions instead of standard Transformer attention.LinGen combines bidirectional Mamba2 with Temporal Swin Attention for stable minute-length video generation with strictly linear scaling.
  • 4) Extrapolation and RoPE: Long-horizon world models require RoPE modifications to simulate beyond the training horizon without distribution drift.This motivates frequency scaling, attention-dispersion control, and moving-reference-frame designs.
  • 4) Extrapolation and RoPE: RIFLEx shifts RoPE frequencies to enable 3× length extrapolation as a training-free baseline.The method targets temporal repetition attributed to high-frequency components.
  • 4) Extrapolation and RoPE: Infinity-RoPE uses Block-Relativistic RoPE to rotate new latent blocks relative to a moving local reference frame for effectively infinite simulation.This reframes extrapolation as a sliding-world paradigm rather than simply extending a fixed window.
  • E. Discussion: Efficient architectures face trade-offs between computational cost and spatiotemporal or causal integrity, including motion decay, temporal loops, and memory stability-plasticity tension.The discussion identifies these limitations across compression, extrapolation, and memory mechanisms.
  • E. Discussion: Future directions include physically constrained latent spaces, hybrid memory hierarchies, interactive causal chains, and hardware-software co-design for dynamic compute allocation.The proposed mechanisms target physics-aware, adaptive, real-time generation.

V. EFFICIENT INFERENCE

Efficient inference distributes computation, reuses intermediate features, and reduces redundant tokens to address the memory and latency costs of long, high-resolution video generation.

  • V. Efficient Inference: Efficient inference targets single-GPU memory bottlenecks and latency through parallelism, redundant-computation reduction, and quantization.The section organizes these strategies around deployment of large, long-duration video models.
  • A. Parallelism: Sequence, pipeline, and hybrid parallelism distribute video generation across devices to support high resolution, long duration, and real-time inference.These approaches partition spatial, temporal, or model workloads and improve device utilization.
  • A. Parallelism: Timestep-forcing pipeline parallelism assigns different denoising timesteps to different devices, converting diffusion into a high-throughput streaming pipeline.This design is illustrated through LiveAvatar’s streaming avatar generation system.
  • A. Parallelism: xDiT combines sequence, PipeFusion-style pipeline, and classifier-free-guidance parallelism in a unified scalable inference engine.The combination reflects the absence of one universally optimal parallel strategy.
  • B. Caching: Caching exploits gradual changes in intermediate activations across diffusion timesteps, avoiding full recomputation at every step.Methods progress from coarse feature reuse toward timestep-aware and fine-grained policies.
  • B. Caching: HERO adapts recomputation frequency by layer, recomputing variable shallow layers more often while linearly extrapolating stable deeper layers.This hybrid scheme targets multimodal video generation with depth and RGB views.
  • C. Pruning: Pruning reduces computation by eliminating redundant tokens, channels, or layers across video content and the diffusion process.Token-level reduction addresses sequence-length explosion caused by video resolution and temporal depth.

1) Token-Level Reduction:

Token-level reduction methods reduce video-generation computation by merging redundant tokens, pruning structure, and lowering numerical precision. These techniques target sequence length, architectural redundancy, and inference memory or arithmetic costs.

  • Token Merging: VidToMe merges redundant self-attention tokens across video frames using bipartite matching, combining locally merged tokens with maintained global tokens.It partitions tokens into source and destination sets and links source tokens to similar destination tokens across adjacent frame chunks.
  • Token Merging: Temporal dimension token merging averages consecutive tokens to halve the sequence length used for attention computation.
  • Structural Pruning: Structural pruning reduces computation by removing redundant temporal blocks and compressing layer widths through learnable gating and channel funneling.
  • Quantization: Quantization accelerates inference and reduces memory usage by lowering the precision of model weights and activations.The reviewed methods include attention-centric optimization, post-training quantization, quantization-aware training, and dynamic scheduling.
  • Quantization: FP4 microscaling attention achieves extreme compression with negligible quality loss on next-generation hardware such as RTX 5090.
  • Quantization: Post-training quantization often degrades video-generation performance at ultra-low precision, motivating quantization-aware training for settings such as W4A4 and W3A3.

E. Discussion

Efficient video inference combines parallelism, caching, pruning, and quantization to reduce latency while supporting high-fidelity, long-horizon world-model generation. The review connects these efficiency techniques to applications in driving and interactive simulation, while emphasizing their interactions and stability risks.

  • Discussion: Efficient inference targets per-step latency and high-fidelity, long-horizon generation through parallelism, caching, pruning, and quantization.
  • Discussion: These inference techniques can interact adversely: caching or pruning may amplify errors, while low-bit quantization can destabilize activations altered by token reduction or feature reuse.
  • Discussion: Future methods should combine parallelism, caching, pruning, and quantization while improving efficiency and stability for long-duration interactive scenarios.
  • Autonomous Driving: Video-based world models support autonomous driving tasks including synthetic data generation, interactive simulation, and generative planning.
  • Interactive Simulation: Closed-loop systems use action-conditioned rollouts for interactive simulation and evaluation, including realistic continuous video generation and virtual autonomous driving.
  • Interactive Simulation: MagicDrive-V2 supports conditional street-scene generation using road maps, object boxes, ego trajectories, and text as control signals.

B. Embodied AI

In embodied AI, video world models augment training data, support action-conditioned simulation and evaluation, and connect visual trajectory generation with robot action learning. Interactive game environments provide another controlled setting for efficient world-model deployment.

  • B. Embodied AI: Video world models augment embodied-AI training data with broader distributions and rare cases to improve policy generalization in dynamic, long-tail tasks.
  • B. Embodied AI: GenMimic reconstructs human motion in 4D, retargets it to humanoid embodiments, and trains reinforcement-learning policies for motion imitation.
  • B. Embodied AI: World-Env couples a video simulator with VLM-guided reflection for dense rewards and completion-based termination, while EVAC generates controllable multi-view observations for evaluation.
  • B. Embodied AI: World action models jointly predict future video frames and actions, using shared representations between video generation and action prediction for robot policy learning.
  • B. Embodied AI: Other systems generate future visual trajectories before predicting actions, enabling imitation learning, receding-horizon control, or long-horizon consistency.
  • Interactive Games: Games offer well-defined interaction interfaces and controllable closed-loop evaluation, making them common deployment settings for efficient interactive world simulation.

VII. MORE RELATED WORK

Related work extends efficient video world modeling across interactive talking heads, content creation, video-driven scene generation, and broader world-model applications. These systems increasingly emphasize streaming, controllability, memory, and real-time interaction, while retaining open consistency challenges.

  • A. Interactive Talking Head Generation: Talking-head generation is shifting from offline portrait synthesis toward causal, streaming, real-time interaction and long-horizon conversation.
  • A. Interactive Talking Head Generation: Existing talking-head approaches still exhibit limited motion diversity or local artifacts despite progress toward real-time streaming generation.
  • B. Interactive Content Creation: Efficient video generation supports interactive content-creation workflows in which creators repeatedly refine prompts, structure, characters, or shot pacing.
  • B. Interactive Content Creation: Video editing methods reduce cost through cached spatial-attention tokens, mask-guided key-value sharing, and latent blending across frames or semantic segments.
  • C. Video-Driven Scene Generation: Video-driven scene-generation pipelines use video models for reference sequences and then recover or expand 3D/4D scene structure.
  • C. Video-Driven Scene Generation: Decoupling video generation from geometric reconstruction provides a stable modular pipeline but can cause errors to accumulate progressively.
  • VIII. Conclusions: The review organizes efficiency techniques into modeling paradigms, architectures, and algorithms, and examines their effects on autonomous driving, embodied AI, and gaming.
Loading 2603.28489v3…