Source-linked AI summary
Pre-Trained Video Generative Models as World Simulators
Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, Ling Pan
TL;DR
Pretrained video generators are effective at producing realistic clips from static prompts, but lack the interactivity and dynamic modeling required for action-conditioned simulation. DWS adapts pretrained generators with action conditioning and motion-focused training, and adds prioritized imagination for downstream learning. The paper reports improvements in action-conditioned video quality and MBRL performance across games and robotics.
Problem
Video generators conditioned on static prompts lack frame-level interactivity and dynamic modeling for interactive, action-conditioned simulation.
Source: §1 · pp. 1–2Method
DWS fine-tunes pretrained video generators with a lightweight action-conditioned module and motion-reinforced loss, and uses prioritized imagination for downstream MBRL.
Source: §1 · pp. 2–3Results
DWS significantly improves action-conditioned video quality and dynamic consistency across game and robotics tasks, and improves MBRL performance.
Source: §1 · p. 3, §6 · p. 12Takeaways & Limitations
Prioritized imagination improves sample efficiency and policy performance when pretrained-model world simulators are applied to downstream MBRL.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Video generative models pre-trained on large-scale internet datasets have achieved remarkable success, excelling at producing realistic synthetic videos. However, they often generate clips based on static prompts (e.g., text or images), limiting their ability to model interactive and dynamic scenarios. In this paper, we propose Dynamic World Simulation (DWS), a novel approach to transform pre-trained video generative models into controllable world simulators capable of executing specified action trajectories. To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module that seamlessly integrates into any existing model. Instead of focusing on complex visual details, we demonstrate that consistent dynamic transition modeling is the key to building powerful world simulators. Building upon this insight, we further introduce a motion-reinforced loss that enhances action controllability by compelling the model to capture dynamic changes more effectively. Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models, achieving significant improvements in generating action-controllable, dynamically consistent videos across games and robotics domains. Moreover, to facilitate the applications of the learned world simulator in downstream tasks such as model-based reinforcement learning, we propose prioritized imagination to improve sample efficiency, demonstrating competitive performance compared with state-of-the-art methods.
1 Introduction
Static-prompt video generators lack the frame-level interactivity and dynamic modeling needed for action-conditioned world simulation. DWS adapts pretrained models with action conditioning and motion-focused training, while prioritized imagination improves sample efficiency in downstream model-based reinforcement learning.
1 Introduction
- Video generators trained for one-shot output from static prompts lack frame-level interactivity and frame-to-frame dynamic modeling needed for interactive simulation.World simulators must also capture fine-grained dynamic changes with frame-level action alignment, especially for MBRL.Source: §1 · pp. 1–2
- World models can let agents learn skills through simulated interaction, avoiding the risks and costs of real-world trials.The paper frames this as a potential application of world models in model-based reinforcement learning.Source: §1 · p. 2
- DWS combines a lightweight action-conditioned module for frame-level action alignment with motion-reinforced training that prioritizes action-induced changes over static visual details.The module is described as two linear layers integrated through scale and shift operations.Source: §1 · p. 2
- Prioritized imagination samples informative transitions instead of well-understood ones, improving sample efficiency during agent–world-model interaction.The approach is proposed to enhance world simulators' practical utility in MBRL.Source: §1 · p. 3
- Across game and robotics tasks, DWS improves action-conditioned video quality and dynamic consistency, while its prioritized-imagination models support more efficient and effective MBRL than prior state-of-the-art approaches.The cited introduction summarizes these outcomes without specifying quantitative values.Source: §1 · p. 3
2 Related Work
Prior work develops action-controllable video models and model-based reinforcement learning systems, but the paper identifies gaps in adaptation cost, architecture coverage, and use of pretrained knowledge. DWS is positioned as an approach to adapt pretrained video models and prioritize imagined samples.
Video World Models
- Action-conditioned video models target generalist world simulators that capture physical dynamics and action consequences, but typically require large-scale from-scratch training with substantial compute and slow inference.The paper contrasts these models with its fine-tuning approach using publicly available pretrained generators.Source: §2 · p. 3
- Prior pretrained-model adaptation work is limited to diffusion models and does not validate downstream model-based RL, while DWS is presented as architecture-agnostic.The paper also reports that DWS reduces training costs and inference latency.Source: §2 · p. 3
Model-Based RL
- Recent transformer- and diffusion-based world models often train from scratch and remain task-specific, while existing methods uniformly weight imagined samples rather than prioritizing significant ones.DWS is described as leveraging pretrained knowledge and selectively prioritizing samples to improve sample efficiency.Source: §2 · p. 4
3 Preliminaries
The preliminaries frame visual world modeling as interactive video generation: an agent acts from observations, and a world model predicts subsequent observations and rewards. The paper uses pretrained video generators as bases for this simulation task.
3.1 Problem Formulation
- In the POMDP formulation, an agent selects actions from observations, the environment transitions to a new state and observation, and the agent seeks to maximize discounted cumulative reward.A trained world model can replace the environment by predicting the next observation and reward from the current observation and action.Source: §3.1 · p. 4
3.2 Pre-Trained Video Generative Models
- The paper formulates visual-control world modeling as interactive video generation to use video data's generalizable visual and motion knowledge, and fine-tunes pretrained Open-Sora and iVideoGPT models.Open-Sora is a rectified-flow diffusion model pretrained on millions of internet videos; iVideoGPT is an autoregressive transformer that tokenizes visual outcomes, actions, and rewards.Source: §3.2 · p. 4
- iVideoGPT is an autoregressive transformer built on LLaMA that compresses visual outcomes, actions, and rewards into tokens for interactive video prediction.Source: §3.2 · p. 5
4 Method
The method adds frame-level action conditioning and motion-focused training to video generative models, then uses the resulting world models for prioritized imagination in model-based reinforcement learning.
4.1 Action-Conditioned Module
- A lightweight add-on conditions each frame on its corresponding action and extends frame-level control across diffusion and transformer architectures.Unlike a single embedding for the full trajectory, each frame is directly modulated by its corresponding action.Source: §4.1 · pp. 5–6
- Discrete actions are translated into language templates for pretrained text encoders, while continuous actions use a trainable linear action embedder.The language mapping is intended to bridge the semantic gap between integer actions and text prompts.Source: §4.1 · p. 5
4.2 Motion-Reinforced Loss
- The motion-reinforced loss weights training by inter-frame pixel changes, emphasizing dynamic content over stable backgrounds.Pixels that change across frames have greater influence on backpropagation than static background elements.Source: §4.2 · pp. 6–7
- On BAIR, removing either the motion-reinforced loss or action-conditioned module significantly degrades action-conditioned video prediction with frame-to-frame dynamics.The reported comparison uses SFT loss curves from fine-tuning iVideoGPT.Source: §4.2 · p. 7
4.3 Model-Based Reinforcement Learning
- Prioritized imagination samples starting observations more often when their transitions have high learning potential, measured by the magnitude of TD loss.The method concentrates imagination on states expected to contribute more to policy learning.Source: §4.3 · p. 8
5 Experiments
DWS improves action-conditioned video prediction across robotics and game domains, and its world models support stronger model-based reinforcement learning. Offline RL results also improve when DWS-generated transitions augment the training data.
5 Experiments
- Evaluation spans Open-Sora diffusion and iVideoGPT autoregression across continuous-action BAIR and discrete-action Procgen and Atari, measuring video prediction and reinforcement-learning returns.The experiments assess action-conditioned prediction quality and online and offline model-based RL.Source: §5.1 · pp. 8–9, §5 · p. 8
5.2 Model-Based Reinforcement Learning
- DWS-trained world models with PPO outperform vanilla PPO and DreamerV3; Breakout shows a reported 7× gain over state-of-the-art methods, with prioritized imagination credited for sample efficiency.Figure 6 reports averaged returns across five random seeds on Atari and Procgen.Source: Figure 6 · p. 11, §5.2 · pp. 11–12
5.1 Action-Conditioned Simulation
- DWS improves video quality and action following across diffusion and autoregressive models, producing coherent videos and precise predictions across robotics and game domains.The evaluations include unseen initial frames; the base models without DWS can show visual distortions and fail to match conditioned actions.Source: Figure 4 · p. 9, §5.1 · pp. 10–11, Figure 5 · p. 10
5.3 Offline Model-Based Reinforcement Learning
- Adding one million DWS-generated transitions to one-million-trajectory CoinRun and Ninja datasets significantly improves CQL and IQL performance across both environments.The generated transitions double the original dataset size.Source: Table 3 · p. 12, §5.3 · p. 12
6 Conclusion and Limitation
DWS adapts pre-trained video generators into action-conditioned world simulators using a lightweight module and motion-reinforced loss. The paper reports improved video prediction and MBRL performance, while noting limits in temporal horizon and spatial resolution.
6 Conclusion and Limitation
- DWS combines a lightweight action-conditioned module for action-frame alignment with motion-reinforced loss to model inter-frame pixel dynamics.The framework adapts pre-trained video generative models for downstream action-conditioned simulation.Source: §6 · p. 12
- DWS is currently limited in modeling videos with extended temporal horizons or high spatial resolutions.The authors leave addressing these limits to future work.Source: §6 · p. 12
A Details of Dataset Collection
The Atari dataset comes from DQN replay buffers collected without sticky actions, while Procgen replay datasets use PPO-collected transitions.
A.1 Atari
- Atari replay data comes from a DQN trained for one million environment steps; environments omit sticky actions and use frame skip 4 and 84 × 84 observations.The replay buffer is used as the offline dataset for tuning.Source: §A.1 · p. 18
A.2 Procgen
- Coinrun and Ninja Procgen datasets contain one million PPO-collected state transitions per game at 64 × 64 resolution.The data is collected at the default Procgen resolution for DWS training.Source: §A.2 · p. 18
B.1 Details of Base Models
The base models are Open-Sora, a text-and-frame-conditioned generator, and iVideoGPT, an autoregressive transformer; DWS action modules are integrated into their layers or blocks.
B.1 Details of Base Models
- Open-Sora 1.2 uses a 12-layer subset of the original 1.1-billion-parameter architecture, with the DWS module integrated into each layer.The implementation retains the Open-Sora VAE and uses T5-small for text encoding.Source: §B.1 · p. 18
- iVideoGPT is an autoregressive transformer with a 114M tokenizer and 138M transformer, and the DWS module is integrated into each transformer block.The model also includes specialized reward-prediction head-layers for video generation and reward estimation.Source: §B.1 · pp. 18–19
B.2 Implementation of Motion-Reinforced Loss
The section adapts motion-reinforced loss to diffusion and autoregressive transformer models. It incorporates motion weights into diffusion’s squared-L2 loss and defines a corresponding loss for transformers trained with cross-entropy.
B.2 Implementation of Motion-Reinforced Loss
- Diffusion models use squared-L2 loss, with motion weights incorporated into the original objective.The prediction and video embedding have identical dimensionality, allowing the weights to be incorporated.Source: §B.2 · p. 19
- For diffusion models, the target signal is ground-truth noise in DDPM-style models and velocity in flow-matching models.Source: §B.2 · p. 19
- Autoregressive transformers use cross-entropy training, and the section specifies a motion-reinforced loss with c = e and I as an indicator function.The final expression is presented as the transformer-specific Lmotion.Source: §B.2 · p. 19
B.3 Implementation of the MBRL Algorithm
The implementation combines a DWS-trained world model with PPO in an MBPO framework and uses prioritized imagination. Its algorithm prioritizes real replay samples for imagined rollouts, while actor-critic learning follows the stated implementation settings.
B.3 Implementation of the MBRL Algorithm
- The method uses a DWS-trained world model within MBPO, with PPO as the base actor-critic algorithm.Source: §B.3 · p. 19
- Actor-critic learning uses the same hyperparameters and architecture as the referenced Procgen and Atari implementations, and reward prediction uses symlog.Model-based RL hyperparameters are listed in Table 4.Source: §B.3 · p. 19, Table 4 · p. 20
- Real replay samples are prioritized using max(δ, 0) + ϵ, then sampled to start k-step policy rollouts added to the imagined buffer.Actor-critic updates use PPO minibatches drawn from imagined and real replay buffers.Source: §B.3 · p. 20