Source-linked AI summary
WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens
Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, Jiwen Lu
TL;DR
Existing world models are largely confined to specific scenarios, whereas general world dynamics matter for video generation. WorldDreamer addresses this gap by predicting masked visual tokens with multimodal prompts, and it performs across natural and driving scenes while supporting several generation and editing tasks.
Problem
Existing world models focus on specific scenarios such as gaming or driving, limiting coverage of general world dynamics needed for video generation.
Method
WorldDreamer maps visual inputs to discrete tokens, predicts randomly masked tokens with STPT, and conditions prediction on text and action embeddings.
Results
WorldDreamer generates videos across natural scenes and driving environments and supports text-to-video, image-to-video, video editing, and action-to-video tasks.
Takeaways & Limitations
WorldDreamer provides a versatile general world-modeling approach for video generation and is approximately 3× faster than diffusion-based methods.
Takeaways & Limitations
Text or action embeddings can be empty, but the paper does not establish that every task requires both modalities.
Abstract
from arXiv · showhide
World models play a crucial role in understanding and predicting the dynamics of the world, which is essential for video generation. However, existing world models are confined to specific scenarios such as gaming or driving, limiting their ability to capture the complexity of general world dynamic environments. Therefore, we introduce WorldDreamer, a pioneering world model to foster a comprehensive comprehension of general world physics and motions, which significantly enhances the capabilities of video generation. Drawing inspiration from the success of large language models, WorldDreamer frames world modeling as an unsupervised visual sequence modeling challenge. This is achieved by mapping visual inputs to discrete tokens and predicting the masked ones. During this process, we incorporate multi-modal prompts to facilitate interaction within the world model. Our experiments show that WorldDreamer excels in generating videos across different scenarios, including natural scenes and driving environments. WorldDreamer showcases versatility in executing tasks such as text-to-video conversion, image-tovideo synthesis, and video editing. These results underscore WorldDreamer's effectiveness in capturing dynamic elements within diverse general world environments.
1. Introduction
WorldDreamer introduces a general world model for video generation that learns motion and physics from visual signals rather than targeting only specific scenarios. It uses masked visual-token prediction with multimodal prompts and demonstrates broad video-generation versatility.
- Masked visual-token prediction models motion and physics by encoding images into discrete tokens, masking some tokens, and predicting them from unmasked tokens.
- STPT focuses attention on localized patches within temporal-spatial windows to learn visual dynamics and accelerate training convergence.
- WorldDreamer integrates language and action signals through cross-attention, forming multimodal prompts for interaction with the world model.
- WorldDreamer parallel-decodes videos in a few iterations and is approximately 3× faster than diffusion-based methods.
- WorldDreamer introduces a general world model for video generation that learns general world motion and physics.
- Experiments cover natural scenes and driving environments, including text-to-video, image-to-video, video editing, and action-to-video generation.
2. Related Work
Video-generation research includes Transformer-based and diffusion-based approaches, while world-model research has largely developed around gaming and structured driving scenarios. The supplied related work positions WorldDreamer against these model families and domain-specific world models.
- Video-generation models are primarily classified as Transformer-based or diffusion-based methods.
- Transformer-based methods: Transformer-based methods derive from LLMs and generate videos through autoregressive next-token prediction or parallel masked-token decoding.
- Diffusion based methods: Diffusion-based methods extend image diffusion into video using temporal latent representations, lightweight 3D latent spaces, or pretrained text-to-image models.
- World models: Early world models focused on gaming, while later Dreamer work validated them across diverse gaming scenarios.
3. WorldDreamer
WorldDreamer converts visual inputs into discrete tokens, masks them, and predicts the missing tokens with STPT conditioned on text and action embeddings. Its architecture combines localized spatiotemporal attention, multimodal cross-attention, and dynamic masking to support efficient video generation and editing.
- WorldDreamer encodes images and videos as discrete visual tokens, masks them, and decodes predicted tokens into videos or edits.
- Model Architecture: VQGAN downsamples visual inputs by 16× into tokens, while text uses pretrained T5 embeddings and actions use MLP-generated embeddings.
- Text and action signals are separately encoded and concatenated as multimodal prompt embeddings, with either modality allowed to be empty for unconditional learning.
- Model Architecture: STPT uses 3D convolution, spatial-temporal patchwise self-attention, and spatial-wise cross-attention to combine visual and multimodal features.
- WorldDreamer generates videos in about 10 steps, compared with approximately 30 diffusion steps and 200 autoregressive steps.
- Model Architecture: STPT maps processed visual features to the VQGAN codebook size, predicts masked-token probabilities, and is optimized with cross-entropy loss.
- Model Architecture: Joint training on videos and images augments training samples and improves comprehension of temporal and spatial visual aspects.
- Masking Strategy: Dynamic cosine-scheduled masking and shared masks across frames support parallel inference by predicting multiple output tokens in one forward pass.
4. Experiment
WorldDreamer is trained on diverse image and video data and evaluated through visual experiments spanning video generation and editing. It produces high-fidelity image-to-video and text-to-video results, realistic inpainting and stylization, and action-conditioned driving videos.
- Image to Video: WorldDreamer generates high-fidelity image-to-video outputs with smooth motion and frame composition that remains consistent with the initial image.The generated videos include cinematic landscape sequences and seamless frame-to-frame motion.
- Text to Video: Text prompts control video content, style, and camera motion across diverse stylistic paradigms, producing visually faithful results.The experiments include prompts describing scenes and stylistic transformations.
- Video Inpainting: Masked regions and text prompts enable realistic video inpainting that modifies specified areas while preserving a convincing video result.The method takes a mask outlining the target area and a prompt specifying the desired modification.
- Video Stylization: Random visual-token masks combined with style prompts enable realistic video stylization of original videos.The stylization experiment evaluates transformations driven by a style prompt and masked visual tokens.
- Action to Video: Different driving actions applied to identical initial frames produce distinct future videos, including separate left-turn and right-turn outcomes.This demonstrates action-conditioned generation in the autonomous-driving setting.
5. Conclusion
WorldDreamer is presented as a general world model that captures complex visual dynamics across natural and driving environments. Its demonstrated scope includes action-conditioned video generation and other video-generation tasks.
- WorldDreamer captures general world dynamics across natural scenes and driving environments, extending beyond scenario-specific world models.
- WorldDreamer supports text-to-video conversion, image-to-video synthesis, and video editing across diverse environments.
- WorldDreamer also generates videos conditioned on actions in autonomous-driving contexts.