Source-linked AI summary
VideoPoet: A Large Language Model for Zero-Shot Video Generation
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming-Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold, Lu Jiang
TL;DR
VideoPoet targets the limited quality of LLM-based video generation relative to video diffusion models. It trains a decoder-only multimodal LLM with two-stage pretraining and task adaptation, achieving state-of-the-art high-fidelity motion and zero-shot generation across video tasks.
Problem
LLM-based video generation had not reached video diffusion models’ quality in tasks such as text-to-video generation.
Method
VideoPoet uses a decoder-only LLM over discrete image, video, and audio tokens, trained with multimodal autoregressive pretraining followed by task adaptation.
Results
VideoPoet demonstrates state-of-the-art generation of large, high-fidelity motions and zero-shot video generation for novel inputs and tasks.
Takeaways & Limitations
A unified multimodal LLM can serve as a foundation for multitask video creation, including multiple forms of editing.
Takeaways & Limitations
Compressed and quantized token reconstruction limits visual fidelity, while small objects and fine-grained details remain difficult, especially with significant motion.
Abstract
from arXiv · showhide
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundation that can be adapted for a range of video generation tasks. We present empirical results demonstrating the model's state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions. Project page: http://sites.research.google/videopoet/
1. Introduction
VideoPoet explores LLM-based video generation as an alternative to dominant diffusion approaches, using one multimodal model for diverse video tasks. It reports state-of-the-art motion quality and zero-shot capabilities.
- VideoPoet addresses the gap between LLM-based video generation and the quality of video diffusion models.
- The approach reuses LLM training, inference, and scaling infrastructure while integrating diverse tasks in a unified model.
- The model is a decoder-only LLM that represents image, video, and audio modalities as discrete tokens.
- Training combines multimodal autoregressive pretraining with task adaptation, producing one model for text-to-video, image-to-video, editing, and stylization.
- VideoPoet demonstrates state-of-the-art generation of large, high-fidelity motions and coherent videos up to 10 seconds.
- The model supports zero-shot generation for inputs outside the training distribution and for new tasks formed by chaining editing operations.
2. Related Work
Related work contrasts diffusion-based video generation with transformer language models that support multimodal task integration and zero-shot behavior. VideoPoet extends video-language modeling beyond paired text-video training.
- Video diffusion methods dominate recent text-to-video and video-to-video editing research, often deriving from text-to-image diffusion models.
- Transformer language models can combine multiple tasks during pretraining and demonstrate zero-shot capabilities across modalities.
- Prior text-to-video transformer work commonly relies on paired data, whereas VideoPoet also uses unpaired videos and multiple tasks from the same video.
- VideoPoet extends video-language modeling to text- and multimodal-conditioned tasks through synergistic pretraining across tasks.
3. Model Overview
VideoPoet converts visual, audio, and text inputs into a shared token-based interface for a decoder-only language model, followed by token-space video super-resolution. Local attention addresses the cost of long high-resolution sequences.
- VideoPoet comprises modality-specific tokenizers, a language-model backbone, and a super-resolution module.
- Image, video, and audio data become discrete tokens in a unified vocabulary, while text enters as embeddings.
- The prefix language model conditions on multimodal tokens and autoregressively predicts visual and audio tokens.
- High-resolution autoregressive generation can produce 35,840 tokens, making direct sampling computationally impractical.
- A custom non-autoregressive token-space super-resolution transformer uses windowed local attention to reduce long-sequence self-attention costs.
- The super-resolution transformer uses token factorization with k = 2 to replace a 262,144-way classification problem with two 512-way problems.
4. LLM Pretraining for Generation
VideoPoet pretraining uses a mixture of multimodal prefix-to-output tasks, shared task signaling, and strategies for varied lengths and modalities. Fine-tuning on higher-quality data adapts the pretrained model to specific or new tasks.
- The pretraining mixture conditions on text, visual, and audio inputs while predicting visual or audio outputs.
- Tasks include unconditioned generation, text-to-video, future prediction, image-to-video, painting, stylization, audio-to-video, and video-to-audio.
- A unique <task> token identifies output types, while compatible tasks can share the same task token across different context signals.
- Omitting end-of-sequence markers during text-to-image pretraining enables longer continuous video generation and improves initial-frame quality.
- Video tokens are generated at 128×128 or 128×224 resolution and at lengths of 17 or 41 frames, encoded at 8 frames per second.
- Stylization reconstructs videos from text, optical flow, and depth during training, then varies the text prompt at inference.
- Alternating Gradient Descent groups tasks by sequence length to achieve a near 0% padding ratio during multitask training.
- Two-stage pretraining shifts sampling from 90% images and 10% videos to 90% videos and 10% images across training.
5. Experiments
VideoPoet is evaluated as a unified, multimodal video-generation model across zero-shot benchmarks, human preferences, runtime, and diverse generation capabilities. Results highlight competitive motion quality, task generalization, long-video extension, and remaining fidelity and aesthetic limitations.
- Evaluation setup: Training-task combinations are compared using 300M-parameter models trained for 300k steps with learning rate 10^-3 and batch size 1024.The analysis covers text-to-video, text-to-image, frame prediction, Painting, audio-video continuation, and other pretraining tasks.
- Evaluation setup: The evaluation uses zero-shot protocols across text-to-video, frame prediction, and inpainting/outpainting, with one model performing all tasks.The evaluation datasets were not used for training.
- Pretraining task analysis: Using all pretraining tasks gives the best average performance across evaluated tasks, while self-supervised-only training shows a significant disparity and requires text-paired data.The authors note that multitask performance can decrease slightly when each task receives insufficient training.
- Zero-shot text-to-video: VideoPoet performs favorably on CLIP similarity and FVD across zero-shot MSR-VTT and UCF-101 text-to-video benchmarks, with higher CLIPSIM after high-quality finetuning.The pretrained foundation model is already competitive on all reported metrics.
- Human evaluation: Human evaluations find VideoPoet competitive with state-of-the-art video diffusion models, with significant wins in motion interestingness and realism; Lumiere alone surpasses it on video quality.The comparison uses proportions of trials in which VideoPoet was preferred or less preferred to alternatives.
- Runtime: On TPUv5p with four chips, the base model takes 34s to generate 17 frames at 8fps, while detokenization takes 1.3s and super-resolution 6.8s.The reported amortized runtime is about 5 seconds per second of output video, and the model was not optimized.
- Diverse capabilities: Autoregressive extension supports coherent videos beyond 10 seconds and iterative clip extension, while task chaining enables novel editing sequences within one multimodal transformer.Examples include image-to-video animation followed by video-to-video stylization and video outpainting followed by editing.
- Diverse capabilities: VideoPoet can apply camera motions and produce reasonable backside visualizations without dedicated 3D-consistency data or losses.The model uses a small proportion of videos and texts describing camera motion.
6. Conclusion
VideoPoet uses discrete visual, text, and audio tokens in a unified large language model to generate compelling, state-of-the-art-quality videos. Its pretrained model supports diverse video-generation capabilities, including multiple forms of editing.
- VideoPoet is a large language model trained on discrete visual, text, and audio tokens.
- The model generates videos with compelling state-of-the-art quality.
- A unified architecture and vocabulary support multitask video creation and diverse capabilities, including multiple forms of editing.
Impact Statement
The paper identifies misuse and ethical risks associated with VideoPoet, including deceptive or harmful generated content. It describes watermarking and transparency as mitigation strategies.
- VideoPoet could be misused to create deceptive or harmful content such as misinformation or deepfakes.
- Ethical considerations include avoiding harmful stereotypes and respecting cultural sensitivities.
- The paper describes digital watermarking for traceability and accountability, alongside transparency in model design.
A.1. Responsible AI and Fairness Analysis
The fairness analysis evaluates generated outputs for protected attributes using standardized prompts spanning professions, people descriptors, and viewing adverbs.
- The analysis evaluates fairness for perceived age, perceived gender expression, and perceived skin tone.
- It constructs 306 prompts using the template “a {profession or people descriptor} looking {adverb} at the camera”.
- Professions are drawn from the US Bureau of Labor and Statistics, while people descriptors include emotion state and socioeconomic class.
A.2. Model Scale and Performance
VideoPoet’s performance improves as model size and training-data volume increase across visual and audiovisual tasks. Larger models also improve temporal consistency, prompt fidelity, and motion dynamics while adding several capabilities.
- Performance improves across visual and audiovisual tasks as model size and training-data volume increase.The evaluated models have 300 million, 1 billion, and 8 billion parameters, trained on 10, 37, and 58 billion visual and audio tokens, respectively.
- Increasing model size improves temporal consistency, prompt fidelity, and motion dynamics.
- Larger models add limited text rendering, spatial understanding, and counting capabilities.
A.2.1. QUALITATIVE COMPARISON OF 1B AND 8B MODELS
The 8B model produces more stable, prompt-faithful, and dynamically consistent videos than the 1B model under the same prompts and settings. Scaling also adds limited text rendering, spatial understanding, counting, and complex layout-following capabilities.
- A.2.1. QUALITATIVE COMPARISON OF 1B AND 8B MODELS: The 8B model is more temporally stable and better follows complex prompts than the 1B model.The 1B model shows large subject changes and misses prompt elements, whereas the 8B model better preserves content over time and follows layouts.
- A.2.1. QUALITATIVE COMPARISON OF 1B AND 8B MODELS: The 8B model renders a single letter clearly, while the 1B model still produces artifacts.
- A.2.1. QUALITATIVE COMPARISON OF 1B AND 8B MODELS: The 8B model learns spatial positioning, including a river in front of the astronaut and horse.
- A.2.1. QUALITATIVE COMPARISON OF 1B AND 8B MODELS: The 8B model follows a stop-motion instruction to make items disappear one by one and handles complicated layouts.
- A.2.1. QUALITATIVE COMPARISON OF 1B AND 8B MODELS: Overall, scaling improves temporal consistency, prompt fidelity, and motion dynamics while adding limited text rendering, spatial understanding, and counting.
A.3. Additional Generated Examples
The additional examples cover stylization, camera movement, human comparison, tokenization, super-resolution, and evaluation procedures. Together, they illustrate VideoPoet’s multimodal pipeline and breadth of video-generation tasks.
- A.4. Video Stylization: Video stylization combines text, optical flow, depth, and sometimes the first video frame to control appearance and structure.Text generally defines output content or appearance, while optical flow and depth control structure; inference varies the text prompt to change style.
- A.4. Video Stylization: VideoPoet outperforms Control-A-Video by a large margin on stylization, and human raters prefer its text fidelity and video quality.The stylization evaluation uses 20 DAVIS videos with two style prompts per video.
- Additional Generated Examples: VideoPoet conditions on text, visual tokens, and audio tokens, then autoregressively predicts visual and audio tokens.Tokenizers map images, video frames, and audio waveforms into a unified vocabulary before multimodal generation.
- Additional Generated Examples: The super-resolution pipeline uses two successive 2× spatial stages, with 1B and 500M models respectively.The stages increase video resolution from 448×256 to 896×512 and then use the same MAGVIT-v2 tokenizer as the base model.