Source-linked AI summary
VideoGPT: Video Generation using VQ-VAE and Transformers
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, Aravind Srinivas
TL;DR
Natural video generation requires modeling complex spatial and temporal dependencies, motivating a compact likelihood-based approach. VideoGPT learns discrete downsampled video latents with a VQ-VAE and models them autoregressively with a GPT-like transformer, achieving competitive BAIR generation and realistic samples on UCF-101 and TGIF.
Problem
High-fidelity natural video generation remains less developed because videos require modeling complex correlations across space and time at high input dimensions.
Method
VideoGPT adapts VQ-VAE and Transformer models to videos, using space-time downsampling, 3D convolutions, axial attention, and a GPT-like prior over discrete latents.
Results
VideoGPT generates BAIR samples competitive with state-of-the-art GAN-based methods and realistic samples from UCF-101 and the Tumblr GIF dataset.
Takeaways & Limitations
VideoGPT provides a simple reference architecture for likelihood-based transformer video generation and reports ablations of key design choices.
Takeaways & Limitations
The authors do not claim likelihood-based models are better than GANs, and VideoGPT overfits UCF-101, suggesting that dataset requires further exploration.
Abstract
from arXiv · showhide
We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html
1. Introduction
VideoGPT addresses the difficulty of modeling high-fidelity natural videos by compressing videos into discrete latent representations and autoregressively modeling those latents. It reports competitive BAIR results, realistic UCF-101 and TGIF samples, ablations, and conditional-generation results.
- Motivation: Natural video generation remains challenging because videos require modeling high-dimensional correlations across both space and time.The paper identifies this complexity and associated compute demands as reasons for slower progress than in image, audio, and text generation.
- Approach: VideoGPT chooses likelihood-based autoregressive modeling and models discrete VQ-VAE latents with a GPT-like architecture.This design reflects the authors’ stated preference for objectives that are convenient to train and evaluate, rather than an empirical claim that likelihood models outperform GANs.
- Approach: VideoGPT uses learned downsampled video representations to remove spatial and temporal redundancies before generative modeling.The paper motivates downsampling as a way to spend computation on fewer, more useful bits while retaining realistic reconstructions.
- Results: 103 FVD against real BAIR samples and 94 FVD* against reconstructions make VideoGPT competitive with existing video-generation methods.These are the reported BAIR Robot Pushing results, with FVD and FVD* benchmarked against different references.
- Results: VideoGPT generates realistic samples from UCF-101 and the Tumblr GIF dataset and supports action-conditional generation on BAIR and VizDoom.The paper also reports ablations covering axial attention, latent-space size, codebooks, and autoregressive-prior capacity.
2. Background
VQ-VAE compresses inputs into discrete codebook representations and reconstructs them, while GPT-like transformers autoregressively model discrete data with masked self-attention. The VQ-VAE objective combines reconstruction, codebook, and commitment losses.
- VQ-VAE: VQ-VAE compresses high-dimensional data into a discretized latent space and reconstructs it from quantized encodings.An encoder maps inputs to latent vectors, nearest-neighbor lookup selects codebook embeddings, and a decoder reconstructs the input.
- VQ-VAE: The VQ-VAE objective combines reconstruction, codebook, and commitment losses.The commitment term is weighted by β and discourages encoder outputs from fluctuating between code vectors.
- VQ-VAE: An EMA codebook update is used because it empirically provides faster VQ-VAE training and convergence.
- Autoregressive models: GPT-like models factorize discrete-data distributions autoregressively and optimize them through maximum likelihood.Their standard architecture uses masked multi-head self-attention followed by pointwise MLP feedforward blocks.
3. VideoGPT
VideoGPT combines a space-time VQ-VAE with a transformer prior over its discrete latent codes. Its encoder uses 3D convolutions and axial attention, while conditioning is implemented through cross-attention or conditional normalization.
- VQ-VAE: VideoGPT’s VQ-VAE encoder downsamples videos over space-time with 3D convolutions and attention residual blocks using axial attention.The decoder reverses this process with attention residual blocks and 3D transposed convolutions that upsample over space-time.
- VQ-VAE: Learned spatio-temporal position embeddings are shared across axial-attention layers in the encoder and decoder.
- Latent prior: The second stage trains an autoregressive prior over the VQ-VAE latent codes using an Image-GPT-style transformer with dropout regularization.
- Conditional generation: Frame conditioning uses cross-attention, whereas action and class conditioning use affine conditional functions in transformer LayerNorm.
4. Experiments
Experiments evaluate VideoGPT across video datasets and architectural choices. The model produces realistic, diverse samples, while ablations identify trade-offs among latent resolution, downsampling, attention, and prior capacity.
- Natural video datasets: VideoGPT captures complex interactions in TGIF without the overfitting observed on the smaller UCF-101 dataset.TGIF samples include camera movement, scene changes, and human-object dynamics; UCF-101 shows lower test than train loss.
- Architecture ablations: Axial attention improves VQ-VAE reconstruction and generation quality compared with a parameter-adjusted no-attention variant.The comparison evaluates reconstruction NMSE and reconstruction FVD.
- Architecture ablations: Larger transformer priors improve BAIR performance, with sample quality gains continuing up to approximately eight layers.The latent representation allows a larger prior under memory constraints.
- Latent-space ablations: A balanced 2 × 2 × 2 temporal-spatial downsampling ratio, corresponding to 8 × 32 × 32 latents, gives the best sample-quality trade-off.Very small latents reconstruct poorly, while the largest latent setting limits prior capacity under compute constraints.
- Latent-space ablations: Increasing codebook size improves BAIR reconstruction but does not improve sample quality after a base generation-quality threshold, while one codebook improves performance.The fixed latent size in the codebook experiment is 8 × 32 × 32.
5. Related Work
Prior video-generation work includes prediction-oriented and GAN-based approaches, while VideoGPT adapts VQ-VAE for video generation.
- Video prediction methods generate future frames from past frames and sometimes robot actions.
- Video-to-video methods condition generation on abstract supervision such as segmentation masks, keypoints, poses, and edges.
- VideoGPT builds on VQ-VAE by adapting it for video generation, addressing the lack of a clean VQ-VAE video-generation architecture.
6. Conclusion
VideoGPT adapts VQ-VAE and Transformer models to video with minimal modifications and synthesizes videos competitive with state-of-the-art GAN-based methods.
- VideoGPT adapts VQ-VAE and Transformer models typically used for image generation to videos with minimal modifications.
- VideoGPT synthesizes videos competitive with state-of-the-art GAN-based video-generation models.
- The paper presents ablations on key VideoGPT design choices for future video-generation architecture design.
A.1. VQ-VAE Encoder and Decoder
The appendix identifies the VQ-VAE encoder and decoder hyperparameters for each dataset.
- Table 8 lists VQ-VAE encoder and decoder hyperparameters for each dataset.
- The table concerns the VQ-VAE encoder models used across datasets.
- The table also concerns the VQ-VAE decoder models used across datasets.
A.2. Prior Networks
The appendix identifies the hyperparameters of the prior networks used for each dataset.
- Table 9 lists hyperparameters for prior networks for each dataset.
- The table organizes prior-network settings by dataset.
- The appendix provides dataset-specific prior-network hyperparameters.
B. ViZDoom Samples
The ViZDoom samples show both unconditional generation and generation conditioned on fixed action sequences in two environments.
- Health gathering supreme: In the health gathering supreme environment, samples are shown unconditionally and conditioned on the same action sequence: turn right and go straight.The figure separates unconditional samples from action-conditioned samples.
- Comparison: Across both ViZDoom environments, the figures pair unconditional samples with action-conditioned samples using fixed movement instructions.The paired presentation is repeated for health gathering supreme and battle2.
- Battle2: In the battle2 environment, samples are shown unconditionally and conditioned on the same action sequence: moving forward and right.The figure compares unconditional samples with samples sharing one specified action sequence.