PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoGPT: Video Generation using VQ-VAE and Transformers

Wilson Yan, Yunzhi Zhang, Pieter Abbeel, Aravind Srinivas

arXiv:2104.10157v2cs.CVcs.LG

TL;DR

Natural video generation requires modeling complex spatial and temporal dependencies, motivating a compact likelihood-based approach. VideoGPT learns discrete downsampled video latents with a VQ-VAE and models them autoregressively with a GPT-like transformer, achieving competitive BAIR generation and realistic samples on UCF-101 and TGIF.

  • Problem

    High-fidelity natural video generation remains less developed because videos require modeling complex correlations across space and time at high input dimensions.

    Source: §1 · p. 1

  • Method

    VideoGPT adapts VQ-VAE and Transformer models to videos, using space-time downsampling, 3D convolutions, axial attention, and a GPT-like prior over discrete latents.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    VideoGPT generates BAIR samples competitive with state-of-the-art GAN-based methods and realistic samples from UCF-101 and the Tumblr GIF dataset.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    VideoGPT provides a simple reference architecture for likelihood-based transformer video generation and reports ablations of key design choices.

    Source: §1 · p. 3, §6 · p. 9

    The authors do not claim likelihood-based models are better than GANs, and VideoGPT overfits UCF-101, suggesting that dataset requires further exploration.

    Source: §1 · p. 2, §4.5 · p. 5

Abstract

from arXiv · show

We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html

1. Introduction

VideoGPT addresses the difficulty of modeling high-fidelity natural videos by compressing videos into discrete latent representations and autoregressively modeling those latents. It reports competitive BAIR results, realistic UCF-101 and TGIF samples, ablations, and conditional-generation results.

Motivation

  • Natural video generation remains challenging because videos require modeling high-dimensional correlations across both space and time.The paper identifies this complexity and associated compute demands as reasons for slower progress than in image, audio, and text generation.Source: §1 · p. 1

Approach

  • VideoGPT chooses likelihood-based autoregressive modeling and models discrete VQ-VAE latents with a GPT-like architecture.This design reflects the authors’ stated preference for objectives that are convenient to train and evaluate, rather than an empirical claim that likelihood models outperform GANs.Source: §1 · p. 2
  • VideoGPT uses learned downsampled video representations to remove spatial and temporal redundancies before generative modeling.The paper motivates downsampling as a way to spend computation on fewer, more useful bits while retaining realistic reconstructions.Source: §1 · p. 2

Results

  • 103 FVD against real BAIR samples and 94 FVD* against reconstructions make VideoGPT competitive with existing video-generation methods.These are the reported BAIR Robot Pushing results, with FVD and FVD* benchmarked against different references.Source: §1 · p. 3
  • VideoGPT generates realistic samples from UCF-101 and the Tumblr GIF dataset and supports action-conditional generation on BAIR and VizDoom.The paper also reports ablations covering axial attention, latent-space size, codebooks, and autoregressive-prior capacity.Source: §1 · p. 3

2. Background

VQ-VAE compresses inputs into discrete codebook representations and reconstructs them, while GPT-like transformers autoregressively model discrete data with masked self-attention. The VQ-VAE objective combines reconstruction, codebook, and commitment losses.

VQ-VAE

  • VQ-VAE compresses high-dimensional data into a discretized latent space and reconstructs it from quantized encodings.An encoder maps inputs to latent vectors, nearest-neighbor lookup selects codebook embeddings, and a decoder reconstructs the input.Source: §2.1 · p. 3
  • The VQ-VAE objective combines reconstruction, codebook, and commitment losses.The commitment term is weighted by β and discourages encoder outputs from fluctuating between code vectors.Source: §2.1 · p. 3
  • An EMA codebook update is used because it empirically provides faster VQ-VAE training and convergence.Source: §2.1 · p. 3

Autoregressive models

  • GPT-like models factorize discrete-data distributions autoregressively and optimize them through maximum likelihood.Their standard architecture uses masked multi-head self-attention followed by pointwise MLP feedforward blocks.Source: §2.2 · p. 3

3. VideoGPT

VideoGPT combines a space-time VQ-VAE with a transformer prior over its discrete latent codes. Its encoder uses 3D convolutions and axial attention, while conditioning is implemented through cross-attention or conditional normalization.

VQ-VAE

  • VideoGPT’s VQ-VAE encoder downsamples videos over space-time with 3D convolutions and attention residual blocks using axial attention.The decoder reverses this process with attention residual blocks and 3D transposed convolutions that upsample over space-time.Source: §3 · p. 4
  • Learned spatio-temporal position embeddings are shared across axial-attention layers in the encoder and decoder.Source: §3 · p. 4

Latent prior

  • The second stage trains an autoregressive prior over the VQ-VAE latent codes using an Image-GPT-style transformer with dropout regularization.Source: §3 · p. 4

Conditional generation

  • Frame conditioning uses cross-attention, whereas action and class conditioning use affine conditional functions in transformer LayerNorm.Source: §3 · p. 4

4. Experiments

Experiments evaluate VideoGPT across video datasets and architectural choices. The model produces realistic, diverse samples, while ablations identify trade-offs among latent resolution, downsampling, attention, and prior capacity.

Natural video datasets

  • VideoGPT captures complex interactions in TGIF without the overfitting observed on the smaller UCF-101 dataset.TGIF samples include camera movement, scene changes, and human-object dynamics; UCF-101 shows lower test than train loss.Source: §4.6 · p. 5, §4.5 · p. 5

Architecture ablations

  • Axial attention improves VQ-VAE reconstruction and generation quality compared with a parameter-adjusted no-attention variant.The comparison evaluates reconstruction NMSE and reconstruction FVD.Source: §4.7 · p. 6
  • Larger transformer priors improve BAIR performance, with sample quality gains continuing up to approximately eight layers.The latent representation allows a larger prior under memory constraints.Source: §4.7 · p. 6

Latent-space ablations

  • A balanced 2 × 2 × 2 temporal-spatial downsampling ratio, corresponding to 8 × 32 × 32 latents, gives the best sample-quality trade-off.Very small latents reconstruct poorly, while the largest latent setting limits prior capacity under compute constraints.Source: §4.7 · p. 7
  • Increasing codebook size improves BAIR reconstruction but does not improve sample quality after a base generation-quality threshold, while one codebook improves performance.The fixed latent size in the codebook experiment is 8 × 32 × 32.Source: §4.7 · pp. 7–8

5. Related Work

Prior video-generation work includes prediction-oriented and GAN-based approaches, while VideoGPT adapts VQ-VAE for video generation.

  • Video prediction methods generate future frames from past frames and sometimes robot actions.Source: §5 · p. 8
  • Video-to-video methods condition generation on abstract supervision such as segmentation masks, keypoints, poses, and edges.Source: §5 · p. 8
  • VideoGPT builds on VQ-VAE by adapting it for video generation, addressing the lack of a clean VQ-VAE video-generation architecture.Source: §5 · p. 8

6. Conclusion

VideoGPT adapts VQ-VAE and Transformer models to video with minimal modifications and synthesizes videos competitive with state-of-the-art GAN-based methods.

  • VideoGPT adapts VQ-VAE and Transformer models typically used for image generation to videos with minimal modifications.Source: §6 · p. 9
  • VideoGPT synthesizes videos competitive with state-of-the-art GAN-based video-generation models.Source: §6 · p. 9
  • The paper presents ablations on key VideoGPT design choices for future video-generation architecture design.Source: §6 · p. 9

A.1. VQ-VAE Encoder and Decoder

The appendix identifies the VQ-VAE encoder and decoder hyperparameters for each dataset.

  • Table 8 lists VQ-VAE encoder and decoder hyperparameters for each dataset.Source: Table 8 · p. 13
  • The table concerns the VQ-VAE encoder models used across datasets.Source: Table 8 · p. 13
  • The table also concerns the VQ-VAE decoder models used across datasets.Source: Table 8 · p. 13

A.2. Prior Networks

The appendix identifies the hyperparameters of the prior networks used for each dataset.

  • Table 9 lists hyperparameters for prior networks for each dataset.Source: Table 9 · p. 13
  • The table organizes prior-network settings by dataset.Source: Table 9 · p. 13
  • The appendix provides dataset-specific prior-network hyperparameters.Source: Table 9 · p. 13

B. ViZDoom Samples

The ViZDoom samples show both unconditional generation and generation conditioned on fixed action sequences in two environments.

Health gathering supreme

  • In the health gathering supreme environment, samples are shown unconditionally and conditioned on the same action sequence: turn right and go straight.The figure separates unconditional samples from action-conditioned samples.Source: Figure 9 · p. 14

Comparison

  • Across both ViZDoom environments, the figures pair unconditional samples with action-conditioned samples using fixed movement instructions.The paired presentation is repeated for health gathering supreme and battle2.Source: Figure 9 · p. 14, Figure 10 · p. 14

Battle2

  • In the battle2 environment, samples are shown unconditionally and conditioned on the same action sequence: moving forward and right.The figure compares unconditional samples with samples sharing one specified action sequence.Source: Figure 10 · p. 14

Related papers

Loading 2104.10157v2…