PaperlayerView exact revision on arXiv

Source-linked AI summary

MAGVIT: Masked Generative Video Transformer

Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, Lu Jiang

arXiv:2212.05199v2cs.CV

TL;DR

Video synthesis requires a model that can handle diverse generation tasks efficiently rather than relying on separate task-specific procedures. MAGVIT addresses this with a 3D video tokenizer and masked video token modeling in one multi-task transformer. It reports strong benchmark quality, substantially faster inference, and support for ten tasks across diverse visual domains.

  • Problem

    Existing video generation approaches motivate a need for efficient multi-task video generation using masked token modeling and transformer-based synthesis.

  • Method

    MAGVIT quantizes videos into spatial-temporal visual tokens with a 3D tokenizer and uses multivariate masked token embeddings for multi-task conditional generation.

  • Results

    MAGVIT achieves best-published FVD on UCF-101, BAIR Robot Pushing, and Kinetics-600 while generating videos two orders of magnitude faster than diffusion models and 60 times faster than autoregressive models.

  • Takeaways & Limitations

    A single MAGVIT model supports ten diverse video generation tasks and synthesizes videos across distinct visual domains.

Abstract

from arXiv · show

We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spatial-temporal visual tokens and propose an embedding method for masked video token modeling to facilitate multi-task learning. We conduct extensive experiments to demonstrate the quality, efficiency, and flexibility of MAGVIT. Our experiments show that (i) MAGVIT performs favorably against state-of-the-art approaches and establishes the best-published FVD on three video generation benchmarks, including the challenging Kinetics-600. (ii) MAGVIT outperforms existing methods in inference time by two orders of magnitude against diffusion models and by 60x against autoregressive models. (iii) A single MAGVIT model supports ten diverse generation tasks and generalizes across videos from different visual domains. The source code and trained models will be released to the public at https://magvit.cs.cmu.edu.

1. Introduction

MAGVIT is a single masked-token video generation model designed for diverse tasks, combining a high-fidelity 3D tokenizer with multi-task masked token modeling. It achieves strong benchmark quality, fast inference, and broad task and domain flexibility.

  • Approach: MAGVIT combines a 3D spatial-temporal quantization model with an embedding method for masked token modeling in multi-task video generation.The tokenizer maps videos to a low-dimensional spatial-temporal token manifold, while the masking scheme models video conditions using multivariate masks.
  • Quality: 77% lower FVD improves class-conditional generation on UCF-101 from 332 to 76.This result compares MAGVIT with the cited state-of-the-art baseline.
  • Quality: 26% and 38% lower FVD produce the best reported results for frame prediction on BAIR Robot Pushing and Kinetics-600, respectively.FVD decreases from 84 to 62 on BAIR and from 16 to 9.9 on Kinetics-600.
  • Efficiency: 0.25 seconds generates a 16-frame 128×128 clip in 12 steps on one TPUv4i, while the base V100 variant reaches 37 fps at 128×128.At the same resolution, MAGVIT is two orders of magnitude faster than a video diffusion model and 60 times faster than an autoregressive video transformer.
  • Flexibility: A single trained MAGVIT model performs ten video generation tasks, including interpolation, prediction, inpainting, and outpainting.The model also synthesizes videos across domains including object actions, autonomous driving, and multi-view object-centric videos.

2. Preliminaries: Masked Image Synthesis

Masked image synthesis uses discrete visual tokens and a transformer trained to reconstruct randomly masked tokens. Non-autoregressive decoding then iteratively predicts tokens in parallel while retaining high-confidence predictions.

  • Two-stage synthesis: A VQ autoencoder quantizes an image into a flattened sequence of discrete tokens before transformer-based masked token modeling.The framework separates tokenization from masked-token prediction into two synthesis stages.
  • Masked token modeling: A binary mask replaces selected tokens with [MASK], and a BERT predicts the masked tokens conditioned on optional prefix tokens.Training minimizes cross-entropy between predictions and ground-truth tokens at masked positions.
  • Mask sampling: Random mask ratios are sampled using a cosine scheduling function after uniformly sampling per-token mask scores.The threshold is set by the selected order statistic, determining which tokens become [MASK].
  • Non-autoregressive decoding: At inference, all tokens are initially masked and are generated through iterative parallel prediction, retaining the highest-scoring tokens at each step.The mask ratio follows the schedule function with a deterministic current-step input.

3. Masked Generative Video Transformer

MAGVIT combines spatial-temporal video tokenization with multi-task masked token modeling to generate videos from diverse conditions. Its COMMIT masking embeds interior conditions, addresses tokenizer-induced information leakage, and supports conditional non-autoregressive decoding.

  • Framework: MAGVIT quantizes videos into discrete spatial-temporal tokens with a 3D-VQ autoencoder, then trains a video transformer using multi-task masked token modeling.The encoder maps videos to latent tokens and the decoder maps tokens back to pixels.
  • Spatial-Temporal Tokenization: The VQ autoencoder sets a generation-quality bound and determines token sequence length, directly affecting inference efficiency.MAGVIT therefore designs its video quantizer to perform favorably against other video VQ models.
  • Spatial-Temporal Tokenization: The 3D-VQ design models temporal dynamics by expanding convolutions into 3D and combining 3D and 2D downsampling layers.Three-dimensional downsampling is used in shallower encoder layers, while the decoder mirrors the encoder.
  • Multi-Task Masked Token Modeling: Directly unmasking interior-condition regions can leak ground-truth information through the tokenizer’s non-local receptive field, causing non-causal masking and poor generalization.COMMIT is designed to provide causal masking for these conditions.
  • Multi-Task Masked Token Modeling: COMMIT embeds interior conditions as corrupted visual tokens within a multivariate mask rather than directly unmasking conditioned regions.Interior conditions include spatial regions or observed frames for inpainting, outpainting, prediction, and interpolation.
  • Multi-Task Masked Token Modeling: The multi-task objective refines condition tokens, predicts masked tokens, and reconstructs target tokens while a bidirectional transformer predicts the target sequence.At inference, non-autoregressive decoding progressively replaces generated tokens and ultimately refines the interior condition tokens.
  • Multi-Task Masked Token Modeling: COMMIT uses fixed-length sequences for conditions with arbitrary regional volume, avoiding padding tokens and improving training and memory efficiency.The framework covers ten tasks, including prediction, interpolation, outpainting, inpainting, and class-conditional generation.

4. Experimental Results

MAGVIT is evaluated for quality, efficiency, and multi-task flexibility across standard video benchmarks and diverse generation settings. It achieves strong fidelity while substantially reducing inference cost and supporting multiple tasks with shared models.

  • Experimental Setup: MAGVIT is evaluated on class-conditional generation, frame prediction, multi-task generation, and additional visual domains.The experiments use UCF-101, BAIR, Kinetics-600, SSv2, nuScenes, Objectron, and Web videos.
  • Single-Task Video Generation: 62 FVD on BAIR and 9.9 FVD on Kinetics-600 improve previous best results of 84 and 16, respectively.The reported relative reductions are 26% on BAIR and 38% on Kinetics-600.
  • Inference-Time Generation Efficiency: 37 fps at 128×128 on a V100 demonstrates fast generation, while MAGVIT is 60 times faster than an autoregressive transformer.MAGVIT also runs at 190 fps for the base model and 65 fps for the large model on a TPUv4i.
  • Multi-Task Generation: A single model performs eight tasks on BAIR or ten tasks on SSv2, with multi-task models achieving better fidelity across all evaluated tasks.The multi-task evaluation is intended to demonstrate a generic video synthesis model rather than compare dedicated task-specific models.
  • Ablation Study: The 3D-VQ tokenizer provides better reconstruction quality than 2D-VQ despite higher compression and fewer parameters.The proposed VQ performs favorably against similarly sized baselines and improves with a larger model.

5. Related Work

Prior video synthesis methods include GANs, autoregressive transformers, non-autoregressive transformers, and diffusion models. MAGVIT differs by using a non-autoregressive transformer designed for greater efficiency and multi-task flexibility.

  • GAN-Based Approaches: GAN-based video synthesis achieved early success but is associated with training instability and limited generation diversity.
  • Autoregressive Transformers: Autoregressive transformers model visual dynamics sequentially, with TATS using hierarchical transformers and 3D-VQGAN tokens for long videos.
  • Non-Autoregressive Transformers: MAGVIT introduces a non-autoregressive transformer with higher efficiency and flexibility than prior autoregressive approaches.
  • Non-Autoregressive Transformers: Compared with related non-autoregressive methods, MAGVIT uses an embedding mask to model multi-task video conditions with better quality.

6. Conclusion

MAGVIT is presented as a generic, efficient mask-based video generation model combining a high-quality 3D-VQ tokenizer with COMMIT for multi-task conditional masked token modeling. Experiments report state-of-the-art quality on three video generation benchmarks.

  • Conclusion: MAGVIT combines a high-quality 3D-VQ tokenizer with COMMIT for multi-task conditional masked token modeling.
  • Conclusion: MAGVIT establishes state-of-the-art quality for class-conditional generation on UCF-101 and frame prediction on BAIR and Kinetics-600.

Supplementary Materials

The supplementary document expands the paper with architectural, implementation, quantitative, multi-task, cross-domain, and qualitative materials. It covers additional benchmarks and generated-video examples.

  • Supplementary Organization: Appendix A describes MAGVIT’s 3D-VQ architectures and transformer models, while Appendix B provides training and evaluation details.
  • Supplementary Organization: Appendix C adds benchmark comparisons, SSv2 multi-task results, and results on nuScenes, Objectron, and Web video datasets.
  • Supplementary Organization: Appendix D provides additional qualitative examples of generated videos, supplemented by a demo webpage.

A. MAGVIT Model Architecture

MAGVIT combines a 3D video tokenizer, BERT-style transformer, and task-specific conditioning to support ten video-generation tasks with one model. Its architecture uses spatial-temporal quantization and masked-token training, with base and large variants for both tokenizer and transformer.

  • 3D-VQ module: MAGVIT compares a 3D-VQ architecture with TATS, using average-pooling downsampling, resizing-plus-convolution upsampling, and a mirrored encoder-decoder design.A single deeper 3D discriminator replaces separate shallow 2D and 3D discriminators, and the vocabulary is reduced to 1,024 tokens.
  • Model variants: The 3D-VQ module has base and large variants with 41M and 158M parameters, respectively, excluding discriminators.
  • Transformer: MAGVIT uses BERT-style transformers with base and large configurations containing 87M and 306M parameters.A huge transformer is used only for the large Web video dataset and demonstration videos.
  • Multi-task conditioning: A single trained model supports ten tasks defined by adjustable interior conditions, padding functions, and optional class-label prefixes.The tasks include prediction, interpolation, inpainting, outpainting, class-conditional generation, and class-conditional frame prediction.

B.3. Evaluation

MAGVIT evaluation uses FVD as the primary metric alongside selected image-quality and inception metrics, with standardized sampling protocols across several datasets. The BAIR evaluation additionally addresses bias caused by its small evaluation set through a debiased protocol.

  • Metrics: FVD is the primary evaluation metric, while UCF-101 additionally uses IS and BAIR uses PSNR, SSIM, and LPIPS.FVD features are extracted with an I3D model trained on Kinetics-400; UCF-101 IS uses a C3D model trained on UCF-101.
  • Sampling protocols: Standard evaluation samples 16-frame clips and computes FVD distributions from generated samples and clips drawn from benchmark videos.The protocol runs through required samples with one data loader and reports means and standard deviations across four folds.
  • BAIR: BAIR evaluation uses 25,600×4 generated samples at 64×64 resolution, with the first 16-frame clip from each of 256 evaluation videos as the real distribution.
  • Debiased FVD: BAIR FVD is difficult because the evaluation target contains only 256 16-frame clips, making the standard real distribution highly biased.A training-set experiment reports FVD 64 from 256 videos versus 13 using all 43K videos.
  • Debiased FVD: Uniformly sampling 16-frame clips from the 256 BAIR evaluation videos yields 3,840 clips and changes MAGVIT-L-FP from FVD 62 to 31.The revised value is described as more aligned with the model’s training-set FVD of 8 and is used for BAIR multi-task evaluation and ablations.

C. Additional Quantitative Evaluation

Additional evaluations test MAGVIT on class-conditional generation, frame prediction, multi-task generation, token reconstruction, and videos from new visual domains. Results support strong benchmark performance, consistent multi-task gains, reconstruction fidelity, and scalable generalization.

  • Class-conditional generation: MAGVIT-L improves UCF-101 class-conditional generation from FVD 332 to 76, a 77% reduction, and raises IS from 79.28 to 89.27.It outperforms TATS and Make-A-Video, despite Make-A-Video using substantially more pretraining data.
  • Frame prediction: MAGVIT achieves state-of-the-art FVD on BAIR and Kinetics-600 frame prediction, with a 39% relative improvement on Kinetics over Video Diffusion.The benchmark uses one conditioned frame on BAIR and five conditioned frames on Kinetics-600.
  • Multi-task generation: Multi-task models achieve consistently better average FVD across all tasks than single-task unconditional-generation and frame-prediction baselines.The SSv2 evaluation covers ten tasks and 174 everyday-object actions, including tasks unseen during training.
  • Cross-domain generalization: MAGVIT generalizes to nuScenes, Objectron, and 12M Web videos across frame prediction, frame interpolation, and an eight-task suite.These evaluations use the same learning recipe on additional video domains.
  • Tokenizer reconstruction: MAGVIT 3D-VQ reconstructs consistent frames with greater details for both static and moving pixels than the compared 2D-VQ and TATS 3D-VQ tokenizers.
  • Scalable tokenization: A tokenizer trained on 16-frame 128×128 videos reconstructs unseen YouTube videos at 240p and arbitrary lengths, including 4,096 frames, with a 614.4× compression rate.The compression combines factors of 4 temporally, 8×8 spatially, and 24 bits to 10 bits per element.

D.2. Single-Task Generation Examples

Visual comparisons show MAGVIT generating higher-quality videos with more substantial motion and stronger visual consistency than prior methods across UCF-101, BAIR, and Kinetics-600.

  • UCF-101: On UCF-101, MAGVIT produces higher-quality frames with substantial motion, whereas CCVS+StyleGAN shows little motion and TATS exhibits artifacts.
  • BAIR: On BAIR frame prediction, MAGVIT clips maintain better visual consistency and spatial-temporal dynamics than RaMViD.The comparison uses a one-frame condition.
  • Kinetics-600: On Kinetics-600, MAGVIT generates plausible actions with greater details than RaMViD given conditioned frames.MAGVIT generates at 128×128 while RaMViD generates at 64×64; standard evaluation uses 64×64.

D.3. Multi-Task Generation Examples

MAGVIT demonstrates multi-task video generation across datasets and visual domains, while its tokenizer preserves detailed, temporally consistent reconstructions. Examples include conditional generation, prediction, and repeated outpainting for panorama videos.

  • Multi-task generation: A single MAGVIT model performs 10 generation tasks when trained on Something-Something-V2, demonstrating multi-task flexibility.Additional models trained on nuScenes, Objectron, and Web videos show flexibility across visual domains.
  • Outpainting: MAGVIT generates panorama videos by repeatedly applying vertical outpainting to a small 64×128 slice, expanding it to 384×128.The examples show five outpainting operations on each side of the initial slice.
  • Tokenization: The 3D tokenizer improves temporal consistency over 2D-VQ while retaining more moving-object detail than TATS 3D-VQ.The tokenizer comparison describes flicker from 2D-VQ and detail loss for moving objects with TATS 3D-VQ.
  • Tokenization: Despite a compression rate of 614.4, reconstructed videos preserve stunning details and are almost indistinguishable from real videos.The representation uses a codebook of size 1024 for discrete spatial-temporal tokens.
  • Conditional generation: On UCF-101, MAGVIT generates higher-quality frames and more motion than comparison methods, whose outputs show weak motion or visible artifacts.CCVS+StyleGAN shows little or no motion, while TATS produces motion with clear artifacts.
  • Conditional generation: MAGVIT produces visually consistent BAIR frame predictions and plausible, detailed actions on unseen Kinetics-600 conditioned-frame evaluations.The BAIR examples emphasize visual consistency and spatial-temporal dynamics, while Kinetics-600 examples emphasize action plausibility and detail.
Loading 2212.05199v2…