MAGVIT: Masked Generative Video Transformer

1. Introduction

• MAGVIT models videos as latent visual-token sequences and predicts masked tokens using a high-fidelity 3D quantizer and a multivariate-mask embedding for video conditions.The embedding method is proposed to support multi-task video generation.Source: §1 · p. 2
• For frame prediction, MAGVIT improves FVD from 84 to 62 (↓26%) on BAIR and from 16 to 9.9 (↓38%) on Kinetics-600.The paper reports best-published fidelity on three widely used benchmarks.Source: §1 · p. 2
• For class-conditional UCF-101 generation, MAGVIT reduces FVD from 332 to 76 (↓77%).The paper identifies this as a state-of-the-art comparison.Source: §1 · p. 2
• MAGVIT generates a 16-frame 128×128 clip in 12 steps in 0.25 seconds on one TPUv4i and runs at 37 fps at 128×128 on a V100.At the same resolution, it is two orders of magnitude faster than a video diffusion model, 60 times faster than an autoregressive video transformer, and 4–16 times more efficient than a contemporary non-autoregressive video transformer.Source: §1 · p. 2
• A single trained MAGVIT model supports ten inference-time tasks, including interpolation, class-conditional frame prediction, inpainting, and outpainting.It also synthesizes complex scenes and motion across action-object, autonomous-driving, and multi-view object videos.Source: §1 · p. 2
• MAGVIT is presented as the first masked multi-task transformer for efficient video generation and manipulation, motivated by advances in generative image and video modeling.The approach draws inspiration from generative image transformers and combines masked-token modeling with multi-task learning.Source: §1 · pp. 1–2

2. Preliminaries: Masked Image Synthesis

• The framework first quantizes and flattens an image into discrete tokens with a VQ auto-encoder, then trains a transformer using masked token modeling [62] [71].For an image I ∈ R^(H×W×3), the encoder produces a token sequence z ∈ Z^N.Source: §2 · p. 2
• MaskGIT corrupts tokens with a binary mask, prepends condition tokens such as class labels, and trains a BERT to minimize cross-entropy on masked positions.The objective sums the negative log-probability of each ground-truth masked token conditioned on the prefix and corrupted sequence.Source: §2 · p. 2
• During training, MaskGIT samples per-token uniform mask scores and uses a cosine-scheduled mask ratio to set the cutoff for masking tokens.It samples r uniformly, then masks tokens whose scores fall below the ⌈γ(r)N⌉-th smallest score.Source: §2 · p. 2
• At inference, MaskGIT generates an image in K = 12 parallel decoding steps, retaining the highest-scoring token predictions and remasking the rest.The mask ratio follows the schedule γ(t/K), where t is the current step.Source: §2 · p. 2

3. Masked Generative Video Transformer

• MAGVIT uses a two-stage pipeline: a 3D VQ autoencoder turns videos into discrete tokens, then a video transformer learns through multi-task masked-token modeling.The model generates video from task-specific inputs such as a frame, a partially observed video, or a class identifier.Source: §3 · p. 2

3.1. Spatial-Temporal Tokenization

• The 3D-VQ tokenizer models temporal dynamics with 3D convolutions and mixed 2D/3D downsampling and upsampling; token sequence length affects generation efficiency, and tokenization sets a quality bound.For small datasets, the model initializes from 2D-VQ weights by filling the central slice of a zero-filled 3D kernel, and uses reflect padding to improve token consistency across locations.Source: §3.1 · p. 3

3.2. Multi-Task Masked Token Modeling

• COMMIT embeds interior conditions as corrupted visual tokens, addressing ground-truth leakage and poor generalization that can result from directly unmasking tokens within the tokenizer’s non-local receptive field.Its loss includes Lrefine for refining condition tokens, in addition to masked-token prediction and target-token reconstruction.Source: §3.2 · pp. 3–4
• COMMIT supports ten video-generation tasks, providing causal masking for interior conditions and fixed-length sequences across conditions with arbitrary regional volume.The tasks are frame prediction, frame interpolation, central, vertical, horizontal, and dynamic outpainting, central and dynamic inpainting, class-conditional generation, and class-conditional frame prediction.Source: §3.2 · p. 4
• At inference, COMMIT uses non-autoregressive decoding to generate video tokens in K steps, gradually reducing the mask ratio and refining interior-condition tokens.The passage gives 12 as an example number of decoding steps.Source: §3.2 · p. 4

4.1. Experimental Setups

• Evaluation covered UCF-101 class generation, BAIR and Kinetics-600 frame prediction, and 8–10-task BAIR/SSv2 synthesis; FVD was primary, with IS and image metrics also reported.Models trained with the same recipe on nuScenes, Objectron, and 12M web videos provide additional generalizability evaluations; quantitative results are in Appendix C.Source: §4.1 · p. 5

4.2. Single-Task Video Generation

• BAIR FVD falls from 84 to 62, while Kinetics-600 FVD improves from 16.2 to 9.9 (39% relative) over ’s prior best.The paper describes the BAIR improvement as a large margin and reports a debiased FVD in parentheses for that dataset.Source: Table 2 · p. 5, §4.2 · p. 5
• On UCF-101, MAGVIT surpasses prior best-published FVD and IS, including Make-A-Video [50] despite using only 9.5K training videos, and produces higher-quality frames with substantial motion.The comparisons describe CCVS+StyleGAN as having little or no motion and TATS as producing motion but with artifacts.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3. Inference-Time Generation Efficiency

• At 128×128, MAGVIT-B runs at 37 fps on one V100 GPU; on one TPUv4i, MAGVIT-B runs at 190 fps and MAGVIT-L at 65 fps.At 128×128, the paper reports MAGVIT as 60 times faster than autoregressive TATS and 4–16 times faster than MaskViT; diffusion models typically require 256–1000 steps.Source: Figure 5 · p. 6, §4.3 · p. 6

4.5. Ablation Study

• COMMIT uses a fixed-length sequence and generalizes better in single- and multi-task setups than latent masking or variable-length prefix conditioning.The paper reports that latent masking leads to poor generalization, especially for multi-task learning, while prefix conditioning makes sequences long and less tractable for multi-task learning.Source: §4.5 · pp. 6–7

4.4. Multi-task Video Generation

• Across eight BAIR and ten SSv2 tasks, multi-task models achieve better fidelity across tasks and better average FVD than single-task models, with a small same-size gain on BAIR frame prediction.Single-task models perform considerably worse on tasks unseen in training, especially tasks differing more from their training task; SSv2 evaluation covers 174 basic actions.Source: §4.4 · p. 6, Table 4 · p. 7

4.5. Ablation Study

• The proposed decoding algorithm gives the best quality with 3D-VQ, uses a sequence 4× shorter than 2D-VQ, and avoids the over 85× more inference steps required by AR.Source: §4.5 · pp. 7–8, Table 6 · p. 8
• 3D-VQ reconstructs video better than 2D-VQ despite higher compression, while ImageNet initialization improves both and central inflation outperforms average inflation.The proposed VQ performs favorably against similarly sized baseline architectures and improves further with a larger model.Source: Table 7 · p. 8, §4.5 · p. 8

GAN-based approaches

• GANs achieved early success in video synthesis [1,6,7,15,27,37,48,51,55,58,65,73], but training instability and limited generation diversity are known issues.Source: §5 · p. 8

Multi-task video synthesis

• MAGVIT presents an efficient non-autoregressive multi-task transformer and verifies its quality and efficiency on ten video-generation tasks, unlike Transframer’s frame-prediction-based autoregressive approach.The paper identifies multi-task video synthesis as under-studied and describes Transframer as the closest prior work.Source: §5 · p. 8

Autoregressive transformers

• Autoregressive video transformers model visual dynamics, with work shifting from raw pixels [61] to discrete latent codes [70]; TATS uses hierarchical transformers and 3D-VQGAN tokens.MAGVIT contrasts this line of work with its non-autoregressive transformer, which it describes as more efficient and flexible.Source: §5 · p. 8

Non-autoregressive transformers

• Concurrent non-autoregressive image methods [?, 12, 42, 75] include MaskGIT; MAGVIT proposes an embedding mask for multi-task video conditions and claims better quality than these approaches.Source: §5 · p. 8

Diffusion models

• Video diffusion extends image denoising diffusion with 3D U-Nets and joint image-video training, but sampling speed limits its application to video synthesis.MAGVIT presents a different approach aimed at high efficiency and compelling quality.Source: §5 · p. 8

Text-to-video

• MAGVIT leaves text-to-video for future work; training such models requires large, sometimes non-public paired text-image datasets [50] [63].All models are trained on public benchmarks except the Web video model.Source: §5 · p. 8

• MAGVIT establishes new state-of-the-art quality for class-conditional generation on UCF-101 and frame prediction on BAIR Robot Pushing and Kinetics-600.Source: §6 · p. 8
• MAGVIT introduces a high-quality 3D-VQ tokenizer and COMMIT for multi-task conditional masked token modeling.The tokenizer quantizes videos into tokens.Source: §6 · p. 8
• The paper presents MAGVIT as a generic, efficient model for video generation and evaluates its quality, efficiency, and flexibility for multi-task generation.Source: §6 · p. 8

Appendix Overview

• Appendix C adds comparisons with more published results on UCF-101, BAIR, and Kinetics-600.These are the three benchmarks discussed in the paper.Source: Appendix Overview · p. 12
• Appendix A describes the 3D-VQ architectures and transformer models used in MAGVIT.Source: Appendix Overview · p. 12
• Appendix B provides additional implementation details for training and evaluation.Source: Appendix Overview · p. 12
• Appendix C reports multi-task results on Something-Something-v2 (SSv2).Source: Appendix Overview · p. 12
• Appendix C also includes results on NuScenes, Objectron, and web video datasets.Source: Appendix Overview · p. 12
• Appendix D presents more qualitative examples of generated videos.Source: Appendix Overview · p. 12
• A demo video and additional generated examples are available on a webpage.Source: Appendix Overview · p. 12

A.1. 3D-VQ Tokenizer

• MAGVIT’s 3D-VQ design differs from TATS through average-pooling downsampling, nearest-resize-and-convolution upsampling, mirrored spatial/temporal sampling, one deeper discriminator, a smaller vocabulary, and different normalization, activation, and regularization choices.Its vocabulary has 1,024 entries versus TATS’s 16,384; it uses group normalization, Swish, and LeCAM regularization.Source: §A.1 · p. 12, Figure 7 · p. 13
• The 3D-VQ tokenizer has base and large variants with 41M and 158M parameters, respectively, excluding discriminators.Source: §A.1 · p. 12

A.2. Transformer

• MAGVIT uses BERT transformers with base and large variants of 87M and 306M parameters; a huge variant is reserved for large Web-video training and demo generation.Source: §A.2 · pp. 12–13, Table 8 · p. 14

B.1. Task Definitions

• The ten-task setup includes frame prediction from one initial frame with last-frame replication, and frame interpolation between one initial and one final frame.Interpolation pads by linearly interpolating between the given endpoint frames.Source: §B.1 · p. 13
• Outpainting tasks extend central, vertical, horizontal, or moving-strip conditions; inpainting tasks condition on everything except a central or moving rectangular region.The moving outpainting and inpainting strips travel left to right.Source: §B.1 · pp. 13–14
• Class-conditional generation uses a class-label prefix, while class-conditional frame prediction adds one initial frame and replicates it for padding.Together with the prediction, interpolation, outpainting, and inpainting variants, the task set contains ten tasks.Source: §B.1 · p. 14, Figure 8 · p. 14

B.2. Training

• MAGVIT training first learns the 3D-VQ tokenizer and then trains the transformer with that tokenizer frozen; the learning recipe is shared across datasets except for epoch count, and models are trained separately per dataset.Source: §B.2 · pp. 14–15
• Training uses 16-frame clips and a 4×16×16 latent with a 1,024-token vocabulary; the transformer sequence length is 1,026.The listed video resolution is 128×128, except 64×64 for BAIR.Source: §B.2 · p. 14

Evaluation metrics

• FVD is the primary metric; evaluation also reports IS on UCF-101 and PSNR, SSIM, and LPIPS on BAIR.FVD features come from an I3D model trained on Kinetics-400; UCF-101 IS uses a UCF-101-trained C3D model, and BAIR LPIPS uses VGG features.Source: §B.3 · p. 15

Sampling protocols

• For standard benchmarks, evaluation uses 16-frame clips sampled without replacement, runs the required sample count, and reports mean and standard deviation across four folds.For other datasets, the protocol is adapted to dataset size to ensure sample diversity.Source: §B.3 · p. 15

Debiased FVD on BAIR

• Using only 256 BAIR training videos gives FVD 64, versus 13 using all 43K videos, illustrating bias from the small evaluation set.The standard evaluation target contains only 256 16-frame clips.Source: §B.3 · p. 16
• Under uniform-sampled debiased BAIR evaluation, MAGVIT-L-FP scores FVD 31 instead of 62, closer to its training FVD=8.The protocol uniformly samples 16-frame clips from 256 evaluation videos, yielding 3,840 clips.Source: Table 11 · p. 16, §B.3 · p. 16

C. Additional Quantitative Evaluation

• Across ten tasks on Something-Something-v2, multi-task models achieve better average FVD than unconditional-generation and frame-prediction single-task baselines.The benchmark covers 174 everyday-object actions, and Table 12 marks tasks unseen during training and reports masked-pixel and masked-token proportions.Source: Table 12 · p. 17, C. Additional Quantitative Evaluation · p. 17
• On UCF-101, MAGVIT-L improves FVD from 332 to 76 (↓77%) and IS from 79.28 to 89.27 (↑13%), while MAGVIT-B compares favorably with TATS.CogVideo and Make-A-Video use additional pretraining data; the other models, including MAGVIT, are trained on UCF-101 data.Source: C. Additional Quantitative Evaluation · pp. 16–17
• MAGVIT achieves state-of-the-art FVD on BAIR Robot Pushing and Kinetics-600, with a 39% relative improvement over Video Diffusion on Kinetics-600.The comparison uses a 1-frame condition on BAIR and a 5-frame condition on Kinetics-600; some published FVDs are unavailable or incomparable.Source: C. Additional Quantitative Evaluation · p. 17
• MAGVIT’s results on nuScenes frame prediction, Objectron frame interpolation, and an eight-task suite on 12M Web Videos support generalization across distinct visual domains.Table 13 reports FVD for these datasets and tasks.Source: C. Additional Quantitative Evaluation · p. 17, Table 13 · p. 17

D.1. High-Fidelity Tokenization

• On UCF-101 reconstructions, MAGVIT 3D-VQ combines frame consistency with greater detail in static and moving pixels than MaskGIT 2D-VQ and TATS 3D-VQ.MaskGIT 2D-VQ shows flicker, while TATS 3D-VQ loses moving-object details.Source: §D.1 · p. 18
• Trained without labels on 16-frame 128×128 videos, MAGVIT 3D-VQ reconstructs unseen YouTube videos at 240p and arbitrary lengths, including 4,096 frames.It compresses by 4× temporally, 8×8 spatially, and 2.4× per element (24 bits to 10 bits), for a 614.4× total rate; reconstructions retain substantial detail.Source: §D.1 · p. 18

D.2. Single-Task Generation Examples

• On UCF-101, MAGVIT produces higher-quality frames with substantial motion than CCVS+StyleGAN and TATS.CCVS+StyleGAN yields little or no motion, while TATS generates motion with clear artifacts.Source: §D.2 · p. 18
• On BAIR frame prediction given a 1-frame condition, MAGVIT clips maintain better visual consistency and spatial-temporal dynamics than RaMViD.Source: §D.2 · p. 18
• On Kinetics-600 frame prediction given a 5-frame condition, MAGVIT generates plausible actions with greater detail than RaMViD.RaMViD generates at 64×64 and MAGVIT at 128×128; standard evaluation is carried out at 64×64.Source: §D.2 · p. 18

D.3. Multi-Task Generation Examples

• One MAGVIT model trained on Something-Something-V2 generates videos for 10 different tasks, demonstrating multi-task flexibility.The shown conditions come from Something-Something-V2 evaluation videos.Source: §D.3 · p. 18, Figure 15 · p. 25
• Three additional MAGVIT models trained on nuScenes [5], Objectron [3], and 12M Web videos also show multi-task generation results.The conditions are taken from each model’s evaluation set.Source: §D.3 · p. 18, Figure 16 · p. 26
• A 64×128 vertical slice can be expanded into a 384×128 panorama video by applying vertical outpainting five times on each side.The examples show MAGVIT generating ultrawide panoramas through repeated outpainting from a small condition.Source: §D.3 · p. 18, Figure 17 · p. 27, Figure 18 · p. 28