Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

• On ImageNet at 256ˆ256 resolution, the best language model (Lee et al., 2022) trails the diffusion model (Gao et al., 2023) by 48% (FID 3.41 vs. 1.79).Source: §1 · p. 1
• With good visual tokens, masked language models beat state-of-the-art diffusion models in fidelity and efficiency across image and video benchmarks under comparable data, model size, and training budgets; MAGVIT-v2 also surpasses MAGVIT in generation and action recognition.The action-recognition comparison spans two setups and three datasets.Source: §1 · pp. 1–2
• Discrete visual tokens share language tokens’ form, allowing reuse of LLM optimizations; they can also act as compressed video representations and self-supervised targets associated with robust, generalizable understanding.Source: §1 · pp. 1–2
• Built on MAGVIT (Yu et al., 2023a) within the VQ-VAE framework, MAGVIT-v2 uses lookup-free quantization for a large vocabulary and tokenizer modifications enabling shared image and video tokens.Source: §1 · p. 2
• User studies find MAGVIT-v2 compression better than MAGVIT and HEVC, and on par with VVC.The paper reports codec comparisons at similar bit rates.Source: §1 · p. 2

Language Model for visual generation

• Language models generate visual content by tokenizing an image or video into discrete codes, flattening them in raster-scan order, and modeling the sequence with a transformer.The tokenizer’s vocabulary size defines the available codes; a decoder converts generated tokens back into pixels.Source: §2 · p. 2
• Autoregressive LMs predict each next token from preceding tokens and conditioning, while masked LMs iteratively fill masked sequences and re-mask low-confidence tokens.Masked-LM decoding begins with a fully masked sequence and follows a decreasing masking-ratio schedule.Source: §2 · pp. 2–3

Denoising Diffusion Models

• Diffusion and language models increasingly overlap through latent-space generation and transformer architectures, but retain a fundamental distinction: continuous versus discrete latent formats.Diffusion models are described as state of the art in image and video generation; recent approaches diffuse over latents rather than raw pixels.Source: §2 · p. 3

Visual tokenization

• VQ-VAE encodes video into embeddings, quantizes each embedding to its nearest codebook entry, and decodes the resulting representation; VQGAN adds adversarial and perceptual losses to improve image quality.Discrete tokens are represented by the codebook indices, with the embedding dimension dropped.Source: §2 · p. 3
• MAGVIT advances video tokenization with a 3D architecture, image-pretraining initialization, and robust losses, enabling leading video-generation quality, but struggles with images and flickers in longer videos.The paper identifies MAGVIT as the state of the art in video tokenization.Source: §2 · p. 3

3.1 LOOKUP-FREE QUANTIZER

• For VQ-VAE, reconstruction FID improves as vocabulary grows while language-model generation FID eventually worsens; LFQ improves both with larger vocabularies.Figure 1 compares the trends on ImageNet at 128×128 resolution using a 306–372M-parameter MLM.Source: §3.1 · p. 4, Figure 1 · p. 4
• LFQ eliminates codebook embedding lookup by setting embedding dimension to zero and, in the described variant, quantizing independent binary dimensions with a sign function.Training adds an entropy penalty to encourage codebook utilization; this paper focuses on the simplest independent-binary variant, while other LFQ methods are left for future research.Source: §3.1 · p. 4

3.2 VISUAL TOKENIZER MODEL IMPROVEMENT

• Causal 3D convolutions make the first frame independent of later frames, enabling image tokenization within a joint image-video tokenizer; among tested designs, causal 3D CNN performs best.The shared-codebook goal addresses MAGVIT’s difficulty tokenizing images because its 3D CNN has a temporal receptive field.Source: §3.2 · p. 5
• Beyond causal layers, MAGVIT changes down- and up-sampling, delays temporal downsampling, adds discriminator blur pooling and decoder adaptive group normalization, and factorizes LFQ tokens for smaller transformers.Factorized subspace tokens are embedded separately and summed for transformer inputs; weight tying shares embedding and softmax weights, with separate prediction heads for subspaces.Source: §3.2 · p. 5

4.1 EXPERIMENTAL SETUPS

• The generation experiments use K600 and UCF-101 for video, ImageNet for images, MCL-JCV for compression, and K400 and SSv2 for video understanding.Training follows Yu et al. (2023a) unless otherwise stated, using LFQ with a default codebook size of 2^18 and an annealed entropy-loss weight.Source: §4.1 · p. 6

4.2 VISUAL GENERATION

• MAGVIT-v2 surpasses prior methods on K600 and UCF-101 video generation; on ImageNet 512×512, its MLM achieves FID 1.91 vs. 2.65 for the best diffusion baseline.The ImageNet comparison uses the same training data and comparable model size and training budget. At 256×256, the FID margin narrows, but the MLM uses 50% less model size and 64 vs. 250 sampling steps.Source: Table 2 · p. 7, §4.2 · p. 7, Table 1 · p. 7

4.3 VIDEO COMPRESSION

• At 0.0384 bpp, MAGVIT-v2 beats MAGVIT on LPIPS, PSNR, and MS-SSIM, and beats all compared methods on LPIPS; raters prefer it to MAGVIT, HEVC, and VVC at multiple bit rates.The subjective study used 30 MCL-JCV videos and 16 raters, each answering roughly 800 pairwise-preference questions. Standard codecs may preserve local details better but introduce block artifacts; CPU efficiency remains an open challenge.Source: §4.3 · pp. 7–8, Figure 6 · p. 7, Table 3 · p. 8

4.4 VIDEO UNDERSTANDING

• For action recognition, MAGVIT-v2 outperforms MAGVIT, and decoded-token performance approaches that of the same ViViT backbone trained on ground-truth pixels, though it remains below state of the art.The reported improvements are attributed to the new tokenizer.Source: §4.4 · pp. 8–9, Table 4 · p. 8

4.5 ABLATION STUDY

• Ablations examine architecture choices, LFQ and other improvements, and vocabulary size across ImageNet and UCF-101.The section also refers to LFQ-versus-VQ and vocabulary-size ablations in Figure 1; no ablation outcomes are supplied in these passages.Source: §4.5 · p. 9, Table 5 · p. 9

Visual tokenization

• Prior visual tokenizers use transformer blocks, independent-frame processing, or 3D CNNs, while MAGVIT-v2 enables an LM-modelable 262k vocabulary linked to improved image and video generation.ViT-VQGAN replaces CNNs with transformer blocks for image tokenization (Yu et al., 2022a), and C-ViViT extends this approach to video (Villegas et al., 2022). Early video methods treat frames as independent images without temporal compression (Wu et al., 2022; Gupta et al., 2022), while later work uses 3D CNNs for spatiotemporal volumes (Yan et al., 2021; Ge et al., 2022; Yu et al., 2023a). Previous VQ codebooks were relatively small, around 8k, whereas MAGVIT-v2 induces a 262k vocabulary that can be modeled effectively by an LM.Source: §5 · p. 9

Text-to-{image, video}

• Language models and diffusion models both advance text-to-image and text-to-video generation, but inconsistent training data and conditions make their performance difficult to compare fairly.The paper therefore prioritizes ImageNet and Kinetics benchmarks for a fairer comparison.Source: §5 · p. 9

Diffusion models

• Diffusion evolved from pixel-space denoising to continuous VAE-latent and binary-latent approaches, while Transformer or hybrid denoisers blur its distinction from language models.The section identifies latent format as a key distinction: diffusion models use continuous latents, whereas language models use discrete latents.Source: §5 · p. 9

• MAGVIT-v2 combines lookup-free quantization with architectural advancements to tokenize images and videos using a shared vocabulary.Source: §6 · p. 9
• MAGVIT-v2 outperforms the previously leading video tokenizer in visual generation, video compression, and action recognition in videos.Source: §6 · p. 9
• The results suggest that strong visual tokenization is key to enabling language models to excel at image and video generation, motivating further exploration of tokenization methods designed for LLMs.Source: §6 · p. 9

A.1 IMAGE AND VIDEO GENERATION

• Two image tokenizers downsample by factors of 16 and 32 for 256x256 and 512x512 generation, with both representing images as 16x16 tokens.Source: §A.1 · p. 15
• Both image tokenizers are trained on ImageNet’s training set for 270 epochs with batch size 256 using 256x256 images.Source: §A.1 · p. 15
• The image-generation masked language model uses token factorization and is trained for 1080 epochs with batch size 1024.The training follows Yu et al. (2023a) and matches the 1080-epoch schedule of MDT (Gao et al., 2023).Source: §A.1 · p. 15
• Image training randomly crops 80–100% of each image while preserving aspect ratio, flips images horizontally, and drops class labels in 10% of batches.Class-label dropping enables classifier-free guidance (Ho & Salimans, 2021).Source: §A.1 · p. 15
• Unguided decoding uses temperatures 30 at 512x512 and 15 at 256x256; guided decoding uses guidance scale 25 and temperature 15.Guided generation adopts Gao et al. (2023)’s guidance schedule with temperature scaling from Lezama et al. (2023).Source: §A.1 · p. 15
• For video modeling, a 128x128 image tokenizer is inflated with causal padding in the temporally last slice; the discriminator is trained from scratch, and the tokenizer trains on Kinetics-600 for 190 epochs at batch size 256.The causal video tokenizer is also used for video-compression and action-recognition evaluations.Source: §A.1 · p. 15
• Kinetics-600 frame prediction conditions on the first 2x16x16 tokens of a 5x16x16-token representation of a 17x128x128 clip; its masked language model trains for 360 epochs at batch size 256 and uses cosine-schedule sampling at temperature 32.The condition corresponds to the first 5 frames in the standard frame-prediction setup.Source: §A.1 · p. 15

A.2 MODEL SETUP AND HYPERPARAMETERS

• The model uses four residual blocks, base channels 128, VQVAE channel multipliers 1, 2, 2, 4, and discriminator multipliers 2, 4, 4, 4, 4.Initialization uses central inflation from a 2D model trained on ImageNet with this setup.Source: §A.2 · p. 15
• Training combines reconstruction loss weight 5.0, entropy loss weight 0.1, perceptual loss weight 0.1, commitment loss weight 0.25, and adversarial loss weight 0.1.The generator uses non-saturating loss; entropy loss anneals over 2000 steps with factor 3, and the discriminator uses r1 gradient penalty cost 10 and LeCAM weight 0.001.Source: §A.2 · p. 15
• Optimization uses peak learning rate 10´4 with linear warmup and cosine decay, Adam with β1 0 and β2 0.99, EMA decay 0.999, and batch size 256.Source: §A.2 · p. 15
• Figure 7 presents the MAGVIT-v2 tokenizer architecture, with T-Causal Conv denoting temporally causal convolution.The accompanying setup uses 17 frames at stride 1 and 128 by 128 resolution, with a latent shape of 5 by 16 by 16 and vocabulary size 218.Source: §A.2 · p. 15, Figure 7 · p. 16

A.3 VIDEO COMPRESSION EVALUATION

• The study used 30 MCL-JCV videos at 640×360 pixels, with 16 raters each answering roughly 800 pairwise-preference questions on average.Raters compared videos without pausing them, using an interface adapted from the Challenge on Learned Image Compression.Source: §A.3 · p. 17
• Pairwise preferences were collected through two-alternative forced choice and converted into Elo scores to quantify relative visual quality.The rating interface extended the one used for the Challenge on Learned Image Compression to video comparisons (Fig. 8).Source: §A.3 · p. 17, Figure 8 · p. 17

Tokens as prediction targets

• MAGVIT-v2 video tokens are effective targets for masked-modeling-based video understanding in a simplified, video-only BEVT setup.The setup drops the image stream and multiple decoders, using an encoder-only architecture similar to BEiT.Source: §A.4 · p. 17

Tokens as inputs

• A factorized ViViT model trained on pixels retained performance with minimal drop when evaluated on MAGVIT-v2 detokenized pixels.The model follows Arnab et al. (2021) hyperparameters and uses 32 frames at 224p resolution; the Kinetics-600 experiment uses the Kinetics-400 hyperparameters.Source: §A.4 · p. 17

• Table 6 describes experimental configurations with tokens as targets, while Table 7 covers class-conditional ImageNet 256×256 generation evaluated under the ADM protocol.Source: Table 6 · p. 18, Table 7 · p. 19
• Figure 9 provides video-compression metrics supplementary to Table 3, and the paper directs readers to its project page for generated video samples.Source: B ADDITIONAL RESULTS · p. 17, Figure 9 · p. 18
• Text-to-image and text-to-video results are unavailable because training is ongoing; comparisons are challenging given different datasets, training conditions, and sometimes proprietary data.The authors intend to add generated videos in a later revision.Source: B ADDITIONAL RESULTS · p. 18
• Table 8 covers class-conditional UCF-101 video generation with AR-LM models; MAGVIT-v2 uses more embedding-table and softmax parameters because vocabulary factorization and weight tying are omitted.Source: Table 8 · p. 19