Source-linked AI summary
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, Lu Jiang
TL;DR
Language models have lagged diffusion models in visual generation, motivating better discrete visual representations for transformer-based modeling. The paper introduces MAGVIT-v2, a shared-vocabulary tokenizer using lookup-free quantization, and reports that it outperforms diffusion models and prior video tokenizers across multiple visual tasks. Its results support further exploration of visual tokenization for LLMs, while the presented LFQ variant remains a deliberately simple design.
Problem
Language models still underperform diffusion models on image and video generation, suggesting a need for visual representations suited to effective language-modeling of visual data.
Method
MAGVIT-v2 uses lookup-free quantization and architectural modifications to tokenize images and videos into compact discrete tokens with a shared vocabulary.
Results
MAGVIT-v2 outperforms prior video tokenizers across visual generation, video compression, and action recognition, and provides evidence that language models can outperform diffusion models on ImageNet.
Takeaways & Limitations
The results identify visual tokenization as key to enabling language models to excel in image and video generation and motivate further research on tokenizer design.
Takeaways & Limitations
The paper focuses on a simple lookup-free quantization variant with independent binary dimensions; other LFQ methods remain for future research.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation, one crucial component is the visual tokenizer that maps pixel-space inputs to discrete tokens appropriate for LLM learning. In this paper, we introduce MAGVIT-v2, a video tokenizer designed to generate concise and expressive tokens for both videos and images using a common token vocabulary. Equipped with this new tokenizer, we show that LLMs outperform diffusion models on standard image and video generation benchmarks including ImageNet and Kinetics. In addition, we demonstrate that our tokenizer surpasses the previously top-performing video tokenizer on two more tasks: (1) video compression comparable to the next-generation video codec (VCC) according to human evaluations, and (2) learning effective representations for action recognition tasks.
1 INTRODUCTION
Visual language models have lagged diffusion models because their discrete visual representations have been inadequate. The paper introduces MAGVIT-v2 to improve token quality and shows stronger performance across generation, compression, and recognition tasks.
- Motivation: 48%: On ImageNet at 256×256, the best language model trails diffusion with FID 3.41 versus 1.79.ImageNet is described as a gold-standard image-generation benchmark.
- Motivation: A good visual tokenizer lets masked language models surpass state-of-the-art diffusion models in image and video generation fidelity and efficiency under comparable resources.The comparison uses the same training data, comparable model size, and training budget.
- Motivation: Discrete visual tokens share the form of language tokens, enabling reuse of language-model training, inference, infrastructure, scaling, and hardware optimizations.The paper presents shared vision-language token space as groundwork for multimodal language models.
- MAGVIT-v2: MAGVIT-v2 maps images and videos into compact discrete tokens using a shared vocabulary.The tokenizer is built within the VQ-VAE framework and extends MAGVIT.
- MAGVIT-v2: Lookup-free quantization enables a large vocabulary, while tokenizer modifications improve generation quality and support joint image-video tokenization.The design targets tokens compatible with language-model learning.
- Results: MAGVIT-v2 outperforms MAGVIT in visual generation, video compression, and action recognition.User studies find compression better than MAGVIT and HEVC, and comparable to VVC.
2 BACKGROUND
Visual generation with language models converts images and videos into discrete token sequences before transformer modeling. The background distinguishes autoregressive and masked language models, explains VQ-VAE-style tokenization, and motivates improved video tokenizers.
- LM visual generation: A visual tokenizer maps an image or video into discrete tokens, flattens them in raster-scan order, and feeds the sequence to an LM transformer.The representation has vocabulary size K and reduced temporal and spatial dimensions.
- Autoregressive LMs: Autoregressive language models predict each next token from preceding tokens and conditioning information, then decode the generated tokens back into pixels.Inference uses standard autoregressive decoding.
- Masked LMs: Masked language models randomly hide tokens, predict the masked positions from observed tokens, and iteratively fill a fully masked sequence during generation.The process repeatedly samples predictions and re-masks low-probability tokens under a decreasing schedule.
- Visual tokenization: VQ-VAE-style decoding supplies token embeddings to a decoder that reconstructs the input, while VQGAN adds adversarial and feature-level perceptual losses for image quality.Discrete tokens are represented by codebook indices.
- Video tokenization: Video tokenization is harder than image tokenization; MAGVIT improves it with a 3D architecture, image-pretraining initialization, and robust training losses.MAGVIT achieves leading generation quality across multiple video benchmarks but struggles with image tokenization.
3 METHOD
MAGVIT-v2 develops a compact discrete visual tokenizer for videos and images, combining lookup-free quantization with causal architectures and training modifications. The method targets language-model generation while supporting a shared image-video tokenization scheme.
- Tokenizer design: MAGVIT-v2 maps videos and images into compact discrete tokens suitable for language models.The tokenizer builds on MAGVIT within the VQ-VAE framework.
- Lookup-free quantization: Unlike standard VQ-VAE methods, LFQ makes reconstruction and language-model generation improve consistently as vocabulary size increases.With conventional methods, generation initially improves but deteriorates for larger vocabularies even as reconstruction improves.
- Lookup-free quantization: Lookup-free quantization replaces the VQ-VAE embedding codebook with an integer set, eliminating embedding lookup and enabling larger vocabularies.The design reduces the code embedding dimension to zero and uses vocabulary size K.
- Lookup-free quantization: The LFQ variant assumes independent binary dimensions and quantizes each latent dimension using the sign of its feature value.The representation decomposes into single-dimensional variables with binary codebook values {-1, 1}.
- Training: An entropy penalty encourages codebook utilization, while reconstruction, GAN, perceptual, commitment, and LeCAM losses train and stabilize the tokenizer.The standard codebook loss is excluded because it is inapplicable to LFQ.
- Joint image-video tokenization: Causally padded 3D convolutions make the first frame independent of later frames, allowing a shared tokenizer to process both videos and single images.The causal 3D CNN performs best among the compared designs, and temporal subsampling and upsampling preserve single-image tokenization.
4 EXPERIMENTS
Experiments evaluate MAGVIT-v2 across visual generation, video compression, and action recognition, using shared or matched settings against prior methods. The tokenizer improves generation quality, compression preferences, perceptual metrics, and learned video representations.
- Experimental setup: Experiments cover video and image generation, video compression, and action recognition using Kinetics-600, UCF-101, ImageNet, MCL-JCV, Kinetics-400, and SSv2.The evaluation follows MAGVIT protocols and uses the same MLM transformers to verify the tokenizer.
- Visual generation: MAGVIT-v2 surpasses diffusion models on ImageNet generation quality and inference-time efficiency.At 512×512, it achieves FID=1.91 versus 2.65 for the best diffusion baseline, a 28% improvement; at 256×256 it uses 64 versus 250 decoding steps.
- Visual generation: MAGVIT-v2 significantly outperforms MAGVIT on video-generation benchmarks while using the same MLM transformer backbone.It also outperforms the non-causal baseline on frame prediction, supporting the contribution of the causal tokenizer.
- Video compression: Raters prefer MAGVIT-v2 to MAGVIT, HEVC, and VVC at multiple bit rates in a 16-rater video-compression study.The study uses pairwise preferences and Elo scores across 30 MCL-JCV videos.
- Video compression: MAGVIT-v2 outperforms MAGVIT on LPIPS, PSNR, and MS-SSIM, and outperforms all methods on LPIPS.The authors note that further research is needed to adapt the model for efficient CPU execution like standard codecs.
5 RELATED WORK
The paper situates visual tokenization and text-to-image or text-to-video generation among approaches using language-model-based and diffusion-model-based systems.
- Related work spans visual tokenization, text-to-image and text-to-video benchmarks, and diffusion models.
6 CONCLUSION AND FUTURE WORK
The paper concludes that MAGVIT-v2 improves visual generation, video compression, and action recognition over the leading prior video tokenizer. These results support further exploration of visual tokenization for language models.
- MAGVIT-v2 uses lookup-free quantization and architectural advances to tokenize images and videos with a shared vocabulary.
- The tokenizer outperforms MAGVIT across visual generation, video compression, and video action recognition.
- The results suggest that visual tokenization is key to enabling language models to excel in image and video generation.
A IMPLEMENTATION DETAILS
Implementation uses separate image and causal video tokenizer setups, followed by masked-language-model training for image and video generation. The configurations specify downsampling, training schedules, and conditioning procedures.
- Image generation: Image tokenizers downsample by 16× and 32× for 256×256 and 512×512 generation, representing both resolutions with 16×16 tokens.Both tokenizers are trained on ImageNet for 270 epochs with batch size 256.
- Image generation: The ImageNet masked language model is trained for 1080 epochs with token factorization, batch size 1024, random crops, and horizontal flipping.Class labels are dropped for 10% of training batches to enable classifier-free guidance.
- Video generation: The causal video tokenizer inflates a 128×128 image tokenizer, modifies temporal initialization and discriminator training, and trains on Kinetics-600 for 190 epochs.Training uses batch size 256.
- Video generation: For a 17×128×128 clip, the causal tokenizer produces 5×16×16 tokens, with the first 2×16×16 tokens conditioning the first five frames.The video MLM is trained for 360 epochs and sampled with a cosine schedule at temperature 32.
A.2 MODEL SETUP AND HYPERPARAMETERS
MAGVIT-v2 uses a video-tokenizer architecture with specified training hyperparameters and a compact latent representation. The architecture includes temporally causal convolution.
- The model processes 17-frame video inputs at 128 × 128 resolution with a latent shape of 5 × 16 × 16.
- The setup uses base channels of 128, four residual blocks, and a vocabulary size of 218.
- MAGVIT-v2’s architecture is illustrated in Figure 7, including temporally causal convolution.T-Causal Conv refers to temporally causal convolution.
A.3 VIDEO COMPRESSION EVALUATION
The video-compression evaluation combines pairwise subjective ratings with Elo scores to quantify relative visual quality. It uses videos from MCL-JCV and responses from sixteen raters.
- Subjective compression quality is evaluated with a two-alternative forced choice rating methodology.The methodology produces binary pairwise decisions.
- Elo scores quantify relative visual quality from the raters’ pairwise preferences.
- The study uses 30 MCL-JCV videos scaled to 640 × 360 pixels and collects responses from 16 raters.
A.4 VIDEO UNDERSTANDING EXPERIMENTS
The video-understanding experiments test MAGVIT-v2 tokens both as masked-modeling targets and as inputs to a model trained on pixel data. The latter evaluates whether pixel-trained models can reuse tokenized video inputs.
- Tokens as prediction targets: A simplified BEVT setup pretrains visual encoders by predicting video tokens in a masked-modeling framework.The image stream is dropped, leaving only the video stream.
- Tokens as inputs: Video tokens can be used as inputs to a factorized ViViT model trained on pixels, with very minimal performance drop.
- Tokens as inputs: The token-input experiment uses a Base-sized ViViT model with 32-frame inputs at 224p resolution.
B ADDITIONAL RESULTS
Additional materials provide access to generated video samples, describe supplementary compression metrics and experimental configurations, and document the absence of text-to-image results. The appendix also includes ImageNet and UCF-101 generation tables.
- Generated video samples are available through the paper’s supplementary website.
- Figure 9 presents video-compression metrics supplementary to Table 3, while Table 6 lists configurations with tokens as targets.
- Text-to-image results are unavailable because the paper focuses on video tokenization, while Table 7 and Table 8 report ImageNet and UCF-101 generation configurations and results.The authors also note that scientific comparison of text-to-image or text-to-video models is challenging when datasets and training conditions differ.