PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, Zehuan Yuan

arXiv:2406.06525v1cs.CV

TL;DR

Image generation has been dominated by diffusion models, while it remained unclear whether vanilla language-model-style autoregression could scale effectively for vision. LlamaGen reexamines image tokenizers, model scaling, and training data using Llama-like next-token prediction, achieving strong class-conditional results and competitive text-conditional quality and alignment.

  • Problem

    It remained unclear whether vanilla autoregressive models without visual inductive biases could achieve state-of-the-art image generation performance when scaled properly.

  • Method

    LlamaGen applies Llama-like next-token prediction to discrete image tokens while reexamining tokenizer design, model scalability, and training-data quality.

  • Results

    The models outperform popular diffusion models for class-conditional generation and demonstrate competitive visual quality and text alignment for text-conditional generation.

  • Takeaways & Limitations

    Vanilla autoregressive models can serve as the basis of image generation systems without inductive biases on visual signals.

  • Takeaways & Limitations

    Text-conditional models exhibit text-rendering, counting, and common-misconception errors due to training data and model-parameter limits.

Abstract

from arXiv · show

We introduce LlamaGen, a new family of image generation models that apply original ``next-token prediction'' paradigm of large language models to visual generation domain. It is an affirmative answer to whether vanilla autoregressive models, e.g., Llama, without inductive biases on visual signals can achieve state-of-the-art image generation performance if scaling properly. We reexamine design spaces of image tokenizers, scalability properties of image generation models, and their training data quality. The outcome of this exploration consists of: (1) An image tokenizer with downsample ratio of 16, reconstruction quality of 0.94 rFID and codebook usage of 97% on ImageNet benchmark. (2) A series of class-conditional image generation models ranging from 111M to 3.1B parameters, achieving 2.18 FID on ImageNet 256x256 benchmarks, outperforming the popular diffusion models such as LDM, DiT. (3) A text-conditional image generation model with 775M parameters, from two-stage training on LAION-COCO and high aesthetics quality images, demonstrating competitive performance of visual quality and text alignment. (4) We verify the effectiveness of LLM serving frameworks in optimizing the inference speed of image generation models and achieve 326% - 414% speedup. We release all models and codes to facilitate open-source community of visual generation and multimodal foundation models.

1 Introduction

LlamaGen examines whether vanilla autoregressive models using the language-model next-token paradigm can scale to competitive image generation. It reworks tokenizers, model scaling, and training data, reporting strong class-conditional results, competitive text-conditional results, and faster inference.

  • LlamaGen adopts the same architecture and next-token prediction paradigm as language models, reducing inductive biases on visual signals.
  • 0.94 rFID and 97% codebook usage are achieved by the downsample-ratio-16 image tokenizer on ImageNet.
  • 2.18 FID is achieved by the largest 3.1B-parameter class-conditional model on ImageNet 256×256, outperforming LDM and DiT.
  • A 775M-parameter text-conditional model trained on LAION-COCO and high-aesthetics images demonstrates competitive visual quality and text alignment.
  • 326% - 414% speedup is achieved by applying the vLLM serving framework to image-generation inference.
  • The released models remain behind state-of-the-art diffusion models, while larger AR models above 7B parameters are left for future exploration.

2 Autoregressive Models for Image Generation

LlamaGen converts images into discrete tokens and generates them autoregressively with a largely Llama-based architecture. The design combines quantized autoencoding, conditional prefilling, classifier-free guidance, scalable model sizes, and LLM-derived serving optimizations.

  • 2.1 Overview: Images are quantized into discrete tokens, reshaped in raster order, generated by next-token prediction, and decoded back into pixels.
  • 2.2 Image Tokenizer: The quantized autoencoder uses an encoder-quantizer-decoder pipeline in which feature vectors are mapped to nearest codebook entries and reconstructed by the decoder.
  • 2.2 Image Tokenizer: L2-normalized codebook vectors, low codebook dimension, and large codebook size improve reconstruction quality and codebook usage.
  • 2.2 Image Tokenizer: The tokenizer trains with straight-through gradients, vector-quantization and commitment losses, plus pixel, perceptual, and adversarial reconstruction losses.
  • 2.3 Llama Architecture: The model follows Llama with RMSNorm, SwiGLU, rotary positional embeddings, and 2D RoPE, while omitting AdaLN to preserve the LLM-like structure.
  • 2.4 Conditional Generation: Class and text conditions are supplied as prefilling embeddings; text conditioning uses FLAN-T5 XL followed by an additional MLP.
  • 2.4 Conditional Generation: Classifier-free guidance combines conditional and unconditional logits during inference to improve visual quality and text-image alignment.
  • 2.5 Serving: Models scale to 3.1B parameters and use LLM-community training and inference techniques, with vLLM producing 326% - 414% speedup over baseline.

3 Experiments

Experiments evaluate tokenizer design, token count, model scaling, sampling, cross-model comparisons, text conditioning, and inference acceleration. Results show that tokenizer choices and scalable autoregressive models support strong image-generation performance, while text quality and sampling involve measurable trade-offs.

  • 3.1 Image Tokenizer: Lower codebook vector dimensions and larger codebooks improve image-tokenizer reconstruction quality and codebook usage.Reducing vector dimension from 256 to 32 to 8 consistently improves both metrics, while increasing codebook size from 4096 to 16384 benefits overall performance.
  • 3.1 Image Tokenizer: 0.99 rFID is achieved by increasing image representation from 256 to 576 tokens at downsample ratio 16.The 256-token representation yields 2.43 rFID, whereas 576 tokens substantially improve reconstruction quality on ImageNet.
  • 3.1 Image Tokenizer: The tokenizer outperforms previous image tokenizers and remains competitive with continuous latent representations on ImageNet and COCO.The comparison includes VQGAN, MaskGIT, and ViT-VQGAN, while COCO evaluation supports applicability to both object-centric and scene-centric images.
  • 3.2 Class-conditional Image Generation: 3.06 FID is the limit for 256 tokens, while 576 tokens can improve image-generation FID further.Image token count is not strongly correlated with reconstruction quality alone: for models below 1B parameters, 256 tokens can outperform 576 tokens for generation, but fewer tokens constrain the best achievable FID.
  • 3.2 Class-conditional Image Generation: Scaling from LlamaGen-B to LlamaGen-XXL produces notable FID improvements, whereas further scaling to 3B yields only marginal gains.The authors suggest dataset size may limit the benefit of additional parameters.
  • 3.2 Class-conditional Image Generation: CFG improves visual quality, with optimal FID at CFG = 2.0, while stronger guidance trades diversity for fidelity.Increasing CFG raises precision and lowers recall; top-k, top-p, and temperature changes likewise expose fidelity–diversity trade-offs, and maximum top-k is selected because FID is the main metric.
  • 3.2 Class-conditional Image Generation: The 3B model outperforms LDM and DiT, while LlamaGen outperforms prior autoregressive models across parameter scales.The models are competitive across FID, IS, Precision, and Recall, supporting vanilla autoregressive models as a basis for advanced image-generation systems.
  • 3.3 Text-conditional Image Generation: Stage II training improves visual aesthetic quality, while the text-conditional model remains competitive for long-caption text-image alignment.Stage I captures text-image alignment; fine-tuning on high-aesthetic images substantially improves visual quality, and PartiPrompts evaluates longer captions.

4 Related Work

Visual generation methods differ in their modeling paradigms, while image tokenization commonly converts pixels into semantic representations. Multimodal foundation models have achieved visual understanding, but unified understanding and generation remain early-stage.

  • Autoregressive models predict the next image token, whereas diffusion models generate images through reverse diffusion from noise.
  • Masked-prediction models use BERT-style masked-token prediction, while autoregressive models use GPT-style next-token prediction.
  • Image tokenization converts pixel-space images into semantic-space representations to ease modeling and improve generation quality.
  • Vision-and-language models have achieved versatile visual understanding through visual instruction tuning, but unifying understanding and generation remains in its early stages.

5 Conclusion

The paper studies vanilla autoregressive models for scalable image generation by reconsidering tokenizers, generation models, and training data. Its class-conditional models outperform popular diffusion models, while text-conditional models show competitive visual quality and text alignment.

  • LlamaGen reexamines image tokenizers, image generation models, and training data for scalable vanilla autoregressive image generation.
  • Class-conditional LlamaGen models outperform popular diffusion models on image generation benchmarks.
  • Text-conditional LlamaGen models demonstrate competitive visual quality and text alignment.

A Examples of Image-Text Pair Data

The text-conditional model uses two training stages: a filtered LAION-COCO subset followed by internal high-aesthetic-quality images with generated captions.

  • Training stage I: Stage I retains about 50M LAION-COCO image-text pairs after filtering for URLs, aesthetics, watermarks, CLIP similarity, and image size.
  • Training stage II: Stage II fine-tunes on 10M internal high-aesthetic-quality images captioned in detail by LLaVA.
  • Training stage II: The first sentence of each long LLaVA caption is used as a short caption to augment text-conditional training.

B More Results on ImageNet Benchmark

The benchmark appendix provides ImageNet evaluation details and sample grids across class-conditional resolutions and text-conditional training stages. Evaluation settings and displayed resolutions are explicitly specified for the reported experiments.

  • Benchmark setup: ImageNet evaluation resizes generated images to 256×256.
  • Benchmark setup: Tables 8, 9, and 10 report detailed class-conditional ImageNet 256×256 performance under specified sampling settings.
  • Class-conditional samples: Figures 8–10 show 384×384 LlamaGen-3B samples conditioned on golden retriever, husky, and cliff drop-off labels.
  • Class-conditional samples: Figures 2–4 show 384×384 LlamaGen-3B samples conditioned on coral reef, space shuttle, and sport car labels.
Loading 2406.06525v1…