PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang

arXiv:2404.02905v2cs.CVcs.AI

TL;DR

Visual autoregressive image models have lagged behind diffusion models, with their scaling and generalization properties underexplored. VAR addresses this gap by replacing raster-scan next-token prediction with coarse-to-fine next-scale prediction, and reports superior image-generation quality, efficiency, data efficiency, scalability, and zero-shot task performance. The authors conclude that VAR initially verifies LLM-like scaling laws and zero-shot generalization in visual transformers.

  • Problem

    Visual autoregressive image models lag behind diffusion models, while their scaling laws and generalization properties remain underexplored.

  • Method

    VAR encodes images into multi-scale token maps and autoregressively predicts each higher-resolution map conditioned on preceding lower-resolution maps.

  • Results

    VAR surpasses strong diffusion models across image quality, diversity, data efficiency, inference speed, and scalability, while achieving FID 1.73 and IS 350.2 on ImageNet 256×256.

  • Takeaways & Limitations

    VAR initially verifies LLM-like power-law scaling and zero-shot generalization for visual transformers, including in-painting, out-painting, and editing.

  • Takeaways & Limitations

    The study keeps the baseline VQVAE tokenizer unchanged and identifies text-prompt generation as ongoing future work.

Abstract

from arXiv · show

We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine "next-scale prediction" or "next-resolution prediction", diverging from the standard raster-scan "next-token prediction". This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and generalize well: VAR, for the first time, makes GPT-like AR models surpass diffusion transformers in image generation. On ImageNet 256x256 benchmark, VAR significantly improve AR baseline by improving Frechet inception distance (FID) from 18.65 to 1.73, inception score (IS) from 80.4 to 350.2, with around 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot task generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.

1 Introduction

Existing visual autoregressive models flatten multi-scale image tokens into raster-scan sequences and lag behind diffusion models. VAR instead predicts progressively finer token maps, achieving strong benchmark performance while exhibiting scaling and generalization properties associated with LLMs.

  • Motivation: Visual AR models discretize images into 2D token grids, flatten them into 1D sequences, and significantly lag behind diffusion models.Their scaling laws remain underexplored.
  • Method: VAR orders images hierarchically by predicting progressively higher-resolution token maps from a 1×1 starting map.Each prediction is conditioned on all previously generated lower-resolution maps.
  • Results: On ImageNet 256×256, VAR reaches FID 1.73 and IS 350.2 with inference speed 20× faster than its AR baseline.The framework uses a GPT-2-like transformer architecture for visual autoregressive learning.
  • Results: VAR surpasses DiT in image quality, inference speed, data efficiency, and scalability, while demonstrating scaling laws and zero-shot generalization.Reported downstream tasks include image in-painting, out-painting, and editing.
  • Contribution: The work releases models and code covering VQ tokenization and autoregressive training pipelines.These resources are intended to support further visual autoregressive research.

2 Related Work

Related work spans scaling and zero-shot generalization in language models, raster-scan and masked visual prediction, and transformer-based diffusion models. VAR is positioned against these existing visual generation paradigms.

  • Large autoregressive language models: Language-model scaling laws relate model, dataset, or compute scale to test loss and help predict larger-model performance from smaller ones.They also describe continued performance growth with increased scale.
  • Large autoregressive language models: Zero-shot generalization is the ability to perform tasks without explicit task-specific training, a capability studied across language and vision foundation models.Examples include CLIP, SAM, DINOv2, Painter, and LVM.
  • Visual generation: Raster-scan visual autoregressive methods encode images as 1D token sequences and include pixel-level, super-resolution, and VQVAE-latent approaches.VQGAN uses a GPT-2 decoder-only transformer for latent autoregressive learning.
  • Visual generation: Masked-prediction models such as MaskGIT, MagViT, MagViT-2, and MUSE generate visual tokens using masked prediction transformers.These methods extend from images to videos and larger model scales.
  • Diffusion models: Diffusion-model research has advanced learning, sampling, guidance, latent learning, and transformer architectures, including DiT and U-ViT.These architectures underpin recent image and video synthesis systems.
  • Visual generation: A related work titled “language model beats diffusion” uses a BERT-style masked-prediction model.

3 Method

VAR replaces raster-scan next-token prediction with coarse-to-fine next-scale prediction over multi-scale token maps. This design addresses limitations of flattened image sequences by preserving spatial structure and generating each scale in parallel.

  • Vanilla autoregressive modeling: Conventional image autoregression tokenizes images into discrete grids, flattens them into one-dimensional sequences, and trains with next-token prediction.The approach introduces mathematical, generalization, structural, and efficiency problems.
  • Limitations of vanilla autoregression: Flattened visual tokens retain bidirectional feature correlations, conflicting with the unidirectional prefix dependency assumed by autoregressive models.The encoder’s feature vectors are inter-dependent before quantization and flattening.
  • Limitations of vanilla autoregression: Conventional self-attention generation incurs O(n2) autoregressive steps and O(n6) computational cost for an n×n image token sequence.Flattening also disrupts spatial locality among neighboring image tokens.
  • Visual autoregressive modeling: VAR quantizes images into K token maps at increasingly higher resolutions and predicts each next map conditioned on all preceding scales.The final map matches the original feature-map resolution.
  • Visual autoregressive modeling: During VAR training, a block-wise causal attention mask restricts each scale to its prefix, while all tokens within the predicted map are generated in parallel.The method uses standard cross-entropy loss in a two-stage pipeline with a multi-scale VQ autoencoder followed by a VAR transformer.
  • Tokenization: The multi-scale quantization autoencoder encodes images into shared-vocabulary token maps using a modified VQGAN-style architecture and extra convolution layers.The encoding and reconstruction procedures use residual designs described in Algorithms 1 and 2.

4 Implementation details

The implementation uses a multi-scale VQVAE tokenizer and a simple GPT-2-like decoder-only VAR transformer. Model dimensions are scaled with depth, while training settings vary by model size.

  • VAR tokenizer: The tokenizer uses a vanilla VQVAE architecture with multi-scale quantization, K extra convolutions, a shared codebook of V = 4096, and 16× spatial downsampling.It is trained on OpenImages with the compound loss.
  • VAR transformer: The VAR transformer adopts a GPT-2-like decoder-only architecture with adaptive normalization and scales width, head count, and drop rate with depth.The scaling rules are w = 64d, h = d, and dr = 0.1 · d/24.
  • VAR transformer: The main VAR transformer parameter count is formulated as a function of depth and model width, including an adaptive-layernorm contribution.The supplied implementation passage introduces this parameter-count formulation without providing its complete expression.
  • Training settings: Training uses a base learning rate of 10^-4 per 256 batch size, AdamW optimization, batch sizes from 768 to 1024, and 200–350 epochs depending on model size.The optimizer uses β1 = 0.9, β2 = 0.95, and decay = 0.05.

5 Empirical Results

VAR establishes a visual autoregressive model class that improves image-generation quality and speed while outperforming diffusion transformers, and exhibits power-law scaling with model size and compute.

  • Overall comparison: VAR establishes a new visual autoregressive model class that achieves the best FID/IS and remarkable generation speed on ImageNet benchmarks.The comparison covers 256×256 and 512×512 conditional synthesis.
  • Compared with popular diffusion transformer: VAR surpasses DiT in image quality, inference speed, data efficiency, and scalability.A 2B-parameter VAR consistently performs better in FID/IS, while DiT-XL/2 requires 45× the wall-clock time and 1400 training epochs versus VAR’s 350.
  • Scaling laws with model parameters N: VAR test loss follows L_last = (2.0 · N)^-0.23 and L_avg = (2.5 · N)^-0.20 as model parameters increase.The study varies model depth from 6 to 30, producing models from 18.5M to 2.0B parameters.
  • Scaling laws with model parameters N: Token error rate also follows power-law trends with model size, and scaling VAR transformers continuously improves test performance.The reported relations are Err_last = (4.9 · 10^2N)^-0.016 and Err_avg = (6.5 · 10^2N)^-0.010.
  • Scaling laws with optimal training compute C_min: Larger VAR transformers are more compute-efficient when trained with sufficient data, reaching the same performance with less computation across six orders of magnitude in C_min.The compute-scaling analysis traces loss and error across 12 models and uses Pareto-frontier estimates of optimal training compute.

6 Zero-shot task generalization

VAR generalizes to image in-painting, out-painting, and class-conditional editing without architectural modifications or parameter tuning.

  • Image in-painting and out-painting: VAR performs in-painting and out-painting by generating masked tokens while teacher-forcing ground-truth tokens outside the mask.No class-label information is injected for these tasks.
  • Generalization: Zero-shot downstream performance is achieved without special design or finetuning.The evaluated tasks include in-painting, out-painting, and class-conditional editing.
  • Class-conditional image editing: VAR produces plausible class-conditional edits that fuse with surrounding context.The model generates tokens inside a bounding box conditioned on a class label.

7 Ablation Study

The ablation study evaluates VAR against a vanilla autoregressive transformer and isolates enhancements that improve image quality and efficiency.

  • The study reports its effectiveness and efficiency results in Table 3.
  • FID improves from 18.65 to 5.22 when the vanilla AR methodology is replaced with VAR under otherwise unchanged settings.The comparison uses the vanilla AR transformer baseline and reports an inference wall-clock cost of 0.013× relative to that AR model.
  • Table 3 compares GPT-2-style transformers trained with AR or VAR and reports inference cost relative to the baseline alongside FID reduction.The table also identifies AdaLN, CFG, and attention normalization as VAR enhancements.
  • AdaLN, top-k sampling, CFG with ratio 2.0, and unit-vector normalization of q and k progressively improve VAR’s FID.The combined configuration reaches FID 3.30, a reduction of 15.35 from baseline.

8 Limitations and Future Work

The paper identifies unchanged tokenizer design, absent text-prompt generation, and unimplemented video generation as boundaries and directions for future work.

  • The study keeps the VQVAE architecture and training unchanged, while identifying advanced tokenizers and sampling techniques as orthogonal improvement directions.
  • Text-prompt generation remains ongoing and is prioritized for future integration with language models through encoder-decoder or in-context approaches.
  • Video generation is not implemented, but the paper proposes extending VAR to 3D next-scale prediction using multi-scale video features.

9 Conclusion

The conclusion presents VAR as a visual generative framework that improves autoregressive image generation and exhibits scaling-law behavior as it grows.

  • VAR is introduced as a framework that addresses issues in standard image autoregressive models and surpasses strong diffusion models across several image-generation dimensions.The cited conclusion names image quality, diversity, data efficiency, and inference speed.
  • Power-law relationships between test performance and model parameters or training compute emerge when VAR is scaled to 2 billion parameters.The reported Pearson coefficients approach −0.998.
  • Figure 9 visualizes normalized attention scores from the last self-attention layer of the VQGAN encoder for four ImageNet validation images.

A Token dependency in VQVAE

The token-dependency analysis examines self-attention in VQVAE before vector quantization using four sampled ImageNet validation images.

  • Attention heat maps show strong bidirectional dependency among VQVAE tokens in the examined self-attention layer.The analysis uses one attention map per image because the layer has a single head.

B Time complexity of AR and VAR generation

The appendix analyzes generation-time complexity, showing standard autoregressive generation scales as O(n^6) while VAR scales as O(n^4) under a geometric multi-scale resolution schedule.

  • Standard AR generation: O(n^6) is the time complexity of standard autoregressive generation for an n×n VQ code map.The derivation sums O(i^2) attention computation across n^2 sequential iterations.
  • VAR generation: VAR generates across a sequence of progressively larger token maps, with each step conditioned on the accumulated token maps.The complexity analysis considers the total tokens in the current maps at each autoregressive generation step.
  • VAR generation: O(n^4) is the time complexity of VAR generation when resolutions grow geometrically by factor a > 1 until reaching the final n×n map.VAR defines intermediate resolutions n_k = a^(k−1), with the final resolution equal to n.
Loading 2404.02905v2…