PaperlayerView exact revision on arXiv

Source-linked AI summary

Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, Yonglong Tian

arXiv:2410.13863v1cs.CVcs.LG

TL;DR

The paper asks why scaling autoregressive vision models is less consistently beneficial than scaling language models, examining token representations, generation orders, and evaluation behavior. It studies discrete versus continuous tokens and raster versus random-order autoregression, then scales Fluid, a random-order continuous-token model. Continuous tokens improve visual quality and scaling, random order improves GenEval alignment, and Fluid 10.5B reaches a zero-shot FID of 6.16 on MS-COCO 30K with a GenEval score of 0.69.

  • Problem

    Scaling autoregressive models in vision has not matched the established benefits of scaling language models, and the roles of token representation, generation order, and evaluation metric remain unclear.

  • Method

    The study compares discrete and continuous tokens with raster-order GPT-like and random-order BERT-like autoregressive models, then scales the random-order continuous-token model Fluid.

  • Results

    Continuous-token models show better visual quality and FID scaling, random-order models outperform raster-order models on GenEval, and Fluid 10.5B achieves a zero-shot FID of 6.16 on MS-COCO and a GenEval score of 0.69.

  • Takeaways & Limitations

    Scaling autoregressive text-to-image models depends on token representation and generation order, with random-order continuous-token models providing the strongest reported combination of scaling behavior and image-generation performance.

  • Takeaways & Limitations

    GenEval plateaus from Fluid 3.1B to 10.5B, although visual quality and FID continue improving; raster-order continuous-token models also occasionally produce unrecoverable gray-token failures.

Abstract

from arXiv · show

Scaling up autoregressive models in vision has not proven as beneficial as in large language models. In this work, we investigate this scaling problem in the context of text-to-image generation, focusing on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed raster order using BERT- or GPT-like transformer architectures. Our empirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends. Models based on continuous tokens achieve significantly better visual quality than those using discrete tokens. Furthermore, the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared to raster-order models. Inspired by these findings, we train Fluid, a random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zero-shot FID of 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage future efforts to further bridge the scaling gap between vision and language models.

1 INTRODUCTION

The paper examines why scaling autoregressive vision models has yielded weaker gains than scaling language models, focusing on token representation, generation order, and evaluation metrics. Its study finds that continuous tokens and random-order generation improve image quality and text alignment, culminating in Fluid’s strong benchmark results.

  • The study evaluates continuous versus discrete tokens and random versus raster generation orders to investigate vision-model scaling.The authors use Diffusion Loss to make autoregressive modeling compatible with continuous tokens and analyze four variants.
  • Continuous-token models improve FID more rapidly with scale and produce higher visual quality than VQ-based models.The authors attribute the quality gap likely to information loss from vector quantization.
  • Random-order models with bidirectional attention significantly outperform raster-order models with causal attention on GenEval, especially for multiple objects.Random-order prediction permits global structural readjustment at each step, unlike raster-order prediction.
  • Validation loss follows a power law with model size for both continuous and discrete tokens, but FID and GenEval need not follow a strict power law.Thus, scaling behavior depends on whether evaluation uses the training metric or a different metric.
  • Fluid 10.5B achieves a zero-shot FID of 6.16 on MS-COCO and a GenEval overall score of 0.69.The model uses random-order autoregression with continuous tokens and compares favorably with leading text-to-image models.

2 RELATED WORK

Related work contrasts diffusion models’ strong but computationally expensive text-to-image generation with autoregressive models’ weaker performance and discrete-token limitations. Prior work introduced continuous-token masked autoregression, but its text-to-image scaling behavior remained unexplored.

  • Diffusion models generate highly realistic text-to-image samples but usually require multiple computationally expensive forward passes.
  • Autoregressive text-to-image models lag behind diffusion models, and discrete tokenizers often limit their modeling capability.Parti scaled to 20B parameters yet reported a slightly lower or better MS-COCO FID than the passage’s comparison context.
  • Continuous-token masked autoregression achieved strong class-conditional ImageNet results, but scaling it for text-to-image generation was unexplored.
  • Language-model scaling studies established power-law relationships with model size, dataset size, and training compute, while compute-optimal training balanced model and data scaling.
  • Vision-model scaling has been less consistent for recognition, although generative models such as DiT show more promising quality improvements with scale.

3 PRELIMINARY: AUTOREGRESSIVE IMAGE GENERATION

Autoregressive image generation models predict tokens sequentially under a chosen representation and order. The paper contrasts discrete and continuous tokens with raster-order GPT-style generation and random-order BERT-style generation, each trading inference efficiency against global editing.

  • Autoregressive models factorize token generation into conditional next-token predictions and generate tokens iteratively.
  • The two central design choices are the token representation—discrete or continuous—and the generation order.
  • Discrete vs. continuous tokens: Discrete tokenization uses a finite vocabulary and categorical cross-entropy modeling, whereas continuous tokenization avoids vector quantization and improves reconstruction quality.A diffusion model can approximate each continuous token’s distribution without discretization.
  • Raster Order + GPT vs. Random Order + BERT: Raster order generates tokens left-to-right and top-to-bottom with GPT-like causal attention, while random order can generate multiple tokens per step with BERT-like bidirectional attention.
  • Raster Order + GPT vs. Random Order + BERT: Raster-order models support fast key-value-cached inference, whereas random-order models enable global editing but cannot use key-value caching.

4 IMPLEMENTATION

The training framework tokenizes images, masks tokens, encodes text, and trains a transformer to reconstruct masked visual tokens conditioned on text. It supports discrete or continuous image representations and uses distinct output heads for their distributions.

  • The framework masks image tokens and trains a transformer to reconstruct them conditioned on text.The image tokenizer, text encoder and aligner, transformer, and output head form the main pipeline.
  • Image Tokenizer: The continuous tokenizer produces 256 grouped tokens with 16 channels and achieves notably higher reconstruction quality than the discrete tokenizer.The discrete tokenizer uses 16×16 tokens from an 8192-entry vocabulary, while continuous tokens originate from grouped 2×2 patches.
  • Text Encoder: Text is encoded with a frozen 4.7B-parameter T5-XXL encoder and aligned by six trainable transformer blocks.
  • Transformer: A decoder-only transformer combines visual self-attention and text-conditioned cross-attention for autoregressive generation.Raster-order models use causal self-attention to predict the next token.
  • Output head: Discrete tokens use a softmax categorical head, whereas continuous tokens use a six-layer MLP diffusion head with a cosine noise schedule.Training uses 1000 diffusion steps and inference resamples the schedule to 100 steps.

5 EXPERIMENTS

The experiments show that token representation and generation order determine how autoregressive text-to-image models scale in evaluation, despite consistently improving validation loss. Fluid, combining continuous tokens with random-order generation, delivers the strongest scaling behavior and benchmark performance.

  • 5.1 SCALING BEHAVIORS: Validation loss scales as a power law with model size from 150M to 3B across continuous- and discrete-token variants.The reported validation-loss relationship is measured in log space and reflects generalization to held-out data under the training metric.
  • 5.1 SCALING BEHAVIORS: Random-order continuous-token models improve consistently through 3B parameters, whereas raster-order discrete-token models plateau around 1B.Validation-loss improvements do not translate linearly to evaluation metrics, and Fluid achieves the best overall FID and GenEval scaling among the four variants.
  • 5.1 SCALING BEHAVIORS: Fluid’s validation loss, FID, and GenEval improve with training steps and compute, but additional training saturates around 1M steps.The results indicate that training larger models for fewer steps is more compute-efficient than training smaller models longer in this setting.
  • 5.1 SCALING BEHAVIORS: Validation loss correlates strongly with FID and GenEval across 150M–3B Fluid models, with Pearson coefficients of 0.917 and -0.931.The correlation weakens slightly for the 3.1B model, and the authors note that bounded metrics prevent the relationship from holding indefinitely.
  • 5.1 SCALING BEHAVIORS: Continuous-token models produce substantially better visual quality than discrete-token models, whose tokenizer information loss persists even when scaling to 3B parameters.Larger models also improve visual quality and image-text alignment; a 3B random-order model succeeds on a prompt that a 0.2B model struggles with.
  • 5.2 BENCHMARKING WITH PREVIOUS SYSTEMS: 6.16 zero-shot FID and 0.692 GenEval overall score are achieved by Fluid at 10.5B parameters, improving on its 369M model’s 7.23 FID and 0.62 GenEval score.The 10.5B model is evaluated on MS-COCO and GenEval; its reported speed is 1.571 seconds per image per TPU.

6 DISCUSSION

The paper concludes that autoregressive image generation scales most effectively when it uses continuous tokens and random-order generation. Scaling Fluid to 10.5B parameters achieves state-of-the-art text-to-image performance, although GenEval plateaus relative to the 3.1B model.

  • 6 DISCUSSION: Fluid combines random-order generation with continuous tokens and achieves state-of-the-art text-to-image generation performance at 10.5B parameters.The paper presents these design choices as the strongest-performing and best-scaling configuration across evaluation metrics and visual quality.
  • 6 DISCUSSION: The findings are intended to provide insight into autoregressive image-generation scaling and help narrow the gap with language-model scaling behavior.This is stated as the paper’s broader intended implication rather than as a demonstrated result beyond the reported experiments.
  • 6 DISCUSSION: GenEval plateaus for 10.5B Fluid relative to 3.1B Fluid, while visual quality and FID continue improving.This identifies a benchmark-specific scaling boundary rather than a general halt in model improvement.

3.1B 10.5B 3.1B 10.5B

Fluid’s larger model improves text rendering, fine detail, and text-image alignment, while the study also documents distinct failure modes and implementation choices for generation and guidance.

  • 3.1B 10.5B 3.1B 10.5B: Adding layers to the trainable text aligner consistently improves FID, leading the authors to select six layers for balancing performance and efficiency.The pilot study used a 277M-parameter random-order continuous-token model with a T5-XL text encoder.
  • 3.1B 10.5B 3.1B 10.5B: The 10.5B Fluid model generates text, finer image details, and text-aligned images better than the 3.1B model.The comparison covers additional qualitative results between the two Fluid model sizes.
  • 3.1B 10.5B 3.1B 10.5B: Raster-order continuous-token generation can produce persistent gray tokens that leave generated images incomplete.The authors suspect learned raster positional embeddings struggle with the discontinuity between adjacent lines and suggest 2D positional embeddings as a possible remedy.
  • 3.1B 10.5B 3.1B 10.5B: Random-order continuous-token generation rarely produces abnormal bright spots, which can be addressed by increasing diffusion steps from 100 to 200.The reported issue is presented as an uncommon failure case for Fluid.
  • 3.1B 10.5B 3.1B 10.5B: Continuous-token sampling uses conditional and unconditional vectors with classifier-free guidance, while guidance scale and temperature are tuned across model variants.The selected guidance parameters were found to remain consistent between 160M and 360M models and were applied up to 3B parameters.
Loading 2410.13863v1…