PaperlayerView exact revision on arXiv

Source-linked AI summary

DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation

Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, Shuangfei Zhai

arXiv:2410.08159v2cs.CVcs.LG

TL;DR

Markovian diffusion sees only the previous denoising step, limiting use of the full generation trajectory. DART combines autoregressive modeling with non-Markovian diffusion and adds token-level autoregression and flow-based refinement. It achieves competitive performance on class-conditioned and text-to-image tasks, though memory and token-count constraints remain.

  • Problem

    Markovian diffusion conditions denoising on the previous step, limiting use of the full generation trajectory and contributing to training and inference inefficiencies.

    Source: §1 · p. 2

  • Method

    DART combines autoregressive modeling with non-Markovian diffusion, adding token-level autoregression and flow-based refinement to its generation process.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    DART achieves competitive performance on class-conditioned and text-to-image tasks; DART-AR achieves FID 3.98 and surpasses DiT at 16 sampling steps.

    Source: §1 · p. 2, §4.2 · p. 10

  • Takeaways & Limitations

    DART offers a unified, flexible framework with competitive performance across class-conditioned and text-to-image generation.

    Source: §1 · p. 2, §5 · p. 12

    Non-Markovian modeling is memory-constrained as T increases; at T = 16, 256 × 256 images create over 4,000 latent tokens, limiting capacity on complex tasks such as text-to-image generation.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Diffusion models have become the dominant approach for visual generation. They are trained by denoising a Markovian process which gradually adds noise to the input. We argue that the Markovian property limits the model's ability to fully utilize the generation trajectory, leading to inefficiencies during training and inference. In this paper, we propose DART, a transformer-based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework. DART iteratively denoises image patches spatially and spectrally using an AR model that has the same architecture as standard language models. DART does not rely on image quantization, which enables more effective image modeling while maintaining flexibility. Furthermore, DART seamlessly trains with both text and image data in a unified model. Our approach demonstrates competitive performance on class-conditioned and text-to-image generation tasks, offering a scalable, efficient alternative to traditional diffusion models. Through this unified framework, DART sets a new benchmark for scalable, high-quality image synthesis.

1 INTRODUCTION

The introduction argues that Markovian diffusion underuses generation history, then presents DART as a non-Markovian autoregressive alternative with two extensions and competitive performance across class-conditioned and text-to-image generation.

1 INTRODUCTION

  • Markovian diffusion conditions each denoising step on the previous step, limiting access to earlier trajectory context and contributing to training and inference inefficiencies.The introduction notes that high-resolution generation also often requires cascaded or multiscale models, or preprocessing into lower-resolution autoencoder codes.Source: §1 · pp. 1–2
  • DART integrates autoregressive modeling with non-Markovian diffusion to use the full generation trajectory during training and inference.The paper describes this as a more efficient and flexible alternative to traditional diffusion and autoregressive approaches.Source: §1 · p. 2
  • DART-AR models image-token dependencies autoregressively, while DART-FM refines the process to improve expressiveness and coherence.DART-AR is described as enabling finer control and improved generation quality; DART-FM smooths transitions between denoising steps.Source: §1 · p. 2
  • DART achieves competitive performance on class-conditioned and text-to-image generation tasks.The introduction presents it as a scalable, unified approach to high-quality, controllable image synthesis.Source: §1 · p. 2

2 BACKGROUND

The background contrasts iterative denoising in diffusion models with autoregressive modeling of discrete image tokens. It outlines each approach’s process and practical tradeoffs.

2.1 DIFFUSION MODELS

  • Diffusion models progressively add noise through a Markovian process and train a backward denoiser; standard approaches require many denoising steps and extensive training data.The denoiser maps a noisy sample to its clean version, with a time-dependent loss weight.Source: §2.1 · p. 3

2.2 AUTOREGRESSIVE MODELS

  • Autoregressive image models encode images as discrete tokens, train with cross-entropy, then sample tokens and decode them into image space.The token sequence is designed to approximately reconstruct the original image.Source: §2.2 · p. 3
  • Autoregressive models can cache prior computations to reduce overhead, but quantization may lose information and stepwise prediction may miss global structure.The passage also identifies difficulty capturing long-range dependencies and holistic coherence in complex scenes or sequences.Source: §2.2 · p. 3

3 DART

DART replaces the standard Markovian transition with a non-Markovian formulation that can use the full generation trajectory. It implements this approach with an autoregressive Transformer and adds methods to improve modeling capacity, while longer trajectories remain memory-constrained.

3.1 NON-MARKOVIAN DIFFUSION FORMULATION

  • A non-Markovian noise schedule can match a comparable diffusion process while letting the model use the full generation trajectory.The formulation reconsiders the generator pθ(x_t−1|x_t:T), rather than restricting it to the previous noisy sample.Source: §3.1 · p. 4

3.2 PROPOSED METHODS

  • DART uses an independent noising process and an autoregressive Transformer to predict the next noisy image from the concatenated trajectory.Images are represented as pretrained-VAE latent maps that are patchified and flattened into continuous tokens.Source: §3.1 · p. 4, §3.2 · p. 5
  • Training shares computations across denoising steps in parallel, using a chunk-based causal mask to preserve autoregressive structure.The objective is similar to the original diffusion objective, and the formulation permits established language-model design principles.Source: §3.2 · p. 5
  • At sampling, DART predicts a mean, adds Gaussian noise for the next step, and uses KV-cache for decoding efficiency without complex solvers.Classifier-free guidance is applied to the prediction for improved visual quality.Source: §3.2 · p. 5
  • Non-Markovian modeling faces increasing memory demands as the number of steps grows, constraining capacity on complex tasks such as text-to-image generation.At T = 16 on 256 × 256 images, the latent sequence can exceed 4,000 tokens.Source: §3.2 · p. 5

1. DART with Token Autoregressive (DART-AR)

DART-AR adds token-wise autoregression to DART’s denoising process. This models dependencies among tokens, but makes sampling more expensive.

1. DART with Token Autoregressive (DART-AR)

  • The independent Gaussian assumption can poorly approximate the true conditional distribution, especially when T is small.DART-AR models the denoised image autoregressively across its flattened tokens, so those tokens are not treated as independent.Source: §1 · pp. 5–6
  • DART-AR replaces block-wise autoregression with token-wise autoregression for finer-grained denoising generation.The token factorization ensures the predicted tokens are not independent.Source: §1 · p. 6
  • DART-AR training takes essentially the same computation as DART, but sampling requires K × T autoregressive steps.The additional training changes are at the input and attention masks.Source: §1 · p. 6

2. DART with Flow Matching (DART-FM)

DART-FM adds a flow-matching refinement to DART’s Gaussian-based predictions. The framework also extends to multi-resolution generation and shared continuous-image and discrete-token modeling.

2. DART with Flow Matching (DART-FM)

  • DART-FM abandons the Gaussian assumption for pθ(x_{t−1}|x_{t:T}) by recursively applying a flow network to bridge sampled x̃_t and x_{t−1}.A small MLP models the velocity field using last-block Transformer context; flow matching trains it, and its objective refines the Gaussian-based prediction.Source: §2 · p. 6

3.3 MULTI-RESOLUTION GENERATION

  • Matryoshka-DART represents each resolution with a separate noisy sequence, then flattens and concatenates the sequences for sequential prediction.The framework supports joint modeling of multiple resolutions.Source: §3.3 · p. 6
  • Not conditioning x_i^0 directly on earlier x^0 avoids boundary shape changes and mitigates error propagation; self-attention processes low-resolution information.Source: §3.3 · pp. 6–7
  • The model balances noise-level count against token count for efficiency and supports progressively higher resolutions by finetuning low-resolution models with extended sequences.Source: §3.3 · p. 7

3.4 MULTI-MODAL GENERATION

  • Kaleido-DART jointly trains continuous denoising and discrete next-token prediction in a shared architecture, reweighting discrete loss by relative token lengths.Source: §3.4 · p. 7
  • Unlike cited unification efforts, DART changes diffusion’s Markovian process to exploit generation history and permits integration into LLM pipelines.Source: §3.4 · p. 7

4 EXPERIMENTS

Experiments evaluate DART on ImageNet class-conditioned generation and CC12M text-to-image generation, using FID and, for text-to-image, CLIP score. DART and its variants show competitive results across tasks, with reported trade-offs in quality, scale, and inference efficiency.

4.1 EXPERIMENTAL SETTINGS

  • Experiments use ImageNet class labels and CC12M image-caption pairs; most models train at 256 × 256 resolution, while Matryoshka-DART uses 512 × 512.FID measures realism and diversity; text-to-image evaluation also uses CLIP score for text alignment and MSCOCO 2017 for zero-shot evaluation.Source: §4.1 · p. 7

4.2 RESULTS

  • At 16 sampling steps, DART-AR reaches an FID of 3.98 and significantly surpasses DiT, while DiT performs better with 250 steps.The paper notes that official DiT was trained for 7M iterations, substantially more than DART.Source: §4.2 · p. 10, Figure 8 · p. 11
  • DART achieves competitive ImageNet-256 performance against diffusion and autoregressive models, though the paper notes a gap from state-of-the-art models that use more training FLOPs or larger models.The comparison includes reproduced DiT, SiT, and MAR results at 16 sampling steps; DART performs better than those reproduced baselines at that step count.Source: §4.2 · p. 10, Table 1 · p. 10
  • DART-FM achieves the best text-to-image FID of 11.12, and both DART-AR and DART-FM improve on default DART.The evaluation uses text-to-image generation, including an in-house text-conditioned DiT comparison at 16 and 250 steps.Source: §4.2 · pp. 10–11
  • DART has comparable inference FLOPs and wall-clock time to DiT at 16 sampling steps, and outperforms DiT at that setting on ImageNet and COCO.DART-AR has higher wall-clock inference time because its autoregressive steps are not well parallelized in the current implementation; DART-FM incurs flow-network overhead.Source: §4.2 · p. 11, Figure 9 · p. 11
  • Across small, base, large, and extra-large CC12M models, CLIP score improves as parameter count increases, and performance also rises with more training iterations.The paper presents this as evidence that the generative paradigm benefits from scaling.Source: §4.2 · p. 11, Figure 9 · p. 11
  • Reducing the denoising schedule from 16 to 4 or 8 noise levels degrades performance, though DART still generates plausible samples with 4 levels.The authors identify varying noise levels as a potential computation–performance trade-off when compute is limited.Source: §4.2 · pp. 11–12, Figure 9 · p. 11
  • Kaleido-DART jointly generates descriptive text and realistic images through one model, using a unified mechanism for both modalities.The paper describes this as preliminary multimodal generation and suggests potential integration with multimodal language models.Source: §4.2 · p. 12, Figure 10 · p. 12

5 CONCLUSION

The paper presents DART as an autoregressive denoising model within non-Markovian diffusion, reporting competitive performance across class-conditioned and text-to-image generation. It also identifies token count in the denoising process as a current restriction.

5 CONCLUSION

  • DART combines autoregressive denoising with non-Markovian diffusion and is presented as improving image-generation efficiency and scalability.The conclusion highlights token-level autoregression and flow matching as components of the approach.Source: §5 · p. 12
  • The paper reports competitive performance for DART on both class-conditioned and text-to-image tasks.It characterizes the approach as a unified and flexible solution for high-quality visual synthesis.Source: §5 · p. 12
  • The current model is restricted by the number of tokens in its denoising process.The authors point to more efficient long-context architectures and broader multimodal modeling as future directions.Source: §5 · p. 12

A PROOF OF PROPOSITION 1

The proof represents the noisy sample using the clean image and a noise term, then relates the resulting sequences. It also examines conditions under which the signal-to-noise ratio is maximal and the forward process is Markovian.

A PROOF OF PROPOSITION 1

  • The proof writes x_t as a scaled clean image plus scaled noise, using the stated Gaussian distribution q(x_t|x_0).The relation is x_t = √γ_t x_0 + √(1 − γ_t)ϵ_t.Source: A PROOF OF PROPOSITION 1 · p. 17
  • The proof studies maximal signal-to-noise ratio for y_t and shows a Markov property when each y_t achieves that maximum.The Markov-property argument invokes an independent noise term added to y_t to obtain y_{t+1}.Source: A PROOF OF PROPOSITION 1 · p. 17
  • The proof states that the x_t and y_t sequences have a one-to-one correspondence.It says the x_t sequence can be uniquely determined from the y_t sequence.Source: A PROOF OF PROPOSITION 1 · p. 17

B.1 ARCHITECTURE

DART uses an encoder-decoder Transformer, with different conditioning paths for text-to-image and class-conditioned generation. The appendix also describes training and parameterization choices, along with distinctions among DART variants and their extensions.

B.1 ARCHITECTURE

  • The standard DART model uses an encoder-decoder Transformer with about 800M parameters; text-to-image uses a frozen pretrained Flan-T5-XL encoder, while class-conditioned generation uses an embedding layer and Adaptive LayerNorm.The architecture configuration specifies 28 layers and hidden sizes of 1280 for text-to-image or 1152 for class-conditioned generation.Source: §B.1 · p. 18
  • Matryoshka-DART combines 256 × 256 and 512 × 512 image tokens in one process, spatially aligning high-resolution rotary positions with low-resolution ones.The described setup contains 8192 tokens, split between 4096 tokens at each resolution.Source: §B.1 · p. 18

B.2 TRAINING

  • Across all experiments, DART shares a training configuration with learning rate 3e-4; its implementation uses v-prediction, maps predictions to clean-image estimates, and computes loss between x and the estimate.The configuration specifies beta2=0.95, epsilon=1e-8, 10,000 warmup steps, and weight decay 0.01.Source: §B.2 · pp. 18–19, §B.3 · p. 19

C COMPARISON OF DART VARIANTS

  • DART denoises block-wise autoregressively, DART-AR uses finer-grained token-wise autoregression, and DART-FM adds flow-network refinement while retaining block-wise autoregression.The authors report that both enhanced variants improve performance over vanilla DART and that DART-FM generally offers a better quality–inference-efficiency trade-off.Source: C COMPARISON OF DART VARIANTS · p. 19, Table 3 · p. 19
  • Kaleido-DART combines next-token prediction for text with next-denoising prediction for images, while Matryoshka-DART concatenates high-resolution tokens after low-resolution ones.Both extensions can use DART, DART-AR, or DART-FM variants.Source: C COMPARISON OF DART VARIANTS · p. 19

D.1 STEP-WISE GENERATIVE RESULTS

Matryoshka-DART first refines a 256 × 256 image, then upsamples it to 512 × 512 with fewer denoising steps. Figure 11 shows latent maps alongside decoded RGB images at both resolutions.

D.1 STEP-WISE GENERATIVE RESULTS

  • DART refines images at 256 × 256 for 16 denoising steps before upsampling them to 512 × 512.The higher-resolution generation uses four denoising steps and is conditioned on the previous low-resolution samples.Source: §D.1 · p. 19
  • Conditioning 512 × 512 generation on the 256 × 256 samples lets it generate realistic images in four denoising steps.The passage describes the higher-resolution generation as iterative denoising.Source: §D.1 · p. 19
  • Figure 11 presents latent maps on the left and decoded RGB images on the right for 256 × 256 and 512 × 512 generation.The caption identifies T = 16 for 256 × 256 and T = 4 for its 512 × 512 upsampling.Source: §D.1 · p. 19, Figure 11 · p. 20

D.2 ADDITIONAL SAMPLES

Additional examples compare DART variants on ImageNet and show text-to-image generations at resolutions up to 1024 pixels. On ImageNet, DART-AR and DART-FM examples have higher fidelity and more details than original DART.

D.2 ADDITIONAL SAMPLES

  • DART-AR and DART-FM generate ImageNet samples with higher fidelity and more details than original DART.The passage describes additional examples from models trained on ImageNet at 256 × 256.Source: §D.2 · p. 20, Figure 12 · p. 21
  • Text-to-image examples show DART-AR and DART-FM at 256 × 256 pixels.Figure 13 contains uncurated samples given various captions.Source: §D.2 · p. 20, Figure 13 · p. 22
  • Matryoshka-DART fine-tuned from DART-FM at 256 × 256 is shown generating text-to-image samples at 512 × 512 and 1024 pixels.Figures 14 and 15 show uncurated examples given various captions.Source: §D.2 · p. 20, Figure 14 · p. 23, Figure 15 · p. 24
Loading 2410.08159v2…