PaperlayerView exact revision on arXiv

Source-linked AI summary

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, Mohammad Norouzi

arXiv:2205.11487v1cs.CVcs.LG

TL;DR

Text-to-image synthesis seeks both photorealistic images and deep language understanding, but Imagen addresses this by combining large pretrained language models with diffusion models. It achieves state-of-the-art zero-shot COCO FID-30K of 7.27, while human raters prefer it over other methods in image fidelity and text alignment.

  • Problem

    Text-to-image synthesis requires models that combine photorealistic generation with deep language understanding.

  • Method

    Imagen combines frozen large pretrained language models as text encoders with high-fidelity diffusion models for text-to-image generation.

  • Results

    Imagen achieves a zero-shot COCO FID-30K of 7.27, while human raters prefer it over other methods in image fidelity and image-text alignment.

  • Takeaways & Limitations

    Scaling pretrained language encoders has greater impact than scaling the image diffusion model, supporting larger language models for text-to-image generation.

  • Takeaways & Limitations

    Imagen is not suitable for public use at this time because its web-scale training data includes inappropriate content and associated dataset risks.

Abstract

from arXiv · show

We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.

1 Introduction

Imagen combines large transformer language models with diffusion models for photorealistic, text-aligned image generation, achieving strong zero-shot COCO results. The paper also introduces dynamic thresholding and DrawBench to improve sampling and evaluate text-to-image systems more deeply.

  • Model and sampling: Imagen uses a frozen T5-XXL encoder, a 64×64 image diffusion model, and two super-resolution diffusion models to generate 256×256 and 1024×1024 images.All diffusion models condition on the text embedding sequence and use classifier-free guidance.
  • Model and sampling: Dynamic thresholding enables large guidance weights while avoiding prior sample-quality degradation, producing higher-fidelity and better-aligned images.The technique is introduced as a new diffusion sampling method for generating more photorealistic and detailed images.
  • Results: 7.27 zero-shot FID-30K on COCO outperforms GLIDE at 12.4, DALL-E 2 at 10.4, and Make-A-Scene at 7.6.Imagen’s score is reported without training on COCO, while Make-A-Scene is described as trained on COCO.
  • Evaluation: DrawBench is a structured prompt suite probing compositionality, cardinality, spatial relations, complex prompts, rare words, and creative generation.It enables multi-dimensional evaluation and deeper insights into text-to-image models.
  • Key contributions: Scaling the frozen text encoder improves sample quality significantly more than scaling the image diffusion model.The paper identifies large frozen language models trained only on text data as effective text encoders for text-to-image generation.

2 Imagen

Imagen combines pretrained text encoders with a cascade of conditional diffusion models that progressively generate higher-resolution images. Its design emphasizes T5-XXL text representations, classifier-free guidance with dynamic thresholding, and efficient super-resolution U-Nets.

  • Architecture: Imagen maps text to embeddings and uses cascaded conditional diffusion models to generate images at increasing resolutions.The cascade consists of a text encoder, a base diffusion model, and subsequent diffusion models for higher-resolution generation.
  • Text encoder: Imagen freezes pretrained BERT, T5, and CLIP text encoders, enabling offline embedding computation with negligible training-time computation or memory overhead.The authors report a clear preference for T5-XXL over CLIP on challenging prompts, while noting similar performance on simple MS-COCO benchmarks.
  • Text encoder: Human evaluators prefer T5-XXL over CLIP for image-text alignment and image fidelity on DrawBench’s challenging compositional prompts.The encoders perform similarly on simpler MS-COCO benchmarks.
  • Guidance and thresholding: Imagen depends critically on classifier-free guidance, which improves text conditioning by jointly training conditional and unconditional objectives with randomly dropped conditioning.Increasing the guidance weight strengthens guidance but can reduce diversity and produce unnatural images.
  • Guidance and thresholding: Dynamic thresholding improves photorealism and image-text alignment at very large guidance weights by rescaling extreme pixel values at each sampling step.It addresses train-test mismatch caused when guided x-predictions exceed the training range [−1, 1].
  • Cascaded generation: Imagen uses a 64×64 base model followed by text-conditional super-resolution models producing 256×256 and 1024×1024 images.Its Efficient U-Net super-resolution variant runs 2-3x faster in steps/second than the referenced U-Net.

3 Evaluating Text-to-Image Models

The section evaluates text-to-image models using COCO, FID, and CLIP, while supplementing their limitations with controlled human assessments of photorealism and image-text alignment. It also introduces DrawBench as a broader benchmark for comparing models through pairwise human judgments.

  • COCO evaluation: COCO validation is the standard benchmark, with FID measuring image fidelity and CLIP score measuring image-text alignment.The evaluation reports zero-shot FID-30K using 30K randomly sampled validation prompts.
  • Human evaluation: Because FID and CLIP have limitations, human evaluation measures image quality and caption similarity against ground-truth caption-image pairs.FID is not fully aligned with perceptual quality, while CLIP is ineffective at counting.
  • Human evaluation: Raters assess photorealism by choosing between model and reference images, and assess alignment by judging whether captions accurately describe images.Alignment responses are “yes”, “somewhat”, or “no”, and photorealism is summarized by the model-generation preference rate.
  • Human evaluation: Human evaluations use 200 randomly chosen COCO image-caption pairs, control trials, and data from raters achieving at least 80% control accuracy.The protocol produced 73 ratings per image for image quality and 51 for image-text alignment.
  • DrawBench: DrawBench is introduced as a comprehensive, challenging prompt set designed to reveal model differences beyond COCO.Its motivation includes evaluating visual reasoning skills and social biases beyond COCO.
  • DrawBench: DrawBench comparisons show raters eight samples from each of two models and choose which model is preferred, or whether they are indifferent, for fidelity and alignment.The three choices are Prefer Model A, Indifferent, and Prefer Model B.

4 Experiments

Imagen achieves state-of-the-art zero-shot COCO performance and is strongly preferred by human raters on DrawBench. Ablations show that text-encoder scaling, dynamic thresholding, noise conditioning augmentation, and sequence-level cross-attention are critical to performance.

  • MS-COCO: 7.27 zero-shot FID on COCO, surpassing DALL-E 2 and models trained on COCO.Imagen is evaluated on the COCO validation set using FID, with human evaluation also assessing image quality and text alignment.
  • DrawBench: Human raters exceedingly prefer Imagen over competing models on DrawBench for image fidelity and image-text alignment.Comparisons include DALL-E 2, GLIDE, Latent Diffusion, and CLIP-guided VQ-GAN, with preference rates aggregated across categories and raters.
  • Ablations: Scaling the text encoder consistently improves image-text alignment and image fidelity, with T5-XXL producing the best results.The largest text encoder, T5-XXL, has 4.6B parameters.
  • Ablations: Text-encoder scaling has significantly greater impact than U-Net scaling, although enlarging the U-Net also improves sample quality.This comparison is reported in the ablation analysis of model scaling.
  • Ablations: Dynamic thresholding substantially improves photorealism and text alignment over static or absent thresholding, especially with large classifier-free guidance weights.The effect is strongest under large guidance weights.
  • Ablations: Noise conditioning augmentation improves super-resolution CLIP and FID scores, while sequence-level cross-attention outperforms mean or attention pooling for fidelity and alignment.Noise conditioning also enables stronger text conditioning and improved scores at higher guidance weights.

5 Related Work

The section situates Imagen among autoregressive, GAN, VQ-VAE Transformer, and diffusion approaches, emphasizing its simpler design and use of large pretrained frozen language models. Compared with DALL-E 2 and GLIDE, Imagen avoids a latent prior while achieving better MS-COCO FID and DrawBench human-evaluation results.

  • Diffusion and text-to-image models: Diffusion models outperform GANs in image fidelity and diversity without training instability or mode collapse issues.The passage also notes substantial progress from autoregressive, GAN, VQ-VAE Transformer-based, and diffusion methods in text-to-image generation.
  • Comparison with related models: DALL-E 2 generates 1024 × 1024 images using a diffusion prior on CLIP text latents and cascaded diffusion models.Imagen is described as simpler because it does not need to learn a latent prior.
  • Comparison with related models: Imagen achieves better MS-COCO FID and human-evaluation results on DrawBench than DALL-E 2 while avoiding a learned latent prior.The comparison is presented as evidence that Imagen’s design is simpler than DALL-E 2’s.
  • Language encoders: Imagen uses large pretrained frozen language models, which are instrumental to both image fidelity and image-text alignment.GLIDE also uses cascaded diffusion models, whereas XMC-GAN uses BERT as a text encoder and Imagen scales to much larger text encoders.

6 Conclusions, Limitations and Societal Impact · A Background

The paper concludes that pretrained language encoders and diffusion-model design choices are central to Imagen’s performance, while documenting substantial limitations and societal risks. It also outlines diffusion as a forward-noising process learned through denoising and reversed for generation.

  • 6 Conclusions, Limitations and Societal Impact: Scaling the language model improves Imagen’s overall performance more than scaling its U-Net, motivating exploration of larger text encoders.Imagen also re-emphasizes classifier-free guidance as important for performance.
  • 6 Conclusions, Limitations and Societal Impact: Large, mostly uncurated web-scraped datasets enable progress but raise ethical concerns about appropriate public-data use and subject awareness.The paper recognizes that generative applications may affect society in complex ways while potentially augmenting human creativity.
  • 6 Conclusions, Limitations and Societal Impact: Imagen’s data filtering removed some undesirable content, but audits of LAION-400M found pornography, racist slurs, and harmful social stereotypes.The authors therefore assess Imagen as unsuitable for public use.
  • 6 Conclusions, Limitations and Societal Impact: Imagen is not released because preliminary assessments identify mode dropping, degraded fidelity for people, and social biases including lighter-skin-tone and stereotype tendencies.The paper leaves in-depth empirical analysis of social and cultural bias to future work.
  • 6 Conclusions, Limitations and Societal Impact: The authors identify text-to-image social-bias evaluation as a critical research gap and propose developing benchmark evaluations for social and cultural bias.They note that comparatively less auditing work exists for text-to-image models than for image-to-text and image-labeling systems.
  • A Background: Diffusion models define latents z = {z_t | t ∈ [0, 1]} through a Gaussian Markov forward process q(z|x), with noise increasing until q(z_1) ≈ N(0, I).The schedule uses α_t and σ_t, with log signal-to-noise ratio λ_t decreasing over time.
  • A Background: Generation is learned by denoising z_t sampled from q(z_t|x) into an estimate x̂_θ(z_t, λ_t, c), optionally conditioned on text embeddings or a low-resolution image.Training uses weighted squared error, and the paper uses ε-prediction with a cosine sampling schedule.
  • A Background: Sampling starts from z_1 ∼ N(0, I) and reverses diffusion using a discrete-time ancestral sampler or DDIM.DDIM is deterministic, whereas the ancestral sampler is stochastic with γ controlling sampler stochasticity.

B Architecture Details · B.1 Efficient U-Net · C DrawBench

Efficient U-Net reallocates capacity toward lower-resolution blocks and reverses sampling order to improve efficiency and speed without performance degradation. DrawBench is a fine-grained benchmark using approximately 200 prompts across 11 categories and structured human comparisons of image quality and text alignment.

  • B.1 Efficient U-Net: Efficient U-Net is simpler, converges faster, and is more memory efficient than some prior U-Net implementations.The paper directs readers to Appendix D.3.2 for results and Figures A.28-A.30 for architecture details.
  • B.1 Efficient U-Net: Efficient U-Net shifts model parameters from high-resolution to low-resolution blocks, increasing capacity where more channels reduce memory and computation costs.The design adds more residual blocks at lower resolutions.
  • B.1 Efficient U-Net: Reversing downsampling and upsampling order significantly improves the U-Net forward-pass speed without performance degradation.Downsampling is moved before convolutions, while upsampling is moved after convolutions.
  • C DrawBench: DrawBench contains approximately 200 text prompts organized into 11 categories for fine-grained text-to-image model analysis.Its size is intended to support thorough testing while keeping human-rating trials manageable.
  • C DrawBench: Each DrawBench comparison shows raters 8 random generations from each of two models and asks which set has higher quality and better represents the caption.The questions measure image fidelity and image-text alignment.
  • C DrawBench: 275 raters provide aggregated scores across the 11 DrawBench categories, with 25 raters assigned to each category.The study uses three response choices: preference for set A, indifference, or preference for set B, without post-filtering unreliable raters.

D Imagen Detailed Abalations and Analysis

This section presents ablation studies and a detailed analysis of Imagen.

  • The section examines Imagen through ablations and detailed analysis.

D.1 Pre-trained Text Encoders

Imagen compares BERT, T5, and CLIP as frozen text encoders and finds that larger T5 models improve image-text alignment and image fidelity. T5-XXL achieves the best CLIP scores and is preferred over CLIP on complex DrawBench prompts.

  • Encoder families and scaling: BERT, T5, and CLIP are evaluated as pre-trained text encoders, spanning different training corpora, objectives, and model sizes.BERT uses approximately 20 GB of text and models up to 340M parameters, while T5 uses approximately 800 GB and models up to 11B parameters.
  • Encoder families and scaling: Larger language-model text encoders generally yield better image-text alignment as measured by CLIP score over training.The comparison uses a 64 × 64, 300M-parameter diffusion model conditioned on embeddings from BERT, T5, or CLIP.
  • Encoder families and scaling: T5-XXL produces the best CLIP scores among the evaluated text encoders.This result is reported as a function of the number of training steps.
  • Encoder families and scaling: Larger T5 variants improve both image-text alignment and image fidelity across guidance-weight trade-offs.The finding emphasizes the effectiveness of large frozen text encoders for text-to-image models.
  • T5-XXL versus CLIP on DrawBench: Raters considerably more often prefer generations from the 1B-parameter T5-XXL model over the CLIP model, especially for image-text alignment.The evaluation uses 64×64 diffusion models and DrawBench, where 300M-parameter models significantly underperformed.

D.2 Classifier-free Guidance and the Alignment-Fidelity Trade-off … D.3.2 Comparison of U-Net vs Efficient U-Net

Imagen uses classifier-free guidance and dynamic thresholding to improve image-text alignment while managing the fidelity trade-off. Ablations show that larger text encoders, sequence-based conditioning, and Efficient U-Net architecture improve quality or efficiency.

  • D.2 Classifier-free Guidance and the Alignment-Fidelity Trade-off: Classifier-free guidance strengthens image-text alignment but creates a trade-off between alignment and image fidelity as guidance weight changes.Imagen uses relatively large guidance weights across all three diffusion models to achieve a good balance of sample fidelity and alignment.
  • D.2 Classifier-free Guidance and the Alignment-Fidelity Trade-off: Dynamic thresholding achieves significantly better CLIP scores and comparable or better FID scores than static thresholding across a wide range of guidance weights.Static thresholding can produce oversaturated samples at high guidance, whereas dynamic thresholding yields more natural-looking images.
  • D.2 Classifier-free Guidance and the Alignment-Fidelity Trade-off: Noise conditioning augmentation is critical for super-resolution quality, while no augmentation gives the best FID score across guidance weights during inference.Training without augmentation generally worsens both CLIP and FID scores and reduces their variation across guidance weights.
  • D.3 Impact of Model Size: Scaling the 64 × 64 U-Net from 300M to 2B parameters improves CLIP-FID trade-off curves, but scaling the frozen text encoder yields greater quality gains.Each U-Net model uses batch size 2048 and 400K training steps.
  • D.3.1 Impact of Text Conditioning Schemas: Cross-attention over contextual text embeddings outperforms mean or attention pooling for the base 64 × 64 text-to-image diffusion model.Explicit cross-attention in the 1024 × 1024 model improves fidelity and image-text alignment with minimal computational costs.
  • D.3.2 Comparison of U-Net vs Efficient U-Net: Efficient U-Net converges significantly faster than U-Net and achieves better overall performance on 64 × 64 →256 × 256 super-resolution.Efficient U-Net is also ×2 −3 faster at sampling.

E Comparison to GLIDE and DALL-E 2

On DrawBench, human raters preferred Imagen over DALL-E 2 for image-text alignment in 7 of 11 categories and for sample fidelity in all 11. Imagen particularly outperformed DALL-E 2 and GLIDE on prompts involving colors, positional relationships, and quoted text.

  • DALL-E 2 comparison: 7 of 11 categories favored Imagen over DALL-E 2 for image-text alignment, while all 11 favored Imagen for sample fidelity.These results come from human ratings on DrawBench.
  • DALL-E 2 comparison: Imagen’s largest advantages over DALL-E 2 occurred in the Colors, Positional, Text, DALL-E, and Descriptions categories.The comparison includes qualitative examples spanning DrawBench categories.
  • DALL-E 2 comparison: DALL-E 2 struggles to bind attributes such as colors to objects and to produce coherent text from prompts, limitations also identified relative to GLIDE.The passage attributes these observations to the authors of the DALL-E 2 study.
  • Text prompts: Imagen is significantly better than DALL-E 2 and GLIDE on DrawBench prompts containing quoted text.The cited qualitative comparisons specifically identify quoted-text prompts as a strong advantage for Imagen.
  • GLIDE comparison: GLIDE is better than DALL-E 2 at assigning colors to objects, while Imagen is reported as significantly better than GLIDE on quoted-text prompts.These observations are based on qualitative comparisons across DrawBench categories.

F Implementation Details

Imagen’s base model uses a scaled architecture with text conditioning at multiple resolutions, Adafactor optimization, and continuous-time cosine diffusion. Its super-resolution stages use Efficient U-Nets, with attention configurations differing between 64 × 64 →256 × 256 and 256 × 256 →1024 × 1024 models.

  • Base model: The base architecture uses text cross attention at resolutions [32] [16] plus attention-pooled text embeddings.Its specification includes embed_dim 512, channel_mult [1] [2] [4], and three residual blocks.
  • Base model: The base model is trained with Adafactor, a 1e-4 learning rate, and 10000 linear warmup steps.The optimizer uses default optax.adafactor parameters.
  • Base model: The base diffusion process uses a cosine noise schedule and continuous time steps t ∼U(0, 1).The configuration marks continuous_time as True.
  • 64 × 64 →256 × 256 super-resolution: The 64 × 64 →256 × 256 super-resolution model uses an Efficient U-Net with self-attention and text cross-attention.Its specification includes 1024 channels, strides (2, 2), eight residual blocks, and eight attention heads.
  • 256 × 256 →1024 × 1024 super-resolution: The 256 × 256 →1024 × 1024 super-resolution model matches the earlier super-resolution configuration but replaces self-attention with cross-attention to text embeddings.It uses a 1000 step linear noise schedule with start 1e-4 and end 0.02, while training uses continuous time steps t ∼U(0, 1).
Loading 2205.11487v1…