PaperlayerView exact revision on arXiv

Source-linked AI summary

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen

arXiv:2112.10741v3cs.CVcs.GRcs.LG

TL;DR

Text-to-image models had not yet achieved reliable photorealism or natural-language editing. GLIDE applies guided diffusion with classifier-free guidance, which human evaluators preferred over DALL-E for photorealism and caption similarity, while also supporting text-driven inpainting.

  • Problem

    Existing text-conditional image models could compose objects plausibly but had not yet generated photorealistic images capturing all prompt aspects.

  • Method

    GLIDE trains a 3.5-billion-parameter text-conditional diffusion model and compares CLIP guidance with classifier-free guidance using human and automated evaluations.

  • Results

    Human judges preferred classifier-free-guided GLIDE samples over DALL-E 87% of the time for photorealism and 69% for caption similarity.

  • Takeaways & Limitations

    Fine-tuned GLIDE can realistically edit existing images with text, including inserting objects, shadows, reflections, and style-matched elements.

  • Takeaways & Limitations

    The unoptimized model takes 15 seconds to sample one image on a single A100 GPU, limiting suitability for real-time applications.

Abstract

from arXiv · show

Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored by human evaluators to those from DALL-E, even when the latter uses expensive CLIP reranking. Additionally, we find that our models can be fine-tuned to perform image inpainting, enabling powerful text-driven image editing. We train a smaller model on a filtered dataset and release the code and weights at https://github.com/openai/glide-text2im.

1. Introduction

GLIDE applies guided diffusion to text-conditional image synthesis, comparing CLIP guidance with classifier-free guidance in a 3.5-billion-parameter text-conditioned model. The system also supports natural-language image inpainting for iterative editing, while its release addresses misuse risks through smaller models trained on filtered datasets.

  • Motivation: Natural-language image generation and editing could make rich visual content easier to create while enabling iterative refinement and fine-grained control.Creating illustrations, paintings, and photographs can require specialized skills and hours of labor.
  • Problem: Existing text-conditional models compose objects plausibly but do not yet generate photorealistic images capturing all aspects of their prompts.Unconditional models can generate photorealistic images, motivating diffusion-based approaches for text conditioning.
  • Approach: GLIDE trains a 3.5 billion parameter diffusion model conditioned on natural-language descriptions and compares CLIP guidance with classifier-free guidance.The model uses a text encoder, and the comparison evaluates two ways to guide diffusion toward text prompts.
  • Results: 87% of human judgments favored GLIDE over DALL-E for photorealism, while 69% favored it for caption similarity.The result concerns samples generated with classifier-free guidance, which were also described as reflecting broad world knowledge.
  • Editing and safety: Fine-tuned image inpainting lets users improve samples for complex prompts, producing edits that match surrounding style and lighting with convincing shadows and reflections.The model can struggle with complex prompts in zero-shot generation, so editing provides an iterative alternative.
  • Editing and safety: Because GLIDE can reduce the effort needed to produce disinformation or Deepfakes, the authors release smaller diffusion and noised CLIP models trained on filtered datasets.The system is named GLIDE, and the small filtered model is called GLIDE (filtered).

2. Background

This section introduces Gaussian diffusion as iterative denoising from Gaussian noise and describes classifier-free and CLIP guidance for steering generation toward conditioning information.

  • Diffusion: Diffusion models generate samples by learning reverse transitions that progressively remove noise from an initially Gaussian-noised latent.The forward process adds Gaussian noise until xT is approximated by N(0, I), while the learned reverse process produces x0 step by step.
  • Classifier-free guidance: Classifier-free guidance replaces conditioning labels with a null label during training and extrapolates toward the conditioned prediction during sampling.For generic text prompts, captions are sometimes replaced with an empty sequence during training, then guidance uses the caption-conditioned prediction relative to the null-conditioned prediction.
  • Classifier-free guidance: Classifier-free guidance uses one model’s knowledge and avoids requiring a separate classifier, simplifying guidance for difficult-to-classify conditions such as text.These are identified as two appealing properties of the technique.
  • CLIP guidance: CLIP guidance steers diffusion toward text captions by perturbing the reverse-process mean using the gradient of image-caption encoding similarity.CLIP consists of separate image and caption encoders trained to produce high similarity for paired images and captions.
  • CLIP guidance: Noise-aware CLIP models are trained on noised images for reverse-process guidance, and the paper reports favorable performance versus public CLIP guidance without extra tricks.The cited comparison is attributed to Appendix D, while the authors hypothesize that public CLIP is harmed by out-of-distribution noisy intermediates.

3. Related Work

Prior work has explored text-conditional image generation with GANs and autoregressive models, while other studies apply diffusion models to inpainting and text-based editing. CLIP has also been used to guide GANs and diffusion-based generation, but the cited works do not explore noised CLIP models.

  • Text-conditional image generation: Text-conditional image generation has been studied with caption-conditioned GANs and autoregressive models built on discrete latent codes.The cited GAN studies use publicly available image-captioning datasets, while Ramesh et al. build on discrete latent codes.
  • Diffusion inpainting: Diffusion-based inpainting can follow rough sketches or color sets and smoothly fill regions without edge artifacts when trained directly on inpainting.These findings come from Meng et al. (2021) and Saharia et al. (2021a), respectively.
  • CLIP guidance and editing: CLIP has guided GAN generation toward text prompts and has also been used in unnoised diffusion and text-conditioned image editing.Related methods include CLIP-guided GAN generation, unnoised CLIP-guided diffusion, and editing by targeting a CLIP loss while reconstructing an image’s DDIM latent.
  • CLIP guidance and editing: The cited CLIP-guided methods do not explore noised CLIP models and often rely on data augmentations and perceptual losses.Other related work conditions GANs on perturbed CLIP image embeddings or uses CLIP and GANs to inpaint images toward text targets.

4. Training

GLIDE’s training uses text-conditioned diffusion models at 64 × 64 and a learned upsampler to reach 256 × 256, with total compute roughly matching DALL-E. The authors additionally fine-tune the base model for unconditional generation and explicitly train it for text-guided inpainting.

  • Base and upsampling models: GLIDE trains a 3.5 billion parameter text-conditional diffusion model at 64 × 64 and a 1.5 billion parameter upsampler to reach 256 × 256.The upsampler increases image resolution from 64 × 64 to 256 × 256.
  • Architecture: The models adopt ADM with text conditioning through token encoding and a Transformer whose output supplies conditioning information.The text is encoded as a sequence of K tokens, and the Transformer’s final token embedding replaces the ADM class embedding.
  • Optimization: The base model trains for 2.5M iterations at batch size 2048, while the upsampler trains for 1.6M iterations at batch size 512, using stable 16-bit training.Total training compute is roughly equal to that used to train DALL-E.
  • Unconditional generation: After initial training, GLIDE fine-tunes the base model with 20% empty text sequences so it supports unconditional as well as text-conditional image generation.The fine-tuning procedure otherwise matches pre-training.
  • Inpainting: GLIDE explicitly fine-tunes for inpainting by erasing random image regions and conditioning on the remaining image content plus a mask channel.The architecture adds four input channels: a second set of RGB channels and a mask channel, with corresponding input weights initialized for the new channels.

5. Results

GLIDE’s classifier-free guidance generally produces more realistic, prompt-consistent images than CLIP guidance, while supporting broad prompt composition and text-driven image editing. Human evaluations also favor GLIDE over prior models, including DALL-E, despite its smaller size and lack of CLIP reranking.

  • Guidance comparison: Classifier-free guidance often produces more realistic samples than CLIP guidance, so the remaining results use classifier-free guidance.This comparison is made visually in Figure 5.
  • Qualitative results: GLIDE generalizes across diverse prompts, producing realistic shadows, reflections, textures, varied artistic styles, and compositions that bind attributes to multiple objects.Examples include corgis with bowties and birthday hats, with colors bound to the objects.
  • Image editing: GLIDE performs realistic text-guided image editing, including inpainting, style matching, sketch transformation, and iterative construction of complex scenes.Edits can insert objects, shadows, and reflections, while matching the style of paintings.
  • Guidance comparison: Classifier-free guidance is nearly Pareto optimal for FID versus Inception Score and Precision versus Recall, whereas CLIP guidance boosts CLIP score more strongly.The trade-offs are evaluated on zero-shot MS-COCO generation at 64 × 64 using Precision/Recall, FID, Inception Score, and CLIP score.
  • Comparisons: 12.24 to 12.89: GLIDE’s FID rises slightly after removing training-similar validation images, while human evaluators prefer GLIDE over DALL-E in all tested settings.GLIDE uses a 3.5-billion-parameter model, compared with DALL-E’s 12 billion parameters, and requires no CLIP reranking.

6. Safety Considerations

GLIDE can enable convincing fake-image creation and editing while reflecting harmful societal biases, motivating filtered training data and misuse evaluations. These evaluations found no recognizable human generations, no significant increase in violent or human-image quality from the filtered CLIP model, and no faithful reproduction of inspected training images.

  • Risks: GLIDE can generate realistic fake images and let unskilled users make convincing edits, lowering the skills needed for disinformation and Deepfakes.Its samples may also perpetuate harmful societal biases, including biases from the dataset.
  • Mitigations: The release models were trained on internet images filtered to remove people, reducing capabilities for many problematic people-centric use cases.The source dataset contained several hundred million images and was largely disjoint from the datasets used to train CLIP and DALL-E.
  • Misuse evaluation: Red teaming GLIDE (filtered) with adversarial prompts found no recognizable human images, suggesting the data filter had a sufficiently low false-negative rate.The investigation evaluated whether open-sourced weights could mitigate misuse risks.
  • Misuse evaluation: The filtered ViT-B CLIP model did not significantly increase violent-image or human-image quality over existing public CLIP models on the tested prompts.Researchers released the noised ViT-B CLIP model after training it on a filtered dataset and used it to guide GLIDE (filtered) and a public 64 × 64 ImageNet model.
  • Training-image regurgitation: For 30K training-set prompts, GLIDE (filtered) did not faithfully reproduce any of the inspected training images with the smallest CLIP-latent distances.Researchers computed distances between generated and original images in CLIP latent space before inspecting the closest pairs.

7. Limitations … B. Hyperparameters

The model has limitations in handling unusual prompts and remains slow to sample, while evaluation uses defined human-comparison and automated-metric procedures. Automated evaluation uses 30,000 validation-prompt samples with center-cropped references, and CLIP scores are scaled by 100.

  • 7. Limitations: The model sometimes fails on prompts describing highly unusual objects or scenarios, despite composing disparate concepts in complex ways.Figure 8 provides examples of these failure cases.
  • 7. Limitations: 15 seconds: the unoptimized model takes this long to sample one image on a single A100 GPU.This is much slower than related GAN methods, which generate images in a single forward pass and are more favorable for real-time applications.
  • A.1. Human Evaluations: 1,000 pairwise comparisons: photorealism evaluations always use this count, while caption-similarity evaluations use 1,000 except guidance-scale sweeps, which use 500.These comparisons are collected during human evaluations.
  • A.1. Human Evaluations: Ties count as half a win for each model when computing wins and Elo scores, effectively diluting both models’ wins.This convention applies to the human-evaluation scoring procedure.
  • A.1. Human Evaluations: Elo scores are computed from matrix A, where Aij counts how often model i beats model j, by minimizing the stated objective from initial scores σi = 0 for i ∈[1, N].The procedure constructs scores for all N models.
  • A.2. Automated Evaluations: 30,000 samples: MS-COCO FIDs and other metrics use this many validation-prompt samples, with the entire validation set as reference unless otherwise stated.Validation images are center-cropped, matching Ramesh et al. (2021) but differing from literature that squeezes images.
  • A.2. Automated Evaluations: 100: CLIP scores are scaled by the CLIP logit scale using the CLIP ViT-B/16 model released by Radford et al. (2021).This scaling factor is used for the reported CLIP score.

B.1. Training Hyperparameters · B.2. Sampling Hyperparameters

The paper specifies training schedules for CLIP, GLIDE, and its upsampler, then uses task-dependent diffusion-step budgets and a strided upsampler schedule for sampling. Evaluation samples use more base-model steps than displayed samples to slightly improve FID.

  • B.1. Training Hyperparameters: The final noised CLIP model uses a ViT-L with patch size 4 × 4, trained for 390K iterations at batch size 32K, then fine-tuned for 30K more.Training used a 50%-50% mixture of datasets from Radford et al. (2021) and Ramesh et al. (2021), with weight decay 0.0125 for the final ViT-L.
  • B.1. Training Hyperparameters: GLIDE (filtered) was pre-trained for 1.1M iterations and fine-tuned for another 500K iterations for classifier-free guidance and inpainting.A small filtered upsampler used 192 base channels and 512 text encoder channels, trained for 400K iterations.
  • B.1. Training Hyperparameters: The small filtered upsampler model uses 192 base channels and 512 text encoder channels during its 400K-iteration training run.This training is reported separately from GLIDE’s 1.1M pre-training and 500K fine-tuning schedule.
  • B.2. Sampling Hyperparameters: Displayed base-model samples use 150 diffusion steps, except inpainting samples use 100 steps, while evaluations use 250 steps for a slight FID boost.The sampling-step count therefore differs between paper visualizations, inpainting, and quantitative evaluations.
  • B.2. Sampling Hyperparameters: The upsampler reaches good sample quality with 27 diffusion steps using a five-segment strided schedule allocating 10, 10, 3, 2, and 2 steps.The schedule samples two timesteps in (800, 1000] but 10 in (0, 200].
  • B.2. Sampling Hyperparameters: The upsampler’s strided sampling schedule was selected by sweeping FID on an internal validation set.This criterion motivated the uneven allocation of sampling steps across timestep ranges.

C. Comparison to Smaller Models

Comparisons with a 300 million-parameter model show that larger models and broader training data improve attribute binding, compositional tasks, and unusual-concept combinations. Human evaluations find classifier-free guidance provides a larger Elo boost than scaling the model by roughly 10x.

  • Model comparison: The study trains a 300 million-parameter GLIDE (small) model on the full dataset using the same hyperparameters as GLIDE (filtered).Samples from the large, small, and safe models are compared to assess gains from larger scale and a large, diverse dataset.
  • Model comparison: Smaller models often fail to bind attributes to objects and perform worse on compositional tasks, while full-dataset models better combine unusual concepts.All models can often produce realistic images, but the full-dataset models better handle examples such as a hedgehog using a calculator.
  • Human evaluation: Classifier-free guidance gives a larger Elo boost than scaling the model by roughly 10x.For the 300M model, human evaluators slightly prefer guidance scale 4.0 to 3.0 before comparing guided and unguided large and small models.

D. Comparison to Unnoised CLIP Guidance · E. Comparison to Blended Diffusion · F. GLIDE (filtered)

The section finds that noised CLIP guidance produces images of equal or higher quality with a simpler procedure than unnoised CLIP guidance, while masked-context conditioning improves caption adherence in image editing but can reduce scene smoothness.

  • D. Comparison to Unnoised CLIP Guidance: Unnoised CLIP guidance typically requires engineered augmentations and auxiliary losses to obtain recognizable samples.The passage attributes this need to CLIP not being trained on the noisy or blurry images encountered during diffusion sampling.
  • D. Comparison to Unnoised CLIP Guidance: The comparison evaluates Crowson’s popular CLIP-guided diffusion program against GLIDE’s noised-CLIP approach.The noised ViT-B CLIP model is trained on 64 × 64 images using the same dataset as Radford et al. (2021).
  • D. Comparison to Unnoised CLIP Guidance: Noised CLIP guidance produces images of equal or higher quality than the more engineered unnoised-CLIP approach.The method uses a noised ViT-B CLIP model, fixed gradient scale 15.0, random class labels per timestep, and diffusion upsampling.
  • E. Comparison to Blended Diffusion: The fine-tuned model sometimes ignores the text prompt and produces an image influenced mainly by surrounding context.This behavior is observed when evaluating prompts used in Blended Diffusion.
  • E. Comparison to Blended Diffusion: Fully masking the context improves caption consistency, but can produce objects that fit less smoothly into the scene.With masked context, the model receives contextual information through the noised masked x_t.

F.1. Data Filtering for GLIDE (filtered) · F.2. Biases and CLIP Guidance for GLIDE (filtered)

GLIDE (filtered) uses CLIP-based SVM classifiers and active learning to remove human, violent-object, and selected hate-symbol images. Despite filtering, it retains biases related to gender, religion, hate symbols, and face-like outputs, with classifier-free guidance amplifying some effects.

  • F.1. Data Filtering for GLIDE (filtered): Human-image filtering resizes images, takes three crops, mean-pools CLIP ViT-B/16 features, and trains an RBF-kernel SVM.The classifier bias was tuned to produce less than a 1% false negative rate.
  • F.1. Data Filtering for GLIDE (filtered): Less than a 1% false negative rate was the target for human and violent-object classifiers, each producing no false negatives on separate batches of 1024 samples.The violent-object classifier was iteratively improved using examples near its decision boundary.
  • F.1. Data Filtering for GLIDE (filtered): Switching from ViT-B/32 to ViT-B/16 remedied previously observed misses of people in low-light or obstructed conditions.The authors attribute the improvement to ViT-B/16’s higher hidden-state resolution.
  • F.1. Data Filtering for GLIDE (filtered): Violent-object filtering began with CLIP searches for terms such as “weapon” and “violence,” followed by SVM training and active-learning iterations.The final classifier was tested on a separate batch of 1024 samples.
  • F.1. Data Filtering for GLIDE (filtered): Hate-symbol filtering used CLIP keyword searches, external images of swastikas and confederate flags, and an SVM trained with active learning.The initial keyword search surfaced few relevant images, suggesting the data sources had already filtered some content.
  • F.2. Biases and CLIP Guidance for GLIDE (filtered): GLIDE (filtered) produces different toy outputs for boys and girls, while “a religious place” tends toward church-like buildings, amplified by classifier-free guidance.These examples demonstrate biases beyond those found in images of people and biases introduced by filtering choices.
  • F.2. Biases and CLIP Guidance for GLIDE (filtered): The hate-symbol classifier likely has strong American and Western bias because it was trained only on two prevalent American hate symbols.The authors therefore expect unfiltered hateful symbols to remain in the training data, while the filtered model is less able to generate non-hate symbols.
  • F.2. Biases and CLIP Guidance for GLIDE (filtered): Combining GLIDE (filtered) with a public CLIP-guided diffusion program generated some face-like objects, highlighting a limitation of the filtering approach.The original program with a public diffusion model often produced more recognizable prompt responses.

G. Additional Samples

Additional samples show that classifier-free guidance most reliably produces the highest-quality images, while GLIDE samples are presented without CLIP reranking unlike DALL-E. The figures also compare filtered and small models and illustrate classifier-free versus CLIP guidance across prompts.

  • Guidance comparisons: Classifier-free guidance produces the highest-quality images most reliably across the displayed random-sample grids.The grids use identical random seeds for no guidance, classifier-free guidance, CLIP guidance, and DALL-E.
  • Sampling protocol: DALL-E samples 512 images per prompt and selects the top 16 by CLIP reranking, whereas other grids show 16 random samples without reranking.GLIDE samples are explicitly shown without CLIP reranking for the example prompts “a stained glass window of a panda eating bamboo” and a cozy living room scene.
  • Model comparisons: For “orange triangle” and “recycling symbol,” GLIDE (filtered) produces less faithful renditions than GLIDE (small), likely because the filtered model has less training data.The symbols were not filtered from the model.
  • Guidance comparisons: Figures for “a happy person” and “a place of worship” compare GLIDE with classifier-free guidance scale 3.0 against CLIP-guided GLIDE and CLIP-guided ImageNet diffusion.The comparisons use a publicly available CLIP model, publicly available ImageNet diffusion model, and GLIDE (filtered).
Loading 2112.10741v3…