PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Diffusion Distillation

Axel Sauer, Dominik Lorenz, Andreas Blattmann, Robin Rombach

arXiv:2311.17042v1cs.CV

TL;DR

Diffusion models deliver high-quality images but are slowed by iterative sampling, while existing few-step methods can sacrifice quality. ADD combines adversarial training with score distillation from a frozen diffusion teacher to generate high-fidelity images in 1–4 steps. It outperforms concurrent few-step methods in one or two steps and surpasses widely used multi-step generators with four steps.

  • Problem

    Diffusion models’ iterative sampling hinders real-time application, while few-step methods and GANs can suffer from reduced quality or artifacts.

  • Method

    ADD distills pretrained diffusion models using a hybrid objective that combines adversarial loss with score distillation from a frozen diffusion teacher.

  • Results

    ADD outperforms concurrent methods in the one- or two-step regime and, with four steps, outperforms widely used generators including SDXL, IF, and OpenMUSE.

  • Takeaways & Limitations

    ADD enables high-quality single-step image generation and supports iterative refinement with foundation models.

  • Takeaways & Limitations

    GAN-based approaches remain difficult to scale because stable adversarial training requires finely balanced architectures.

Abstract

from arXiv · show

We introduce Adversarial Diffusion Distillation (ADD), a novel training approach that efficiently samples large-scale foundational image diffusion models in just 1-4 steps while maintaining high image quality. We use score distillation to leverage large-scale off-the-shelf image diffusion models as a teacher signal in combination with an adversarial loss to ensure high image fidelity even in the low-step regime of one or two sampling steps. Our analyses show that our model clearly outperforms existing few-step methods (GANs, Latent Consistency Models) in a single step and reaches the performance of state-of-the-art diffusion models (SDXL) in only four steps. ADD is the first method to unlock single-step, real-time image synthesis with foundation models. Code and weights available under https://github.com/Stability-AI/generative-models and https://huggingface.co/stabilityai/ .

1. Introduction

Diffusion models provide high-quality, scalable image synthesis but require many iterative sampling steps, while GANs are fast yet often produce lower-quality samples. ADD combines adversarial and distillation objectives to reduce pretrained diffusion inference to 1–4 steps while preserving fidelity.

  • Diffusion models support high-quality image synthesis but their iterative inference hinders real-time application.GANs offer single-step speed but often fall short of diffusion models in sample quality.
  • ADD reduces pretrained diffusion-model inference to 1–4 sampling steps while maintaining high sampling fidelity.
  • ADD combines adversarial loss with score distillation from a fixed pretrained diffusion teacher.The adversarial objective targets realistic image-manifold samples, while distillation preserves the teacher’s compositionality.
  • ADD produces high-fidelity, real-time image generation and significantly outperforms LCM, LCM-XL, and single-step GAN baselines at one inference step.
  • With four sampling steps, ADD-XL outperforms its SDXL-Base teacher at 512^2 px resolution.

2. Background

Existing acceleration methods reduce diffusion sampling but can produce blurry artifacts, while GANs are fast yet lag in quality and face scaling challenges. ADD addresses these issues by combining adversarial training with score distillation in a hybrid objective.

  • Four-step samples from existing methods often look blurry and exhibit noticeable artifacts, with degradation amplified at fewer steps.
  • GANs provide impressive sampling speed but lag behind diffusion-based models in performance and are difficult to scale reliably.Their training depends on finely balanced architectures, and current text-to-image GANs lack classifier-free guidance.
  • Score Distillation Sampling distills foundational text-to-image model knowledge into other generative models, including 3D synthesis.
  • ADD combines adversarial training and score distillation in a hybrid objective to address issues in leading few-step generative models.

3. Method

ADD targets high-fidelity image generation in as few sampling steps as possible by combining adversarial training with score distillation. Its training uses a student, discriminator, and frozen diffusion-model teacher, with additional conditioning and reconstruction choices supporting the low-step regime.

  • ADD aims to generate high-fidelity samples in as few sampling steps as possible while matching state-of-the-art model quality.
  • The adversarial objective trains the student to produce samples on the image manifold, while score distillation uses a pretrained diffusion model as a teacher signal.
  • Training uses an ADD student initialized from a pretrained UNet-DM, a trainable discriminator, and a frozen diffusion-model teacher.
  • The formulation supports exponential or SDS weighting and can be adapted to latent diffusion models, with pixel-space distillation yielding more stable gradients in their experiments.
  • The discriminator can use text and image conditioning, encouraging the student to use input-image information when that signal is available.
  • The distillation loss compares generated samples with the teacher’s denoising predictions after diffusing the samples, rather than applying the teacher directly to them.

4. Experiments

Experiments evaluate ADD across ablations, distillation comparisons, user preferences, inference speed, and qualitative results. The studies identify effective design choices and show strong few-step performance, including advantages over established baselines and the teacher model.

  • Experimental Setup: ADD experiments use 512x512 outputs, with ADD-M and ADD-XL models containing 860M and 3.1B parameters, respectively.ADD-M uses Stable Diffusion backbones, while ADD-XL uses an SDXL backbone.
  • Ablation Study: DINOv2 is the best discriminator feature network among the evaluated alternatives.The ablation compares feature networks based on CLIP and DINO objectives.
  • Ablation Study: Combining text and image conditioning for the discriminator yields the best results, with image conditioning outperforming text conditioning alone.The conditioning ablation reports that image conditioning is stronger than text conditioning, while their combination performs best.
  • Ablation Study: Pretraining the ADD student is important and lets ADD leverage larger pretrained diffusion models, unlike the reported saturation behavior of pure GAN approaches.The paper identifies stable diffusion-model pretraining as a significant advantage over standalone GAN training.
  • Ablation Study: Both adversarial and distillation losses are essential; distillation alone is ineffective, while weighting schedules trade off diversity against quality and text alignment.The exponential schedule tends to produce more diverse samples, whereas SDS and NFSD schedules improve quality and text alignment; NFSD is used for the final model.
  • Quantitative Comparison: ADD-XL outperforms LCM-XL in one step, beats SDXL with four steps in most comparisons, and outperforms other evaluated approaches including an eight-step DPM solver.User studies assess prompt adherence and image quality using pairwise win percentages and ELO scores; four-step ADD-XL also surpasses its teacher in human preference.

5. Discussion

ADD combines adversarial and score distillation objectives to produce high-fidelity images in one or two steps while retaining multi-step refinement. With four steps, ADD-XL outperforms widely used multi-step generators, including SDXL, IF, and OpenMUSE.

  • ADD combines adversarial and score distillation objectives to distill pretrained diffusion models into fast, few-step image generators.The approach leverages real data through a discriminator and structural information from a frozen diffusion teacher.
  • ADD performs particularly well in the ultra-fast regime of one or two sampling steps, outperforming concurrent methods there.
  • Four sampling steps enable ADD to outperform widely used multi-step generators such as SDXL, IF, and OpenMUSE.
  • Single-step generation with high-quality images opens possibilities for real-time image generation with foundation models.

A. SDS As a Special Case of the Distillation Loss

The distillation loss becomes equivalent to the score distillation objective under a specific weighting function and squared Euclidean distance.

  • Setting c(t) = α_t^2σ_t w(t) and d(x, y) = ||x − y||2 makes the distillation loss equivalent to the score distillation objective.

B. Details on Human Preference Assessment

The human preference studies compare ADD-M at one step and ADD-XL at four steps against established baselines. The evaluation uses selected PartiPrompts categories that standard generative-model metrics capture insufficiently.

  • Human evaluation is used instead of FID and CLIP-score because those metrics insufficiently capture aesthetics and scene composition.The study evaluates 100 selected prompts from the PartiPrompts benchmark.
  • The single-step preference study compares ADD-M against established baselines.
  • The multiple-step preference study compares four-step ADD-XL against established baselines.

B.1. Experimental Setup

The human preference assessment uses pairwise comparisons of model outputs for identical prompts, collects multiple annotator votes, and aggregates rankings with ELO scores.

  • Evaluators compare two images from different models generated from the same text prompt in pairwise 1v1 studies.Each comparison receives an average of four votes from different annotators for visual quality and prompt following.
  • ELO scores aggregate rankings when more than two models are compared through a sequence of pairwise results.Higher ELO scores indicate better rankings.
  • The study sets K = 1, bootstraps final rankings over 1000 randomly shuffled calculations, and initializes every model at Rinit = 1000.

C. GAN Baselines Comparison

The comparison evaluates StyleGAN-T++ against prior GANs using FID–CLIP tradeoffs, while additional figures document fast diffusion-model samples.

  • GAN baseline setup: StyleGAN-T++ uses extended training, a revised discriminator architecture, and R1 regularization at each discriminator head.The training schedule is approximately 2M iterations with batch size 2048.
  • GAN comparison: StyleGAN-T++ achieves comparable zero-shot FID to GigaGAN at a significantly higher CLIP score.The comparison concerns GAN baselines rather than diffusion models, whose automatic-metric comparisons are described as less informative.
  • GAN comparison: Figure 11 compares FID–CLIP score curves for StyleGAN-T, StyleGAN-T++, and GigaGAN across truncation values ψ = {1.0, 0.9, . . . , 0.3}.Increasing CLIP score is obtained through decreasing truncation.

D. Additional Samples

Additional figures show ADD-XL qualitative samples, comparisons with other fast samplers, and how increasing sampling steps affects output diversity and spelling.

  • Additional samples: Additional figures present one-step ADD samples and prompts including robots, teddy bears, and a kangaroo portrait.The samples are presented as further qualitative examples of the method’s outputs.
  • Fast-sampler comparisons: Figure 13 provides qualitative comparisons between ADD-XL and LCM-XL, InstaFlow, and OpenMuse.The comparison covers state-of-the-art one- and few-step models.
  • Sampling-step effects: Single-step ADD-XL samples are already often high quality, while additional steps can improve diversity and spelling capabilities.The qualitative comparison uses 1, 2, and 4 sampling steps with constant seeds within columns, preserving general layout.
Loading 2311.17042v1…