PaperlayerView exact revision on arXiv

Source-linked AI summary

SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation

Thuan Hoang Nguyen, Anh Tran

arXiv:2312.05239v7cs.CV

TL;DR

Text-to-image diffusion is slowed by iterative sampling, and prior distillation approaches can require training images. SwiftBrush adapts a text-to-3D distillation loss for image-free one-step generation, reporting 16.6 FID and 0.29 CLIP on MS COCO 2014 (30K images).

  • Problem

    Text-to-image diffusion models are slowed by iterative sampling, while prior distillation methods can require substantial real or teacher-generated image data.

    Source: §1 · p. 2

  • Method

    SwiftBrush adapts text-to-3D distillation principles to distill a pretrained multi-step text-to-image model into an image-free, one-step student.

    Source: §1 · p. 2

  • Results

    16.6 FID and 0.29 CLIP on MS COCO 2014 (30K images); the paper reports outperforming prior distilled one-step approaches without training images.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports that SwiftBrush makes text-to-image generation more efficient and accessible through image-free distillation.

    Source: §5 · p. 8

    The current design is limited to single-step generation and cannot support few-step generation to improve image fidelity.

    Source: §5 · p. 8

Abstract

from arXiv · show

Despite their ability to generate high-resolution and diverse images from text prompts, text-to-image diffusion models often suffer from slow iterative sampling processes. Model distillation is one of the most effective directions to accelerate these models. However, previous distillation methods fail to retain the generation quality while requiring a significant amount of images for training, either from real data or synthetically generated by the teacher model. In response to this limitation, we present a novel image-free distillation scheme named $\textbf{SwiftBrush}$. Drawing inspiration from text-to-3D synthesis, in which a 3D neural radiance field that aligns with the input prompt can be obtained from a 2D text-to-image diffusion prior via a specialized loss without the use of any 3D data ground-truth, our approach re-purposes that same loss for distilling a pretrained multi-step text-to-image model to a student network that can generate high-fidelity images with just a single inference step. In spite of its simplicity, our model stands as one of the first one-step text-to-image generators that can produce images of comparable quality to Stable Diffusion without reliance on any training image data. Remarkably, SwiftBrush achieves an FID score of $\textbf{16.67}$ and a CLIP score of $\textbf{0.29}$ on the COCO-30K benchmark, achieving competitive results or even substantially surpassing existing state-of-the-art distillation techniques.

1. Introduction

Text-to-image diffusion offers high-quality generation but is slowed by iterative sampling, while prior one-step distillation methods can require complex training, produce blurry outputs, or depend on many image pairs. SwiftBrush adapts text-to-3D distillation principles to train an image-free one-step generator, reporting competitive benchmark results.

1. Introduction

  • Iterative sampling is the primary reason text-to-image diffusion inference is slow, challenging deployment on consumer devices.Source: §1 · p. 2
  • Prior methods use complex training (at least 108 A100 GPU days), yield blurry one-step results, or rely heavily on text-image pairs.The cited methods include two-stage distillation, LCM, and four-stage InstaFlow training.Source: §1 · p. 2
  • SwiftBrush adapts text-to-3D principles to distill a pretrained text-to-image model into a one-step generator without 2D image supervision.The approach draws on using a pretrained 2D model to assess rendered outputs, as in text-to-3D synthesis.Source: §1 · p. 2
  • 16.6 FID and 0.29 CLIP on MS COCO 2014 (30K images); SwiftBrush is the first distilled one-step model reported to outperform prior approaches without training images.The paper reports high-quality images with remarkable details.Source: §1 · p. 2

2. Related Work

Related work contrasts the speed of GANs with diffusion-model realism, and traces SwiftBrush's connections to text-to-3D priors and diffusion distillation. It also describes earlier training-data requirements and image-free distillation approaches.

2. Related Work

  • Earlier text-to-image models relied on GANs in small object-centric domains, while later systems drew on web-scale text-image datasets and large language and vision-language models.Source: §2 · pp. 2–3
  • Text-to-image GANs avoid iterative sampling but are described as less scalable and realistic than diffusion models, which produce high-quality sharp samples more slowly.Source: §2 · p. 3
  • Text-to-3D methods use 2D text-to-image priors, including DreamFusion [28] and SJC [44]; SwiftBrush adapts VSD, also used by Score GAN [6] and Diff-Instruct [24].Magic3D [18] uses two stages, while ProlificDreamer [45] introduces a secondary teacher in VSD.Source: §2 · p. 3
  • Knowledge distillation transfers teacher knowledge to a student [9]; diffusion acceleration moved from costly 50–1000-step supervision to bootstrapping, with [20] [23] [25] image-dependent and concurrent [8] image-free.The paper characterizes its own design as image-free and simpler than the concurrent approach.Source: §2 · p. 3

3. Proposed Method

The method adapts score distillation from text-to-3D synthesis to train a generalized one-step text-to-image student. It uses a frozen pretrained teacher alongside an alternating LoRA teacher, and reparameterizes the student to predict clean images.

3.1. Preliminary

  • Diffusion models add noise to data in the forward process and learn a reverse denoising process; latent diffusion performs this process in a pretrained autoencoder’s smaller latent space.The smaller latent representations enhance training and inference efficiency.Source: §3.1 · p. 3
  • Score Distillation Sampling uses a pretrained text-to-image diffusion model to optimize a NeRF from rendered views, but it can yield oversaturation, oversmoothing, and reduced diversity.The paper reports observing the same degradations when SDS is applied naively in its framework.Source: §3.1 · p. 4
  • Variational Score Distillation adds a score function fine-tuned to the rendered-image distribution, and ProlificDreamer alternates that fine-tuning with optimization of the NeRF.The added score model is initialized from the pretrained diffusion model and conditioned on camera view in the cited text-to-3D setting.Source: §3.1 · p. 4

3.2. SwiftBrush

  • SwiftBrush replaces NeRF rendering in score distillation with a one-step text-to-image generator, converting text-to-3D training into one-step diffusion distillation.The generator directly synthesizes a text-guided image for the distillation objective.Source: §3.2 · p. 4
  • The design uses a pretrained text-to-image teacher and a LoRA teacher without camera conditioning, alternating their training with a generalized one-step student while freezing the pretrained teacher.Both the student and LoRA teacher start from the pretrained teacher’s weights, and classifier-free guidance is used for both teachers.Source: §3.2 · p. 4
  • The student reparameterization converts the pretrained model’s noise-prediction output into a predicted clean-image x0 output, which is empirically helpful for learning.The pretrained model predicts added noise, whereas the student objective is to predict noise-free x0.Source: §3.2 · pp. 4–5

4. Experiments

The experiments evaluate SwiftBrush against text-to-image baselines on COCO 2014 and HPSv2, then analyze its training components and latent-space behavior. SwiftBrush performs strongly in one-step generation, while ablations highlight the roles of the LoRA teacher and student parameterization.

4.1. Experimental Setup

  • The evaluation uses COCO 2014 FID and CLIP metrics plus HPSv2 across anime, concept-art, paintings, and photos; training uses deduplicated captions rather than images.The captions come from JourneyDB: 4M were deduplicated to 1.38M; the two teachers use Stable Diffusion v2.1.Source: §4.1 · p. 5

4.2. Results

  • SwiftBrush’s outputs are described as high quality and text-aligned, comparable to 25-step Stable Diffusion; LCM is blurry, while InstaFlow struggles with some prompts.Figure 4 compares methods on shared prompts, with rows representing prompts and columns representing models.Source: §4.2 · p. 6, Figure 4 · p. 7
  • SwiftBrush improves one-step realism and alignment on COCO, with FID exceptions for InstaFlow and 25-step Stable Diffusion; it leads HPSv2 across aspects and beats InstaFlow in the user survey.The tables compare COCO FID and CLIP scores and HPSv2 scores, respectively.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.3. Analysis

  • With the LoRA teacher, the student progressively produces realistic images; without an effective LoRA teacher, outputs remain prompt-aligned but overly saturated and unrealistic, with flat FID and CLIP curves.These comparisons use early training outputs after 10K iterations.Source: §4.3 · pp. 6–7
  • With rank-4 LoRA, FID starts worsening after 8K steps despite an initial decline, and CLIP similarly rises slightly before worsening as the model undergoes mode collapse.The reported later outputs are poor despite initially reasonable results.Source: §4.3 · p. 7, Figure 5 · p. 8
  • Without student re-parameterization, outputs are noisy and FID and CLIP performance are consistently worst among the tested configurations.The ablation evaluates four configurations during the early 10K training iterations.Source: §4.3 · p. 7, Figure 5 · p. 8
  • With the noise input held fixed while prompts vary, the distilled student shows seamless image transitions and seemingly greater control and editability than the teacher.The authors attribute the apparent control and editability to the student’s one-step nature.Source: §4.3 · pp. 7–8

5. Conclusion and Discussion

SwiftBrush is an image-free distillation scheme for efficient text-to-image generation, drawing on text-to-3D synthesis. Its current design is limited to one-step generation, which produces lower-quality samples than the multi-step teacher.

5. Conclusion and Discussion

  • SwiftBrush is an image-free distillation scheme inspired by text-to-3D synthesis, intended to make text-to-image generation more efficient and accessible.The authors hope that lowering inference cost will encourage broader use of text-to-image generation systems.Source: §5 · p. 8
  • The one-step student produces lower-quality samples than the teacher using multi-step inference.The authors identify extending SwiftBrush to few-step generation as future work to trade computation for image fidelity.Source: §5 · p. 8
  • Figure 7 presents prompt-interpolation examples using four animals and three actions in a DSLR-photo template.The animal prompts are owl, raccoon, cat, or panda; the actions are reading a book, drinking a latte, or eating a pizza.Source: Figure 7 · p. 8
  • The current design supports only single-step generation, unlike few-step generation explored in Luo et al. [23].The authors also propose exploring training with only one teacher and integrating DreamBooth, ControlNet, or InstructPix2Pix.Source: §5 · p. 8

6. Additional Details

The additional details compare SwiftBrush’s training cost, inference time, and memory footprint with other methods. Its reported inference time is 110 ms, and training takes about 4.11 A100 GPU days.

Training Cost

  • 4.11 A100 GPU days are required to distill Stable Diffusion with SwiftBrush, compared with about 5.56 days for the BOOT reimplementation.SwiftBrush training uses 65,000 gradient updates; the BOOT reimplementation converges in about 100,000 iterations.Source: §6 · p. 11

Inference Speed

  • 110 ms is SwiftBrush’s total inference time, compared with 115 ms for BOOT.SwiftBrush’s text encoder, UNet, and VAE take 80 ms, 25 ms, and 5 ms, respectively; BOOT has a 30 ms UNet overhead.Source: §6 · p. 11

Memory Requirement

  • Training only a student and an additional low-rank LoRA weight for the LoRA teacher gives SwiftBrush a minimal memory footprint.The passage reports training at batch size 16 and inference at batch size 1 on an A100 GPU, but does not provide memory values here.Source: §6 · p. 11

Compare with Others

  • SwiftBrush is described as combining efficiency with high-quality output, while one-step LCM is described as producing poor-quality images.Guided Distillation and InstaFlow are described as requiring considerable training time, limiting their practical use.Source: §6 · p. 11

7. Additional Visual Results

The additional visuals include uncurated samples, component comparisons across training stages, and prompt or noise interpolation. The section also identifies a comparison of inference speed, training time, and memory requirements.

Uncurated Samples

  • Uncurated outputs conditioned on 25 random prompts are shown for SwiftBrush, InstaFlow, BOOT, and LCM.The SwiftBrush and LCM samples are identified as uncurated; the InstaFlow samples use its provided pretrained model, while BOOT is reimplemented.Source: §7 · p. 11

Additional Details

  • Table 3 compares methods on inference speed, training time, and memory usage.Its units are milliseconds, A100 GPU days, and gigabytes, respectively.Source: Table 3 · p. 11

Additional Visuals of Analysis

  • The ablation visuals compare four configurations across training stages using the prompt “An DSLR photo of a tiger in the city.”The configurations are Full, SDS, Small-Rank LoRA, and No Parameterization.Source: §7 · p. 11

Additional Visuals of Interpolation

  • Text-prompt and noise interpolation produce seamless output transitions, which the authors describe as showing editability and controllability.The prompt interpolation holds noise fixed, while the noise interpolation holds the text input fixed.Source: §7 · p. 11

8. Additional Quantitative Results

On CIFAR-10 and class-conditional ImageNet using EDM teachers, SwiftBrush is reported to be on par with CD-LPIPS and to significantly outperform the other compared methods. Additional figures show uncurated samples, interpolation, and ablation configurations.

Quantitative Benchmarks

  • SwiftBrush is on par with CD-LPIPS and significantly outperforms the other compared methods on CIFAR-10 and class-conditional ImageNet benchmarks.The comparison uses EDM teachers and reports FID; Table 4 covers CIFAR-10 at 32 × 32 and ImageNet at 64 × 64.Source: §8 · p. 11, Table 4 · p. 11

Uncurated Samples

Visual Analyses

  • The figures visualize prompt and noise interpolation, plus four ablation configurations across training iterations.Prompt interpolation varies the text embedding with the same noise input; noise interpolation varies the noise with the same text input. The ablation configurations are Full, SDS, Small-Rank LoRA, and No Parameterization.Source: Figure 12 · p. 16, Figure 13 · p. 17, Figure 14 · p. 18
Loading 2312.05239v7…