PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Step Diffusion via Score identity Distillation

Mingyuan Zhou, Yi Gu, Zhendong Wang

arXiv:2505.12674v1cs.CVcs.LGstat.ML

TL;DR

High-resolution text-to-image diffusion distillation still often depends on real or teacher-synthesized images, while classifier-free guidance poses an alignment–diversity trade-off. This paper extends data-free SiD to few-step generation using a theoretically justified uniform mixture of outputs, and adds real-data adversarial training with Zero-CFG and Anti-CFG. Experiments on SD1.5 and SDXL report state-of-the-art one- and few-step performance.

  • Problem

    High-resolution T2I distillation often relies on real or teacher-synthesized images, while CFG leaves a persistent trade-off between text-image alignment and generation diversity.

    Source: §1 · p. 2

  • Method

    The paper extends data-free SiD to few-step generation by matching a uniform mixture of outputs and adds a Diffusion GAN loss with Zero-CFG or Anti-CFG when real text-image pairs are available.

    Source: §5 · p. 10

  • Results

    Experiments on SD1.5 and SDXL report state-of-the-art performance in both one- and few-step generation, surpassing prior methods in FID and CLIP.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The uniform-mixture formulation avoids step-specific networks and integrates into existing training pipelines, while the data-free approach remains robust without real images.

    Source: §5 · p. 10

    The real-data training set is limited to 480k images and becomes substantially oversampled around the point when CLIP scores tend to peak.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

Diffusion distillation has emerged as a promising strategy for accelerating text-to-image (T2I) diffusion models by distilling a pretrained score network into a one- or few-step generator. While existing methods have made notable progress, they often rely on real or teacher-synthesized images to perform well when distilling high-resolution T2I diffusion models such as Stable Diffusion XL (SDXL), and their use of classifier-free guidance (CFG) introduces a persistent trade-off between text-image alignment and generation diversity. We address these challenges by optimizing Score identity Distillation (SiD) -- a data-free, one-step distillation framework -- for few-step generation. Backed by theoretical analysis that justifies matching a uniform mixture of outputs from all generation steps to the data distribution, our few-step distillation algorithm avoids step-specific networks and integrates seamlessly into existing pipelines, achieving state-of-the-art performance on SDXL at 1024x1024 resolution. To mitigate the alignment-diversity trade-off when real text-image pairs are available, we introduce a Diffusion GAN-based adversarial loss applied to the uniform mixture and propose two new guidance strategies: Zero-CFG, which disables CFG in the teacher and removes text conditioning in the fake score network, and Anti-CFG, which applies negative CFG in the fake score network. This flexible setup improves diversity without sacrificing alignment. Comprehensive experiments on SD1.5 and SDXL demonstrate state-of-the-art performance in both one-step and few-step generation settings, along with robustness to the absence of real images. Our efficient PyTorch implementation, along with the resulting one- and few-step distilled generators, will be released publicly as a separate branch at https://github.com/mingyuanzhou/SiD-LSG.

1 Introduction

High-resolution text-to-image distillation still faces dependence on real or teacher-synthesized images, a need for multiple refinement steps, and a CFG-related alignment–diversity trade-off. The paper extends SiD to few-step generation and introduces data-dependent loss and guidance strategies to address these challenges.

1 Introduction

  • High-resolution T2I distillation, including SDXL at 1024×1024, remains challenging because existing methods often depend on real or teacher-synthesized image data.Such dependence limits data-free applicability and constrains effectiveness even when real data is available.Source: §1 · p. 2
  • Few-step generation is needed because a one-step generator may be insufficient to reduce artifacts and refine high-resolution outputs.Reverse diffusion sampling itself remains iterative despite accelerated solvers and approximation methods.Source: §1 · p. 2
  • CFG creates an alignment–diversity trade-off: high-CLIP models often differ from low-FID models, and stronger guidance tends to favor alignment over diversity.The passage attributes the discrepancy to alignment- and fidelity-focused models typically requiring high CFG, whereas diversity-focused models benefit from lower CFG.Source: §1 · pp. 2–3
  • DMD2 is a strong SDXL distillation framework, but it relies on teacher-synthesized noise–image pairs for one-step generation and real data for four-step generation.Its distribution-matching loss improves CLIP at the cost of FID, while omitting GAN loss degrades FID.Source: §1 · p. 3
  • The paper optimizes data-free SiD for multistep generation and adds Zero-CFG or Anti-CFG, which disable teacher CFG and remove or invert fake-score text guidance.The authors position these strategies as adaptive ways to balance alignment and fidelity when real data is available.Source: §1 · p. 3
  • Experiments on SD1.5 and SDXL report state-of-the-art performance for the multistep SiD framework, real-data enhancements, and Zero-CFG and Anti-CFG guidance strategies.The paper presents key findings in the main text and defers detailed ablations and full results to the Appendix.Source: §1 · p. 3

2 Preliminaries and related works

The section establishes how noisy inputs, score networks, and generators are represented in T2I distillation. It situates distribution matching and Fisher divergence among data-free and data-dependent approaches.

2 Preliminaries and related works

  • Forward diffusion corrupts a clean image into a noisy image at timestep t, and the pretrained score network estimates the conditional data score from that noisy image and text.The model’s noise prediction is also described as an approximation to the true data score.Source: §2 · p. 3
  • A distilled generator maps noise and a text condition to generated samples, requiring one or a few evaluations instead of the teacher’s many iterative refinements.The generator’s diffused versions follow semi-implicit distributions that support score-related identities.Source: §2 · p. 3
  • Distribution matching compares noisy model and data distributions, whose overlapping supports make the divergence well-defined and yield meaningful gradients.The section names Jensen–Shannon, KL, and Fisher divergence as possible measures.Source: §2 · p. 4
  • KL-based distillation often needs real-data adversarial or teacher-synthesized regression losses, whereas Fisher-divergence SiD motivates a data-free few-step T2I solution and a real-data diversity enhancement.The related-work discussion classifies the proposed approach as score-based, with an optional Diffusion GAN loss and either data-free or real-data use.Source: §2 · p. 4

3 Method

The method extends SiD to multistep T2I generation by matching a uniform mixture of outputs to the data distribution with a shared fake score network. With real images, it adds a Diffusion GAN loss and Zero-CFG or Anti-CFG guidance.

3 Method

  • SiD’s Fisher-divergence approach is extended to few-step T2I generation, where the authors describe multiple steps, CFG, and training images as components often considered essential for high-resolution performance.The method is framed as an extension of data-free SiD, which had shown strong one-step results in other diffusion settings.Source: §3 · p. 4
  • The objective uses a pretrained score network as an approximation to the true diffused-data score and alternates optimization of the generator and fake score network.The objective is minimized when the model distribution matches the teacher, although the optimal fake score network is unknown.Source: §3 · pp. 4–5

3.1 Data-free multistep distillation

  • Final-Step Matching backpropagates through the entire generation chain, while Uniform-Step Matching samples a generation step and avoids that full-chain backpropagation.The final-step variant becomes harder to optimize as K grows; increasing K can improve FID but slow convergence and tend to reduce CLIP.Source: §3.1 · p. 5
  • Uniform-Step Matching aims to align intermediate and final outputs with the data distribution, providing dense supervision across generation steps.Stop-gradient prevents gradients from propagating into earlier steps.Source: §3.1 · pp. 5–6
  • A single shared fake score network suffices because the lemma makes the step index unnecessary and identifies the optimal score estimator for noisy images from a uniform mixture of all generation-step outputs.This provides the theoretical basis for matching the mixture to the data distribution rather than training separate score networks for each step.Source: §3.1 · p. 6

3.2 Enhancements with real images, guidance strategies, and coding optimizations

  • When real text–image pairs are used, the Diffusion GAN adversarial loss can be applied to the uniform mixture and primarily improves diversity, with reduced FID and maintained or slightly improved CLIP.The discriminator reuses the fake score network without adding model parameters.Source: §3.2 · p. 6
  • Zero-CFG removes text conditioning from the fake score network, whereas Anti-CFG applies negative CFG; both leave the teacher with standard text conditioning without CFG amplification.Combined with the adversarial loss, the strategies reinforce alignment and rank among the top-performing methods for SD1.5 and SDXL.Source: §3.2 · pp. 6–7
  • The implementation incorporates AMP, FSDP, and tailored multistep generation strategies to improve efficiency and scalability while minimizing reliance on non-native PyTorch libraries.The authors describe these optimizations as supporting stable training of high-performing one- and few-step models.Source: §3.2 · p. 7

4 Experiments

Experiments evaluate one- and few-step SiD on SD1.5 and SDXL, with data-free training and enhancement using limited real text–image pairs. SiD performs competitively without real images, while real-data variants improve FID but can show early CLIP declines.

4 Experiments

  • Data-free training uses Aesthetics6+ prompts, while data-dependent enhancement uses 480K text–image pairs; evaluations use COCO-2014 prompts and report FID and CLIP.The experiments cover SD1.5 and SDXL; the paper also distinguishes data-free SiD, data-enhanced SiDa, and the two-stage SiDa 2 variant.Source: §4 · p. 7

4.1 Ablation studies on multistep generation, real data usage, and guidance strategies

  • 7.89 FID with 0.304 CLIP: SD1.5 one-step SiDa at LSG scale 1.5 improves on SiD’s 8.15 FID at the same CLIP and is the lowest reported FID among SD1.5 one-step generators above 0.3 CLIP.At LSG scale 4.5, SiDa reaches 10.10 FID and 0.321 CLIP, compared with SiD’s 16.59 FID and 0.317 CLIP.Source: §4.1 · p. 8
  • With four-step matching, uniform-step matching maintains high CLIP while benefiting FID as steps increase, whereas final-step matching improves FID but lowers CLIP.Visual inspection shows image-quality gains, particularly in fine details, from one to four steps, with diminishing returns thereafter; quantitative FID and CLIP improvements are not consistent.Source: §4.1 · pp. 7–8
  • For four-step SiDa 2 on SD1.5, Anti-CFG with early stopping favors higher CLIP (0.325) with moderately lower FID (13.35), while Zero-CFG gives 0.318 CLIP and 9.86 FID.The two-stage strategy initializes real-data training from a four-step, data-free SiD generator trained with LSG at guidance scale 4.5.Source: §4.1 · p. 8, Figure 2 · p. 8

4.2 Distilling SDXL with SiD and its enhanced versions

  • 13.25 FID with 0.335 CLIP: four-step SiDa 2 under Zero-CFG achieves the best FID, while Anti-CFG with early stopping achieves the best CLIP (0.344) and a competitive 15.82 FID on SDXL.The Anti-CFG variant is reported to provide a balanced FID–CLIP trade-off; early stopping is typically within 1M fake images for SDXL.Source: Table 2 · p. 9, §4.2 · pp. 9–10
  • SiD optimized for four-step SDXL generation performs on par with Rectified Diffusion-XL in CLIP and delivers competitive FID; real data further improves FID, often with lower CLIP for LSG and Zero-CFG.The authors report that SiD remains highly competitive for one- and few-step generation without real images; 480K real images can further lower FID while maintaining high CLIP scores.Source: §4.2 · pp. 9–10

4.3 Qualitative evaluations, limitations, and future work

  • Qualitative evaluations report strong visual fidelity, text–image alignment, and diversity across SiD variants enhanced with multistep generation, real images, and new guidance strategies.The paper notes that FID and CLIP do not always reflect human-perceived quality because distillation targets distributional fidelity rather than subjective preference.Source: §4.3 · p. 10
  • Real-data SiDa and SiDa 2 typically continue improving FID as CLIP peaks early and declines; the authors associate this pattern with oversampling the limited 480K-image dataset.The full training dataset contains billions of images and is beyond the authors’ current computational and storage capacity.Source: §4.3 · p. 10

5 Conclusion

The paper extends SiD to few-step generation by matching a uniform mixture of outputs from all generation steps, avoiding step-specific networks. It combines limited-real-data adversarial enhancement with Zero-CFG and Anti-CFG, and reports state-of-the-art one- and few-step results on SD1.5 and SDXL.

5 Conclusion

  • The multistep algorithm matches a uniform mixture of outputs from all generation steps to the data distribution, avoiding step-specific networks.The authors describe the formulation as theoretically grounded and readily integrated into existing training pipelines.Source: §5 · p. 10
  • Zero-CFG and Anti-CFG alter text conditioning in the fake score network while disabling teacher CFG; paired with a Diffusion GAN loss, they improve diversity without compromising alignment or fidelity.The approach also includes data-dependent optimization when real text-image pairs are available.Source: §5 · p. 10
  • Experiments on SD1.5 and SDXL report state-of-the-art performance in both one- and few-step generation, surpassing prior methods in FID and CLIP.The authors report robustness with or without real images, as well as efficiency and scalability.Source: §5 · p. 10

A Additional Qualitative Examples

The appendix provides four-step, 1024×1024 examples from data-free SiD-LSG and SiD initialized with SiD and enhanced using limited real text-image pairs. The broader-impact discussion notes both reduced sampling cost and potential misuse concerns for deployment.

A Additional Qualitative Examples

  • Four-step SiD-LSG examples are shown at 1024×1024 resolution, and SiD does not require access to real images.Source: Figure 3 · p. 15
  • Four-step SiDa 2 examples use Zero-CFG after initializing from data-free SiD and continuing training with limited real text-image pairs.Source: Figure 4 · p. 16

B Broader Impact

  • The authors say the framework reduces diffusion sampling cost, while its efficiency and scalability may lower deployment barriers for models trained on uncurated or sensitive data and raise misuse concerns.They call for clear ethical guidelines and practical safeguards.Source: B Broader Impact · p. 16

C Related Work

This section surveys score- and trajectory-based approaches to diffusion distillation and situates SiD among methods using adversarial losses, teacher trajectories, or specialized generators. It also includes qualitative comparisons, reported metrics, and the SiD training algorithm.

C Related Work

  • Diffusion distillation work includes consistency distillation, teacher-ODE noise-to-image supervision, rectified flow, and Diffusion GAN-based adversarial methods.The section also discusses progressive distillation, preference optimization, and specialized discriminator heads as extensions or related approaches.Source: C Related Work · pp. 16, 19
  • For SD1.5, four-step SiD-LSG is reported at FID 16.60 and CLIP 0.319; the accompanying comparison says SiD-based enhancement with limited real data improves diversity and alignment.The Anti-CFG variant is reported as achieving the highest CLIP score in the comparison.Source: Figure 7 · p. 18, C Related Work · p. 18
  • Rectified-flow performance remains constrained by the capacity of the teacher ODE solver, while mixtures of same-sized or smaller generators have improved performance and efficiency when combined with DMD.Source: C Related Work · p. 18
  • Four-step SDXL SiD-LSG is reported at FID 21.04 and CLIP 0.340, while the qualitative comparison describes SiD-LSG as on par with leading SDXL distillation methods.Source: C Related Work · p. 19, Figure 8 · p. 19
  • The training algorithm supports K-step inference with either Final-Step or Uniform-Step Matching and can use data-dependent enhancement during training.The procedure outputs a generator optimized for K-step inference.Source: C Related Work · p. 20
  • The authors focus on improving teacher-model distillation, optionally using the real data used to train the teacher, and leave preference optimization and other post-training techniques beyond scope.Source: C Related Work · p. 20

D SiD Enhanced with Limited Real Images

The limited-real-image variant combines SiD and adversarial learning while retaining shared training across generation steps. The section describes its loss inputs, prompt sampling, loss weighting, and the use of Zero-CFG and Anti-CFG to balance diversity and alignment.

D SiD Enhanced with Limited Real Images

  • With an optimal teacher score network, all K step-specific Fisher-divergence objectives share the same optimum: matching the generated conditional distribution to the data distribution.Source: D SiD Enhanced with Limited Real Images · p. 20
  • The fake score network loss takes paired text and real noisy images together with noisy fake images generated for those prompts.The loss combines data-free SiD-LSG with an adversarial component, with λadv,ψ balancing their scales.Source: D SiD Enhanced with Limited Real Images · pp. 20–21
  • Fake-image prompts need not match prompts associated with observed real images; the experiments use 480k real text-image pairs and allow prompts not drawn from those pairs.Source: D SiD Enhanced with Limited Real Images · p. 21
  • The default weighting γt = 4 + SNRt gives low-SNR timesteps contribution to discriminator learning; γt = SNRt and γt = 10SNRt yielded comparable performance without significant differences.Source: D SiD Enhanced with Limited Real Images · p. 21
  • The generator loss uses λsid = 1 and λadv,θ = 0.001 by default, while the adversarial loss complements SiD regularization against teacher-score error and GAN instability.Source: D SiD Enhanced with Limited Real Images · p. 21
  • Zero-CFG and Anti-CFG are tuning-free strategies intended to balance diversity and text-image alignment when SiD is enhanced with limited real data.The authors motivate them after finding room to improve on SiD's existing LSG guidance strategies.Source: D SiD Enhanced with Limited Real Images · pp. 21–22

E.1 Implementation-level Optimizations

The implementation uses mixed precision to reduce training cost while retaining FP32 model weights, then combines AMP with FSDP and gradient checkpointing to scale SDXL training. These changes address memory and numerical-stability constraints while preserving performance.

E.1 Implementation-level Optimizations

  • FP16/BF16 SDXL training produced reasonable but suboptimal models, while FP32 on 80GB GPUs ran out of memory, motivating more efficient optimization.AMP alone does not resolve the SDXL FP32 out-of-memory issue; the authors add FSDP and gradient checkpointing.Source: §E.1 · p. 22
  • AMP reduces memory use and enables SiD training on SD1.5 with 1 million fake images in three to five hours rather than one day, with no noticeable performance difference from FP32.The reported training time depends on LSG settings and uses a node with eight H100 80GB GPUs.Source: §E.1 · p. 22
  • FP16 can be faster than BF16, and the authors choose AMP-FP16 despite its need for gradient scaling before backpropagation.They report no clear early-training performance difference between AMP-FP16 and AMP-BF16.Source: §E.1 · p. 22
  • AMP with FSDP and gradient checkpointing enables SDXL training under FP32 while avoiding numerical instability and achieving state-of-the-art performance.The approach also avoids regularization tricks commonly required by previous methods.Source: §E.1 · p. 22

E.2 Memory and Speed.

The resource comparisons cover one-step and four-step SiD distillation under different guidance strategies, using eight H100 GPUs with 80GB memory as a reference. Reported time and memory exclude periodic evaluation and checkpointing overhead.

E.2 Memory and Speed

  • Reported time and memory exclude periodic single-step FID and CLIP evaluations and checkpoint-saving costs, which vary with software, storage, evaluation frequency, and job restarts.These omitted costs can vary significantly.Source: §E.2 · p. 22
  • Table 3 compares one-step SiD distillation time and memory across CFG, LSG, and Zero-CFG strategies, using eight H100 80GB GPUs as a reference.The table includes hyperparameter settings alongside the resource comparison.Source: §E.2 · p. 23, Table 3 · p. 23
  • Table 4 gives the analogous time and memory comparison for four-step SiD distillation enhanced with real data, including CFG, LSG, Zero-CFG, and Anti-CFG.Zero-CFG's higher memory use primarily reflects its efficiency allowing twice the per-GPU batch size.Source: §E.2 · p. 23, Table 4 · p. 24

F Sudo Code for Custom UNet for SiDa

The custom guided SiDa U-Net exposes encoder outputs and can return encoder, decoder, or combined outputs according to a return flag. Its interface supports either tuple or structured returns.

F Sudo Code for Custom UNet for SiDa

  • The custom U-Net is introduced for guided SiDa and subclasses the pretrained conditional U-Net, with a loader that obtains the pretrained model before changing its class.The listing names the class UNet2DConditionModelWithEncoder.Source: F Sudo Code for Custom UNet for SiDa · p. 25
  • For the encoder_decoder option, the listing returns both the sample and encoder output, with tuple or structured formatting controlled by return_dict.The model's forward interface accepts sample, timestep, encoder hidden states, and optional conditioning arguments.Source: F Sudo Code for Custom UNet for SiDa · p. 25
  • The return_flag selects encoder, decoder, or encoder_decoder output, and invalid flag values trigger an assertion.The encoder-only path returns the encoder output, either as a tuple or a structured output.Source: F Sudo Code for Custom UNet for SiDa · p. 25

G Distillation of SD1.5 under Multistep SiD

The SD1.5 experiments compare one-step guidance variants and multistep matching strategies. Adversarial enhancement and Zero-CFG improve reported one-step behavior, while multistep trends vary with the matching strategy.

G Distillation of SD1.5 under Multistep SiD

  • Adding the Diffusion GAN loss transforms poor no-CFG data-free SiD performance into SiDa with approximately FID 10 and CLIP 0.30, competitive without CFG.SiD converges similarly under FP32 and AMP in this setting.Source: G Distillation of SD1.5 under… · pp. 25–26
  • With low-guidance LSG, adding the Diffusion GAN loss provides limited benefit when SiD already has low FID but low CLIP, with either FID or CLIP improving somewhat.The reported SiDa, AMP, κ = 1.5 checkpoint reaches FID 7.89, but the supplied passage does not provide its CLIP value.Source: G Distillation of SD1.5 under… · p. 26
  • Under high-guidance LSG, AMP-based SiDa improves both FID and CLIP, while FP16 can reduce FID but is limited by gradient underflow or overflow and difficult loss scaling.The authors recommend AMP, using it with FSDP when DDP memory constraints prevent its use.Source: G Distillation of SD1.5 under… · p. 26
  • SiDa-Zero-CFG significantly improves both CLIP and FID over SiD-Zero-CFG, and the tuning-free strategy offers good memory and computational efficiency, particularly for SDXL with AMP and FSDP.Conventional large CFG gives satisfactory CLIP under SiD but relatively poor FID; converting to SiDa improves both metrics.Source: G Distillation of SD1.5 under… · p. 26
  • SiDa-Zero-CFG yields FID 9.63 and CLIP 0.321 at γt = 4 + SNRt, versus FID 10.28 and CLIP 0.322 at γt = 10SNRt.The authors describe performance as robust across a wide range of γt choices.Source: G Distillation of SD1.5 under… · p. 26
  • Final-step matching tends to lower FID as steps increase but often reduces CLIP, whereas uniform-step matching improves FID gradually and initially raises CLIP before decline.Visual quality improves notably from one to two steps, especially in high-frequency details, with subtler gains from two to four.Source: G Distillation of SD1.5 under… · p. 26, Figure 10 · p. 29

H Prompts for qualitative examples

Related papers

Loading 2505.12674v1…