Source-linked AI summary
SDXL-Lightning: Progressive Adversarial Diffusion Distillation
Shanchuan Lin, Anran Wang, Xiao Yang
TL;DR
Diffusion models provide high-quality generation but are computationally slow, and existing distillation methods remain inadequate at one or two inference steps. SDXL-Lightning combines progressive and adversarial distillation, producing state-of-the-art one-step and few-step SDXL generation at 1024px resolution. The models are released as LoRA and full UNet weights.
Problem
Diffusion models are slow because of iterative generation, while existing distillation methods do not provide production-acceptable quality at one or two inference steps.
Method
The method combines progressive and adversarial distillation, using latent-space adversarial objectives and a pretrained diffusion UNet encoder as discriminator backbone.
Results
The method produces state-of-the-art SDXL models supporting one-step and few-step text-to-image generation at 1024px resolution.
Takeaways & Limitations
SDXL-Lightning models are open-sourced as LoRA and full UNet weights for one-step and few-step generation.
Takeaways & Limitations
Separate checkpoints are produced for each inference-step setting, although LoRA modules can mitigate switching when flexible step counts are needed.
Abstract
from arXiv · showhide
We propose a diffusion distillation method that achieves new state-of-the-art in one-step/few-step 1024px text-to-image generation based on SDXL. Our method combines progressive and adversarial distillation to achieve a balance between quality and mode coverage. In this paper, we discuss the theoretical analysis, discriminator design, model formulation, and training techniques. We open-source our distilled SDXL-Lightning models both as LoRA and full UNet weights.
1. Introduction
Diffusion models deliver strong results but are slow because generation follows a complex probability flow in many small steps. SDXL-Lightning combines progressive and adversarial distillation to improve one-step and few-step 1024px generation while balancing quality and mode coverage.
- Diffusion models achieve state-of-the-art results across text-to-image, text-to-video, and image-to-video applications, but their iterative generation is slow and computationally expensive.
- Because the probability flow is complex and curved, high-quality generation typically requires more than 50 inference steps.
- Existing ODE-solver and flow-straightening approaches generally still require more than 20 inference steps.
- Model distillation reduces generation to under 10 steps, but existing methods remain not production-acceptable at one or two steps.
- Progressive adversarial distillation combines probability-flow and mode-coverage preservation with adversarial training to mitigate blurry low-step results.
- The method uses a pretrained diffusion UNet encoder as a latent-space discriminator and introduces adversarial objectives and training techniques for quality and mode-coverage trade-offs.
- The resulting SDXL-Lightning models support state-of-the-art one-step and few-step text-to-image generation at 1024px resolution.
2. Background
Diffusion models generate samples by following a learned flow from noise toward data, while latent diffusion performs this process in a compact latent space. Progressive and adversarial distillation reduce inference steps, but existing approaches retain quality, compatibility, or scalability limitations.
- 2.1. Diffusion Model: The forward diffusion process transforms data samples into Gaussian noise, and the learned network predicts the flow gradient used for generation.Generation moves a noisy sample through the learned gradient field toward the data distribution.
- 2.2. Latent Diffusion Model: Latent diffusion trains diffusion models on compact VAE latents and decodes generated latent codes into final samples, enabling efficient high-resolution generation.SDXL generates 1024px images from a 128px latent space.
- 2.3. Progressive Distillation: Progressive distillation trains a student to predict a direction toward a farther flow location, repeating the process until few-step generation is possible.A frozen teacher performs multiple inference steps, and the student learns to jump directly to the resulting location; the original objective uses MSE loss.
- 2.4. Adversarial Distillation: Adversarial diffusion distillation uses a discriminator to distinguish generated from real samples, but prior designs can require costly pixel-space training and alter multi-step behavior.SDXL-Turbo uses a pretrained DINOv2 image encoder, limiting it to t = 0 and making high-resolution distillation impractical; it supports up to 512px resolution.
- 2.4. Adversarial Distillation: The proposed approach uses the diffusion U-Net encoder as a latent-space discriminator, supporting discrimination at all timesteps and control over quality–mode-coverage trade-offs.The method combines progressive and adversarial distillation and introduces adversarial objectives and training-stabilization techniques.
- 2.5. Other Distillation Methods: Existing alternatives retain limitations: consistency methods reduce plugin compatibility, rectified flow harms arbitrary-noise editing, and VSD adds a dynamic negative-distribution model.The paper reports preliminary experiments in which its method achieves better quality than VSD, while releasing distilled models as LoRAs and full models.
3. Method
The method analyzes why few-step students struggle to match complex teacher flows and uses conditional adversarial discrimination to preserve the probability flow and mode coverage during distillation.
- Theoretical Analysis: The learned probability flow depends on the dataset, forward function, loss function, and model capacity.Finite data makes the underlying distribution ambiguous, while infinite-capacity maximum likelihood can overfit observed samples.
- Why Distillation with MSE Fails: Few-step students have less capacity than multi-step models to approximate the teacher’s complex distribution, making exact output matching difficult.Sharp teacher changes under small noise perturbations contrast with smoother distilled latent traversals.
- Why Distillation with MSE Fails: MSE, L1, and perceptual losses produce undesirable or blurry results because the student cannot perfectly match the teacher’s outputs.Adversarial objectives are effective in mitigating this issue.
- Adversarial Objective: The discriminator distinguishes teacher-generated from student-predicted future flow locations conditioned on the current state and text condition.Training alternates discriminator and student updates using non-saturated adversarial loss.
- Adversarial Objective: Conditioning the discriminator on the current state preserves the teacher’s deterministic probability flow, requiring the student to follow that flow to fool it.The formulation preserves probability flow and mode coverage when used for distillation.
3.3. Discriminator Design
The discriminator uses a pretrained SDXL UNet encoder in latent space, while staged conditional and unconditional objectives trade off flow preservation, sharpness, and semantic correctness.
- Discriminator Design: The discriminator reuses the pretrained SDXL UNet encoder and midblock, processing current and future latents independently before feature concatenation and prediction.The prediction head reduces spatial dimensions with convolution, group normalization, and SiLU layers.
- Discriminator Design: The discriminator backbone is fully trainable from pretrained initialization and remains stable without expansive R1 regularization or L2 attention.Additional stabilization techniques are discussed separately.
- Failure Mode: Adversarial objectives encourage sharp, flow-preserving predictions but expose capacity limits as “Janus” artifacts rather than blur.These artifacts arise when students cannot match the teacher’s sudden layout changes.
- Relax the Mode Coverage: Because semantic correctness is preferred over mode coverage, the method relaxes flow preservation by finetuning without conditioning on the current latent.The unconditional objective focuses on per-sample quality.
- Relax the Mode Coverage: The unconditional finetuning removes “Janus” artifacts while preserving the original flow to a great extent in practice.Each progressive stage applies conditional training first, followed by unconditional finetuning.
3.5. Fix the Schedule
The schedule correction addresses a mismatch between training and inference noise, which is especially detrimental for few-step generation, while retaining compatibility with existing sampling procedures.
- Fix the Schedule: SDXL’s diffusion schedule does not reach pure noise at t = T during training although inference supplies pure noise, creating a train–inference discrepancy.The discrepancy is less apparent with many steps but particularly harmful for few-step generation.
- Fix the Schedule: Hard-swapping pure noise ϵ into the model input at t = T trains the model to expect inference-time noise without changing the old inference schedule.The approach requires minimal sampling changes and is also used by SDXL-Turbo.
3.6. Distillation Procedure
Distillation first compresses 128 steps to 32 with MSE, then uses adversarial stages through one step, combining conditional flow preservation with unconditional quality finetuning and LoRA/full-UNet training.
- Distillation Procedure: The initial 128 →32 stage uses MSE loss and classifier-free guidance with guidance scale 6 and no negative prompts.MSE is considered sufficient for this early stage.
- Distillation Procedure: Adversarial distillation then reduces the step count through 32 →8 →4 →2 →1.Each stage trains conditionally to preserve probability flow, then unconditionally to relax mode coverage.
- Model Training: Each stage begins with LoRA training, after which the LoRA weights are merged and the whole UNet is trained further with the unconditional objective.Whole-UNet finetuning achieves better performance, while LoRA supports use with other base models.
- Data: The models are distilled on filtered LAION and COYO images above 1024px, with aesthetic, sharpness, and prompt-cleaning filters.Although training uses square images, the method generalizes well to other aspect ratios at inference.
- Training Configuration: Training uses batch size 512 across 64 A100 80G GPUs, with stage-specific learning rates and mixed-precision and memory-efficiency techniques.Adversarial stages use learning rates of 1e-6 with LoRA and 5e-7 without LoRA for both student and discriminator.
3.7. Stable Training Techniques
The section stabilizes one-step and two-step adversarial distillation by training discrimination across multiple timesteps and switching one-step prediction from ε to x0.
- Train Discriminator at Multiple Timesteps: Training the one-step discriminator directly at t = 0 fails to critique image structure, producing bad shapes and possible divergence.The diffusion encoder emphasizes high-frequency details at lower timesteps and low-frequency structures at higher timesteps.
- Train Discriminator at Multiple Timesteps: Adding noise to teacher and student predictions at timesteps {10, 250, 500, 750} lets the discriminator critique both details and structures.The method samples these timesteps uniformly during stabilization.
- Train Discriminator at Multiple Timesteps: 5:1:1:1 timestep weighting after stabilization further improves details and removes noisy artifacts.The weighting is changed only after the model has been trained stably.
- Switch to x0 Prediction: The one-step model switches from ε-prediction to x0-prediction because ε-prediction tends to produce noise artifacts from numerical instability.MSE gradually guides the copied network toward x0-prediction before adversarial training.
- Switch to x0 Prediction: MSE conversion can produce blurry results, which the subsequent adversarial objectives are intended to fix.After conversion, the student uses x0-prediction while the teacher remains in ε-prediction, so one-step LoRA is not provided.
4. Evaluation
The evaluation compares SDXL-Lightning with open-source distillation models using qualitative, specification, and quantitative analyses. The results report stronger visual quality and high-resolution details, while maintaining similar diversity and text alignment.
- Qualitative Comparison: Qualitatively, SDXL-Lightning is substantially better in overall quality, details, style preservation, and layout preservation than SDXL-Turbo and LCM.The comparison uses fully trained models and controls the initial noise for each prompt except SDXL-Turbo.
- Qualitative Comparison: SDXL-Lightning’s 4-step and 8-step models can often outperform original SDXL sampled for 32 steps.The paper attributes this to progressive distillation starting from 128 steps.
- Qualitative Comparison: Fully trained models have better structures and details than LoRA models, with the difference more observable for 2-step models than 8-step models.The paper still characterizes the LoRA models as very high quality and provides no 1-step LoRA.
- Quantitative Comparison: FID-Patch shows significantly better high-resolution details, while FID-Whole and CLIP indicate similar diversity, quality, and text alignment compared with other methods.FID-Whole measures high-level diversity and quality, FID-Patch measures high-resolution details, and CLIP measures text alignment.
5. Ablation
The distilled LoRA transfers across cartoon, anime, and realistic base models while preserving their style and layout, supports varied aspect ratios, and remains compatible with ControlNet.
- 5.1. Apply LoRA on Other Base Models: The distillation LoRA preserves the style and layout of cartoon, anime, and realistic third-party base models.
- 5.2. Inference with Different Aspect Ratios: The models generate different resolutions and aspect ratios despite training only on square images.Figure 6 shows 1:2 aspect-ratio examples at 720×1440px from the 4-step model.
- 5.3. Compatibility with ControlNet: The models are compatible with ControlNet conditioning on canny edges and depth.
6. Limitation
The method has practical limitations involving checkpoint management and the inherited UNet architecture.
- Separate checkpoints are produced for each inference-step setting, although LoRA modules can mitigate switching when flexible step counts are needed.
- The student retains the teacher’s UNet architecture, which the authors consider non-optimal for one-step generation.Feature inspection indicates that most generation occurs in the decoder, motivating future architectural work.
7. Conclusion
The paper presents SDXL-Lightning, a progressive adversarial diffusion-distillation method for state-of-the-art one-step and few-step text-to-image generation. Its evaluation reports superior image quality versus prior work, and the models are open-sourced.
- SDXL-Lightning uses progressive adversarial diffusion distillation to produce state-of-the-art one-step and few-step text-to-image models.
- The evaluation finds that the models produce superior image quality compared with prior works.
- The authors open-source the models to advance research in generative AI.