Source-linked AI summary
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, Robin Rombach
TL;DR
Diffusion models are slow because their iterative inference requires many evaluations. LADD addresses this by using generative features and latent-space adversarial distillation, producing SD3-Turbo, which matches its teacher’s image quality in four steps and extends to editing and inpainting.
Problem
Diffusion models require many inference evaluations, while existing adversarial distillation such as ADD is limited by fixed DINOv2 features and costly pixel-space decoding.
Method
LADD uses pretrained diffusion-model generative features, synthetic teacher data, and latent-space adversarial distillation instead of pixel-based ADD.
Results
SD3-Turbo matches its SD3 teacher’s image quality in four sampling steps and supports high-resolution multi-aspect generation, editing, and inpainting.
Takeaways & Limitations
LADD provides a simpler distillation approach for fast, high-resolution image synthesis and demonstrates versatility beyond text-to-image generation.
Takeaways & Limitations
Maintaining the teacher’s image quality in four steps comes at the expense of prompt alignment, including challenges with duplication, merging, spatial prompting, and negation.
Abstract
from arXiv · showhide
Diffusion models are the main driver of progress in image and video synthesis, but suffer from slow inference speed. Distillation methods, like the recently introduced adversarial diffusion distillation (ADD) aim to shift the model from many-shot to single-step inference, albeit at the cost of expensive and difficult optimization due to its reliance on a fixed pretrained DINOv2 discriminator. We introduce Latent Adversarial Diffusion Distillation (LADD), a novel distillation approach overcoming the limitations of ADD. In contrast to pixel-based ADD, LADD utilizes generative features from pretrained latent diffusion models. This approach simplifies training and enhances performance, enabling high-resolution multi-aspect ratio image synthesis. We apply LADD to Stable Diffusion 3 (8B) to obtain SD3-Turbo, a fast model that matches the performance of state-of-the-art text-to-image generators using only four unguided sampling steps. Moreover, we systematically investigate its scaling behavior and demonstrate LADD's effectiveness in various applications such as image editing and inpainting.
1 Introduction
Diffusion models offer strong synthesis and editing but are slow because inference typically requires many network evaluations. LADD simplifies adversarial distillation in latent space, enabling SD3-Turbo to generate high-resolution multi-aspect images in four steps while matching its teacher’s image quality.
- Diffusion inference usually requires dozens of network evaluations, limiting speed and real-time applications, especially for large models.
- LADD uses pretrained diffusion-model generative features and latent-space distillation to simplify training, support megapixel multi-aspect synthesis, and outperform prior single-step approaches.
- SD3-Turbo matches Stable Diffusion 3’s image quality in four sampling steps rather than 50, while supporting high-resolution multi-aspect text-to-image generation.
- The work also studies LADD’s scaling behavior and demonstrates applications in image editing and image inpainting.
2 Background
Diffusion models learn to denoise noise into data, but generation requires repeatedly solving a nonlinear reverse-time process. Distillation methods seek simpler trajectories that preserve the final distribution while allowing larger integration steps.
- 2.1 Diffusion Models: Diffusion models denoise Gaussian noise into data by predicting clean-image information from noisy inputs at different timesteps.
- 2.1 Diffusion Models: Generation numerically solves a reverse-time differential equation from noise to data while iteratively evaluating the learned denoiser.
- 2.1 Diffusion Models: The denoising dynamics are highly nonlinear, so sufficiently small step sizes and many expensive network evaluations are needed.
- 2.1 Diffusion Models: Distillation learns simpler, straighter differential-equation trajectories that reach the same final distribution with fewer evaluations.
3 Method
LADD distills large latent diffusion models using their own generative features, synthetic teacher data, and latent-space adversarial training. These choices avoid pixel decoding, provide noise-level-specific feedback, and support high-resolution multi-aspect synthesis.
- 3 Method: Latent diffusion models reduce training memory requirements, facilitating scaling to large models and high resolutions such as the 8B-parameter MMDiT family.
- 3 Method: LADD unifies the teacher and discriminator while replacing pixel-space ADD objectives with synthetic-data training and latent-space adversarial distillation.
- 3 Method: The teacher generates synthetic latents, and its intermediate generative features provide discriminator feedback without decoding to RGB space.
- 3 Method: Generative features provide structured global feedback at high noise levels and texture-related feedback at low noise levels, unlike fixed discriminative features.
- 3 Method: Targeted noise-level sampling lets LADD bias discriminator features toward global or local behavior.
- 3 Method: The discriminator reshapes token sequences into spatial layouts and uses 2D convolutions to accommodate varying aspect-ratio strides.
- 3 Method: Classifier-free guidance can oversaturate one-shot samples, motivating constant-CFG synthetic teacher generation for aligned training data.
- 3 Method: Synthetic data significantly outperforms real data for image-text alignment, and synthetic-data training needs only an adversarial loss.
4 Experiments
Experiments examine teacher noise, training data, latent distillation, and scaling choices for single-step synthesis. LADD benefits from synthetic data, outperforms LCM, and transfers larger-student advantages to distilled models.
- Experimental setup: Single-step evaluation starts from pure noise and uses an approximately 2B-parameter MMDiT with depth 24 by default.The reported qualitative outputs in this section are lower quality than those of the final larger model.
- Teacher noise distribution: π(t; m = 1, s = 1) balances global coherence against texture and fine-detail preservation when sampling teacher noise levels.Lower noise bias can lose global coherence, while excessively high noise harms textures and fine details.
- Synthetic data: Synthetic-data training significantly outperforms real-data training for image-text alignment, making the additional distillation loss unnecessary.The distillation loss improves details and alignment for real-data training but offers no advantage with synthetic data.
- Latent distillation approaches: LADD outperforms LCM by a large margin in single-step distillation, while LCM training shows substantially higher volatility across hyperparameters, seeds, and training iterations.LADD uses one training run and the final checkpoint, whereas LCM uses a broad hyperparameter search and selects its best checkpoint.
- Scaling behavior: Student model size has a greater effect on performance than data quality or teacher model size in the scaling study.The study varies student, teacher, and data-generator depth one dimension at a time while keeping the others at depth 24.
- Scaling behavior: Larger student-model advantages transfer to distilled versions, while teacher-model and data-quality benefits plateau beyond certain thresholds.This supports prioritizing larger students and permitting smaller teachers when memory is constrained.
5 Comparison to State-of-the-Art
The paper evaluates SD3-Turbo across text-to-image, editing, and inpainting tasks, showing strong quality with one to four sampling steps. LADD also supports DPO refinement and high-resolution multi-aspect generation.
- Text-to-Image Synthesis: Applying pretrained DPO-LoRA matrices after LADD adds detail, fixes duplicate objects, improves hands, and increases overall visual appeal.
- Text-to-Image Synthesis: Single-step user studies show SD3-Turbo outperforming established baselines in both image quality and prompt alignment, with four steps improving results further.
- Text-to-Image Synthesis: SD3-Turbo matches SD3 image quality in four instead of 50 sampling steps, while slightly reducing prompt alignment but outperforming strong baselines such as Midjourney v6.
- Image-to-Image Synthesis: For instruction-based editing, SD3-edit Turbo matches its teacher’s performance in one step on the CLIP image-similarity versus edit-direction trade-off.
- Image-to-Image Synthesis: The distilled editing model is especially effective for style edits and object swaps, integrating edited objects into scenes; its speed comes with reduced guidance controllability.
- Image-to-Image Synthesis: For inpainting, the distilled model performs on par with its teacher in one step, while LaMa achieves better LPIPS but worse FID and qualitative results for large non-homogeneous masks.
6 Limitations
The paper identifies prompt-alignment and controllability limitations in distilled models, including failures with spatial relations, negation, duplicated entities, and large image edits.
- SD3-Turbo can retain teacher image quality while losing prompt alignment, causing entity merging, weaker detailed spatial accuracy, and missed negations on some seeds.
- The speed–quality trade-off is accompanied by difficulties in object duplication, entity merging, fine-grained spatial prompting, and negation.
- SD3-edit Turbo lacks adjustable image and text guidance strengths, limiting control over the editing trade-off available in comparative methods.
- In some cases, the model adheres too closely to the input, making large image changes difficult.
Appendix A Details on Human Preference Assessment
The human evaluation compares models through pairwise judgments on prompts from the PartiPrompts benchmark, focusing on image quality and prompt fidelity.
- The study uses human evaluation because standard generative-model metrics insufficiently measure aesthetics and scene composition.
- Samples are generated from PartiPrompts, excluding its Basic category to focus on more complex prompts.
- Each comparison presents samples from two models for the same prompt, requiring evaluators to choose one in a 1v1 judgment.
- Prompt-following judgments display the prompt above both images and ask which image most faithfully represents the text.
A.1 List of Prompts used for Human Evaluation
The human-evaluation prompt set spans abstract concepts, rare words, detailed scenes, object relations, artistic styles, and everyday compositions.
- Prompts include abstract or unusual language, such as four-dimensional space-time and a highly technical medical term.
- Many prompts test detailed spatial and relational descriptions involving animals, objects, positions, and interactions.
- The set includes stylistic and compositional prompts covering paintings, drawings, food arrangements, architecture, and imaginative scenes.