Source-linked AI summary
LCM-LoRA: A Universal Stable-Diffusion Acceleration Module
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, Hang Zhao
TL;DR
Slow reverse sampling and the extra training needed for specialized datasets motivate a training-free acceleration method for Stable-Diffusion models. The paper introduces LoRA-based LCM distillation and combines its learned acceleration parameters with style LoRAs, extending efficient inference to larger models and customized styles without further training.
Problem
Slow latent-diffusion sampling and the additional training required for specialized datasets limit fast deployment across custom image-generation domains.
Method
The paper applies LoRA distillation to LCMs and linearly combines the resulting acceleration parameters with customized style LoRAs.
Results
LCM-LoRA extends LCM distillation to SDXL and SSD-1B while reducing trainable parameters and enabling specific-style generation with minimal sampling steps without further training.
Takeaways & Limitations
LCM-LoRA serves as a universal training-free acceleration module for fine-tuned Stable-Diffusion models and SD LoRAs.
Abstract
from arXiv · showhide
Latent Consistency Models (LCMs) have achieved impressive performance in accelerating text-to-image generative tasks, producing high-quality images with minimal inference steps. LCMs are distilled from pre-trained latent diffusion models (LDMs), requiring only ~32 A100 GPU training hours. This report further extends LCMs' potential in two aspects: First, by applying LoRA distillation to Stable-Diffusion models including SD-V1.5, SSD-1B, and SDXL, we have expanded LCM's scope to larger models with significantly less memory consumption, achieving superior image generation quality. Second, we identify the LoRA parameters obtained through LCM distillation as a universal Stable-Diffusion acceleration module, named LCM-LoRA. LCM-LoRA can be directly plugged into various Stable-Diffusion fine-tuned models or LoRAs without training, thus representing a universally applicable accelerator for diverse image generation tasks. Compared with previous numerical PF-ODE solvers such as DDIM, DPM-Solver, LCM-LoRA can be viewed as a plug-in neural PF-ODE solver that possesses strong generalization abilities. Project page: https://github.com/luosiallen/latent-consistency-model.
1 INTRODUCTION
Latent diffusion models offer high-quality image generation but remain slow, while LCMs address this bottleneck through efficient PF-ODE prediction and minimal-step sampling. LCM-LoRA extends fast, training-free inference to fine-tuned Stable-Diffusion models and LoRAs.
- Slow reverse sampling in latent diffusion models limits real-time image generation, especially on standard consumer GPUs.Existing acceleration approaches use numerical ODE solvers or distillation.
- LCMs predict PF-ODE solutions in latent space instead of relying on iterative numerical ODE solvers.This design enables efficient synthesis while avoiding iterative solver steps.
- 1 to 4 inference steps enable LCMs to synthesize high-resolution images efficiently.LCM distillation requires merely 32 A100 training hours for minimal-step inference.
- Specialized datasets traditionally require additional distillation or fine-tuning, complicating rapid deployment across custom domains.The passage frames training-free custom-dataset inference as an open question.
- LCM-LoRA is a universal training-free module that plugs into fine-tuned Stable-Diffusion models or LoRAs for fast, minimal-step inference.It is presented as a neural-network PF-ODE solver with robust generalization across such models and LoRAs.
2 RELATED WORK
Related work establishes consistency models, parameter-efficient LoRA adaptation, and task arithmetic as foundations for efficient generation and weight-space customization. These approaches motivate LCM-LoRA’s combination of acceleration and specialized adaptation.
- Consistency Models: Consistency models improve sampling efficiency by mapping points along ODE trajectories to their origins for one-step generation.Prior demonstrations targeted ImageNet 64x64 and LSUN 256x256 image generation.
- Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning limits retraining to a small parameter set for task customization.PEFT reduces computational load and storage demands.
- Parameter-Efficient Fine-Tuning: LoRA fine-tunes models by learning low-rank matrices while keeping the bulk of pretrained weights unchanged.This reduces the parameters requiring retraining, lowering computational and storage demands.
- Task Arithmetic in Pretrained Models: Task arithmetic edits pretrained models in weight space by applying fine-tuned weights from different tasks.The approach can improve task performance or induce forgetting by negating task weights.
3 LCM-LORA
LCM-LoRA applies LoRA distillation to latent consistency models, reducing trainable parameters and memory requirements while extending distillation to larger Stable-Diffusion models. Its learned acceleration parameters can be linearly combined with style LoRAs without further training.
- 3.1 LORA DISTILLATION FOR LCM: LCM distillation uses guided PF-ODE distillation in a pretrained autoencoder’s latent space, with skipping steps to accelerate convergence.Algorithm 1 specifies sampled latent data, noise schedules, guidance, an ODE solver, and EMA updates.
- 3.1 LORA DISTILLATION FOR LCM: LoRA distillation updates only low-rank matrices A and B while keeping the pretrained weight matrix W0 fixed.The forward pass adds the low-rank update BAx to the base output W0x.
- 3.1 LORA DISTILLATION FOR LCM: Significantly fewer trainable parameters reduce memory requirements during LCM distillation.Table 3.1 compares full-model parameters with LoRA trainable parameters for the evaluated Stable-Diffusion models.
- 3.1 LORA DISTILLATION FOR LCM: LCM distillation extends from SD-V1.5 and SD-V2.1 to larger models including SDXL and SSD-1B.Experiments report that the LCD paradigm adapts well to these larger models.
- 3.2 LCM-LORA AS UNIVERSAL ACCELERATIION MODULE: LCM-LoRA parameters act as an acceleration vector that can be combined with style LoRA parameters without further training.The resulting model supports specific-style image generation with minimal sampling steps.
4 CONCLUSION
The paper presents LCM-LoRA as a universal, training-free Stable-Diffusion acceleration module. It functions as a neural PF-ODE solver for fast inference across fine-tuned models and LoRAs, with experiments reporting strong generalization and superiority.
- LCM-LoRA is an independent neural-network solver module for predicting PF-ODE solutions during Stable-Diffusion inference.It enables fast inference with minimal steps on various fine-tuned Stable-Diffusion models and LoRAs.
- Extensive text-to-image experiments demonstrate strong generalization capabilities and superiority for LCM-LoRA.
5 CONTRIBUTION & ACKNOWLEDGEMENT
The work builds on prior LCM research and reports contributions spanning implementation, training, and library integration.
- Acknowledgement: The work builds upon Simian Luo and Yiqin Tan’s Latent Consistency Models.
- Acknowledgement: Simian Luo wrote the original LCM-SDXL distillation code and primarily completed the technical report with Yiqin Tan.
- Acknowledgement: Yiqin Tan discovered the arithmetic property of LCM parameters.
- Acknowledgement: Suraj Patil completed LCM-LoRA training, discovered its generalization abilities, and conducted most of the training.
- Acknowledgement: Suraj Patil and Daniel Gu refactored the LCM-SDXL codebase, improved training efficiency, and integrated it into Diffusers.