PaperlayerView exact revision on arXiv

Source-linked AI summary

TLCM: Training-efficient Latent Consistency Model for Image Generation with 2-8 Steps

Qingsong Xie, Zhenyi Liao, Zhijie Deng, Chen chen, Haonan Lu

arXiv:2406.05768v6cs.CVcs.AI

TL;DR

Latent diffusion models can produce high-quality images but require many model evaluations, while prior acceleration methods face quality and training-data challenges. TLCM uses data-free distillation and additional techniques to support 2–8-step generation; its three-step SDXL model reports strong MSCOCO-2017 5K scores against accelerated baselines.

  • Problem

    Latent diffusion models require tens to hundreds of model evaluations per sample, while existing acceleration methods can involve quality degradation and large data and training requirements.

    Source: §1 · p. 2

  • Method

    TLCM combines data-free multistep latent consistency distillation and latent consistency distillation with preference learning, distribution matching, and adversarial learning.

    Source: §1 · pp. 2–3

  • Results

    33.68 CLIP Score (CS) and 5.97 Aesthetic Score (AS) on MSCOCO-2017 5K at 3 steps substantially surpass 4-step LCM, 8-step SDXL-Lightning, and 8-step HyperSD, and are comparable to 25-step DDIM.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TLCM supports sampling from 2 to 8 steps and is demonstrated for image stylization, controllable generation, and Chinese-to-image generation.

    Source: §1 · p. 3, §6 · p. 11

Abstract

from arXiv · show

Distilling latent diffusion models (LDMs) into ones that are fast to sample from is attracting growing research interest. However, the majority of existing methods face two critical challenges: (1) They hinge on long training using a huge volume of real data. (2) They routinely lead to quality degradation for generation, especially in text-image alignment. This paper proposes a novel training-efficient Latent Consistency Model (TLCM) to overcome these challenges. Our method first accelerates LDMs via data-free multistep latent consistency distillation (MLCD), and then data-free latent consistency distillation is proposed to efficiently guarantee the inter-segment consistency in MLCD. Furthermore, we introduce bags of techniques, e.g., distribution matching, adversarial learning, and preference learning, to enhance TLCM's performance at few-step inference without any real data. TLCM demonstrates a high level of flexibility by enabling adjustment of sampling steps within the range of 2 to 8 while still producing competitive outputs compared to full-step approaches. Notably, TLCM enjoys the data-free merit by employing synthetic data from the teacher for distillation. With just 70 training hours on an A100 GPU, a 3-step TLCM distilled from SDXL achieves an impressive CLIP Score of 33.68 and an Aesthetic Score of 5.97 on the MSCOCO-2017 5K benchmark, surpassing various accelerated models and even outperforming the teacher model in human preference metrics. We also demonstrate the versatility of TLCMs in applications including image style transfer, controllable generation, and Chinese-to-image generation.

1 INTRODUCTION

Latent diffusion models generate high-quality images but require many model evaluations, and prior acceleration methods face quality or training-cost challenges. TLCM targets competitive 2–8-step generation with data-free distillation and reports strong three-step benchmark results.

1 INTRODUCTION

  • LDM inference requires tens to hundreds of model evaluations per sample, creating high cost and a poor user experience.Source: §1 · p. 2
  • Prior acceleration methods trade off sampling flexibility, image quality, text-image alignment, or reliance on large volumes of data and long training.Progressive distillation requires separate models for different step counts; one-step methods weaken the benefits of additional steps, while LCM and HyperSD have reported quality or alignment issues.Source: §1 · p. 2
  • TLCM combines data-free multistep latent consistency distillation to reduce steps with latent consistency distillation for global consistency, plus preference, distribution-matching, and adversarial techniques.It targets sampling in 2–8 steps with training data and computation kept low.Source: §1 · pp. 2–3
  • 33.68 CLIP Score (CS) and 5.97 Aesthetic Score (AS) at 3 steps substantially surpass 4-step LCM, 8-step SDXL-Lightning, and 8-step HyperSD, and are comparable to 25-step DDIM.The model is trained in 70 A100 hours without real data.Source: §1 · p. 3
  • TLCM is also demonstrated for image stylization, controllable generation, and Chinese-to-image generation.Source: §1 · p. 3

2 RELATED WORKS

Prior diffusion acceleration methods reduce sampling effort, but reported limitations include degraded few-step image quality, constrained multi-step use, and resource-intensive training. The related work also describes preference metrics and reward learning as ways to tune text-to-image models toward human preference.

2 RELATED WORKS

  • Slow inference involving tens to hundreds of model evaluations is a primary challenge to the practical adoption of diffusion models.Source: §2 · p. 3
  • Progressive distillation and classifier-aware distillation can produce blurry samples within four steps, while consistency-model approaches have not verified effectiveness on large-scale text-to-image generation.Source: §2 · p. 3
  • LCM can synthesize blurry four-step images, and several one-step text-to-image methods cannot extend their samplers to multiple steps for improved quality.Source: §2 · p. 3
  • SDXL-Turbo, SDXL-Lightning, and HyperSD rely on large high-quality text-image datasets and expensive online training; TLCM is described as using no images in training.Source: §2 · p. 3
  • ImageReward and Aesthetic Score evaluate human preference, while TLCM incorporates reward learning to guide model tuning toward that preference.Source: §2 · pp. 3–4

3 PRELIMINARY

The preliminaries introduce diffusion sampling through forward and probability-flow processes, then describe consistency models and their multistep extension. These foundations frame TLCM’s training overview as staged distillation followed by techniques to improve performance.

3.1 DIFFUSION MODELS

  • A forward SDE progressively distorts clean data toward Gaussian noise, with drift and diffusion coefficients defining its dynamics.Source: §3.1 · p. 4
  • The probability-flow ODE shares the forward SDE’s marginal distributions; estimating the score enables sampling by solving the ODE with a solver such as DDIM.The sampling iterates over predefined discretization timesteps.Source: §3.1 · p. 4

3.2 CONSISTENCY MODELS

  • A consistency model learns to map points on a diffusion ODE trajectory directly to its endpoint, with its parameterization designed to satisfy the boundary condition fθ(x0, 0) = x0.Source: §3.2 · p. 4
  • Consistency distillation learns from a pretrained diffusion model by matching predictions at states linked by an ODE solver, using an exponential moving average of the model parameters.Source: §3.2 · p. 4
  • Multistep consistency models divide the timestep range into segments and perform consistency distillation within each segment; TLCM’s overview adds a later global-consistency stage and data-free performance techniques.The overview identifies multistep latent consistency distillation as the first stage, followed by improved latent consistency distillation, MPS optimization, distribution matching, and adversarial learning.Source: §3.2 · p. 4, Figure 2 · p. 5

4 METHODOLOGY

TLCM uses data-free, staged latent consistency distillation to target 2–8-step LDM inference, then adds techniques to improve performance without real training data.

4.1 DATA-FREE MULTISTEP LATENT CONSISTENCY DISTILLATION

  • Splitting the ODE trajectory into segments and enforcing consistency within each lets MLCD distill a pre-trained LDM while using a skip of 20 and removing EMA to save memory.The authors motivate segmentation as a response to the difficulty of mapping arbitrary trajectory states directly to the endpoint.Source: §4.1 · p. 5
  • Teacher-generated training latents come from denoising Gaussian noise; multistep denoising uses more iterations for smaller timesteps to obtain cleaner states.A single denoising step was empirically insufficient because the resulting latent quality was poor.Source: §4.1 · pp. 5–6

4.2 IMPROVED DATA-FREE LATENT CONSISTENCY DISTILLATION

  • Stage-two data-free LCD enforces consistency across predefined timesteps, using synthetic clean samples generated by a few-step TLCM sampler instead of real data.The method avoids enforcing consistency over the entire timestep range, which the authors describe as difficult to learn.Source: §4.2 · p. 6
  • A multistep DDIM solver with classifier-free guidance estimates the next latent across each T/M interval, and the improved stage converges in 2K training iterations.The interval is uniformly divided into p parts; the multistep solver addresses discretization error associated with a one-step solver at large skips.Source: §4.2 · p. 6

4.3 INCORPORATING BAG OF TECHNIQUES INTO TLCM IN DATA-FREE MANNER

  • Latent LPIPS replaces direct latent-space MSE consistency with perceptual features from a VGG-based model trained from scratch on BAPPS.The model uses four input channels and removes three max-pooling layers to account for the LDM latent space being 8× downsampled.Source: §4.3 · p. 6
  • Multi-dimensional preference optimization raises scores for relevant text conditions above a margin and lowers scores for irrelevant conditions; ImageReward and Aesthetic Score are excluded due to observed overexposure and oversaturation.Gradients are back-propagated from the scorer to the model parameters.Source: §4.3 · pp. 6–7
  • Data-free distribution matching optimizes TLCM at sparse-step inference using noisy latents, with real and fake score models initialized from SDXL.The fake score model is fine-tuned on synthetic samples using a diffusion noise-prediction loss.Source: §4.3 · p. 7
  • Adversarial learning adds a GAN loss to enforce distribution consistency, using synthetic latents rather than real data.The paper motivates this choice by noting that MSE can imperfectly constrain consistency for high-resolution generation with complex distributions.Source: §4.3 · p. 7

5 EXPERIMENTS

Experiments evaluate TLCM’s training setup, benchmark performance, and component ablations. The results show strong few-step comparisons and identify contributions from multistep distillation, consistency refinement, and additional training losses.

5.1 IMPLEMENTATION DETAILS

  • Evaluation uses zero-shot MSCOCO-2017 5K validation for all methods except HyperSD, which trained on MSCOCO-2017; SDXL-Turbo alone outputs 512-pixel images, others 1024; FID is reference only.Training prompts come from the LAION-Aesthetics-6+ subset of LAION-5B, and the whole procedure uses 4 A100 GPUs.Source: §5.2 · p. 7, §5.1 · p. 7

5.2 MAIN RESULTS

  • On MSCOCO-2017 5K validation, 3-step TLCM has superior CS, AS, and IR to 4–8-step LCM; versus the 25-step teacher, it outperforms on AS and IR and achieves comparable CS.It also has much higher CS than 4–8-step HyperSD; the authors attribute HyperSD’s higher IR to using the IR model to optimize it.Source: §5.2 · pp. 7–8, Table 1 · p. 8
  • SDXL-Turbo’s CS and IR drop as sampling steps increase, whereas TLCM improves at least one metric with additional steps.The authors frame TLCM’s step flexibility as useful when image quality is prioritized given available compute.Source: §5.2 · p. 8
  • TLCM’s images show better quality and semantic consistency on challenging prompts, leading to greater human preference under the same conditions.The comparison is presented in Figure 3.Source: §5.2 · p. 8, Figure 3 · p. 9

5.3 ABLATION STUDY

  • Data-free MLCD outperforms single-step data-free LCD, and multistep denoising further improves MLCD by producing better initial latents.The ablation uses a 4-step sampler and SDXL backbone.Source: §5.3 · p. 8, Table 2 · p. 10
  • Using latent LPIPS rather than raw latent space improves all reported metrics, consistent with its perceptual-consistency objective.The comparison concerns the L-LPIPS variant in Table 2.Source: §5.3 · p. 10, Table 2 · p. 10
  • Stage-2 LCD improves inter-segment consistency and sparse-timestep iLCD improves further; MHP lifts CS and IR, data-free DM lifts all metrics, and GAN improves CS, AS, and IR.These component ablations use the 4-step SDXL setup.Source: §5.3 · p. 10, Table 2 · p. 10
  • Stage-2 teacher sampling steps from 1 to 4 consistently improve performance, supporting multistep denoising to reduce discretization error at large skips.Table 3 examines the teacher’s sampling steps for data-free LCD in stage 2.Source: §5.3 · p. 10, Table 3 · p. 10

6 CONCLUSION

TLCM accelerates text-to-image latent diffusion models with 2–8 sampling steps and 70 A100 training hours, without text-image paired data. The conclusion reports better image quality than baseline methods and compatibility with several generation applications.

6 CONCLUSION

  • TLCM accelerates text-to-image latent diffusion models using only 70 A100 hours and no text-image paired data.The conclusion characterizes the approach as training-efficient.Source: §6 · p. 10
  • TLCM generates high-quality images with 2–8 sampling steps and achieves better image quality than baseline methods.Source: §6 · pp. 10–11
  • The method is compatible with image style transfer, controllable generation, and Chinese-to-image generation.Source: §6 · p. 11

A.1 ALGORITHMS

The appendix gives two data-free distillation procedures: multistep latent consistency distillation and stage-2 latent consistency distillation. Both use teacher denoising to construct student consistency targets.

A.1 ALGORITHMS

  • Algorithm 1 initializes the latent from Gaussian noise and uses the teacher model to iteratively denoise toward a target latent before applying MLCD.The iteration count and time interval depend on the segment index and segment number.Source: §A.1 · p. 14
  • Algorithm 2 first denoises from Gaussian noise with the student for q steps, then samples a timestep milestone and forward-diffuses the resulting latent.Source: §A.1 · p. 14
  • For stage-2 LCD, the teacher takes p denoising steps within the selected segment to estimate the target, after which the algorithm performs LCD.Source: §A.1 · p. 14

A.2.1 ACCELERATION OF IMAGE STYLE TRANSFER

The TLCM LoRA is compatible with the image style-transfer pipeline, and the paper presents examples generated with two sampling steps.

A.2.1 ACCELERATION OF IMAGE STYLE TRANSFER

  • The TLCM LoRA is compatible with the image style-transfer pipeline.Source: §A.2.1 · p. 14
  • The demonstrated style-transfer examples use 2-step sampling.Source: §A.2.1 · p. 14

A.2.2 ACCELERATION OF CONTROLLABLE GENERATION

TLCM LoRA is presented as compatible with controllable generation, image style transfer, and Chinese-to-image generation at few sampling steps. The examples report satisfactory ControlNet results and images aligned with Chinese semantic meaning.

A.2.2 ACCELERATION OF CONTROLLABLE GENERATION

A.2.3 ACCELERATION OF CHINESE-TO-IMAGE GENERATION

  • Two-step TLCM LoRA supports image style transfer and produces highly stylized images.Source: Figure 4 · p. 15
  • TLCM accelerates Chinese-to-image generation, with three-step examples aligned to Chinese semantics in general and cultural settings.Source: §A.2.3 · p. 14, Figure 6 · p. 16

Related papers

Loading 2406.05768v6…