PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models

Yefei He, Jing Liu, Weijia Wu, Hong Zhou, Bohan Zhuang

arXiv:2310.03270v4cs.CV

TL;DR

Diffusion models remain computationally costly, and low-bit quantization can trade generation quality against efficiency. EfficientDM uses data-free, parameter-efficient fine-tuning to target QAT-level performance with PTQ-like efficiency. The authors report strong low-bit results, while noting that gradient-based optimization demands more GPU memory than PTQ, especially for massive models.

  • Problem

    Diffusion models have substantial computational costs and latency, while PTQ can lose performance at low bit-width and QAT requires more time, compute, and data.

    Source: §1 · pp. 1–2

  • Method

    EfficientDM fine-tunes low-bit models without training data, using quantization-aware low-rank adapters to distill full-precision noise estimates and improve quantization.

    Source: §1 · p. 2

  • Results

    EfficientDM is reported to achieve QAT-level performance with PTQ-level time and data efficiency for low-bit diffusion models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Jointly quantizing adapter and model weights supports efficient fine-tuning and low-bit inference.

    Source: §6 · p. 9

    Gradient descent on QALoRA requires more GPU memory than PTQ, particularly for diffusion models with massive parameter counts.

    Source: §6 · p. 9

Abstract

from arXiv · show

Diffusion models have demonstrated remarkable capabilities in image synthesis and related generative tasks. Nevertheless, their practicality for real-world applications is constrained by substantial computational costs and latency issues. Quantization is a dominant way to compress and accelerate diffusion models, where post-training quantization (PTQ) and quantization-aware training (QAT) are two main approaches, each bearing its own properties. While PTQ exhibits efficiency in terms of both time and data usage, it may lead to diminished performance in low bit-width. On the other hand, QAT can alleviate performance degradation but comes with substantial demands on computational and data resources. In this paper, we introduce a data-free and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency. Specifically, we propose a quantization-aware variant of the low-rank adapter (QALoRA) that can be merged with model weights and jointly quantized to low bit-width. The fine-tuning process distills the denoising capabilities of the full-precision model into its quantized counterpart, eliminating the requirement for training data. We also introduce scale-aware optimization and temporal learned step-size quantization to further enhance performance. Extensive experimental results demonstrate that our method significantly outperforms previous PTQ-based diffusion models while maintaining similar time and data efficiency. Specifically, there is only a 0.05 sFID increase when quantizing both weights and activations of LDM-4 to 4-bit on ImageNet 256x256. Compared to QAT-based methods, our EfficientDM also boasts a 16.2x faster quantization speed with comparable generation quality. Code is available at \href{https://github.com/ThisisBillhe/EfficientDM}{this hrl}.

1 INTRODUCTION

Diffusion models face substantial latency and computational costs, while low-bit quantization trades quality against efficiency. EfficientDM aims to retain QAT-level performance with PTQ-like efficiency through data-free, parameter-efficient fine-tuning.

1 INTRODUCTION

  • PTQ is time- and data-efficient but incurs low-bit quantization errors; QAT can recover performance but requires more resources and may require unavailable training data.For LDM-4 on ImageNet 256 × 256, QAT uses 31.4GB versus 11.7GB GPU memory and 54.5 versus 2.88 GPU hours.Source: §1 · p. 2
  • EfficientDM targets QAT-level performance with PTQ-level time and data efficiency using data-free fine-tuning and quantization-aware low-rank adapters.It distills noise estimates from the full-precision model into its quantized counterpart.Source: §1 · p. 2
  • EfficientDM also introduces scale-aware LoRA optimization and TALSQ, which addresses changing activation distributions across denoising steps.Experiments on CIFAR-10, LSUN, and ImageNet are reported to achieve state-of-the-art low-bit quantization performance.Source: §1 · pp. 2–3
  • Figure 1 compares efficiency and quality for LDM-8 with 4-bit weights and activations on LSUN-Churches, encoding GPU memory consumption as circle size.Quantization can reduce computation: the cited CUTLASS implementation reports 2.03× speedup at 8-bit and 3.34× at 4-bit over full precision.Source: Figure 1 · p. 2, §1 · p. 2

2 RELATED WORK

Prior quantization methods balance efficiency against low-bit performance: PTQ is fast but can degrade, while QAT can preserve quality at greater data and computational cost. EfficientDM combines low-rank adaptation with joint quantization to address deployment overhead and ineffective learning across varying layer scales.

2 RELATED WORK

  • PTQ uses a small calibration dataset without fine-tuning weights, while QAT fine-tunes weights but has greater data and computational costs.For ResNet-18 ImageNet classification, QAT is reported to be 240× slower than PTQ.Source: §2 · p. 3
  • Diffusion-model quantization has focused largely on data-free PTQ, but existing methods degrade significantly at 4-bit or lower; TDQ fine-tunes with the original dataset and extra MLPs.EfficientDM instead uses a few trainable quantization scales per layer to pursue QAT-level performance with PTQ-level efficiency.Source: §2 · p. 3
  • PEFT methods tune a small subset of parameters while keeping most pretrained parameters frozen; examples include Prefix Tuning, Prompt Tuning, and LoRA.These approaches have been widely used in large language models.Source: §2 · pp. 3–4
  • LoRA freezes pretrained weights and learns low-rank updates, whereas QLoRA’s full-precision adapter weights cannot be integrated with quantized model weights for deployment.EfficientDM merges LoRA weights with full-precision model weights before jointly quantizing them, and addresses ineffective learning caused by layer-wise scale variation.Source: §2 · p. 4

3 BACKGROUND

The background describes diffusion as forward noise addition followed by reverse denoising, then introduces low-precision quantization and gradient estimation for optimizing quantization parameters.

3.1 DIFFUSION MODELS

  • The forward diffusion process is a Markov chain that adds noise, while the reverse process models the distribution of the preceding state given the current noisy state.The reverse model estimates Gaussian statistics; quantizing its neural networks can make those statistics inaccurate and compromise denoising and sample quality.Source: §3.1 · p. 4

3.2 MODEL QUANTIZATION

  • Uniform quantization maps floating-point values to low-precision representations using a trainable scale and bit-width-dependent thresholds; a straight-through estimator approximates gradients through rounding.The indicator function is 1 for elements in set A and 0 otherwise.Source: §3.2 · p. 4

4 EFFICIENT QUANTIZATION-AWARE FINE-TUNING OF DIFFUSION MODELS

EfficientDM fine-tunes diffusion models with quantized weights and activations using a quantization-aware adapter and data-free denoising distillation. Scale-aware LoRA optimization and temporal activation quantization address differences across layers and denoising steps.

4 EFFICIENT QUANTIZATION-AWARE FINE-TUNING OF DIFFUSION MODELS

  • EfficientDM combines a quantization-aware adapter and noise distillation with scale-aware optimization and a temporal-aware quantizer for low-bit models.The framework targets PTQ-like efficiency and QAT-like accuracy.Source: §4 · p. 5
  • QALoRA merges LoRA updates with full-precision weights before jointly quantizing them to the target bit-width.Only the quantized updated weights need to be saved after fine-tuning.Source: §4 · p. 5
  • The framework distills full-precision denoising into the quantized model by minimizing the difference between their denoising outputs on the same noise input.The input noise is obtained by iteratively denoising random Gaussian noise with the full-precision model.Source: §4 · p. 5
  • Scale-aware optimization scales LoRA gradients by the average weight quantization scale, supporting effective training across most layers.Scale-agnostic training leaves most quantized weight updates at zero because LoRA updates can be too small to cross quantization steps.Source: §4 · pp. 5–6, Figure 3 · p. 6
  • TALSQ learns activation quantization scales individually for each denoising step to address variation in activation distributions over time.The learned scales are interpolated when inference uses a different number of sampling steps than fine-tuning.Source: §4 · p. 6

5 EXPERIMENTS

Experiments evaluate EfficientDM on CIFAR-10, ImageNet, and LSUN using diffusion-model benchmarks and compare quantization quality, component effects, and efficiency. Results show strong low-bit performance alongside reduced data and time demands relative to QAT.

5.1 IMPLEMENTATION DETAILS

  • Evaluation covers DDIM on CIFAR-10 and LDM on ImageNet and LSUN, reporting IS and FID, plus sFID and Precision for ImageNet.The stated setup uses 50,000 sampled images, rank-32 adapters, 16K fine-tuning iterations, and 100 denoising steps.Source: §5.1 · pp. 6–7

5.2.1 EVALUATION OF UNCONDITIONAL GENERATION

  • On CIFAR-10 at W4A8, EfficientDM achieves FID 3.80, outperforming TDQ by 0.33 while requiring substantially less time.It also surpasses prior methods at W8A8 and W4A8, according to the reported comparison.Source: §5.2.1 · p. 7, Table 1 · p. 7

5.2.2 EVALUATION OF CONDITIONAL GENERATION ON IMAGENET 256 × 256

  • On ImageNet 256 × 256, EfficientDM denoises at W4A4 with DDIM, achieving FID 6.17 and sFID 7.75 where PTQ methods fail to denoise.It also supports 2-bit weight quantization with a sFID increase of less than 1 under DDIM.Source: §5.2.2 · pp. 7–8, Table 2 · p. 8

5.3 ABLATION STUDY

  • Ablations on LDM-4/ImageNet show QALoRA reaches FID 11.42; scale-aware optimization reduces FID by 1.87 and sFID by 10.95, and TALSQ yields sFID 7.75.The PTQD baseline has FID 259.73 at W4A4 and fails to denoise effectively.Source: §5.3 · p. 8, Table 3 · p. 9
  • On CIFAR-10, EfficientDM’s FID is comparable to QAT (10.48 vs. 7.30) while quantization time is similar to PTQ (0.97 GPU hours vs. 0.95 GPU hours).For LDM-8 on LSUN-Churches, the cited QAT method uses 16.8× the time and 2.2× the memory of the comparison baseline.Source: §5.3 · p. 9, Table 4 · p. 9

6 CONCLUSION

EfficientDM combines quantization-aware low-rank adapters and distillation with scale-aware optimization and temporal quantization. Experiments report strong low-bit results, while gradient-based optimization raises GPU-memory demands and video and 3D generation remain underexplored.

6 CONCLUSION

  • EfficientDM uses QALoRA and distills full-precision noise-estimation capabilities into a quantized model, with scale-aware optimization and TALSQ addressing layer-scale and timestep activation variation.QALoRA can be jointly quantized with model weights.Source: §6 · p. 9
  • On ImageNet 256 × 256, quantizing LDM-4 weights and activations to 4-bit produces only a 0.05 increase in sFID.The conclusion reports that prior methods failed to denoise in this setting.Source: §6 · p. 9
  • Gradient descent for QALoRA requires more GPU memory than PTQ, especially for diffusion models with massive parameters; video and 3D generation remain underexplored.The authors identify memory-efficient optimization as a possible response.Source: §6 · p. 9

A.1 EVALUATION WITH 100 DENOISING STEPS

On high-resolution LSUN generation, EfficientDM maintains image quality at low bit-widths where other PTQ methods fail; a separate 20-step evaluation reports small FID increases with interpolated scales.

A.1 EVALUATION WITH 100 DENOISING STEPS

  • On LSUN-Bedrooms, EfficientDM limits the W6A6 FID increase to 0.19 and attains FID 10.60 at W4A4, while other PTQ methods fail to generate meaningful images.At W8A8, it surpasses previous methods, including the QAT-based LSQ, in FID; at W6A6 its FID is only 0.07 higher than LSQ.Source: §A.1 · p. 13

A.2 EVALUATION WITH 20 DENOISING STEPS

  • With 20 denoising steps, interpolated quantization scales add only 0.01 FID at W8A8 and 0.64 FID at W6A6.The passage presents these results as evidence of effectiveness in the fewer-step regime.Source: §A.2 · p. 13

B DEPLOYMENT EFFICIENCY

Quantized deployment reduces model size and latency relative to full precision, while activation distributions vary across denoising steps, motivating learned step-specific quantization scales.

B DEPLOYMENT EFFICIENCY

  • W8A8 yields 3.74× smaller models and 2.03× lower latency; W4A4 yields 6.58× smaller models and 3.34× speedup; W2A8 reduces size 10.57×.Nvidia GPUs do not support W2A8, so those calculations are conducted at W8A8 precision.Source: B DEPLOYMENT EFFICIENCY · pp. 13–14, Table C · p. 14

C VARIATION OF ACTIVATION DISTRIBUTIONS ACROSS STEPS

  • A single activation-quantization parameter set can cause errors at steps with distinct distributions, while TALSQ learns separate quantization parameters for each step.Learned scales vary across steps in layers whose activation distributions shift substantially; the comparison is visualized for W4A4 LDM-4 on LSUN-Bedrooms.Source: C VARIATION OF ACTIVATION… · p. 14, Figure C · p. 15

D.1 ABLATION STUDY OF TALSQ

TALSQ improves over time-step-wise post-training quantization, and the LoRA rank study selects r = 32 as the default based on performance and trainable-parameter cost.

D.2 SELECTION OF RANK r

  • At r = 32, 2.39% of model parameters are trainable for a 4.61 FID decrease; raising rank to 64 adds 2.27% trainable parameters for only 0.09 further FID reduction.Ranks below 32 substantially decrease performance after fine-tuning, motivating r = 32 as the default.Source: §D.2 · p. 15

D.1 ABLATION STUDY OF TALSQ

  • TALSQ reduces FID by 1.64 and sFID by 2.03 versus time-step-wise post-training quantization on W4A4 LDM-4.Naive Min-Max dynamic quantization fails to denoise images even with fine-tuned QALoRA weights.Source: §D.1 · p. 15, Table D · p. 16

E GRADIENT ANALYSIS OF QALORA

The gradient analysis attributes ineffective QALoRA learning to how quantization scales interact with rounded LoRA updates.

E GRADIENT ANALYSIS OF QALORA

F ADDITIONAL VISUALIZATION RESULTS

The visualizations show quantized LDM samples across ImageNet and LSUN-Bedrooms settings, plus Stable Diffusion samples for two prompts; one comparison reports PTQD failure and EfficientDM recovery on 4-bit LSUN-Bedrooms.

F ADDITIONAL VISUALIZATION RESULTS

Related papers

Loading 2310.03270v4…