PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Diffusion: Quantizing Diffusion Models

Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, Kurt Keutzer

arXiv:2302.04304v3cs.CVcs.LG

TL;DR

Diffusion-model inference is slow and resource-intensive, and conventional PTQ does not directly handle its multi-timestep pipeline and architecture. Q-Diffusion introduces timestep-aware calibration and specialized quantization for the noise-estimation network. It achieves comparable low-bit performance, including 4-bit unconditional models and qualitatively comparable Stable Diffusion generation.

  • Problem

    Diffusion-model PTQ must handle changing noise-estimation distributions across time steps, accumulated iterative quantization error, and bimodal shortcut-layer distributions.

  • Method

    Q-Diffusion is a data-free PTQ method using timestep-aware calibration, tailored calibration objectives, and weight and activation quantizers for the noise-estimation architecture.

  • Results

    0.39-2.34 FID increment enables W4A8 PTQ for pixel-space and latent-space unconditional diffusion models, with qualitatively comparable Stable Diffusion images.

  • Takeaways & Limitations

    Q-Diffusion demonstrates that diffusion noise-estimation networks can be compressed to 4 bits while maintaining comparable generation results.

  • Takeaways & Limitations

    The work focuses on unconditional image generation and does not systematically analyze quantization of the text encoder or cross-attention mechanism for fully quantized conditional generation.

Abstract

from arXiv · show

Diffusion models have achieved great success in image synthesis through iterative noise estimation using deep neural networks. However, the slow inference, high memory consumption, and computation intensity of the noise estimation model hinder the efficient adoption of diffusion models. Although post-training quantization (PTQ) is considered a go-to compression method for other tasks, it does not work out-of-the-box on diffusion models. We propose a novel PTQ method specifically tailored towards the unique multi-timestep pipeline and model architecture of the diffusion models, which compresses the noise estimation network to accelerate the generation process. We identify the key difficulty of diffusion model quantization as the changing output distributions of noise estimation networks over multiple time steps and the bimodal activation distribution of the shortcut layers within the noise estimation network. We tackle these challenges with timestep-aware calibration and split shortcut quantization in this work. Experimental results show that our proposed method is able to quantize full-precision unconditional diffusion models into 4-bit while maintaining comparable performance (small FID change of at most 2.34 compared to >100 for traditional PTQ) in a training-free manner. Our approach can also be applied to text-guided image generation, where we can run stable diffusion in 4-bit weights with high generation quality for the first time.

1. Introduction

Diffusion models generate high-quality images but are slowed by iterative, compute-intensive noise estimation, making compression important. Q-Diffusion addresses diffusion-specific PTQ challenges with timestep-aware calibration and architecture-tailored quantization, achieving strong low-bit results.

  • Motivation: 50 to 1,000 time steps make diffusion sampling slow, while each noise-estimation network is compute- and memory-intensive.These costs are separate from the number of denoising steps and contribute to high inference memory footprints.
  • Challenges: Diffusion-specific PTQ is challenging because noise-estimation outputs vary across time steps and quantization errors accumulate during iterative inference.Calibration at an arbitrary time step can therefore produce poor performance.
  • Method: Q-Diffusion uses data-free timestep-aware calibration and quantizers tailored to the common noise-estimation architecture.The method samples calibration inputs across time steps and reduces quantization error through specialized objectives and quantizers.
  • Results: 0.39-2.34 FID increment enables W4A8 PTQ for pixel-space and latent-space unconditional diffusion models relative to full precision.The method also produces qualitatively comparable images when applied to Stable Diffusion for text-guided synthesis.

2. Related work

Diffusion models use forward noise addition and learned reverse denoising in a Markov-chain generation process. Related work covers accelerated sampling, PTQ calibration and quantization, and the underlying diffusion formulation.

  • Diffusion Models: β_t ∈ (0, 1) is a variance schedule controlling Gaussian-noise strength at each forward step.As T approaches infinity, x_T approaches an isotropic Gaussian distribution.
  • Diffusion Models: Forward diffusion repeatedly adds Gaussian noise to data, while the reverse process gradually denoises Gaussian noise into high-fidelity images.The reverse conditional distribution is learned because the real reverse conditional is unavailable.
  • Accelerated diffusion process: The same noise-estimation network computes noise at every time step, with UNet dominating common diffusion-model architectures.Q-Diffusion targets this network for acceleration, while related methods also reduce sampling steps or modify the diffusion process.
  • Post-training quantization: Quantization uses scale parameters, clipping bounds, and rounding, calibrated from estimated weight and activation distributions.Prior PTQ work studies calibration objectives and calibration-data acquisition, including data-driven, synthetic, random, and Fisher-information-based approaches.

3. Method

Q-Diffusion addresses PTQ challenges caused by multi-step denoising and UNet shortcut layers. Its pipeline uses time step-aware calibration, block reconstruction, and split quantization to reduce quantization error.

  • Multi-step denoising: Quantization errors accumulate across denoising time steps because each input depends on the previous step’s model output.This effectively multiplies the number of layers involved by the number of denoising steps.
  • Multi-step denoising: INT4 quantization produces dramatically increasing MSE during iterative DDIM denoising, making error reduction necessary at every time step.The comparison uses INT8, INT5, and INT4 models on CIFAR-10 with batch size 64.
  • Multi-step denoising: Activation distributions vary across time steps, so calibration from only a few steps can overfit those distributions and hurt overall performance.The calibration data must represent the full range of activations encountered by the shared noise estimation network.
  • UNet architecture: UNet shortcut layers can have input activations up to 200 times larger than neighboring layers because they concatenate deep and shallow features with different ranges.These differences also produce bimodal weight distributions that a shared quantizer handles poorly.
  • Q-Diffusion pipeline: Q-Diffusion combines time step-aware calibration sampling with shortcut-splitting quantization to address multi-step and UNet-specific quantization challenges.The calibration pipeline reconstructs residual or transformer blocks, calibrates other parts per layer, and tunes quantizers using adaptive rounding.

4. Experiments

Q-Diffusion preserves image quality under low-bit quantization across unconditional and text-guided generation, while calibration choices and shortcut splitting strongly affect results.

  • Unconditional Generation: Q-Diffusion preserves most perceptual quality under 4-bit weight quantization, with at most a 2.34 FID increase and imperceptible sample distortions.
  • Unconditional Generation: Q-Diffusion outperforms Linear Quantization by a large margin across tested resolutions and diffusion-model types at low bit widths.
  • Text-guided Generation: Q-Diffusion produces higher-quality Stable Diffusion images with more realistic details and better semantic information than Linear Quantization under W4A8 precision.
  • Sampling Strategies: Uniform calibration sampling across all time steps outperforms sampling from partial time-step ranges, while tuning sampling intervals or sample counts has marginal impact.
  • Split Shortcut Quantization: Splitting shortcut paths improves DDIM CIFAR-10 quantization from FID 188.11 with linear W4A8 quantization to FID 4.93.

5. Conclusion

The conclusion presents Q-Diffusion as a multi-timestep PTQ scheme that enables comparable 4-bit diffusion-model quality.

  • Q-Diffusion calibrates across multiple denoising time steps and achieves comparable results to full-precision models under 4-bit quantization.

A. Extended Experimental Settings

The extended settings describe implementation choices, calibration procedures, and activation-distribution observations underlying the experiments.

  • Experimental Settings: The experiments use model-specific denoising-step configurations for DDIM, LSUN latent diffusion, and Stable Diffusion.
  • Implementation Details: Quantization applies to weights and activations in matrix multiplications, while activation functions and normalization layers remain full precision.
  • Text-guided Calibration: Stable Diffusion calibration adds conditional and unconditional features for sampled MS-COCO text prompts.
  • Calibration Procedure: The calibration algorithm samples intermediate inputs across denoising time steps, updates weight quantizers block by block, and then updates activation quantizers when enabled.
  • Calibration Settings: Uniformly sampling 20 time steps with 5,120 unconditional calibration samples provides high-quality quantization without hyperparameter tuning.
  • Activation Analysis: Residual Conv layers exhibit wider activation ranges, reaching 1200 in the first Conv layer, while activation distributions also vary significantly across time steps.

C. Quantitative Evaluation on Text-guided Image Generation

The text-guided evaluation measures generated-image quality with FID and CLIP score on MS-COCO prompts. Q-Diffusion shows minimal degradation, though slight artifacts occur more often when both weights and activations are quantized.

  • FID and CLIP score are computed on 10k Stable Diffusion images generated from MS-COCO 2017-val prompts.
  • Q-Diffusion causes minimal quality degradation across settings, whereas Linear Quantization produces significant quality drops, especially with activation quantization.
  • Slight artifacts appear more often in Q-Diffusion images when both weights and activations are quantized.

D. Study of Combining with Fast Samplers

Q-Diffusion can be combined with fast samplers, but changing sampling trajectories can reduce quantized generation quality. The method remains better than Linear Quantization and SQuant in the tested settings.

  • D. Study of Combining with Fast Samplers: Fast sampling trajectories reduce denoising steps, providing an acceleration direction orthogonal to Q-Diffusion’s quantization-based compression.The paper investigates combining Q-Diffusion with DPM-Solver++ to exploit both directions.
  • D. Study of Combining with Fast Samplers: Q-Diffusion results outperform Linear Quantization and SQuant under W4A8 precision for the tested CIFAR-10 and LSUN settings.The comparison uses 100, 200, and 500 sampling steps for CIFAR-10, LSUN-Bedrooms, and LSUN-Churches, respectively.
  • D. Study of Combining with Fast Samplers: Changing sampling trajectories can make activation distributions inconsistent with Q-Diffusion’s calibration, degrading W4A8 generation quality.The authors leave a systematic pipeline combining quantization and fast sampling for future work.

E. Comparing with PTQ4DM [40]

Q-Diffusion is compared with PTQ4DM using evaluation settings that account for attention matrix multiplication precision. It consistently outperforms PTQ4DM, including when PTQ4DM uses INT8 attention matmuls.

  • E. Comparing with PTQ4DM [40]: The evaluation reports Inception Score, Frechet Inception Distance, and sFID over 10k generated samples.Q-Diffusion computes attention q ∗ k and attn ∗ v matrix multiplications in full integer, unlike the cited PTQ4DM implementation’s FP16/327 setting.
  • E. Comparing with PTQ4DM [40]: Activation distributions in DDIM vary significantly across time steps, complicating calibration for diffusion-model quantization.This distributional variation is shown for multiple DDIM layers on CIFAR-10.
  • E. Comparing with PTQ4DM [40]: Q-Diffusion consistently outperforms PTQ4DM when attention matrix multiplications are reproduced with integer arithmetic.The comparison includes PTQ4DM results with INT8 attention act-to-act matmuls and the original FP16/32 settings.
  • E. Comparing with PTQ4DM [40]: Directly quantizing PTQ4DM attention matmuls to 8-bit further degrades generation quality.The degraded results are reported in parentheses in the comparison.

F. Limitations of this work

The paper’s main scope is unconditional diffusion-model noise-estimation quantization, while several extensions remain open. It also examines alternative calibration sampling schemes without improving on uniform sampling.

  • F. Limitations of this work: The work focuses on PTQ for diffusion-model noise-estimation networks in unconditional image generation.A systematic study of quantizing the text encoder and cross-attention remains necessary for fully quantized conditional generation.
  • F. Limitations of this work: Q-Diffusion can be applied to Stable Diffusion, but the paper does not provide a systematic analysis of quantization effects on text encoding and cross-attention.These components are identified as requirements for a fully quantized conditional-generation framework.
  • F. Limitations of this work: The evaluated figures include Stable Diffusion FID and CLIP results across precisions, plus INT4 image samples using PNDM and DPM-Solver++.The figures cover text-guided synthesis and unconditional LSUN-Bedroom and LSUN-Church samples.
  • F. Limitations of this work: Non-uniform time-step sampling and additional quantizer designs for attention softmax outputs are left for future work.The paper lists these as other design choices requiring further investigation.

G. Additional Random Samples

Additional figures show random samples from Q-Diffusion’s 4-bit quantized unconditional and text-guided models, using fixed random seeds.

  • G. Additional Random Samples: Additional random samples are provided for weight-only and fully quantized diffusion models under 4-bit quantization.The samples are intended to supplement the reported experimental results.
  • G. Additional Random Samples: INT4 quantized 256 × 256 LSUN-Bedroom models are shown with a fixed random seed.
  • G. Additional Random Samples: INT4 quantized 256 × 256 LSUN-Church models are shown with a fixed random seed.
  • G. Additional Random Samples: INT4 quantized Stable Diffusion produces 512 × 512 text-guided images from LAION-5B with a fixed random seed.
Loading 2302.04304v3…