Source-linked AI summary
Post-training Quantization on Diffusion Models
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, Yan Yan
TL;DR
Diffusion models are slow because generation requires lengthy iterative sampling and costly noise-estimation networks, while prior acceleration methods mainly shorten trajectories. This paper introduces PTQ4DM, a training-free post-training quantization method designed for timestep-varying diffusion outputs. It directly quantizes pretrained models to 8-bit without significant performance loss and can complement fast-sampling methods such as DDIM.
Problem
Diffusion-model generation is slow because denoising repeatedly uses cumbersome networks across many timesteps, while existing acceleration methods mainly address sampling length and standard PTQ assumes single-timestep scenarios.
Method
PTQ4DM applies post-training quantization to diffusion noise-estimation networks and designs timestep-aware calibration across quantized operations, datasets, and metrics.
Results
PTQ4DM directly quantizes pretrained diffusion models to 8-bit without significant performance loss, and an 8-bit DDPM setting outperforms its full-precision counterpart.
Takeaways & Limitations
PTQ4DM accelerates network inference in diffusion denoising and can be added as a plug-and-play module to fast-sampling methods such as DDIM.
Takeaways & Limitations
The calibration dataset must reflect activation and sample-distribution changes across diffusion timesteps, unlike common single-timestep calibration.
Abstract
from arXiv · showhide
Denoising diffusion (score-based) generative models have recently achieved significant accomplishments in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data into noise and a backward denoising process for sampling data from noise. Unfortunately, the generation process of current denoising diffusion models is notoriously slow due to the lengthy iterative noise estimations, which rely on cumbersome neural networks. It prevents the diffusion models from being widely deployed, especially on edge devices. Previous works accelerate the generation process of diffusion model (DM) via finding shorter yet effective sampling trajectories. However, they overlook the cost of noise estimation with a heavy network in every iteration. In this work, we accelerate generation from the perspective of compressing the noise estimation network. Due to the difficulty of retraining DMs, we exclude mainstream training-aware compression paradigms and introduce post-training quantization (PTQ) into DM acceleration. However, the output distributions of noise estimation networks change with time-step, making previous PTQ methods fail in DMs since they are designed for single-time step scenarios. To devise a DM-specific PTQ method, we explore PTQ on DM in three aspects: quantized operations, calibration dataset, and calibration metric. We summarize and use several observations derived from all-inclusive investigations to formulate our method, which especially targets the unique multi-time-step structure of DMs. Experimentally, our method can directly quantize full-precision DMs into 8-bit models while maintaining or even improving their performance in a training-free manner. Importantly, our method can serve as a plug-and-play module on other fast-sampling methods, e.g., DDIM. The code is available at https://github.com/42Shawn/PTQ4DM .
1. Introduction
Diffusion models generate high-quality, diverse data but remain costly because denoising combines lengthy sampling iterations with cumbersome noise-estimation networks. This paper introduces training-free PTQ4DM to compress those networks and address time-step-dependent output distributions.
- Diffusion models achieve strong quality and diversity across generative tasks, including image, audio, video, graph, super-resolution, inpainting, and translation applications.
- Thousands of iterative noise-estimation steps and expensive neural networks make diffusion-model generation slow and difficult to deploy broadly.The denoising process repeatedly applies a cumbersome network across many timesteps.
- Existing acceleration methods mainly shorten sampling trajectories while overlooking the cost of noise estimation at each iteration.
- Training-aware compression is unsuitable for many diffusion models because training data may be inaccessible and fine-tuning can require hundreds of thousands of GPU hours.
- PTQ4DM directly quantizes pretrained noise-estimation networks without retraining, targeting output-distribution discrepancies across denoising timesteps.The method explores quantized operations, calibration datasets, and calibration metrics for diffusion models.
- PTQ4DM quantizes pretrained diffusion models to 8-bit without significant performance loss and can operate as a plug-and-play module for other acceleration methods.
2. Related Work
Prior diffusion-model acceleration methods primarily seek shorter sampling trajectories, while PTQ4DM adds orthogonal network compression. The paper also situates PTQ within broader post-training quantization practice and calibration design.
- Diffusion Model Acceleration: Most diffusion-model acceleration methods reduce sampling steps through faster schedules, implicit phases, analytical approximations, or dynamically adjusted trajectories.
- Diffusion Model Acceleration: Some alternative acceleration methods require additional training after pretraining, making them less desirable when retraining is costly or impractical.
- Diffusion Model Acceleration: PTQ4DM complements trajectory-based acceleration by compressing the noise-estimation network used at every denoising iteration.
- Post-training Quantization: PTQ quantizes a trained network after training and generally requires less time and computation than quantization-aware training.
- Post-training Quantization: PTQ selects weight and activation quantization parameters, often by minimizing tensor reconstruction error using metrics such as MSE, L1 distance, cosine distance, or KL divergence.
- Post-training Quantization: Calibration samples are used to calculate activations, and their selection affects the quantization parameters; diffusion models require calibration across multiple timesteps.
3. PTQ on Diffusion Models
Diffusion models reverse a long noise-injection process through thousands of denoising steps, each using a costly noise-estimation network. This multi-time-step structure makes straightforward PTQ calibration ineffective, motivating a calibration method that covers changing activation distributions.
- Preliminaries: Diffusion models gradually add Gaussian noise to data, then generate samples by iteratively denoising from Gaussian noise.The forward process uses a Markov chain, while the learned reverse process samples x_{t-1} from p_θ(x_{t-1}|x_t).
- Preliminaries: DDPM requires 4000 denoising steps for one sample, and each step relies on a cumbersome noise-estimation network.These two characteristics jointly create high inference cost.
- Challenges: PTQ calibration is challenging because diffusion-network inputs and activations vary substantially across denoising time-steps.The model receives x_t and t, produces distribution parameters, and repeatedly samples x_{t-1}.
- Challenges: Previous single-time-step calibration methods become inapplicable when activation distributions differ across time-steps.Figure 3 examines both overall channel ranges and layer-wise activation histograms to expose this discrepancy.
- Normally Distributed Time-step Calibration: PTQ4DM addresses the discrepancy with Normally Distributed Time-step Calibration, which samples time-steps from a skew normal distribution for calibration.The method generates calibration samples through the denoising process at the sampled time-steps.
4. More Experiments
Experiments evaluate 8-bit PTQ4DM on CIFAR10 and ImageNet under DDPM and DDIM settings. The method directly quantizes diffusion models and compares generated-image quality against full-precision and other baselines.
- CIFAR10 32×32 and ImageNet 64×64 experiments cover DDPM with 4000 steps and DDIM with 100 or 250 steps.
- 1024 calibration samples are used to quantize the network to 8-bit, followed by 10,000 generated images for evaluation.
- The 8-bit DDPM quantized by PTQ4DM outperforms full-precision DDPM for CIFAR10 32×32 image generation.The reported evaluation uses 10,000 samples, so results may differ slightly from papers using 50,000 samples.
- PTQ4DM’s results indicate redundancy in the noise-estimation network, complementing prior evidence of redundancy in sampling-trajectory length.
5. Conclusion
PTQ4DM targets the network-computation cost that slows diffusion denoising, directly quantizing pretrained models to 8 bits without significant performance degradation. It can also be combined with fast-sampling methods such as DDIM.
- PTQ4DM addresses the cumbersome noise-estimation network, the second factor slowing diffusion generation alongside lengthy sampling iterations.
- A pretrained diffusion model can be directly quantized into 8 bits without significant performance degradation.
- PTQ4DM can be added to other fast-sampling methods, including DDIM.
6. Appendix
Appendix experiments examine non-uniform timestep sampling and measure inference latency for the original and quantized networks. The 8-bit model achieves about a twofold speedup on an Nvidia RTX A6000 GPU.
- The timestep-sampling study compares normal, Poisson, and exponential distributions and conducts hyperparameter-selection experiments.
- 8-bit quantization achieves about 2x speedup on an Nvidia RTX A6000 GPU compared with the original network.
- The measured speedup may be more significant on an NPU than on the tested GPU.