Source-linked AI summary
Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers
Lei Chen, Yuan Meng, Chen Tang, Xinzhu Ma, Jingyan Jiang, Xin Wang, Zhi Wang, Wenwu Zhu
TL;DR
DiT quantization research remains limited, while existing diffusion quantization methods primarily target UNets despite the high inference costs of DiTs. Q-DiT combines automatic granularity allocation with sample-wise dynamic activation quantization. It reports only a 1.09 increase in FID at W4A8 on ImageNet 256×256.
Problem
DiTs have high inference costs, while diffusion quantization research remains limited and existing methods primarily target UNets.
Source: §1 · p. 1Method
Q-DiT combines fine-grained group quantization with sample-wise dynamic activation quantization to address channel variation and activation changes across timesteps and samples.
Source: §1 · p. 1Results
At W4A8 on ImageNet 256×256, Q-DiT increased FID by only 1.09.
Source: §7 · p. 8Takeaways & Limitations
Q-DiT reports lossless compression at W6A8 and minimal degradation at W4A8 for image and video generation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recent advancements in diffusion models, particularly the architectural transformation from UNet-based models to Diffusion Transformers (DiTs), significantly improve the quality and scalability of image and video generation. However, despite their impressive capabilities, the substantial computational costs of these large-scale models pose significant challenges for real-world deployment. Post-Training Quantization (PTQ) emerges as a promising solution, enabling model compression and accelerated inference for pretrained models, without the costly retraining. However, research on DiT quantization remains sparse, and existing PTQ frameworks, primarily designed for traditional diffusion models, tend to suffer from biased quantization, leading to notable performance degradation. In this work, we identify that DiTs typically exhibit significant spatial variance in both weights and activations, along with temporal variance in activations. To address these issues, we propose Q-DiT, a novel approach that seamlessly integrates two key techniques: automatic quantization granularity allocation to handle the significant variance of weights and activations across input channels, and sample-wise dynamic activation quantization to adaptively capture activation changes across both timesteps and samples. Extensive experiments conducted on ImageNet and VBench demonstrate the effectiveness of the proposed Q-DiT. Specifically, when quantizing DiT-XL/2 to W6A8 on ImageNet ($256 \times 256$), Q-DiT achieves a remarkable reduction in FID by 1.09 compared to the baseline. Under the more challenging W4A8 setting, it maintains high fidelity in image and video generation, establishing a new benchmark for efficient, high-quality quantization in DiTs. Code is available at \href{https://github.com/Juanerx/Q-DiT}{https://github.com/Juanerx/Q-DiT}.
1. Introduction
DiTs offer performance and scalability but incur high inference costs, while existing diffusion-model quantization work is limited and largely targets UNets. Q-DiT combines fine-grained group quantization with sample-wise dynamic activation quantization to address channel and timestep variation.
1. Introduction
- DiT inference is computationally intensive, and existing diffusion quantization methods primarily target UNets and can be difficult to scale to large models.PTQ is attractive for large models because it avoids retraining.Source: §1 · p. 1
- Q-DiT pairs fine-grained group quantization for channel-wise weight and activation variance with sample-wise dynamic quantization for activation changes across timesteps and samples.The runtime adjustment adapts quantization parameters to changing activation distributions.Source: §1 · pp. 1–2
- An evolutionary search selects layer-wise group sizes using FID and FVD to guide quantization toward generated-sample quality.The search also observes predefined computational constraints.Source: §1 · pp. 1–2
- Figure 1 depicts shared within-layer group sizes for weights and activations, search-guided layer configurations, and runtime dynamic activation quantization.The search is guided by FID/FVD scores between real and generated samples.Source: Figure 1 · p. 2
- Experiments on ImageNet and VBench show Q-DiT achieves lossless compression with W6A8 and minimal degradation with W4A8 for image and video generation.Source: §1 · p. 2
2. Related Work
Related work covers quantization approaches for models and diffusion systems, while noting that DiTs combine transformer architecture with activation dynamics that existing methods may not address together.
2. Related Work
- PTQ adjusts quantization parameters using a small calibration dataset without retraining, but low-bit quantization can degrade performance.Reconstruction-based methods reduce layer or block reconstruction error but are difficult to scale to large models.Source: §2 · p. 2
- Transformer quantization includes methods for activation distributions, salient weights, and outliers, but these techniques may not directly transfer to DiTs.The passage describes PTQ4ViT, RepQ-ViT, GPTQ, AWQ, LLM.int8(), and Outlier Suppression.Source: §2 · p. 3
- Diffusion quantization methods address denoising-step variation through reconstruction, variance calibration, timestep prediction, or timestep-sequence reduction, while PTQ4DiT targets channel magnitudes and temporal variability.The passage states that these methods may not handle transformer characteristics and denoising dynamics simultaneously.Source: §2 · p. 3
3. Observations of DiT Quantization
The observations identify channel-wise weight and activation variation and activation distribution shifts across denoising timesteps and samples as distinct DiT quantization challenges.
3. Observations of DiT Quantization
- Applying recent UNet-based quantization methods directly to DiTs leads to significant performance degradation.The section investigates differences in weight and activation distributions between DiTs and UNet-based architectures.Source: §3 · p. 3
- Weights and activations vary more across input channels than output channels, and activation outliers can distort tensor-wise quantization parameters for non-outliers.Figure 2 shows distributions across DiT-XL/2 layers, with red peaks marking higher values and blue areas lower values.Source: §3 · p. 3, Figure 2 · p. 3
- Activation distributions shift substantially across denoising timesteps, and the temporal shift also varies across samples.Figures 3 and 4 present activation distributions across timesteps and activation standard deviations across blocks.Source: §3 · p. 3
4. Preliminary
The preliminary section introduces uniform quantization as the hardware-friendly approach used for weights and activations, mapping floating-point values to discrete levels.
4. Preliminary
- Uniform quantization divides the floating-point range into equally spaced intervals and maps each interval to a discrete value.The paper selects uniform quantization for both weights and activations as a hardware-friendly choice.Source: §4 · p. 3
- The quantization function maps an input tensor x to a b-bit integer tensor.The passage introduces this mapping through function Q.Source: §4 · p. 3
- The quantization notation defines rounding, the scaling factor s, and zero point Z.The rounding operation is denoted ⌊·⌉.Source: §4 · p. 4
5. Method: Q-DiT
Q-DiT combines automatic layer-wise quantization-granularity allocation with sample-wise dynamic activation quantization. It searches group sizes against generation-quality objectives under a BitOps constraint and adapts activation quantization during inference.
5.1. Automatic Quantization Granularity Allocation
- Input-channel-wise quantization can require repeated intermediate rescaling that limits low-precision hardware efficiency; group quantization offers a compromise with tensor quantization.Weights and activations are divided into groups and quantized separately.Source: §5.1 · p. 4
- Reducing group size from 128 to 96 increased FID by about 11.8%, from 17.87 to 19.97, showing that finer groups do not always improve generation quality.Source: §5.1 · p. 4, Table 1 · p. 4
- Because smaller layer MSE did not necessarily correspond to less DiT quantization degradation, Q-DiT evaluates generated-sample quality directly with FID or FVD.Source: §5.1 · p. 5
- Evolutionary search selects layer-wise group sizes using FID for image generation or FVD for video generation, subject to a BitOps constraint.The approach is intended to balance generation quality and efficiency.Source: §5.1 · p. 5
5.2. Sample-wise Dynamic Activation Quantization
- Earlier per-timestep activation parameters could require up to 39% of full-precision model size at 50 steps; Q-DiT computes per-sample group parameters on-the-fly and fuses min-max computation for negligible overhead.Source: §5.2 · p. 5
6. Experiments
Experiments evaluate Q-DiT on DiT-XL/2 image generation and STDiT3 video generation, then test its components through ablations. Q-DiT retains strong image and video quality under low-bit quantization.
6.1. Experimental Setup
- Evaluation uses DiT-XL/2 at 256×256 and 512×512 with 50 or 100 DDIM steps, and STDiT3 videos assessed across 16 VBench dimensions.Image evaluation samples 10K images per setting; video evaluation uses five 2-second videos per prompt.Source: §6.1 · pp. 5–6
6.2. Main Results
- At W4A8, Q-DiT substantially outperforms five image-generation baselines with FID 15.76 and IS 98.78, and beats G4W+P4A on 15 of 16 VBench metrics.The image results cover ImageNet at 256×256; the video results use STDiT3 on VBench.Source: Table 2 · p. 6, §6.2 · p. 7, Table 3 · p. 7
6.3. Ablation Studies
- Under W4A8, adding dynamic activation quantization improves the group-size-128 baseline, and subsequent group-size allocation brings FID to 6.40, near full precision.With dynamic activation quantization, FID is 6.64, sFID is 19.29, and IS is 211.27.Source: §6.3 · p. 7, Table 4 · p. 8
- At W4A8, Q-DiT produces cleaner, more realistic images with better texture preservation than G4W+P4A in the ImageNet examples.The figure includes examples at 256×256 and 512×512.Source: Figure 5 · p. 8
- At W16A8, our activation quantization achieves FID 5.34 versus TFMQ-DM’s 7.74, isolating activation quantization; the proposed group-quantization search significantly outperforms ILP, Hessian-based search, and the baseline.Source: §6.3 · p. 8
7. Conclusion
Q-DiT combines automatic quantization-granularity allocation with dynamic activation quantization to address channel-wise weight and activation variance and timestep-varying activations. Experiments report strong low-bit generation quality, while evolutionary group-size search remains costly.
7. Conclusion
- Q-DiT combines automatic quantization-granularity allocation for input-channel variance with runtime dynamic activation quantization for activation-range variation across timesteps.The conclusion reports an FID increase of only 1.09 when quantizing to W4A8 on ImageNet 256×256.Source: §7 · p. 8
- The dynamic activation comparison contrasts Q-DiT's timestepwise and sample-wise quantization with TFMQ-DM's timestep-aware approach.The comparison uses W16A8.Source: Table 5 · p. 8
- Evolutionary search for optimal group sizes is computationally expensive and time-consuming, increasing optimization cost and duration.The authors identify optimizing this search as future work.Source: §7 · p. 8