PaperlayerView exact revision on arXiv

Source-linked AI summary

SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, Song Han

arXiv:2411.05007v4cs.CVcs.LG

TL;DR

Large diffusion models create memory and latency challenges, and weight-only quantization cannot accelerate their computationally bounded inference. SVDQuant absorbs outliers with a high-precision low-rank branch while quantizing residuals to 4 bits, with Nunchaku fusing the branches to reduce overhead. Across diffusion models, it preserves image quality while substantially reducing memory and improving speed over weight-only baselines.

  • Problem

    Scaling diffusion models increases memory and latency demands, while 4-bit quantization is difficult because both weights and activations are sensitive to outliers.

  • Method

    SVDQuant migrates activation outliers into weights, decomposes the updated weights with SVD, and uses a 16-bit low-rank branch alongside a 4-bit residual branch; Nunchaku fuses their kernels.

  • Results

    SVDQuant preserves image quality across SDXL, PixArt-Σ, and FLUX.1, reducing 12B FLUX.1 memory usage by 3.5× and achieving 3.0× and 3.1× speedups over weight-only baselines on RTX 4090 and RTX 5090, respectively.

  • Takeaways & Limitations

    The approach supports 4-bit diffusion inference across UNet and DiT architectures, INT4 and FP4 data types, and pretrained LoRAs without re-quantization.

  • Takeaways & Limitations

    A separately executed low-rank branch incurs latency overhead of approximately 50% of the 4-bit branch latency, motivating kernel fusion; increasing rank also presents a stated trade-off.

Abstract

from arXiv · show

Diffusion models can effectively generate high-quality images. However, as they scale, rising memory demands and higher latency pose substantial deployment challenges. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where existing post-training quantization methods like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing, which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights. Then, we use a high-precision, low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD), while a low-bit quantized branch handles the residuals. This process eases the quantization on both sides. However, naively running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without re-quantization. Extensive experiments on SDXL, PixArt-$Σ$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5$\times$, achieving 3.0$\times$ speedup over the 4-bit weight-only quantization (W4A16) baseline on the 16GB laptop 4090 GPU with INT4 precision. On the latest RTX 5090 desktop with Blackwell architecture, we achieve a 3.1$\times$ speedup compared to the W4A16 model using NVFP4 precision.

1 INTRODUCTION

Scaling diffusion models increases computational cost, memory demands, and latency, while weight-only quantization cannot accelerate their computationally bounded inference. SVDQuant addresses aggressive 4-bit weight-and-activation quantization with a low-rank branch, and Nunchaku fuses its computation to preserve speedups.

  • Motivation: 12B FLUX.1 models create substantial deployment barriers as diffusion-model computation and memory demands grow with scale.The paper contrasts FLUX.1’s 12B parameters with smaller diffusion models and notes especially high costs for interactive, low-latency use cases.
  • Motivation: Weight-only quantization cannot accelerate diffusion models because their inference remains computationally bounded even at batch size one.This differs from LLM inference, where latency is primarily constrained by loading model weights on modern GPUs.
  • SVDQuant: SVDQuant migrates activation outliers into weights, then decomposes the updated weights into a 16-bit low-rank branch and a 4-bit residual branch.The low-rank branch absorbs difficult outliers while the residual is quantized at low precision.
  • Nunchaku: Nunchaku fuses low-rank computation into 4-bit quantization and computation kernels to remove redundant memory access overhead.This enables measured inference speedup despite adding the low-rank branch.
  • Results: 3.5× lower memory usage and 3.0× speedup over NF4 weight-only quantization are reported for 12B FLUX.1 on the stated NVIDIA GPU setups.The results also support INT4 and FP4 models across multiple diffusion architectures while preserving image quality and integrating LoRA without re-quantization.

2 RELATED WORK

Diffusion-model research has progressed from convolutional UNet architectures toward transformer-based DiT models as generation quality and control demands increase. Recent work also motivates low-bit quantization and efficient handling of low-rank adapters.

  • Diffusion models: Diffusion models generate high-quality samples through iterative denoising, and text-to-image advances have expanded content-generation applications.The related work describes diffusion models as a powerful generative-model family and notes their growing use in text-to-image generation.
  • Architectures: Research has shifted from convolution-based UNet architectures toward transformer-based DiT architectures while scaling diffusion models.The passage places this architectural transition alongside recent advances in text-to-image diffusion models.
  • Efficient adaptation: Nunchaku supports integer and floating-point 4-bit quantization across UNet and DiT backbones while fusing low-rank branches to avoid re-quantization overhead for LoRA.Kernel fusion removes the memory-access overhead that would otherwise arise from separately executing the LoRA branch.

3 QUANTIZATION PRELIMINARY

The section introduces quantization notation and explains why W4A4 diffusion inference is difficult: both weights and activations contain outliers, while existing suppression methods are costly or incompatible with diffusion architectures.

  • Quantization basics: Quantization represents a tensor with a low-bit tensor and a scaling factor, with qmax determining the maximum quantized value.For signed k-bit integer quantization, qmax = 2^k−1 −1; for the stated 4-bit floating-point format, qmax = 6.
  • Quantization basics: A quantized linear layer approximates XW using the product of dequantized input and weight representations, and the same approximation applies to convolutions.Modern GPUs generally require weights and activations to use the same bit width to retain low-bit speed advantages.
  • W4A4 challenges: W4A4 quantization is difficult because outliers in both weights and activations substantially increase quantization error.The section identifies quantization-aware training and rotation as alternative suppression methods, but neither is convenient here.
  • W4A4 challenges: Quantization-aware training requires massive resources for models above 10B parameters, while rotation is incompatible with adaptive normalization and can add runtime overhead.Runtime-generated normalization weights prevent offline rotation, and rotating activations and weights online is expensive.

4 METHOD

SVDQuant reduces 4-bit quantization error by moving activation outliers into weights, absorbing the resulting weight outliers with a low-rank branch, and quantizing the residual. Nunchaku fuses low-rank and low-bit kernels to reduce the memory-access overhead that would otherwise undermine speedups.

  • 4.1 PROBLEM FORMULATION: The quantization error depends on input and weight magnitudes together with their respective quantization errors, motivating optimization of all four terms.
  • 4.2 SVDQUANT: ABSORBING OUTLIERS VIA LOW-RANK BRANCH: Smoothing reduces activation outliers but magnifies weight outliers, limiting overall error reduction when both sides are quantized.The smoothed input has lower magnitude and fewer outliers, while the transformed weight has greater magnitude and more outliers.
  • 4.2 SVDQUANT: ABSORBING OUTLIERS VIA LOW-RANK BRANCH: SVDQuant decomposes the transformed weight as ˆ W = L1L2 + R, computing the low-rank branch in 16-bit precision and quantizing the residual in 4 bits.The low-rank factors have rank r, typically set to 16 or 32, and add only mr+nr over mn to the overall costs.
  • 4.3 NUNCHAKU: FUSING LOW-RANK AND LOW-BIT BRANCH KERNELS: Nunchaku fuses Down Projection with Quantize and Up Projection with 4-Bit Compute because each pair shares an input or output, reducing redundant data movement.For rank 32, naïve execution introduces 57% latency overhead; kernel fusion targets the extra 16-bit reads and writes.
  • 4.2 SVDQUANT: ABSORBING OUTLIERS VIA LOW-RANK BRANCH: SVD identifies dominant singular components in the smoothed weight, whose removal substantially reduces residual magnitude and compresses its value range.The first 32 singular values of the smoothed weight show a steep drop, while the residual has fewer outliers and a narrower range.
  • 4.3 NUNCHAKU: FUSING LOW-RANK AND LOW-BIT BRANCH KERNELS: A separate low-rank branch adds approximately 50% latency overhead because unchanged activation sizes shift the bottleneck from computation to memory access.The overhead is especially problematic when activations exceed the GPU L2 cache, causing additional DRAM loads and stores.

5 EXPERIMENTS

Experiments across diffusion architectures and benchmarks show that SVDQuant preserves quality at low precision while Nunchaku reduces memory use and inference latency. The method also supports LoRAs without requantization, with ablations supporting its low-rank outlier-absorption design.

  • Setups: Experiments cover FLUX.1, PixArt-Σ, SANA, SDXL, and SDXL-Turbo across UNet and DiT backbones.Evaluation uses COCO Captions 2024 for calibration and MJHQ-30K and sDCI prompts for benchmarking.
  • Setups: Image quality is evaluated with FID and Image Reward, while LPIPS and PSNR measure perceptual and pixel-level similarity.FID is lower-is-better, whereas Image Reward is higher-is-better.
  • Quality results: Our 8-bit results closely match 16-bit quality, while 4-bit results outperform other 4-bit baselines across models.On FLUX.1-dev, INT8 PSNR reaches 27 on MJHQ; on PixArt-Σ, NVFP4 is comparable to FP16 and both 4-bit variants outperform ViDiT-Q’s W4A8 results.
  • Quality results: On SDXL-Turbo, 4-bit models substantially outperform MixDQ W4A8 while achieving FID scores on par with FP16.On SDXL, INT4 and NVFP4 achieve quality comparable to TensorRT’s W8A8 results.
  • Efficiency: SVDQuant reduces the 12B FLUX.1 transformer from 22.2 GiB to 6.1 GiB, a 3.6× reduction including low-rank overhead.Nunchaku provides 3.0× speedup on RTX 4090 GPUs with INT4 and 3.1× on RTX 5090 with NVFP4; laptop offloading avoidance yields 10.1× speedup.
  • Adaptation and ablation: The 4-bit model integrates off-the-shelf LoRAs without requantization and matches the original 16-bit FLUX.1-dev image quality.The ablation study reports severe degradation for naïve quantization and SVD-only approaches, while smoothing remains unsatisfactory.
  • Adaptation and ablation: Increasing the low-rank branch rank involves a documented trade-off.The paper directs readers to Appendix E.5 for details.

6 CONCLUSION

The paper introduces SVDQuant for 4-bit diffusion-model quantization and Nunchaku for efficient fused inference. Together, they preserve image quality while reducing memory use and accelerating deployment on NVIDIA GPUs.

  • Conclusion: SVDQuant absorbs weight and activation outliers with a low-rank branch, while Nunchaku fuses low-rank and low-bit kernels to reduce redundant data movement.The approach targets 4-bit post-training quantization of diffusion models.
  • Conclusion: SVDQuant preserves image quality while Nunchaku achieves 3.5× lower memory usage and 3.0× speedup over W4A16 on NVIDIA RTX 4090 and 5090 GPUs.The conclusion frames this as enabling more efficient deployment of large-scale diffusion models on edge devices.

CHANGELOG

The changelog records the progression from the initial preprint through a typo fix to the ICLR 2025 camera-ready version.

  • Changelog: V1 marks the initial preprint release.
  • Changelog: V2 fixes a typo.
  • Changelog: V3 is the ICLR 2025 camera-ready version, combining SVDQuant with GPTQ and updating NVFP4 and SANA results.

A.1 PROOF OF PROPOSITION 4.1

The proof decomposes the error of separately quantizing inputs and weights into an input-dependent weight-error term and an activation-error term weighted by quantized-weight magnitude.

  • Error decomposition: Proposition 4.1 defines quantization error as E(X, W) = ∥XW − Q(X)Q(W)∥F.
  • Final bound: The resulting bound is ∥X∥F∥W − Q(W)∥F + ∥X − Q(X)∥F(∥W∥F + ∥W − Q(W)∥F).
  • Error decomposition: The proof inserts and subtracts XQ(W), separating weight quantization error from activation quantization error.
  • Distributional assumptions: Proposition 4.2 assumes a regularity condition on tensor-element distributions and specializes the analysis to normally distributed elements.The proof references Gaussian maximal inequalities, Cauchy–Schwarz, and the expected half-normal distribution.

B BENCHMARK MODELS

The benchmark covers six text-to-image diffusion models spanning DiT and UNet architectures, including both full and distilled variants.

  • The evaluation includes FLUX.1, PixArt-Σ, SANA, SDXL, and SDXL-Turbo across DiT and UNet backbones.FLUX.1-dev and FLUX.1-schnell are both evaluated, while SDXL is paired with its distilled SDXL-Turbo variant.
  • FLUX.1 is a 12B-parameter DiT model evaluated in 50-step and 4-step variants.
  • PixArt-Σ is a 600M-parameter DiT model evaluated with its default 20-step setting.
  • SANA is a 1.6B-parameter DiT model using linear attention and a 32× compression autoencoder.
  • SDXL is a 2.6B-parameter UNet model evaluated in 30-step and 4-step settings through SDXL-Turbo.

C BENCHMARK DATASETS

The benchmark uses two prompt sets to test generalization across artistic and realistic image-generation styles.

  • The evaluation uses MJHQ-30K and Densely Captioned Images prompt sets with varying styles.
  • MJHQ-30K contributes 5K randomly selected prompts for artistic image generation.The source dataset contains 30K Midjourney samples across 10 categories.
  • The summarized DCI dataset contributes 5K randomly selected prompts for realistic image generation.Its captions are condensed to 77 tokens for diffusion-model use.

D IMPLEMENTATION DETAILS

Implementation uses distinct quantization configurations for 8-bit and 4-bit inference, with offline smoothing-factor selection.

  • The 4-bit configuration uses per-group symmetric quantization for weights and activations with a rank-32 low-rank branch.
  • INT4 uses group size 64 with 16-bit scales, while NVFP4 uses group size 16 with FP8 scales on Blackwell GPUs.
  • The 8-bit configuration uses per-token dynamic activation quantization, per-channel weight quantization, and a rank-16 low-rank branch.
  • The per-channel smoothing vector is computed from activation and weight maxima, with layerwise α selected offline to minimize calibrated post-SVD output MSE.

E.1 VISUAL QUALITY RESULTS

Across the evaluated models and precision settings, SVDQuant preserves or improves visual quality while exposing quality–latency trade-offs from precision, rank, and kernel optimization.

  • LoRA compatibility: The INT4 model matches the original BF16 image quality when applying community-contributed LoRAs.
  • Precision ablation: 21.5 PSNR is achieved by NVFP4+SVDQuant on FLUX.1-dev, closely matching the original 16-bit model.
  • Precision ablation: NVFP4 outperforms INT4, and adding a low-rank branch significantly enhances image quality across settings.GPTQ can further improve quality in most cases.
  • Quantitative quality: SVDQuant reaches LPIPS 0.322 for FLUX.1-dev BF16 in the reported comparison.
  • Qualitative results: NVFP4+SVDQuant yields the highest image fidelity among the precisions shown in the qualitative comparison.
  • Latency: Nunchaku removes redundant memory access, reducing SVDQuant’s unoptimized 18% latency overhead to latency comparable with naive INT4.The same comparison reports that 4-bit quantization is 1.3× faster than INT8.
  • Rank trade-off: Increasing rank from 16 to 64 improves image quality but increases parameter and latency overhead; rank 32 provides decent quality with minor overhead.
  • Bitwidth trade-off: SVDQuant consistently outperforms naive quantization and SmoothQuant across bitwidths on PixArt-Σ and FLUX.1-schnell.Its 4-bit results match 7-bit naive quantization on PixArt-Σ and 6-bit naive quantization on FLUX.1-schnell.
Loading 2411.05007v4…