PaperlayerView exact revision on arXiv

Source-linked AI summary

SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, Song Han

arXiv:2411.05007v4cs.CVcs.LG

TL;DR

Scaling diffusion models raises deployment memory and latency barriers, while 4-bit quantization is challenged by outliers in both weights and activations. SVDQuant uses a low-rank branch to absorb outliers, and Nunchaku fuses its kernels with the low-bit branch; on FLUX.1, the paper reports 3.5× lower memory use and 3.0× speedup over W4A16.

  • Problem

    Diffusion-model scaling raises memory and latency barriers, while existing smoothing methods are less effective for W4A4 because both weights and activations are outlier-sensitive.

    Source: §1 · p. 2

  • Method

    SVDQuant shifts activation outliers into weights, uses a 16-bit low-rank branch to absorb weight outliers, and quantizes the residual; Nunchaku fuses low-rank and low-bit kernels.

    Source: §4.2 · p. 5, §6 · p. 10

  • Results

    3.5× reduction in memory usage over the original 16-bit model and 3.0× speedup over W4A16 on NVIDIA RTX 4090 and 5090 GPUs; experiments show SVDQuant preserves image quality.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors conclude this enables efficient deployment of large-scale diffusion models on edge devices, with potential for interactive AI applications.

    Source: §6 · p. 10

    Running the low-rank branch separately incurs latency overhead of approximately 50% of the 4-bit branch latency.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Diffusion models can effectively generate high-quality images. However, as they scale, rising memory demands and higher latency pose substantial deployment challenges. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where existing post-training quantization methods like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing, which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights. Then, we use a high-precision, low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD), while a low-bit quantized branch handles the residuals. This process eases the quantization on both sides. However, naively running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without re-quantization. Extensive experiments on SDXL, PixArt-$Σ$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5$\times$, achieving 3.0$\times$ speedup over the 4-bit weight-only quantization (W4A16) baseline on the 16GB laptop 4090 GPU with INT4 precision. On the latest RTX 5090 desktop with Blackwell architecture, we achieve a 3.1$\times$ speedup compared to the W4A16 model using NVFP4 precision.

1 INTRODUCTION

Diffusion models’ growing computational demands make deployment difficult, while weight-only quantization cannot speed up their compute-bound inference. SVDQuant addresses aggressive 4-bit weight-and-activation quantization by moving outliers into a low-rank branch and fusing its computation with low-bit kernels.

1 INTRODUCTION

  • Diffusion models scale from 800M-parameter SD 1.4 to 12B-parameter FLUX.1, and their computational costs rise more rapidly with model size, creating memory and latency barriers.Because diffusion inference is compute-bound even at batch size one, weight-only quantization cannot accelerate it; weights and activations must use the same bit width to avoid upcasting.Source: §1 · p. 2
  • SVDQuant migrates outliers from activations to weights, then uses SVD to isolate a 16-bit low-rank branch so the residual can be quantized to 4 bits.The approach targets outlier sensitivity on both sides, where smoothing alone is less effective.Source: §1 · p. 2, Figure 3 · p. 3
  • Nunchaku fuses low-rank branch computation into 4-bit quantization and computation kernels to avoid memory-access overhead that can offset inference speedup.This co-designed inference engine enables measured speedup despite the additional branch.Source: §1 · pp. 2–3
  • 3.5× lower memory use than the original BF16 model, with INT4 and FP4 delivering 3.0× and 3.1× speedups over the NF4 weight-only baseline on RTX 4090 and RTX 5090, respectively.These results are reported for the 12B FLUX.1 model; the RTX 4090 is the 16GB laptop-level GPU and the RTX 5090 is the desktop-level GPU.Source: §1 · p. 3
  • SVDQuant supports UNet and DiT architectures, INT4 and FP4, and pretrained LoRAs without re-quantization while maintaining visual quality.The introduction describes this as the first 4-bit post-training quantization of both diffusion-model weights and activations with measured speedup on NVIDIA GPUs.Source: §1 · p. 3

2 RELATED WORK

Prior diffusion acceleration work spans sampling, architecture, sparsity, distribution, and quantization, while diffusion quantization research had largely focused on 8-bit methods. This work targets 4-bit inference and uses Nunchaku to fuse low-rank branches, distinguishing its inference goal from prior compression and fine-tuning approaches.

2 RELATED WORK

  • Prior diffusion acceleration methods include few-step sampling, distillation, efficient architectures, quantization, sparse inference, and distributed inference.The paper focuses on 4-bit quantization and notes that it can also be applied to few-step diffusion models to further reduce latency (Section 5.2).Source: §2 · p. 3
  • Diffusion quantization progressed from initial 8-bit methods to sensitivity-aware, timestep-aware, and broader model and training settings, but only MixDQ and ViDiT-Q report measured 8-bit GPU speedups.The initial methods were Q-Diffusion and PTQ4DM; later work extended quantization to text-to-image and DiT models, training, video, and other data types.Source: §2 · p. 3
  • Unlike low-rank compression and quantized-model fine-tuning, this work targets inference acceleration through joint weight-and-activation quantization and Nunchaku kernel fusion.Prior direct weight decomposition can degrade performance, while prior quantization-plus-low-rank work uses a branch to compensate for quantization error; the paper reports that directly applying these methods degrades image quality and adds overhead.Source: §2 · p. 4
  • Nunchaku supports integer or floating-point 4-bit quantization across UNet and DiT backbones and runs LoRA as a separate low-rank branch without re-quantization.Kernel fusion reduces the memory-access overhead that otherwise motivates fusing LoRA into the main branch and re-quantizing.Source: §2 · p. 4

3 QUANTIZATION PRELIMINARY

The section introduces low-bit quantization notation and explains why SVDQuant focuses on W4A4 despite outlier-related obstacles and limitations of existing suppression methods.

Quantization Preliminary

  • For 4-bit floating-point quantization with a 1-bit mantissa and 2-bit exponent, qmax is 6, and dequantization multiplies the low-bit tensor by its scaling factor.For signed k-bit integer quantization, qmax = 2k−1 −1; linear-layer computation is approximated using quantized inputs and weights.Source: §3 · p. 4
  • Matching weight and activation bit widths lets modern GPUs avoid upcasting that would negate low-bit speed gains; WxAy denotes x-bit weights and y-bit activations.The same approximation applies to convolutional layers; INT and FP denote integer and floating-point types.Source: §3 · p. 4
  • W4A4 outliers challenge quantization; QAT is resource-intensive for models above 10B parameters, while adaptive normalization makes rotation inapplicable without costly online overhead.Runtime-generated normalization weights prevent offline rotation with projection weights, and online rotation of activations and weights adds significant runtime overhead. Prior work includes QAT (He et al., 2024) and rotation (Ashkboos et al., 2024; Liu et al., 2024c; Lin et al., 2025).Source: §3 · p. 4

4 METHOD

SVDQuant targets 4-bit diffusion quantization by using a low-rank branch to absorb outliers, then fusing its kernels with the low-bit branch in Nunchaku to reduce memory overhead.

4.1 PROBLEM FORMULATION

  • Quantization error is bounded by input and weight magnitudes and their respective quantization errors, so the formulation seeks to optimize all four terms.Source: §4.1 · p. 5

4.2 SVDQUANT: ABSORBING OUTLIERS VIA LOW-RANK BRANCH

  • Smoothing lowers activation magnitude and outliers but increases them in weights, limiting overall quantization-error reduction.The method therefore seeks to reduce both input and weight magnitudes and their respective quantization errors.Source: §4.2 · p. 5, Figure 4 · p. 5
  • A 16-bit low-rank branch absorbs weight outliers while the residual is quantized to 4 bits; the rank is typically 16 or 32.SVDQuant decomposes the smoothed weight into low-rank factors and a residual; the residual has fewer outliers and a narrower value range.Source: §4.2 · pp. 5–6, Figure 4 · p. 5
  • SVD selects the dominant singular values to reduce residual magnitude, and iterative updates can further reduce quantization error.The method decomposes the transformed weight and selects the result with the smallest error after several iterations.Source: §4.2 · p. 6

4.3 NUNCHAKU: FUSING LOW-RANK AND LOW-BIT BRANCH KERNELS

  • Kernel fusion reduces the low-rank branch’s latency overhead from approximately 50% of the 4-bit branch latency to 5–10%.Nunchaku fuses the down projection with quantization and the up projection with 4-bit computation, sharing activations and eliminating extra memory access.Source: §4.3 · pp. 6–7, Figure 6 · p. 6

5 EXPERIMENTS

Across diffusion-model benchmarks, SVDQuant preserves image quality at low precision while reducing FLUX.1 memory use and inference time; Nunchaku also supports off-the-shelf LoRAs without requantization.

5.1 SETUPS

  • The evaluation spans FLUX.1, PixArt-Σ, SANA, SDXL, and SDXL-Turbo, measuring image quality and similarity against 16-bit results.Calibration prompts are sampled from COCO Captions 2024, with 5K prompts from MJHQ-30K and sDCI used for benchmarking. Metrics include FID, Image Reward, LPIPS, and PSNR; baselines include NF4, ViDiT-Q, MixDQ, and TensorRT.Source: §5.1 · p. 7

5.2 RESULTS

  • SVDQuant’s 8-bit results closely match 16-bit quality, with PSNR above 21 and FLUX.1-dev INT8 PSNR reaching 27 on MJHQ.The reported PSNR values are higher than 21 for the 8-bit results and 27 for FLUX.1-dev INT8 on MJHQ.Source: §5.2 · p. 7, Table 1 · p. 8
  • SVDQuant 4-bit surpasses NF4 W4A16 on every FLUX.1 metric, ViDiT-Q W4A8 on PixArt-Σ, and MixDQ W4A8 on SDXL-Turbo, where FID matches FP16; on SDXL it matches TensorRT W8A8 quality.Source: §5.2 · p. 8
  • SVDQuant reduces 12B FLUX.1 model size 3.6×, from 22.2 GiB to 6.1 GiB, and 16-bit memory usage 3.5×; Nunchaku achieves 3.0× and 3.1× speedups.The 3.0× speedup is on desktop and laptop RTX 4090 GPUs with INT4; the 3.1× speedup is on RTX 5090 with NVFP4. On the 16GB laptop RTX 4090, INT4 avoids CPU offloading for a 10.1× speedup.Source: §5.2 · pp. 8–9, Figure 8 · p. 9
  • Nunchaku integrates off-the-shelf LoRAs without requantization, and the resulting INT4 FLUX.1-dev images match the original 16-bit model’s quality.The low-rank branch can absorb the LoRA branch by slightly increasing its rank; examples cover five styles, and FLUX.1-schnell also supports pix2pix-turbo LoRAs for controls such as sketch.Source: §5.2 · p. 9, Figure 9 · p. 10
  • On PixArt-Σ, SVDQuant’s decomposition and outlier-handling steps substantially improve image quality over SVD-only, naïve, smoothed, and LoRC alternatives.SVDQuant decomposes weights and quantizes the residual, shifts the largest singular values to a low-rank branch, then uses smoothing to consolidate outliers for absorption; LoRC’s low-rank compensation is less effective because quantization errors are spread across singular values.Source: §5.2 · p. 10, Figure 10 · p. 10

6 CONCLUSION

SVDQuant uses a low-rank branch to absorb weight and activation outliers for 4-bit diffusion-model quantization, while Nunchaku fuses low-rank and low-bit kernels. The paper reports preserved image quality and more efficient deployment of large-scale diffusion models.

6 CONCLUSION

  • 3.5× lower memory usage than the original 16-bit model and 3.0× speedup over W4A16 are reported for Nunchaku on NVIDIA RTX 4090 and 5090 GPUs.The conclusion presents these results as enabling efficient deployment of large-scale diffusion models on edge devices.Source: §6 · p. 10
  • SVDQuant uses a low-rank branch to absorb outliers in both weights and activations, easing 4-bit quantization of diffusion models.The conclusion states that extensive experiments demonstrate preserved image quality.Source: §6 · p. 10
  • Nunchaku fuses the low-rank and low-bit branch kernels to reduce memory usage and redundant data-movement overhead.The paper says this supports efficient deployment and broader potential for interactive AI applications.Source: §6 · p. 10

CHANGELOG · A PROOFS

The changelog records the initial preprint, a typo correction, and a camera-ready update that combines SVDQuant with GPTQ and updates NVFP4 and SANA results.

CHANGELOG

A.1 PROOF OF PROPOSITION 4.1 · A.2 PROOF OF PROPOSITION 4.2

The appendix bounds the combined weight-and-activation quantization error and outlines a distribution-dependent analysis, including a normal-distribution case.

A.1 PROOF OF PROPOSITION 4.1

  • The combined error is bounded by a weight-quantization term plus an activation-quantization term scaled by the weight norm and weight quantization error.The bound is E(X, W) ≤ ||X||F ||W − Q(W)||F + ||X − Q(X)||F (||W||F + ||W − Q(W)||F).Source: §A.1 · p. 17
  • The proof splits the product error into weight and activation perturbations, then bounds the quantized weight norm using the triangle inequality.It applies the Frobenius norm product bound to each term in the decomposition.Source: §A.1 · p. 17

A.2 PROOF OF PROPOSITION 4.2

  • Proposition 4.2 considers quantization of a tensor R as Q(R) = sR · QR and assumes a regularity condition with constant c.The supplied passage says size(R) denotes the number of elements, but does not include the full regularity condition.Source: §A.2 · p. 17
  • The proof specializes its distribution-dependent analysis to normally distributed tensor elements.The supplied text does not include the full regularity condition or the resulting bound.Source: §A.2 · p. 17
  • For the normal case, Equation 12 uses the maximal inequality for Gaussian variables, with σ denoting the distribution’s standard deviation.The passage attributes the inequality to Lemma 2.3 in Massart (2007).Source: §A.2 · p. 18
  • The subsequent derivation invokes Cauchy–Schwarz and the expectation of a half-normal distribution, then combines the results for the normal case.The supplied passages omit the equations and the explicit resulting bound.Source: §A.2 · p. 18

B BENCHMARK MODELS

The benchmark covers six text-to-image models: FLUX.1-dev and FLUX.1-schnell, PixArt-Σ, SANA, SDXL, and SDXL-Turbo, spanning DiT- and UNet-based architectures.

B BENCHMARK MODELS

  • The six benchmark models are FLUX.1-dev, FLUX.1-schnell, PixArt-Σ, SANA, SDXL, and SDXL-Turbo.The benchmark introduces these as six text-to-image models.Source: B BENCHMARK MODELS · p. 18
  • FLUX.1 is a 12B-parameter DiT with 19 joint and 38 parallel attention blocks; evaluation uses its 50-step dev and 4-step schnell variants.The paper describes FLUX.1 as a state-of-the-art open-sourced DiT-based diffusion model.Source: B BENCHMARK MODELS · p. 18
  • PixArt-Σ has 600M parameters across 28 self-attention, cross-attention, and feed-forward blocks, evaluated at 20 steps.It is a DiT-based model that stacks attention blocks rather than using joint attention.Source: B BENCHMARK MODELS · p. 18
  • SANA (Xie et al., 2025) is a 1.6B-parameter DiT using a 32× compression autoencoder and linear attention instead of Softmax attention.Source: B BENCHMARK MODELS · p. 18
  • SDXL is a 2.6B-parameter UNet evaluated at 30 steps and as the 4-step SDXL-Turbo variant (Podell et al., 2024; Sauer et al., 2024).Its three-resolution noise-prediction stages use ResBlocks throughout, with attention layers in the two lower-resolution stages; text conditioning uses cross-attention.Source: B BENCHMARK MODELS · p. 18

C BENCHMARK DATASETS

To assess generalization, the benchmark uses two prompt sets with different styles: one for artistic image generation and one for realistic image generation.

C BENCHMARK DATASETS

  • 30K Midjourney samples across 10 categories (3K each) form MJHQ-30K; 5K randomly selected prompts evaluate artistic image generation.Source: C BENCHMARK DATASETS · p. 18
  • ∼8K images with detailed human-annotated captions averaging over 1,000 words make up DCI; experiments use LLM-summarized 77-token captions and 5K randomly sampled prompts for realistic generation (Urbanek et al., 2024).Source: C BENCHMARK DATASETS · pp. 18–19

D IMPLEMENTATION DETAILS … E.2 LORA RESULTS

The appendix details quantization and smoothing choices, reports additional quality metrics and visual comparisons, and shows INT4 models with community LoRAs matching BF16 image quality. These passages provide implementation settings and supplementary results across the evaluated model families.

D IMPLEMENTATION DETAILS

  • The 4-bit setup uses per-group symmetric activation and weight quantization, rank-32 low-rank branches, and group size 64 with 16-bit scales for INT4.The 8-bit setup uses per-token dynamic activation quantization, per-channel weight quantization, and rank-16 low-rank branches; GPTQ quantizes residual weights. NVFP4 uses group size 16 with FP8 scales on Blackwell GPUs (NVIDIA Corporation, 2025).Source: D IMPLEMENTATION DETAILS · p. 19
  • For FLUX.1, adaptive-normalization linear-layer inputs remain 16-bit; other models retain 16-bit cross-attention key and value projections, whose latency is under 5% of total runtime.The latter exception applies to models other than FLUX.1.Source: D IMPLEMENTATION DETAILS · p. 19
  • The smoothing factor is computed per channel using activation and weight maxima, with migration strength α selected offline per layer to minimize post-SVD output MSE on calibration data.The method follows SmoothQuant (Xiao et al., 2023).Source: D IMPLEMENTATION DETAILS · p. 19

E.1 VISUAL QUALITY RESULTS

  • Additional quality comparisons use CLIP IQA for image quality, CLIP Score for text-image alignment, and SSIM against the original 16-bit model; Table 2 presents the quantitative comparisons.The appendix also provides qualitative results for FLUX.1-dev, FLUX.1-schnell, PixArt-Σ, SDXL, and SDXL-Turbo on MJHQ.Source: §E.1 · p. 20, Table 2 · p. 20, Figure 11 · p. 21
  • Figure 15 presents qualitative visual results of SDXL-Turbo on MJHQ, with the caption identifying the model, dataset, figure, and qualitative nature of the displayed results.Source: Figure 15 · p. 23

E.2 LORA RESULTS

  • The five community-contributed LoRAs cover Realism, Ghibsky Illustration, Anime, Children Sketch, and Yarn Art styles on INT4 models.The results are presented in Figure 16.Source: §E.2 · p. 24, Figure 16 · p. 24
  • Figure 16 reports that the INT4 FLUX.1-dev model matches the original BF16 model’s image quality when community LoRAs are applied.The figure shows additional LoRA results; prompts are detailed in Appendix F.Source: Figure 16 · p. 24

E.3 ADDITIONAL ABLATION OF SVDQUANT … F TEXT PROMPTS

The additional ablations report image-quality, latency, rank, and bitwidth trade-offs for SVDQuant; the final appendix subsection lists prompts used for figures 9 and 16.

E.3 ADDITIONAL ABLATION OF SVDQUANT

  • NVFP4+SVDQuant reaches a PSNR of 21.5 on FLUX.1-dev, the best reported result and close to the original 16-bit model.Across settings, the low-rank branch absorbs outliers and enhances image quality; GPTQ instead of round-to-nearest further improves quality in most cases.Source: Table 3 · p. 25, §E.3 · p. 25

E.4 LATENCY RESULTS

  • 4-bit quantization delivers a 1.3× speedup over INT8, while unoptimized SVDQuant incurs 18% overhead and Nunchaku reduces latency to comparable with naive INT4.The passage attributes the overhead reduction to eliminating redundant memory access.Source: §E.4 · p. 27, Table 4 · p. 27

E.5 TRADE-OFF OF INCREASING RANK

  • Increasing rank from 16 to 64 improves image quality but raises parameter and latency overhead; the experiments select rank 32 for decent quality with minor overhead.Figure 18 presents this trade-off on PixArt-Σ.Source: §E.5 · p. 27, Figure 18 · p. 27

E.6 TRADE-OFF BETWEEN QUALITY AND BITWIDTH

  • On PixArt-Σ and FLUX.1-schnell, SVDQuant’s 4-bit results match 7-bit and 6-bit naive quantization, respectively.The comparison evaluates LPIPS on MJHQ; above 4 bits it uses per-channel quantization, and at 4 bits or below it uses per-group quantization with group size 64.Source: §E.6 · p. 27, Figure 19 · p. 27
  • SVDQuant generates images at 3 bits on both PixArt-Σ and FLUX.1-schnell, outperforming SmoothQuant, but cannot generate images below that precision.The passage attributes the limitation to 2-bit symmetric quantization being essentially ternary, which prior work says requires quantization-aware training.Source: §E.6 · p. 27

F TEXT PROMPTS

  • The cited material specifies that the text prompts used for Figure 16 are presented in rasterizing order.Source: F TEXT PROMPTS · p. 28

Related papers

Loading 2411.05007v4…