PaperlayerView exact revision on arXiv

Source-linked AI summary

Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers

Shiwen Wang, Pengxiang Zhao, Xiaoming Yuan

arXiv:2610.11315v1cs.LGcs.CVmath.OC

TL;DR

Diffusion-transformer W4A4 post-training quantization faces a gap in jointly modeling low-rank compensation, residual quantization, and activation-side error. H-SVDQuant couples these in a calibration objective and derives optimization-based solvers. Rank-4 H-SVDQuant improves fidelity over rank-4 SVDQuant across diffusion backbones and matches or surpasses rank-32 SVDQuant on SANA and FLUX variants, with lower calibration cost.

  • Problem

    Existing low-rank PTQ approaches do not jointly model low-rank compensation, residual quantization, and activation-side quantization under a small rank budget.

    Source: §1 · p. 1

  • Method

    H-SVDQuant formulates low-rank-assisted W4A4 PTQ as a joint calibration problem and derives optimization-based solvers, including a deflated Hessian and an activation-noise surrogate.

    Source: §1 · pp. 1–2

  • Results

    Rank-4 H-SVDQuant consistently improves PSNR and LPIPS over rank-4 SVDQuant and matches or outperforms rank-32 SVDQuant on SANA-1.6B and FLUX variants.

    Source: §1 · p. 2, §5 · p. 6, §7 · p. 10

  • Takeaways & Limitations

    The reported results show that H-SVDQuant achieves improved W4A4 performance with substantially lower rank and quantization cost.

    Source: §1 · p. 2, §7 · p. 10

    The reconstruction objective's vanishing cross term relies on standard conditional zero-mean uniform-rounding assumptions.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

In diffusion transformers, low-rank branches can mitigate 4-bit weight--activation (W4A4) post-training quantization (PTQ) loss by decomposing each weight into a low-bit residual and a high-precision low-rank component. Existing low-rank PTQ approaches, however, either optimize low-rank compensation and residual quantization separately, often requiring higher ranks, or rely on second-order weight updates without explicitly modeling activation quantization error, which becomes particularly pronounced under 4-bit quantization. To address these limitations, we present \method{}, a unified framework modeling low-rank-assisted W4A4 PTQ as a coupled calibration problem and deriving optimization-based solvers from the joint objective. Eliminating the output-side low-rank factor yields a \emph{deflated Hessian} that discounts residual errors already captured by the low-rank component, while an activation-noise surrogate is incorporated to suppress activation quantization error. Across five diffusion backbones, rank-4 \method{} consistently outperforms rank-4 SVDQuant in PSNR and LPIPS. It further surpasses rank-32 SVDQuant on SANA-1.6B, FLUX.1-schnell, and FLUX.1-dev with an $8\times$ smaller rank and up to $6.25\times$ faster quantization. Furthermore, on the Qwen3-8B LLM, rank-4 \method{} improves MMLU accuracy from 61.50\% to 68.17\% over rank-32 SVDQuant. Overall, \method{} achieves better W4A4 performance with substantially lower rank and quantization cost.

1 INTRODUCTION

H-SVDQuant formulates low-rank-assisted W4A4 quantization as a coupled calibration problem and derives solvers that account for weight and activation quantization. Across diffusion backbones, rank-4 H-SVDQuant improves on rank-4 SVDQuant and matches or surpasses rank-32 SVDQuant on key models.

1 INTRODUCTION

  • Existing approaches do not jointly model low-rank compensation, residual quantization, and activation-side error under a small rank budget.W4A4 quantization can accumulate errors across layers and amplify them over denoising steps.Source: §1 · p. 1
  • Eliminating the output-side low-rank factor yields a deflated Hessian that removes errors compensable by the protected subspace from the residual objective.Residual quantization then focuses on errors outside that subspace.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The framework incorporates an activation-noise surrogate and diagonal smoothing to address activation quantization during calibration.The smoothing initialization uses geometric programming rather than costly grid searches or online rotation.Source: §1 · p. 2
  • Under W4A4, rank-4 H-SVDQuant consistently improves reconstruction fidelity across five diffusion backbones, reaching up to +1.61 dB PSNR on PixArt-Σ and rivaling or surpassing 8×-larger rank-32 SVDQuant on SANA-1.6B and FLUX models.On Qwen3-8B, rank-4 H-SVDQuant achieves 10.44 WikiText-2 PPL and 68.17% MMLU accuracy, versus rank-32 SVDQuant’s 11.21 PPL and 61.50% MMLU.Source: §1 · p. 2
  • The authors report three contributions: a joint W4A4 calibration objective, optimization-based solvers, and empirical validation under small-rank budgets.Offline PTQ calibration is accelerated by up to 6.25×.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses diffusion quantization, low-rank correction, and activation-aware calibration through distinct approaches. H-SVDQuant focuses on coupling low-rank compensation with activation-aware second-order information under small rank budgets.

2 RELATED WORK

  • Diffusion quantization methods address timestep variation, accumulated noise, activation outliers, and layer sensitivity through calibration, correction, mixed precision, and rotations.The related work describes Q-Diffusion, PTQD, ViDiT-Q, DiRotQ, and OrbitQuant.Source: §2 · p. 3
  • Low-rank PTQ methods differ in what they optimize, from SVD branches and joint corrections to quantization-error approximations and augmented-Hessian updates.The passage describes SVDQuant, LRC, LoRaQ, and GPTQ-intrinsic LoRA.Source: §2 · p. 3
  • Activation-aware calibration reshapes quantization using curvature, salient-weight scaling, diagonal rescaling, or orthogonal rotations.The passage covers GPTQ, AWQ, SmoothQuant, QuaRot, and SpinQuant.Source: §2 · p. 3
  • H-SVDQuant studies the coupled W4A4 formulation of low-rank compensation and activation-aware second-order information under small auxiliary-rank budgets.Its stated focus is efficient PTQ in diffusion transformers.Source: §2 · p. 3

3 METHOD

H-SVDQuant models low-rank-assisted W4A4 quantization as a coupled calibration objective and derives tractable updates for its factors, residual, and smoothing transform. Its implemented workflow combines deflation, residual quantization, and closed-form refitting.

3.1 PROBLEM MODELING

  • The low-rank-assisted operator keeps a rank-r high-precision branch separate from the quantized residual and activation path.The reconstruction objective includes remaining weight error and activation error; its cross term vanishes in expectation under conditional zero-mean uniform rounding assumptions.Source: §3.1 · pp. 3–4
  • A shared calibration objective combines weight-error minimization with activation-error suppression through the residual, with λA controlling the activation-aware term.The activation-noise covariance is represented by a block-constant diagonal surrogate.Source: §3.1 · p. 4

3.2 H-SVDQUANT

  • Analytically eliminating L2 reduces residual-code optimization to a second-order metric-and-target problem; Hessian deflation removes errors compensable by L1.The deflated metric directs residual quantization toward remaining reconstruction error.Source: §3.2 · p. 4
  • The method initializes the protected subspace with a calibration-weighted rank-r approximation and uses diagonal smoothing to redistribute channel ranges without changing the full-precision map.Smoothing scales are obtained through a closed-form ℓp relaxation of a grouped geometric-program objective.Source: §3.1 · p. 3, §3.2 · p. 5

3.3 IMPLEMENTED ALGORITHM

  • Each outer iteration updates smoothing, initializes L1, constructs the deflated Hessian, quantizes the residual, and refits L2 in closed form.For activation-aware calibration, a GPTQ-style solver uses MQ and TQ in place of conventional curvature and weight targets; small diagonal damping stabilizes practical solves.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUP

The evaluation tests H-SVDQuant on diffusion and language-model workloads under W4A4 PTQ, using matched calibration and teacher-output comparisons. It compares against PTQ baselines at matched and higher rank and reports fidelity, quality, and language-model metrics.

4 EXPERIMENTAL SETUP

  • The evaluation covers PixArt-Σ, SANA-1.6B, FLUX.1-dev, Qwen-Image, accelerated few-step variants, and Qwen3-8B.The listed diffusion backbones range from 0.6B to 20B parameters.Source: §4 · p. 5
  • Diffusion backbones use W4A4 PTQ with group size 64, rank 4, and per-token, per-group 4-bit activation quantization.Primary DiT experiments use reference activations with λA = 0, except PixArt-Σ at λA = 0.05.Source: §4 · p. 5
  • Generation fidelity is evaluated on fixed MJHQ-5K and sDCI-1K subsets using paired teacher outputs with matched seeds, latents, resolution, and scheduler steps.Diffusion calibration uses 128 prompts from the official SVDQuant repository.Source: §4 · p. 6
  • Baselines include rank-4 SVDQuant as the matched-capacity comparison and rank-32 SVDQuant as an 8×-larger-rank reference.The comparison also includes W4A4 DiRotQ, OrbitQuant, and ViDiT-Q adaptations.Source: §4 · p. 6
  • Diffusion metrics include PSNR, LPIPS, CLIP Score, and ImageReward; Qwen3-8B is assessed with WikiText-2 perplexity and five zero-shot accuracy benchmarks.PixArt-Σ ablations additionally report SSIM.Source: §4 · p. 6

5 MAIN RESULTS ON DIFFUSION TRANSFORMERS

Under W4A4, H-SVDQuant improves teacher fidelity over rank-4 SVDQuant across the evaluated diffusion backbones, with especially strong results on PixArt-Σ, SANA-1.6B, and FLUX.1-dev. It also preserves visual detail and reduces calibration time in the reported SANA and FLUX runs.

Quantitative Comparison

  • PSNR and LPIPS measure teacher fidelity, whereas CLIP and ImageReward measure standalone generation quality, so the best fidelity score need not imply the highest quality score.Table 1 reports W4A4 results across diffusion-transformer backbones.Source: §5 · p. 6, Table 1 · p. 8

Qualitative Comparison

  • H-SVDQuant preserves sharp object boundaries and subtle textures faithful to the 16-bit teacher, while OrbitQuant and ViDiT-Q show noise, distortion, and contrast loss.Figure 2 compares methods under identical prompts and seeds; its blue boxes highlight rank-4 H-SVDQuant.Source: §5 · pp. 6, 8, Figure 2 · p. 7

Quantitative Comparison

  • H-SVDQuant improves teacher fidelity over rank-4 SVDQuant on every evaluated backbone, with PixArt-Σ gains up to +1.61 dB PSNR and −0.099 LPIPS.On FLUX.1-dev, it also exceeds rank-32 SVDQuant in PSNR and LPIPS on both subsets; comparisons on Qwen-Image are mixed on MJHQ but improve both metrics on sDCI.Source: §5 · p. 6, Table 1 · p. 8

Rank Efficiency and Calibration Cost

  • On SANA-1.6B, recorded calibration time falls from 30.0 to 4.8 minutes; FLUX.1-schnell and FLUX.1-dev show 1.81× and 1.85× speedups.The reported times are offline PTQ calibration costs for the recorded base runs.Source: §5 · p. 8, Table 2 · p. 8

6 ANALYSIS AND ABLATION STUDIES

The ablations examine protected rank, activation-noise weighting, Hessian deflation and protected-subspace initialization, and diagonal smoothing. They report gains from the complete low-rank recipe, while showing that activation-noise weighting is model- and regime-dependent.

Rank Sweep Ablation

  • Rank-0 H-SVDQuant reaches 10.72 perplexity versus 11.21 for rank-32 SVDQuant; adding a small protected branch lowers perplexity to 10.44.The rank-4 configuration also has lower local and propagated activation error than the SVDQuant baselines; the passage attributes the result to the complete recipe, not one component.Source: §6 · p. 9, Table 4 · p. 9

Activation-Noise Sensitivity

  • On PixArt-Σ, λA = 0.05 gives 15.99 dB PSNR, 0.583 SSIM, and 0.414 LPIPS; on Qwen3-8B, it gives the lowest perplexity, 10.433.Larger weighting can regress performance: PixArt-Σ at λA = 0.50 falls slightly below the unweighted baseline, and the reported benefit depends on model and regime.Source: §6 · p. 9, Table 5 · p. 10

Hessian Deflation and Protected-Subspace Initialization

  • Combining calibration-weighted initialization with Hessian deflation gives the lowest Qwen3-8B PPL, 10.486, reducing baseline PPL by 0.073.In the controlled ablation, deflation lowers PPL with either initialization, and weighted initialization lowers PPL with either GPTQ metric.Source: §6 · p. 9, Table 6 · p. 10

Closed-Form Diagonal Smoothing

  • For W4A4, closed-form smoothing reduces PPL from 10.895 to 10.469 and raises average accuracy from 67.84 to 68.44.For W4A16, it reduces PPL from 10.078 to 9.976 and raises average accuracy from 71.45 to 71.61; the comparison also reports eliminating grid-search overhead.Source: §6 · p. 10, Table 7 · p. 10

7 CONCLUSION

H-SVDQuant jointly models low-rank compensation, residual quantization, and activation noise for W4A4 PTQ. Across diffusion transformers, rank 4 improves PSNR and LPIPS over rank-4 SVDQuant, with rank-32 comparisons and calibration-time savings also reported.

7 CONCLUSION

  • Rank-4 H-SVDQuant consistently improves PSNR and LPIPS over rank-4 SVDQuant, matches or outperforms rank-32 SVDQuant on SANA-1.6B and FLUX variants, and reduces calibration time by up to 6.25×.On Qwen3-8B, it also improves perplexity and downstream accuracy over rank-32 SVDQuant.Source: §7 · p. 10

A PROBLEM REFORMULATION AND ERROR DECOMPOSITION

  • The appendix reformulates calibration around reconstruction and quantization errors, holding the residual fixed while refitting L2 once codes are selected.It distinguishes activation and weight quantization errors and uses an additive-noise assumption under which their cross term vanishes in expectation.Source: A PROBLEM REFORMULATION AND ERROR… · p. 14, §A.2 · p. 14

A.1 SMOOTHING AND LOW-RANK DECOMPOSITION

  • Diagonal smoothing reparameterizes the linear map and transforms the calibration Gram matrix, while the quantized forward pass combines the low-rank branch with quantized activations and residual weights.The appendix defines D as a positive diagonal matrix and relates the smoothed Gram matrix to the original one.Source: §A.1 · p. 14

B ELIMINATION OF L2 AND HESSIAN DEFLATION

  • Eliminating the output-side low-rank factor yields a deflated metric that removes errors in the protected input subspace from the residual quantization objective.The low-rank branch can compensate for errors along that subspace, leaving the residual to account for remaining errors.Source: B ELIMINATION OF L2 AND HESSIAN… · pp. 15–16

C PROTECTED-SUBSPACE ALLOCATION / C.1 WEIGHTED RANK-r APPROXIMATION

  • The protected rank-r branch is allocated using calibration-weighted approximation, prioritizing directions that matter to layer output perturbations rather than treating all input directions equally.The weighted approximation protects directions important in the weight matrix and strongly excited by calibration activations.Source: §C.1 · pp. 16–17

C.2 HESSIAN EIGENSPACE LIMIT / C.3 BETA-FAMILY

  • Hessian deflation favors directions with the largest layer-output distortion, while the paper’s weight-aware β-family includes ordinary truncated SVD at β=0 and uses β=1/2 in its main recipe.The experiments fix β=1/2 and do not use an adaptive activation-rank split or global rank allocator.Source: §C.2 · p. 17, §C.3 · p. 18

D SMOOTHING OPTIMIZATION / D.1 GROUPED ACTIVATION-NOISE MODEL / D.2 GEOMETRIC-PROGRAM FORMULATION

  • The smoothing procedure models grouped activation noise and formulates the fixed-branch smoothing subproblem as a geometric program; its global-optimum claim applies to a converged modeled solution, not finite-step implementations or measured quantized-layer error.The grouped objective balances activation range against residual weight carried by each group.Source: §D.1 · p. 19, §D.2 · pp. 19–20

D.3 CLOSED-FORM RELAXATION

  • A closed-form ℓp relaxation provides a smoothing initialization whose stationary point is a global minimizer of that relaxation, followed by geometric-mean normalization before refinement.The square-root rule is exact for the relaxation; grouped maxima and the activation-noise term motivate later refinement.Source: §D.3 · p. 20

D.4 THE ACTIVATION-AWARE METRIC AND TARGET

The activation-aware objective eliminates the output-side low-rank factor to define a metric and target for residual quantization. The calibration algorithm iteratively updates smoothing and low-rank factors, quantizes the residual, and retains the best calibration state.

D.4 THE ACTIVATION-AWARE METRIC AND TARGET

  • The activation-aware metric combines the deflated Hessian with an activation-noise term, and its target is obtained using the metric pseudoinverse.At zero activation-noise weight, the metric is generally singular, so an ordinary inverse requires regularization.Source: §D.4 · p. 20
  • The weight-error code path quantizes an initialized residual under a stabilized deflated Hessian, while the activation-aware path uses a target and metric derived from the chosen target.Their added penalties are centered differently, so equal continuous undamped losses need not yield equal grids, greedy updates, or codes.Source: §D.4 · p. 21
  • The zero-mean conditional noise surrogate removes the exact error's cross term, while the diagonal range-based covariance omits off-diagonal correlations and does not exactly model four-bit rounding.The deployed activation error multiplies the quantized residual, not the unquantized provisional residual.Source: §D.4 · p. 21

E ALGORITHM

  • Calibration runs two update iterations, refining smoothing and low-rank initialization before residual quantization, low-rank refitting, and FP32 output-MSE evaluation.It retains the best visited state on cached calibration inputs and returns the corresponding smoothing, low-rank factors, and quantized residual.Source: E ALGORITHM · p. 21

F EVALUATION ON ACCELERATED DIFFUSION BACKBONES

The accelerated-backbone evaluation measures W4A4 fidelity and generation quality for four-step FLUX.1-schnell and Qwen-Image Lightning. The reported metrics capture distinct trade-offs, rather than a single measure of performance.

F EVALUATION ON ACCELERATED DIFFUSION BACKBONES

  • Table 8 evaluates W4A4 on four-step FLUX.1-schnell and Qwen-Image Lightning, complementing base-model evaluations.It reports teacher fidelity alongside generation-quality scores.Source: F EVALUATION ON ACCELERATED DIFFUSION… · p. 21, Table 8 · p. 22
  • The method with the best PSNR need not have the highest CLIP or ImageReward score, so teacher fidelity and standalone generation quality are distinct properties.CLIP and ImageReward are absolute scorer means, not teacher-relative differences.Source: F EVALUATION ON ACCELERATED DIFFUSION… · p. 21, Table 8 · p. 22
  • Table 8 marks the best and second-best unrounded means per subset and metric; H-SVDQuant uses group-64 residuals and DiRotQ uses matched A4/group-64.Both models use four-step sampling on the specified dataset.Source: Table 8 · p. 22

G QUALITATIVE RESULTS OF QWEN-IMAGE

The Qwen-Image qualitative examples compare reconstructions under matched prompts and seeds, while the paired-distribution figure summarizes fidelity differences across five diffusion backbones. The figures describe complementary visual and distribution-level comparisons.

G QUALITATIVE RESULTS OF QWEN-IMAGE

  • Figures 3 and 4 show representative reconstructions for the 50-step base model and four-step Lightning variant, evaluated against DiRotQ and both SVDQuant ranks under identical prompts and seeds.Figure 3's first column is the high-precision reference, and its blue outline identifies H-SVDQuant; Figure 4 follows the same column order.Source: G QUALITATIVE RESULTS OF QWEN-IMAGE · p. 22, Figure 3 · p. 23, Figure 4 · p. 24
  • The Qwen-Image examples complement complete-subset results in Table 1, while accelerated quantitative comparisons for Lightning appear in Table 8.Figure 3 shares the displayed prompt within each row.Source: Figure 3 · p. 23, Figure 4 · p. 24

H FULL PAIRED PROMPT DISTRIBUTIONS

  • Figure 5 compares rank-4 H-SVDQuant with rank-4 and rank-32 SVDQuant across five backbones, with positive values favoring H-SVDQuant.Rows show PSNR gain and LPIPS reduction; columns separate baseline rank, and colors distinguish MJHQ from sDCI.Source: Figure 5 · p. 25, H FULL PAIRED PROMPT DISTRIBUTIONS · p. 25
  • The Figure 5 distributions use identical prompts and seeds, with boxes for interquartile ranges, white median bars, 5th–95th percentile whiskers, and black mean dots.The compared checkpoints retain their respective activation quantizers and are complete-recipe comparisons.Source: Figure 5 · p. 25

I BETWEEN-BLOCK AND WITHIN-BLOCK PROPAGATION PILOT

The calibration replay pilot compares rank-4 W4A4 checkpoints on matched MJHQ prompts. Its results do not establish a PixArt ranking, and its FLUX example illustrates that local module MSE may not predict full-image behavior.

I BETWEEN-BLOCK AND WITHIN-BLOCK PROPAGATION PILOT

  • The replay uses rank 4, W4A4/group-64, λA = 0, and the weight-error code path; each model is evaluated on the same 16 MJHQ prompts and image identifiers.These pilot checkpoints are not substituted into Table 1.Source: Table 9 · p. 25
  • PixArt's two PSNR differences versus reference/sequential were −0.177 dB (95% paired interval [−0.839, 0.412]) and +0.133 dB ([−0.239, 0.529]); the pilot establishes no ranking.Source: I BETWEEN-BLOCK AND WITHIN-BLOCK… · p. 25
  • For FLUX-schnell, the corresponding PSNR differences were −0.232 dB ([−0.500, 0.016]) and −0.644 dB ([−1.395, −0.003]); both LPIPS intervals include zero.Source: I BETWEEN-BLOCK AND WITHIN-BLOCK… · p. 25
  • Despite worse image fidelity in the independent FLUX run, its largest local module MSE was slightly smaller, illustrating local MSE's limits as a predictor of full-image behavior.Local errors were measured on each route's own calibration activations, not as paired errors on a shared held-out activation set.Source: I BETWEEN-BLOCK AND WITHIN-BLOCK… · p. 25
Loading 2610.11315v1…