PaperlayerView exact revision on arXiv

Source-linked AI summary

TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization

Haodong Wang, Junjie Liu, Zicong Hong, Qianli Liu, Jian Lin, Song Guo, Xu Chen

arXiv:2606.01556v1cs.DC

TL;DR

Aggressive 4-bit quantization can degrade LLM accuracy, and prior weight decompositions do not optimize the post-quantization errors of both components. TwinQuant learns quantization-friendly low-rank and residual subspaces and a fused kernel; across mainstream models it reports near-FP16 accuracy with up to 1.8× speedup.

  • Problem

    Prior decompositions target real-valued residual energy rather than the post-quantization errors of both components, while aggressive low-bit quantization can reduce accuracy.

    Source: §1 · pp. 1–2

  • Method

    TwinQuant learns a quantization-friendly split into low-rank and residual components and jointly optimizes component-specific transformations.

    Source: §1 · p. 2

  • Results

    0.6–2.6 average-accuracy points from FP16 is the reported 4-bit accuracy gap, with up to 1.8× speedup across mainstream LLMs and GPUs.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The reported framework improves low-precision LLM inference throughput without sacrificing accuracy across mainstream models and GPUs.

    Source: §7 · p. 9

    Evaluation mainly covers dense language models; effectiveness on MoE and multimodal models and under ultra-long contexts requires further study.

    Source: §6 · p. 9

Abstract

from arXiv · show

4-bit quantization reduces the memory footprint and latency of large language model inference, but its aggressive precision reduction can severely degrade accuracy. Prior methods address this by decomposing each weight matrix into two components (e.g., via singular value decomposition) and quantizing them separately, assigning the bulk of values to a low-precision residual component while handling outliers with a high-precision low-rank component. However, such decompositions are designed to minimize the real-valued energy of the residual, rather than the post-quantization error of the residual and low-rank components. We propose TwinQuant, a 4-bit quantization framework that learns quantization-friendly decomposed subspaces and jointly reshapes both the low-rank and residual components. TwinQuant learns component-specific transformations via a joint optimization over the Stiefel and general linear manifolds, flattening their distributions and reducing dynamic-range imbalance. To enable efficient end-to-end execution, we further design a fused dual-component kernel that pipelines the two-stage low-rank computation on-chip and merges both components with a single epilogue, avoiding intermediate global-memory traffic. Across LLaMA3 and Qwen3 models, TwinQuant preserves near-FP16 accuracy and delivers up to $1.8\times$ end-to-end speedup over an FP16 baseline.

1. Introduction

TwinQuant addresses the accuracy and deployment costs of aggressive low-bit LLM quantization by learning a quantization-friendly split of each weight into low-rank and residual components. It combines component-specific transforms and a fused kernel; reported evaluations preserve near-FP16 accuracy with speedups up to 1.8×.

1. Introduction

  • Aggressive low-bit quantization distorts heavy-tailed, anisotropic weights and activations because a few directions dominate the available 4-bit scales.Coordinate redistribution alone does not remove this intrinsic anisotropy.Source: §1 · p. 1
  • Small-rank SVD cannot absorb enough LLM outliers, while increasing rank adds substantial memory and compute overhead.The paper identifies this rank trade-off as a limitation of prior decomposition approaches.Source: §1 · p. 1
  • TwinQuant learns a limited-rank split into complementary low-rank and residual components, reshaping both to reduce 4-bit distortion.The decomposition aims to improve quantization friendliness without increasing the truncation rank or adding substantial compute and memory overhead.Source: §1 · p. 2
  • Joint optimization over the Stiefel and general linear manifolds learns component-specific transforms to minimize end-to-end quantization error.The transforms can be folded into weights offline, avoiding additional inference overhead.Source: §1 · p. 2
  • A fused dual-component kernel pipelines the two-stage low-rank computation on-chip and merges both components in one epilogue.This design reduces kernel launches and avoids intermediate global-memory traffic.Source: §1 · p. 2
  • TwinQuant preserves near-FP16 accuracy while achieving up to 1.8× speedup over an FP16 baseline.The reported evaluations include mainstream LLMs and workstation- and server-class GPUs.Source: §1 · p. 2

2. Background & Related Work

The background reviews symmetric quantization, scaling, and low-rank decomposition, then distinguishes TwinQuant's learnable decomposed subspaces from fixed decompositions and single-space calibration methods.

2. Background & Related Work

  • Symmetric quantization represents tensor X with integer QX and scale sX, then dequantizes it as Q(X) = sX QX.QX and QW usually use the same bit-width to exploit low-precision compute units without on-the-fly dequantization.Source: §2 · p. 2
  • Per-channel scaling balances activation and weight magnitudes, with scaling factors selected by grid search or learned as parameters.The factors can be folded into model weights offline.Source: §2 · p. 2
  • SVD-based decomposition places outliers in a low-rank component and quantizes the remaining residual at low precision.The low-rank branch adds parameters and computation proportional to r(m + n).Source: §2 · pp. 2–3
  • Learnable quantization methods can calibrate clipping thresholds or learn rotations, while TwinQuant instead learns a decomposed subspace for 4-bit quantization.TwinQuant's stated goal is to quantize the low-rank component as well as learn a quantization-friendly decomposition, rather than rely on fixed offline SVD.Source: §2 · p. 3

3. Motivation

The motivation is that LLM weight spectra decay slowly, so small-rank decompositions leave difficult residuals while larger ranks increase overhead. Directly quantizing SVD low-rank components also produces skewed errors, motivating learnable transformations.

3. Motivation

  • LLaMA3-8B singular-value spectra remain relatively flat beyond rank 256, so a small low-rank component cannot capture all substantial-energy directions.This contrasts with the rapid spectral decay assumption used by SVD-based approaches for diffusion models.Source: §3 · p. 3
  • Rank 32 leaves substantial residual quantization errors, while rank 256 reduces them but raises parameter, compute, and memory-access costs.For the LLaMA3-8B qproj layer, the low-rank parameters rise from about 6.25% to about 50% of 4-bit residual storage across those ranks.Source: §3 · p. 3
  • Direct 4-bit quantization of SVD low-rank components creates skewed errors, and a fixed Hadamard transform helps little across layers with varying statistics.The passage attributes poor representation to scale-imbalanced, heavy-tailed low-rank components and motivates learnable subspace decomposition.Source: §3 · p. 3

4. Method

TwinQuant learns transforms that reshape low-rank and residual components for quantization, then optimizes them under structural constraints. Its fused kernel pipelines the low-rank computation and combines both branches in one epilogue.

4. Method

  • The method section presents a learnable decomposition and derives an error decomposition to motivate its transforms and fully 4-bit execution.The framework overview says the transforms are folded offline, enabling merged 4-bit inference without extra runtime overhead.Source: §4 · p. 4, Figure 3 · p. 5

4.1. Theory-Guided Re-parameterization over Dual Decomposed Subspaces

  • TwinQuant learns global orthogonal and layer-specific invertible transforms to reshape activations, residuals, and low-rank factors for lower quantization error.The calibration reconstruction loss measures the joint effect of activation, low-rank, and residual quantization errors on layer output; transformed factors can be preprocessed offline.Source: §4.1 · p. 4

4.2. Joint Optimization on Stiefel and General Linear Manifolds

  • The hybrid optimization keeps global rotations orthogonal while allowing each layer’s invertible transform additional scaling and shearing degrees of freedom.A three-stage schedule trains global rotations, then the invertible transform, then all parameters jointly to reduce interference and yield smoother convergence.Source: §4.2 · p. 5

4.3. Dual-Component Kernel Design

  • The fused kernel keeps the low-rank intermediate on-chip, runs both components in 4-bit, and combines their outputs in a shared epilogue.It avoids writing and reloading the intermediate from global memory.Source: §4.3 · p. 6

5. Experiments

Across LLaMA3 and Qwen3, TwinQuant reports stronger W4A4 accuracy than prior methods and faster inference than FP16 TensorRT-LLM. Experiments also examine the rank trade-off, component-wise quantization error, and low-rank overhead.

5.1. Setups

  • The evaluation covers six LLaMA3 and Qwen3 models, six zero-shot tasks, WikiText2 perplexity, eight PTQ baselines, and two GPU environments; both weights and activations use group-wise quantization with groups of 128.Source: §5.1 · p. 6

5.2. Accuracy Results

  • Under W4A4, TwinQuant improves average zero-shot accuracy over SVDQuant by 0.9–2.7 points across models and reduces perplexity by 0.6–3.1 points across models.Source: Table 1 · p. 7, §5.2 · p. 7

5.4. Sensitivity Analysis of the Rank Number.

  • From rank 32 to 128, average zero-shot accuracy rises from 68.0 to 70.9 and perplexity falls from 11.3 to 9.4; rank 256 adds only 0.2 accuracy and 0.1 perplexity improvement.The reported low-rank memory and latency overheads increase from 4.9% and 4.3% at rank 128 to 9.9% and 8.7% at rank 256.Source: §5.4 · p. 7, Table 2 · p. 8, Figure 6 · p. 8

5.3. Speed Measurement

  • 1.63–1.8× acceleration over FP16 TensorRT-LLM in Environment 1 and 1.32–1.73× in Environment 2 is reported for TwinQuant.Figure 5 measures throughput across batch sizes at 1024 input and 256 output tokens.Source: §5.3 · p. 7, Figure 5 · p. 8

5.5. Ablation study

  • On LLaMA3-8B, SVD decomposition raises zero-shot average from 41.7 to 61.1 and reduces perplexity from 91.6 to 19.6; TwinQuant reaches 70.9 average accuracy and 9.4 perplexity.The ablation reports the same final scores for Qwen3-8B as 70.2 average accuracy and 13.2 perplexity.Source: Table 3 · p. 8, §5.5 · p. 8

5.6. Quantization Error Analysis

  • Learnable subspace decomposition suppresses long-tail quantization-error spikes in both branches; the low-rank branch uses global rotation Q and layer-specific transform G, while the residual uses Q.Source: §5.6 · pp. 8–9, Figure 7 · p. 9

5.7. Overhead of Low-Rank Component

  • Across six models, low-rank memory overhead is 3.5–7.3% and latency overhead is 3.8–8.2%, with the dual-component kernel co-scheduling paths to limit launches and memory traffic.Figure 8 covers the six models at rank 128 in Environment 1.Source: Figure 8 · p. 9, §5.7 · p. 9

6. Limitations and Future Work

The authors identify limits in evaluation scope, accelerator coverage, workload validation, and optimization requirements, with corresponding directions for future study.

6. Limitations and Future Work

  • Evaluation mainly covers dense language models; extending the study to MoE architectures and multimodal models remains future work.Source: §6 · p. 9
  • The fused dual-component kernel is optimized for the evaluated GPU platforms, so deployment on other accelerator architectures may require adaptation and tuning.Source: §6 · p. 9
  • TwinQuant’s effectiveness and system behavior under ultra-long-context workloads require further validation.Source: §6 · p. 9
  • The learnable transforms rely on calibration data and add offline optimization time; lighter or more data-efficient optimization is left for future work.Source: §6 · p. 9

7. Conclusion

TwinQuant combines learnable subspace decomposition and component-specific transformations with a fused 4-bit kernel. Across mainstream LLMs and GPUs, it keeps accuracy within 0.6–2.6 average-accuracy points of FP16 and reaches up to 1.8× speedup.

7. Conclusion

  • TwinQuant keeps 4-bit accuracy within 0.6–2.6 average-accuracy points of FP16, with up to 1.8× speedup across mainstream LLMs and GPUs.Source: §7 · p. 9
  • TwinQuant splits each weight into low-rank and residual components, learning component-specific transformations over the Stiefel and general linear manifolds to reduce dynamic range and quantization error.Source: §7 · p. 9
  • A fused 4-bit kernel pipelines the two-stage low-rank path on-chip and merges it with the residual path in one epilogue and write-back.Source: §7 · p. 9

A.1. Theoretical Analysis

The analysis expresses quantized reconstruction error through activation and weight-error terms, then derives an expected-error decomposition under assumptions about quantization noise. It also defines gains for transformed activation and low-rank terms as error-proxy reductions.

A.1. Theoretical Analysis

  • Assuming independent, zero-mean quantization noises makes cross-product expectations vanish, leaving the nonzero terms in the expected squared-error decomposition.Source: §A.1 · p. 12
  • The low-rank component's quantization error is approximated by the two factor-error terms, neglecting the second-order product of factor noises.Source: §A.1 · p. 12
  • The total reconstruction error is expressed as activation error, low-rank weight error, and residual weight error.Source: §A.1 · p. 12
  • The activation-flattening and low-rank re-parameterization gains are inverse error-proxy gains, so larger values indicate stronger reductions.Source: §A.1 · p. 13

A.2. Baselines

The baselines span rounding, Hessian-aware quantization, scaling and clipping, rotation-based methods, low-rank decomposition, and distribution flattening.

A.3. Baselines

  • RTN uses fixed-scale rounding; GPTQ applies Hessian-aware greedy weight quantization with error compensation; AWQ uses activation-aware clipping and per-channel scaling.Source: §A.2 · p. 13
  • SmoothQuant shifts activation outliers into weights, while QuaRot and SpinQuant use orthogonal rotations to disperse outliers or homogenize magnitudes.Source: §A.2 · p. 13
  • SVDQuant isolates outlier-dominated components in a low-rank branch and quantizes the residual, adding low-rank computation and storage.Source: §A.2 · p. 13
  • FlatQuant uses learnable per-channel affine rescaling to equalize dynamic ranges and reduce quantization error.Source: §A.2 · p. 14

A.3. Zero-shot Accuracy

Across LLaMA3 and Qwen3, TwinQuant remains robust under W4A4, with competitive results under W4A8. Its reported accuracy and perplexity comparisons cover multiple model sizes and baselines.

A.3. Zero-shot Accuracy

  • Under W4A4, TwinQuant reaches 65.6 average accuracy on LLaMA3-3B and 70.9 on LLaMA3-8B, and 65.0/70.2/72.8/73.3 on Qwen3-4B/8B/14B/32B.Source: §A.3 · p. 14
  • On LLaMA3-3B W4A4, TwinQuant's 65.6 average accuracy exceeds SpinQuant's 63.6 and matches FlatQuant's 65.5; on 8B, its 70.9 and 9.4 perplexity improve on SpinQuant's 69.4 and 9.5.Source: §A.3 · p. 14
  • Under W4A8, TwinQuant reaches 65.4/72.8 average accuracy on LLaMA3-3B/8B and 65.6 average accuracy with 18.3 perplexity on Qwen3-4B.Source: §A.3 · p. 14
  • On Qwen3 W4A4, TwinQuant improves average accuracy over SpinQuant by 1.9 points on 8B and 1.3 points on 32B, with 32B perplexity falling from 12.1 to 10.4.Source: §A.3 · p. 14
  • RTN and SmoothQuant show severe instability under W4A4, including Qwen3 perplexities often in the thousands and near-random accuracies.Source: §A.3 · p. 14

A.4. Kernel Profile

Kernel fusion reduces latency across attention and MLP projections in two hardware/software environments. The appendix also reports modest offline quantization time and describes the kernel and training implementation.

A.5. Offline Overhead

  • 8–38 minutes per model for quantization across six models, typically around 10–30% of the overall optimization budget.Optimization time grows from 27 minutes for LLaMA3-3B to 389 minutes for Qwen3-32B on 2× NVIDIA RTX Pro 6000; the authors say offline optimization can be amortized when models are served for long periods.Source: §A.5 · p. 14, Table 8 · p. 17

A.6. Implement Detail

  • The fused dual-component kernel extends CUTLASS INT4 Tensor Core GEMM templates in CUDA/C++.The implementation adds roughly 2K lines for the kernel and uses a custom trainer of about 4K lines. Training totals 1,000 steps: 400 Global Alignment, 400 Invertible Adaptation, and 200 Joint Refinement; migration strength α is selected per layer to minimize calibration-set layer-output MSE.Source: §A.6 · p. 15

A.4. Kernel Profile

  • About 1.8–2.0× speedup for attention qproj and k/vproj in both phases on LLaMA3-8B.In Environment 1, MLP speedups are 1.23–1.91× prefill and 1.42–2.00× decode; in Environment 2, they are 1.22–1.93× prefill and 1.42–2.10× decode. Environment 2 also reports qproj speedups of 1.87–2.00× prefill and 1.91–1.99× decode, and up to 2.23× decode for k/vproj.Source: §A.4 · p. 14, Table 6 · p. 17, Table 7 · p. 17

Related papers

Loading 2606.01556v1…