PaperlayerView exact revision on arXiv

Source-linked AI summary

SpinQuant: LLM quantization with learned rotations

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, Tijmen Blankevoort

arXiv:2405.16406v4cs.LGcs.AIcs.CLcs.CV

TL;DR

Outliers make low-precision LLM quantization less accurate despite its memory and latency benefits. SpinQuant learns rotation matrices that reduce outliers while preserving full-precision outputs, achieving strong results across quantization settings. The paper leaves closed-form optimal rotation design for future work.

  • Problem

    Outliers expand quantization ranges and increase reconstruction errors for normal LLM weights and activations, limiting low-precision quantization.

  • Method

    SpinQuant inserts rotation matrices using rotational invariance and optimizes them with Cayley SGD while keeping full-precision outputs unchanged.

  • Results

    64.0 average accuracy in W4A4KV4 quantization on LLaMA-2 7B leaves a 2.9-point gap to full precision.

  • Takeaways & Limitations

    SpinQuant provides learned, robust rotations for improving quantized LLM accuracy and remains compatible with advanced weight quantization such as GPTQ.

  • Takeaways & Limitations

    Whether a theoretically calculated closed-form rotation yields the best quantization performance remains open for future research.

Abstract

from arXiv · show

Post-training quantization (PTQ) techniques applied to weights, activations, and the KV cache greatly reduce memory usage, latency, and power consumption of Large Language Models (LLMs), but may lead to large quantization errors when outliers are present. Rotating activation or weight matrices helps remove outliers and benefits quantization. In this work, we identify a collection of applicable rotation parameterizations that lead to identical outputs in full-precision Transformer architectures while enhancing quantization accuracy. In addition, we find that some random rotations lead to much better quantization than others, with an up to 13 points difference in downstream zero-shot reasoning performance. As a result, we propose SpinQuant, a novel approach that incorporates learned rotation matrices for optimal quantized network accuracy. With 4-bit quantization of weight, activation, and KV-cache, SpinQuant narrows the accuracy gap on zero-shot reasoning tasks with full precision to merely 2.9 points on the LLaMA-2 7B model, surpassing LLM-QAT by 19.1 points and SmoothQuant by 25.0 points. Furthermore, SpinQuant also outperforms concurrent work QuaRot, which applies random rotations to remove outliers. In particular, for LLaMA-3 8B models that are hard to quantize, SpinQuant reduces the gap to full precision by up to 45.1% relative to QuaRot. Code is available at https://github.com/facebookresearch/SpinQuant.

1 INTRODUCTION

SpinQuant addresses the inference cost and outlier-related errors of low-precision LLM quantization by learning rotations that preserve full-precision outputs while improving quantizability. Across several model families and settings, its rotation strategies and learned matrices improve quantized performance.

  • Post-training quantization reduces LLM memory usage and can improve latency, supporting both server-side and on-device inference.
  • SpinQuant uses rotation invariance to integrate rotation matrices with nearby weights, reducing outliers without changing the full-precision network output.
  • Up to 13 points separate zero-shot reasoning accuracy across different rotation matrices, motivating optimization rather than relying on random rotations.
  • Cayley SGD learns orthonormal rotations against quantized-network loss while keeping the full-precision output unchanged and making intermediate values more quantization-friendly.
  • SpinQuantno had absorbs shortcut and value-output rotations into weights, whereas SpinQuanthad adds online Hadamard rotations for activation and KV-cache outliers.
  • Experiments cover seven LLMs, including LLaMA-2, LLaMA-3, and Mistral models, with low-bit weight, activation, and KV-cache quantization.
  • In extreme W4A4KV4 quantization, SpinQuanthad reaches 64.0 average accuracy on LLaMA-2 7B, only 2.9 points below full precision.
  • Rotation removes channel outliers and reduces quantization error in LLaMA-2 7B activation distributions, supporting token-wise or tensor-wise quantization.

2 MOTIVATION

The paper motivates rotation as a way to reduce outlier-driven quantization error and investigates why rotation quality varies. It finds that optimized rotations provide more accurate and stable quantization than random alternatives.

  • Quantization saves memory and latency, but LLM outliers expand value ranges and increase reconstruction errors for ordinary values.
  • Random rotations blend large and small weights across axes, producing distributions with fewer outliers that are easier to quantize.
  • After rotation, activation kurtosis falls from above 200 in many layers to approximately 3 across all layers, indicating a more Gaussian-shaped distribution.
  • Cayley-optimized rotations outperform the best random and Hadamard matrices across 100 seeds while showing minimal initialization variance.
  • 13 points separate the best and worst random rotations in zero-shot average accuracy for W4A4-quantized LLaMA-2 7B.
  • SpinQuant’s learned rotations consistently achieve high accuracy across seven models and four low-bit quantization settings.

3 METHOD

SpinQuant parameterizes rotations that preserve full-precision Transformer outputs while improving quantization, then learns mergeable rotations with Cayley SGD against quantization loss.

  • Rotation parameterization: R1 rotates residual activations and is reversed before nonlinear attention and feed-forward blocks, allowing absorption into corresponding weights.Without quantization, the full-precision network remains unchanged regardless of the applied rotation.
  • Rotation parameterization: R2 applies head-wise rotations to value matrices and matching output activations, improving value-cache and out-projection input quantization without new parameters.The paired rotations offset each other in the full-precision network because no operators intervene between them.
  • Rotation parameterization: SpinQuantno had, using only R1 and R2, narrows the W4A8 zero-shot commonsense reasoning accuracy gap to 0.1–2.5 points versus full precision.The rotations are merged into pretrained weights, so inference requires replacing the rotated quantized weights rather than modifying the forward pass.
  • Rotation parameterization: SpinQuanthad adds online Hadamard rotations R3 and R4 for low-bit KV-cache and activation quantization, using fast transforms with marginal inference overhead.R4 reduces outliers entering the feed-forward down-projection layer, while R3 is used when low-bit KV-cache quantization is required.
  • Rotation parameterization: Four rotations R1–R4 improve quantization while preserving numerical consistency in the full-precision network.R1 and R2 are mergeable; R3 and R4 are online Hadamard rotations retained for efficient outlier suppression.
  • Cayley-optimized rotation: Cayley SGD optimizes R1 and R2 on the Stiefel manifold using fixed pretrained weights and calibration-set task loss.The rotations comprise approximately 0.26% of the weight size and remain orthonormal during optimization.
  • Cayley-optimized rotation: Cayley updates preserve orthonormality and can be computed with fixed-point iteration at approximately twice the per-iteration computation time of naive SGD.After 100 iterations on an 800-sample WikiText2 calibration set, the optimized rotation outperforms the best of 100 random seeds and has minimal seed variance.

4 EXPERIMENTS

Experiments evaluate SpinQuant across seven LLMs and multiple low-bit settings, comparing rotation schemes, quantization methods, and implementation trade-offs. Learned rotations improve accuracy, especially under 4-bit activation quantization, while online Hadamard rotations add latency for further gains.

  • Experimental setup: Experiments cover seven LLMs, eight zero-shot commonsense reasoning tasks, WikiText2 perplexity, and four commonly used bit-width settings.The models include LLaMA-2 7B/13B/70B, LLaMA-3 1B/3B/8B, and Mistral 7B.
  • Rotation schemes: SpinQuantno had merges learned rotations into model weights, requiring no forward-pass modification or additional kernel support.SpinQuanthad additionally uses online Hadamard rotations, which are computed with a fast Hadamard kernel.
  • Main results: 10.5 points: SpinQuantno had improves 4-8-8 quantized Mistral 7B, while LLaMA-3 8B reaches a 1.0-point gap to full precision on 4-8-16.When activations are not extremely quantized, adding online Hadamard rotation provides marginal benefit.
  • Main results: 2.9/1.4/1.7 points: SpinQuanthad leaves these accuracy gaps to full precision for 4-4-4 LLaMA-2 7B/13B/70B models.It surpasses previous state-of-the-art methods by 19.1/16.4/15.3 points, respectively.
  • Main results: SpinQuant achieves similar WikiText2 perplexity to OmniQuant, AWQ, and QuIP# with 4-bit weights and 8-bit activations, without advanced vector quantization.The results support applicability across weight-only and weight-plus-activation quantization scenarios.
  • Learned rotation versus random rotation: 15.7 points: optimized rotations improve Mistral-7B over random Hadamard rotations, with optimization taking 30 minutes for small models to 3.5 hours for 70B.Across models and bit-width configurations, learned rotations consistently outperform random rotations.
  • Comparison with QuaRot: SpinQuanthad improves over QuaRot by 2.0 to 28.6 points while using two online Hadamard matrices per block instead of four.QuaRot exhibits 28.1- and 33.2-point drops for 70B W4A4 and W4A4KV4 quantization, respectively.
  • Speed measurement: Online Hadamard processing adds 8% latency, while 4-bit quantization provides an approximately 3× speedup over the 16-bit model on a MacBook M1 Pro CPU.This presents a trade-off between SpinQuantno had simplicity and SpinQuanthad accuracy for lower-bit activations.

5 RELATED WORK

LLM quantization reduces model size and storage, but outliers make quantization difficult by dominating the representable range. Prior work addresses this challenge through several mitigation strategies.

  • Quantization compresses neural networks and reduces model size and storage requirements.
  • LLM quantization is especially challenging because numerous outliers dominate the quantization range.Most values therefore receive only a few effective bits.
  • Prior strategies mitigate outliers through techniques discussed in earlier quantization research.The supplied passage introduces these strategies without specifying them in full.

6 CONCLUSIONS

SpinQuant uses learned rotations and Cayley SGD to reduce outliers while preserving full-precision outputs under rotation invariance. It targets the gap between full precision and 4-bit quantization.

  • SpinQuant uses learned rotations to bridge full-precision and 4-bit weight, activation, and KV-cache quantization.
  • Rotation matrices diminish outliers in weights and intermediate activations while keeping full-precision outputs numerically identical.
  • Cayley SGD optimizes the rotation matrices, producing improved and robust quantization outcomes.
  • SpinQuant is compatible with advanced weight quantization methods such as GPTQ and demonstrates state-of-the-art performance.

A.1 COMPLETE RESULTS OF MAIN RESULT TABLE

The complete results compare perplexity and averaged zero-shot commonsense reasoning accuracy across LLaMA-2 models. Eight reasoning tasks are included alongside WikiText2 evaluation.

  • The evaluation reports perplexity on the WikiText2 test set.
  • It reports averaged accuracy across eight zero-shot commonsense reasoning tasks.The tasks include ARC-easy, ARC-challenge, BoolQ, PIQA, SIQA, HellaSwag, OBQA, and WinoGrande.
  • The complete tables compare SpinQuant with previous quantization works.The supplied passage introduces the comparison but does not enumerate all compared methods.

A.2 RESULTS ON 3-BIT WEIGHT QUANTIZATION

SpinQuant remains effective under 3-bit weight and 8-bit activation quantization across seven models, with robustness to calibration choices and manageable optimization and inference overhead.

  • Results on 3-bit weight quantization: 1.2−5.3 points: SpinQuant reduces the full-precision accuracy gap from 9.0−28.0 points under 3-bit weight and 8-bit activation quantization.The result covers seven models.
  • Results on 3-bit weight quantization: Minimal calibration data and iterations are sufficient to optimize rotations for better quantization.The optimization is resilient to changes in sample count.
  • Results on 3-bit weight quantization: Asymmetric quantization outperforms symmetric quantization for both activation and KV-cache quantization.
  • Results on 3-bit weight quantization: C4 calibration data produces results consistent with WikiText calibration on LLaMA-2 7B.This supports robustness to calibration-data choice.
  • Results on 3-bit weight quantization: Hadamard rotation causes only a marginal latency difference when SpinQuant is implemented carefully.This result is reported for LLaMA-3 70B end-to-end speed tests.
  • Results on 3-bit weight quantization: SpinQuant requires optimization time on a similar scale to GPTQ.The authors characterize the additional optimization time as worthwhile given the improvements over GPTQ.

A.8 ABLATION STUDY ON RTN VS GPTQ

SpinQuant’s accuracy gains primarily come from learned rotations, while GPTQ provides an additional smaller improvement. The method also remains effective for weight-only and instruction-finetuned models.

  • RTN versus GPTQ: 6.5 ∼20.9 percentage points: learned rotations improve accuracy over previous methods, including GPTQ, in W4A4KV16 quantization.Adding GPTQ afterward further boosts performance by up to 2.3 percentage points.
  • Weight-only quantization: SpinQuant consistently achieves higher accuracy than AWQ and other previous work under 4-bit weight-only quantization.
  • Instruction-finetuned models: SpinQuant W4A8 improves few-shot MMLU accuracy and TLDR9 ROUGE for instruction-finetuned LLaMA 3.2 1B and 3B models.These improvements significantly close the gap to the BF16 baseline.

B.1 GRADIENT ANALYSIS

The gradient analysis explains why rotations can be optimized for quantized models despite leaving full-precision outputs unchanged. Quantization creates the nonzero gradient signal needed to learn useful rotation matrices.

  • Gradient analysis: Equation (5) is generally nonzero, validating backpropagation to learn the rotation matrix R.
  • Gradient analysis: Without quantization, equation (5) reduces to 0, so learning R is meaningful only for quantized models.
  • Gradient analysis: Two discrepancies move the gradient with respect to R away from 0: quantized versus unquantized rotated weights and activations.

B.2 LOSS ANALYSIS

Loss and distribution analyses connect learned rotations to improved quantization fidelity. Rotations attenuate extreme values, improve signal-to-noise ratio, and support accuracy gains across evaluated settings.

  • Training and SNR: Figure 7 tracks training, activation SNR, and layerwise SNR improvement as the rotation changes from random initialization to learned optimization.R0 denotes a randomly initialized rotation, while RT denotes the learned rotation after T = 200 iterations.
  • Loss analysis: Learning R improves the end-to-end signal-to-quantization-noise ratio, bringing quantized outputs closer to floating-point outputs.Introducing a random R produces a 3.8 dB SNR improvement in the reported analysis.
  • Quantization ablations: In W4A4KV16 ablations, learned rotations provide the primary accuracy gains, while GPTQ adds up to 2.3 percentage points.
  • Distribution analysis: After rotation, extreme activation and weight values are attenuated and no noteworthy outliers remain across the token dimension.The comparison uses activation Figures 8–9 and weight Figures 10–11.
Loading 2405.16406v4…