PaperlayerView exact revision on arXiv

Source-linked AI summary

SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, Dan Alistarh

arXiv:2306.03078v1cs.CLcs.LG

TL;DR

Three- to four-bit quantization can substantially reduce LLM accuracy, motivating compression methods that preserve predictive quality for memory-limited deployment. SpQR isolates sensitive outlier weights in higher precision while quantizing most weights and metadata at low bitwidth, achieving near-lossless perplexity with compact inference. Its main evaluation boundary is that generative quality was not directly measured.

  • Problem

    3–4-bit quantization often causes significant accuracy degradation, despite its value for fitting LLMs onto memory-limited devices.

  • Method

    SpQR isolates outlier weights with disproportionately large quantization errors, stores them in higher precision, and compresses the remaining weights and quantization metadata to low bitwidth.

  • Results

    SpQR reaches 3–4 bits per parameter with less than 1% relative perplexity error versus the dense baseline.

  • Takeaways & Limitations

    SpQR provides a hybrid sparse-quantized format for near-lossless compression of accurate pretrained LLMs at low bitwidth.

  • Takeaways & Limitations

    The paper evaluates predictive performance through zero-shot accuracy and perplexity, but does not evaluate the generative quality of quantized LLMs.

Abstract

from arXiv · show

Recent advances in large language model (LLM) pretraining have led to high-quality LLMs with impressive abilities. By compressing such LLMs via quantization to 3-4 bits per parameter, they can fit into memory-limited devices such as laptops and mobile phones, enabling personalized use. However, quantization down to 3-4 bits per parameter usually leads to moderate-to-high accuracy losses, especially for smaller models in the 1-10B parameter range, which are well-suited for edge deployments. To address this accuracy issue, we introduce the Sparse-Quantized Representation (SpQR), a new compressed format and quantization technique which enables for the first time near-lossless compression of LLMs across model scales, while reaching similar compression levels to previous methods. SpQR works by identifying and isolating outlier weights, which cause particularly-large quantization errors, and storing them in higher precision, while compressing all other weights to 3-4 bits, and achieves relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs. This makes it possible to run 33B parameter LLM on a single 24 GB consumer GPU without any performance degradation at 15% speedup thus making powerful LLMs available to consumer without any downsides. SpQR comes with efficient algorithms for both encoding weights into its format, as well as decoding them efficiently at runtime. Specifically, we provide an efficient GPU inference algorithm for SpQR which yields faster inference than 16-bit baselines at similar accuracy, while enabling memory compression gains of more than 4x.

1 Introduction

SpQR targets the accuracy degradation caused by 3–4-bit quantization by preserving sensitive outlier weights in higher precision while compressing the remaining weights. It combines outlier-aware post-training quantization with efficient sparse and quantized computation for compact inference.

  • Motivation: 3–4-bit quantization can significantly degrade accuracy, and sequential generation may accumulate small errors into severely corrupted outputs.The paper therefore targets low-bitwidth compression that preserves predictive performance relative to a 16-bit model.
  • Contribution: SpQR compresses pretrained LLMs to 3–4 bits per parameter while inducing less than 1% relative perplexity error versus a dense baseline.It is presented as the first weight-quantization method to combine these compression ratios with near-lossless end-to-end accuracy.
  • Contribution: SpQR isolates outlier weights that cause disproportionately high quantization errors, stores them in high precision, and compresses other weights to low bitwidth.It also uses very small quantization groups and quantizes the scales themselves to 3 bits.
  • Method: SpQR converts models using an extended GPTQ-style post-training procedure with calibration data, layer-wise output-error minimization, outlier detection, and compression of quantization metadata.Most (≥99%) of weights are compressed to low bitwidth, while extracted outliers remain separate.
  • Analysis: The method analyzes vertical and horizontal correlations in quantization errors, including outliers associated with input feature dimensions and output hidden dimensions.The paper identifies output hidden-dimension outliers as a new counterpart to previously observed input-feature outliers.
  • Inference: A specialized CSR-based sparse multiplication algorithm is combined with dense quantized multiplication for token-by-token generation.This design reduces LLM memory footprints by about 3.4x or more without degradation in language-modeling loss or perplexity.

2 Related Work

Prior post-training quantization methods trade memory savings against accuracy using calibration data, quantization granularity, and optimization-based solvers. The paper positions SpQR as a fine-grained hybrid format addressing persistent accuracy losses, particularly in smaller deployable LLMs.

  • Post-training quantization: Post-training quantization compresses models using limited calibration data and solvers focused on layer-wise or group-wise subproblems.The related methods discussed include AdaRound, BitSplit, and AdaQuant.
  • Existing approaches: Direct-rounding methods customize quantization granularity, while LLM.int8() separately quantizes outlier features at higher bit-width.These approaches seek lower quantization error by trading off space against accuracy.
  • Accuracy–compression trade-off: Existing methods can still incur significant accuracy loss, especially for relatively small 7–13B models where accuracy drops are described as drastic.The paper investigates near-lossless 3–4-bit compression for this deployment-relevant regime.
  • Outlier structure: Prior activation-and-weight quantization work reports relatively low accuracy impact at 8 bits and identifies high-valued outlier features in large LLM inputs and outputs.SpQR instead analyzes outlier structure from the perspective of weight quantization.
  • Hybrid formats: Earlier hybrid sparse-quantized formats use fixed sparse blocks or groups, whereas the paper motivates a finer-grained format for partially structured weight outliers.The related formats described include blocks of four weights and groups containing two zero weights.

3 Quantization sensitivity of LLM weights

SpQR’s sensitivity analysis treats quantization error as dependent on weight interactions and reveals structured, layer-specific patterns of sensitive weights. These patterns motivate isolating structured and unstructured outliers during compression.

  • Sensitivity definition: Sensitivity measures the minimum squared prediction difference after quantizing one weight while allowing other weights to compensate.The analysis uses calibration inputs per layer and a closed-form solution based on an inverse Hessian.
  • Sensitivity estimation: GPTQ approximates sensitivity dynamically while quantizing columns and compensating with not-yet-quantized weights.It reuses inverse Hessian subsections already computed by the solver.
  • Observed structures: Sensitive weights form structured patterns rather than appearing randomly, including row, column, attention-head, rotary-embedding, and unstructured outliers.The analysis uses GPTQ 3-bit quantization on LLaMA-65B with C4 calibration data.
  • Observed structures: Row outliers affect selected output units, while column outliers span rows across selected input dimensions and correlate with outlier features.Partial row patterns can correspond to subsets of attention heads.
  • Observed structures: Sensitive attention heads appear as regular width-128 stripes, but individual weight sensitivity still varies substantially within those heads.The stripe orientation depends on the projection type, and the pattern is absent from value projections and MLP weights.
  • Observed structures: Rotary embeddings produce a repeating sensitivity pattern with period 64, which is absent in layers without rotary embeddings.The paper attributes higher sensitivity to lower-frequency sine and cosine components.

4 SpQR: A Sensitivity-aware compressed representation

SpQR allocates precision according to weight sensitivity by combining very small-group quantization with separate high-precision storage for outliers. Its representation uses multilevel quantization statistics and CSR-based sparse handling to support GPU inference.

  • Design motivation: SpQR addresses scattered sensitive weights with separate mechanisms for small sensitive groups and individual outliers.This extends quantization beyond methods that treat low- and high-sensitivity weights equally.
  • Bilevel quantization: Bilevel quantization uses groups of β1=8–32 weights with separate scales and zero-points, then quantizes those statistics across β2=16 consecutive values.Quantizing the statistics reduces the storage overhead of very small groups.
  • Outlier handling: Some sensitive weights account for over 75% of total quantization error despite representing only 1% of weights, so SpQR stores them at 16-bit precision.The remaining base weights are quantized to 3–4 bits, with residual quantization effects transferred into the high-precision outliers.
  • Outlier handling: Outlier detection filters weights using the sensitivity criterion, excludes them when computing base-weight statistics, and then applies GPTQ to the remaining weights.Excluding outliers can substantially reduce quantization scales such as min-max ranges.
  • Representation: The SpQR representation stores quantized weights, two levels of quantized statistics, and CSR outlier indices and values.A representative configuration uses 3-bit weights and statistics with 16-weight groups and four 16-bit scalars per 256-weight block.
  • GPU inference: SpQR’s GPU decoder loads statistics and quantized weights into shared memory, dequantizes them to 16-bit, and combines dense multiplication with sparse outlier processing.The sparse algorithm exploits row-like outlier patterns and uses load balancing across GPU thread blocks.

5 Experimental Validation

SpQR is evaluated across LLaMA and Falcon models against RTN and GPTQ using perplexity, zero-shot accuracy, compression, and inference-speed measures. The experiments show near-baseline perplexity at under 4.71 bits per parameter, improved 4-bit performance, and faster inference with specialized sparse multiplication.

  • Experimental setup: The evaluation covers LLaMA 7B–65B and Falcon 7B–40B models, comparing SpQR with GPTQ and RTN using perplexity and zero-shot accuracy.Calibration data comes from RedPajama for LLaMA and RefinedWeb for Falcon.
  • Main results: 4.6 to 4.71 bits per parameter brings SpQR within 1% perplexity of 16-bit models across all evaluated models.The comparison uses actual model size versus perplexity and zero-shot task accuracy.
  • Main results: At 4 bits per parameter, SpQR halves the error relative to the 16-bit baseline compared with GPTQ.The comparison is reported for LLaMA and Falcon results controlled for model size.
  • Ablations: Small quantized groups and unstructured outliers independently improve perplexity over alternative strategies, while activation ordering provides a smaller additional gain.The ablations evaluate bilevel quantization, outlier types, zero-point precision, and the GPTQ activation-order heuristic.
  • Inference time: The specialized sparse matrix multiplication algorithm yields 20–30% speedups, whereas standard PyTorch sparse multiplication is not faster than 16-bit inference.Inference speed is measured for batch size 1 on a single A100 GPU.

6 Discussion & Limitations

SpQR preserves near-lossless 16-bit accuracy at low average precision, but the paper identifies limits in generative-quality evaluation and inference-kernel fusion.

  • Discussion: Less than 4.75 bits per parameter achieves near-lossless 16-bit accuracy on average.Compressing to 3.36 bits provides an even better quality-size tradeoff.
  • Limitations: The evaluation measures zero-shot accuracy and perplexity, not the generative quality of quantized LLMs.The authors treat the relationship between perplexity and generation quality as a hypothesis for future investigation.
  • Limitations: SpQR does not fuse sparse matrix multiplication with regular quantized matrix multiplication, limiting potential inference-time improvements.The authors leave such a fused implementation to future work because it is difficult to implement.

A Additional weight sensitivity analysis

Weight sensitivity varies with quantization scheme, layer depth, matrix role, and location, revealing structured outliers and algorithm-dependent error patterns.

  • Sensitivity factors: Sensitivity matrices vary with quantization scheme, layer depth, weight role, and location within the matrix.The appendix organizes its additional observations around these four factors.
  • Weight location: A particularly sensitive group contains a vertical outlier whose corresponding input feature has significantly higher variance than other input dimensions.The vertical outlier is associated with an input feature dimension.
  • Quantization scheme: Blockwise GPTQ with size 128 concentrates increased sensitivity in a 8192 × 128 weight group, whereas per-row statistics produce high error across the entire row.The comparison uses 3-bit GPTQ configurations with per-row versus blockwise statistics.
  • Attention structure: Rotary embeddings produce a sensitivity pattern repeating every 64 rows in attention query and key weights.The authors attribute this pattern to how LLaMA implements rotary position embeddings.
  • Quantization order: Later columns are more sensitive on average, while rightmost-weight errors arise from GPTQ’s left-to-right quantization and disappear when the order is shuffled.Earlier layers show this difference most strongly.
  • Layer depth: The first LLaMA layers generally have lower GPTQ-3bit OBC error, although absolute error must be interpreted alongside each layer’s input and output variance.Figures compare sensitivity behavior in the first, middle, and last layers.

B Experimental Configurations

SpQR exposes bit-width, grouping, outlier, and threshold controls, and evaluates how grouping choices affect compressed size and perplexity.

  • Hyperparameters: SpQR’s configurable hyperparameters include weight, scale, and zero-point bit widths; outlier rate; two block sizes; and an outlier threshold.The notation names these controls bw, bs, bz, ro, β1, β2, and τ.
  • Outlier threshold: Increasing τ reduces the number of outliers for a fixed configuration, with τ tuned by binary search over 0.1 to 1.0.Most reported configurations use τ between 0.1 and 0.45.
  • Example configuration: A near-lossless LLaMA-30B configuration uses bw = 4, bs = bz = 3, β1 = β2 = 16, and τ = 0.1.The supplementary command specifies corresponding four-bit weights, 16-weight groups, and permutation ordering.
  • Group structure: β1 groups weights for the same input unit, while β2 groups output units when quantizing quantization scales.The visualizations show β1 blocks horizontally.
  • Evaluation: Table 5 evaluates varying β1 and β2 settings for 3-bit SpQR on LLaMA-65B using WikiText2, Penn Treebank, and C4 perplexity.The table reports effective bits for each configuration and dataset-specific perplexities.

D Estimating model size

The appendix estimates SpQR storage in average bits per parameter by accounting for quantized weights, statistics, outliers, and indexing overhead.

  • Size estimation: Average bits per parameter provides a pre-quantization estimate of compressed model size.The estimate is expressed using total model storage in bits divided by the number of parameters.
  • Storage components: Model storage includes quantized weights, first- and second-order statistics, outliers, and the outlier index.Each outlier requires approximately 32 bits of storage.
  • Matrix-level accounting: For a weight matrix R^(d_out×d_in), the average-bit calculation accounts for weight precision, statistic precision, block sizes, and outlier storage.Transformer linear projections dominate storage and computation in the considered matrices.
  • Size trade-offs: Increasing precision increases model size, whereas increasing block size decreases it under the storage estimate.The converse changes reduce or increase size, respectively.
  • Worked example: With bw = 3, bs = 3, bz = 3, β1 = 16, β2 = 32, and 0.4% outliers, the appendix gives a concrete average-bit example.The example illustrates how configuration choices and outlier rate enter the estimate.

E Choice of optimal configuration for fixed average number of bits

The section evaluates how bitwidth, quantization groupsizes, and outlier rates trade model performance against compression. SpQR stays within 1% of baseline perplexity below 4.3 bits per parameter and improves over GPTQ at 4 bits.

  • Smaller groups and limited outliers considerably improve model performance, but higher-precision storage is preferable to extremely small groups or many outliers.
  • Within 1% of baseline perplexity is achieved by SpQR with less than 4.3 bits per parameter, while 4-bit SpQR significantly improves on GPTQ.

F Additional results for near-lossless compression

This section reports additional evaluations, robustness checks, generated-sample comparisons, and deployment recommendations for near-lossless SpQR compression. The experiments include OPT, LLaMA, and Falcon models under varied memory and randomness settings.

  • Additional evaluations: OPT evaluations compare SpQR with RTN and GPTQ on WikiText2, C4, and Penn Treebank.
  • Additional evaluations: LLaMA and Falcon results are reported using the LM eval harness.
  • Memory-constrained deployment: The practitioner-facing evaluation recommends LLaMA models and compression levels that fit specified RAM or VRAM budgets without parameter or activation offloading.
  • Robustness and samples: 3.625 bits per parameter yields LLaMA-65B perplexities of 3.75 ± 0.003 on WikiText2, 7.03 ± 0.01 on Penn Treebank, and 5.75 ± 0.00086 on C4.The reported standard deviations are at least one order of magnitude smaller than the difference between SpQR and GPTQ.
  • Robustness and samples: Generated samples compare original LLaMA-65B with SpQR and RTN-4bit under greedy autoregressive inference.

J Broader impact

SpQR is presented as a way to deploy high-quality LLMs on memory-limited phones and laptops while preserving reliable performance. The broader-impact discussion also notes that it improves access to existing models rather than creating new capabilities.

  • Access and deployment: SpQR enables deployment of high-quality 7-13B parameter LLMs on memory-limited laptops and phones.The paper specifically describes applying SpQR after developing specialized 7B models in 16-bit.
  • Generated samples: The generated-sample comparison uses a shared prompt to contrast 16-bit, SpQR, and RTN 4-bit outputs.
  • Sensitivity analysis: The sensitivity figures organize LLaMA-65B weight log-sensitivities by layer type and layer depth under per-row or group-wise quantization.
  • Broader impact: SpQR is described as making larger and generally more accurate existing language models accessible to users with low-end devices.
  • Broader impact: The paper states that SpQR does not create models with new capabilities or risks, but makes already-openly-available models more accessible.
Loading 2306.03078v1…