PaperlayerView exact revision on arXiv

Source-linked AI summary

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, James Hensman

arXiv:2404.00456v2cs.LG

TL;DR

Fully quantizing LLM weights, activations, and KV caches is difficult because activation and attention outliers hinder low-bit inference. QuaRot uses Hadamard rotations under computational invariance to remove these outliers while preserving model outputs, enabling end-to-end 4-bit quantization. On LLaMA2-70B, it retains 99% of zero-shot performance, while 6- and 8-bit LLaMA-2 quantization is lossless.

  • Problem

    Activation outliers make joint quantization of weights, activations, and KV caches difficult, especially at 4 bits, while prior methods retain selected features in higher precision.

  • Method

    QuaRot uses randomized Hadamard transformations and computational invariance across transformer, feed-forward, attention, and KV-cache components to remove outliers without changing model outputs.

  • Results

    99% of zero-shot performance is retained for 4-bit LLaMA2-70B, with at most 0.47 WikiText-2 perplexity loss; 6- and 8-bit LLaMA-2 quantization is lossless.

  • Takeaways & Limitations

    QuaRot enables end-to-end 4-bit LLM inference with all weights, activations, and KV caches quantized using integer quantization.

  • Takeaways & Limitations

    The authors identify quantizing residuals and extending QuaRot to mixture-of-experts architectures as opportunities for future work.

Abstract

from arXiv · show

We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLaMa2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLaMa2 models without any calibration data using round-to-nearest quantization. Code is available at: https://github.com/spcl/QuaRot.

1 Introduction

LLM inference is constrained by computation, memory, and energy costs, while activation outliers make end-to-end low-bit quantization difficult. QuaRot uses rotations and computational invariance to remove outliers and quantize weights, activations, and KV caches in 4 bits with limited accuracy loss.

  • LLM inference requires substantial computation, memory, and energy, particularly during prefill, motivating lower-precision computation.
  • Activation outliers make joint quantization harder than weight quantization, especially at 4 bits, prompting prior methods to retain selected features in higher precision.
  • QuaRot applies randomized Hadamard transformations and computational invariance to produce an equivalent network without outlier features.
  • QuaRot extends online Hadamard transformations to attention keys and values, enabling KV-cache quantization.
  • 3.33× prefill speedups and 3.89× decoding-stage memory savings were achieved on LLaMA2-70B, with at most 0.47 WikiText-2 perplexity loss.

2 Related Work

Prior work largely compresses weights or separately quantizes KV caches, while activation outliers remain a central obstacle to fully low-precision inference. QuaRot combines Hadamard-based incoherence processing with fused transformations and attention handling to support broader end-to-end quantization.

  • Most quantization schemes use weight-only compression, leaving the main computation in high precision.
  • Activation quantization is difficult because LLM activations contain outlier features unlike weights.
  • Earlier rotation-based methods applied orthogonal transformations to weights, Hessians, or each linear layer, primarily for weight quantization or training.
  • KV-cache research progressed from 4-bit to 3-bit and promising 2-bit quantization, but keys also contain outliers and require complex techniques.
  • QuaRot fuses Hadamard transformations into weights, reduces forward-pass overhead, incoherence-processes activations, and quantizes the KV cache in 4 bits.

3 Background

This section introduces orthogonal and randomized Hadamard transformations, incoherence processing, and the computational invariance that lets QuaRot alter internal representations without changing model outputs.

  • 3.1 Orthogonal, Rotation and Hadamard Matrices: Orthogonal matrices preserve structure through QQ⊤ = I, while rotation matrices are orthogonal and Hadamard matrices use entries from {+1,−1}.Walsh-Hadamard matrices have size d = 2^n and support fast matrix-vector products.
  • 3.1 Orthogonal, Rotation and Hadamard Matrices: Randomized Hadamard matrices multiply a Hadamard matrix by a random sign diagonal matrix while remaining orthogonal.The random sign vector contains draws from {+1,−1}.
  • 3.2 Incoherence Processing: Incoherence processing multiplies weight matrices by orthogonal matrices to reduce outliers and improve weight and activation quantization.The method applies incoherence processing to activations in addition to weights.
  • 3.2 Incoherence Processing: Gated feed-forward blocks normalize and rescale inputs, apply Wup and Wgate, gate one activation with σ, multiply element-wise, and produce Y through Wdown.Before quantization, operations use either single-precision or half-precision arithmetic.
  • 3 Background: Pre-norm, gated transformer blocks are assumed, with each block preceded by LayerNorm or RMSNorm.The methodology is also stated to apply straightforwardly to MLP architectures.
  • 3.4 Computational Invariance: Computational invariance inserts an orthogonal rotation before a block and cancels it with Q⊤ in the output matrix, preserving the model output through RMSNorm.RMSNorm depends on activation norms, which rotations do not change.

4 Method

QuaRot modifies transformer weights and inserts Hadamard operations so activations, weights, and KV-cache components can be quantized while preserving the model’s forward computation. The method combines offline weight transformations with online activation and cache quantization.

  • Overview: QuaRot has a weight-modification stage followed by quantization and added online operations for activations and caches.Weights are manipulated in full precision before existing weight quantization is applied.
  • Weight modification: Randomized Hadamard matrices are fused into adjacent weights through computational invariance, changing representations without changing the model output at sufficient precision.LayerNorm or RMSNorm linear scaling is first fused into neighboring weight matrices.
  • FFN activation rotation: An online Hadamard transform before the FFN down-projection rotates intermediate activations, while the corresponding transform is fused into Wdown.The modified down-projection becomes HWdownQ when combined with the global rotation Q.
  • Attention rotations: Attention value projections are transformed head-wise by combining Wv and Wout modifications with Hadamard operations and Kronecker-structured head transforms.The identity Hnh×dh = (I ⊗Hdh)(Hnh ⊗I) enables an efficient reshape-based implementation.
  • KV-cache rotation: Queries and keys receive online head-wise Hadamard rotations so attention scores remain unchanged despite rotated key and query representations.QuaRot uses Post-RoPE caching to rotate a single query token during each decoding step rather than repeatedly rotating many cached vectors.
  • Quantization: After the structural transformations, GPTQ quantizes weights and symmetric per-token INT4 operations quantize linear-layer inputs online.Input row scales divide each token’s maximum absolute value by 7, the largest representable INT4 value.

5 Experimental Validation

QuaRot is evaluated on LLAMA-2 for language generation, zero-shot accuracy, kernel speed, decoding memory, round-to-nearest quantization, and group-wise quantization. Across these experiments, it combines low-bit accuracy with substantial performance and memory improvements, while results are partly measured on a single transformer block.

  • Accuracy Results: QuaRot outperforms prior 4-bit methods on WikiText-2, with at most 0.63 perplexity loss and 0.47 loss on LLAMA2-70B.It requires neither retraining nor higher-precision outlier features in the cited comparison.
  • Accuracy Results: QuaRot preserves LLAMA-2 zero-shot accuracy with at most 4.18% average score loss, including 1.09% for the 70B model.The evaluation covers PIQA, WinoGrande, HellaSwag, LAMBADA, and Arc Easy and Challenge.
  • Performance Analysis: 3.33× is the maximum reported prefill speedup on LLAMA2-70B, while LLAMA2-7B achieves 1.97×–2.16× over FP16 at sequence length 2048.Speedups are measured with the QuaRot kernel across batch sizes on an RTX 3090.
  • Performance Analysis: 3.89× is the reported LLAMA2-70B decoding memory saving, compared with FP16, while both evaluated models achieve at least 3.63× saving.These measurements decode 50 tokens and use varying prefill lengths with batch size 16; the 70B result is measured on a single layer.
  • Ablation Studies: Round-to-nearest weight quantization fully maintains FP16 accuracy in 8 bits, while its 4-bit gap versus GPTQ decreases from 2.27 points on 7B to 0.34 on 70B.RTN requires no calibration set or hyper-parameter, whereas GPTQ is the default weight-quantization method.
  • Ablation Studies: Smaller group sizes improve accuracy but require more scale-storage bits and more complex matrix-multiplication kernels.The group-wise experiments vary activation and weight group sizes while keeping the KV-cache group size at 128.

6 Conclusion

QuaRot enables end-to-end 4-bit quantization of pretrained LLM activations and KV caches using Hadamard matrices, while preserving most baseline task performance. The conclusion also identifies residual quantization and mixture-of-experts architectures as areas for further work.

  • QuaRot uses Hadamard matrices to eliminate outliers in activations and KV caches, enabling end-to-end 4-bit quantization.
  • 99% of FP16 baseline downstream task performance is retained, with 2.16× prefill speedup on RTX 3090 GPUs and up to 3.39× decoding memory saving.
  • Quantizing all LLAMA-2 models to 6 and 8 bits is lossless.
  • Future work includes quantizing residuals and extending QuaRot to mixture-of-experts architectures.

A.1 QuaRot on the Attention Module

QuaRot modifies the attention module by absorbing RMSNorm scaling and inserting rotations around attention projections and the KV cache. The figures distinguish ordinary computation flow from generation-time cache access and identify RoPE as the source of relative positional embeddings.

  • The input is rotated by Q, inverse matrices are fused into attention input layers, and Hadamard matrices are fused into V_projection columns by head.
  • The attention flow includes prefill and per-token inference paths, while dashed arrows represent generation-time access to and from the KV cache.
  • RoPE computes relative positional embeddings within the self-attention block.
  • QuaRot absorbs RMSNorm scaling α into input weight matrices and rotates the hidden state with Q in the attention component.

A.2 Clipping Ratio Ablation

The clipping-ratio ablation compares weight and activation quantization under different optimization constraints. Weight clipping is searched per column, whereas online input quantization requires a constant ratio.

  • Weight quantization uses a linear MSE search to select the best clipping ratio for each weight-matrix column.
  • Online input quantization cannot use per-column clipping because inputs are quantized during inference and require a constant ratio.
  • Ratios of 0.95 for asymmetric KV-cache quantization and 0.9 for symmetric input quantization are identified as suitable.
  • Table 5 evaluates WikiText perplexity for LLAMA2-7B across clipping ratios while keeping the rest of the model in full precision.

A.3 KV Cache Quantization Ablation

The KV-cache ablation evaluates QuaRot with different cache precisions while keeping the rest of the model in high precision. Perplexity degradation remains small even with low-bit KV-cache quantization.

  • KV-cache quantization produces at most 0.21 perplexity degradation up to 3-bit precision, including 0.07 for LLAMA2-70B.
  • The experiment keeps weights and activations in high precision and applies group-wise asymmetric quantization with group-size 128 to keys and values.
  • Table 6 reports WikiText-2 perplexity for various KV-cache precisions using QuaRot.

A.4 Weight-only Quantization Ablation

QuaRot improves weight-only quantization quality by removing outlier features through Hadamard transformations, with larger gains at lower bitwidths.

  • Up to 2.65 lower WikiText-2 perplexity is achieved with QuaRot under 4-bit GPTQ weight-only quantization.The comparison uses asymmetric per-column quantization while inputs and KV cache remain FP16.
  • QuaRot improves weight-only quantization more strongly at 2–3 bits across all evaluated models.GPTQ still has lower perplexity at 2–3 bits than RTN, but QuaRot makes 2-bit GPTQ non-trivially effective.
  • 0.24 perplexity points is the maximum improvement reported for RTN weight-only quantization with QuaRot.

A.5 Random Orthogonal Matrices Ablation

Random orthogonal matrices can be fused into QuaRot’s weights without changing computation, but randomized Hadamard transformations generally provide better perplexity. The broader evaluations also examine precision, model families, and runtime effects.

  • Random Orthogonal Matrices Ablation: Random orthogonal matrices are fused into weights, with online Hadamard transformations retained for selected layer operations.QuaRot starts from a uniformly random matrix and applies QR decomposition to make it orthogonal.
  • Random Orthogonal Matrices Ablation: A 1.35 perplexity gap appears on LLAMA2-7B with random orthogonal matrices, shrinking to 0.28 on LLAMA2-70B relative to randomized Hadamard transformations.The same fast Hadamard kernel remains usable for down-projection and out-projection layers.
  • FP16 Hadamard Transformation Ablation: FP16 rather than FP32 online Hadamard transformations changes WikiText-2 perplexity by less than 0.1 and averaged zero-shot accuracy by less than 0.6% on LLAMA2-7B.On LLAMA2-13B, perplexities are identical and averaged zero-shot results differ by 0.07%.
  • Model and Precision Evaluations: QuaRot evaluates 4-bit quantization on LLAMA-3 and Phi-3-mini-4k-instruct, while LLAMA-2 experiments also report 6- and 8-bit results across perplexity and zero-shot tasks.The task evaluations include PIQA, WinoGrande, HellaSwag, Arc-Easy, Arc-Challenge, and LAMBADA.
  • Runtime Evaluation: QuaRot’s 4-bit linear layer reaches 3.2x speedup on LLAMA2-7B and 4.3x on LLAMA2-70B relative to FP16.The Hadamard operation adds at most 7% overhead to the forward pass, and the reported linear-layer benchmarks use 2048 sequence lengths.
  • Runtime Evaluation: Up to 1.72x decoding speed improvement is reported for the LLAMA2-7B attention layer at larger batch sizes with the 4-bit KV cache.The comparison evaluates decoding a single token when 2047 tokens are already cached.
Loading 2404.00456v2…