PaperlayerView exact revision on arXiv

Source-linked AI summary

QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, Christopher De Sa

arXiv:2402.04396v2cs.LGcs.AIcs.CL

TL;DR

Large language models require substantial memory, while existing PTQ methods struggle with extreme compression or fast decoding. QuIP# combines Randomized Hadamard incoherence processing, E8 lattice codebooks, and inter-layer fine-tuning, achieving state-of-the-art results with stronger 3-bit than 4-bit scaling and fast inference.

  • Problem

    Existing PTQ methods either do not scale to extreme compression ratios or rely on expensive decoding schemes, limiting efficient low-precision LLM deployment.

  • Method

    QuIP# is a weight-only PTQ method combining Randomized Hadamard incoherence processing, E8 lattice-based codebooks, adaptive rounding, and inter-layer fine-tuning.

  • Results

    QuIP# achieves state-of-the-art results at 2, 3, and 4 bits per weight, with superior 3-bit over 4-bit scaling and over 50% peak memory bandwidth in a proof-of-concept CUDA implementation.

  • Takeaways & Limitations

    QuIP# demonstrates that extreme PTQ compression can combine strong model quality with fast inference and new scaling behavior across bitrates.

  • Takeaways & Limitations

    Vector quantization has exponential cost in bitrate and vector dimension, and practical GPU cache constraints limit unstructured codebooks.

Abstract

from arXiv · show

Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision. In this work, we introduce QuIP#, a weight-only PTQ method that achieves state-of-the-art results in extreme compression regimes ($\le$ 4 bits per weight) using three novel techniques. First, QuIP# improves QuIP's (Chee et al., 2023) incoherence processing by using the randomized Hadamard transform, which is faster and has better theoretical properties. Second, QuIP# uses vector quantization to take advantage of the ball-shaped sub-Gaussian distribution that incoherent weights possess: specifically, we introduce a set of hardware-efficient codebooks based on the highly symmetric $E_8$ lattice, which achieves the optimal 8-dimension unit ball packing. Third, QuIP# uses fine-tuning to improve fidelity to the original model. Our experiments show that QuIP# outperforms existing PTQ methods, enables new behaviors in PTQ scaling, and supports fast inference. Our code can be found at https://github.com/Cornell-RelaxML/quip-sharp.

1. Introduction

QuIP# addresses the deployment cost of large language models by combining efficient incoherence processing, lattice-based vector quantization, and inter-layer fine-tuning. It reports state-of-the-art quantization results at extreme compression, including stronger 3-bit scaling than 4-bit models and fast inference.

  • Motivation: 140GB of GPU memory is required for Llama 2 70B at native 16-bit precision, motivating lower-memory compression.Post-training quantization can reduce Llama 2 70B below 20GB at 2 bits and improve throughput in memory-bound autoregressive decoding.
  • Motivation: Existing quantization methods either fail to scale to extreme compression ratios or use expensive decoding schemes.These limitations motivate PTQ methods that are both accurate and fast.
  • Results: QuIP# significantly outperforms OmniQuant, QuIP, and AQLM, while its 3-bit models scale better than theoretically lossless 4-bit models.A proof-of-concept CUDA implementation achieves over 50% of peak memory bandwidth on an NVIDIA RTX 4090.
  • Method: QuIP# uses the computationally efficient Randomized Hadamard Transform for incoherence processing, improving theoretical incoherence properties and runtime over QuIP’s Kronecker factorization.The transform is part of QuIP#’s outlier-suppression pipeline.
  • Method: QuIP# rounds incoherence-processed weights with block adaptive rounding and hardware-friendly codebooks based on the highly symmetric E8 lattice.E8 provides the highest 8-dimensional unit-ball packing density, while its structure supports fast inference.
  • Method: QuIP# adds inter-layer fine-tuning to further improve quantization quality and fidelity to the original model.The method is presented as a weight-only PTQ approach with optional fine-tuning.

2. Background / Related Work

Prior LLM compression work uses pruning, QAT, and PTQ, but extreme PTQ faces quality, decoding-cost, and vector-quantization constraints. QuIP# addresses these with incoherence processing, structured E8 codebooks, and inter-layer fine-tuning.

  • Incoherence Processing: Incoherence processing suppresses outliers in weights and Hessians, enabling quantization with provably bounded error.QuIP applies structured random orthogonal transformations to both matrices while preserving the proxy objective.
  • Vector Quantization: QuIP# replaces scalar quantization with vector quantization using structured codebooks based on the 8-dimensional E8 lattice.The E8-based codebook addresses vector quantization’s exponential cost and cache constraints; E8P achieves k_d = 16 while allowing 256× compression.
  • Fine-Tuning: Inter-layer fine-tuning captures interactions between layers with substantially less data and compute than full QAT.QuIP# can quantize a 70B-parameter model in a few hours on one 8-GPU node, while LLM-QAT requires 960 GPU-hours to generate training data alone.

3. Incoherence Processing with the Randomized Hadamard Transform

QuIP# replaces QuIP’s Kronecker-based incoherence processing with a randomized Hadamard transform that improves theoretical incoherence bounds and reduces transform cost. RFFT provides an alternative with similar guarantees when suitable Hadamard factorizations are unavailable.

  • RHT Advantages: RHT improves QuIP’s incoherence processing by strengthening the theoretical bound on µ and reducing asymptotic transform cost from Θ(n√n) to Θ(n log n).Hadamard multiplication also avoids floating-point multiplies because its entries are ±1.
  • RHT Construction: RHT applies randomized signs and a Hadamard transform to vectors, achieving O(n log n) computation when dimensions are powers of two.The same construction is used to conjugate weight and Hessian matrices for incoherence processing.
  • Theoretical Guarantees: RHT achieves a logarithmic dependence on matrix size for incoherence, improving on QuIP’s log-squared dependence while preserving the proxy-loss theory.The improved incoherence rates propagate through QuIP’s existing analysis.
  • Non-Power-of-Two Dimensions: For non-power-of-two dimensions, Hadamard factorizations can provide faster multiplication than QuIP’s two-factor Kronecker approach when p ≫ q.The construction uses V = H_p ⊗ H_q and computes the transform in O(q^2p log p).
  • RFFT Alternative: RFFT offers similar runtime and concentration properties when an appropriate Hadamard factorization does not exist, while remaining widely supported by hardware.In practice, RFFT performs slightly worse than RHT but still achieves strong results.

4. BlockLDLQ and Lattice Codebooks

QuIP# extends adaptive rounding to vector quantization and uses structured E8 lattice codebooks to match incoherent weights while avoiding VQ's hardware and scaling costs. Residual vector quantization then supports higher bitrates.

  • 4.2. The E8P (“E8 Padded”) Codebook: VQ can better match the roughly ball-shaped distribution of incoherent weights, but unrestricted codebooks become intractable because their size is exponential in dimension and bitrate.GPU inference also constrains unstructured codebooks to fit in L1 cache after bank-conflict duplication.
  • 4.1. Adaptive Rounding for Vector Quantization: BlockLDLQ extends LDLQ to vector quantization by using a g-block LDL decomposition and rounding weight blocks with a vector quantizer.The decomposition uses a unit block lower triangular matrix and block diagonal matrix, with U = L^T − I.
  • 4.1. Adaptive Rounding for Vector Quantization: Theorem 4.1 bounds the Hessian-weighted quantization error when block LDLQ uses a µ-incoherent Hessian and a vector quantizer with bounded covariance error.The quantizer must satisfy E[(Q(x) − x)(Q(x) − x)^T] ⪯ σ^2I for vectors of dimension g.
  • 4.2. The E8P (“E8 Padded”) Codebook: E8P is a 2-bit, 8-dimensional lattice codebook with 2^16 entries that requires lookup into only a 2^8-entry table, reducing storage and decoding cost.Its 1 KiB footprint fits in modern GPU L1 cache, including duplicated copies for bank conflicts.
  • 4.2. The E8P (“E8 Padded”) Codebook: E8P exploits E8 symmetry: absolute-value entries combine with parity-constrained sign flips and shifts to represent codewords efficiently.The construction uses 8 bits for the source entry, 7 bits for sign flips, and 1 bit for the shift.
  • 4.2. The E8P (“E8 Padded”) Codebook: Figure 3 shows lower Gaussian quantization MSE for E8-based codebooks than for the other presented codebooks, reflecting E8's packing density and dimensionality.The comparison includes D4-based codebooks and illustrates that increasing vector dimension can reduce distortion.
  • 4.3. Scaling E8 to Higher Bitrates: Residual vector quantization scales E8 lattice codebooks to higher bitrates by repeatedly quantizing the remaining residual.QuIP# uses two 2-bit E8P rounds for 4 bits and combines 2-bit E8P with a 1-bit E8 codebook for 3 bits.

5. Fine-Tuning During Quantization

QuIP# uses fine-tuning during quantization to recover the original model's behavior by compensating for quantized layers within transformer blocks and across the full model. The approach is reported to support scaling behavior in which 3-bit models outperform 4-bit models for Llama 1.

  • 5. Fine-Tuning During Quantization: QuIP# fine-tunes on a small development set to recover the original unquantized model during quantization.For a 70B-parameter model, this process takes around 50 GPU-hours.
  • 5. Fine-Tuning During Quantization: Within each transformer block, unquantized layers are fine-tuned before quantization to compensate for layers that have already been quantized.This approach can be parallelized across transformer blocks.
  • 5. Fine-Tuning During Quantization: For Llama 1, QuIP# 3-bit models scale better than 4-bit models, while 2-bit models scale similarly to higher bitrates.The caption reports this scaling pattern as analogous to Llama 2.
  • 5. Fine-Tuning During Quantization: After all linear layers are quantized, remaining unquantized parameters are fine-tuned to minimize activation error over the entire model.The method optimizes sign vectors as real rather than binary vectors in both fine-tuning steps.

6. Experiments

Experiments evaluate QuIP# on Llama models across perplexity, zeroshot accuracy, bit scaling, inference speed, and component ablations. QuIP# performs strongly at 2–4 bits while retaining fast decoding through its hardware-efficient design.

  • 6.1. QuIP# on Llama Models: QuIP# produces high-quality 2-bit models, whereas AWQ falls apart near 2.15 bits and OmniQuant produces unusable 2-bit models.The comparison uses Wikitext2 and C4 perplexity at context length 2048.
  • 6.1. QuIP# on Llama Models: QuIP# and AQLM significantly outperform OmniQuant on zeroshot accuracy, while QuIP# tends to outperform AQLM at lower bitrates and model sizes.Both methods perform close to FP16 at higher bitrates and for larger models; zeroshot results can vary by up to 0.5%.
  • 6.2. QuIP# Bit Scaling: 3-bit QuIP# outperforms 4-bit QuIP#, while 2-bit QuIP# offers similar scaling to 3- and 4-bit models on Llama 1 and 2.On Llama 2, 3-bit QuIP# also outperforms a theoretical lossless 4-bit model.
  • 6.3. Efficient Inference with QuIP#: AQLM inference is slower than FP16 because its 1MiB codebooks exceed current GPU L1 caches and cause high cache miss rates.The H100 has a 256KB L1 cache, smaller than AQLM’s codebook.
  • 6.3. Efficient Inference with QuIP#: QuIP# reaches over 50% of peak memory bandwidth with a 2-bit model despite minimal randomized-Hadamard kernel fusion.The result validates the method’s fast-inference design.
  • 6.3. Efficient Inference with QuIP#: Compared with QuIP at the same bitrate on an A6000, QuIP# achieves roughly twice the inference throughput.The paper reports this as a speed-quality frontier for PTQ.
  • 6.4. Ablations: At all tested bitrates, removing fine-tuning or E8 lattice codebooks reduces performance, and randomized Hadamard processing improves over QuIP’s Kronecker factorization.The ablation attributes the latter improvement to stronger incoherence properties.

7. Conclusion

QuIP# is a weight-only PTQ method for 2–4-bit LLM compression that combines randomized Hadamard incoherence processing, E8 lattice codebooks, and inter-layer fine-tuning. It achieves state-of-the-art results while supporting efficient inference and unusual bit-scaling behavior.

  • 7. Conclusion: QuIP# achieves state-of-the-art LLM results at 2, 3, and 4 bits per weight.It is a weight-only post-training compression method.
  • 7. Conclusion: Randomized Hadamard processing suppresses outliers efficiently, while the symmetric E8P codebook improves quantization and supports fast inference.The proof-of-concept CUDA implementation exceeds 50% of peak memory bandwidth.
  • 7. Conclusion: Inter-layer fine-tuning further improves quantization quality.The conclusion presents fine-tuning as an additional component beyond incoherence processing and lattice codebooks.
  • 7. Conclusion: QuIP# shows superior scaling at 3 bits over 4 bits and similar scaling at 2 bits to higher bitrates.The paper projects that 2-bit models are likely to scale better than 3-bit models in the near future.

A.1. Incoherence Processing with the Randomized Hadamard Transform

The appendix formalizes randomized orthogonal transforms for making weights and Hessians incoherent, then connects block LDLQ and vector quantization to quantization-error control. It also describes FFT alternatives and practical E8P codebook decoding.

  • Randomized transform guarantees: Randomized Hadamard or FFT multiplication makes orthogonal matrices incoherent with high probability under the stated randomization assumptions.The guarantees are established for vectors and for matrices transformed on both sides.
  • FFT alternative: The randomized Fourier transform can be computed in O(n log n) time and offers theoretical guarantees similar to the randomized Hadamard transform.The paper notes that FFT is a practical alternative when a fast Hadamard implementation is unavailable.
  • Quantization-error bound: Under Hessian incoherence and a bounded vector-quantizer error covariance, Theorem 4.1 controls the expected Hessian-weighted quantization error.The theorem assumes E[(Q(x) − x)(Q(x) − x)^T] ⪯ σ^2I for each g-dimensional input.
  • E8P decoding: The E8P codebook uses structured, symmetric lattice representations and bit fields for an entry index, sign flips, and a shift decision.The decoding example illustrates how these fields reconstruct an E8 or E8 + 1 vector.
  • E8P codebook design: E8P is motivated by approximately Gaussian post-incoherence weights, while symmetry is needed to keep the codebook compressible.The paper contrasts this design with learned K-means codebooks.

E. Additional Results

Additional results indicate that QuIP# extends beyond the main Llama experiments to different model architectures, including Mixtral and Falcon.

  • E. Additional Results: QuIP# scales to different architectures without issue.The additional-results table includes 2-bit QuIP# without fine-tuning on Mixtral 8x7B and Falcon 180B.

E.3. Zeroshot performance for ablation on lattice codebooks and fine-tuning

The ablation compares QuIP# variants that progressively add lattice codebooks and fine-tuning, while Figure 5 shows scaling against AQLM across Llama benchmarks.

  • QuIP# no FT and E8 uses randomized Hadamard incoherence processing without lattice codebooks or fine-tuning.
  • QuIP# No FT adds lattice codebooks but omits fine-tuning.
  • QuIP# combines lattice codebooks with fine-tuning.
  • Figure 5 compares Llama 2 Wikitext 2 perplexity against AQLM at 2 and 3 bits, using context length 4096.
  • Figure 5 also reports Llama 2 C4 perplexity at context length 4096 and Llama 1 C4 perplexity at context length 2048.

F. Implementation Details

The experiments use specified datasets, calibration procedures, quantization settings, fine-tuning configurations, and GPU environments for Llama, Mixtral, and Falcon models.

  • The implementation details primarily describe Llama experiments, with most settings also applying to Mixtral and Falcon except for Falcon’s publicly available dataset.
  • Hessian matrices were generated from 6144 native-context-length sequences drawn from RedPajama 1T.
  • Perplexity uses OPTQ’s Wikitext2 and C4 sampling functions, while zero-shot scores use LM Eval.
  • Weights are scaled by ρ|W|F before quantization, with codebook-specific ρ values selected by minimizing Gaussian quantization error.
  • Fine-tuning uses Adam at learning rate 5 × 10^-5 on RedPajama 1T development and validation sequences, with five epochs and best-validation checkpoint selection.
  • Quantizing Llama 2 70B takes under 10 GPU-hours without fine-tuning and around 100 GPU-hours with fine-tuning, excluding reusable Hessian generation.

G. Example Generation

The section presents example outputs from Llama 2 70B chat quantized to 2 bits, including CUDA code, a tongue-twister response, and a song about language models.

  • The examples are generated by Llama 2 70B chat quantized with QuIP# to 2 bits and truncated to 256 tokens.
  • For the woodchuck tongue-twister, the response explains that woodchucks move soil while offering no scientific evidence that they chuck wood.
  • For a Mariah Carey tune prompt, the response produces verses and choruses about wanting a large language model for writing assistance.
  • For a matrix-multiplication prompt, the output includes CUDA headers, a GPU kernel, indexing logic, accumulation, and example matrix dimensions.
  • The matrix-multiplication example continues with the output assignment and initializes a 3 × 3 matrix configuration.
Loading 2402.04396v2…