PaperlayerView exact revision on arXiv

Source-linked AI summary

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De Sa

arXiv:2307.13304v2cs.LGcs.CL

TL;DR

Large language model quantization needs methods that reduce parameter storage and runtime costs while preserving useful behavior at very low bit widths. QuIP combines adaptive quadratic-proxy rounding with incoherence processing, and reports the first viable two-bit LLM quantization results alongside theory covering QuIP and OPTQ.

  • Problem

    Post-training quantization seeks more efficient inference for massive LLMs, but two-bit quantization requires handling weight and Hessian structure effectively.

  • Method

    QuIP combines adaptive rounding that minimizes a quadratic proxy objective with random-orthogonal pre- and post-processing to ensure weight and Hessian incoherence.

  • Results

    QuIP achieves the first viable two-bit quantization results for LLMs, especially at large model sizes.

  • Takeaways & Limitations

    For models larger than 2B parameters, the observed gap between 2-bit and 4-bit compression decreases with model size, hinting at accurate 2-bit inference.

  • Takeaways & Limitations

    The theoretical optimality analysis does not directly extend to finite-grid rounding with clamping, although experiments found OPTQ soundly beats nearest rounding and the issue did not appear practically.

Abstract

from arXiv · show

This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from $\textit{incoherent}$ weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found at https://github.com/Cornell-RelaxML/QuIP.

1 Introduction

QuIP targets efficient inference for massive LLMs through incoherence-aware two-bit quantization, combining adaptive rounding with random-orthogonal pre- and post-processing. Its theory covers QuIP and OPTQ, while experiments report viable two-bit LLM compression.

  • LLMs can reach hundreds of billions of parameters, motivating sophisticated deployment methods and efficient inference algorithms.
  • QuIP uses incoherence to make weights even in magnitude and important rounding directions less aligned with coordinate axes.The method treats incoherence as a principled form of outlier reduction that facilitates adaptive rounding to compressed values.
  • QuIP combines adaptive quadratic-error rounding with pre- and post-processing based on Kronecker products of random orthogonal matrices.The adaptive step minimizes ℓ(Ŵ) = tr((Ŵ − W)H(Ŵ − W)^T) using an estimate of the Hessian.
  • The theoretical analysis covers QuIP and OPTQ and shows QuIP’s rounding procedure is optimal within a general class of rounding methods.
  • QuIP makes two-bit LLM compression viable, with small 2-bit versus 4-bit gaps for models larger than 2B parameters that decrease with model size.The paper presents this as evidence hinting at accurate 2-bit inference in LLMs.

2 Related Work

Prior work includes adaptive-rounding methods based on a quadratic proxy objective, post-training approaches for large models, and OPTQ’s iterative error-updating procedure. Existing methods vary in retraining requirements, model scale, and rounding strategy.

  • Many adaptive-rounding methods optimize a principled quadratic proxy objective, but some require expensive further retraining or lack evaluation on the largest open LLMs.
  • Large-model post-training quantization methods often reduce weight or activation ranges while retaining nearest rounding.SmoothQuant rescales activations and weights to remove activation outliers, while ZeroQuant is another cited approach in this setting.
  • OPTQ iteratively quantizes weight columns, computes rounding error, updates remaining weights with scaled error, and repeats in a fixed order.
  • Other quantization procedures either do not optimize the adaptive-rounding proxy objective or are not designed for the largest language models.

3 Quantization With Incoherence Processing: Adaptive Rounding Step

QuIP’s adaptive rounding step minimizes a Hessian-weighted quadratic proxy objective, while LDLQ selects optimal linear error feedback within the considered rounding class. The analysis shows how incoherence and Hessian structure improve proxy-loss guarantees and motivates QuIP’s preprocessing pipeline.

  • Adaptive rounding: QuIP rounds each layer by minimizing a quadratic proxy objective weighted by an estimated Hessian.The formulation supports parallel quantization across neurons, making it tractable for large language models.
  • Adaptive rounding: Adaptive rounding iteratively quantizes columns and adds a linear correction based on residual errors from previously rounded columns.The correction vectors define an upper-triangular feedback matrix, so each column depends only on earlier quantized columns.
  • LDLQ: LDLQ chooses the feedback matrix from the LDL decomposition of H, canceling the corresponding factors in the transformed proxy loss.This makes the resulting procedure optimal within the analyzed family of methods whose feedback depends on H rather than W.
  • Optimality guarantees: LDLQ is worst- and average-case optimal for nearest or stochastic rounding, with c = 12 for nearest and c = 6 for stochastic rounding.Nearest rounding has the same worst-case proxy loss as stochastic rounding but a better average proxy loss.
  • Optimality guarantees: Across OPT-125m to 2.7b models, tr(D) / tr(H) ≤ 0.65, indicating a nontrivial gap between LDLQ and plain rounding baselines.For non-diagonal positive semidefinite Hessians, LDLQ achieves strictly lower worst- and average-case proxy loss than the corresponding baselines.
  • Incoherence and spectral bounds: Incoherence processing reduces weight and Hessian-eigenvector outliers, while low-rank Hessians enable spectral bounds on LDLQ’s proxy loss.The paper reports that this yields asymptotically better loss than plain rounding for sufficiently low-rank H by a factor of µ2k/n.

4 Quantization With Incoherence Processing: Incoherence Processing Step

QuIP's incoherence-processing step pre- and post-processes weights and Hessians with fast random orthogonal transformations, preserving the proxy quadratic form while making matrices incoherent. The full procedure combines this processing with adaptive rounding and optional heuristics such as diagonal rescaling and greedy local search.

  • Incoherence via Orthogonal Multiplication: Random orthogonal multiplications make the weight and Hessian matrices incoherent while preserving the proxy quadratic form.The transformations are ˜H ← VHV^T and ˜W ← UWV^T, with tr(˜W˜H˜W^T) = tr(WHW^T).
  • Incoherence via Efficient Orthogonal Multiplication: Kronecker-factored random orthogonal matrices reduce multiplication to O(n(p + q)) = o(n^2) operations.For n = pq, vectors are reshaped into p × q matrices, multiplied on the left and right, and reshaped back.
  • Incoherence Guarantees: With independent Kronecker factors, the transformed Hessian and weights are incoherent with high probability and a poly-logarithmic incoherence parameter.The experiments use k = 2 factors for constructing U and V.
  • QuIP Procedure: QuIP applies fast orthogonal preprocessing, randomly permutes entries, then performs LDLQ with clamping before orthogonal post-processing.The algorithm also includes heuristic improvements beyond the core preprocessing and post-processing steps.
  • Incoherence-Based Heuristics: Diagonal rescaling trades off matrix spectra, while quantization ranges use ||W||_F rather than the largest absolute weight.These heuristics target the proxy loss and exploit incoherence in the weights.
  • Greedy Local Search: Greedy coordinate descent after LDLQ can further lower the proxy loss before post-processing.Updates follow the same coordinate order as the initial LDLQ pass.

5 Extensions and Further Analyses

The paper establishes an equivalence between LDLQ and OPTQ and analyzes a limitation that appears when adaptive rounding is clamped to a finite grid. In that setting, clamped LDLQ can be asymptotically worse, motivating a bounded procedure that is theoretically stronger but usually not used in practice.

  • OPTQ Equivalence: OPTQ falls within the adaptive-rounding class with linear feedback and is equivalent to LDLQ.This equivalence supplies the paper's theoretical analysis of OPTQ.
  • OPTQ Equivalence: LDLQ is more efficient than OPTQ because it avoids matrix inversion and one of OPTQ's two Cholesky decompositions.OPTQ requires a matrix inversion of H and two Cholesky decompositions, whereas LDLQ uses only one Cholesky decomposition.
  • OPTQ Equivalence: The OPTQ and LDLQ implementations produce exactly identical quantized outputs on synthetic random data.The experiment uses W ∼ Unif[0, 1]^(1000×1000).
  • Finite-Grid Limitation: When rounding is restricted to a finite 4-bit grid [0, 15], clamped LDLQ with nearest rounding is asymptotically worse in a constructed example.The passage attributes this behavior to clamping to the finite grid.
  • Finite-Grid Limitation: Although OPTQ beats nearest rounding in practice, the clamping issue may need to be addressed to prove useful end-to-end bounds.The authors state that clamping rarely causes practical problems but can matter for theory.
  • Bounded Procedure: A bounded procedure constrains |Ŵ_ij − W_ij| so rounded weights remain inside the finite grid when W is sufficiently far from its boundary.The method solves a convex optimization problem and then uses stochastic rounding with U = R^-1 − I.
  • Bounded Procedure: Despite its theoretical role, the bounded procedure is not used in practice because clamping rarely causes issues and solving the program adds substantial computation.The practical implementation instead uses QuIP with large c and nearest rounding.

6 Experiments

Experiments show that QuIP’s incoherence processing enables strong low-bit quantization across model sizes, tasks, and rounding methods, including viable two-bit results. The method outperforms OPTQ broadly, with an inference-throughput cost of about 1.5×.

  • Setup: QuIP is evaluated on OPT models up to 66B and Llama 2 70B across language-generation and zero-shot tasks.The evaluation includes WikiText2, PTB, C4, LAMBADA, ARC Easy, PiQA, and StoryCloze.
  • Main Results: As model size increases, the performance gap between two-bit and full-precision weights becomes small.The reported trend is observed for large LLMs and is described as evidence toward accurate two-bit inference.
  • Main Results: QuIP is superior to OPTQ across model sizes and evaluation tasks, while three-bit quantization reasonably matches full precision.At two bits, performance approaches full precision for larger LLMs above 2B parameters.
  • Incoherence Processing Ablation: Two-bit QuIP quantization is viable across rounding methods, including nearest rounding, whereas other two-bit methods fail at moderate model sizes.Incoherence processing dramatically improves lower-bit performance across all evaluation tasks.
  • Throughput Comparison: QuIP’s implementation is about 1.5× slower than OPTQ during inference.The added incoherence-processing operations are implemented in PyTorch within OPTQ’s efficient forward pass.
  • Further Ablation: All evaluated incoherence-processing sub-steps contribute to the full improvement, and random permutation significantly reduces perplexity.These findings come from ablations on OPT-350m and OPT-125m.

7 Conclusion

The paper concludes that QuIP combines adaptive rounding with incoherence processing and provides a theoretical foundation for scalable LLM quantization. Empirically, it makes viable two-bit LLM quantization possible, especially for large models, while leaving broader usage implications outside its scope.

  • 7 Conclusion: QuIP combines optimal adaptive rounding with efficient pre- and post-processing that enforces weight and Hessian incoherence.The processing multiplies matrices by a Kronecker product of random orthogonal matrices.
  • 7 Conclusion: The theory establishes optimality for QuIP within a general class of adaptive rounding methods with linear feedback.The analysis is presented as the first for a quantization algorithm scaling to LLM-sized models.
  • 7 Conclusion: QuIP achieves the first viable two-bit quantization results for LLMs, particularly at large model sizes.The authors describe this result as hinting at accurate two-bit inference in LLMs.
  • 7 Conclusion: The paper aims to improve LLM efficiency but does not address the purposes for which those LLMs are used.This scope statement limits the paper’s discussion of downstream use.

A.2 Limitations

The adaptive-rounding proxy objective treats each layer independently, leaving the value of other tractable cross-layer proxies unresolved. Existing methods that model layer interactions remain too expensive for the largest open LLMs.

  • A.2 Limitations: QuIP’s adaptive-rounding proxy objective considers each layer in isolation.The authors leave open whether other computationally tractable proxies could improve quantization.
  • A.2 Limitations: Cross-layer quantization methods exist, but their computational cost has prevented application to the largest open LLMs.This marks the practical boundary of the paper’s layerwise proxy approach.

B Additional Method Clarifications

The supplementary material clarifies greedy local search, proxy-loss comparisons, and implementation details for quantization methods. It also reports a counterexample where OPTQ can underperform nearest rounding.

  • Greedy local search: Greedy local search performs restricted coordinate descent on the proxy loss over the quantization grid.Each update changes one weight while holding the others fixed; a full pass performs mn updates in LDLQ order.
  • Greedy local search: A single greedy-update pass fits adaptive rounding with linear feedback U = (H⊙M) diag(H)^−1.Here M is the strictly upper triangular mask, and nearest rounding with clamping projects each update onto the representable grid.
  • Greedy local search: Greedy updates can operate standalone or refine another quantizer, with multiple passes used for post-processing.When used after an initial quantization method, each update cannot increase the proxy loss; multiple passes such as 10 are possible.
  • Proxy-loss analysis: For OPT models, H becomes approximately lower-rank as model size increases, with OPT-2.7b having approximate fractional rank ≈0.02(±0.02).The reported statistics are computed across layers, alongside tr(D)/tr(H) and absolute and approximate fractional ranks.
  • Counterexample: A constructed finite-grid example makes OPTQ round in the wrong direction, whereas nearest rounding avoids that error under the stated perturbation.The construction uses c=0.01; with c=0, nearest rounding also performs poorly.

C.5 Section 6 (Main Results on Additional Evaluations)

Additional evaluations reinforce QuIP’s advantage over OPTQ across model sizes, tasks, and quantization settings. Incoherence processing enables a marked improvement at two bits, although some more elaborate processing adds computational cost without consistent gains.

  • Additional evaluations: QuIP is superior to OPTQ across OPT model sizes and additional WikiText2, PiQA, and StoryCloze evaluation tasks.These evaluations cover 2- and 3-bit quantization for OPT models up to 30B parameters.
  • Additional evaluations: Incoherence processing enables a step-function change in quantization at 2 bits across OPT models from 125m to 30b and across rounding methods.The reported tables evaluate combinations of quantization and pre/post-processing methods on language-generation and zero-shot tasks.
  • Rounding choices: Unbiased rounding performs worse than biased nearest rounding for LDLQ/OPTQ across WikiText2, PTB, and C4, with the gap increasing at lower bit widths.The comparison uses average perplexity difference defined as unbiased minus biased rounding across OPT models 125m to 2.7b.
  • Clamping-aware processing: Algorithm 5 yields modest 2-bit improvements on OPT-125m and OPT-350m but diminishing improvements on OPT-1.3b.Because its OPT-1.3b performance is relatively equivalent to QuIP while requiring more computation, the authors do not use it.
  • Clamping-aware processing: In practice, the authors do not observe constructions that make LDLQ/OPTQ vulnerable to the clamping issue, especially as model size increases.This is presented as an empirical observation rather than a guarantee for all possible weight and Hessian matrices.

D Proofs for Section 3 (Quantization With Incoherence Processing: Adaptive Rounding Step )

The proofs establish optimality properties for adaptive rounding under a quadratic proxy loss and relate LDLQ’s behavior to Hessian structure. They also characterize the losses of nearest and stochastic rounding and compare methods across Hessian orientations.

  • Adaptive rounding optimality: LDLQ is worst- and average-case optimal among rounding methods whose linear feedback depends on H rather than W.The theorem applies to positive semidefinite H, integer rounding, and either nearest or stochastic rounding.
  • Adaptive rounding optimality: The recurrence underlying adaptive rounding achieves its global minimum when each a_k is the kth column of the LDL factor L, with minimum proxy loss tr(D).This identifies the LDL assignment as the optimizer of the recurrence’s loss.
  • Incoherence and spectral bounds: For incoherent positive semidefinite Hessians, the LDL decomposition supports a spectral bound connecting tr(D) with the Hessian spectrum.The proof uses the incoherence condition and continuity to extend the result from positive definite to positive semidefinite matrices.
  • Baseline rounding: Nearest and stochastic rounding have average-case proxy losses m/12 tr(H) and m/6 tr(H), respectively, while stochastic rounding has worst-case loss (m/4) tr(H).These expressions are stated for symmetric positive definite H.
  • Fixed-spectrum comparison: Across all Hessian matrices with a fixed spectrum, LDLQ matches stochastic rounding in worst-case loss and matches the corresponding routine in average-case loss.The comparison is over Hessian orientations sharing the same eigenvalues.

E Proofs for Section 4 (Quantization With Incoherence Processing: Incoherence Processing Step )

The incoherence-processing proof shows that structured random orthogonal transformations make both Hessian and weight matrices incoherent with high probability. Kronecker structure provides an efficient way to obtain these transformations.

  • Randomized incoherence processing: Independent random orthogonal matrices combined through Kronecker products transform H and W into incoherent matrices with probability at least 1−δ.The construction uses separate products for Hessian and weight dimensions, with incoherence parameters controlled by global constants.
  • Hessian transformation: The proof analyzes transformed Hessian eigenvectors as products of independent random unit vectors and applies concentration bounds with a union bound.This controls the entries of the transformed orthogonal eigenvector matrix.
  • Weight transformation: For weights, flattening W turns W/∥W∥F into a unit vector, allowing the same random-product argument to bound entries after transformation.The weight analysis uses independent random unit vectors on both sides of the matrix transformation.

F Proofs for Section 5 (Extensions and Further Analyses)

The proofs establish how adaptive rounding with linear feedback relates to OPTQ and how constrained rounding avoids clamping. They also derive probabilistic in-range guarantees for finite-grid quantization, including with incoherence processing.

  • OPTQ equivalence: OPTQ is equivalent to an adaptive rounding procedure with linear feedback, using U from the LDL decomposition of the Hessian.The proof tracks OPTQ’s sequential minimization, nearest rounding, and clamping updates.
  • Clamping correction: Algorithm 5 addresses OPTQ’s clamping issue by restricting |Ŵij − Wij| within the optimization objective.
  • Finite-grid rounding: The finite-grid rounding analysis bounds the quadratic objective while enforcing constraints on the unit upper triangular feedback matrix L.The construction selects hyperparameters so the constraints hold and derives a bound involving n · min(1, c) and tr terms.
  • In-range guarantees: With probability at least 1 − δ, constrained quantization can keep all quantized weights in range without overflow or clipping under bounded-entry assumptions.The result rescales weights before quantization and undoes the scaling afterward.
  • Incoherence extension: With probability at least 1 − δ, the same in-range guarantee extends to weights processed by two-factor orthogonal matrices under a bounded Frobenius norm assumption.The proof combines bounds on weight magnitude, Hessian incoherence, and stochastic rounding.
Loading 2307.13304v2…