PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, Kwang-Ting Cheng

arXiv:2310.16836v1cs.CLcs.AIcs.ARcs.CV

TL;DR

Transformer compression is needed because model size and computational costs challenge use where resources are limited, while existing PTQ approaches are primarily integer-based and struggle below 8 bits. The paper combines searched FP quantization parameters with per-channel activation scaling reparameterized as weight exponential biases. It reports a 5.8-point zero-shot reasoning degradation from full precision for a 4-bit LLaMA-13B model.

  • Problem

    Transformer model size and computational costs challenge use in resource-limited settings, and existing PTQ solutions primarily use integers that often break down below 8 bits.

    Source: §1 · p. 1

  • Method

    The method searches exponent bits and maximum values for FP quantization, then handles activation-channel variance with per-channel scales reparameterized as weight exponential biases.

    Source: §1 · p. 2

  • Results

    5.8-point degradation on zero-shot reasoning tasks versus full precision was reported for the first usable FP4 weight-and-activation quantized LLaMA-13B model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach demonstrates 4-bit floating-point PTQ across language transformers, BERT, and vision transformers.

    Source: §6 · p. 9

    Generalizability to extremely long sequences or streaming data is unverified, and applicability beyond language and vision remains to be investigated.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We propose LLM-FP4 for quantizing both weights and activations in large language models (LLMs) down to 4-bit floating-point values, in a post-training manner. Existing post-training quantization (PTQ) solutions are primarily integer-based and struggle with bit widths below 8 bits. Compared to integer quantization, floating-point (FP) quantization is more flexible and can better handle long-tail or bell-shaped distributions, and it has emerged as a default choice in many hardware platforms. One characteristic of FP quantization is that its performance largely depends on the choice of exponent bits and clipping range. In this regard, we construct a strong FP-PTQ baseline by searching for the optimal quantization parameters. Furthermore, we observe a high inter-channel variance and low intra-channel variance pattern in activation distributions, which adds activation quantization difficulty. We recognize this pattern to be consistent across a spectrum of transformer models designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models. To tackle this, we propose per-channel activation quantization and show that these additional scaling factors can be reparameterized as exponential biases of weights, incurring a negligible cost. Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63.1 on the common sense zero-shot reasoning tasks, which is only 5.8 lower than the full-precision model, significantly outperforming the previous state-of-the-art by 12.7 points. Code is available at: https://github.com/nbasyl/LLM-FP4.

1 Introduction

The paper addresses the challenge of compressing transformers with floating-point post-training quantization, whose results depend on choosing suitable quantization parameters. It proposes a search-based FP-PTQ recipe and a per-channel activation approach, reporting 4-bit results across language and vision transformers.

1 Introduction

  • Post-training quantization is convenient, but existing transformer methods primarily use integers and often break down below 8 bits.Floating-point quantization is more flexible for varied weight and activation distributions and is supported as a default choice on some hardware platforms.Source: §1 · p. 1
  • A search-based recipe jointly selects exponent bits and maximum values through layer-wise reconstruction, providing a more stable FP-PTQ baseline than gradient updates.Source: §1 · p. 2
  • High inter-channel and low intra-channel activation variance appears across language and vision transformers; per-channel scales are reparameterized as weight exponential biases at negligible cost.Source: §1 · p. 2
  • A 5.8-point degradation on zero-shot reasoning tasks versus full precision accompanies the first usable FP4 weight-and-activation quantized LLaMA-13B model.The reported approach reduces the gap to the previous state of the art by approximately 70%.Source: §1 · p. 2
  • The method exceeds the previous best 4-bit quantized BERT by 7.8 points on GLUE and the previous 4-bit DeiT-S ImageNet method by 31.4 points in accuracy.Source: §1 · p. 2

2 Related Works

Related work describes limited transformer PTQ research and the potential of floating-point quantization, motivating a low-bit FP-PTQ baseline for language transformers.

2.1 Post-Training Quantization

  • Transformer PTQ research has been limited, with prior work focused mainly on vision transformers and weaker performance below 8 bits.Source: §2.1 · p. 2
  • Most PTQ studies target CNNs, while only a limited number address transformer models.Source: §2.1 · p. 2

2.2 Floating-Point Quantization

  • Floating-point quantization offers flexibility for long-tail distributions and hardware support, but prior FP work was sparse and the paper proposes an FPQ baseline for low-bit language-transformer PTQ.Source: §2.2 · p. 2

3 Preliminaries

These preliminaries define floating-point formats and describe how real-valued variables are quantized and used in matrix multiplication. The process combines scaling and clipping with selection among floating-point quantization levels.

3.1 Formulation of Floating-Point Variables

  • A floating-point format encodes sign, exponent, and mantissa bits; E_jM_k denotes a format with j exponent bits and k mantissa bits.Source: §3.1 · pp. 2–3

3.2 Floating-Point Quantization Process

  • Floating-point quantization scales and clips real-valued variables, then compares them with quantization levels and selects a power-of-two step size according to magnitude.Source: §3.2 · p. 3

3.3 Floating-Point Matrix Multiplication

  • Matrix multiplication supports per-tensor activation quantization and per-channel weight quantization, with each output entry formed from quantized vectors and their real-valued scaling factors.Source: §3.3 · pp. 3–4

4 Method

The method combines a search-based floating-point quantization baseline with pre-shifted exponent bias to address activation variance while keeping matrix multiplication nearly as efficient as per-tensor quantization.

4 Method

  • The method combines joint format and maximum-value search with pre-shifted exponent bias to push transformer quantization to 4-bit.The combined method is named Floating Point Quantization (FPQ).Source: §4 · p. 4, §4.2 · p. 6

4.1 Joint Format and Max Value Search

  • The FPQ baseline jointly searches for the quantization format and clipping range, and achieves state-of-the-art results at 8-bit and 6-bit settings.The search proceeds layer by layer, minimizing a reconstruction metric.Source: §4.1 · p. 4

4.2 Pre-Shifted Exponent Bias

  • Per-channel activation biases address high inter-channel variance but are reparameterized into corresponding weights to maintain nearly identical efficiency to per-tensor quantization.The biases are applied during calibration through precomputed weights, avoiding per-channel bias shifts during inference.Source: §4.2 · pp. 4–6

5 Experiments

FPQ is evaluated on language and vision transformers, with experiments covering low-bit accuracy, calibration and search robustness, and hardware area. Its strongest reported results are at 4 bits, where it outperforms prior methods across LLaMA, BERT, and ViT evaluations.

5.1 Experiments Details

  • Experiments use layer reconstruction and parallel quantization; LLaMA quantizes fully connected layers, while BERT and ViT also quantize self-attention activation multiplications.Activations use per-tensor quantization and weights use per-channel quantization.Source: §5.1 · p. 6

5.2.1 LLM Zero-Shot Reasoning

  • LLaMA-7B/13B show only 8.2/5.8% accuracy drop with FPQ at 4/4/4 bits, and FPQ improves 5.3% over LLM-QAT in that setting.FPQ also improves 1.5% over GPTQ at 4/4/16 bits on LLaMA-7B; the reported LLaMA comparison is in Table 1.Source: §5.2.1 · p. 7, Table 1 · p. 7

5.2.2 BERT Model

  • At 4/4/4 bits, FPQ improves average BERT accuracy by 44.3% over BrecQ and 7.9% over QDrop; at 4-bit weights and 8-bit activations, it nearly matches full precision with 128 calibration points.The corresponding BERT results are in Table 2.Source: §5.2.2 · p. 8, Table 2 · p. 8

5.2.3 Generalizability on Vision Transformer

  • 40.9% and 31.5% absolute accuracy improvements over PTQ4ViT and APQ-ViT, respectively, are reported for 4-bit DeiT-S on ImageNet.FPQ also outperforms both methods at 6 bits; the ImageNet comparison is in Table 3.Source: §5.2.3 · p. 8, Table 3 · p. 9

5.3 Ablation Study

  • Across calibration sizes of 32, 64, 128, and 256 data points, FPQ remains robust and maintains competitive accuracy even with limited calibration data, including as few as 32 points.MNLI and QQP are more robust to calibration-size changes than CoLA, where variance is more significant.Source: §5.3 · pp. 8–9
  • FPQ is robust across search ranges unless the range is overly aggressive, although no single tested range wins consistently across MNLI, QQP, and CoLA.The comparison tests (0.01, 1.2), (0.1, 1.2), and (0.5, 1.5).Source: §5.3 · p. 9, Table 5 · p. 9

5.4 Hardware Cost

  • FP operators have hardware cost comparable to INT, and mixed-format FP4 has area comparable to standard FP4.The multiplier is the primary INT cost and the adder the primary FP cost; the area comparison uses synthesized operators.Source: §5.4 · p. 9, Table 6 · p. 9

6 Conclusion

The paper presents 4-bit floating-point PTQ for transformer weights, activations, and embeddings, and extends the approach to vision transformers. It combines search-based quantization with pre-shifted exponent bias to improve low-bit quantization.

6 Conclusion

  • The paper reports the first successful 4-bit floating-point PTQ demonstration for weights, activations, and embeddings in language transformers, including LLMs and BERT.The method is also extended to vision transformers.Source: §6 · p. 9
  • Search-based quantization establishes a strong FP-PTQ baseline and achieves state-of-the-art results at 6 and 8 bits.The approach is described as practical and robustly generalizing to vision transformers.Source: §6 · p. 9
  • Pre-shifted exponent bias addresses high inter-channel variance and is reported as effective for accurate 4-bit quantization.The method targets the activation-variance challenge in transformers.Source: §6 · p. 9

Limitations

The method’s generalizability beyond the evaluated settings remains unverified, including for very long or streaming sequences and domains beyond language and vision.

Limitations

  • Experiments use publicly available datasets with finite sentence lengths, leaving extremely long sequences and streaming data unverified.The authors say these settings may require further investigation.Source: Limitations · p. 10
  • Generalization to domains beyond language and vision, including audio, remains untested.The authors identify audio as an example for further study.Source: Limitations · p. 10
  • Applicability to generative tasks and other applications remains an open question.The paper identifies these as potential areas for future investigation.Source: Limitations · p. 10

A Hessian-Based Loss Metric

The reconstruction objective minimizes quantization-induced perturbation to the pretrained network. A Hessian-based metric approximates this loss using layer-output changes and substitutes a diagonal Fisher Information Matrix for the costly output Hessian.

  • PTQ minimizes the perturbation δX = XFP − XR introduced when quantizing the pretrained real-valued network.Source: A Hessian-Based Loss Metric · p. 12
  • The Taylor expansion assumes the pretrained model is well converged, so its gradient is near zero in every element.Source: A Hessian-Based Loss Metric · p. 12
  • The input-space Hessian is computed using the layer-output Jacobian and the Hessian with respect to the output.Source: A Hessian-Based Loss Metric · p. 12
  • When δX is relatively small, the quantized-versus-original output difference is approximated by JO(X)δX using a first-order Taylor expansion.Source: A Hessian-Based Loss Metric · p. 12
  • Because calculating the output Hessian is burdensome, the method substitutes its diagonal Fisher Information Matrix and assumes output entries are independent.The resulting Hessian-based metric is used to search for optimal FP weight and activation quantization functions through layer-wise reconstruction in BERT and Vision Transformer models.Source: A Hessian-Based Loss Metric · p. 12

B Quantization Error of Different Floating-Point Formats

The appendix compares floating-point formats and activation distributions across transformer modules and models. It reports module-dependent format optima and inter-channel variance across language and vision transformers.

B Quantization Error of Different Floating-Point Formats

C Inter-Channel Variance Visualization

D Efficient Matrix Multiplication

For efficient matrix multiplication, the method uses per-tensor activation quantization because per-token precision gains are minimal relative to added overhead. Its search-based format selection also outperforms a gradient-based alternative on DeiT-S.

D Efficient Matrix Multiplication

E Learning Format and Maximum Value

  • FPQ reaches 79.88% ImageNet accuracy on 8-bit DeiT-S, versus 74.38% for the learnable gradient-based method.Source: E Learning Format and Maximum Value · p. 12
  • The gradient-based format search is unstable because gradients for exponent bits vary exponentially whenever the exponent-bit count changes.The authors use this observation to argue that search-based selection of optimal formats is crucial in PTQ.Source: E Learning Format and Maximum Value · p. 12

F Reconstruction Choices

The section contrasts sub-module reconstruction with the choice of layer reconstruction for the FPQ baseline. Its figures identify quantization-error, activation-magnitude, and efficient-matrix-multiplication visualizations.

Loading 2310.16836v1…