Source-linked AI summary
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
TL;DR
Transformer models are difficult to deploy efficiently because of their large memory footprint and high latency, while their activation ranges challenge low-bit quantization. The paper analyzes these challenges, proposes several quantization strategies including per-embedding-group quantization, and reports strong GLUE results alongside substantial low-bit compression.
Problem
Transformer models have large memory footprints and high latency, and their activation tensors have dynamic ranges that are difficult to represent with low-bit fixed-point formats.
Method
The paper studies transformer quantization and proposes post-training, mixed-precision, quantization-aware-training, and per-embedding-group quantization solutions.
Results
4-bit weight and 2-bit token-embedding quantization produced less than a 0.8% GLUE-score drop, while the techniques achieved state-of-the-art results for post-training quantization and per-tensor QAT on GLUE.
Takeaways & Limitations
Transformer weights and embeddings can reach ultra-low bit-widths with significant memory and compute savings and minimal accuracy loss.
Takeaways & Limitations
The proposed methods involve trade-offs: higher bit-widths reduce efficiency and may lack hardware support, while finer-grained activation quantization adds compute and potential latency.
Abstract
from arXiv · showhide
Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges -- namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme -- per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. Our source code is available at~\url{https://github.com/qualcomm-ai-research/transformer-quantization}.
1 Introduction
Transformer models deliver strong results but are costly to deploy because of their size, latency, memory footprint, and energy use. This paper studies transformer quantization, identifies activation-range challenges, and proposes solutions evaluated on GLUE.
- Transformer models can exceed billions of parameters, making efficient deployment difficult on resource-constrained systems because of latency, memory, and energy demands.
- Quantization reduces memory consumption, inference time, and energy use by representing weights and activations with low-bit fixed-point arithmetic.
- Standard 8-bit post-training quantization causes significant performance degradation in transformer encoder models.
- Activation tensors in residual connections have mismatched dynamic ranges and structured outliers associated with attention patterns such as attending to [SEP].The paper reports that this issue is inherent to many architectures and pre-training objectives.
- The paper proposes post-training, mixed-precision, quantization-aware training, and per-embedding-group quantization as solutions with different trade-offs.The per-embedding-group scheme addresses activation quantization without significant compute overhead or increased complexity.
- On eight GLUE tasks, the techniques establish state-of-the-art results for BERT post-training quantization and per-tensor quantization-aware training.The method is presented as applicable beyond BERT to other pre-trained transformer models.
2 Background and related work
This section reviews efficient-transformer research and explains quantization methods, including post-training, quantization-aware training, mixed precision, and varying quantization granularity. Prior transformer quantization work largely relies on quantization-aware training or gives limited attention to post-training alternatives.
- Efficient Transformers: Efficient-transformer research includes architectural changes such as sparse attention and low-rank or kernel-based approximations to reduce computation.
- Quantization: Quantization uses low-bit weight or activation representations; moving from 32 to 8 bits reduces tensor-storage overhead fourfold and matrix-multiplication cost sixteenfold.Fixed-point operations can also reduce energy consumption relative to floating-point operations.
- Quantization: Uniform affine quantization maps tensors using a bitwidth, scale factor, and zero-point, with de-quantization approximately recovering the real-valued input.
- Quantization: Per-tensor quantization uses one parameter set for a tensor, while finer-grained quantization can improve accuracy at additional compute and memory cost.
- Quantization: Post-training quantization converts a pre-trained FP32 network directly to fixed point without the original training pipeline, while range estimation selects quantization parameters.Common estimators include current min-max, running min-max, and MSE-based calibration.
- Quantization: Quantization-aware training simulates quantization during training so models adapt to quantization noise, but requires longer training, labeled data, and hyperparameter search.
- Quantization: Mixed precision assigns different bit-widths to different layers or network parts.
- Transformer quantization: Prior BERT quantization methods used quantization-aware training, while often omitting post-training alternatives or treating them as weak baselines.
3 Problem investigation
The study evaluates standard 8-bit post-training quantization on BERT and traces its degradation primarily to activation quantization in residual connections. Structured outliers, especially in deeper layers, create mismatched dynamic ranges and influence attention toward [SEP].
- Evaluation setup: The evaluation applies standard 8-bit post-training quantization to BERT across eight GLUE downstream tasks.The study reports joint, activation-only, and weight-only quantization configurations.
- Quantization results: Joint 8-bit quantization causes significant performance degradation, while weight-only quantization incurs almost no error.The reported results indicate that most degradation comes from activation quantization, although task robustness varies.
- Quantization results: The smallest performance drop occurs when the residual sum after the FFN is left unquantized, especially implicating deeper encoder layers 10 and 11.This ablation identifies the post-FFN residual sum as the most problematic activation among those tested.
- Outlier analysis: In the 11th layer, FFN inputs and outputs have radically different dynamic ranges because of strong output outliers, making per-tensor quantization trade precision against clipping.A single range must represent both small- and large-magnitude values, producing either rounding or clipping error.
- Outlier analysis: Outliers correlate with [SEP] tokens and recur in a few embedding dimensions across data points, while structured residual outliers drive query-key outliers that make tokens attend to [SEP].The issue appears across all BERT-base layers and GLUE tasks, and is also present in BERT-large, RoBERTa, DistilRoBERTa, and MobileBERT.
4 Methodology
The paper presents three quantization approaches for BERT-like models—mixed-precision PTQ, PEG activation quantization, and QAT—each trading off accuracy, model efficiency, and implementation requirements. PEG groups embedding dimensions, uses range-based permutation, and can be simulated with per-tensor hardware operations.
- Proposed techniques: The proposed methods combine post-training mixed precision, per-embedding-group activation quantization, and quantization-aware training.They use uniform affine quantization with static activation ranges estimated during PTQ or learned during QAT.
- Mixed precision: Mixed-precision quantization assigns 16-bit activations to problematic tensors and explores 2–4-bit weights and token embeddings.The targeted higher activation precision represents the differing FFN input, output, and residual-sum ranges, while low-bit weights and embeddings reduce model size.
- Per-embedding-group quantization: Per-embedding-group quantization shares activation parameters among K evenly sized groups along the embedding dimension, reducing accumulator rescalings from d to K.Full per-embedding quantization requires repeated intermediate rescaling, whereas grouping reduces that overhead.
- Per-embedding-group quantization: PEG uses calibration ranges to deterministically permute embedding dimensions so outlier dimensions are placed in the same groups.Groups are formed by sorting dimensions according to their observed dynamic ranges.
- Implementation: PEG adds less than 0.04% to BERT-base model size and can be rewritten as split, summed, and concatenated linear operations for per-tensor hardware.The hardware simulation decomposes the linear layers into K smaller layers and applies corresponding permutations when needed.
5 Experiments
Experiments evaluate mixed-precision, per-embedding-group, and quantization-aware methods for BERT on GLUE, showing that dynamic-range problems can be mitigated with limited accuracy loss and practical trade-offs.
- The experiments evaluate proposed quantization techniques on BERT across GLUE downstream tasks.
- Mixed precision PTQ: Mixed-precision PTQ approaches near-FP32 performance while retaining 8-bit weights and most activations.For classification tasks, only a few problematic components need 16-bit precision; STS-B additionally requires a higher-precision output.
- Mixed precision PTQ: 22% of activations retained in 16-bit achieves performance close to FP32, with all other activations and weights in 8-bit.
- Per-embedding-group PTQ: Per-embedding-group PTQ recovers most performance degradation with K = 3 groups when range-based permutation concentrates outliers.Applying the scheme to problematic network parts limits rescaling and supports efficient execution on resource-constrained devices.
- Comparison of proposed methods: Across GLUE, the proposed methods achieve strong PTQ and per-tensor QAT results while trading accuracy, ease of use, model size, and hardware compatibility.Per-embedding-group quantization adds minor compute and latency and may lack native support on some fixed-point platforms.
- Low-bit weight and token embeddings: Token embeddings can be quantized to 2 bits with less than a 0.8% GLUE-score drop and an 8.85× model-size reduction.QAT recovers most performance, even with quantized activations.
6 Conclusions
The paper identifies structured activation outliers as a distinctive transformer-quantization challenge and proposes three methods to address it. On GLUE, these methods achieve strong quantization results, including ultra-low-bit representations with small accuracy loss.
- Transformer activations have high dynamic ranges and structured residual outliers that encourage attention to special [SEP] tokens.
- The paper proposes mixed precision, per-embedding-group quantization, and quantization-aware training, each with trade-offs in accuracy, ease of use, and model size.
- The techniques overcome dynamic-range issues and establish state-of-the-art PTQ and per-tensor QAT results on GLUE downstream tasks.
- 4-bit weights and 2-bit token embeddings produce less than a 0.8% GLUE-score drop while providing significant memory and compute savings.
Supplementary materials
Supplementary analysis connects structured activation outliers with recurring attention to special [SEP] tokens in deeper BERT layers, while noting a hypothesis about why these patterns arise.
- In a problematic 11th-layer attention head, most tokens attend to special [SEP] tokens, alongside a consistent vertical attention pattern.
- Prior analysis reported that attending to special tokens can act as a “no-op” and often occupies more than half of attention in deeper-layer heads.
- The authors hypothesize that the attention pattern supports predictive performance, while structured outliers facilitate it.
- They further hypothesize that architectural choices and long pretraining without explicit activation regularization contributed to the outliers.
B Experimental details
The experiments fine-tune BERT-base on GLUE with standardized sequence processing, task-specific hyperparameter selection, repeated seeds, and exclusion of unstable WNLI.
- BERT-base uncased with 109M parameters is fine-tuned using standard practices, with sequences truncated or padded to length 128.
- Fine-tuning runs for 3 epochs with Adam, while learning rates and batch sizes are selected per task from predefined search spaces.
- Each experiment is repeated with 5 random seeds, and the configuration with the best median development-set score is selected.
- Quantization is applied to the median checkpoint for each task.
- WNLI is excluded because its small dataset and construction issues produce unstable behavior.
B.2 Range setting for 8-bit post-training quantization
The 8-bit post-training quantization procedure selects range estimators and activation-calibration settings, then evaluates the best configurations on GLUE tasks.
- Weights are evaluated with min-max and MSE range estimators, while activations use current min-max, running min-max, and MSE.
- Activation calibration searches batch sizes and batch counts, uses momentum 0.9 for running min-max, and repeats each experiment five times with different random seeds.Current min-max uses a single batch; configurations are selected by the best median development-set score for each task.
- Table 9 lists the best range-estimator configurations for post-training quantization of BERT-base on GLUE tasks.The table reports batch size and number of batches for the selected configurations.
- The best configurations for joint weight and activation 8-bit post-training quantization are listed in Table 9.
C Detailed results for low-bit weight and embedding quantization
The section reports detailed GLUE development-set results for low-bit BERT-base weights and token embeddings, while examining recurring activation outliers across data points.
- Only a few designated embedding dimensions generate activation outliers across many data points in BERT-base FFN inputs and outputs.The visualizations cover the first ten MNLI, STS-B, and MRPC development-set sequences, with outliers defined as values exceeding six standard deviations from the activation-tensor mean.
- The recurring outlier pattern suggests that activation outliers are pre-determined by the pretrained BERT model’s weights and embeddings.
D.2 Activation tensors for different architectures
Activation dynamic-range behavior is examined across several transformer architectures and training settings using full-precision FFN activation distributions and outlier visualizations.
- The dynamic-range issue is reported for BERT-base, BERT-large, RoBERTa-base, DistilRoBERTa-base, and MobileBERT-base.The listed evaluations use pretrained checkpoints from the HuggingFace library.
- Figures 6–8 visualize FFN activation values exceeding six standard deviations from the tensor mean across embedding dimensions for BERT-base.The figures cover MNLI, STS-B, and MRPC development-set sequences and separately show FFN inputs and outputs.
- Figures 9–13 plot FFN input and output activation ranges across GLUE development-set sequences for the five evaluated architectures.The figures use different numbers of sequences for some architectures and note that input and output axes have different scales.