PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort

arXiv:2306.12929v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Transformer activation outliers hinder efficient quantization, motivating methods that avoid higher bitwidths and other workarounds. The paper identifies no-op attention behavior as their source and introduces clipped softmax and gated attention. These methods reduce outliers while preserving or sometimes improving floating-point performance and improving quantized performance, though important scalability and consistency limits remain.

  • Problem

    Transformer activation outliers make quantization difficult, often requiring higher bitwidths, different numeric formats, retraining, or other workarounds.

  • Method

    The paper modifies attention with clipped softmax and gated attention to address the outlier root cause during model training.

  • Results

    The methods substantially dampen outliers, improve quantized performance toward FP16/32 performance, and sometimes improve floating-point task performance across language and vision models.

  • Takeaways & Limitations

    The architectural changes produce transformers that are easier to quantize and can support efficient integer inference without post-processing.

  • Takeaways & Limitations

    The methods were studied only up to 1.3B models, and the authors do not claim that the small floating-point improvement generalizes across architectures and larger models.

Abstract

from arXiv · show

Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways to reduce the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a "no-op" or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism - clipped softmax and gated attention. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers.

1 Introduction

Transformer activation outliers make quantization difficult because existing remedies often require retraining, specialized hardware, or higher-bit activations. The paper analyzes their origin and proposes architectural changes intended to reduce outliers before quantization.

  • Activation outliers make transformer quantization difficult because large ranges create a trade-off between rounding precision and clipping error.Existing fixes may require retraining, specialized input-channel hardware, or retaining some activations at higher bitwidth.
  • The paper changes the transformer architecture itself to make models easier to quantize without post-processing.
  • The analysis finds that attention heads attempting not to update residual representations produce strong outliers through the softmax function.The authors report this behavior across language and vision transformers and different transformer architectures.
  • Two architectural tweaks, clipped softmax and gated attention, are proposed to remove the outlier problem at its source.

2 Background and related work

Quantization uses low-bit representations to reduce neural-network cost, but transformer outliers make post-training quantization error-prone. Prior approaches mainly adapt quantization to accommodate outliers, whereas this paper targets the underlying transformer behavior.

  • Quantization reduces neural-network computation, memory consumption, and potentially energy use through low-bit weight and activation representations.The paper focuses on post-training quantization, which converts a pretrained FP32 network without the original training pipeline.
  • Uniform affine quantization maps inputs using a scale factor, zero point, and bitwidth, with rounding to the nearest integer.
  • Transformer outliers occur in a small set of embedding dimensions across layers and sequences, and clipping them can substantially degrade task performance.
  • Outliers make per-tensor quantization difficult because a large range increases rounding error while a small range increases clipping error.
  • Existing remedies use finer granularity, higher bitwidths, different numeric formats, or extra fine-tuning, often reducing general applicability or adding inference overhead.
  • The paper instead addresses the root cause during pre-training, aiming for models whose outliers are smaller and whose post-training quantization causes less performance degradation.

3 Outlier analysis

The outlier analysis links transformer outliers to attention heads that approximate no-update or selective-update behavior by concentrating attention on low-information tokens or patches. Softmax, normalization, and residual connections jointly drive the associated magnitudes upward.

  • In BERT, more than 97% of detected outliers correlate with delimiter-token positions, and only a few hidden dimensions contain outliers.
  • The BERT hidden dimension with the highest outlier count corresponds to attention head #3, whose attention patterns were analyzed through probabilities, values, and their product.
  • Attention heads often assign most probability to [SEP], dots, or commas, whose small value outputs yield a small product and a soft no-update of the hidden representation.
  • When some attention probability remains on other tokens, the same pattern produces a soft selective update rather than a complete no-update.
  • These attention patterns act as a learned workaround for softmax and residual connections when a head should not update some representations.
  • In ViT, outliers correlate with uninformative background patches, which receive most attention and have smaller values than non-outlier patches.
  • Exact zero softmax outputs require an unbounded input dynamic range, while LayerNorm and persistent softmax gradients drive other activations toward larger magnitudes.

4 Method

The method modifies softmax attention so attention outputs can become very small or exactly zero without generating outliers. It introduces clipped softmax and gated attention as alternative mechanisms for nullifying updates.

  • 4.1 Clipped softmax: Clipped softmax stretches and clips softmax outputs, enabling exact zeros and ones with a finite softmax-input range.The stretch factors satisfy ζ ≥ 1 and γ ≤ 0; values are clipped back to (0, 1).
  • 4.1 Clipped softmax: Clipping prevents further outlier growth because clipped values receive no gradient.This preserves the finite-range mechanism while stopping clipped activations from driving larger outliers.
  • 4.2 Gated attention: Gated attention uses an explicit learned gate to keep or nullify token updates instead of relying on attention probabilities and values.The gate is applied through element-wise multiplication across the token axis, while the remaining attention formulation is unchanged.
  • 4.2 Gated attention: The gating function is defined per attention head, shared across token positions within a head, and not shared across heads.The gating modules are learned jointly with the transformer and operate on head-specific representations.
  • 4.2 Gated attention: A single-layer gating module adds approximately dmodel parameters per attention layer, making the mechanism computationally inexpensive.Its stated memory overhead is nheads · (dhead + 1), approximately equivalent to one extra token.

5 Experiments

Experiments evaluate clipped softmax and gated attention across language and vision transformers, measuring floating-point performance, outlier magnitude, kurtosis, and quantized performance. Both methods generally reduce outliers and improve quantized performance, while results depend on hyperparameters and architecture.

  • Experimental setup: Experiments evaluate the proposed attention modifications on BERT, OPT, and ViT, followed by 8-bit post-training quantization.The evaluation reports task performance and quantization behavior across language and vision transformer models.
  • Experimental setup: Outlier magnitude is assessed using maximum ∥x∥∞ and kurtosis of attention-layer outputs, metrics associated with model quantizability.Networks are trained with multiple random seeds, and these metrics are averaged as described in the experimental protocol.
  • Clipped softmax: γ < 0 provides the main clipped-softmax benefit: γ = −0.03 yields smaller infinity norm, lower kurtosis, and lower quantized perplexity than the baseline.Clipping at one alone behaves similarly to vanilla softmax, whereas lower-range clipping enables exact zeros.
  • Clipped softmax: α ∈ [2, 4] significantly dampens outliers while maintaining good FP16 perplexity across sequence lengths T ∈ {32, 64, 128, 192, 256}.This result uses the parameterization γ = −α/T on BERT-6L.
  • Gated attention: For gated attention, reasonable bias-initialization ranges are [0.25, 0.9] for BERT and [0.1, 0.5] for ViT, indicating relative hyperparameter robustness.Very high initialization preserves vanilla-like floating-point behavior but leaves strong outliers, while very low initialization degrades performance.
  • Main results: Across almost all cases, both methods reduce outlier magnitude and kurtosis while bringing quantized performance close to FP16/32 performance.At least one method also improves floating-point task performance for each model, but the improvement is not consistent across hyperparameters or architectures.
  • Scalability: Gated attention remains effective for OPT models with 350m and 1.3B parameters, reducing outliers and improving quantized performance.These larger-model experiments used 105 training steps with batch size 256 because of compute constraints.
  • Qualitative results: Both methods represent partial or soft no-op attention behavior without requiring strong outliers elsewhere in the network.Clipped softmax diffuses smaller weights and saturates higher weights, while gated attention further modulates the update using gating probabilities.

6 Discussion

The discussion connects learned no-op behavior to residual noise that may regularize networks, while noting limits in scale, evidence, and hyperparameter simplicity.

  • No-op behavior: Attention heads may learn no-op behavior beyond transformers, and residual noise may provide a form of network regularization.The paper suggests investigating this connection could help explain generalization in overparameterized networks.
  • Limitations: The methods were studied only on models up to 1.3B parameters, leaving very large, longer-trained transformers unexplored.The authors expect the same effect at larger scale based on their understanding, but do not report experiments there.
  • Limitations: The reported FP16/FP32 improvement is small and not considered exhaustive evidence that performance gains generalize.The authors explicitly avoid claiming that the observed improvement will hold in general.
  • Limitations: Each proposed method has a hyperparameter, although both methods appear relatively robust to its value.The discussion identifies having a hyperparameter as a remaining imperfection rather than a demonstrated failure mode.
  • Potential impact: The authors expect efficiency gains to reduce inference power consumption and support moving inference from cloud systems to edge devices.They connect edge inference with potentially reduced privacy concerns.

7 Conclusions

The paper explains transformer activation outliers through attention heads that avoid updating residuals and introduces clipped softmax and gated attention to address the mechanism structurally. These changes preserve or improve floating-point performance while improving post-training quantization.

  • Mechanism: Attention heads learn not to update residuals, and the interaction of softmax, residual connections, and LayerNorm produces significant transformer outliers.The analysis identifies this mechanism as the core explanation for the activation outlier problem.
  • Methods: The paper proposes clipped softmax and gated attention as two structural methods addressing the outlier mechanism at its source.Both methods modify the transformer architecture rather than relying on post-training correction.
  • Results: The structural changes yield similar or better floating-point performance after training and significantly improve post-training quantization results.The conclusion frames the methods as enabling high-performance transformers that are easier to quantize.

A Additional graphs from outlier analysis

The additional analyses localize outliers and compare method behavior across BERT and ViT, while extended experiments examine gating choices, clipped-softmax settings, and quantized performance.

  • A.1 BERT: BERT outliers concentrate in seven hidden dimensions, mainly dimensions #180 and #720, corresponding to attention heads #3 and #12.Additional attention-pattern figures compare these outlier-associated heads with non-associated heads and trained models using the proposed methods.
  • A.2 ViT: In ViT, the strongest outliers peak in layers #10 and #11, with over 99% occurring in 10 hidden dimensions, primarily #48 and #43.These dimensions correspond to attention head #1, and outliers are concentrated near image boundaries in the ImageNet validation set.
  • Gated attention: Gated-attention initialization balances closeness to the original network against the risk of slow gate closure or performance degradation.High initial openness preserves vanilla-like behavior, whereas fully closed gates can deviate too far from the original training dynamics.
  • BERT results: For BERT, both methods generally dampen outliers, reduce kurtosis, improve quantized performance, and maintain or sometimes improve FP16 perplexity.The main results report these trends across most tested settings.
  • OPT results: For OPT-125m, gated attention further reduces outlier magnitude and kurtosis while producing quantized performance close to the original FP16 performance.The comparison includes settings with and without weight decay on LayerNorm weights.
  • ViT results: For ViT-S/16, adding LayerNorm after patch embeddings with either proposed method yields quantized performance within 1% of original FP32 accuracy.The same configuration also greatly reduces outlier magnitude and kurtosis.
  • Clipped-softmax ablation: For ViT clipped softmax, most improvement comes from γ < 0, while ζ > 1 behaves similarly to vanilla softmax.Combining γ < 0 and ζ > 1 produces results similar to clipping at zero alone.

B.6 Fine-tuning experiment

The fine-tuning experiment tests whether gated attention can reduce outliers in a larger pretrained model, addressing the cost of training the proposed framework from scratch. On OPT-1.3B, gated-attention fine-tuning improves perplexity and reduces outlier statistics relative to vanilla softmax fine-tuning.

  • Motivation: Training the proposed framework from scratch can be expensive for very large models, motivating gated-attention fine-tuning experiments.The paper treats this training requirement as a drawback of the framework.
  • Setup: OPT-1.3B was fine-tuned on Bookcorpus and Wikipedia for 4000 steps using gated attention and an activation regularization term.The regularization term was added to encourage lower activation magnitudes because pretrained outliers were already present.
  • Results: Gated-attention fine-tuning produces better perplexity and lower maximum infinity norm and average kurtosis than vanilla-softmax fine-tuning.The comparison is reported for the causal language-modeling evaluation on the English Wikipedia validation set.
  • Quantization context: The broader experiments use post-training quantization and examine compatibility with different bitwidths and other compression methods.The paper states that the proposed methods are not limited to 8-bit quantization.
  • Low-bit results: Across tested bitwidths, both methods improve perplexity over vanilla softmax pre-training, while lower bitwidths generally degrade performance.The reported W8A8-to-W6A8 comparison reduces average kurtosis from 3406±547 to 631±94 and maximum infinity norm from 577±80 to 158±40.

C.2 OPT pre-training

OPT-125m pre-training uses Wikipedia and BookCorpus with a 512-token sequence length, 125,000 steps, and a single-A100 experimental setup. Quantization uses symmetric uniform weights and calibrated activation ranges, with percentile ranges selected for OPT when beneficial.

  • Training setup: OPT-125m is trained on concatenated Wikipedia and BookCorpus sequences of length 512 for 125,000 steps using an effective batch size of 192.The setup uses batch size 48 with four gradient-accumulation steps and 500,000 forward passes on one A100 80GB GPU.
  • Model configuration: All experiments place LayerNorm before the attention block, differing from the HuggingFace OPT-350m checkpoint's post-attention placement.
  • Quantization setup: Weights use symmetric uniform quantization, with min-max scaling except for OPT, where an MSE estimator performs better.
  • Quantization setup: Activations use static range calibration with a running min-max estimator, 0.9 momentum, and 16 randomly sampled training batches.
  • Quantization setup: For OPT, 99.99% and 99.999% activation percentiles are also tested, with 99.999% producing the lowest W8A8 perplexity in almost all cases.

D Compute cost

The proposed attention modifications add limited runtime overhead relative to vanilla attention. The reported project required about 320 A100 GPU-days for the main results and 1,400 GPU-days overall.

  • Runtime: Clipped softmax is only marginally more expensive than vanilla softmax attention.
  • Runtime measurement: Table 11 reports proposed-method and vanilla-pretraining runtimes in hours measured on Nvidia-A100 GPUs.
  • Runtime: Adding the linear G for gated attention increases compute overhead by 3% to 8%, depending on the model.
  • Runtime: Weight decay on LayerNorm γ for OPT and LayerNorm after ViT patch embeddings have negligible effects on runtime.
  • Compute cost: 320 A100 GPU-days were estimated for the paper's main results, versus 1,400 GPU-days for the full project including preliminary and ablation experiments.
  • Attention visualizations: Figures 10–15 visualize attention probabilities, values, and their products across vanilla, clipped-softmax, and gated-attention BERT models.
  • Attention visualizations: Figures 16 and 17 summarize ViT outlier analyses using input images, layer-output outliers, cumulative attention weights, attention matrices, and value magnitudes.
Loading 2306.12929v2…