PaperlayerView exact revision on arXiv

Source-linked AI summary

Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models

Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, Xianglong Liu

arXiv:2209.13325v3cs.LG

TL;DR

Large Transformer models are costly to deploy, and structured activation outliers limit low-bit quantization while existing work can add computation overhead. This paper analyzes outlier causes and clipping importance, then introduces Gamma Migration and Token-Wise Clipping. The framework suppresses outliers efficiently and pushes 6-bit post-training BERT quantization to the full-precision level.

  • Problem

    Structured Transformer outliers severely damage low-bit quantization, while existing methods increase computation overhead and leave the outliers present.

  • Method

    The framework combines equivalent Gamma Migration with token-wise coarse-to-fine clipping to suppress outliers and obtain quantization-friendly activations.

  • Results

    The framework surpasses existing works and pushes 6-bit post-training BERT quantization to the full-precision level.

  • Takeaways & Limitations

    Gamma Migration suppresses LayerNorm-induced outliers without extra inference time, while Token-Wise Clipping efficiently focuses clipping on influential ranges.

  • Takeaways & Limitations

    The paper leaves open whether its conclusions benefit computer vision and calls for further study of outliers during pre-training.

Abstract

from arXiv · show

Transformer architecture has become the fundamental element of the widespread natural language processing~(NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To fundamentally address this problem, this paper delves into the inherent inducement and importance of the outliers. We discover that $\boldsymbol γ$ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts. Motivated by these findings, we propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping. The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more quantization-friendly model without any extra burden. The Token-Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining a clipping range with minimal final quantization loss in an efficient way. This framework effectively suppresses the outliers and can be used in a plug-and-play mode. Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit post-training BERT quantization to the full-precision (FP) level. Our code is available at https://github.com/wimh966/outlier_suppression.

1 Introduction

Transformer quantization addresses the deployment costs of large NLP models, but structured outliers create severe accuracy losses and existing work adds computation overhead. The paper analyzes their causes and clipping sensitivity, then proposes Gamma Migration and Token-Wise Clipping to suppress them efficiently.

  • Quantization compresses Transformer models for deployment on resource-limited devices, where memory and computation overheads are a concern.
  • Structured outliers can approach 100 and cause a 12% drop even with 8-bit quantization, while finer-grained quantization increases computation cost.
  • LayerNorm’s scaling parameter γ amplifies output outliers, while clipping impact varies: some broad outliers are safely clipped, but important ones can sharply reduce accuracy.
  • The framework combines Gamma Migration, which transfers γ into subsequent modules equivalently, with Token-Wise Clipping, which finds clipping ranges through a coarse-to-fine token-wise procedure.
  • Experiments cover BERT, RoBERTa, and BART across classification, question answering, and summarization, establishing the framework as efficient, plug-and-play, and state of the art.

2 Preliminaries

The paper defines matrix, vector, token, and quantization notation used to describe low-precision Transformer computation. Quantization maps continuous values to discrete integers and dequantization reconstructs values after multiplication.

  • Matrices are denoted X, vectors x, · denotes scalar multiplication, and ⊙ denotes element-wise multiplication.
  • For tokenized NLP inputs, X_t,j is the element at token t and embedding j, while x_t denotes token t’s embedding.
  • Quantization uses two operations: Quant maps continuous x to discrete x̄, and DeQuant reconstructs x̂ after multiplication.
  • The quantizer is parameterized by step size s, zero point z, and bit setting b.

3 Outlier analysis

Transformer quantization is hindered by structured outliers, especially in LayerNorm and GELU outputs. The analysis finds that LayerNorm’s γ amplifies outliers and that token-specific clipping can remove some aggressive outliers with little or no full-precision accuracy loss.

  • Outlier characteristics: LayerNorm and GELU outputs contain sharp outliers that cause severe degradation for standard low-bit quantization.The reported problem concerns standard 6/8-bit post-training quantization and 4-bit quantization-aware training.
  • Outlier characteristics: Outliers cluster in certain embedding dimensions, with unique tokens producing especially large values on those dimensions.The paper notes that token frequency during pre-training might contribute to sharper token-specific outliers.
  • Inducement of outliers: LayerNorm’s scaling parameter γ aligns with output outlier dimensions and amplifies outliers across tokens as a shared parameter.The additive parameter β has a smaller range than the LayerNorm output and is therefore not the key source identified here.
  • Inducement of outliers: Non-scaling LayerNorm produces weaker outliers and a milder activation distribution than normal LayerNorm.The transformation removes the amplification effect associated with γ.
  • Inducement of outliers: 6-bit cosine similarity improves for Non-scaling LayerNorm relative to the original output across 12 BERT-SST-2 LayerNorms.Table 1 evaluates cosine similarity between quantized and real signals; higher similarity indicates less quantization error.
  • Impact of outlier clipping: Clipping values from 10–100 to 10 preserves full-precision accuracy at 91.02, whereas cutting too many outliers lowers accuracy to 85.93.The result shows that outlier importance varies substantially, so the clipping range must account for final performance.
  • Impact of outlier clipping: Outliers spanning 10–100 correspond to only 3% of tokens, and clipping these sharper token-specific values does not affect full-precision performance.The analysis uses token-range variation to identify an unimportant long-tail region efficiently.

4 Method

The framework suppresses Transformer activation outliers through Gamma Migration and Token-Wise Clipping. Gamma Migration preserves full-precision equivalence while moving scaling, and Token-Wise Clipping efficiently searches token-dependent ranges for low quantization loss.

  • Gamma Migration: Gamma Migration moves LayerNorm’s γ into subsequent modules, producing a more quantization-friendly activation without extra computation overhead.The transformation separates the Non-scaling LayerNorm output from γ while preserving the full-precision model’s equivalence.
  • Gamma Migration: With residual connections, γ is migrated into the shortcut branch and absorbed by the next layer’s weight.Because γ is shared across tokens, the transfer into the subsequent linear-layer weight is valid.
  • Gamma Migration: Gamma Migration greatly reduces the activation quantization burden while causing little disparity between original and changed 6-bit weights.The activation range is reduced by |max(γ)|, whereas cosine similarity changes little for weights across 12 BERT-SST-2 intermediate layers.
  • Token-Wise Clipping: Token-Wise Clipping selects a clipping range by minimizing final quantized-output loss through a token-wise coarse-to-fine search.The method searches step size s using the distance between quantized output f̂(s) and real output f, then refines the result by gradient descent.
  • Token-Wise Clipping: The coarse stage represents each token by its maximum or minimum embedding signal, then uses quantiles of these bounds to identify clipping values.Upper and lower representative collections support efficient skipping of broad outlier regions with limited accuracy influence.

5 Experiments

Experiments evaluate the framework across PTQ and QAT settings, multiple Transformer models, and classification, question-answering, and summarization tasks. The results show strong low-bit performance, including near-full-precision behavior in several settings.

  • Experimental Setup: Experiments cover GLUE, SQuAD, XSum, and CNN/DailyMail across BERT, RoBERTa, and BART models.The study evaluates both PTQ and QAT, with 4-4-4 denoting 4-bit weights, embeddings, and activations.
  • Ablation Study: Gamma Migration and Token-Wise Clipping improve RoBERTa 6-bit PTQ, with gains of 17.53% on QNLI and 13.22% on MRPC over the baseline.Gamma Migration helps both MinMax and Token-Wise Clipping in the ablation study.
  • PTQ Results: 6-bit PTQ remains within 2.64% of FP overall on BERT, while RoBERTa and BART achieve average accuracy improvements of up to 8.64% and 11.79%.On CoLA and STS-B, the reported gains are 4.49% and 1.33%, respectively.
  • QAT Results: QAT at 4-4-4 bits achieves near-floating-point performance, with an average reduction of 2.70% on 4-bit quantization.On QQP and MNLI, the framework limits accuracy drops to 0.7% and 1.7% without distillation or data augmentation, compared with 4.19% and 3.16% for LSQ+.
  • Question Answering: On SQuAD v1.1, the framework outperforms other methods by over 4.73% on BERT and 15.55% on RoBERTa at 6-bit quantization.On SQuAD v2.0, the reported boosts are 12.31% on RoBERTa and 4.96% on BART.
  • Summarization: For BART summarization, the approaches reach near-floating-point performance at 8 bits and provide about 4% enhancement at 6 bits.The evaluation uses ROUGE 1/2/L on CNN/DailyMail and XSum.

6 Conclusions and Discussions of Limitations

The conclusion presents the framework as an outlier-suppression approach grounded in analyses of outlier inducement and clipping impact. It identifies transfer to other fields and understanding outliers during pre-training as open problems.

  • Conclusions: The framework is based on analyzing how outliers arise and how clipping them affects Transformer language models.The paper frames Gamma Migration and Token-Wise Clipping as responses to these analyses.
  • Limitations and Future Work: Whether the conclusions benefit other fields such as computer vision remains an open question.The paper identifies systematic exploration of this issue as valuable future work.
  • Limitations and Future Work: The paper leaves the pre-training process of outliers for further investigation, although outliers occur in both fine-tuned and pre-trained models.The authors point to pre-training analysis as meaningful for better understanding.

Checklist

The checklist reports affirmative documentation for contributions, limitations, theoretical assumptions and proofs, reproducibility materials, training details, compute resources, and asset licensing. Error bars were not reported because the evaluation was broad and computationally expensive to repeat.

  • Transparency and Scope: The authors report that the paper describes its contributions, scope, and limitations, while leaving some topics as future work.The checklist also marks discussion of potential negative societal impacts as not applicable.
  • Theoretical Results: Theoretical assumptions and complete proofs are reported, with detailed proofs available in supplementary materials.The checklist marks these items affirmatively.
  • Reproducibility: Code, data, experiment instructions, and training details are reported as available through supplementary materials or the paper’s assets.The authors defer detailed training settings to the supplementary materials.
  • Evaluation Reporting: Error bars were not reported because evaluating robust generalization across many models and datasets was considered computationally expensive.The checklist otherwise reports the total compute amount and resource types.
  • Ethics and Assets: Existing assets are reported as cited and licensed, while consent, personally identifiable information, offensive content, and human-subject procedures are marked not applicable.The checklist marks the corresponding ethics and asset-use items affirmatively or not applicable.
  • Supplementary Materials: Supplementary materials provide equivalent-transformation proofs, quantization-challenge illustrations, outlier analyses, experiments, related work, and implementation details.The appendix also gives migration procedures for FFN-LN and MHA-LN.

A.1 Proof of equivalent transformation

The equivalent transformation moves LayerNorm’s scaling parameter into subsequent modules while preserving the computation. The section also specifies quantization-node placement and identifies LayerNorm and GELU outputs as key quantization obstacles.

  • Equivalent Transformation: The proof examines each element of a matrix-multiplication output h and distributes γ across the corresponding weight terms.The elementwise contribution is expressed as (γ_j · W_i,j) · x_j.
  • Equivalent Transformation: Because γ is shared across samples and tokens, the next-layer weight can absorb it while preserving the transformation.This establishes the equivalence used by Gamma Migration.
  • Migration Procedures: Gamma Migration is illustrated for FFN-LN and MHA-LN in the Cross-Attention module, with quantization-flow comparisons before and after migration.The original LayerNorm is represented as a Non-scaling LayerNorm multiplied by γ.
  • Inference Flow: Adjacent DeQuant and Quant operations are merged into one ReQuant operation during inference for faster computation.The figure describes this as the practical inference implementation.
  • Quantization Nodes: Different quantization-node choices complicate fair comparisons and hardware development, so the study adopts the FasterTransformer scheme.The scheme quantizes embeddings, matrix-multiplication weights and activations, and uses one quantizer for the same activation.
  • Quantization Obstacles: Outputs of LayerNorm and GELU are identified as problematic because their quantized outputs show low cosine similarity and disabling their quantization recovers accuracy.Activations below 0.99 cosine similarity are treated as problematic positions.

C Analysis of outliers

The analysis attributes Transformer outliers to LayerNorm scaling and finds that their quantization importance varies substantially across embeddings and tokens. Removing the scaling effect yields milder activations, while token-aware clipping can remove some aggressive values with limited accuracy impact.

  • Embedding and token patterns: LayerNorm outputs contain large outliers concentrated at specific embedding dimensions and intensified for special or frequent tokens.Dimensions 308 and 381 show outliers across almost all tokens, while tokens such as [SEP], [CLS], commas, periods, “the”, “and”, and “of” can exhibit sharper values.
  • Outlier inducement: The scaling parameter γ amplifies outliers in LayerNorm outputs, whereas removing it produces a milder, more quantization-friendly distribution.The comparison uses γ, f X, and X′ across LayerNorm modules, with X′ showing weaker outliers.
  • Clipping impact: Outlier importance varies sharply: some very large values can be clipped without substantial accuracy loss, while clipping others causes rapid degradation.For MHA-LN outputs, clipping from -60 to -45 is described as reliable, whereas clipping from -40 to -35 induces about 5% performance loss.
  • Token-wise clipping: Because aggressive outliers are concentrated in a small subset of tokens, token-perspective range search can bypass relatively unimportant activation regions efficiently.For values in (-60, -45), clipped tokens remain 3% for most layers.
  • Quantitative evidence: Table 12 measures cosine-similarity improvement after extracting γ on 256 development-set samples across LayerNorm modules.BART contributes six LayerNorm layers, leaving the table’s right half empty.

D.3 Comparisons among Token-Wise Clipping and existing methods

The comparisons show that Token-Wise Clipping’s coarse stage is both more effective than several clipping baselines and efficient at searching ranges aligned with final performance. Competing approaches can select locally favorable or costly ranges when unimportant outliers occupy a broad value region.

  • Efficiency and effectiveness: Token-Wise Clipping evaluates 30 ratios on GLUE tasks in about 2 minutes while searching clipping ranges toward final performance.The method uses an efficient coarse-to-fine search strategy.
  • Comparison with OMSE: OMSE may select a locally optimal but inferior clipping range: it chooses 40 where 10 performs better for the distribution in Fig. 2.OMSE also runs slowly even with fast golden section search.
  • Comparison with learning-based methods: Direct step size learning can require careful initialization, learning-rate selection, and substantial tuning time before reaching the useful clipping region.Even after sufficient QAT step-size tuning, the quantized model may still benefit from further clipping.
  • Comparison with percentile search: Percentile search is time-costly when unimportant outliers cover a large value range because it searches from the activation-value perspective.Both Percentile and direct step size learning remain vulnerable to broad regions of unimportant outliers.

D.4 Supplementary results of QAT

Supplementary QAT experiments apply the framework to RoBERTa and BART and report improvements over strong baselines on GLUE and BART tasks. The supplied results also describe comparisons across QAT strategies and extend the framework to integer-only quantization.

  • RoBERTa: RoBERTa QAT results surpass LSQ+ by 2.54% on QNLI and 7.53% on STS-B.These results are reported on the GLUE benchmark.
  • QAT comparison: Table 16 compares QAT strategies with low-bit activation on the GLUE benchmark for RoBERTa and BART.The supplied passage identifies the table’s model and benchmark scope but does not provide its individual values.
  • Clipping evaluation: Table 48 evaluates dev-set accuracy after clipping output activations and records the clipped-token ratio for each layer.For BART, the evaluation also includes the Decoder GELU module, with bold values marking accuracy inflection points.
  • BART and extensions: BART QAT results achieve an absolute improvement of 1.73–32.11 points against the best baseline.The framework is also described as extendable to integer-only quantization.

F Supplementary implementation details

The implementation uses specified quantization nodes, calibration data, and training settings across PTQ and QAT experiments. Token-Wise Clipping is implemented as a coarse grid search followed by fine-grained step-size optimization, with model-size and timing details reported separately.

  • Quantizer configuration: Activations use asymmetric per-layer quantization, while weights use symmetric per-channel quantization.Quantization nodes are inserted according to the paper’s stated implementation scheme.
  • PTQ settings: PTQ calibration samples 256 examples, while the fine-grained clipping stage runs for 3 epochs at learning rate 1e-5 across datasets.Calibration batch size is 32 on GLUE and SQuAD, and 4 on CNN/DailyMail and XSum.
  • QAT settings: For QAT on GLUE, the coarse clipping stage initializes quantization parameters and the fine-grained stage is omitted because LSQ+ learns step sizes.Learning rates are searched over 1e-5 through 5e-5, with batch-size adjustments on small datasets.
  • Token-Wise Clipping algorithm: Token-Wise Clipping first computes per-token maxima and minima, derives quantile-based bounds, clips activations, and selects an initialization step size.The fine-grained stage then optimizes the step size before returning it.
Loading 2209.13325v3…