PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer

arXiv:2208.07339v2cs.LGcs.AI

TL;DR

Large transformer models require substantial inference memory, and conventional quantization loses precision as systematic outlier features emerge. The paper introduces LLM.int8(), combining vector-wise quantization with mixed-precision decomposition, and reports preserved inference performance for models up to 175B parameters.

  • Problem

    Large transformer models require substantial inference memory, while 8-bit quantization can degrade performance when systematic outlier features emerge at scale.

  • Method

    LLM.int8() combines vector-wise quantization with separate inner-product normalization and mixed-precision decomposition that isolates outlier dimensions for 16-bit multiplication.

  • Results

    LLM.int8() preserves inference performance for transformer models with up to 175B parameters, while maintaining favorable perplexity and zeroshot scaling trends.

  • Takeaways & Limitations

    The method enables immediate Int8 inference for very large models, including OPT-175B and BLOOM-175B, on a single server with consumer GPUs.

  • Takeaways & Limitations

    The study evaluates Int8 rather than FP8, covers models only up to 175B parameters, and does not use Int8 multiplication for the attention function.

Abstract

from arXiv · show

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.

1 Introduction

Large transformer models make inference memory-intensive, while conventional 8-bit quantization can lose performance as systematic outliers emerge. LLM.int8() combines vector-wise quantization with mixed-precision handling of outlier features to preserve 16-bit performance at large scales.

  • 95% of consumed parameters and 65-85% of computation in large transformers come from feed-forward and attention projection layers and their matrix multiplications.
  • LLM.int8() is presented as a multi-billion-scale Int8 procedure that avoids performance degradation and supports immediate inference after converting 175B-parameter checkpoints.
  • Vector-wise quantization assigns separate normalization constants to each matrix-multiplication inner product, retaining performance up to 2.7B parameters.
  • At around 6.7B parameters, extreme outliers affect all transformer layers and 75% of sequence dimensions while concentrating in only 6 feature dimensions.
  • Mixed-precision decomposition multiplies outlier feature dimensions in 16-bit and the other 99.9% of dimensions in 8-bit.
  • Figure 2 depicts decomposition, vector-wise scaling and quantization, Int32 dequantization, and accumulation of regular and outlier outputs in 16-bit.

2 Background

The background reviews symmetric and asymmetric 8-bit quantization and describes matrix multiplication with 16-bit inputs and outputs. Absolute-maximum quantization uses tensorwise scaling, while zeropoint quantization shifts distributions to use the available integer range.

  • The paper compares zeropoint quantization and absolute-maximum quantization when examining where transformer quantization fails as models scale.
  • Absolute-maximum quantization scales inputs into the Int8 range [−127, 127] using 127 divided by the tensor's absolute maximum.
  • Rounding to the nearest integer is used when mapping FP16 inputs to quantized values.
  • Zeropoint quantization scales and shifts the input distribution so asymmetric tensors use the full [−127, 127] range.
  • Zeropoint matrix products combine an Int8 product with zeropoint correction terms, accumulate in Int32, and dequantize using scaling constants.
  • The described matrix multiplication accepts FP16 hidden states and weights, applies either absmax or zeropoint quantization, and returns FP16 outputs.

3 Int8 Matrix Multiplication at Scale

LLM.int8() combines vector-wise quantization with mixed-precision decomposition to preserve transformer performance at large scales while reducing memory use. Its evaluation shows favorable scaling through 175B parameters, though smaller models can incur runtime overhead.

  • Vector-wise quantization: Vector-wise quantization assigns separate normalization constants to inner products, improving precision over a single scaling constant.Matrix multiplication is treated as independent row–column inner products, with outputs denormalized using the corresponding constants.
  • Mixed-precision decomposition: Mixed-precision decomposition multiplies outlier dimensions in 16-bit precision while multiplying more than 99.9% of values in 8-bit.For models up to 13B parameters, the outlier set contains no more than seven dimensions and adds about 0.1% memory.
  • LLM.int8(): LLM.int8() is the combination of absmax vector-wise quantization and mixed-precision decomposition.The method targets memory-efficient matrix multiplication while preserving high-precision treatment of outliers.
  • Mixed-precision decomposition: Extreme outlier features make vector-wise quantization insufficient beyond 6.7B parameters, motivating mixed-precision decomposition.These outliers are sparse and systematic, affecting specific hidden dimensions across sequence positions.
  • Language-modeling results: LLM.int8() is the only evaluated method with a favorable language-modeling scaling trend from 125M to 13B parameters.On C4 validation perplexity, competing quantization methods degrade as model size increases, especially at 13B.
  • Zeroshot results: LLM.int8() maintains full 16-bit zeroshot performance from 125M to 175B parameters, while absmax vector-wise quantization degenerates into random performance.The evaluation uses OPT models and the EleutherAI language model evaluation harness.
  • Runtime: Quantization overhead can slow inference below 6.7B parameters, whereas LLM.int8() is about two times faster for matrix multiplications equivalent to those in 175B models.The runtime evaluation compares against an FP16 baseline, while the paper’s primary focus remains memory reduction.

4 Emergent Large Magnitude Features in Transformers at Scale

The analysis identifies sparse, systematic outlier feature dimensions that emerge across transformer layers and strongly affect predictive performance and quantization. Their prevalence and magnitude change with model scale and perplexity, motivating specialized treatment.

  • Outlier features: Outlier features are particular hidden-state dimensions that recur across layers and sequence positions, especially in attention projections and FFN expansion layers.The analysis excludes the attention function and FFN contraction layer because outliers occur in the projection and expansion components.
  • Outlier features: Outliers are defined by magnitude at least 6.0, occurrence in at least 25% of layers, and presence in at least 6% of sequence dimensions.These thresholds were chosen to capture systematic features while limiting detection to a small number of features in the smallest model.
  • Emergence at scale: 65% to 100% of layers and 35% to 75% of sequence dimensions become affected suddenly between 6B and 6.7B parameters.When measured by perplexity, emergence instead follows a smooth exponential trend as perplexity decreases.
  • Quantization effects: Outlier magnitude rises rapidly after outliers reach all layers, making quantization ranges too broad and forcing many small values to zero.The paper identifies this asymmetric magnitude growth as the core reason conventional quantization methods fail at the 6.7B scale.
  • Emergence at scale: Outlier counts increase monotonically with decreasing C4 perplexity, while their relationship with model size is non-monotonic.This indicates that perplexity, rather than model size alone, tracks the phase shift more consistently.
  • Predictive importance: 150k outlier features in a 6.7B transformer sequence are concentrated in only 6 hidden dimensions, showing that the outliers are sparse and systematic.Removing at most 7 such dimensions sharply worsens predictive measures, whereas removing 7 random dimensions has little effect.
  • Quantization effects: Zeropoint quantization benefits from the outliers' asymmetric distributions, but accumulated errors and rapidly growing magnitudes cause it to fail at 13B.With mixed-precision decomposition, the advantage of zeropoint quantization disappears, while vector-wise quantization retains an advantage over row-wise quantization.

5 Related work

Related work covers alternative 8-bit data types, prior studies of language-model outliers, and concurrent methods for multi-billion-scale transformer quantization.

  • 8-bit data types: The paper studies Int8 because GPUs support it, while related 8-bit alternatives include fixed-point and floating-point formats such as FP8.FP8 variants allocate bits among sign, exponent, and fraction components.
  • Outlier features: Prior work analyzed large-magnitude language-model outliers and their relationships to layer normalization and token-frequency distributions.Other studies attributed outlier behavior to properties of BERT-family models and examined related transformer phenomena.
  • Transformer quantization: nuQmm and ZeroQuant concurrently used group-wise quantization, offering finer normalization granularity than vector-wise quantization but requiring custom CUDA kernels.Both methods targeted multi-billion-scale transformer quantization for inference.

6 Discussion and Limitations

LLM.int8() enables multi-billion-parameter transformers to run with Int8 inference while preserving full performance, but the study leaves several precision, scale, operation, and training questions open.

  • Discussion: 175B parameters is the largest tested scale at which the method recovers full inference performance.The authors report empirical results for models up to 175B parameters.
  • Limitations: The analysis studies Int8 but not 8-bit floating-point, and larger scales may introduce emergent properties that disrupt the method.The authors leave FP8 and scales beyond 175B for future work.
  • Limitations: The method does not use Int8 multiplication for the attention function because addressing it appeared to require additional quantization methods.The paper focuses on parameter-containing projections, while attention itself does not use parameters.
  • Limitations: The study focuses on inference and does not study training or finetuning at scale.The authors identify trade-offs among quantization precision, training speed, and engineering complexity.

7 Broader Impacts

The work may broaden access to large pretrained models for resource-constrained researchers, while also creating uncertain societal effects and potentially widening organizational disparities.

  • Access: The main impact is enabling large models that previously could not fit into GPU memory.The paper connects this access to research and applications that limited memory had prevented.
  • Risks: Serving more models on the same hardware could increase disparities between resource-rich and resource-poor organizations.The paper identifies this as a possible consequence of improved access for organizations with many GPUs.
  • Access: Public release of large pretrained models with Int8 inference may enable academic research previously blocked by resource constraints.The authors specifically mention OPT and zero- and few-shot prompting.
  • Risks: The widespread accessibility of large-scale models may have beneficial and detrimental societal effects that are difficult to predict.

Checklist

The checklist records the paper’s compliance answers, experimental reporting choices, asset disclosures, societal-impact discussion, and model-access evidence.

  • For all authors: The authors report that the paper’s main claims accurately reflect its contributions and scope, and that limitations and societal impacts are discussed.
  • Reproducibility: The paper states that code will be included in supplemental material and training details are described in the experimental setup.
  • Experiments: The experiments are deterministic for each model, so the authors do not report error bars from repeated runs.
  • Assets and data: The paper reports using existing assets and datasets, with creator citations, while marking several licensing and human-subject disclosures as not applicable or omitted.
  • Model access: LLM.int8() allows OPT-175B and BLOOM-176B to run on a single node with consumer-grade GPUs.
  • Related work: Prior quantization work largely targeted sub-billion-parameter masked language models, whereas this paper studies vector-wise quantization for autoregressive large-scale models.
  • Related work: The paper situates its approach among low-bit convolutional quantization and learned quantization methods, including mixed-precision optimization.
  • Outlier analysis: The outlier analysis summarizes magnitudes, prevalence, asymmetry, scale trends, and effects on softmax probabilities and quantization behavior.

C Detailed Outlier Feature Data

The outlier-feature data tabulate prevalence, distributional asymmetry, magnitudes, and scale-related changes associated with transformer validation perplexity.

  • Outlier statistics: Table 4 reports quartiles for the most common outlier and counts of one-sided outliers in each transformer.One-sided outliers have asymmetric distributions that do not cross zero.

D.1 Matrix Multiplication benchmarks

Int8 matrix multiplication reduces memory use, but practical speed gains depend strongly on model size, GPU saturation, quantization overhead, and mixed-precision decomposition. End-to-end BLOOM-176B inference is slightly slower but close to 16-bit latency per token.

  • Matrix multiplication benchmarks: Int8 matrix multiplication yields an advantage only when the entire GPU is well saturated, occurring in LLMs with model dimension 4096 or larger.Quantization and decomposition overheads are significant.
  • Matrix multiplication benchmarks: 2x faster raw Int8 matrix multiplication begins at model size 5140, but quantization and dequantization reduce the speedup to 1.6x.The comparison is against 16-bit multiplication at model size 5140.
  • Matrix multiplication benchmarks: Models with model size 2560 or smaller are slowed down, while mixed-precision decomposition leaves speedups only for the 13B and 175B models.Adding mixed-precision decomposition slows inference further.
  • Matrix multiplication benchmarks: Optimized CUDA kernels could significantly improve mixed-precision decomposition, while default quantization kernels slow all matrix multiplications except for a 175B model.Existing custom CUDA kernels are faster than default PyTorch and NVIDIA-provided kernels.
  • End-to-end inference: BLOOM-176B end-to-end Int8 inference is slightly slower but close to 16-bit inference in millisecond latency per token.Distributed inference includes fixed communication overhead, which reduces the overall speed difference.

E Training Results

Training experiments show that 8-bit FFN layers can preserve performance more reliably than 8-bit attention components, especially as model scale increases. Attention linear projections degrade performance at larger scales, and mixed precision is insufficient to recover full performance in most settings.

  • Training results: 8-bit FFN layers are straightforward for training, whereas other layers require additional techniques or different data types to avoid performance degradation at scale.Mixed-precision decomposition improves results slightly but usually does not restore full performance.
  • Training results: At large scale, performance degrades for 8-bit attention linear projections even when small-scale models remain close to baseline performance.The table summary contrasts small- and large-scale language modeling.

F Fine-tuning Results

Fine-tuning results distinguish robust 8-bit FFN quantization from more fragile attention-projection quantization. Vector-wise quantization improves over prior methods, while mixed-precision decomposition is needed to avoid degradation in 8-bit attention projections.

  • Fine-tuning results: 8-bit FFN layers lead to no degradation in fine-tuning, while 8-bit attention linear projections degrade performance without mixed-precision decomposition.The results use median performance over five random seeds.
  • Fine-tuning results: At least the top 2% magnitude dimensions must be computed in 16-bit instead of 8-bit to prevent degradation from 8-bit attention linear projections.Mixed-precision decomposition is identified as critical for fine-tuning attention projections.
  • Fine-tuning results: Vector-wise quantization improves over other 8-bit methods for GLUE fine-tuning of the feedforward layer.The comparison keeps the rest of the model in 16-bit and uses no mixed-precision decomposition.
  • Fine-tuning results: Fine-tuning with 8-bit attention layers appears to require a higher decomposition percentage than inference.The table reports median scores over five random seeds.
Loading 2208.07339v2…