PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He

arXiv:2206.01861v1cs.CLcs.LG

TL;DR

Large language models are increasingly difficult to deploy because their memory and computational requirements strain even powerful cloud servers. ZeroQuant addresses this with fine-grained post-training quantization, affordable layer-by-layer distillation, and an optimized inference backend, achieving strong efficiency results across large Transformer models.

  • Problem

    Large language models’ growing memory and computational requirements make efficient deployment difficult, while conventional QAT requires unavailable or costly training data, compute, and retraining.

  • Method

    ZeroQuant combines hardware-friendly weight and activation quantization, layer-by-layer knowledge distillation without original training data, and optimized inference support.

  • Results

    ZeroQuant reports up to 5.19x speedup on BERT models, 5.2x better efficiency on GPT-NeoX20B, and 3x model-size reduction with mixed INT4/INT8 quantization.

  • Takeaways & Limitations

    The pipeline supports INT8 quantization for models up to GPT-NeoX20B while maintaining similar accuracy to FP16 in the reported evaluations.

  • Takeaways & Limitations

    The work focuses on natural language models and leaves performance on computer vision models for future work.

Abstract

from arXiv · show

How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memory/computation requirements. In this work, we present an efficient and affordable post-training quantization approach to compress large Transformer-based models, termed as ZeroQuant. ZeroQuant is an end-to-end quantization and inference pipeline with three main components: (1) a fine-grained hardware-friendly quantization scheme for both weight and activations; (2) a novel affordable layer-by-layer knowledge distillation algorithm (LKD) even without the access to the original training data; (3) a highly-optimized quantization system backend support to remove the quantization/dequantization overhead. As such, we are able to show that: (1) ZeroQuant can reduce the precision for weights and activations to INT8 in a cost-free way for both BERT and GPT3-style models with minimal accuracy impact, which leads to up to 5.19x/4.16x speedup on those models compared to FP16 inference; (2) ZeroQuant plus LKD affordably quantize the weights in the fully-connected module to INT4 along with INT8 weights in the attention module and INT8 activations, resulting in 3x memory footprint reduction compared to the FP16 model; (3) ZeroQuant can be directly applied to two of the largest open-sourced language models, including GPT-J6B and GPT-NeoX20, for which our INT8 model achieves similar accuracy as the FP16 model but achieves up to 5.2x better efficiency.

1 Introduction

ZeroQuant targets the deployment bottlenecks of large language models with an end-to-end post-training quantization pipeline that combines fine-grained quantization, affordable layer-wise distillation, and optimized inference. It reports substantial speedups, memory reduction, and scalability across BERT, GPT-3-style, GPT-J6B, and GPT-NeoX20B models.

  • Large language models’ growing memory footprint and computational cost make deployment difficult even on powerful cloud GPUs.
  • Quantization reduces weight and activation precision for lower memory usage and faster computation, but conventional QAT requires training data, compute resources, and time.
  • ZeroQuant combines hardware-friendly weight and activation quantization, layer-by-layer knowledge distillation without original training data, and an optimized inference backend.
  • Up to 5.19x/4.16x speedup over FP16 inference is achieved for BERTbase/GPT-3350M with INT8 models on A100 GPUs.
  • 3x memory footprint reduction is achieved through INT4/INT8 mixed-precision quantization with marginal accuracy loss, while quantization takes 33s for BERTbase and 10 minutes for BERTlarge.
  • For GPT-J6B, ZeroQuant achieves 3.67x speedup, while GPT-NeoX20B inference falls from 2 GPUs and 65ms latency to 1 GPU and 25ms latency.

2 Related Work

Prior NLP quantization work largely relies on retraining or distillation with original training data, while PTQ reduces retraining needs but has often shown accuracy or deployment-efficiency limitations.

  • Quantization is a prominent model-compression direction because it directly reduces memory footprint and computational intensity for NLP models.
  • Early BERT quantization methods use integer weights and activations, with some applying Hessian information and group-wise quantization to reach INT2/INT4 weights.
  • Knowledge distillation and data augmentation have supported very low-bit compression, but these methods generally use original training datasets and retrain or finetune the full model.
  • Post-training quantization avoids retraining but can cause significant accuracy drops because Transformer networks are sensitive to quantization errors.

3 Background and Challenge

PTQ offers efficient compression without retraining, but applying it to large Transformers exposes accuracy challenges from heterogeneous activation and weight ranges, especially for low-bit settings.

  • PTQ usually quantizes models without retraining and commonly calibrates activation scaling factors from training data using running means.
  • Existing BERT PTQ work leaves lower-bit BERT quantization and large-scale GPT-3-style models insufficiently investigated.
  • GPT-3350M experiments evaluate post-training configurations across 20 zero-shot datasets, including W4/8 with INT8 MHSA weights and INT4 FFC weights.
  • INT8 activation quantization causes the primary accuracy loss, while W4/8 can make GPT-3350M unable to generate meaningful Wikitext-2 text.
  • Different tokens and layers exhibit substantially different activation ranges, motivating token-wise rather than static activation quantization.
  • The resulting accuracy gap motivates lightweight knowledge distillation because hidden-state distillation is expensive for large-scale models.

4 Methodology

ZeroQuant combines fine-grained quantization with token-wise activation handling, layer-by-layer distillation, and an optimized inference backend. These choices target accuracy retention and practical latency reduction for large Transformer models.

  • Fine-grained Hardware-friendly Quantization Scheme: INT8 PTQ can substantially degrade BERT and GPT-3-style model accuracy because one scale cannot capture different row and token numerical ranges.This motivates finer-grained quantization for weights and activations.
  • Fine-grained Hardware-friendly Quantization Scheme: Group-wise weight quantization and token-wise activation quantization reduce representation error while preserving hardware acceleration properties.Token-wise activation ranges are dynamically calculated per token rather than calibrated statically across tokens.
  • Inference Backend: The optimized backend addresses data-movement overhead introduced by token-wise quantization and its associated quantization/dequantization operations.The implementation is designed to avoid losing latency benefits to token-wise processing overhead.
  • Layer-by-layer Knowledge Distillation with Affordable Cost: Layer-by-layer distillation quantizes each transformer block using the original block as teacher and the preceding layers' output as input.The loss compares the unquantized and quantized block outputs on an accessible dataset.
  • Layer-by-layer Knowledge Distillation with Affordable Cost: LKD reduces distillation cost by optimizing only the current layer, avoiding a separate teacher model and eliminating dependence on labels or the original training data.The method also reduces optimizer-state memory overhead because only one layer is optimized at a time.
  • Inference Backend: ZeroQuant's system optimizations produce significant latency reduction for BERT and GPT-3-style models, while INT8 quantization reduces the data volume loaded from memory.The paper emphasizes that lower precision alone does not guarantee improved latency without backend support.

5 Results

ZeroQuant maintains strong accuracy across BERT and GPT-3-style models while enabling aggressive quantization, substantial speedups, and reduced memory or training costs. Its benefits extend to large open-source models and are supported by fine-grained quantization, lightweight distillation, and optimized inference.

  • BERT results: ZeroQuant achieves 83.75 average accuracy on BERTbase with W8A8, only 0.2 below baseline and without activation-range calibration cost.Its average score is 1.29 higher than the compared method [6].
  • BERT results: ZeroQuant saves INT8 retraining time from 2900s to 0s, while LKD raises BERTbase W4/8A16 accuracy from 81.65 to 82.35 at 31s per task.The LKD cost is reported as 93.5x cheaper than INT8 QAT quantization.
  • BERT results: 3x smaller model size is achieved for BERTlarge with W4/8A16 or W4/8A8 while retaining good accuracy except on RTE.LKD takes about 550s per task on BERTlarge, reported as 13x cheaper than QAT.
  • GPT-3-style results: ZeroQuant reduces GPT-3350M W8A8 accuracy and perplexity gaps versus PTQ from 1.1% to 0.2% and from 4.7 to 0.2, respectively.This comparison uses 19 accuracy-based tasks and Wikitext-2, with no activation-range calibration cost.
  • Latency reduction: 4.16x/4.06x speedup is achieved for GPT-3350M/GPT-31.3B, while BERTbase and BERTlarge reach 2.27–5.19x and 2.47–5.01x speedups.These are end-to-end comparisons of INT8 or W8A8 inference against FP16 or W16A16 baselines.
  • Scalability: 3.67x speedup is achieved on GPT-J6B, whose W8A8 perplexity remains similar to FP16 across PTB, Wikitext-2, and Wikitext-103.For GPT-NeoX20B, INT8 quantization reduces inference requirements from two GPUs to one and latency from 65ms to 25ms, yielding 5.2x system efficiency.
  • Ablation study: 14.54 points are gained by adding token-wise activation quantization after group-wise weight quantization, while LKD adds a further 0.56 points.The ablation uses BERTlarge with W4/8A8.
  • No original training data: Wikipedia data raises GPT-3350M W4/8A8 accuracy to 36.2 and lowers perplexity to 30.4, comparable to using the original data.Random data also improves accuracy by 1.1% and reduces perplexity from 92.1 to 40.6.

6 Conclusions

ZeroQuant targets the practical difficulty of serving increasingly large language models by combining fine-grained compression with affordable post-training quantization and inference support. It reports INT8 scalability to 20B-scale models, reduced model size, and substantial efficiency gains.

  • ZeroQuant addresses the memory and computational bottlenecks that make serving rapidly growing language models difficult in practice.
  • The approach offers fine-grained compression for weights and activations, layer-by-layer distillation for ultra-low precision, and backend support for online inference.
  • INT8 quantization scales to GPT-NeoX20B, while ultra-low precision reduces model size by 3x versus FP16 with minimal accuracy degradation.
  • The system reports up to 5.19x speedup on BERT and 5.2x better efficiency on GPT-NeoX20B.

A.1 Transformer Architecture

The paper focuses quantization on Transformer encoder/decoder blocks, whose attention and feed-forward sublayers are major memory and compute components. It uses uniform symmetric quantization with hardware-oriented handling of weights and activations.

  • Transformer Architecture: ZeroQuant focuses on encoder/decoder Transformer blocks because they are often the architecture’s most memory- and compute-intensive components.
  • Transformer Architecture: A Transformer block contains multi-head self-attention and feed-forward connection sublayers within encoder-only and decoder-only model families.
  • Quantization Background: Quantization maps FP16/FP32 values to INT4/INT8 values to reduce model footprint and improve compute performance using uniform symmetric scalar quantizers.
  • Quantization Background: Weight scales are generally computed from the maximum absolute weight, whereas activation scales require dynamic inference calculation or calibrated static estimation.

B Experimental Details

The experiments evaluate BERT and GPT-3-style models across GLUE and zero-shot tasks using specified calibration, grouping, distillation, training, and scoring procedures. The reported tables cover model metrics and mixed-precision PTQ configurations.

  • Calibration: Activation calibration uses 100 iterations with 0.95 momentum for BERT and the same momentum method with batch size 8 and sequence length 2048 for GPT-3-style models.
  • Models and Tasks: BERT experiments cover BERTbase and BERTlarge across GLUE tasks, while GPT-3-style experiments evaluate 20 zero-shot tasks including language modeling.
  • Quantization Configuration: Weight quantization uses 48 or 64 groups for BERT and 64 or 128 groups for GPT-3-style models, depending on model size.
  • Distillation: Layer-wise distillation uses 100 iterations for BERTbase, 400 for BERTlarge, and 1600 for both GPT-3-style models, with a fixed learning rate of 5e-6.
  • Compute: BERT and quantized models are trained using a single 40G-A100 GPU, while GPT-3-style pretraining uses 128 A100 GPUs.
  • Evaluation: BERT’s final average score averages task metrics, first averaging the two metrics when a task reports two metrics.

C PTQ challenge of BERTbase

BERTbase PTQ is especially sensitive to activation quantization and extreme weight compression. ZeroQuant’s backend addresses inference overhead through hardware-aware scheduling, fused operations, and optimized dequantization.

  • PTQ Accuracy: INT8 weight quantization causes negligible accuracy degradation, whereas INT8 activation quantization reduces accuracy from 84.06 to 79.61.
  • PTQ Accuracy: INT4 FFC weights with INT8 MHSA weights produce purely random predictions without meaningful accuracy.
  • Inference Backend: The inference backend adapts CUTLASS GeMM schedules to input sizes and uses runtime schedule selection to reduce padding during Tensor-core multiplication.
  • Inference Backend: Activation quantization is fused with bias addition, GeLU, and LayerNorm to remove overhead before GeMM operations.
  • Inference Backend: A custom epilogue combines dequantization with GeMM, while quantization scales are read in advance to overlap parameter access with computation.
  • Inference Backend: CUDA Graph support reduces CPU overhead when optimized INT8 kernels make kernel launches and CPU-to-GPU communication significant bottlenecks.

E Tuned Results on BERT

The tuned BERT experiments select hyperparameters for ZeroQuant and QAT, report best single-run results, and expose a performance weakness for BERTlarge on RTE.

  • Hyperparameter tuning: Learning rates and iteration counts were tuned for BERTbase and BERTlarge to show ZeroQuant’s best possible performance.Learning rates were selected from four values, while iteration counts were selected from seven values including zero.
  • Tuning cost: 4.5 hours on 8 40G-A100 GPUs was the total tuning time for BERTbase, including data loading, evaluation, tokenization, and three quantization schemes.BERTlarge tuning took around 16 hours on the same hardware.
  • Reported results: The appendix summarizes the best tuned results for BERTbase and BERTlarge in Tables E.1 and E.2.Both tables cover GLUE development-set results except WNLI, with W4/8 using INT8 MHSA weights and INT4 FFC weights.
  • QAT comparison: The reported paper results were chosen from the best single run among four QAT learning rates for BERTlarge.The selected rates were 5e-6, 1e-5, 2e-5, and 5e-5.
  • QAT comparison: Even after learning-rate tuning, QAT did not achieve good performance for BERTlarge on RTE.This is reported as a limitation of the tuned QAT results.
  • Tuning cost: Including tuning cost would make the reported single-run time equal to 4 × 7181s.The main-text time cost is based on a single run.

G Limitations and Future Work

The paper identifies three limitations: evaluation is restricted to natural language models and models up to 20B parameters, while activation sensitivity could not be checked across other extra-large models.

  • Scope: ZeroQuant was evaluated only on natural language models, leaving its performance on computer vision models for future work.The paper explicitly identifies computer vision evaluation as an open direction.
  • Scale: ZeroQuant’s scalability was verified only up to 20B-scale models.The authors propose testing larger open-sourced models when available.
  • Quantization sensitivity: GPT-NeoX20B showed greater quantization sensitivity in the self-attention activation input, but this finding was not verified on other extra-large models.The paper attributes the missing verification to the lack of other open-sourced models at that scale.

H Full Zero-shot Evaluation of GPT-3-style Models

The appendix provides full zero-shot evaluation results for GPT-3-style models, including GPT-1.3B and GPT-NeoX20B, plus data-resource comparisons for GPT-3350M.

  • Evaluation coverage: The appendix includes all zero-shot evaluation results for GPT-3-style models, including GPT-NeoX20B.The supplied passage identifies the evaluation coverage but does not state the numerical outcomes.
  • Model results: Table H.2 reports the full results for GPT-1.3B.No individual metric values are supplied in the passage.
  • Data-resource comparison: Table H.3 reports full results for W4/8A8 GPT-3350M using different data resources.The passage identifies the table’s comparison dimension without reporting its numerical findings.
  • Model results: Table H.4 reports the full results for GPT-NeoX20B.No individual metric values are supplied in the passage.
Loading 2206.01861v1…