Source-linked AI summary
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, Paulius Micikevicius
TL;DR
Integer quantization targets faster and smaller neural-network inference by using integer weights and activations with high-throughput integer math pipelines. The paper reviews quantization choices, evaluates them across diverse architectures and tasks, and presents an int8 workflow that maintains accuracy within 1% of floating-point baselines, including challenging models.
Problem
Lower-precision inference can improve performance, but quantization must minimize its impact on model accuracy across diverse neural networks.
Method
The paper reviews integer-quantization fundamentals and accuracy-recovery techniques, then evaluates parameter choices and combines them into an int8 workflow.
Results
The workflow maintains accuracy within 1% of floating-point baselines for all studied models, including MobileNets and BERT-large.
Takeaways & Limitations
Post-training quantization, partial quantization, and quantization-aware fine-tuning were sufficient for the studied models; more complex methods were not required.
Abstract
from arXiv · showhide
Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are more difficult to quantize, such as MobileNets and BERT-large.
1 Introduction
The paper motivates integer quantization as a way to accelerate neural-network inference with high-throughput integer hardware while preserving accuracy. It reviews quantization choices and evaluates a workflow across diverse models and application domains.
- Low-precision formats can accelerate math-intensive operations such as convolutions and matrix multiplications through higher-throughput processor pipelines.
- 8-bit integer tensor operations can provide up to a 16x speed-up over fp32 for math-intensive operations on NVIDIA Turing GPUs.
- Integer quantization modifies trained networks to use integer weights and activations so that integer math pipelines handle many inference operations.
- The paper reviews quantization fundamentals, accuracy-recovery techniques, and a recommended workflow, evaluating models across vision, language, translation, and speech tasks.
- The proposed int8 workflow maintains accuracy within 1% of each floating-point baseline, including challenging MobileNets and BERT-large models.
2 Related Work
Prior work established post-training int8 quantization and explored training-based methods, especially as quantization extended beyond image CNNs to recurrent and attention-based architectures.
- Earlier neural networks were quantized after training for int8 CPU inference while maintaining floating-point accuracy.
- Later work showed that some modern networks require training to maintain accuracy when quantized for int8 inference.
- Very low-bit quantization studies found that training was required for high accuracy, which remained below floating-point performance on harder tasks such as ImageNet classification.
- Quantization research expanded beyond CNN image classification to LSTM-based translation systems and Transformer architectures.
- Distillation and ADMM were proposed as more complex approaches for training quantized models, generally targeting lower bit-widths.
3 Quantization Fundamentals
The paper formalizes uniform integer quantization through range selection and real-to-integer mapping, then examines granularity, computational cost, and calibration choices. It favors configurations that preserve integer-math efficiency while limiting accuracy impact.
- 3.1 Range Mapping: Uniform quantization first clamps values to a chosen real range, then maps them to representable integers by rounding.Values outside the selected range are clipped to the nearest bound.
- 3.1 Range Mapping: Affine quantization uses a scale and zero-point, while scale quantization uses only a scale transformation.The zero-point maps real zero to an integer value in affine quantization.
- 3.2 Tensor Quantization Granularity: Quantization granularity ranges from per-tensor sharing to per-element parameters, with intermediate options such as per-row, per-column, and per-channel.Granularity affects both model accuracy and computational cost.
- 3.2 Tensor Quantization Granularity: Integer matrix multiplication supports per-row or per-tensor activation quantization and per-column or per-tensor weight quantization.For performance, activations should use per-tensor granularity, while weights can use per-column granularity for linear layers and per-channel granularity for convolutions.
- 3.3 Computational Cost of Affine Quantization: Scale quantization leaves an integer matrix multiply followed by one point-wise floating-point multiplication, whereas affine quantization adds extra computation involving quantized inputs.The paper recommends scale quantization for weights and reports it sufficient for int8 activations in all studied networks.
- 3.4 Calibration: Calibration chooses real ranges using maximum magnitude, KL-divergence-based entropy, or distribution-percentile methods.Percentile calibration intentionally clips the largest-magnitude values to reduce rounding errors for most values.
4 Post Training Quantization
This section evaluates post-training quantization parameter choices across diverse neural-network tasks and architectures. It finds that weight quantization is robust with per-channel max calibration, while activation calibration requires network-specific choices.
- Method: Post-training quantization calibrates parameters offline from trained weights and sample-dataset activations without further training.The evaluation covers convolutional, recurrent, and attention-based networks, using task-specific accuracy metrics.
- Weight Quantization: Per-channel int8 weight quantization with max calibration maintains accuracy, including after batch-normalization folding.Per-tensor quantization can cause substantial losses, especially when batch-normalization parameters create different channel-wise weight distributions.
- Activation Quantization: Activation calibration is network-dependent: entropy, 99.99%, or 99.999% percentile methods generally perform best, but no single method wins universally.Max calibration is inconsistent, while 99.9% percentile calibration clips large values too aggressively.
- Activation Quantization: MobileNets, EfficientNets, Transformer, and BERT experience accuracy drops larger than 1% under activation quantization.These models are more sensitive than most evaluated networks to activation calibration choices.
5 Techniques to Recover Accuracy
The paper presents partial quantization, quantization-aware training, and learned activation ranges as techniques for recovering accuracy lost during integer quantization. Across challenging models, these methods generally restore accuracy to near the floating-point baseline, although BERT requires approaches beyond simple layer skipping.
- 5.1 Partial Quantization: Partial quantization leaves sensitive layers in floating-point to trade some performance for improved accuracy.Because layer quantization affects downstream inputs, the paper uses one-at-a-time sensitivity analysis instead of evaluating every possible layer subset.
- 5.1 Partial Quantization: 0.65% relative top-1 accuracy drop remained for EfficientNet b0 after skipping its 10 most sensitive layers.This retains 72 of 82 convolution layers in quantized form while keeping the 10 sensitive layers in floating-point.
- 5.1 Partial Quantization: Within 1% of fp32 accuracy was recovered for MobileNet v1, EfficientNets, and Transformer by skipping only a few sensitive layers.BERT was the exception: sensitivity analysis did not identify a small subset of layers to leave in floating-point, motivating training-based recovery methods.
- 5.2 Quantization-Aware Training: Quantization-aware training inserts quantization operations before fine-tuning so the network can adapt to quantized weights and activations.Fake quantization approximates quantize-dequantize behavior in floating point, while the STE supplies an approximate derivative for backpropagation.
- 5.2 Quantization-Aware Training: Fine-tuning improved accuracy in most cases, and all networks maintained accuracy within 1% of their pretrained fp32 models.ResNeXt101, Mask R-CNN, and GNMT showed marginally better post-training quantization results, but the differences were described as run-to-run noise.
- 5.3 Learning Quantization Parameters: Learning activation ranges improved accuracy over fixed max ranges for most networks, especially where fixed max ranges caused substantial drops.When initialized from each network’s best PTQ calibration, learned and fixed ranges produced very similar int8 results.
6 Recommended Workflow
The recommended workflow progressively applies post-training quantization, partial quantization, and quantization-aware fine-tuning. It begins with calibrated int8 quantization and escalates only when accuracy or performance requirements are not met.
- Quantization parameters: The workflow recommends scale quantization with per-column/per-channel granularity and a symmetric integer range of [-127, 127].For activations, the recommendations also include per-tensor granularity and max calibration.
- PTQ: Post-training quantization first quantizes computationally intensive layers and calibrates activations with max, entropy, and percentile methods.If these calibrations do not achieve the desired accuracy, the workflow proceeds to partial quantization or QAT.
- Partial Quantization: Partial quantization uses sensitivity analysis to leave the most sensitive layers in floating-point.The workflow continues to QAT if the resulting performance impact is unacceptable or the target accuracy remains out of reach.
- QAT: QAT starts from the best calibrated quantized model and fine-tunes for around 10% of the original training schedule.The recommended schedule uses an annealing learning rate beginning at 1% of the initial training learning rate.
7 Conclusions
The paper reviews integer-quantization fundamentals, evaluates int8 choices across diverse models, and derives a workflow that preserves near-baseline accuracy. The workflow succeeds on challenging networks without requiring more complex techniques such as ADMM or distillation.
- Conclusion: The workflow quantized all studied models to int8 with accuracy matching or within 1% of the floating-point model.The evaluation included challenging networks such as MobileNets and BERT.
- Conclusion: The proposed workflow combines post-training quantization, partial quantization, and quantization-aware fine-tuning.More complex methods, including ADMM and distillation, were not required for the studied int8 models.
- Conclusion: Evaluation covered mathematical and performance-related choices for int8 quantization across a variety of neural network models.The paper leaves evaluation of more complex techniques for even lower-bit integer representations to future work.
A Evaluation Details
The evaluation spans diverse model families and application settings, using repository implementations and pretrained weights where available. Fine-tuning schedules and configuration details are summarized separately from the network descriptions.
- Network details: The evaluated models include a large variety of CNNs for image classification, including MobileNets and EfficientNets.MobileNet experiments use width multiplier 1 and 224x224 resolution.
- Evaluation setup: Calibration samples came from each dataset’s training set, except Jasper, which used the development set for calibration and the test set for evaluation.The study used pretrained repository weights except for MobileNet v1 and EfficientNets, whose weights were unavailable.
- Fine-tuning configuration: Table 10 summarizes the fine-tuning schedule and configuration.The complete fine-tuning methodology is presented as a separate evaluation detail.
- Fine-tuning results: Table 11 reports fine-tuned quantization accuracy, highlighting the best accuracy and underlining accuracy from the best PTQ calibration.The table distinguishes fine-tuning outcomes from the corresponding post-training quantization baseline.
A.2 Quantization Aware Training
Quantization-aware training improves quantized accuracy by fine-tuning with quantization effects and can favor wider loss minima. Batch normalization can be folded into preceding layer parameters to simplify integer inference.
- A.2 Quantization Aware Training: Fine-tuning uses roughly one-tenth of the original training steps, changing primarily the initial learning rate and schedule.For multi-dataset networks, fine-tuning uses only the final dataset; the learning rate starts at 1/100th of the initial training value.
- A.2 Quantization Aware Training: Fine-tuning improves accuracy in almost all tested cases, especially models with large post-training quantization drops.The best PTQ activation calibration is often also best for QAT, making PTQ calibration evaluation a heuristic for selecting QAT calibration.
- A.2 Quantization Aware Training: QAT can move optimization away from narrow minima toward wide minima whose quantization points have lower loss and potentially higher accuracy.In the example, PTQ maps w ≈ −0.5 to w_q = −1 with increased loss, whereas QAT finds a wider minimum.
- C Batch normalization folding: Batch normalization folding replaces the batch-normalization affine operation with modified weights and biases in the preceding fully connected layer.The resulting layer performs z = xW′ + b′, and the same optimization applies to convolutions mapped to fully connected layers.
D Novel activation functions
Swish and GELU are smooth, ReLU-like activations with bounded negative outputs that complicate uniform quantization. For BERT, clipping GELU outputs to 10 substantially improves post-training int8 accuracy and nearly matches QAT.
- D Novel activation functions: Swish and GELU have output ranges [−0.2785, ∞] and [−0.1700, ∞], respectively, combining small negatives with large positives.This asymmetric range makes uniform quantization difficult.
- D Novel activation functions: Clipping GELU outputs to 10 with max calibration exceeds the previous best activation calibration by 0.46 F1.The result is reported for BERT int8 post-training quantization.
- D Novel activation functions: A symmetric quantization range of [-10, 10] represents two negative GELU values, whereas [-50, 50] rounds all negative values to zero.The narrower range preserves more of GELU’s bounded negative tail.
- D Novel activation functions: GELU10 post-training quantization almost matches the best QAT result of 90.67 F1.The comparison uses clipped GELU outputs and simple max calibration.