PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Quantization Methods for Efficient Neural Network Inference

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, Kurt Keutzer

arXiv:2103.13630v3cs.CV

TL;DR

The paper addresses how to represent neural-network values with fewer bits while balancing computational efficiency and accuracy, especially under resource constraints. It surveys quantization methods, concepts, trade-offs, and deployment implications, reporting practical speedups from INT8, INT4, and mixed-precision inference across models and hardware. The survey also identifies accuracy degradation, retraining costs, data requirements, and implementation overheads as important boundaries of current methods.

  • Problem

    Neural-network models are often over-parameterized, making efficient low-precision representation important for resource-constrained inference with real-time, energy, memory, and accuracy requirements.

  • Method

    The paper surveys quantization concepts and methods, including precision schemes, NN-specific granularities, training relationships, data availability, vector quantization, and hardware implications.

  • Results

    INT8 quantization yields reported speedups across GPUs and non-GPU hardware, while INT4 and mixed-precision methods provide additional or comparable acceleration in selected models.

  • Takeaways & Limitations

    Quantization is a highly active approach for improving neural-network inference efficiency, with reported benefits across model architectures and hardware platforms.

Abstract

from arXiv · show

As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.

I. INTRODUCTION

Over-parameterized neural networks create deployment challenges for resource-constrained applications requiring real-time inference, low energy consumption, and high accuracy. The survey situates quantization among efficiency techniques and focuses on its use for neural-network inference.

  • I. INTRODUCTION: Over-parameterized neural networks are often too large to deploy in resource-constrained applications.This limits pervasive deep learning applications requiring real-time inference, low energy consumption, and high accuracy.
  • I. INTRODUCTION: Efficient neural-network design seeks latency, memory-footprint, and energy reductions while preserving accuracy and generalization.
  • I. INTRODUCTION: Efficiency efforts include architecture design, hardware–architecture co-design, pruning, knowledge distillation, and quantization.Architecture and hardware co-design can be manual or automated; pruning removes low-saliency neurons, while distillation trains a compact student from a larger teacher.
  • I. INTRODUCTION: Quantization has succeeded in neural-network training and inference, while this survey primarily focuses on inference.Half-precision and mixed-precision training enabled an order of magnitude higher throughput in AI accelerators, but going below half precision is difficult without significant tuning.
  • I. INTRODUCTION: The survey introduces current quantization methods and concepts, discusses challenges and opportunities, and organizes relevant research for evaluation.Its structure covers history, basic concepts, advanced low- and mixed-precision methods, hardware accelerators, and conclusions.

II. GENERAL HISTORY OF QUANTIZATION

Quantization has a long history spanning numerical approximation, digital signal processing, coding theory, and numerical analysis. For neural networks, the survey introduces basic mappings and notation while connecting recent methods to earlier quantization ideas and NN-specific challenges.

  • II. GENERAL HISTORY OF QUANTIZATION: Quantization maps values from a large, often continuous input set to a small, often finite output set.Rounding and truncation are typical examples, and the practice predates digital computing.
  • II. GENERAL HISTORY OF QUANTIZATION: Shannon argued that assigning the same number of bits to events with non-uniform probabilities is wasteful.A more optimal coding approach varies the number of bits according to probability.
  • II. GENERAL HISTORY OF QUANTIZATION: Quantization developed through applications in communication, pulse-code modulation, high-resolution signal processing, numerical analysis, and digital computation.
  • II. GENERAL HISTORY OF QUANTIZATION: Neural networks introduce distinct quantization opportunities because their computations are intensive, models are over-parameterized, and layer sensitivities differ.Different layer impacts on the loss motivate mixed-precision quantization.
  • II. GENERAL HISTORY OF QUANTIZATION: The survey sets up neural-network quantization using layer parameters, activations, inputs, labels, and an empirical-risk objective.It assumes trained parameters are stored in floating-point precision before quantization reduces parameter and activation precision.

B. Uniform Quantization

Uniform quantization maps floating-point weights and activations to uniformly spaced integer levels using a scale, zero point, and clipping range. Symmetric and asymmetric choices trade implementation simplicity against fit to skewed distributions, while calibration must manage outliers and resolution.

  • Symmetric vs asymmetric: 8-bit symmetric quantization uses either the restricted range [-127, 127] or the full range [-128, 127], with the full range described as more accurate.Symmetric quantization is widely adopted for weights because removing the zero point can reduce inference computation and simplify implementation.
  • Uniform quantization: Uniform quantization maps real-valued weights or activations to lower-precision integer values with uniformly spaced quantization levels.The quantization operator uses a real-valued scaling factor and integer zero point, while rounding introduces reconstruction error.
  • Scaling and clipping: The scaling factor partitions the clipped real-value range, whose endpoints [α, β] and bit width b determine the quantization resolution.The clipping range must be selected before the scaling factor is defined; min/max is one straightforward calibration choice.
  • Calibration: Min/max calibration is vulnerable to activation outliers, which can widen the range and reduce quantization resolution.Percentile-based endpoints and KL-divergence minimization are alternatives for reducing information loss.
  • Symmetric vs asymmetric: Symmetric quantization sets Z = 0 and simplifies implementation, but asymmetric quantization is preferred when the value range is skewed or imbalanced.Symmetric ranges are convenient for weights, whereas asymmetric ranges can better represent distributions such as non-negative ReLU activations.

D. Range Calibration Algorithms: Static vs Dynamic Quantization

Activation-range calibration varies along two axes: when ranges are computed and how many parameters share them. Dynamic ranges often improve accuracy but add runtime cost, while channelwise granularity improves resolution with negligible overhead.

  • Dynamic vs static: Dynamic quantization computes each activation map’s clipping range at runtime, often achieving higher accuracy but requiring expensive signal-statistics computation.The range can be calculated from statistics such as min, max, or percentile for every input activation map.
  • Dynamic vs static: Static quantization fixes activation clipping ranges during inference, avoiding computational overhead but typically reducing accuracy relative to dynamic quantization.Calibration inputs are commonly used beforehand to estimate typical activation ranges.
  • Quantization granularity: Layerwise quantization applies one clipping range across a layer, making implementation simple but often producing sub-optimal accuracy when filter ranges differ substantially.Channels with narrower distributions can receive poor quantization resolution under a shared range.
  • Quantization granularity: Groupwise quantization shares ranges across channel groups, while sub-channelwise quantization can incur considerable processing overhead from multiple scaling factors.Groupwise quantization can serve as a compromise between quantization resolution and overhead.
  • Quantization granularity: Channelwise quantization assigns each convolutional filter its own scaling factor, improving quantization resolution and often increasing accuracy.This granularity is described as the current standard for convolutional kernels because its overhead is negligible.

F. Non-Uniform Quantization

Non-uniform quantization uses unevenly spaced thresholds and levels to represent important value regions more accurately at a fixed bit width. Despite this potential, uniform quantization remains the de-facto choice because non-uniform schemes are harder to deploy efficiently on general hardware.

  • Definition: Non-uniform quantization allows both quantization steps and levels to be unevenly spaced, assigning discrete values according to the input distribution.A value between consecutive thresholds is projected to the corresponding quantization level.
  • Motivation: Non-uniform levels may improve fixed-bit-width accuracy by focusing representation on important regions or adapting to dynamic ranges and bell-shaped distributions.Many methods target weight and activation distributions with long tails.
  • Method families: Rule-based methods include logarithmic spacing, while binary-code-based methods approximate a real vector using scaled binary vectors.Logarithmic methods increase steps and levels exponentially rather than linearly; binary representations use scaling factors and vectors with entries in {−1, +1}.
  • Optimization and clustering: Optimization-based methods adjust quantization steps and levels to minimize the difference between original and quantized tensors, and learnable quantizers train these parameters iteratively or by gradient descent.Clustering methods include k-means and Hessian-weighted k-means for reducing quantization-related performance loss.
  • Practical trade-off: Uniform quantization remains the de-facto method because non-uniform schemes are typically difficult to deploy efficiently on GPUs and CPUs.The survey attributes uniform quantization’s practical status to its simplicity and efficient hardware mapping.

G. Fine-tuning Methods

Fine-tuning methods adjust neural-network parameters after quantization, chiefly through QAT, which trains with quantized parameters while retaining floating-point gradient computation. QAT can work with STE, but its main cost is retraining, especially at low precision.

  • Quantization-Aware Training: QAT retrains the model with quantized parameters after each floating-point gradient update to recover quantization-induced loss.The forward and backward passes remain in floating point, while parameters are quantized after updates.
  • Quantization-Aware Training: STE approximates the zero-almost-everywhere gradient of rounding by replacing the quantization operator with an identity function.This enables backpropagation through the non-differentiable quantization step.
  • Quantization-Aware Training: STE often works in practice despite its coarse approximation, except for ultra-low-precision settings such as binary quantization.Theory indicates that, for a suitable STE, its coarse gradient can correlate in expectation with the population gradient.
  • Quantization-Aware Training: QAT can learn quantization parameters during training, including activation clipping ranges, quantization steps, levels, and scaling factors.Examples include PACT, QIT, LSQ, and LSQ+.
  • Quantization-Aware Training: QAT’s main disadvantage is retraining cost, which may require several hundred epochs to recover accuracy, particularly for low-bit quantization.The investment is more likely to be worthwhile for models deployed over extended periods with strong efficiency and accuracy requirements.

2) Post-Training Quantization:

Post-training quantization determines quantization parameters without retraining, offering fast, low-overhead deployment and applicability when data are limited or unlabeled. Its speed and data flexibility can come with lower accuracy than QAT, motivating calibration, adaptive rounding, and zero-shot approaches.

  • Post-Training Quantization: PTQ quantizes and adjusts weights without fine-tuning, so its overhead is often negligible and it can operate with limited or unlabeled data.Unlike QAT, PTQ does not require a sufficient training set for retraining.
  • Post-Training Quantization: Adaptive PTQ methods correct quantization bias or replace naive rounding to reduce loss, with some allowing quantized weights to move beyond ±1 from full-precision values.Examples include bias correction, AdaRound, and AdaQuant.
  • Post-Training Quantization: PTQ determines all weight and activation quantization parameters without retraining, making it very fast but often less accurate than QAT.This is the central speed–accuracy trade-off identified in the survey.
  • Post-Training Quantization: Calibration uses representative inputs to estimate activation ranges for clipping and scaling, while fine-tuning can recover quantization-related accuracy degradation.Access to all or part of the training data is often needed for minimal degradation.
  • Zero-shot Quantization: Zero-shot quantization performs quantization without training or validation data, addressing cases where datasets are too large, proprietary, sensitive, or inaccessible.The survey distinguishes no-data PTQ from no-data QAT with fine-tuning.

H. Stochastic Quantization

Stochastic quantization randomly maps values to nearby discrete levels and has been explored for training and reduced-precision learning. QuantNoise extends this idea by quantizing random weight subsets, but random-number overhead limits practical adoption.

  • Stochastic Quantization: Stochastic quantization can let neural networks explore more than deterministic quantization during training.The motivation is that small updates may otherwise produce no quantized weight change.
  • Stochastic Quantization: Stochastic rounding maps a value down or up with probabilities determined by its fractional position between neighboring quantization levels.The displayed rule assigns probability ⌈x⌉−x to the lower value and x−⌊x⌋ to the upper value.
  • Stochastic Quantization: The standard stochastic-rounding definition cannot be used for binary quantization, requiring an extension based on binarization and a sigmoid function.The survey identifies this as a limitation of the displayed definition.
  • Stochastic Quantization: QuantNoise quantizes a different random subset of weights during each forward pass and trains with unbiased gradients.It supports lower-bit quantization without significant accuracy loss in many vision and language models.
  • Stochastic Quantization: Creating random numbers for every weight update is a major overhead, so stochastic quantization has not been widely adopted in practice.

A. Simulated and Integer-only Quantization

Quantized inference can use floating-point operations on dequantized values or integer-only arithmetic, with the latter generally offering greater hardware efficiency. Mixed precision preserves higher precision for sensitive layers while assigning lower precision elsewhere to balance accuracy and efficiency.

  • Simulated and Integer-only Quantization: Simulated quantization stores low-precision parameters but dequantizes them before floating-point matrix multiplications and convolutions.Integer-only quantization instead performs the inference arithmetic using low-precision logic.
  • Simulated and Integer-only Quantization: 30× greater energy efficiency and 116× greater area efficiency are reported for INT8 addition compared with FP32 addition.
  • Simulated and Integer-only Quantization: Dyadic quantization uses rational scaling with power-of-two denominators, requiring integer addition, multiplication, and bit shifting but no integer division.Shared dyadic scales across additions can simplify addition logic and improve efficiency.
  • Simulated and Integer-only Quantization: Integer-only and dyadic quantization are generally more desirable than fake quantization because their arithmetic uses lower-precision logic.Fake quantization can remain acceptable for bandwidth-bound tasks such as recommendation systems, where memory loading is the bottleneck.
  • Mixed-precision Quantization: Mixed-precision quantization assigns different bit widths across layers, preserving higher precision for sensitive layers and lower precision for insensitive layers.This approach targets lower accuracy degradation while retaining reduced memory footprint and faster low-precision computation.
  • Mixed-precision Quantization: Mixed-precision bit allocation can be selected through reinforcement learning, neural architecture search, regularization, or second-order sensitivity analysis.The search space is exponential in the number of layers, motivating automated selection methods.

C. Hardware Aware Quantization

Hardware-aware quantization accounts for hardware-dependent speedups while extreme low-bit methods trade substantial memory and latency savings against accuracy degradation. Approaches address this trade-off through error minimization, improved loss functions, and improved training methods.

  • Hardware-aware quantization: Quantization speedups depend on on-chip memory, bandwidth, and cache hierarchy, motivating hardware-aware mixed-precision settings.One approach uses reinforcement learning with latency look-up tables, though the hardware latency is simulated.
  • Extreme quantization: Binarization constrains values to 1 bit, reducing memory requirements by 32× and enabling efficient bit-wise arithmetic.Peak binary arithmetic on NVIDIA V100 GPUs is 8x higher than INT8, but naive binarization can substantially reduce accuracy.
  • Extreme quantization: Ternarization permits +1, 0, and -1 values, eliminating costly matrix multiplications; Ternary-Binary Networks combine binary weights with ternary activations.The combination is reported to achieve a tradeoff between accuracy and computational efficiency.
  • Accuracy recovery: Three solution branches target extreme-quantization degradation: quantization-error minimization, improved loss functions, and improved training methods.Examples include multiple binary matrices, loss-aware binarization or ternarization, knowledge distillation, and continuous sign-function derivatives.
  • Applications: Extreme quantization has reduced inference and training latency and model size for many CNN vision models, with emerging applications to NLP inference at the edge.The motivation includes the prohibitive size and latency of models such as BERT, RoBERTa, and GPT-family systems.
  • Extreme quantization: Extreme quantization methods often incur high accuracy degradation relative to baseline unless extensive tuning and hyperparameter search is performed.The degradation may be acceptable for less critical applications.

F. Vector Quantization

Vector quantization clusters weights and uses cluster centroids as quantized values, while product quantization applies this process to weight submatrices. These methods can reduce model size with limited accuracy degradation and can be combined with other compression techniques.

  • Quantization objective: Machine-learning quantization prioritizes small loss rather than minimal change from the original signal, so quantized weights and activations may differ substantially.The objective is a reduced-precision representation with as small a loss as possible.
  • Vector quantization: Vector quantization clusters weights into groups and uses each group’s centroid as the quantized inference value.The centroids are indexed as c1 through c_k for the weight tensor.
  • Results: k-means clustering can reduce model size up to 8× without significant accuracy degradation.Combining k-means vector quantization with pruning and Huffman coding can reduce model size further.
  • Product quantization: Product quantization divides a weight matrix into submatrices and applies vector quantization independently to each submatrix.Residuals can be recursively quantized, and more clusters can be assigned to important quantization ranges to preserve information.

V. QUANTIZATION AND HARDWARE PROCESSORS

Quantization supports neural-network deployment on resource-constrained edge processors by reducing model demands and enabling low-precision computation. Its practical benefits depend on hardware support, while software availability and quantized training remain important limitations and research directions.

  • Edge deployment: Quantization reduces model size, increases speed, and lowers power requirements, making it especially important for edge devices with constrained compute, memory, and power budgets.Many edge processors lack sufficient resources for large deep neural networks without such efficiency measures.
  • Edge processors: ARM Cortex-M devices may require quantized models because some cores lack dedicated floating-point units.CMSIS-NN uses fixed-point quantization with power-of-two scaling so quantization and dequantization can use bit shifting.
  • Edge processors: Quantization, low-precision logic, and dedicated deep-learning accelerators have helped drive the evolution of commercial edge processors.Figure 9 compares throughput across commercial processors used for edge inference.
  • Non-edge processors: Quantization also benefits non-edge processors by supporting latency requirements such as 99th-percentile SLA targets through low-precision matrix multiplication.NVIDIA Turing Tensor Cores are specialized execution units for efficient low-precision matrix multiplications.
  • Software: INT8 deployment is broadly supported without accuracy loss, but software for lower precisions is limited; TensorRT lacks sub-INT8 support and TVM only recently added INT4.The survey identifies efficient lower-precision software APIs as an important research need.
  • Coupled compression: Quantization can be combined with pruning and other compression methods, but the best combinations of structured or unstructured pruning and quantization remain a future direction.The survey also identifies hardware and neural-network architecture co-design as a related opportunity.
  • Quantized training: Half-precision quantized training accelerates training, whereas INT8 training often requires extensive tuning or works only on limited models and easier tasks.INT8 training can become unstable and diverge.

VII. SUMMARY AND CONCLUSIONS

Quantization addresses the efficient representation of numerical values in computationally intensive, over-parameterized neural networks, with low-precision inference delivering substantial speedups across hardware. The survey organizes diverse quantization methods and training relationships, emphasizing edge deployment and future evaluation.

  • VII. SUMMARY AND CONCLUSIONS: Low-precision integer representations can reduce neural-network memory footprint and latency, with INT8 speedups reported across major vision models.Reported speedups include 3.89× for ResNet50, 3.32× for VGG-19, and 5.02× for InceptionV3 on NVIDIA GTX 1080.
  • VII. SUMMARY AND CONCLUSIONS: INT4 ResNet50 inference achieved an additional 50–60% speedup over INT8 on NVIDIA T4 and RTX, while mixed precision produced 23% speedup without accuracy degradation.BERT inference with INT8 also reached up to 4.0× faster inference than FP32.
  • VII. SUMMARY AND CONCLUSIONS: INT8 quantization improved latency on non-GPU platforms, achieving 2.35× speedup on Intel Cascade Lake and 1.40× on Raspberry Pi4.These results contributed to sustained research activity in neural-network quantization.
  • VII. SUMMARY AND CONCLUSIONS: The survey structures diverse efforts around general quantization types, neural-network-specific granularity, training relationships, data availability, and zero-shot quantization.It contrasts quantization-aware training with post-training quantization and considers cases where training data are unavailable.
  • VII. SUMMARY AND CONCLUSIONS: Edge-focused coverage includes sub-8-bit parameters and integer-only quantization for low-end microprocessors without floating-point units.These topics address deployment environments with particularly constrained hardware resources.
  • VII. SUMMARY AND CONCLUSIONS: The survey aims to provide a useful snapshot and conceptual organization that eases evaluation of future neural-network quantization research.The broader motivation is efficient numerical representation for computationally intensive, over-parameterized neural networks.
Loading 2103.13630v3…