Source-linked AI summary
A White Paper on Neural Network Quantization
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, Tijmen Blankevoort
TL;DR
Neural-network inference is costly for latency- and power-constrained devices, while quantization can introduce accuracy loss. The paper presents PTQ and QAT pipelines for mitigating quantization noise, achieving near-floating-point 8-bit PTQ and competitive 4-bit QAT results.
Problem
Neural networks require reduced inference power and latency for edge devices, but lower-bit quantization can degrade accuracy through added noise.
Method
The paper develops hardware-motivated PTQ and QAT pipelines using state-of-the-art algorithms and extensive experimentation.
Results
8-bit weight-and-activation PTQ stays within 1% of floating-point accuracy across all evaluated networks, while QAT achieves 4-bit weights and, for some models, 4-bit activations with only a small accuracy drop.
Takeaways & Limitations
PTQ provides a lightweight route to near-floating-point 8-bit quantization, while QAT supports more aggressive low-bit quantization when retraining is available.
Takeaways & Limitations
Group-wise quantization methods are not considered because most existing fixed-point accelerators lack support for accumulators handling varying scale factors.
Abstract
from arXiv · showhide
While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.
1 Introduction
Neural networks increasingly run on devices constrained by latency, power, and computation. This paper presents PTQ and QAT pipelines to reduce these costs while limiting quantization-related accuracy loss.
- Motivation: Edge devices require neural networks with low latency, small resource demands, and energy efficiency.The paper cites smartphones, smart glasses, appliances, drones, robots, and self-driving cars as examples of affected devices.
- Motivation: Quantization stores weights and activation tensors at lower bit precision than the 16- or 32-bit precision commonly used during training.
- Motivation: Moving from 32 to 8 bits reduces tensor memory overhead by 4× and matrix-multiplication computational cost by 16×.
- Approaches: PTQ quantizes trained networks with little or no data, minimal tuning, and no end-to-end training, whereas QAT retrains with simulated quantization.QAT requires more training effort but generally closes the gap to full-precision accuracy more effectively for low-bit quantization.
- Contribution: The paper provides literature-based pipelines, extensive experiments, and a debugging workflow targeting state-of-the-art performance across common vision and language models.
2 Quantization fundamentals
This section introduces neural-network quantization and the fixed-point accelerators on which quantized networks execute. It connects hardware motivation, quantization schemes, and practical layer considerations.
- The paper introduces neural-network quantization fundamentals alongside the fixed-point accelerators used to run quantized networks.
- The discussion proceeds from hardware motivation to standard quantization schemes and their properties.
- It also addresses practical considerations for common modern-network layers and their implications for fixed-point accelerators.
2.1 Hardware background
Quantized accelerator inference uses parallel processing elements and accumulators to implement matrix-vector multiplication efficiently. Quantization replaces floating-point data with integer representations while retaining wider accumulators to avoid overflow.
- Accelerator operation: A neural-network accelerator computes y = Wx + b using processing elements for products and accumulators for summation.The toy example contains 16 processing elements arranged in a square grid and 4 accumulators.
- Accelerator operation: The accelerator repeats multiply-accumulate operations across cycles, then returns accumulated values to memory for the next layer.The processing elements compute W_n,m x_m in a single cycle before the results are added in the accumulators.
- Fixed-point motivation: Floating-point inference would require floating-point logic and 32-bit transfers between memory and processing units.The paper motivates fixed-point computation because digital arithmetic costs scale linearly to quadratically with bit count and fixed-point addition is more efficient than floating-point addition.
- Quantized computation: Quantized vectors represent floating-point values approximately as a scalar times integer values, with separate weight and activation scale factors.Factoring these scale factors outside the summation enables fixed-point MAC operations.
- Quantized computation: INT8 accelerator arithmetic retains typically 32-bit accumulators to prevent overflow during accumulation, then requantizes activations back to INT8 before the next layer.
2.2 Uniform affine quantization
Uniform quantization maps floating-point values onto an integer grid using scale, zero-point, and bit-width parameters. The paper discusses affine and symmetric variants, error trade-offs, and tensor-versus-channel granularity.
- Uniform quantization: Uniform quantization is used because it supports efficient fixed-point arithmetic implementation.
- Uniform affine quantization: Uniform affine quantization uses scale factor s, zero-point z, and bit-width b to map floating-point values onto an integer grid.The scale factor sets the quantizer step size, while the zero-point ensures real zero is represented without error.
- Quantization errors: Values outside the quantization range are clipped, while increasing the scale factor reduces clipping error but increases rounding error.The paper examines range-setting methods to balance these two errors.
- Symmetric quantization: Symmetric quantization fixes the zero-point at 0, reducing accumulation overhead but restricting the floating-point-to-integer mapping.Unsigned symmetric quantization suits one-tailed distributions such as ReLU activations, whereas signed symmetric quantization suits distributions roughly symmetric around zero.
- Symmetric quantization: Power-of-two quantization restricts s to 2^-k, enabling bit-shifting but complicating the rounding-versus-clipping trade-off.
- Quantization granularity: Per-tensor quantization uses one quantizer per tensor for simpler hardware, while per-channel and finer group granularity can improve accuracy with additional overhead.The paper excludes group-wise methods because most existing fixed-point accelerators do not support their varying-scale accumulator logic.
2.3 Quantization simulation
Quantization simulation inserts quantizer blocks into floating-point compute graphs to approximate fixed-point inference, enabling practical testing without quantized hardware. The section also describes hardware-aware handling of nonlinearities and common layers.
- Quantized forward pass: Figure 4 contrasts actual on-device fixed-point inference with floating-point simulation using quantizer blocks around convolutional operations.The simulation quantizes weights before convolution and activations after the activation function; biases are often retained in higher precision.
- Quantization simulation: Quantization simulation approximates fixed-point operations on general-purpose floating-point hardware, making quantization options easier to test.It also enables GPU-accelerated quantization-aware training.
- Batch normalization folding: Batch normalization folding fuses batch-normalization calculations into adjacent linear layers, removing batch-normalization operations from the inference network.The folding uses training-derived mean and variance together with learned per-channel scale and offset parameters.
- Activation function fusing: Hardware units may apply nonlinearities before requantization, avoiding an intermediate memory write and reload.ReLU can be represented by setting the minimum activation value to 0.
- Activation function fusing: Unsupported sigmoid or Swish functions may require quantization before and after the nonlinearity, potentially harming accuracy and deployment efficiency.Floating-point accuracy improvements from newer activations may disappear after quantization.
- Common layers: Element-wise addition requires matching input quantization ranges, while concatenated branches may require requantization or shared quantization parameters.Average pooling requires a quantization step because averages of integers are not necessarily integers; max pooling does not require activation quantization here.
2.4 Practical considerations
Practical quantization choices span bit-width, scheme, and granularity, with this white paper focusing on homogeneous bit-width. Asymmetric quantization adds expressiveness but can create data-dependent inference overhead, while per-channel weight quantization can improve accuracy.
- Design choices: Quantizing multilayer networks requires choices about quantization scheme, granularity, and bit-width, creating a large search space.The section presents practical considerations intended to reduce that space.
- Bit-width: The white paper considers homogeneous bit-width, keeping weight or activation precision constant across layers.Homogeneous bit-width is more widely supported by hardware than heterogeneous or mixed precision.
- Symmetric vs. asymmetric quantization: Asymmetric quantization is more expressive because it includes an offset parameter, but that flexibility can add computational overhead.The overhead arises from an additional input-dependent term during inference.
- Symmetric vs. asymmetric quantization: A common compromise uses asymmetric activation quantization with symmetric weight quantization to avoid the additional data-dependent term.This retains asymmetric activations while removing the corresponding weight offset from the extra computation.
- Granularity: Per-channel weight quantization can improve accuracy when weight distributions vary substantially between channels.Per-tensor quantization remains broadly supported by fixed-point accelerators.
3 Post-training quantization
Post-training quantization converts pretrained FP32 networks without the original training pipeline, but requires careful range setting and targeted corrections to control quantization error. The paper presents a best-practice PTQ pipeline that achieves competitive results across models and tasks, including near-floating-point performance in many 8-bit and selected 4-bit settings.
- PTQ overview: PTQ converts a pretrained FP32 network directly to fixed-point without requiring the original training pipeline.The approach is intended as a lightweight quantization regime with little or no data and no end-to-end training.
- Quantization range setting: Range setting balances clipping and rounding error, with methods typically optimizing local costs rather than task loss.The selected clipping thresholds define qmin and qmax for each quantizer.
- Quantization range setting: MSE outperforms min-max at lower bit-widths for weights, while MSE + Xent performs especially well for activations.Cross-entropy is particularly useful for preserving the ordering of large logits in the final classification layer.
- Cross-Layer Equalization: Depth-wise separable layers can contain large within-tensor magnitude differences, making a suitable clipping-versus-rounding trade-off difficult even for INT8.This issue arises because only a few weights may determine each output feature.
- Cross-Layer Equalization: CLE brings MobileNetV2 INT8 performance within 2% of FP32, while CLE with bias absorption and per-tensor quantization outperforms per-channel quantization.Bias absorption slightly reduces FP32 performance but improves quantized performance by 1% through more precise activation quantization.
- Weight rounding: AdaRound optimizes weight rounding using a second-order approximation, requiring only a small amount of unlabeled data and no hyperparameter tuning or end-to-end fine-tuning.The approach applies to fully connected and convolutional layers.
- Standard PTQ pipeline: The standard PTQ pipeline keeps W8A8 accuracy within 0.7% of floating-point across all evaluated models, while W4A8 degradation depends strongly on architecture and granularity.For EfficientNet lite, per-channel quantization improves accuracy by 2.8% over per-tensor quantization, reaching within 1.4% of full precision.
4 Quantization-aware training
QAT addresses the limitations of PTQ at low bit-widths by simulating quantization during training and using a standardized pipeline. The reported results show near-floating-point performance for several low-bit settings, while depth-wise separable networks remain more difficult.
- Motivation: PTQ is fast and requires no retraining with labeled data, but may fail to mitigate quantization error for low-bit activations such as 4-bit and below.QAT is introduced for cases where PTQ is insufficient.
- Method: QAT simulates quantization during training and uses a best-practice pipeline built from literature and extensive experimentation.The pipeline starts from a pretrained model and applies PTQ steps to improve convergence and accuracy.
- Backward path: The straight-through estimator approximates the rounding operator’s gradient as 1, making gradient-based training through simulated quantization possible.The exact round-to-nearest gradient is zero or undefined everywhere.
- Batch normalization: Static batch-normalization folding is simpler and, in reported experiments, performs on par with or better than a more computationally costly alternative for per-channel quantization.The alternative uses updated running statistics and a double forward pass.
- Initialization: Better MSE-based initialization helps early QAT training, but its advantage usually becomes small or disappears as training continues.This pattern is reported for 4-bit activation quantization and is also observed for weight-range initialization.
5 Summary and Conclusions
The paper presents quantization as a practical route to lower-power and lower-latency inference, organizing its solutions around PTQ and QAT. It reports lightweight 8-bit PTQ near floating-point accuracy and more aggressive low-bit results with QAT, while recommending the choice based on application requirements.
- Motivation: Quantization targets the power and latency constraints of neural-network inference on devices and services with limited resources.The paper motivates quantization for applications ranging from smartphones and appliances to autonomous systems.
- Approaches: The paper studies Post-Training Quantization and Quantization-Aware Training as its two main classes of quantization algorithms.PTQ converts pretrained FP32 networks without the original training pipeline, while QAT models quantization noise during training.
- PTQ results: The proposed PTQ pipeline achieves 8-bit weight-and-activation quantization within 1% of floating-point accuracy across all evaluated networks.Many networks can also use 4-bit weights with only a small additional performance drop.
- QAT results: The proposed QAT pipeline achieves 4-bit weight quantization, and for some models 4-bit activations, with only a small accuracy drop relative to floating point.QAT also includes attention to batch-normalization folding and a standard training pipeline.
- Practical conclusion: PTQ and QAT should be selected according to the application’s accuracy and power requirements.The paper presents both approaches as components of a model-efficiency toolkit for deploying quantized models.