Source-linked AI summary
Quantizing deep convolutional networks for efficient inference: A whitepaper
Raghuraman Krishnamoorthi
TL;DR
Edge inference requires models that use less compute, memory, power, and communication. The paper reviews quantization methods, training procedures, and implementation considerations for integer CNN weights and activations. Its reported results include 2x-3x CPU speedups, close to 10x speedups on specialized processors, 4x model-size reduction, and improved low-bit accuracy with quantization-aware training.
Problem
Edge devices have constrained compute, memory, and power, motivating techniques that reduce model size, inference time, and energy use.
Method
The paper reviews uniform affine quantization, per-layer and per-channel quantization, quantized convolution implementation, and quantization-aware training for CNN inference.
Results
2x-3x CPU speedups and close to 10x speedups on specialized processors are reported for 8-bit quantized inference, while quantization-aware training narrows the accuracy gap at 4-bit precision.
Takeaways & Limitations
A 4x model-size reduction with no accuracy loss is reported for uniform quantization, and quantization-aware training substantially improves low-precision model accuracy.
Takeaways & Limitations
Naive zero-point handling can reduce convolution throughput by 2x to 4x, requiring optimized convolution kernels; exponential moving averages of weights may also under-perform instantaneous estimates.
Abstract
from arXiv · showhide
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.
1 Introduction
Quantization is presented as a broadly applicable way to optimize edge-deployed deep networks for smaller models, faster inference, and lower power consumption. Reducing precision can improve efficiency without requiring new architectures or, in some cases, retraining.
- Edge devices constrain compute, memory, power, and cloud communication, creating a need for smaller and faster models.
- Quantization reduces weight and activation precision while remaining broadly applicable to existing floating-point models and hardware platforms.
- 8-bit quantization reduces model size by a factor of 4 with negligible accuracy loss and requires no data when only weights are quantized.
- Lower-precision activations reduce working memory and improve cache reuse during intermediate computations.
- 8-bit processing and reduced data movement support faster computation and lower power consumption.Moving 8-bit data is 4 times more efficient than moving 32-bit floating-point data.
- 2-3x typical speedups result from reduced-precision memory accesses and computations, with further gains possible on specialized hardware.
2 Quantizer Design
The section presents uniform, stochastic, and simulated quantization designs, emphasizing how quantizer parameters, granularity, and backward-pass approximations affect efficient convolutional inference and training.
- Uniform Affine Quantizer: For 8-bit quantization, scale specifies the quantizer step size and zero-point maps floating-point zero to an integer without error.This preserves zero padding without introducing quantization error.
- Uniform Affine Quantizer: Naive zero-point handling in convolution reduces throughput by 2x to 4x because it requires wider 16/32-bit operands.Optimized kernels can exploit constant weights and shared activation sums to reduce the additional computation.
- Stochastic quantizer: Stochastic quantization models quantization as additive noise followed by rounding, but it is not considered for inference because most hardware does not support it.In expectation, its de-quantized output passes through floating-point weights, with saturation outside the representable range.
- Modeling simulated quantization in the backward pass: Quantization-aware training uses simulated quantizer-dequantizer operations and a straight-through derivative approximation because the exact derivative is zero almost everywhere.The backward-pass error is propagated through the simulated quantizer using the specified approximation.
- Quantizer parameter selection: Post-training quantizer parameters use actual weight extrema and moving averages of activation extrema, while per-layer quantization applies one scale and zero-point to an entire tensor.Per-channel quantization instead adapts parameters to individual convolutional kernels.
- Granularity of quantization: Per-channel weight quantization assigns parameters to each convolutional kernel, while activations remain per-layer quantized for efficient inner-product computation.Per-channel quantization improves accuracy by adapting to kernel-specific ranges without complicating activation computations.
3 Quantized Inference: Performance and Accuracy
This section compares multiple approaches for model quantization and evaluates their performance impact.
- The paper discusses multiple model-quantization approaches and shows their performance impact.
3.1 Post Training Quantization
Post-training quantization reduces model size and supports efficient inference while retaining accuracy across diverse convolutional networks. The results favor per-channel weight quantization with per-layer activation quantization, especially using asymmetric ranges.
- Per-channel quantization with asymmetric ranges produces accuracies close to floating point across a wide range of networks.
- Weight-only quantization: 8-bit weight-only quantization reduces model size by a factor of 4 without validation data, but floating-point inference remains necessary.
- Weight and activation quantization: Activation quantization requires calibration data; about 100 mini-batches typically suffice for activation-range estimates to converge.
- Weight-only quantization: Per-channel quantization is required for small accuracy loss in weight-only quantization, with asymmetric per-layer quantization providing the best accuracy.
- Weight and activation quantization: Per-channel weights and per-layer activations work well across the evaluated networks, with asymmetric quantization slightly improving accuracy.
- Weight and activation quantization: 8-bit activation quantization causes almost no accuracy loss, partly because batch normalization and bounded activations reduce dynamic-range variation.
- Experimental observations: Networks with more parameters are more robust to quantization than smaller MobileNet architectures.
- Experimental observations: Layer-granularity weight quantization causes large accuracy drops, particularly for MobileNets, largely because batch normalization creates kernel-scale range variation.
3.2 Quantization Aware Training
Quantization-aware training simulates quantization during training and inference, while preserving floating-point weights for updates. The section presents graph-rewriting, batch-normalization, and quantizer-design practices that improve low-precision accuracy.
- Quantization-aware training models quantization during training and can provide higher accuracies than post-training quantization.
- Simulated quantized weights and activations are used in forward and backward passes, while floating-point weights receive gradient updates.The updated weights are quantized for subsequent computation, avoiding underflow from minor gradient updates.
- Automatic TensorFlow graph rewriting inserts fake-quantization operations during training and evaluation, with fine-tuning from a floating-point model recommended.The workflow adds quantization operations, trains scale and zero-point information, converts the model, and executes it with TensorFlow Lite.
- Quantization-aware training must model rescaling for addition and concatenation and preserve inference-time operation fusion.For an add followed by ReLU, fake quantization should not be inserted between the operations when they are fused at inference.
- Batch-normalization corrections and freezing reduce quantized-weight jitter and improve accuracy by aligning training behavior with inference statistics.Batch statistics vary between batches, whereas inference uses long-term statistics; correction and freezing address this mismatch.
- Training closes the gap between symmetric and asymmetric quantization and enables per-layer quantization to achieve accuracy close to floating point.Per-channel quantization remains significantly better than per-layer quantization at 4 bits, while fine-tuning 4-bit weights can approach 8-bit accuracy within 5% for most networks.
4 Training best practices
The section evaluates training choices for quantized models, including stochastic versus deterministic quantization, initialization from floating-point checkpoints, batch normalization, and weight averaging. It identifies practices associated with better accuracy and stability.
- Stochastic quantization underperforms deterministic quantization because training-time variation mismatches deterministic inference.
- Fine-tuning from a floating-point checkpoint provides better accuracy than training a quantized model from scratch.
- Batch-normalization corrections stabilize quantized training, while freezing moving statistics provides an additional accuracy gain.For MobileNet-v2, the evaluation-accuracy jitter drops significantly after moving averages are frozen at 400000 steps.
- Exponential moving averaging of weights requires caution because quantized weights can change substantially near quantization decision boundaries.
5 Model Architecture Recommendations
The architecture recommendations examine activation ranges and the tradeoff between network width and quantization precision. They favor unconstrained ReLU activations and greater width when using lower-precision weights.
- Replacing ReLU6 with ReLU and learning activation ranges can slightly improve accuracy for floating-point and quantized MobileNet-v1 networks.
- Batch-normalization corrections and freezing improve MobileNet accuracy by reducing jitter during quantized training.
6 Run-time measurements
Runtime measurements compare floating-point and quantized inference on a Google Pixel 2 CPU and Qualcomm DSPs. Quantization provides substantial speedups, especially on hardware optimized for low-precision arithmetic.
- 2x to 3x speedup is observed for quantized inference versus floating point on a Pixel 2 large CPU core, with almost 10x on Qualcomm DSPs.Measurements are reported in milliseconds for the CPU and Android NN-API DSP execution.
7 Neural network accelerator recommendations
The paper recommends accelerator support for aggressive fusion, compressed memory access, flexible low-precision arithmetic, per-layer bitwidth selection, and per-channel weight quantization.
- Aggressive operator fusion can reduce memory-access costs and improve runtime and power consumption by performing more operations in one pass.
- Lower-precision storage and on-the-fly decompression can optimize memory bandwidth for weights and activations.
- Accelerators should support 4-, 8-, and 16-bit arithmetic for weights and activations.The paper states that 4- and 8-bit precision suffices for classification, while higher precision may be needed for regression applications.
- Per-layer bitwidth selection can reduce model size and processing time by allowing many layers to operate at lower precision.
- Per-channel weight quantization is considered critical for easier hardware deployment and lower-precision computation.
8 Conclusions and further work
The conclusions recommend practical quantization workflows, report speed and compression benefits, and identify training and architecture choices that affect accuracy and future improvements.
- Symmetric per-channel weight quantization with post-training quantization is recommended as a starting point, with optional fine-tuning after accuracy drops.
- 2x-3x CPU speedup and close to 10x specialized-processor speedup are reported for 8-bit quantized inference versus floating point.
- 4x model-size reduction with no accuracy loss is reported for uniform quantization, while non-uniform methods can provide higher compression.
- Quantization-aware training models quantized weights and activations during training and can substantially improve quantized-model accuracy.
- Exponential moving averages of weights may underperform instantaneous estimates during quantization-aware training and require caution.
- Larger models tolerate quantization error better, and additional feature maps can support lower-bitwidth kernels within one architecture.
- Leaving activation ranges unconstrained during training and using ReLU rather than ReLU6 improved accuracy in the reported experiments.
- Regularization, distilled training, and per-layer weight-and-activation quantization are proposed as future improvement areas.
A Impact of Batch Normalization on Quantization
The batch-normalization analysis evaluates SQNR and weight-power distributions, finding advantages for per-channel quantization and harmful outliers after folding.
- Per-channel quantization provides significantly higher SQNR than per-layer quantization, even with symmetric per-channel quantization.
- SQNR histograms count kernels by SQNR bin for depthwise and pointwise MobileNet layers, with 8 and 128 total kernels respectively.
- After batch-normalization folding, weight distributions contain much larger outliers that severely degrade performance.
- The folded-weight histogram for MobileNet V1 1 224 Conv2d 2 depthwise highlights long distribution tails.