Data-Free Quantization Through Weight Equalization and Bias Correction

Markus Nagel; Mart van Baalen; Tijmen Blankevoort; Max Welling

Source-linked AI summary

Data-Free Quantization Through Weight Equalization and Bias Correction

Markus Nagel, Mart van Baalen, Tijmen Blankevoort, Max Welling

arXiv:1906.04721v3Jun 11, 2019cs.LGcs.CVstat.ML

TL;DR

INT8 quantization supports efficient inference but can substantially reduce accuracy, particularly when weight ranges differ across channels. DFQ addresses this by equalizing weight ranges and correcting quantization-induced bias without data, fine-tuning, or hyperparameter optimization. It achieves near-original model accuracy across common computer vision architectures and tasks while supporting practical per-tensor quantization.

Problem
INT8 quantization introduces noise that can substantially reduce model performance, while difficult models often require data, fine-tuning, or hardware-limited per-channel quantization.
Method
DFQ rescales pretrained network weights to equalize channel ranges and uses batch-normalization-based estimates to correct quantization-induced output bias without data.
Results
DFQ achieves near-original model accuracy across common computer vision architectures and tasks, including image classification, semantic segmentation, and object detection.
Takeaways & Limitations
DFQ enables practical INT8 deployment through a straightforward API call and can support more efficient per-tensor quantization than per-channel quantization.
Takeaways & Limitations
The data-free estimates assume normally distributed pre-activation outputs and batch normalization before a clipped linear activation such as ReLU or ReLU6.

Abstract

from arXiv · show

We introduce a data-free quantization method for deep neural networks that does not require fine-tuning or hyperparameter selection. It achieves near-original model performance on common computer vision architectures and tasks. 8-bit fixed-point quantization is essential for efficient inference on modern deep learning hardware. However, quantizing models to run in 8-bit is a non-trivial task, frequently leading to either significant performance reduction or engineering time spent on training a network to be amenable to quantization. Our approach relies on equalizing the weight ranges in the network by making use of a scale-equivariance property of activation functions. In addition the method corrects biases in the error that are introduced during quantization. This improves quantization accuracy performance, and can be applied to many common computer vision architectures with a straight forward API call. For common architectures, such as the MobileNet family, we achieve state-of-the-art quantized model performance. We further show that the method also extends to other computer vision architectures and tasks such as semantic segmentation and object detection.

1. Introduction

Fixed-point quantization improves inference efficiency but can substantially reduce model performance, while existing practical methods often require data, backpropagation, fine-tuning, or hyperparameter tuning. The paper introduces a data-free approach that targets near-original INT8 performance through weight adaptation and bias-error correction.

Motivation: Fixed-point quantization approximates floating-point values with integers, a scaling factor, and optionally a zero point for more efficient inference.It enables faster and more power-efficient integer operations, at the expense of lower representational power.
Motivation: INT8 quantization introduces weight and activation noise that can cause performance degradation ranging from minor to catastrophic.The resulting engineering burden is especially relevant for organizations deploying models on quantized hardware.
Practical applicability: Existing quantization approaches commonly rely on data and fine-tuning, with some also requiring hyperparameter tuning or a full training pipeline.The paper categorizes these approaches as methods requiring data with or without backpropagation.
Contribution: The proposed approach requires no data, fine-tuning, or hyperparameter tuning and can improve accuracy through a simple API call.It adapts pretrained weight tensors for quantization and corrects bias in quantization error.
Contribution: The method achieves near-original model performance when quantizing FP32 models to INT8 across computer vision models that are difficult to quantize without fine-tuning.Its practical applicability is contrasted with more involved methods requiring architecture changes or training from scratch.

2. Background and related work

The related-work discussion distinguishes quantization methods by their practical requirements, emphasizing data, backpropagation, model generality, architecture changes, and training overhead. DFQ is positioned as a level 1 method that avoids these requirements while retaining efficient integer inference.

Comparison with related methods: Per-channel quantization improves applicability to difficult models but is not supported on all hardware and adds scale-and-offset overhead for each output channel.The paper states that DFQ improves on per-channel quantization while retaining one scale-and-offset set for the whole tensor.
Comparison with related methods: Other approaches require architecture changes, training with quantization in mind, or training from scratch, making them more involved and adding training overhead or hyperparameters.These approaches are categorized as level 4 methods.

3. Motivation

MobileNetV2 is difficult to quantize because output-channel weight ranges differ sharply and quantization error can be biased. The paper motivates equalizing channel ranges and correcting output bias to improve quantization without changing the full-precision model output.

3.1. Weight tensor channel ranges: 70.9% to 0.1%: MobileNetV2 top-1 ImageNet validation accuracy reportedly drops sharply when quantized without per-channel quantization or fine-tuning.Near-original performance can be restored by per-channel quantization, fine-tuning, or both.
3.1. Weight tensor channel ranges: Shared per-tensor quantization parameters can erase small-range channels when other output channels have much larger weight ranges.For example, weights in the range (−0.5, 0.5) can all quantize to 0 when another channel spans [−128, 128].
3.1. Weight tensor channel ranges: MobileNetV2 contains layers with strongly differing output-channel weight ranges, making several layers and the overall model difficult to quantize accurately.The first inverted residual block’s depthwise-separable layer exhibits this problem.
3.1. Weight tensor channel ranges: Equalizing output-channel ranges is proposed as a way to improve quantization while preserving the FP32 model output.The approach exploits rescaling and reparameterization so each channel can use the available weight range more effectively.
3.2. Biased quantization error: Quantization error on weights can be biased rather than canceling, shifting the next layer’s input distribution and producing unpredictable effects.Depthwise-separable layers are especially susceptible because each output channel has only 9 corresponding weights.
3.2. Biased quantization error: Batch-normalization parameters can support a data-free method for correcting bias in quantization-induced output error.The paper introduces this correction as a level 1 method.

4. Method

DFQ reparameterizes consecutive layers to equalize channel weight ranges, absorbs high biases, and corrects quantization-error bias without data. Its data-free estimates use batch-normalization parameters and activation-distribution assumptions.

4. Method: DFQ consists of three steps added to normal quantization, with the overall procedure represented by a flow diagram.The supplied method passage identifies the three-step structure but does not enumerate all three steps there.
4.1.1. Scaling equivariance in neural networks: ReLU and related piecewise-linear activations provide scaling equivariance that permits reparameterizing consecutive layers without changing their computation.For two layers, the method rescales one layer’s weights by S^-1 and the next layer’s weights by S, while adjusting the first bias.
4.1.2. Equalizing ranges over multiple layers: Channel-range equalization seeks scaling factors that maximize total channel precision by matching corresponding ranges across adjacent weight tensors.The procedure iterates over connected layer pairs without intervening input or output splits until convergence.
4.1.3. Absorbing high biases: High biases are absorbed into the subsequent layer to avoid large differences between per-channel activation ranges.For ReLU, the method chooses c = max(0, β − 3γ) under a Gaussian assumption, preserving the equality for 99.865% of values greater than c.
4.2. Quantization bias correction: Bias correction subtracts the expected quantization error from a layer’s bias, preserving each output unit’s mean without requiring data.The expected error is computed from the expected input, which can be estimated using batch-normalization parameters and clipped-normal activation statistics.

5. Experiments

The experiments evaluate data-free quantization through ablations on MobileNetV2 and extensions across architectures and computer vision tasks. Equalization and bias correction together recover near-full-precision performance while retaining hardware-friendly per-tensor quantization.

5.1.1. Cross-layer equalization: Equalization brings MobileNetV2 INT8 performance within 2% of FP32 performance, close to per-channel quantization.Replacing ReLU6 with ReLU does not significantly degrade performance; combining equalization with high-bias folding improves over per-channel quantization.
5.1.1. Cross-layer equalization: Cross-layer equalization makes most MobileNetV2 output-channel weight ranges similar and equalizes strong outliers.Several near-zero channels remain and can be pruned with hardly any accuracy loss.
5.1.2. Bias correction: Bias correction reduces quantization-error bias to very close to zero for most output channels.Its effectiveness supports biased error as part of the quantization problem, although bias correction alone does not achieve near-floating-point performance.
5.1.2. Bias correction: 0.53% reduction from full precision is achieved by the complete DFQ method on MobileNetV2 ImageNet validation.The method combines cross-layer equalization, bias folding, and bias correction.
5.2.1. Other tasks: Less than 1% drop in mIOU is achieved by DFQ for semantic segmentation, while outperforming per-channel quantization.The experiment uses DeeplabV3+ with a MobileNetV2 backend on Pascal VOC.
5.2.1. Other tasks: Less than 1% drop in mAP is achieved by DFQ for SSDLite object detection, again outperforming per-channel quantization.Quantizing the original model causes a significant performance drop, whereas DFQ recovers almost all of it.
5.2.2. Comparison to other approaches: DFQ keeps MobileNetV1 and MobileNetV2 close to full-precision performance and outperforms per-channel quantization and most level 3 and 4 approaches.Those higher-level approaches require significant fine-tuning, training, or architecture changes.
5.2.2. Comparison to other approaches: DFQ maintains full-precision performance for 8-bit fixed-point quantization on ResNet18.It outperforms traditional per-layer quantization but remains slightly below per-channel quantization and higher-level approaches.

6. Conclusion

DFQ improves INT8 quantization without data, fine-tuning, or hyper-parameter optimization, while supporting practical deployment across common computer vision tasks. It achieves near-original accuracy for almost every tested model and compares favorably with more efficient per-tensor quantization and training-based methods.

DFQ improves quantized model performance without requiring data, fine-tuning, or hyper-parameter optimization.
DFQ applies to image classification, semantic segmentation, and object detection using a straight-forward API call.
DFQ achieves near-original model accuracy for almost every tested model and competes with more complicated training-based methods.
DFQ compares favorably to per-channel quantization, allowing the more efficient per-tensor quantization approach in practice.
Quantization levels organize methods by practical applicability to support fairer comparison and practical deployment.

A. Optimal range equalization of two layers

The range-equalization analysis rescales adjacent weight matrices with reciprocal channel factors to optimize symmetric quantization ranges. The resulting condition matches corresponding channel ranges as closely as possible while avoiding increases in the full-matrix range.

Adjacent weight matrices are rescaled as cW(1) = S^-1W(1) and cW(2) = W(2)S using a positive diagonal matrix S.
The optimization considers symmetric quantization, which the paper reports also gives good results for asymmetric quantization.
Each channel's scaling factor cancels from the objective when it does not increase R, the full weight-matrix range.
The optimal solution is constrained by a limiting channel identified by the largest first-layer range.
Per-channel ranges are matched as closely as possible so quantization error is spread equally across both weight tensors.

B. Bias correction for convolutional layers

For convolutional layers, the method treats quantization error as a constant convolutional contribution under an equal-input-channel expectation assumption. This expected error can then be absorbed into the layer bias, while clipped-normal statistics provide analytic moments for the activation distribution.

Under equal expected values across spatial dimensions, the expected convolutional quantization error is constant across each output channel.
The constant expected error can be folded into the convolutional layer's bias parameter.
For a normally distributed input and clipped-linear function f, the mean and variance of f(X) are computed using standard rules.
Because the clipped-linear function is constant outside [a, b], its moments separate contributions from clipped and linear regions.
The standard normal CDF Φ and PDF φ parameterize the analytic expressions for the clipped distribution's moments.

C.2. Variance of Clipped Normal Distribution

The clipped-normal variance analysis decomposes the distribution into clipped tails and a truncated linear region. It uses truncated-normal quantities to evaluate the linear contribution and reports a MobileNetV2 INT8 validation comparison for bias-correction variants.

Outside [a, b], the clipped function is constant, so the variance calculation separates tail and interior contributions.
The interior contribution uses Z = Φ(β) − Φ(α) and the mean µ_t of the truncated normal distribution.
Table 6 reports MobileNetV2 Top1 ImageNet validation results for INT8 weights and activations across analytic and empirical bias-correction configurations.
The section evaluates the first term of the variance expression after decomposing the clipped and linear contributions.

D. Empirical quantization bias correction

The empirical bias-correction procedure estimates pre-activation mean differences between FP32 and quantized models using representative unlabeled data, then corrects quantized biases. It is applied after BatchNorm folding and cross-layer range equalization, and produces results similar to analytic bias correction.

The procedure subtracts the quantization-induced mean difference from each quantized layer’s bias parameter.This correction uses the difference between the two models’ pre-activation means, represented as E[ϵ].
The correction can use unlabeled representative data and should follow BatchNorm folding and cross-layer range equalization.Clipping is applied in the quantized network but not in the corresponding FP32 procedure.
Empirical correction compares per-channel pre-activation means from FP32 and quantized models on the same examples.The FP32 means are collected first, followed by quantized-model means for each layer.

E. Additional experiments

Additional experiments show that DFQ can precede short quantization-aware fine-tuning, performs similarly with symmetric and asymmetric quantization, and improves per-channel quantization when its components are combined.

Combination with fine-tuning: One epoch of quantization-aware fine-tuning after DFQ raises MobileNetV2 accuracy from 71.19% to 71.42%, nearing the 71.72% FP32 result.
Symmetric vs asymmetric quantization: Symmetric quantization’s advantage over asymmetric quantization is almost negligible for all three models when combined with DFQ.Cross-layer equalization removes outliers, often making weight distributions close to symmetric.
DFQ combined with per-channel quantization: Per-channel quantization experiments evaluate DFQ with cross-layer equalization, bias absorption, and bias correction while activations remain per-tensor quantized.The comparison concerns hardware that supports per-channel quantization efficiently.
DFQ combined with per-channel quantization: 0.39% total quantization error results when DFQ components reduce per-channel quantization error from 1.07%.Each individual component incrementally improves per-channel quantization in the reported MobileNetV2 evaluation.