Source-linked AI summary
LSQ+: Improving low-bit quantization through learnable offsets and better initialization
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, Nojun Kwak
TL;DR
Low-bit quantization struggles with negative, skewed activations and unstable training initialization. LSQ+ addresses these issues with learnable asymmetric quantization and MSE-based parameter initialization, improving low-bit results for modern efficient architectures. The method reports accuracy gains over LSQ on EfficientNet-B0 and lower performance variance across runs.
Problem
Unsigned activation quantization discards negative values, while signed quantization requires an extra sign bit; learnable quantization is also sensitive to initialization.
Method
LSQ+ extends LSQ with trainable scale and offset parameters and uses MSE-based initialization for quantization parameters.
Results
LSQ+ reports state-of-the-art results for EfficientNet and MixNet, including a 1.8% accuracy improvement over LSQ with W4A4 and upto 5.6% with W2A2 on EfficientNet-B0.
Takeaways & Limitations
The method enables extremely low-bit quantization of modern efficient architectures with negative activations without using an extra sign bit.
Takeaways & Limitations
The reported experiments use pretrained floating-point weights and the proposed scale/offset initialization in the main experimental sections.
Abstract
from arXiv · showhide
Unlike ReLU, newer activation functions (like Swish, H-swish, Mish) that are frequently employed in popular efficient architectures can also result in negative activation values, with skewed positive and negative ranges. Typical learnable quantization schemes [PACT, LSQ] assume unsigned quantization for activations and quantize all negative activations to zero which leads to significant loss in performance. Naively using signed quantization to accommodate these negative values requires an extra sign bit which is expensive for low-bit (2-, 3-, 4-bit) quantization. To solve this problem, we propose LSQ+, a natural extension of LSQ, wherein we introduce a general asymmetric quantization scheme with trainable scale and offset parameters that can learn to accommodate the negative activations. Gradient-based learnable quantization schemes also commonly suffer from high instability or variance in the final training performance, hence requiring a great deal of hyper-parameter tuning to reach a satisfactory performance. LSQ+ alleviates this problem by using an MSE-based initialization scheme for the quantization parameters. We show that this initialization leads to significantly lower variance in final performance across multiple training runs. Overall, LSQ+ shows state-of-the-art results for EfficientNet and MixNet and also significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations (e.g.: 1.8% gain with W4A4 quantization and upto 5.6% gain with W2A2 quantization of EfficientNet-B0 on ImageNet dataset). To the best of our knowledge, ours is the first work to quantize such architectures to extremely low bit-widths.
1. Introduction
LSQ+ targets low-bit quantization of modern networks whose negative, skewed activations are mishandled by unsigned activation quantization. It adds learnable asymmetry and MSE-based initialization to recover accuracy and improve training stability.
- Motivation: Swish-like activations contain negative values, but PACT and LSQ quantize them to zero under unsigned activation ranges, losing information.This is less problematic for ReLU-based networks but harms architectures such as EfficientNet and MixNet.
- Method: LSQ+ introduces learnable scale and offset parameters in a general asymmetric quantization scheme for low-bit activations.The offset accommodates negative activations without requiring the extra sign bit associated with naive signed quantization.
- Results: 1.8% accuracy improvement over LSQ is reported with W4A4 quantization, rising to upto 5.6% with W2A2 on EfficientNet-B0.These gains are reported for low-bit quantization of networks with Swish activations.
- Training stability: MSE-based initialization for the offset and scale parameters reduces variance in final performance across training runs compared with min-max initialization.Gradient-based learnable quantization is sensitive to initialization, particularly in the low-bit regime.
- Contribution: LSQ+ extends LSQ with a learnable activation offset and emphasizes proper initialization for stable low-bit training.The paper reports state-of-the-art results for EfficientNet and MixNet and presents the method as the first to quantize these architectures to extremely low bit-widths.
2. Related Work
Related work includes post-training and quantization-aware training methods, with learned quantization parameters improving low-bit quantization. LSQ+ builds on this training-based line while remaining compatible with other quantization techniques.
- Quantization methods: Quantization methods are commonly divided into post-training methods and quantization-aware training methods requiring fine-tuning.Post-training methods optimize quantized networks with limited or no data, whereas quantization-aware training simulates quantization during training.
- Post-training quantization: Post-training methods that work well for typical 8-bit quantization have not achieved good accuracy on very low-bit 2-, 3-, and 4-bit quantization.This motivates training-based approaches for low-bit tasks.
- Quantization-aware training: Quantization-aware training generally outperforms post-training methods on low-bit tasks given enough optimization time.Recent methods learn quantization parameters, including QIL, TQT, and LSQ, which is the line of work LSQ+ extends.
- Parallel approaches: Knowledge distillation, automatic bit-width learning, and alternative quantization grids represent parallel approaches to improving quantization.The paper states that LSQ+ is orthogonal to these works and can be jointly used with them.
3. Method
LSQ+ extends LSQ with learnable asymmetric activation quantization and MSE-based initialization for scale and offset parameters. The method addresses skewed activation ranges and initialization sensitivity while retaining symmetric signed quantization for weights.
- 3.1. Learnable asymmetric quantization: LSQ uses symmetric quantization with a trainable scale, but unsigned ranges discard negative activations and signed ranges waste precision on skewed distributions.For Swish, the negative range is limited while the positive range is unbounded, making both conventional choices suboptimal.
- 3.1. Learnable asymmetric quantization: LSQ+ learns both scale and offset parameters for asymmetric activation quantization, allowing different layers to accommodate negative activations.Its parameterizations include signed and unsigned ranges; configurations without offsets correspond to LSQ-style symmetric quantization.
- 3.1. Learnable asymmetric quantization: The activation quantizer uses a coded value obtained by shifting and scaling the input, then reconstructs the quantized value by multiplying by scale and adding the offset.Straight-through estimation is used for gradients of the quantization parameters, and the offset can be folded into the bias at compilation time.
- 3.2. Initialization of quantization parameters: LSQ+ uses an initialization scheme for scale and offset designed to improve stability in extremely low-bit quantization, where performance is sensitive to initialization.This sensitivity is amplified by depthwise separable convolutions; the proposed scheme is reported to be more stable and sometimes better than prior initializations.
- 3.2.1 Scale initialization for weight quantization: For weights, LSQ+ uses symmetric signed quantization and initializes scale from layer weight-distribution statistics, addressing LSQ initializations observed to be far from converged values.The initialization uses a Gaussian approximation based on the layer mean and standard deviation rather than the actual weight values.
- 3.2.2 Scale/offset initialization for activation quantization: For activations, LSQ+ initializes scale and offset by minimizing quantization MSE rather than relying directly on dynamic extrema, which are prone to outliers.The parameters are optimized over a few data batches because the MSE objective has no closed-form solution.
4. Experiments
Experiments evaluate LSQ+ on Swish-based EfficientNet-B0 and MixNet-S at low bit-widths, compare configurations on ReLU-based ResNet18, and assess initialization stability. Learnable offsets improve low-bit performance for Swish activations, while MSE-based initialization reduces training variance.
- 4. Experiments: LSQ+ is evaluated on ImageNet with EfficientNet-B0 and MixNet-S using W2A2, W3A3, and W4A4 quantization.The experiments use the proposed initialization for the main architecture comparisons.
- 4.1. Results on Swish activation: 1.6-1.8% and 1.2-1.3% performance improvements are observed for W4A4 quantization on EfficientNet-B0 and MixNet-S, respectively, with learnable offsets over LSQ.The gain is most prominent for W2A2 quantization.
- 4.1. Results on Swish activation: Signed and unsigned quantization with learnable offsets achieve almost similar performance across bit-widths because the learned offset maps activation ranges appropriately.This mapping makes the choice between signed and unsigned ranges less consequential in the tested configurations.
- 4.1. Results on Swish activation: Configuration 2 consistently performs worse because it lacks an offset and uses only 2^b−1 levels for the positive activation range.Configurations 3 and 4 allocate 2^b levels across the entire activation range, better matching Swish’s larger positive portion.
- 4.2. Results on ReLU activation: On ResNet18, LSQ reaches 70.7% accuracy with W4A4 quantization versus 70.1% full-precision accuracy, and Configurations 1, 3, and 4 outperform PACT, DSQ, and QIL.The comparison uses standard ResNets rather than the higher-accuracy pre-activation ResNet used in the original LSQ results.
- 4.3. Effect of quantization parameter initialization: MSE-based initialization produces more stable final performance across five EfficientNet-B0 training runs than min-max and LSQ initialization.LSQ initialization has higher variation because its initial scale is far from the converged value.
5. Discussion
LSQ+ learns layer-specific offsets that accommodate negative activations while balancing represented range against quantization resolution. The discussion also compares learned offsets with fixed offsets and examines performance variation across initialization methods.
- 5.1. Learned offset values: Layerwise β values are shown after convergence for EfficientNet-B0.
- 5.1. Learned offset values: Most learned β values are negative, shifting activations upward before scaling and clamping them to the quantization range.None of the learned β values are below the Swish minimum.
- 5. Discussion: Table 4 compares all quantization configurations for ResNet18 against 70.1% full-precision accuracy.
- 5. Discussion: Table 5 reports accuracy variation around the mean across five EfficientNet training runs with different initialization methods.These values differ from best accuracies reported elsewhere after hyperparameter grid search.
- 5.1. Learned offset values: The offset gradient becomes zero when β falls below the minimum activation value.
- 5.2. Learned vs Fixed offset: Learned offsets generally differ from the Swish minimum because covering the full activation range with fixed bits produces a coarser grid and higher quantization error.Learning s and β balances grid resolution against the proportion of activation range represented.
6. Conclusion
The work targets low-bit quantization by addressing signed activations and training-performance stability. It proposes asymmetric activation quantization with trainable scale and offset parameters to accommodate negative activations without an extra sign bit.
- The method addresses low-bit quantization of deep neural networks with signed activation functions and training-performance stability.
- LSQ+ uses asymmetric quantization with trainable scale and offset parameters to accommodate negative activations without an extra sign bit.
- The paper reports that asymmetric activation quantization incurs zero runtime overhead.