Source-linked AI summary
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, Gang Hua
TL;DR
Quantized DNNs can reduce model cost but may lose prediction accuracy, while fixed quantizers may not suit differing weight and activation distributions. LQ-Nets jointly train bit-operation-compatible quantizers with the network, achieving strong accuracy across datasets and architectures compared with prior quantization methods.
Problem
Quantized DNNs face a prediction-accuracy gap, while fixed handcrafted quantizers may not adapt to differing weight and activation distributions across networks and layers.
Method
LQ-Nets jointly train a quantized DNN and learnable bit-operation-compatible quantizers for weights and activations, supporting arbitrary bit-widths and backpropagation.
Results
Across CIFAR-10 and ImageNet, LQ-Nets performed well across AlexNet, VGG-Net, GoogLeNet, ResNet, and DenseNet, surpassing previous quantization methods.
Takeaways & Limitations
Learnable quantizers provide a flexible quantization approach that preserves bitwise-operation compatibility and supports fast inference for low-precision DNNs.
Takeaways & Limitations
A naive strategy for learning quantization functions would make them incompatible with bitwise operations, undermining the fast-inference benefit of quantized neural networks.
Abstract
from arXiv · showhide
Although weight and activation quantization is an effective approach for Deep Neural Network (DNN) compression and has a lot of potentials to increase inference speed leveraging bit-operations, there is still a noticeable gap in terms of prediction accuracy between the quantized model and the full-precision model. To address this gap, we propose to jointly train a quantized, bit-operation-compatible DNN and its associated quantizers, as opposed to using fixed, handcrafted quantization schemes such as uniform or logarithmic quantization. Our method for learning the quantizers applies to both network weights and activations with arbitrary-bit precision, and our quantizers are easy to train. The comprehensive experiments on CIFAR-10 and ImageNet datasets show that our method works consistently well for various network structures such as AlexNet, VGG-Net, GoogLeNet, ResNet, and DenseNet, surpassing previous quantization methods in terms of accuracy by an appreciable margin. Code available at https://github.com/Microsoft/LQ-Nets
1 Introduction
DNN quantization reduces model complexity but retains a substantial accuracy gap at very low bit-widths. LQ-Nets address this gap by jointly learning bit-operation-compatible quantizers and demonstrate strong performance across datasets and network structures.
- Motivation: Quantization represents network weights with very low precision, producing compact DNN models compared with floating-point counterparts.
- Motivation: Quantized DNNs remain substantially less accurate than full-precision models, particularly at 1- or 2-bit precision.A 1-bit-weight, 2-bit-activation 50-layer ResNet achieves 64.6% top-1 accuracy versus 75.3% for its full-precision reference.
- Motivation: Hand-crafted or fixed quantizers may not suit the differing weight and activation distributions across networks and layers.The paper motivates adaptive quantizers because uniform and logarithmic schemes may be suboptimal for these distributions.
- LQ-Nets: LQ-Nets jointly train a quantized DNN with learnable quantizers that support bitwise operations, weights and activations, arbitrary bit-widths, and layer-wise parameters.The quantizers are optimized through standard backpropagation, with an additional quantization-error-minimization algorithm.
- Evaluation: Experiments on CIFAR-10 and ImageNet show strong performance across AlexNet, VGG-Net, GoogLeNet, ResNet, and DenseNet, surpassing previous quantization methods.
2 Related Work
Prior work reduces DNN size and computation through compact architectures, parameter reduction, and network quantization. LQ-Nets differ from most quantization methods by adaptively learning quantizers for both weights and activations at arbitrary bit-widths.
- Compact network design: Compact network design reduces inference cost through architectural choices such as 1×1 convolutions, bottlenecks, depth-wise convolution, and group convolution.
- Network parameter reduction: Network parameter reduction uses methods including low-rank approximations, connection pruning, and hashing to reduce existing model parameters.
- Network quantization: Network quantization trains low-precision DNNs by quantizing weights only or both weights and activations.
- Network quantization: Prior quantization methods include binary, uniform, logarithmic, fixed-distribution, and residual multi-bit schemes.
- LQ-Nets: LQ-Nets adaptively learn quantizers during training for both weights and activations with arbitrary bit-widths.
3 LQ-Nets: Networks with Learned Quantization
LQ-Nets learns bit-operation-compatible quantizers jointly with the network, adapting quantization levels to weight and activation distributions while supporting arbitrary bit-widths. The method uses learnable basis vectors and quantization-error minimization during training.
- 3.1 Preliminaries: Network Quantization: Quantization represents floating-point weights and activations with few bits, reducing model storage and potentially enabling bitwise inner products.With n-bit encoding, weight compression is relative to 32-bit or 64-bit floating-point representation; quantized weights and activations can support xnor and popcnt operations.
- 3.1 Preliminaries: Network Quantization: Fixed quantizers may be suboptimal because weight and activation distributions can be complex and differ across network layers.The paper specifically notes that uniform quantization is not optimal for the observed distributions and that an improper quantizer can jeopardize final accuracy.
- 3.2 Learnable Quantizers: The learnable quantizer generates 2^K levels from a K-dimensional basis vector and binary encodings, preserving compatibility with bitwise operations.Weights use {-1,1} encodings and activations use {0,1} encodings in the implementation; the remaining exposition uses {-1,1}.
- 3.3 Training Algorithm: LQ-Nets jointly train floating-point network parameters and quantizers, quantizing weights before convolution and retaining binary codes and quantizer bases after training.The quantizers are applied layer-wise for activations and channel-wise for weights, with negligible additional parameters compared with network weights.
- 3.3 Training Algorithm: QEM alternates between computing binary encodings and updating the quantizer basis to minimize quantization error during forward passes.The procedure uses block coordinate descent and iterates the alternation T times; moving averages update basis vectors during network training.
4 Experiments
Experiments evaluate LQ-Nets across datasets, architectures, bit-widths, and optimization settings. The results show that jointly learned quantizers and QEM provide strong low-precision accuracy, including improvements over prior methods.
- Experimental setup: Experiments cover CIFAR-10 and ImageNet using AlexNet, ResNet, DenseNet, VGG variants, and GoogLeNet structures.The implementation quantizes convolutional and fully connected layers except the first and last layers, and evaluates both weight-only and weight-and-activation settings.
- Optimization analysis: QEM-trained quantizers outperform naive backpropagation for both weight-only and weight-and-activation quantization on ResNet-20.The authors use QEM for quantizer optimization in the remaining experiments.
- Optimization analysis: T = 1 QEM iteration is sufficient, since T = 2, 3, or 4 provides no significant benefit over T = 1.The solver starts each iteration from the previous training result, which the authors identify as a good starting point.
- Quantizer comparisons: Learnable quantizers outperform HWGQ and DoReFa-Net for ResNet-18 on ImageNet with 1-bit weights and 2-bit activations.Under the same setting, LQ-Nets also outperform DoReFa-Net with 4-bit activations.
- Bit-width analysis: 7.7% is the largest ResNet-18 accuracy drop on ImageNet, occurring at “1/2” bits, while the gap nearly closes at “4/32” bits with a 0.3% absolute difference.The accuracy drop in the “4/4” case is as low as 1%.
- Comparisons with previous methods: 93.4% vs. 92.5% is the VGG-Small accuracy comparison between LQ-Nets and HWGQ with 1-bit weights and 2-bit activations on CIFAR-10.For ResNet-50, “4/4” bits reaches 75.1%, up from 68.7% with “1/2” bits, a 6.4% absolute increase and a 1.3% gap to full precision.
5 Conclusions
The paper concludes that jointly trained learnable quantizers achieve state-of-the-art accuracy across network structures while supporting bitwise inference. Quantizer optimization adds computational cost, but its QEM subproblem is relatively small compared with convolution operations.
- Conclusions: Learnable quantizers jointly trained with network parameters provide flexibility for quantizing both weights and activations.The quantizers are compatible with bitwise operations, supporting fast inference.
- Conclusions: The method achieves state-of-the-art accuracy across various network structures.
- Training cost: O(K^2N) time complexity for solving v is relatively small compared with convolution operations.The analysis assumes K ≪ N.
1 Statistics of Weights and Activations
The supplementary statistics examine ResNet-20 with 3/3-bit quantization, comparing floating-point distributions with values grouped by the learned quantization levels. The figure highlights that quantization uses eight learned levels and different histogram binning after quantization.
- Experimental setting: The statistics use a ResNet-20 model with 3/3 quantization for weights and activations.
- Quantization levels: The learned quantizers represent the values with eight quantization levels.These levels are indicated by orange diamonds in the figure.
- Histogram interpretation: Floating-point histograms use equal step-size bins, whereas post-quantization histograms group values by their corresponding quantization levels.
2 Detailed Hyper-Parameter and Other Setups
The supplementary material specifies data augmentation and optimization settings for CIFAR-10 and ImageNet experiments. These setups include dataset-specific crops, flips, training schedules, momentum, weight decay, and batch sizes.
- CIFAR-10 data augmentation: CIFAR-10 training pads images by four pixels, then randomly crops 32×32 samples and may horizontally flip them.Original images are used at test time.
- CIFAR-10 optimization: CIFAR-10 models train for up to 200 epochs with momentum 0.9 and dataset-specific learning-rate schedules, weight decay, and batch sizes.ResNet-20 uses learning-rate drops at epochs 82 and 123, while VGG-Small uses drops at 80 and 160.
- ImageNet data augmentation: ImageNet training resizes the shorter side to 256, samples 224×224 crops, and applies random horizontal flipping.AlexNet uses 227×227 crops, while testing uses a single centered crop.
- ImageNet optimization: ImageNet models train for up to 120 epochs with momentum 0.9, batch size 256, weight decay 1e−4, and scheduled learning-rate reductions.The stated schedule divides the learning rate by 10 at epochs 30, 60, 85, 95, and 105 for bit-widths larger than 1/2.