Source-linked AI summary
Quantization Networks
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, Xiansheng Hua
TL;DR
High computational and memory costs make DNN deployment difficult on portable, resource-constrained devices, while existing quantization methods face gradient, scope, or training-cost limitations. The paper instead formulates low-bit quantization as a differentiable non-linear function learned end-to-end for weights and activations. Experiments on image classification and object detection show that the resulting quantization networks outperform state-of-the-art methods.
Problem
High computational and memory costs challenge DNN deployment on portable devices, while existing quantization approaches face gradient mismatch or weight-only and training-cost limitations.
Method
The paper formulates low-bit quantization as a differentiable non-linear function that supports end-to-end learning for neural-network weights and activations.
Results
Quantization networks outperform state-of-the-art quantization methods on image classification and object detection tasks.
Takeaways & Limitations
The proposed quantization function provides a simple, uniform approach for arbitrary-bit quantization of both weights and activations.
Takeaways & Limitations
Optimization-based alternatives remain limited to weight quantization and incur high computational costs during training.
Abstract
from arXiv · showhide
Although deep neural networks are highly effective, their high computational and memory costs severely challenge their applications on portable devices. As a consequence, low-bit quantization, which converts a full-precision neural network into a low-bitwidth integer version, has been an active and promising research topic. Existing methods formulate the low-bit quantization of networks as an approximation or optimization problem. Approximation-based methods confront the gradient mismatch problem, while optimization-based methods are only suitable for quantizing weights and could introduce high computational cost in the training stage. In this paper, we propose a novel perspective of interpreting and implementing neural network quantization by formulating low-bit quantization as a differentiable non-linear function (termed quantization function). The proposed quantization function can be learned in a lossless and end-to-end manner and works for any weights and activations of neural networks in a simple and uniform way. Extensive experiments on image classification and object detection tasks show that our quantization networks outperform the state-of-the-art methods. We believe that the proposed method will shed new insights on the interpretation of neural network quantization. Our code is available at https://github.com/aliyun/alibabacloud-quantization-networks.
1. Introduction
Deep neural networks are difficult to deploy on resource-constrained devices, motivating low-bit quantization. The paper proposes differentiable quantization networks that learn weights and activations end-to-end and outperform existing methods on classification and detection.
- High computational and memory costs limit DNN deployment when computation and storage resources are constrained.
- Existing quantization methods use approximation or optimization formulations, with limitations involving gradient mismatch, weight-only applicability, or training cost.
- The proposed quantization function is learned in a lossless, end-to-end manner and avoids the severe gradient mismatch problem.
- The paper formulates quantization as a differentiable non-linear mapping function that provides a simple, general solution for arbitrary-bit weight and activation quantization.
- Quantization networks outperform state-of-the-art quantization methods on image classification and object detection tasks.
2. Related Work
Prior work quantizes networks through approximation or discretely constrained optimization, while neural-network activation functions provide a differentiable-function perspective. The paper’s related-work context contrasts these approaches with smooth nonlinear mappings that support back-propagation.
- Low-Bit Quantization: Low-bit quantization methods are broadly divided into approximation-based and optimization-based approaches.
- Low-Bit Quantization: Approximation-based methods replace full-precision values with discrete low-bit values and use step-function approximations during the forward pass.
- Low-Bit Quantization: Approximation-based methods require backward-process approximations because step-function gradients saturate, creating gradient mismatch and unstable optimization.
- Low-Bit Quantization: Optimization-based methods formulate quantization as a discretely constrained problem, but are limited to weights and incur high computational costs during training.
- Non-Linear Functions: Sigmoid functions provide smooth, non-zero gradients for back-propagation and approach unit-step behavior when the input magnitude is large.
- Non-Linear Functions: ReLU is zero on half its domain and linear on the other half, while Maxout generalizes ReLU by combining multiple simple linear functions.
3. Quantization Networks
Quantization networks formulate low-bit quantization as a differentiable non-linear function applied uniformly to weights and activations. Sigmoid-based training gradually approaches discrete inference quantization while preserving end-to-end learning.
- 3.1. Reformulation of Quantization: The ideal quantization function maps continuous weights or activations to discrete integers by combining binary unit-step functions with specified biases and scales.The output is constrained to a predefined integer set, with the number of intervals determined by that set.
- 3.2. Training and Inference with Quantization Networks: Replacing unit-step functions with sigmoids yields a differentiable soft quantization function that supports end-to-end backpropagation.This formulation avoids needing gradient approximations during training.
- 3.2. Training and Inference with Quantization Networks: Temperature T is gradually increased during training: larger T narrows the gap to ideal step quantization but produces zero gradients in more cases.Starting with small T supports learning, while increasing it reduces the training–inference gap by the end of training.
- 3.2. Training and Inference with Quantization Networks: The quantization function independently transforms each full-precision weight or activation, and its output replaces the original value for subsequent computation.The same mechanism can be inserted wherever weights or activations are quantized.
- 3.2. Training and Inference with Quantization Networks: After training, sigmoid functions are replaced with unit-step functions for inference, producing discrete quantized weights and activations.Training uses the smooth function for efficient optimization, whereas inference uses the ideal quantization function.
- 3.3. Experiments: The reported AlexNet ImageNet evaluation uses Top-1 and Top-5 accuracy (%) and distinguishes weight bits W from activation bits A.The full-precision reference performance is 61.8/83.5.
4. Experiments
Experiments evaluate quantization networks across ImageNet classification and Pascal VOC detection, including weight/activation bitwidths and ablations of quantization design and training. The method reports strong results, including lossless 3-bit performance on ResNet-18 and slightly better detection performance than ADMM.
- Image Classification: ImageNet experiments evaluate AlexNet, ResNet-18, and ResNet-50 using binary, ternary, and higher-bit weight and activation quantization.Performance is reported with Top-1 and Top-5 accuracy, while the listed configurations vary weight and activation bitwidths.
- Image Classification: Our quantization network outperforms state-of-the-art methods for both weight and activation quantization and supports arbitrary bitwidths.The method is applied flexibly to both weights and activations rather than only one parameter type.
- Image Classification: Even in compact ResNet-18, the method achieves lossless results with only 3 bits and surpasses the full-precision model with 3-bit weight quantization.The authors report greater degradation for compact architectures than for AlexNet, while still obtaining these results on ResNet-18.
- Object Detection: For SSD on Pascal VOC, the quantization network is slightly better than ADMM in mAP, compared with 77.8 for the full-precision model.The comparison uses ADMM because other baseline quantization methods did not report object-detection performance.
- Ablation Experiments: Non-uniform quantization outperforms linear quantization, while adaptive bias learning does not outperform the fixed-bias version.Non-uniform intervals are motivated by the roughly Gaussian distribution of pretrained parameters and implemented using clustering centers.
- Ablation Experiments: Training from a pretrained model greatly improves performance over training from scratch, and low-bit quantization significantly reduces inference time and space complexity.The inference complexity result is based on evaluation using a VU9P FPGA.
5. Conclusion
The paper interprets low-bit quantization as a differentiable non-linear function embedded as an activation function. This enables end-to-end learning with flexible quantization of arbitrary-bit weights and activations.
- The paper proposes a soft quantization function based on differentiable non-linear functions.
- Embedding the differentiable quantization function as an activation function enables end-to-end training of quantization networks.
- The method supports arbitrary-bit quantization for both weights and activations.
- Experiments on image classification and object detection verified the effectiveness of the proposed method.