Source-linked AI summary
Low-bit Quantization of Neural Networks for Efficient Inference
Yoni Choukroun, Eli Kravchik, Fan Yang, Pavel Kisilev
TL;DR
Large neural networks increase computation and storage demands, while aggressive quantization can reduce accuracy or require retraining. This paper formulates linear quantization as an MMSE problem with hardware-aware parameter partitioning and multiple low-precision tensors. Its INT4 method achieves state-of-the-art results with minimal accuracy loss, including 3% top-1 and 1.7% top-5 mean degradation on ImageNet versus FP32 models.
Problem
Large neural networks challenge real-time inference on resource-constrained hardware, while aggressive low-bit quantization often reduces accuracy or requires retraining, mixed precision, or non-uniform quantization.
Method
The paper uses layer-wise constrained MMSE optimization for weights and activations, hardware-aware kernel-wise partitioning, multiple low-precision tensors, and calibration-based scaling-factor refinement.
Results
3% top-1 and 1.7% top-5 mean accuracy degradation were achieved with INT4 quantization of weights and activations on ImageNet versus FP32 models across various architectures.
Takeaways & Limitations
The framework supports fast deployment of pretrained neural networks on hardware that does not allow mixed-precision operations, with an adjustable compression-complexity trade-off.
Takeaways & Limitations
Multiple quantized tensors for high-MSE layers require more parameters and computations, creating an accuracy-performance trade-off that depends on hardware design.
Abstract
from arXiv · showhide
Recent machine learning methods use increasingly large deep neural networks to achieve state of the art results in various tasks. The gains in performance come at the cost of a substantial increase in computation and storage requirements. This makes real-time implementations on limited resources hardware a challenging task. One popular approach to address this challenge is to perform low-bit precision computations via neural network quantization. However, aggressive quantization generally entails a severe penalty in terms of accuracy, and often requires retraining of the network, or resorting to higher bit precision quantization. In this paper, we formalize the linear quantization task as a Minimum Mean Squared Error (MMSE) problem for both weights and activations, allowing low-bit precision inference without the need for full network retraining. The main contributions of our approach are the optimizations of the constrained MSE problem at each layer of the network, the hardware aware partitioning of the network parameters, and the use of multiple low precision quantized tensors for poorly approximated layers. The proposed approach allows 4 bits integer (INT4) quantization for deployment of pretrained models on limited hardware resources. Multiple experiments on various network architectures show that the suggested method yields state of the art results with minimal loss of tasks accuracy.
1. Introduction
The paper targets efficient low-bit inference of pretrained neural networks on constrained hardware, addressing accuracy loss and deployment limitations through hardware-aware quantization. Its INT4 approach combines MMSE optimization, fine-grained partitioning, and multiple tensors for sensitive layers.
- Motivation: Low-bit quantization reduces model storage, power consumption, memory bandwidth, computational complexity, and chip area, but often sacrifices task accuracy.INT8 operations can save up to 30x energy and 116x area compared with FP32 operations.
- Problem setting: The framework targets pretrained networks on hardware that cannot tolerate complete retraining or mixed-precision calculations.It is designed for highly constrained hardware such as smartphones and IoT devices.
- Method: MMSE optimization finds layer-specific quantization parameters for weights and activations, while kernel-wise partitioning preserves dot-product linearity.The method also identifies quantization-sensitive layers and applies multiple low-precision tensors where needed.
- Method: Fine-grained weight partitioning, optimal MMSE quantization, multiple tensors, and refined quantization factors support hardware-compliant deployment with minimal task-accuracy loss.These components are presented as the paper’s main contributions.
- Results: 3% top-1 and 1.7% top-5 mean accuracy degradation were reported for INT4 quantization of weights and activations on ImageNet.The comparison is against FP32 models across various popular architectures.
2. Related Work
Related work includes training-based and direct quantization approaches for reducing neural-network resource demands. The paper focuses on direct quantization of pretrained FP32 networks, motivated by the deployment difficulty of solutions requiring retraining, mixed precision, or non-uniform quantization.
- Training-based quantization: Training-based quantization can quantize weights, activations, and gradients to very low precision, but discrete parameters make backpropagation difficult.Prior work uses straight-through estimators, stochastic quantization, variational Bayes, or smooth approximations.
- Direct quantization: The paper instead studies direct quantization of pretrained FP32 networks without full training, extending beyond relatively robust INT8 weight quantization toward lower-bit deployment.INT8 activation quantization is described as more challenging under real-time constraints.
- Deployment constraint: Quantization solutions involving mixed precision or non-uniform representations can make fast deployment difficult on highly constrained mobile and IoT hardware.This deployment constraint motivates the paper’s linear low-bit approach.
3. Proposed MMSE Quantization Method
The paper formulates low-bit quantization as layer-wise MMSE optimization and preserves efficient hardware execution through kernel-wise partitioning and adaptive multiple-tensor representations. Scaling-factor refinement further improves approximation using a small unlabeled calibration set.
- MMSE formulation: Quantization minimizes weight and activation noise through MSE optimization to approximate the original network without full retraining.The analysis models quantization noise and uses expected output MSE as the optimization criterion.
- Linear quantization: Each tensor is represented by a low-precision integer tensor and a floating-point scaling factor, preserving linear tensor multiplication.The product of two scaled quantized tensors remains a scaled low-precision tensor product.
- Hardware-aware partitioning: Kernel-wise scaling factors improve INT4 approximation while preserving dot-product linearity and efficient multiplication on dedicated hardware.Kernel-wise quantization adds only negligible storage proportional to the number of kernels.
- Multiple tensor quantization: The method uses multiple low-precision tensors for key layers whose MSE exceeds a threshold, trading additional computation and parameters for accuracy.This avoids mixed-precision inference while targeting only layers that are harder to approximate.
- MMSE optimization: Dual line-search improves MSE by 5x on average over tested dual layers in the INT4 setting.The approach addresses the highly non-convex objective over two scaling factors.
- Scaling-factor refinement: Scaling-factor refinement uses unlabeled calibration data and requires only a few optimization steps, improving accuracy by up to 23 percent.The procedure optimizes one scaling value per convolutional kernel without gradients through non-differentiable quantization.
4. Experiments
Experiments evaluate the framework across popular architectures, including models sensitive to quantization, and examine INT4 settings and accuracy–compression trade-offs. The proposed framework reports mean accuracy degradation for regular and offset versions while using INT4 weights and activations.
- The evaluation covers popular architectures, including SqueezeNet and DenseNet, and considers signed and unsigned INT4 representations.
- 3% mean top-1 and 1.7% mean top-5 accuracy decay are reported for the offset unsigned INT4 version.
- 6.7% mean top-1 and 4% mean top-5 accuracy decay are reported for the regular INT4 version.
- The experiments analyze accuracy–compression trade-offs and report compression ratios for weights and activations alongside the percentage of dual layers.
5. Conclusion
The framework combines hardware-aware parameter partitioning, quantized filter banks, calibration-based scaling refinement, and residual activation quantization for efficient pretrained-model deployment. It supports adjustable precision and reports state-of-the-art INT4 inference quantization results.
- The framework uses hardware-aware parameter partitioning and quantized filter banks to refine layers with high mean squared error.These components target better approximation during constrained low-bit deployment.
- A small calibration set refines quantization scaling factors to improve approximation of the original model.
- Residual quantization improves approximation of the most sensitive network layers' activations.
- The framework can be adjusted to any desired precision according to the compression-complexity trade-off.
- The approach enables fast and efficient deployment of pretrained models and produces state-of-the-art INT4 inference quantization results.