Source-linked AI summary
Deep Learning with Low Precision by Half-wave Gaussian Quantization
Zhaowei Cai, Xiaodong He, Jian Sun, Nuno Vasconcelos
TL;DR
The paper addresses the difficulty of quantizing neural-network activations without undermining backpropagation. It proposes HWGQ for forward ReLU approximation and piecewise backward approximators to reduce gradient mismatch, yielding HWGQ-Net with performance closer to full-precision networks at 1-bit weights and 2-bit activations.
Problem
Activation quantization introduces non-differentiable operators and gradient mismatch, while binary methods approximate the less-used saturating tanh rather than ReLU.
Method
The paper uses a half-wave Gaussian quantizer for forward ReLU approximation and studies piecewise backward approximators using activation statistics and batch normalization.
Results
HWGQ-Net achieves much closer performance to full-precision AlexNet, ResNet, GoogLeNet, and VGG-Net than previously available low-precision networks, using 1-bit weights and 2-bit activations.
Takeaways & Limitations
Combining forward HWGQ with suitable backward operations produces efficient low-precision networks with performance closer to continuous models.
Takeaways & Limitations
Binary forward and backward approximations suffer gradient mismatch, particularly when activation values are large or lie in distribution tails.
Abstract
from arXiv · showhide
The problem of quantizing the activations of a deep neural network is considered. An examination of the popular binary quantization approach shows that this consists of approximating a classical non-linearity, the hyperbolic tangent, by two functions: a piecewise constant sign function, which is used in feedforward network computations, and a piecewise linear hard tanh function, used in the backpropagation step during network learning. The problem of approximating the ReLU non-linearity, widely used in the recent deep learning literature, is then considered. An half-wave Gaussian quantizer (HWGQ) is proposed for forward approximation and shown to have efficient implementation, by exploiting the statistics of of network activations and batch normalization operations commonly used in the literature. To overcome the problem of gradient mismatch, due to the use of different forward and backward approximations, several piece-wise backward approximators are then investigated. The implementation of the resulting quantized network, denoted as HWGQ-Net, is shown to achieve much closer performance to full precision networks, such as AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision networks, with 1-bit binary weights and 2-bit quantized activations.
1. Introduction
Deep networks offer strong vision performance but remain costly to deploy because of large memory footprints and floating-point computation. The paper develops forward and backward approximations for ReLU-based low-precision networks to improve learning efficiency.
- Motivation: Large deep-network models and floating-point dot-products impede deployment when memory or computation is limited.Examples include model sizes from 50MB for GoogLeNet to 500M for VGG-Net, alongside typically GPU-based computation.
- Motivation: Quantized weights can reduce memory substantially, but quantized activations are needed to replace dot-products with logical and bit-counting operations.The paper frames activation quantization as a route to computational speedups in addition to weight compression.
- Problem: Activation quantization is harder than weight quantization because non-differentiable operators interfere with backpropagation.Prior work reported nontrivial losses when activations were additionally binarized on large-scale classification tasks.
- Approach: The paper treats feedforward quantization and backpropagation approximation as separate functions approximating each unit’s activation function.For binary quantization, these functions correspond to discrete and continuous approximations of tanh.
- Contribution: HWGQ-Net combines a half-wave Gaussian forward quantizer with backward approximations designed to reduce gradient mismatch in low-precision networks.The design exploits activation statistics and batch normalization while investigating linearization, clipping, and suppression of gradients.
2. Related Work
Prior compression methods reduce model size or computation through factorization, pruning, architecture design, and quantization. Activation quantization offers additional benefits, but earlier low-precision methods were not competitive with full-precision models on large-scale classification.
- Model compression: Model compression has used quantization, low-rank factorization, pruning, and architecture design to reduce network size.Pruning was reported to reduce parameters by an order of magnitude without considerable classification-accuracy loss.
- Weight quantization: Weight quantization can provide large memory savings, including representations as small as 1 bit per weight, with marginal classification-accuracy cost.Other approaches explored fixed-point, vector, k-means, and layer-specific bit-width allocation methods.
- Activation quantization: Quantized activations can replace inner-products with logical and bit-counting operations and reduce training memory by avoiding full-precision activation caches.These benefits motivated growing interest in activation quantization.
- Open limitation: Earlier methods addressed gradient mismatch or quantized gradients, but none produced low-precision networks competitive with full-precision models on ImageNet-scale classification.The cited methods had produced good results on datasets such as CIFAR-10, but the stated limitation concerned large-scale classification.
3. Binary Networks
Binary networks reduce computation by using binary weights and activations, but their sign-based activation approximation creates vanishing-gradient and gradient-mismatch problems. The section establishes these limitations as motivation for improved low-precision activation methods.
- 3.1. Goals: A neural-network unit computes a nonlinear activation from a weight–input dot-product, making repeated floating-point operations a major source of model complexity.The same computation drives both memory demands for weights and computational demands for dot-products.
- 3.2. Weight Binarization: Binary weight matrices approximate full-precision weights as αB, where B contains ±1 values and α is a positive scaling factor.The resulting convolution uses a multiplication-free operation.
- 3.2. Weight Binarization: Binary weights alone do not eliminate floating-point computation because inputs and resulting activations remain full precision.Binarizing inputs enables further reductions in computational complexity.
- 3.3. Binary Activation Quantization: Binary activation quantization replaces the activation nonlinearity with a sign function for simplified feedforward computation.The binary sign approach is presented as an approximation of the classical tanh nonlinearity.
- 3.3. Binary Activation Quantization: The sign function has derivative zero almost everywhere, producing very small gradients and preventing gradient descent from converging to cost minima.Hard tanh is therefore used during backpropagation as an alternative continuous approximation.
- 3.3. Binary Activation Quantization: Using different forward and backward approximations creates gradient mismatch, making backpropagation potentially highly suboptimal.The mismatch combines with the limitations of approximating the saturating tanh nonlinearity.
4. Half-wave Gaussian Quantization
The paper approximates ReLU activations with a half-wave Gaussian quantizer for feedforward computation and piecewise functions for backpropagation, addressing efficiency and gradient mismatch. It compares vanilla, clipped, and log-tailed ReLU backward approximations to balance fidelity and stable optimization.
- 4.1. ReLU: The proposed strategy uses a quantizer Q(x) for forward ReLU approximation and a piecewise linear approximation eQ(x) during backpropagation.The ReLU is selected because it improves backpropagation efficiency relative to squashing nonlinearities.
- 4.2. Forward Approximation: Non-uniform quantizers store activation indices using log2 m bits, whereas uniform quantizers can use a shared scaling step for storage and arithmetic.The distinction concerns how quantization levels and indices are represented under reduced precision.
- 4.2. Forward Approximation: An optimal quantizer minimizes expected squared error, with its solution depending on the dot-product probability distribution and obtainable using Lloyd’s algorithm.Applying this procedure separately to every changing network unit would be computationally intractable.
- 4.2. Forward Approximation: The half-wave Gaussian quantizer exploits approximately Gaussian, symmetric dot-product statistics and ReLU’s half-wave structure to provide nonnegative quantization levels and thresholds.Its parameters are designed for a Gaussian distribution and can be shared after batch normalization, which produces approximately zero-mean, unit-variance responses across layers.
- 4.3. Backward Approximation: The vanilla ReLU has zero gradient mismatch below the quantizer tail boundary but increasingly mismatches Q(x) for large x, producing inaccurate and potentially unstable gradients.The tail error is unbounded beyond the largest quantization threshold because the ReLU continues increasing while Q(x) remains quantized.
- 4.3.2. Clipped ReLU: The clipped ReLU becomes constant beyond qm, eliminating tail mismatch and restricting nonzero gradients to dot-products in (0, qm].This improves matching to HWGQ and was found useful for stable neural-network optimization, while clipping can discard information above qm.
- 4.3.3. Log-tailed ReLU: The log-tailed ReLU preserves vanilla-ReLU behavior below qm while assigning decreasing derivative weight above qm and approaching clipped-ReLU behavior at infinity.Its logarithmic tail offers an intermediate growth rule between the unbounded vanilla ReLU and constant clipped ReLU.
5. Experimental Results
Experiments on ImageNet and CIFAR-10 evaluate HWGQ-Net across activation quantization levels, backward approximations, and network architectures. The results show that clipped backward approximations and very low-bit HWGQ activations substantially improve quantized-network performance and stability.
- Experimental setup: ImageNet experiments evaluated AlexNet, ResNet, VGG-Net, and GoogLeNet using top-1 and top-5 classification accuracy.The dataset contained approximately 1.2M training images across 1K categories and 50K validation images.
- Activation and weight quantization: Activation quantization caused a nontrivial accuracy loss, whereas weight binarization caused only a minor degradation in the compared AlexNet, ResNet-18, and VGG-Variant experiments.Combining binary weights and activation quantization reduced recognition performance further.
- Forward quantization: HWGQ produced lower training error than sign(x), but pairing it with vanilla ReLU backward propagation caused unstable optimization, especially in deeper networks.For ResNet-18 and VGG-Variant, HWGQ with vanilla ReLU performed worse than HWGQ with sign-based backward approximation.
- Backward approximations: Clipped ReLU and log-tailed ReLU backward approximations enabled more stable learning and reached better optima than vanilla ReLU across the evaluated networks.Log-tailed ReLU was slightly better on AlexNet but slightly worse on ResNet-18 and VGG-Variant; clipped ReLU was selected thereafter.
- Bit-width impact: For AlexNet, two non-uniform positive quantization levels plus zero sufficed to achieve recognition rates close to full precision, with performance saturating as levels increased.At equal bit width, the uniform quantizer was noticeably superior to the non-uniform quantizer when comparing “2” and “3∗”.
6. Conclusion
The paper designs forward and backward ReLU approximators for low-precision deep-network training, combining them into HWGQ-Net. Experiments show improved performance over prior low-precision methods and a smaller gap from full-precision networks.
- HWGQ-Net combines a half-wave Gaussian forward quantizer with a piece-wise continuous backward approximator for ReLU.The forward quantizer supports feedforward computation, while the backward function is used during learning.
- Gradient mismatch is most affected by activation outliers, motivating clipped ReLU and log-tailed ReLU backward approximators.
- HWGQ-Net significantly outperforms previous low-precision approaches and substantially reduces the gap from full-precision state-of-the-art networks.