Source-linked AI summary
Towards Effective Low-bitwidth Convolutional Neural Networks
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, Ian Reid
TL;DR
Low-precision networks with quantized weights and activations are difficult to optimize because noisy gradient approximations can trap training in poor local minima. The paper proposes progressive and guided training strategies, reporting that 4-bit weights and activations can outperform 32-bit models on ImageNet and CIFAR-100 with general frameworks.
Problem
Training networks with low-precision weights and activations is difficult because quantization introduces noisy gradient signals and increases the risk of poor local minima.
Method
The paper uses two-stage weight-then-activation quantization, progressively decreases bit-width, and jointly trains full-precision and low-precision models with guidance.
Results
Using 4-bit weights and activations for all layers outperforms the 32-bit model on ImageNet and CIFAR-100 with general frameworks.
Takeaways & Limitations
The methods support effective low-bitwidth training across general architectures, including AlexNet and ResNet-50, and are presented as scalable to deeper networks.
Takeaways & Limitations
A pretrained full-precision model may not be optimal or suitable for quantization when used to guide training.
Abstract
from arXiv · showhide
This paper tackles the problem of training a deep convolutional neural network with both low-precision weights and low-bitwidth activations. Optimizing a low-precision network is very challenging since the training process can easily get trapped in a poor local minima, which results in substantial accuracy loss. To mitigate this problem, we propose three simple-yet-effective approaches to improve the network training. First, we propose to use a two-stage optimization strategy to progressively find good local minima. Specifically, we propose to first optimize a net with quantized weights and then quantized activations. This is in contrast to the traditional methods which optimize them simultaneously. Second, following a similar spirit of the first method, we propose another progressive optimization approach which progressively decreases the bit-width from high-precision to low-precision during the course of training. Third, we adopt a novel learning scheme to jointly train a full-precision model alongside the low-precision one. By doing so, the full-precision model provides hints to guide the low-precision model training. Extensive experiments on various datasets ( i.e., CIFAR-100 and ImageNet) show the effectiveness of the proposed methods. To highlight, using our methods to train a 4-bit precision network leads to no performance decrease in comparison with its full-precision counterpart with standard network architectures ( i.e., AlexNet and ResNet-50).
1. Introduction
The paper focuses on improving training for networks with low-precision weights and activations, whose optimization is difficult and can lose accuracy. It proposes progressive and jointly guided training approaches that scale to deeper networks.
- Low-precision networks reduce memory and computation costs but are difficult to train accurately.The paper targets training improvements rather than quantization alone.
- Two-stage training first quantizes weights, then quantizes activations after obtaining a good intermediate solution.The first stage provides an initial point for optimizing the target network.
- Progressive bit-width training decreases precision incrementally from full precision to the target bit-width.
- Jointly training a full-precision network with the low-precision network provides guidance without requiring a pretrained guidance model.The two models can be trained jointly from scratch so they adjust to each other.
- The methods are experimentally effective across AlexNet and ResNet-50, addressing scalability issues of some layer-wise alternatives.
2. Related work
Related work compresses and accelerates deep networks through limited precision, low-rank approximation, efficient architecture design, and pruning or sparsity. The paper builds on these directions by emphasizing low-precision training for hardware-efficient inference.
- Existing compression methods span limited numerical precision, low-rank approximation, efficient architecture design, and network pruning.
- Limited numerical precision: Limited-precision methods constrain weights to binary, ternary, or other low-bit representations to improve computation and storage efficiency.Examples include replacing multiply-accumulate operations with simpler accumulations and constraining weights to powers of two or zero.
- Low-rank approximation: Low-rank methods approximate filters or tensor structures to reduce computational cost and remove redundancy in convolutional kernels.
- Efficient architecture design: Efficient architecture designs use smaller convolutions or residual connections to reduce complexity and facilitate training of deep networks.
- Pruning and sparsity: Pruning and sparsity reduce storage and bandwidth requirements, with deep compression combining pruning, trained quantization, and Huffman coding.
3. Methods
The methods progressively simplify optimization for low-precision CNNs and use jointly trained full- and low-precision models for guidance.
- 3.1. Quantization function revisited: The quantization function maps normalized full-precision values to k-bit values, while activations are clipped to [0, 1] before quantization.Weights use tanh-based normalization to reduce large-value effects; activations use f(x)=clip(x,0,1).
- 3.1. Quantization function revisited: The straight-through estimator enables back-propagation through nondifferentiable quantization, but its noisy gradient can lead to poor local minima.Quantizing both weights and activations further worsens the optimization difficulty.
- 3.2. Two-stage optimization: Two-stage optimization first quantizes weights with full-precision activations, then quantizes activations and retrains from the converged weight-quantized model.The first stage is an easier related subproblem intended to provide a better initialization for the target problem.
- 3.3. Progressive quantization: Progressive quantization lowers precision during training, using each higher-precision solution to initialize the next lower-precision model.The example sequence is 32-bit →16-bit →4-bit →2-bit, with the final stage at the target precision.
- 3.4. Guided training with a full-precision network: Guided training jointly updates full-precision and low-precision networks, adding a guidance loss after quantizing the full-precision model's weights and activations.The guidance loss regularizes both classification objectives, and the two models adapt to each other during training.
- 3.4. Guided training with a full-precision network: The guidance strategy avoids fixing a pretrained full-precision model because such a model may be suboptimal or unsuitable for quantization.The approach instead trains the full-precision and low-precision models jointly from scratch.
4. Experiment
Experiments on CIFAR-100 and ImageNet evaluate two-stage optimization, progressive quantization, guided training, and their combinations across AlexNet and ResNet-50 at low precision. The methods improve low-bit training, with particularly strong 4-bit results and consistent gains over baselines.
- Experimental setup: Experiments evaluate AlexNet and ResNet-50 on CIFAR-100 and ImageNet using top-1 and top-5 validation accuracy across multiple precision settings.ImageNet experiments report 4-bit and 2-bit models, with precision schedules including {32, 8, 4, 2}.
- Experimental setup: The ablations compare a DoReFa-Net baseline with two-stage optimization, progressive quantization, guided training, and combinations of these components.The full model combines PQ, TS, and Guided, while additional variants test layer precision sensitivity.
- Main results: 4-bit AlexNet on ImageNet outperforms the full-precision reference by 0.7% in top-1 accuracy, while 2-bit performance drops relative to 32-bit but improves 2.8% over the baseline.The reported results attribute these gains to progressive optimization and guidance from the full-precision model.
- Main results: 4-bit ResNet-50 is comparable with the full-precision reference with no accuracy loss, and its 2-bit model improves over the competing baseline despite an accuracy gap to full precision.This supports effectiveness beyond AlexNet and across deeper architectures.
- Main results: On CIFAR-100, the proposed approach outperforms DoReFa-Net, and its 4-bit full model surpasses the full-precision model.The paper speculates that 4-bit weights and activations may provide suitable model capacity and prevent overfitting.
- Ablation study: Fine-tuning from a full-precision model yields higher overall accuracy than learning from scratch, with a 2.7% gap between the baseline and two-stage optimization under scratch training.The ablation interprets this as evidence that initialization affects low-bit accuracy and that two-stage optimization helps reach a better local minimum.
- Ablation study: Stage 1 causes minimal accuracy loss, while Stage 2 remains consistently better than the baseline across epochs despite an apparent decrease from Stage 1.The study uses 2-bit ResNet-50 on ImageNet to examine the two-stage strategy.
- Ablation study: Guided training improves the baseline, and PQ+TS+Guided outperforms PQ+TS in all reported settings; joint optimization benefits both full- and low-precision models.Keeping the full-precision model unchanged causes an apparent performance drop in the joint-training ablation.
5. Conclusion
The paper proposes three approaches for training networks with low-precision weights and activations, and reports that 4-bit models can outperform 32-bit models on ImageNet and CIFAR-100.
- The paper proposes three approaches to address optimization for networks with low-precision weights and activations.
- The approaches combine two-stage quantization, progressive reduction from high to low precision, and joint learning with a full-precision counterpart.