Source-linked AI summary
Value-aware Quantization for Training and Inference of Neural Networks
Eunhyeok Park, Sungjoo Yoo, Peter Vajda
TL;DR
Neural-network training and inference need lower memory and computation costs, but aggressive quantization can increase errors and impair accuracy. The paper introduces value-aware quantization, which uses low precision for most small values and high precision for a few large values, with specialized training and inference techniques. Experiments report comparable full-precision accuracy for 4-bit inference within 1% top-1 accuracy and substantial training-memory reductions versus the state-of-the-art.
Problem
Neural-network training and inference face runtime, energy, computation, and memory constraints, especially from storing intermediate activations during training.
Method
Value-aware quantization profiles values, applies reduced precision to the majority of small data, retains high precision for large data, and uses quantized backpropagation and fine-tuning.
Results
The method achieves accuracy comparable to full precision for 4-bit weights and activations with 1% large data within 1% top-1 accuracy across the evaluated networks.
Takeaways & Limitations
Separately handling a small amount of large data supports aggressively low-precision training and inference while reducing quantization errors for the majority of values.
Takeaways & Limitations
Identifying large activations requires sorting, whose inference overhead motivates low-cost approximate solutions left for further study.
Abstract
from arXiv · showhide
We propose a novel value-aware quantization which applies aggressively reduced precision to the majority of data while separately handling a small amount of large data in high precision, which reduces total quantization errors under very low precision. We present new techniques to apply the proposed quantization to training and inference. The experiments show that our method with 3-bit activations (with 2% of large ones) can give the same training accuracy as full-precision one while offering significant (41.6% and 53.7%) reductions in the memory cost of activations in ResNet-152 and Inception-v3 compared with the state-of-the-art method. Our experiments also show that deep networks such as Inception-v3, ResNet-101 and DenseNet-121 can be quantized for inference with 4-bit weights and activations (with 1% 16-bit data) within 1% top-1 accuracy drop.
1 Introduction
Neural-network training and inference require lower runtime, energy, and memory costs, with training especially constrained by activation storage. Reduced-precision quantization addresses these pressures, and the paper proposes value-aware quantization for more aggressive precision reduction.
- Training and inference need improved runtime, energy consumption, and memory cost for server and edge applications.The motivation includes online training, sub-10ms click prediction, and real-time video processing.
- Training is especially constrained by GPU memory because intermediate activations from large batches and deep models must be stored.Activation storage can exceed the memory capacity of state-of-the-art GPUs.
- Reduced precision lowers data size, enabling more parallel and energy-efficient computation at a smaller memory footprint.The paper cites four int8 operations replacing one fp32 operation as an example.
- Existing quantization commonly uses 16-bit training and 8-bit inference, motivating further optimization toward 4-bit precision.
- Value-aware quantization applies reduced precision to concentrated narrow regions while separately handling scattered large data in high precision.The approach is proposed for very deep networks, including 3-bit activations with 2% large data during training.
2 Related Work
Prior work reduces memory or precision through checkpointing, reversible networks, and quantization, but aggressive methods have struggled to preserve accuracy in very deep models. The paper positions its linear method as enabling accurate 4-bit inference in such networks.
- Checkpointing and reversible networks reduce activation memory by storing selected activations or recomputing them during back-propagation.
- State-of-the-art quantization methods include 16-bit training and 8-bit inference, while more aggressive training methods have been investigated.
- Aggressive quantization methods have been limited to small networks and have not preserved full-precision accuracy for very deep models such as ResNet-152.
- Inference quantization methods span hardware-friendly linear schemes with uniform levels and non-linear schemes based mainly on clustering.
- Prior results include 4-bit weights and 5-bit activations for AlexNet and sub-1% additional top-5 loss with 5-bit weights and 6-bit activations on ResNet-101.
- The proposed linear method enables accurate 4-bit inference in very deep networks such as ResNet-101 and DenseNet-121, where prior accurate 4-bit results were not reported.
3 Motivation
Weight and activation distributions are wide because a small number of large values spread across large regions, causing conventional low-bit quantization to waste levels and increase errors. The proposed motivation is to quantize the majority of small values at low precision while retaining high precision for the few large values.
- GoogLeNet’s second convolutional-layer activation and weight distributions are wide because of a small number of large data.With a fixed low bitwidth, wider distributions produce larger quantization errors.
- Conventional 3-bit linear quantization allocates many levels to sparsely populated large values, leaving most quantization levels underutilized.
- Value-aware quantization applies low precision only to the majority of small data, producing smaller spacing between quantization levels.
- The remaining large data are handled in high precision, such as 32 or 16 bits, while their ratio is kept small.The cited example gives an activation ratio of 1–3% of total activation data.
4 Proposed Method
The proposed value-aware quantization profiles values, stores most activations at reduced precision, and separately handles sparse large values at high precision. Training-specific mechanisms preserve accuracy while reducing activation memory and computation overhead, and inference requires low-cost alternatives to sorting.
- Value-aware quantization profiles values, applies reduced precision to mostly small data, and retains high precision for sparse large data.The method targets quantization error by assigning precision according to value magnitude.
- Quantized Back-Propagation: Quantized back-propagation applies quantization to activations used in the backward pass, while local sorting identifies large values independently on each GPU.Local sorting avoids global communication between GPUs during training.
- Quantized Back-Propagation: The training pipeline quantizes stored activations after a full-precision forward pass and restores them before weight updates, leaving forward and backward computations unchanged.Only quantized activations for the network are stored, while full-precision working activations are retained for one layer at a time.
- Back-Propagation of Full-Precision Loss: With ReLU activations, aggressive intermediate-activation quantization affects weight updates but not local gradients because the ReLU derivative makes them independent of activations.This mechanism is presented as the reason full-precision training accuracy can be maintained with very low activation precision.
- Activation Annealing: Activation annealing decreases the ratio of large activations across training phases, reducing average memory cost while maintaining training quality.Early training uses more large activations, whereas later phases require fewer.
- Quantized Inference: Sorting large activations can be expensive during inference, so approximate low-cost sorting is identified as necessary; detailed implementations remain outside the paper’s scope.The limitation concerns identifying the large-value subset rather than the value-aware precision assignment itself.
5 Experiments
Experiments show that value-aware quantization preserves training accuracy while reducing activation memory, and supports low-bit inference with limited accuracy loss. Results also indicate that training benefits from high precision early and progressively smaller large-value ratios.
- Training memory: 7.5X reduction in stored-activation memory is achieved by 3-bit 2% RV-Quant on ResNet-50, with similar accuracy to V-Quant.Lempel-Ziv compression further reduces the 3-bit data by 24.4%, yielding 9.0x reduction in whole stored activations.
- Training accuracy: 3-bit 2% RV-Quant gives almost the same training accuracy as full-precision training across all evaluated networks.The configuration uses 3-bit activations with 2% large activations.
- Training memory: 41.6% and 53.7% reductions in total activation memory are obtained for ResNet-152 and Inception-v3 versus checkpointing.The reductions are from 5.29GB to 3.09GB and from 3.87GB to 1.79GB, respectively.
- Training runtime: 8.8% runtime overhead is required relative to full-precision training, compared with 32.4% for checkpointing.The method may reduce training time further on hardware supporting reduced-precision computation.
- Training schedules: 75.454% versus 50.360% accuracy shows that high precision at the beginning of training is important.Activation annealing maintains nearly the same result while reducing the large-data ratio across training phases.
- Inference accuracy: Within 1% of top-1 accuracy, 4-bit V-Quant with 1% large 16-bit weights and activations matches full precision across the evaluated inference networks.Fine-tuning closes the accuracy gap; without fine-tuning, larger large-value ratios are needed.
6 Conclusions
The paper presents value-aware quantization to reduce training and inference costs by applying low precision to most data while separately handling a small amount of large data. It reports substantial training-memory savings and low-accuracy-loss inference at very low precision, while identifying clipping-aware quantization as future work.
- Contributions: 41.6% and 53.7% smaller memory cost is reported for ResNet-152 and Inception-v3, respectively, compared with the state-of-the-art low-cost training method.The method targets memory cost in training and computation/memory cost in inference.
- Contributions: 4-bit inference with 1% large data stays within 1% of additional top-1 accuracy loss for ResNet-101 and DenseNet-121.The paper also reports 5-bit inference within the same accuracy-loss bound for SqueezeNet-1.1 and MobileNet-v2.
- Limitations and future work: Clipping-aware quantization remains an open direction because clipped activations can have large populations near their maximum and minimum values.The authors propose further investigation to account for these values.