Source-linked AI summary
Towards Unified INT8 Training for Convolutional Neural Network
Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, Junjie Yan
TL;DR
Quantized gradients promise faster training but can destabilize optimization, and prior approaches lacked a unified solution across networks and tasks. This paper analyzes gradient behavior, derives convergence principles, and introduces two stabilization techniques for INT8 training. Experiments report accurate training across varied networks and tasks, plus a 22% speedup on Pascal GPUs.
Problem
Prior low-bit training methods lacked a successful unified framework for diverse networks and tasks because inappropriate gradient quantization can destabilize or crash training.
Method
The paper analyzes four gradient characteristics, derives a convergence bound, and proposes Direction Sensitive Gradient Clipping with Deviation Counteractive Learning Rate Scaling.
Results
The framework achieves comparable accuracy to full-precision training across varied networks and tasks and reduces Pascal-GPU training time by 22%.
Takeaways & Limitations
The proposed framework supports flexible INT8 training for varied networks and tasks on off-the-shelf hardware, including low-end Pascal GPUs.
Takeaways & Limitations
The current framework mainly quantizes convolutional layers and does not quantize BatchNorm layers.
Abstract
from arXiv · showhide
Recently low-bit (e.g., 8-bit) network quantization has been extensively studied to accelerate the inference. Besides inference, low-bit training with quantized gradients can further bring more considerable acceleration, since the backward process is often computation-intensive. Unfortunately, the inappropriate quantization of backward propagation usually makes the training unstable and even crash. There lacks a successful unified low-bit training framework that can support diverse networks on various tasks. In this paper, we give an attempt to build a unified 8-bit (INT8) training framework for common convolutional neural networks from the aspects of both accuracy and speed. First, we empirically find the four distinctive characteristics of gradients, which provide us insightful clues for gradient quantization. Then, we theoretically give an in-depth analysis of the convergence bound and derive two principles for stable INT8 training. Finally, we propose two universal techniques, including Direction Sensitive Gradient Clipping that reduces the direction deviation of gradients and Deviation Counteractive Learning Rate Scaling that avoids illegal gradient update along the wrong direction. The experiments show that our unified solution promises accurate and efficient INT8 training for a variety of networks and tasks, including MobileNetV2, InceptionV3 and object detection that prior studies have never succeeded. Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, and reduces the training time by 22% on Pascal GPU without too much optimization effort. We believe that this pioneering study will help lead the community towards a fully unified INT8 training for convolutional neural networks.
1. Introduction
The paper develops a unified INT8 training framework because quantized gradients can destabilize optimization, while low-bit training offers substantial acceleration potential. It combines gradient analysis, convergence principles, and two stabilization techniques to support diverse networks and tasks.
- Motivation: Backward propagation offers greater acceleration potential than forward propagation because it occupies more training time.INT8 arithmetic is theoretically and practically 2× faster than FP16 and 4× faster than FP32, while supporting Pascal, Volta, and Turing GPUs.
- Challenge: Quantized gradients can distort optimization direction, causing unstable or crashed training, especially in deep networks.Prior methods were limited to quantization-friendly networks and often required structural adjustments or additional operations.
- Gradient analysis: The authors identify four gradient characteristics—sharp and wide, evolutionary, depth-specific, and structure-specific—that increase quantization error.These characteristics distinguish gradient quantization from naive weight or activation quantization.
- Method: The framework derives a convergence bound and two principles for stabilizing INT8 training.The principles motivate Direction Sensitive Gradient Clipping and Deviation Counteractive Learning Rate Scaling.
- Results: The unified framework supports MobileNetV2, InceptionV3, object detection, and other tasks with accuracy comparable to full-precision training.It is designed to replace original full-precision training across varied networks and tasks.
- Results: 22% speedup is achieved on a Pascal GPU without too much code optimization.The result is reported for practical INT8 training on an NVIDIA GeForce GTX 1080Ti.
2. Related Work
Prior work explored quantized training but remained limited in network coverage, accuracy, or gradient representation. The cited methods therefore did not establish a broadly successful INT8 training solution.
- Prior limitations: Prior quantized-training studies covered limited cases, including AlexNet with 4- and 6-bit gradients.DoReFa-Net quantized gradients but only evaluated AlexNet with low-precision gradients.
- Prior limitations: WAGE and WAGEUBN used 8-bit integer gradients but incurred accuracy losses greater than 5%.This contrasts with methods targeting accuracy comparable to full-precision models.
- Prior limitations: RangeBN and FP8 training reached comparable accuracy but retained floating-point gradients rather than fully integer gradients.Their gradient representation did not provide the same all-INT8 training setting.
3. Unified INT8 Training
The paper builds a unified INT8 training framework by quantizing gradients and addressing the distinctive, evolving sources of gradient distortion that destabilize deep-network training. Its theoretical principles motivate direction-sensitive clipping and deviation-counteractive learning-rate scaling to reduce direction errors and stabilize convergence.
- Framework: INT8 training accelerates both forward and backward propagation by applying 8-bit arithmetic to convolutional neural network training.The framework targets the expensive training process, including both propagation directions.
- Quantization: Symmetric uniform quantization clips data to a bounded range, scales it to fixed-point 8-bit integers, and subsequently dequantizes the result.The scheme is selected for hardware-friendly computation and applies to weights, activations, and gradients.
- Gradient quantization challenges: Quantized gradients perturb optimization direction, potentially destabilizing training and causing rapid crashes in architectures such as MobileNetV2.The paper attributes this failure to accumulated deviation between quantized and original gradients.
- Gradient quantization challenges: Gradients are sharp and wide, evolve during training, vary by depth, and differ by structure, making fixed quantization settings inadequate.Small values dominate while extreme values remain; shallow and depth-wise layers can exhibit larger ranges and sharper distributions.
- Theoretical perspective: The convergence analysis identifies gradient quantization error and learning-rate-dependent terms as central to the regret bound, yielding principles to reduce direction deviation and control updates.The analysis motivates reducing quantization error and scaling the learning rate as quantization perturbations persist.
- Stabilization techniques: Direction Sensitive Gradient Clipping and Deviation Counteractive Learning Rate Scaling jointly lower the average regret bound and stabilize INT8 training.Clipping selects an evolving value that reduces direction deviation, while layer-wise scaling counteracts deviations across depth- and structure-specific gradients.
4. Experiments
Experiments show that the proposed INT8 framework stabilizes training across diverse networks and tasks while delivering accuracy comparable to full precision and practical GPU acceleration. Ablations support both gradient clipping and learning-rate scaling, and Pascal GPU measurements demonstrate end-to-end speed gains.
- Overall evaluation: The framework supports diverse network structures and image-classification and object-detection tasks with state-of-the-art accuracy and practical deployment on mainstream devices.The experiments cover classification and detection settings and compare favorably with full-precision training.
- Ablation studies: Gradient clipping prevents INT8 training from crashing by optimizing clipping to minimize gradient direction deviation.The ablation explicitly links the clipping objective to stable training.
- Ablation studies: Exponential learning-rate scaling counteracts direction deviation more effectively than linear or quadratic scaling and prevents optimization from leaving the convergence boundary.Linear and quadratic forms led to crashes in the reported comparison.
- Ablation studies: Updating the clipping value less frequently causes little accuracy loss, while combining clipping and learning-rate scaling improves performance.The reported practical benefit is reduced clipping-update overhead without substantial accuracy degradation.
- Image classification: On CIFAR-10, MobileNetV2 loses 1.01% accuracy and InceptionV3 performs better than the full-precision model.The method also achieves comparable accuracy to FP8 training on ResNet-20 with lower memory and computation consumption.
- Image classification: On ImageNet, INT8 training improves AlexNet accuracy by 5.84% over DoReFa-Net, gains about 3% over WAGE in the 2-bit-weight setting, and nearly preserves ResNet accuracy.The method also improves accuracy by nearly 3% over FP8 training and runs on off-the-shelf devices.
- Scope of quantization: The framework achieves comparable accuracy to full precision while quantizing convolutional layers, because convolution dominates training time and BatchNorm and ReLU are not computation-intensive.BatchNorm is not quantized in the reported implementation.
- New application settings: The method reports first successful INT8 training results for MobileNetV2, InceptionV3, and object detection, including approximately 1% MobileNetV2 accuracy loss.InceptionV3 reaches comparable accuracy to full precision, and CIFAR-10 InceptionV3 can outperform full precision.
5. Conclusions
The paper concludes that its gradient analysis and convergence-based design principles support a unified INT8 training framework for varied networks and tasks. The proposed techniques achieve broad experimental versatility and reduce Pascal GPU training time by 22% with limited optimization.
- Contributions: The framework is built from four distinctive gradient characteristics and two convergence-bound principles for stabilizing INT8 training.These foundations motivate the proposed quantization techniques.
- Contributions: Direction Sensitive Gradient Clipping and Deviation Counteractive Learning Rate Scaling form the proposed stabilization approach.The techniques target gradient-direction deviation and unstable updates.
- Contributions: Extensive experiments demonstrate versatility across various networks and tasks, while Pascal GPU training time decreases by 22% with only trivial optimization.The paper presents this as a step toward fully unified INT8 training.
6. Supplementary Material
The supplementary material derives the regret bound, describes INT8 convolution implementation choices, and reports stable accuracy comparable to FP32 training. It also details GPU-oriented optimizations for data layout, fused quantization, stochastic rounding, and random-number generation.
- Convergence analysis: The regret analysis combines per-entry weight updates, quantization error, and bounded-distance assumptions to derive an aggregate bound over dimensions and iterations.The derivation proceeds by rearranging the update, defining quantization error, applying convexity, and summing across dimensions and iterations.
- Empirical validation: The method makes INT8 training smooth and achieves accuracy comparable to FP32 training on MobileNetV2/CIFAR-10 and ResNet-50/ImageNet.The supplementary results use accuracy and loss curves to assess training stability.
- Hardware efficiency: On GTX 1080Ti, peak integer throughput reaches 44 Tera Operations Per Second versus 11 Tera Float Operations Per Second for floating-point computation.The hardware comparison motivates using integer instructions for efficient training kernels.
- Kernel optimization: Fusing quantization with convolution avoids an extra global-memory read and write during INT8 computation.The fusion targets the overhead caused by reading and writing the full tensor before convolution.
- INT8 convolution implementation: The implementation uses DP4A-based INT8 convolution, NC4HW data layout, and PTX prmt instructions for efficient 4×4 byte-block transposition.The convolution is converted to dot products, while prmt reassembles bytes from registers for layout conversion.
- Stochastic rounding: Stochastic rounding preserves unbiased gradient expectations, while an LCG generates thread-specific random numbers with slight INT8-convolution overhead.Each thread derives a distinct seed by combining the first input data with its thread index.