PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable Methods for 8-bit Training of Neural Networks

Ron Banner, Itay Hubara, Elad Hoffer, Daniel Soudry

arXiv:1805.11046v3cs.LGstat.ML

TL;DR

The paper addresses the computational cost of neural-network training, especially the substantial computation required by gradient calculations. It develops an 8-bit training scheme that selectively retains higher precision where needed and introduces Range BN; experiments show large-scale 8-bit training without harming accuracy.

  • Problem

    Neural-network training requires substantially more computation than evaluation, making gradient quantization important for improving training efficiency.

  • Method

    The method quantizes parameters, activations, and layer gradients to 8-bit while retaining higher precision for weight-gradient computation, and replaces standard batch normalization with Range BN.

  • Results

    8-bit training on a large-scale dataset does not harm accuracy, while the approach provides speed, memory, and energy benefits.

  • Takeaways & Limitations

    Range BN is a viable alternative for low-precision training because it is comparable to traditional batch normalization in accuracy and convergence rate.

  • Takeaways & Limitations

    The Range BN scale adjustment relies on the assumption that neural inputs follow an approximately Gaussian distribution.

Abstract

from arXiv · show

Quantized Neural Networks (QNNs) are often used to improve network efficiency during the inference phase, i.e. after the network has been trained. Extensive research in the field suggests many different quantization schemes. Still, the number of bits required, as well as the best quantization scheme, are yet unknown. Our theoretical analysis suggests that most of the training process is robust to substantial precision reduction, and points to only a few specific operations that require higher precision. Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients. Additionally, as QNNs require batch-normalization to be trained at high precision, we introduce Range Batch-Normalization (BN) which has significantly higher tolerance to quantization noise and improved computational complexity. Our simulations show that Range BN is equivalent to the traditional batch norm if a precise scale adjustment, which can be approximated analytically, is applied. To the best of the authors' knowledge, this work is the first to quantize the weights, activations, as well as a substantial volume of the gradients stream, in all layers (including batch normalization) to 8-bit while showing state-of-the-art results over the ImageNet-1K dataset.

1 Introduction

Training deep networks is computationally expensive because of extensive multiply-accumulate operations, while prior quantization work mainly addressed inference or parameters and activations. This work targets nearly all training computations at 8-bit, retaining higher precision for batch normalization and weight-gradient computation.

  • Conventional network compression typically quantizes, prunes, shares weights, or applies low-rank approximations after full-precision training.
  • Training requires approximately three times more computation than evaluation, making gradient quantization important for faster training.
  • Earlier results found 16-bit sufficient for most training, whereas 8-bit quantization caused severe degradation.
  • The proposed approach addresses batch normalization and gradient computation as the two main obstacles to almost exclusively 8-bit training.
  • Range BN replaces traditional batch normalization with input-range normalization, while gradient bifurcation uses separate precision levels for backward propagation and weight-gradient computation.

2 Previous Work

Prior work reduced precision in training, but aggressive gradient quantization degraded accuracy. The paper positions 8-bit precision as a possible balance between efficiency, accuracy, and hardware suitability.

  • Binarized networks constrain weights and activations to -1 and 1, but extreme compression can reduce accuracy on complex models and challenging datasets.
  • Previous gradient quantization either used 16-bit without accuracy loss or reduced precision to 6–8 bits with noticeable degradation.
  • The authors claim 8-bit precision may enable training without accuracy loss or architectural modification while matching available hardware capabilities.

3 Range Batch-Normalization

Range BN replaces variance-based normalization with range-based normalization to improve tolerance to quantization. Under a Gaussian input assumption, the scale adjustment C(n) estimates the standard deviation from the input range.

  • Traditional batch normalization uses sums of squares, square roots, and reciprocals that require high precision and large dynamic range.
  • Range BN normalizes each input dimension using max(x) − min(x) instead of variance-based normalization.
  • C(n) = 1/√(2·ln(n)) serves as a scale adjustment for converting the input range into an estimate of standard deviation.
  • The method assumes approximately Gaussian neural inputs, for which input range is highly correlated with standard-deviation magnitude.
  • Removing or modifying C(n) degraded performance in simulations, indicating that precise scale adjustment is important to Range BN success.

4 Quantized Back-Propagation

The quantized back-propagation scheme bifurcates layer gradients so sequential backward propagation uses 8-bit precision while weight-gradient computation remains higher precision. This preserves the performance-critical backward stream while retaining accurate updates.

  • GEMMLOWP provides the training quantization scheme, and Range BN supplies activation extrema without additional O(n) operations for normalization-scale computation.
  • Stochastic rounding was required for good convergence during gradient quantization because unbiased quantization avoids noise accumulation in weight updates.
  • Each layer produces activation gradients for back-propagation and weight gradients for updating the layer’s weights.
  • The sequential activation-gradient multiplication uses 8-bit precision, while the non-sequential weight-gradient multiplication remains full precision.
  • The extra full-precision weight-gradient computation is considered small relative to communicating the layer gradients and can run in parallel with most backward computations.
  • The straight-through estimator approximates differentiation through discrete variables because their exact derivative is zero almost everywhere.

5 When is quantization of neural networks possible?

The analysis asks when quantization preserves vector direction and concludes that high-dimensional vectors can tolerate substantial precision reduction under specific distributional assumptions. For Gaussian-distributed weights, the angle is preserved when the number of quantization levels greatly exceeds 2 ln(N).

  • Batch normalization makes weight direction, rather than weight norm, the quantity that must be preserved during quantization.The paper states BN(C · W · x) = BN(W · x) for any constant C.
  • For an M-bit fixed-point representation, angle preservation requires 2^M to be much larger than 2 ln(N), where N is the vector size.
  • 8-bit precision suffices for the joint product of a 1024-example batch at the last layer of ResNet-50, where 2 ln(3 · 3 · 2048 · 1024) = 5.7 << 2^8.
  • The angle-preservation conclusion depends heavily on W following a Gaussian distribution, explaining why weights and activations may be more robust than gradients.
  • The result models quantization as additive uniform noise and estimates the angle between W and W + ϵ in high dimensions.The noise components are modeled as independently distributed over [−∆/2, ∆/2].

6 Experiments

Experiments evaluate Range Batch-Norm and Quantized Back-Propagation across datasets and models, linking cosine similarity to quantization quality. Range BN matches standard batch normalization in accuracy, while the full scheme quantizes nearly all training operations to 8-bit and retains only limited higher-precision operations.

  • 6.1 Experiment results on cifar-10 dataset: CIFAR-10 experiments found high correlation between cosine similarity, quantization bit-width, and final accuracy.
  • 6.1 Experiment results on cifar-10 dataset: Backward-pass vector directions were less preserved than forward-pass directions, and validation accuracy tracked backward-pass cosine similarity closely.The results identify gradient quantization as the primary bottleneck in these experiments.
  • 6.1 Experiment results on cifar-10 dataset: Layer gradients did not follow a Gaussian distribution, so their values were bifurcated into low- and high-precision copies to reduce noise accumulation.
  • 6.2 Experiment results on ImageNet dataset: Range Batch-Normalization: Range BN and standard batch normalization produced equivalent accuracy for ResNet-50 on ImageNet.Additional results reported 32.5% vs 32.4% for ResNet-18 on ImageNet and 10.5% vs 10.7% for ResNet-56 on CIFAR-10.
  • 6.3 Experiment results on ImageNet dataset: Putting it all together: The 8-bit scheme quantized the vast majority of operations, retaining float32 minibatch updates and a 16-bit layer-gradient copy for computing weight gradients.For minibatches of 256 examples, float32 updates constituted less than 0.4% of the training effort.

7 Discussion

The paper identifies which training operations require higher precision and combines selective precision with Range BN for efficient 8-bit training. Experiments report comparable accuracy and convergence to traditional batch normalization while retaining parallel gradient computation.

  • Gradient quantization: Gradient direction is less preserved during backpropagation, and validation accuracy closely follows the resulting cosine similarity.This motivates retaining higher precision for the weight-gradient computation while quantizing the next-layer gradient more aggressively.
  • Gradient quantization: Layer gradients are bifurcated: g_l remains 8-bit for computing g_{l−1}, while an unquantized version is used at 16-bit for g_W.The slower weight-gradient computation can run in parallel with layer-gradient propagation.
  • Range BN: Range BN normalizes inputs using max(x) − min(x), avoiding the square and square-root operations required by traditional batch normalization.Its backward derivative is one at maximal or minimal coordinates and zero elsewhere.
  • Range BN: Range BN is comparable to traditional batch normalization in accuracy and convergence rate.The paper presents this as evidence that Range BN is viable for low-precision training.
  • Efficiency: Replacing float32 with int8 makes multiplications 16 times faster and at least 15 times more energy efficient for the forward pass and layer-gradient calculations.Weight-gradient multiplications use 8-bit inputs with 16-bit gradients, yielding 8x speedup and at least 2x power savings.

B Quantization methods

The quantization method uses GMMLOWP with tensor-specific clamping, while stochastic rounding is required for good convergence.

  • Quantization scheme: The implementation uses the GMMLOWP quantization scheme for an input tensor x, given clamping values and a bit count M.The scheme follows Google’s open-source library implementation.
  • Clamping: Weights and activations use absolute extrema for clamping, while activations instead average extrema across K chunks to reduce dynamic-range variance.Smaller quantization steps result from the chunk-based activation range.
  • Rounding: Good convergence was achieved only with stochastic rounding, which prevents accumulated quantization bias in weight updates.The paper links this requirement to the gradients’ eventual role in updating weights.

C Additional Experiments

Additional experiments test more aggressive Quantized BackPropagation, including ternarized gradients and 1-bit weights and activations. On CIFAR10, widening convolutional filters substantially recovers performance.

  • Ternarized Back-Propagation: Ternarized Back-Propagation uses 1-bit weights and activations, with ternarized gradients and XNOR-popcount replacements for forward MAC operations.Stochastic ternarization and wider convolutional layers are used to limit accuracy degradation.
  • CIFAR10: On CIFAR10, tripling convolutional filter widths produced results similar to BNN and full-precision models.The experiment used a VGG-like network with the original work’s hyperparameters.
  • CIFAR10: On an 18-layer ResNet, widening the network improved performance until it came within 1% of the original performance.The passage presents widening as a way to mitigate low-precision training’s accuracy drop.

C.2 ImageNet

The ImageNet experiments evaluate TBP on an inflated AlexNet using multiple activation and gradient precisions. Lower-precision configurations achieve substantial but configuration-dependent top-1 and top-5 accuracy.

  • Dataset and metrics: ImageNet contains 1.2M training samples, 50K test samples, and 1,000 labeled categories; evaluation reports top-1 and top-5 error rates.The passage defines top-k error as the fraction of images whose correct label is absent from the k most probable predictions.
  • Results: With binarized weights and 4-bit activations and gradients, inflated AlexNet reached 53.3% top-1 accuracy and 75.84% top-5 accuracy.The first and last layers were kept in full precision.
  • Results: With 2-bit activations, inflated AlexNet reached 49.6% top-1 accuracy and 73.1% top-5 accuracy.This configuration used the same AlexNet inflation setting described in the experiment.
  • Results: Table 1 reports top-1 validation error for TBP BNNs on ImageNet with AlexNet topology, indexed by inflation, activation bits, weight bits, and gradient bits.The table’s abbreviations are I, A, W, and G.

C.3 Additional experiments

These experiments examine how aggressively ternarized back-propagation can reduce computation while preserving training accuracy. They focus on ternarizing update gradients and using stochastic sampling to recover convergence and accuracy.

  • Additional experiments: Quantizing update gradients can reduce communication bandwidth, power consumption, and training time, but the update phase may require hardware support for full-precision MACs.The experiments distinguish sequential back-propagation from the parameter-update stage and assess both stages' computational costs.
  • Ternarizing both stages: Ternarizing both stages initially reaches only approximately 80% accuracy when the update stage uses less than 3-bit precision.The authors attribute this degradation to ternarization noise distorting the update-gradient direction.
  • Ternarizing both stages: Stopping gradient ternarization after accuracy ceases increasing allows convergence to the same accuracy as TBP, using full-precision weight gradients only for the last couple of epochs.This provides a staged training schedule: ternarized updates first, followed by full-precision updates.
  • Multiple stochastic ternarization sampling: Multiple stochastic ternarization samples are averaged through gated XNOR operations to approximate update gradients without floating-point MACs.The number of samples is selected by comparing update-gradient correlations with and without ternarization.
  • Multiple stochastic ternarization sampling: 12 samples for the first two convolutional layers and approximately 6 for the remaining layers keep the correlation coefficient above 0.7 and produce 87.5% accuracy.ResNet-18 and BNN models were inflated by 5 and 3, respectively, in the corresponding CIFAR10 experiments.
Loading 1805.11046v3…