PaperlayerView exact revision on arXiv

Source-linked AI summary

Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers

Yukuan Yang, Shuang Wu, Lei Deng, Tianyi Yan, Yuan Xie, Guoqi Li

arXiv:1909.02384v2cs.LGstat.ML

TL;DR

Large-scale DNN training quantization lacks a complete low-bit integer approach that includes batch normalization and avoids floating-point dependencies. WAGEUBN quantizes all training data paths, BN, and Momentum, and reports competitive ImageNet accuracy on ResNet18/34/50. The paper presents full INT8 quantization as a step toward lower memory and compute costs.

  • Problem

    Existing schemes often leave parts of training unquantized or retain floating-point values, and most leave batch normalization untouched, while a complete low-bit INT solution for large-scale DNNs is lacking.

    Source: §I · p. 1

  • Method

    WAGEUBN decomposes DNN computations and quantizes W, A, G, E, U, BN, and the Momentum optimizer in low-bit integers.

    Source: §I · pp. 1–2

  • Results

    Experiments report competitive accuracy with fewer overheads on ImageNet ResNet18/34/50 models.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors present full INT8 quantization as a route to bit-wise operations and reduced memory overhead, processing time, circuit area, and energy consumption.

    Source: §V · p. 12

Abstract

from arXiv · show

Deep neural network (DNN) quantization converting floating-point (FP) data in the network to integers (INT) is an effective way to shrink the model size for memory saving and simplify the operations for compute acceleration. Recently, researches on DNN quantization develop from inference to training, laying a foundation for the online training on accelerators. However, existing schemes leaving batch normalization (BN) untouched during training are mostly incomplete quantization that still adopts high precision FP in some parts of the data paths. Currently, there is no solution that can use only low bit-width INT data during the whole training process of large-scale DNNs with acceptable accuracy. In this work, through decomposing all the computation steps in DNNs and fusing three special quantization functions to satisfy the different precision requirements, we propose a unified complete quantization framework termed as ``WAGEUBN'' to quantize DNNs involving all data paths including W (Weights), A (Activation), G (Gradient), E (Error), U (Update), and BN. Moreover, the Momentum optimizer is also quantized to realize a completely quantized framework. Experiments on ResNet18/34/50 models demonstrate that WAGEUBN can achieve competitive accuracy on the ImageNet dataset. For the first time, the study of quantization in large-scale DNNs is advanced to the full 8-bit INT level. In this way, all the operations in the training and inference can be bit-wise operations, pushing towards faster processing speed, decreased memory cost, and higher energy efficiency. Our throughout quantization framework has great potential for future efficient portable devices with online learning ability.

I. INTRODUCTION

WAGEUBN targets complete low-bit integer quantization of large-scale DNN training, including batch normalization and the optimizer. The introduction situates this approach against incomplete prior methods and outlines its framework and reported ImageNet validation.

I. INTRODUCTION

  • Existing training schemes often leave data paths partly quantized, retain floating-point values, or omit batch normalization, while WAGE lacks powerful BN layers for large-scale DNNs.The paper argues that a complete solution must also quantize the optimizer.Source: §I · p. 1
  • WAGEUBN constrains W, A, G, E, U, BN, and the optimizer to low-bit integers, using three fused quantization functions for differing precision requirements.The introduction also describes a flag bit for expanding data coverage and addressing non-convergence from insufficient representation.Source: §I · pp. 1–2
  • ResNet18/34/50 experiments on ImageNet report competitive accuracy with fewer overheads under WAGEUBN.The introduction presents this as validation of the framework on large-scale models and a dataset.Source: §I · p. 2

II. RELATED WORK

  • Prior inference quantization focuses on forward-pass data, while training methods vary in which paths and precisions they quantize and often leave BN untouched.The related-work discussion contrasts these schemes with the stated need for complete INT8 quantization of large-scale DNN training.Source: §II · p. 2

III. WAGEUBN FRAMEWORK

  • The framework quantizes all data in DNN training to INT8 and organizes its description around STE, notation and quantization functions, per-data schemes, and forward/backward implementation.The paper identifies Momentum and BN among the data and operations covered by its scheme.Source: §III · p. 2

A. Straight-through estimator

  • WAGEUBN uses STE to estimate gradients for quantized data, and its notation distinguishes activation gradients used in error backpropagation from weight gradients used in updates.The forward pass is divided into four steps, and the backward pass into five; BN is split into normalization/QBN and scale/offset in the forward pass.Source: A. Straight-through estimator · pp. 2–3, B. Notations · p. 3

C. Quantization Functions

WAGEUBN combines three quantization functions to meet different precision needs across training data paths. Direct quantization handles W, A, and BN; constant quantization handles G; shift quantization handles E.

Direct-quantization function

  • Direct quantization maps continuous values to their nearest discrete states, with bit width k and rounding to the nearest integer.It is used for W, A, and BN.Source: C. Quantization Functions · p. 4

Constant-quantization function

  • Constant quantization normalizes a tensor, limits it to integers, and restores its magnitude, with stochastic rounding and clipping.Its range parameter dr decreases during training, reducing the allowed range; for gradients, the cited examples use 8-bit values in epochs 0–30 and 7-bit values in epochs 30–60.Source: C. Quantization Functions · p. 4

Shift-quantization function

  • Shift quantization normalizes E, converts it to fixed-point values, and restores its magnitude with a layer-wise scaling factor.Unlike constant quantization, it does not use stochastic rounding or a constant to maintain magnitude.Source: C. Quantization Functions · p. 4

D. Quantization Schemes in WAGEUBN

The section introduces detailed quantization schemes after presenting the functions used in WAGEUBN.

D. Quantization Schemes in WAGEUBN

(1) Weight Quantization

The schemes quantize weights, BN, activations, errors, gradients, and the Momentum optimizer. They use different bit widths and quantization choices to constrain the data paths.

(1) Weight Quantization

  • Weights are stored at kWU bits, mapped to kW-bit integers for convolution, and range-limited; they are initialized discretely using an MSRA method.The cited passage identifies MSRA initialization as helpful for faster training.Source: (1) Weight Quantization · p. 4

(2) Batch Normalization Quantization

  • BN operands are quantized to fixed-point values, including the batch mean and standard deviation, scale and offset parameters, and BN output.The BN output uses QBN(x) = Q(x, kBN).Source: (1) Weight Quantization · pp. 4–5

(3) Activation Quantization

  • Activations are quantized after convolution and BN to reduce the increased operand bit width and keep each layer's input width consistent.The activation bit width is kA.Source: (1) Weight Quantization · p. 5

(4) Error Quantization

  • Shift quantization is used for error E because direct quantization would require a large operand width for convergence.For QE2, kE2 ≤ 8 causes non-convergence in large-scale DNNs; kE2 = 16 is reported as suitable with minimum accuracy degradation.Source: (1) Weight Quantization · p. 5
  • Flag QE2 adds a layer-wise scaling factor and a flag bit to expand coverage while retaining an effective INT8 computation value.The 9-bit format is reported to cover almost the same data range as the direct 15-bit quantization; the flag indicates whether |x| is below Sc.Source: (1) Weight Quantization · p. 6

(5) Gradient Quantization

  • Gradient calculations use fixed-point operands, allowing conventional floating-point multiply-accumulate operations to be replaced with bit-wise operations.Gradient quantization reduces bit width in preparation for the optimizer.Source: (1) Weight Quantization · p. 6

(6) Momentum Optimizer Quantization

  • WAGEUBN quantizes Momentum optimizer operands, including the accumulation and gradient, to fixed-point values.The learning rate is also fixed-point, and its bit width is related to the update bit width.Source: (1) Weight Quantization · p. 6
  • Update precision has the greatest reported impact on DNN accuracy, so its bit width must balance model accuracy and memory cost.The learning rate must be fixed-point, and update bit width is directly related to its bit width.Source: (1) Weight Quantization · pp. 6–7

E. Quantization Framework

The framework divides each layer's forward and backward passes into operations that quantize data across the training paths. It uses bit-wise operations in place of conventional floating-point multiply-accumulates.

E. Quantization Framework

  • In the forward pass, weights are stored at kWU bits and mapped to kW bits, while batch statistics and BN operands are quantized before activation.Activation quantization reduces the increased bit width and prepares kA-bit inputs for the next layer.Source: E. Quantization Framework · p. 7
  • The framework replaces conventional floating-point multiply-accumulates with bit-wise operations by quantizing the operands and computation steps.The framework includes W, A, G, E, U, BN, and Momentum optimizer parameters.Source: E. Quantization Framework · p. 7
  • The backward pass covers error propagation, weight and BN gradients, Momentum optimization, and parameter updates.Error propagation has two quantization locations using QE1 and QE2.Source: E. Quantization Framework · p. 7
  • The learning rate is discrete so that the bit width of weight updates can be controlled.The γ and β updates are similar to weight updates, though omitted from the figure for simplicity.Source: E. Quantization Framework · p. 7

F. Overall Algorithm

The complete WAGEUBN quantization process is summarized in forward- and backward-pass pseudocode.

  • Algorithm 1 and Algorithm 2 give the complete quantization process in pseudocode for the forward and backward passes, respectively.Source: F. Overall Algorithm · p. 7

A. Experimental Setup

WAGEUBN is evaluated on ResNet18/34/50 using full 8-bit INT and a 16-bit E2 variant, with the first and last layers left unquantized. The framework also uses quantized mechanisms intended to mitigate saturated learning and vanishing gradients.

A. Experimental Setup

  • The experiments apply WAGEUBN to ResNet18/34/50 on ImageNet, comparing full 8-bit INT with a variant using 16-bit E2.In the full 8-bit version, kW, kA, kGW, kE1, kE2, kγ, and kβ are 8; the E2 variant differs only in QE2.Source: A. Experimental Setup · p. 7
  • The first and last layers remain unquantized because quantizing them reportedly degrades accuracy significantly while saving little overhead.The paper says these layers have few neurons and cites prior work for the same choice.Source: A. Experimental Setup · p. 8
  • Shift quantization is intended to keep backpropagated activation errors from vanishing, while a larger learning rate compensates for smaller quantized gradients.The minimum learning rates are 1.95×10−3 for WAGEUBN and 5 × 10−6 for the floating-point network; quantized Momentum, fixed-point updates, and batch-size selection are also presented as ways to reduce the chance of local optima.Source: A. Experimental Setup · pp. 8–9

B. Training Curve

WAGEUBN’s ImageNet accuracy is reported as competitive with vanilla DNNs and other quantization approaches, with about 4× lower memory size. Training curves show little accuracy difference from vanilla DNNs through epoch 60, after which the gap grows as learning rates differ.

B. Training Curve

  • Through epoch 60, WAGEUBN and vanilla DNNs show little accuracy difference; the gap grows later as their learning rates differ.The paper notes that vanilla DNNs use a much lower learning rate, making their updates more precise, and says reducing WAGEUBN’s learning rate could improve accuracy at additional overhead.Source: B. Training Curve · p. 9, Figure 6 · p. 9
  • The 16-bit E2 WAGEUBN loses 3.46% mean accuracy compared with vanilla DNNs, while WAGEUBN accuracy is reported as comparable to FP8 and QBP2.The paper also reports about 4× memory-size shrink, faster processing, and less energy and circuit area than vanilla DNNs.Source: B. Training Curve · p. 9

C. Quantization Strategies for W, A, G, E, and BN

WAGEUBN assigns direct quantization to weights, activations, and BN, constant quantization to gradients, and shift quantization to errors. These strategies reflect different data distributions and precision requirements, with a flag-based function used for errors between convolution and BN.

C. Quantization Strategies for W, A, G, E, and BN

  • Direct quantization is used for W, A, and BN, constant quantization for G, and shift quantization for E, based on data distribution, sensitivity, and hardware friendliness.Source: C. Quantization Strategies for W, A,… · p. 9
  • Direct quantization barely changes W, BN, and A distributions, while constant quantization changes G greatly without much reported accuracy decline.The paper attributes convergence guidance to gradient orientation rather than magnitude and notes fixed update bit width is more hardware-friendly when kGC is fixed.Source: C. Quantization Strategies for W, A,… · p. 11
  • Eight-bit shift quantization retains magnitude order but omits values with absolute magnitude below 2−7, and is used for errors after activation.It is not sufficient for errors between convolution and BN, so the paper introduces 8-bit Flag QE2 for that case.Source: C. Quantization Strategies for W, A,… · p. 11
  • The data distribution of E3 is reported as almost unchanged after applying 8-bit Flag QE2.Source: C. Quantization Strategies for W, A,… · p. 11

D. Accuracy Sensitivity Analysis

Under WAGEUBN, quantizing E—especially e_l^3—has the greatest impact on accuracy. Accuracy is also more sensitive to a batch size of 16 than full-precision accuracy, which the authors relate partly to omitted moving-statistic momentum and quantization loss.

D. Accuracy Sensitivity Analysis

  • The single-data tests quantized W, A, G, E, or BN to 8-bit INT individually while keeping the update in FP32.These comparisons were intended to reflect the relative difficulty of quantizing each data type.Source: D. Accuracy Sensitivity Analysis · p. 11
  • E, especially e_l^3, demands the highest precision and is the most accuracy-sensitive component under WAGEUBN.The authors report that constraining e_l^3 to 8-bit INT also causes accuracy to fluctuate heavily during training.Source: D. Accuracy Sensitivity Analysis · p. 11
  • At batch size 16, full 8-bit WAGEUBN accuracy declines relatively more than full-precision accuracy; the authors cite omitted moving-statistic momentum and quantization loss as reasons.They describe robustness as still good, with a significant accuracy decline only when batch size is below 16.Source: Figure 8 · p. 11, D. Accuracy Sensitivity Analysis · p. 11

E. Analysis of the Error Quantization between Conv and BN

The analysis connects error quantization to non-convergence when 8-bit QE2 zeros small error values, and reports lower simulated hardware costs for INT8 than FP32. The conclusion identifies specialized hardware design and more applications as remaining work.

E. Analysis of the Error Quantization between Conv and BN

  • Quantization error E, especially e_l^3, is critical to accuracy because it is directly related to the weight update of the l-th layer.The authors state that excessive quantization error can degrade DNN convergence.Source: E. Analysis of the Error Quantization… · p. 11
  • 8-bit QE2 forces e1^3 values in [−2−8R(e1^3), 2−8R(e1^3)] to zero, while Flag QE2 and full-precision distributions are almost the same.The authors identify this zeroing interval as the major distribution difference between QE2 and full precision.Source: E. Analysis of the Error Quantization… · p. 11, Figure 9 · p. 12
  • 8-bit QE2 covers fewer nonzero values than Flag QE2, omitting useful small values and resulting in non-convergence; data ratios also tend to decrease in shallower layers.The authors note that larger values have greater impact on accuracy during error propagation, but smaller values also contain useful information and make up the majority.Source: Figure 10 · p. 12, E. Analysis of the Error Quantization… · p. 12

F. Cost Discussion

  • INT8 multiplication is >3× faster, 10× lower in power, and 9× smaller in circuit area than FP32; INT8 accumulation is about 9× faster with >30× reductions in energy consumption and circuit area.These processing-speed, power, and area results come from simulations of single multiplication and accumulation on an FPGA platform.Source: F. Cost Discussion · p. 12

V. CONCLUSIONS

  • WAGEUBN is presented as complete INT8 quantization for large-scale DNN training and inference, but still needs specialized hardware architecture design and more applications.The conclusion names computing architecture, memory hierarchy, interconnection infrastructure, and mapping tools as future work.Source: §V · p. 12
Loading 1909.02384v2…