Training and Inference with Integers in Deep Neural Networks

Shuang Wu; Guoqi Li; Feng Chen; Luping Shi

Source-linked AI summary

Training and Inference with Integers in Deep Neural Networks

Shuang Wu, Guoqi Li, Feng Chen, Luping Shi

arXiv:1802.04680v1Feb 13, 2018cs.LG

TL;DR

Existing approaches reduced precision mainly for inference, leaving simultaneous low-bitwidth training and inference unresolved. WAGE constrains weights, activations, gradients, and errors to integers, replacing difficult floating-point components with simpler operations. Across multiple datasets, it reports comparable or state-of-the-art accuracy and potential energy, memory, and hardware benefits, while noting limitations around normalization and more aggressive bitwidth reduction.

Problem
Previous methods reduced inference precision but generally retained higher-precision training, limiting simultaneous integer training and inference for embedded systems.
Method
WAGE shifts and linearly constrains weights, activations, gradients, and errors to low-bitwidth integers, replacing batch normalization with constant scaling and simplifying other components.
Results
Comparable accuracy is reported across MNIST, CIFAR10, SVHN, and ImageNet, with 2-8-8-8 bitwidth configurations achieving state-of-the-art accuracies on multiple datasets.
Takeaways & Limitations
WAGE demonstrates potential for pure integer training and inference on lightweight ASIC or FPGA systems with on-site learning capability.
Takeaways & Limitations
WAGE demonstrations avoid or remove normalization layers, although the authors consider normalization important for multi-modal and cross-branch representations.

Abstract

from arXiv · show

Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as "WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

1 INTRODUCTION

WAGE addresses the gap between low-precision inference and high-precision training by constraining weights, activations, gradients, and errors to low-bitwidth integers across both processes. Evaluations report comparable accuracy, reduced overfitting, and potential hardware benefits.

Existing methods mainly compress networks for inference, while training retains higher precision because stochastic-gradient accumulation demands it.
WAGE constrains weights, activations, gradients, and errors to low-bitwidth integers during both training and inference.
Ternary weights, 8-bit activations, and 8-bit gradient accumulation are combined with constant scaling instead of batch normalization.
Orientation-preserved shifting exploits the relative values of errors, while higher bitwidth remains necessary for gradient-update accumulation.
Comparable accuracy across MNIST, CIFAR10, SVHN, and ImageNet is reported against methods discretizing weights and activations only at inference.
WAGE produces pure bidirectional low-precision integer dataflow and is presented as potentially suitable for training and inference in dedicated hardware.

2 RELATED WORK

Prior work reduced precision for weights, activations, or gradients, but generally retained floating-point training state and did not discretize the complete training graph.

Network compression, pruning, and compact architectures reduce complexity but are complementary techniques outside WAGE's scope.
Binary and low-precision weight or activation methods train with real-valued gradient accumulation, indicating that high-precision accumulation remains common.
DoReFa-Net and TernGrad quantize backward-pass gradients or updates, but weights remain stored and updated with float32 during training.

3 WAGE QUANTIZATION

WAGE quantization defines four low-bitwidth integer operands across inference and backpropagation, using linear mapping, shifting, saturation, and stochastic rounding for fixed-point dataflow.

WAGE quantizes weights and activations for inference, and errors and gradients for backpropagation training.Error is the activation gradient, whereas gradient denotes accumulated weight gradients.
Weight and activation gradients follow different paths within each layer, forming a MAC-related fork in inference and backward training.
Figure 1 summarizes four precision-reducing operators, while Figure 2 summarizes the quantization methods and notation used in WAGE.
The framework maps continuous values to uniformly spaced k-bit signed integers using linear quantization and saturation.Rounding selects nearby discrete states, while clipping bounds values to [−1 + σ, 1 − σ].
Shift(x) = 2^round(log2 x) adjusts operand magnitudes before quantization so values avoid saturation or being cleared.
Stochastic rounding replaces small real-valued gradient updates, constraining higher-bitwidth gradients with a 16-bit random-number generator.

3.2 WEIGHT INITIALIZATION

WAGE's weight initialization addresses failures caused by overly large or vanishing quantized weights by imposing limits linked to fan-in and minimum representable steps.

Direct binarization or ternarization can fail without batch normalization because ±1 weights are large for typical deep networks.
The initialization uses fan-in-dependent limits and a minimum uniform-distribution bound, with β > 1 creating overlap between minimum step and maximum value.
Small bitwidths or wide fan-in can produce all-zero tensors when initialized weights do not reach the minimum fixed-point step.

3.3 QUANTIZATION DETAILS

WAGE quantizes weights, activations, errors, and gradients through integer-compatible shifting and scaling. Its error quantization preserves orientations, while gradient shifting and stochastic rounding constrain updates to low-bitwidth integers.

3.3.1 WEIGHT QW (·): The modified initialization and layer-wise shift-based factor α attenuate amplified integer weights while approximating floating-point representations.α is predefined for each layer from the network structure and acts after activations to preserve weight precision.
3.3.2 ACTIVATION QA(·): Batch normalization is replaced by a constant scaling layer because hidden-layer batch outputs are hypothesized to be approximately zero-mean.The scaling parameters are replaced by the layer-wise α used for integer implementation.
3.3.3 ERROR QE(·): Error quantization preserves relative orientations rather than absolute magnitudes, removing the need for inverse transformation after quantization.Errors are first scaled by a shift factor and then quantized; values smaller than σ may be discarded.
3.3.4 GRADIENT QG(·): Gradient updates are rescaled and shifted by an integer-power-of-2 learning rate η before low-bitwidth quantization.Shifted gradients encode minimum update steps and directions for changing weights.
3.3.4 GRADIENT QG(·): Stochastic rounding separates shifted gradients into integer and decimal parts before constraining them to kG-bit integers.The Bernoulli sampler handles decimal parts, while clipping keeps updated weights within their representable range.

3.4 MISCELLANEOUS

WAGE simplifies the training pipeline to reduce storage and implementation demands for integer hardware. It removes or replaces components whose precision or memory requirements are difficult to support, while retaining comparable accuracy in later experiments.

3.4 MISCELLANEOUS: The quantization methods are integrated into a computation graph intended to support training with only integers.The framework first quantizes weights, activations, gradients, and errors before addressing the remaining training-process design choices.
3.4 MISCELLANEOUS: WAGE uses pure mini-batch SGD without momentum or adaptive learning rates to avoid storing gradient updates or moving averages.Those optimizer states can double weight-related memory consumption during training.
3.4 MISCELLANEOUS: WAGE treats quantization effects, including removed small values and stochastic updates, as certain types of regularization.L2 weight decay and dropout remain as supplementary regularization methods.
3.4 MISCELLANEOUS: For tasks with few categories, WAGE omits Softmax and mean reduction, using a sum-square-error criterion instead.The change avoids exponential computation and relies on shifted errors retaining the same values.

4 EXPERIMENTS

Experiments evaluate WAGE across multiple datasets, bitwidth settings, training variants, and hardware-oriented comparisons. The results examine accuracy, error and gradient precision requirements, and the effects of integer-only training choices.

Experimental setup: WAGE uses a default 2-8-8-8 configuration, with ternary weights and 8-bit activations, errors, and gradients.The 2-bit weight setting implies ternary weights and no multiplications during inference.
Datasets and models: WAGE is evaluated on MNIST, CIFAR10, SVHN, and ImageNet using CNN or AlexNet-based models.ImageNet experiments use six bitwidth and quantization patterns, including higher-precision and floating-point references.
Accuracy evaluation: Table 1 compares test or validation error rates for previous methods and WAGE across multiple datasets, including optimizer, normalization, and precision settings.The table reports ImageNet results in top1/top5 format and distinguishes float32 and batch-normalized configurations.
Training curves and regularization: WAGE variations and a vanilla CNN are compared on CIFAR10 using training curves, with the vanilla model retaining floating-point operations, batch normalization, and conventional optimization.The 28ff variation removes quantization nodes in backpropagation.
Bitwidth of errors: 4-8 bits of errors are sufficient for CIFAR10 classification, and 8 bits is selected as the default error width.The error-window experiments vary kE from 4 to 15 and use ten-run accuracy comparisons.
Bitwidth of errors: Large error values are rare but critical for backpropagation, whereas the majority of small error values act as noise; shifting the quantization window therefore affects training.The shifted window uses a right boundary formulated as max{|e|}/γ.
Bitwidth of gradients: Although ternary inference weights provide 16× compression relative to float32 weights, storing and accumulating them at 8 bits during training reduces overall compression to 4×.The higher training width provides buffer space for weight updates.
Bitwidth of gradients: Gradient experiments vary kG from 2 to 12 on CIFAR10 and compare ImageNet top-5 error rates across different kG and kE patterns.The CIFAR10 study adjusts the learning rate as kG changes to maintain approximately equal accumulated weight updates.

5 DISCUSSION AND FUTURE WORK

The discussion frames WAGE as a hardware-oriented low-bitwidth approach while identifying unresolved limitations in computation, quantization, and normalization. It also reports potential cost reductions for 8-bit integer operations relative to FP16.

Hardware implications: Relative to FP16, 8-bit integer operations are estimated to reduce IC energy and area costs by about 5× and halve memory-access costs and memory-size requirements during training.These estimates motivate mobile devices with on-site learning capability.
MAC operation: WAGE still requires multiply-accumulate operations to calculate training gradients, despite eliminating inference multiplications with ternary weights.The framework is mainly tested with the 2-8-8-8 configuration.
MAC operation: Using ternary activations could dramatically slow convergence and hurt accuracy because Q(x, 2) clears most layer outputs early in training.The same phenomenon was observed in the authors’ BNN replication.
Non-linear quantization: WAGE uses uniform linear quantization for simplicity, while logarithmic representations may provide greater value range with fewer bits for naturally logarithmic-normal weights and activations.The paper identifies logarithmic integer encoding as a promising direction for future training methods.
Normalization: Some WAGE demonstrations avoid or remove Softmax and batch normalization, but the paper considers normalization important for multi-modal inputs and cross-branch feature integration.It calls for improved quantization methods for normalization layers.

6 CONCLUSION

WAGE enables pure low-bitwidth integer dataflow for both training and inference, while simplifying normalization and other training components. It achieves state-of-the-art accuracies on multiple datasets with a 2-8-8-8 bitwidth configuration and leaves further compression opportunities in training.

The method replaces batch normalization with layer-wise constant scaling and simplifies other training components through alternative solutions.The conclusion identifies batch normalization as a quantization pain point and notes that several training components are simplified.
Weight-update accumulation remains indispensable for stable convergence and final accuracy, while training compression and memory reduction remain open opportunities.
WAGE enables pure low-bitwidth integer dataflow for both training and inference.The framework targets integer-based lightweight ASIC or FPGA implementations with on-site learning capability.
WAGE achieves state-of-the-art accuracies on multiple datasets with a 2-8-8-8 bitwidth configuration.

A ALGORITHM

The algorithm assumes network structures are defined and initialized before training, then supports implementation on floating-point or integer-based devices. Quantized data and saved weights are supplied with layer-specific shifts and a learning-rate schedule.

The WAGE training algorithm assumes that network structures are defined and initialized with Equation 5.The pseudo-code annotations identify potential corresponding operations for fixed-point dataflow implementation.
Training can run on floating-point-based or integer-based devices using quantized weights, activations, gradients, and errors.The algorithm quantizes these quantities according to Equations 6–12.
The procedure requires quantized mini-batch inputs and targets, layer-specific shift-based α values, a learning-rate scheduler η, and previous weights saved in kG bits.

1. Forward propagation: 1: for i = 1 to I do 2: W i

The algorithm’s forward and backward propagation uses quantization, clipping, MAC operations, and shifts, while Figure 5 visualizes layerwise distributions under a 2-8-8-8 configuration.

Forward propagation: Forward propagation applies MAC, shift, and clipping operations before quantizing activations with QA.
Back propagation: Back propagation computes errors from the loss derivative and processes layers in reverse order.
Back propagation: Gradient updates use MAC, maximum, shift, randomization, and clipping operations before weights are updated according to Equation 12.
Figure 5 shows layerwise histograms for a trained VGG-like network using a 2-8-8-8 bitwidth configuration and learning rate η = 8.The y-axis represents probability for weights and gradients, and logarithmic probability for activations and errors.