PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Deep Neural Networks with 8-bit Floating Point Numbers

Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, Kailash Gopalakrishnan

arXiv:1812.08011v1cs.LGstat.ML

TL;DR

Training DNNs below 16-bit precision challenges gradient fidelity and accuracy. This paper introduces mixed-precision FP8 training techniques that preserve FP32-baseline accuracy across benchmarks while enabling 2−4× speedup.

  • Problem

    Reduced-precision DNN training below 16 bits must preserve gradient fidelity, while large models require substantial computation and storage.

  • Method

    The paper combines an FP8 format, chunk-based accumulation, and floating-point stochastic rounding to use 8-bit multiplications and 16-bit additions during training.

  • Results

    Across a wide spectrum of DNN benchmarks and datasets, FP8 training achieves the same accuracy as the FP32 baseline, with 2−4× speedup.

  • Takeaways & Limitations

    The techniques open opportunities for training hardware platforms with 2−4× improved energy efficiency and throughput over state-of-the-art systems.

Abstract

from arXiv · show

The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems.

1 Introduction

Reduced precision improves Deep Learning hardware efficiency, but training below 16 bits is difficult because back-propagated gradients require high fidelity. This paper introduces techniques enabling accurate DNN training with 8-bit floating point representations and reduced-precision additions.

  • Motivation: Quadratic energy-efficiency gains from lower bit precision motivate reduced-precision representations and computation for costly DNN training.DNN training can require up to 100s of ExaOps and gigabytes of storage.
  • Challenge: Inference has reached 2-4-bit precision with mostly preserved accuracy, whereas training below 16 bits remains challenging because gradient fidelity must be maintained during back-propagation.Recent studies reported that at least 16 bits are necessary for training without impacting model accuracy.
  • Challenge: Using 8-bit operands for GEMM and convolution computations causes noticeable accuracy degradation in many DNNs.The challenge is illustrated for ResNet18 training on ImageNet.
  • Approach: The paper introduces an FP8 format, chunk-based computations, and floating point stochastic rounding to reduce representations, multiplications, additions, and weight updates while preserving accuracy.The proposed scheme uses 8-bit multiplications, 16-bit additions, and 16-bit weight updates.
  • Implication: 2-4× improved energy efficiency and throughput over state-of-the-art training systems is the hardware opportunity enabled by these techniques.The claim concerns opportunities for future hardware platforms.

2 8-bit floating point training

The paper enables DNN training with FP8 representations by addressing accumulation errors and rounding losses that make reduced-precision training difficult. FP8 data and multiplications, FP16 accumulation and updates, and specialized numerical techniques are combined to preserve accuracy.

  • Related work: Earlier reduced-precision training methods generally retained at least 16-bit data or computation precision, often accumulating partial products in 32-bit arrays.Inference had reached lower bit-widths, but training required greater gradient fidelity during back-propagation.
  • FP8 and FP16 formats: The proposed FP8 format uses 1 sign, 5 exponent, and 2 mantissa bits, while FP16 uses 1, 6, and 9 bits for accumulation and AXPY additions.The formats were selected to balance representation range and precision for weights, activations, errors, gradients, and weight updates.
  • Floating-point accumulation: Swamping truncates smaller addends during floating-point addition and becomes severe with non-zero-mean accumulations, long-tailed values, or aggressively reduced accumulation precision.Under FP16 accumulation, truncation occurs when magnitudes differ beyond the swamping threshold 2^(mantissa+1).
  • Floating-point accumulation: Chunk-based accumulation divides long dot-products into smaller chunks and hierarchically combines partial sums, reducing theoretical error bounds from O(N) to O(N/CL + CL).The method reduces large-to-small additions with little additional computational and memory overhead.
  • Floating-point accumulation: Floating-point stochastic rounding preserves information from discarded mantissa bits when reducing precision, complementing chunk-based accumulation for weight-update AXPY operations.Its rounding error is proportional to 2^e, and the paper reports robust numerical and empirical behavior for DNNs.
  • Floating-point accumulation: FP16 accumulation with nearest rounding suffers significant error, whereas chunk sizes of at least 32 and stochastic rounding follow the FP32 baseline.The comparison is reported for reduced-precision accumulation across different chunk sizes and rounding modes.

3 Experimental Results

The FP8 training scheme was evaluated across diverse CNNs and DNNs, datasets, and optimizers, using FP8 GEMMs with FP16 accumulation and updates. Across the tested networks, it matched FP32 baseline errors while reducing weight memory footprint.

  • Experimental setup: FP8 operands and multiplication with FP16 accumulation, chunk size 64, were used for the three GEMM computations.AXPY computations used FP16 with floating point stochastic rounding, and loss scaling preserved small back-propagated errors.
  • Experimental setup: The evaluation covered six CNN and DNN configurations spanning image and speech classification across multiple datasets.The tested models included CIFAR10-CNN, CIFAR10-ResNet, ImageNet-ResNet18, ImageNet-ResNet50, ImageNet-AlexNet, and BN50-DNN.
  • Results: Every tested network achieved almost identical test errors to the FP32 baseline without changing architectures, preprocessing, or hyper-parameters.CIFAR10-CNN also reached baseline accuracy with ADAM while using FP8 GEMMs and FP16 weight updates.
  • Results: 2× lower weight memory footprint was achieved through FP8 weights and an FP16 master copy.The reported reduction applies to both the weights and the master copy.
  • Results: Equivalent top-1 accuracy to prior state-of-the-art reduced-precision training was achieved for AlexNet on ImageNet using half the bit-precision for representations and accumulations.The comparison reports top-1 accuracy in percent.
  • Results: Reliable convergence across the model and dataset spectrum was obtained with FP8 representations and FP16 accumulation and updates.The convergence results used chunk size 64.

4 Discussion & Insight

The discussion identifies precision-sensitive layers and accumulation errors as central constraints in FP8 training, then presents targeted choices that preserve accuracy while reducing arithmetic precision. Chunking and stochastic rounding are especially important for retaining gradient information and convergence.

  • 4.1 Bit-Precisions for First and Last Layer: FP16 input images were retained because FP8 lacks enough mantissa bits to represent intensity values from 0 to 255.This choice was particularly critical for high ImageNet accuracy with ResNet18 and ResNet50.
  • 4.1 Bit-Precisions for First and Last Layer: The last layer remained in FP16 for robust training because FP8 quantization caused noticeable degradation, while its computation is below 1% of FLOPS.Preserving the last-layer Forward GEMM in FP16 could also support high accuracy in the reported AlexNet experiment.
  • 4.2 Accumulation Error: Chunk-based accumulation restored ResNet50 convergence to baseline after FP8 training failed without chunking.The technique addresses information loss in long dot-products while keeping accumulation at FP16.
  • 4.2 Accumulation Error: Gradient GEMM was the most accumulation-sensitive operation: without chunking, test error diverged to 99% despite converged training loss.Small gradients across minibatch samples can be lost through swamping, impairing generalization.
  • 4.2 Accumulation Error: Chunk sizes between 64 and 256 produced Gradient GEMM results closest to the FP32 baseline for CIFAR10-ResNet.The comparison used normalized L2-distance between reduced-precision and full-precision results.
  • 4.3 Weight Updates: Stochastic rounding preserved baseline accuracies for FP16 weight updates, whereas nearest rounding degraded AlexNet and ResNet18 accuracy by 2–4%.The experiment kept GEMM in FP32 to isolate the impact of weight-update rounding.
  • Hardware Implementation: A 14 nm dataflow core implemented chunk-based FP16 accumulation with FP8 engines that were 2–4× more efficient than FP16 implementations.Chunk-based computation added less than 5% energy overhead for chunk sizes above 64.

5 Conclusions

The paper demonstrates FP8 DNN training at FP32-baseline accuracy across diverse benchmarks by using chunk-based accumulation and stochastic rounding to support FP16 additions and updates. The resulting approach reports 2–4× speedup while preserving accuracy.

  • 5 Conclusions: 2–4× speedup was achieved without compromising accuracy in FP8 DNN training.Across a wide spectrum of benchmarks and datasets, mixed-precision training reached the same accuracy levels as the FP32 baseline.
  • 5 Conclusions: Chunk-based accumulation and floating point stochastic rounding reduced addition precision to 16 bits for partial-product accumulation and weight updates.These techniques address swamping errors that can degrade training accuracy.

A Network Architectures

The evaluation used six network configurations spanning convolutional and fully connected architectures, with model depth and output dimensions varying across CIFAR10, ImageNet, and speech-oriented benchmarks.

  • Network Architectures: CIFAR10-CNN uses three convolutional layers, one fully connected layer, and a 10-way Softmax.The convolutional layers use 5x5 filters and ReLU activation.
  • Network Architectures: CIFAR10-ResNet contains 31 convolutional layers across 15 ResNet blocks and ends with a 1K Softmax.Its layers use 3x3 filters, batch normalization, and ReLU activation.
  • Network Architectures: AlexNet contains five convolutional layers and three fully connected layers with a 1K Softmax output.
  • Network Architectures: ResNet18 contains 16 convolutional layers across eight ResNet blocks and ends with a 1K Softmax.The network uses 3x3 filters, batch normalization, and ReLU activation.
  • Network Architectures: ResNet50 contains 48 convolutional layers across 16 bottleneck ResNet blocks and ends with a 1K Softmax.Its filters are 3x3 or 1x1, with batch normalization and ReLU activation.
  • Network Architectures: BN50-DNN uses six fully connected layers, including a 1024x1024 stack, and a 5999-way Softmax.

B Datasets •

The paper evaluates its approach on image and speech benchmarks spanning CIFAR10, ImageNet, and BN50. These datasets cover image classification at different scales and English broadcast-news speech recognition.

  • CIFAR10 contains 32x32-pixel RGB images, with 50K training and 10K test examples.
  • ImageNet is a 1000-category object-classification benchmark with over 1.2M training and 50K validation images.Images are resized to 256x256 and randomly cropped to 224x224 before network input.
  • BN50 is an English Broadcast News speech-recognition benchmark with a 45-hour training set and 5-hour holdout set.
Loading 1812.08011v1…