Source-linked AI summary
Mixed Precision Training
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu
TL;DR
Training larger neural networks increases memory and compute demands, motivating reduced-precision training that preserves accuracy. The paper introduces FP16 mixed-precision techniques and demonstrates accuracy matching across diverse models, with 2-6x speedups for bandwidth-limited operations. Full-network speedups remain dependent on library and framework optimizations.
Problem
Training neural networks with growing model sizes requires increasing memory and compute resources, while reduced precision must maintain model accuracy.
Method
The method trains networks with FP16 tensors and arithmetic while maintaining FP32 master weights, scaling losses, and accumulating arithmetic in FP32.
Results
Mixed-precision training matches FP32 accuracy across diverse architectures and applications, while DeepBench operations on Volta GPUs achieve 2-6x speedups when limited by memory or arithmetic bandwidth.
Takeaways & Limitations
FP16 mixed precision can reduce deep-learning memory and arithmetic or memory-operation costs without accuracy loss across the demonstrated models.
Takeaways & Limitations
Full-network training and inference speedups depend on library and framework optimizations, which were early versions in these experiments.
Abstract
from arXiv · showhide
Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulates the gradients after each optimizer step. This single-precision copy is rounded to half-precision format during training. Secondly, we propose scaling the loss appropriately to handle the loss of information with half-precision gradients. We demonstrate that this approach works for a wide variety of models including convolution neural networks, recurrent neural networks and generative adversarial networks. This technique works for large scale models with more than 100 million parameters trained on large datasets. Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x. In future processors, we can also expect a significant computation speedup using half-precision hardware units.
1 INTRODUCTION
As neural networks and training datasets grow, training demands more memory and computation. The paper addresses reduced-precision training while maintaining model accuracy using FP16 and supporting techniques.
- Larger models require increasing compute and memory resources to train.
- Reduced precision lowers memory-bandwidth pressure by storing values with fewer bits and can reduce arithmetic time on suitable processors.Recent GPUs provide half-precision throughput 2× to 8× higher than single-precision.
- The paper trains neural networks with IEEE FP16 while addressing its narrower dynamic range relative to FP32.
- FP32 master weights, loss scaling, and FP32 accumulation of FP16 products are used to prevent accuracy loss.The approach is reported across convolutional and recurrent architectures and classification, regression, and generative tasks without changing models or training hyper-parameters.
2 RELATED WORK
Prior reduced-precision methods quantized selected tensors or recurrent components, sometimes with accuracy loss or unclear scalability. This paper instead uses FP16 throughout forward and backward computation while matching FP32 baseline accuracy without hyper-parameter adjustment.
- Earlier methods binarized or quantized CNN weights, activations, and sometimes gradients using fewer than full-precision bits.
- Prior RNN approaches reduced weight and activation precision, but some reported small accuracy losses or uncertain applicability to larger networks.
- The proposed method uses reduced precision for all tensors and arithmetic in forward and backward passes.
- Models trained with the proposed techniques match single-precision baselines without adjusting hyper-parameters across varied applications and large-scale datasets.
3 IMPLEMENTATION
The implementation stores training tensors in FP16 while retaining selected FP32 computations and a master weight copy. It addresses underflow and accumulation errors through loss scaling, FP32 updates, and FP32 accumulation where needed.
- 3.1 FP32 MASTER COPY OF WEIGHTS: FP32 master weights are updated by gradients, while an FP16 copy is used for each forward and backward pass.The FP16 copy halves storage and bandwidth needed by FP32 training for those weights.
- 3.1 FP32 MASTER COPY OF WEIGHTS: Approximately 5% of weight-gradient values have exponents smaller than −24 and would become zero in FP16 after multiplication by the learning rate.The master copy preserves these updates in FP32.
- 3.1 FP32 MASTER COPY OF WEIGHTS: Updating an FP32 master copy matched FP32 Mandarin speech results, whereas updating FP16 weights caused 80% relative accuracy loss.
- 3.1 FP32 MASTER COPY OF WEIGHTS: Overall training memory consumption is roughly halved because activations, which dominate training memory, are stored in half precision.The additional master copy increases weight memory requirements by 50% compared with single-precision training, but has a smaller overall effect.
- 3.2 LOSS SCALING: Loss scaling shifts small gradients into the FP16 representable range, preserving values that would otherwise become zeros.For the Multibox SSD example, gradients in the [2−27, 2−24) range were important to preserve, while values below 2−27 were irrelevant.
- 3.2 LOSS SCALING: Scaling the forward-pass loss scales all back-propagated gradients, which are unscaled before weight updates and gradient-related computations.This preserves FP32 update magnitudes without changing hyper-parameters such as gradient-clipping thresholds or weight decay.
- 3.3 FP16 ARITHMETIC WITH FP32 ACCUMULATION: Some FP16 vector dot-products require FP32 accumulation, and large reductions are performed in FP32 to maintain model accuracy.Point-wise operations may use either FP16 or FP32 arithmetic because they are memory-bandwidth limited.
4 RESULTS
The experiments evaluate mixed-precision training across classification, detection, speech recognition, translation, language modeling, and GANs. With FP32 master weights and, where needed, loss scaling, mixed precision generally matches FP32 training, while preserving accuracy across varied models and tasks.
- 4.1 CNNS FOR ILSVRC CLASSIFICATION: Mixed precision matched FP32 top-1 accuracy across AlexNet, VGG-D, GoogLeNet, Inception v2, Inception v3, and pre-activation ResNet-50 on ILSVRC classification.The experiments used identical hyper-parameters for the mixed-precision and baseline training sessions.
- 4.1 CNNS FOR ILSVRC CLASSIFICATION: Loss-scaling was unnecessary for the evaluated classification networks, while FP32 master weights were updated during training.Forward and backward tensors were stored in FP16, with the master weight copy updated in FP32.
- 4.2 DETECTION CNNS: FP16 SSD training diverged without loss-scaling, but a scaling factor of 8 recovered relevant gradients and matched FP32 mAP.The failure was attributed to small gradient values becoming zeros during FP16 training.
- 4.3 SPEECH RECOGNITION: Mixed precision worked for recurrent speech models with approximately 115 million English parameters and 215 million Mandarin parameters; pseudo-FP16 results were roughly 5 to 10% better than baseline.The English model used 6000 hours of speech data, and the authors suggest half-precision storage may act as a regularizer.
- 4.4 MACHINE TRANSLATION: Mixed precision with loss-scaling matched FP32 results for English-to-French translation, whereas omitting loss-scaling caused slight degradation.The same training behavior appeared in both 3-layer and 5-layer encoder-decoder models.
- 4.5 LANGUAGE MODELING: For the bigLSTM language model, loss-scaling was required: without it FP16 perplexity diverged after 300K iterations, while a scaling factor of 128 matched the baseline.The model was trained on the 1 billion word dataset for 50 epochs.
5 CONCLUSIONS AND FUTURE WORK
Mixed precision training reduces memory use and time spent in memory and arithmetic operations while preserving accuracy across many deep learning models. The paper reports operation-level speedups under bandwidth limits and identifies library, framework, and application coverage as remaining work.
- Mixed precision training reduces deep neural networks’ memory consumption and time spent in memory and arithmetic operations without loss in accuracy.The authors report this result across many models and without hyper-parameter tuning.
- 2-6x speedups over FP32 implementations are observed for DNN operations limited by memory or arithmetic bandwidth on a Volta GPU.Speedups are lower for latency-limited operations.
- Full-network training and inference speedups depend on library and framework optimizations for mixed precision.The experiments used early versions of both libraries and frameworks.
- The authors identify extending mixed precision training to generative models and deep reinforcement learning as future work.They also propose automating loss-scale selection using gradient-overflow checks and skipped updates.