Source-linked AI summary
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Urs Köster, Tristan J. Webb, Xin Wang, Marcel Nassar, Arjun K. Bansal, William H. Constable, Oğuz H. Elibol, Scott Gray, Stewart Hall, Luke Hornof, Amir Khosrowshahi, Carey Kloss, Ruby J. Pai, Naveen Rao
TL;DR
Low-bit-width training needs numerical formats that reduce computational resources while preserving full-precision behavior across modern deep networks. The paper presents Flexpoint, which uses dynamically managed shared exponents, and shows that 16-bit Flexpoint matches 32-bit floating point across several models without hyperparameter changes.
Problem
Low-bit-width neural-network training remains challenging, despite potential hardware gains from numerical formats tailored to deep learning.
Method
Flexpoint combines fixed-point and floating-point arithmetic by assigning each tensor a dynamically managed shared exponent to reduce computational and memory requirements.
Results
16-bit Flexpoint achieved numerical parity with 32-bit floating point across several deep-learning models without modifying models or hyperparameters, outperforming 16-bit floating point under the same conditions.
Takeaways & Limitations
Flexpoint is a promising numerical format for specialized hardware supporting training and inference of existing deep-learning models.
Takeaways & Limitations
Flexpoint may provide lower performance and efficiency than more aggressive quantization schemes, and other Flexpoint-family hardware formats remain unexplored.
Abstract
from arXiv · showhide
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet, a deep residual network and a generative adversarial network, using a simulator implemented with the neon deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.
1 Introduction
The paper motivates numerical formats tailored to deep learning because low-bit training remains challenging despite progress in low-bit inference. Flexpoint is proposed to address this gap while preserving full-precision training behavior and supporting existing network topologies.
- More efficient data formats could reduce hardware footprint, power consumption, speed, and memory requirements.
- Low-bit training remains an open challenge, whereas low-bit fixed-point inference has advanced substantially.
- Flexpoint is presented as a flexible low-bit-width format intended to maintain parity with full-precision floating-point training across diverse deep network topologies.
- The proposed evaluation trains AlexNet, a ResNet, and a Wasserstein GAN, comparing Flexpoint with floating-point approaches.
2 Related Work
Prior work established substantial progress in low-precision inference and partial success in low-precision training, but existing approaches often rely on model-dependent quantization or higher-precision accumulation. Flexpoint is positioned against dynamic fixed-point methods that react to overflows only after they occur.
- Low-precision inference has reached 8-bit, 4-bit, 2-bit, ternary, and binary representations, often through model-dependent tensor quantization.
- Low-precision training has used binarized or ternarized weights, binarized gradients, logarithmic quantization, and higher-precision accumulation.
- DoReFa-Nets quantize gradients to 6 bits while reporting performance comparable to related quantized neural-network approaches.
- Dynamic fixed-point training adjusts exponents after monitoring overflow fractions, whereas Flexpoint-related criticism identifies this as reactive rather than anticipatory.
3 Flexpoint
Flexpoint combines integer mantissas with a tensor-shared exponent that is automatically managed to preserve useful dynamic range. Its design reduces hardware costs but depends on narrow tensor ranges and accurate prediction of future extrema.
- 3.1 The Flexpoint Data Format: Flexpoint stores an N-bit two’s-complement mantissa and an M-bit exponent shared across every tensor element.
- 3.1 The Flexpoint Data Format: Shared exponents reduce memory and bandwidth requirements, while fixed-point multiplication and simpler multipliers reduce hardware power and area requirements.
- 3.1 The Flexpoint Data Format: The design trades these hardware benefits for added exponent-management complexity and dynamic-range limitations.
- 3.1 The Flexpoint Data Format: A flex16+5 tensor uses a 16-bit mantissa and 5-bit shared exponent instead of per-element floating-point exponents.
- 3.2 Exponent Management: Faithful floating-point representation requires sufficiently narrow tensor ranges and sufficient mantissa overlap when combining tensors during training.
- 3.4 Autoflex Initialization: Autoflex predicts output exponents from historical tensor-wide statistics and adjusts them preemptively to prevent overflow or improve range utilization.
- 3.4 Autoflex Initialization: Autoflex initialization repeatedly performs each operation, increasing the exponent for overflow or decreasing it when the available bits are underutilized.
- 3.6 Autoflex Example: In an example, a sliding window of 16 values predicted an exponent-boundary crossing about 20 iterations early, preventing overflow while sometimes leaving unused mantissa bits.
4 Experimental Results
Flexpoint training closely matched float32 across convolutional and generative models, while float16 underperformed under unchanged hyperparameters. The experiments also compare Flexpoint’s numerical-parity design with more aggressive low-precision approaches.
- Convolutional Networks: Learning curves for AlexNet and a 110-layer ResNet matched closely between flex16+5 and float32, whereas unchanged-hyperparameter float16 substantially underperformed.The networks were evaluated using validation-set misclassification error; the ResNet was trained on CIFAR-10 for 165 epochs.
- Generative Adversarial Networks: WGAN Wasserstein-distance estimates in flex16+5 closely tracked float32, while float16 deviated significantly from the float32 baseline.The three formats used exactly the same hyperparameter settings.
- Generative Adversarial Networks: Flexpoint and float32 produced no differences in generated-image quality at specific training stages according to FID, while float16 produced significantly higher FIDs.The WGAN used the LSUN bedroom dataset, and FID was estimated from 5000 generator samples.
- Discussion: Flexpoint maintains numerical parity with high-precision floating-point training, unlike below-8-bit schemes that significantly alter quantitative behavior and require different training algorithms.This design preserves prior network-design and hyperparameter knowledge and avoids intermediate high-precision representations except commonly needed higher-precision accumulation.
- Discussion: Flexpoint’s Autoflex algorithm manages tensors uniformly and accommodates diverse deep-network topologies without model-specific hyperparameter tuning.The format uses shared tensor exponents managed automatically rather than user-specified update intervals.
6 Conclusion
The conclusion reports that flex16+5 achieved float32-level training behavior across several models without modifying model architectures or hyperparameters, while outperforming float16 under the same conditions. It presents Flexpoint and Autoflex as promising directions for specialized deep-learning hardware.
- Conclusion: 16-bit mantissa plus 5-bit shared-exponent Flexpoint achieved numerical parity with 32-bit floating point across several deep-learning models without model or hyperparameter modifications.The reported comparison also includes better performance than 16-bit floating point under the same conditions.
- Conclusion: Flexpoint and Autoflex are presented as a promising foundation for future research on specialized hardware for deep-neural-network training.The paper frames Flexpoint as a numerical format that complements hardware designed for greater compute capability and lower memory and power consumption.