Source-linked AI summary
FP8 Formats for Deep Learning
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, Hao Wu
TL;DR
Growing deep-learning models require substantial training and inference resources, motivating lower-precision representations that preserve result quality. The paper proposes E4M3 and E5M2 FP8 formats and evaluates simulated FP8 training across architectures and tasks, reporting matches to 16-bit results, including a 175B-parameter language model.
Problem
Increasing model sizes and compute requirements motivate lower-precision representations that can accelerate deep-learning training and inference while preserving result quality.
Method
The paper proposes E4M3 and E5M2 FP8 encodings and evaluates simulated FP8 training by clipping GEMM inputs, applying scaling, and retaining baseline architectures and hyperparameters.
Results
FP8 training matches 16-bit training results across image, translation, and language tasks, including a language model with up to 175B parameters.
Takeaways & Limitations
FP8 can support training across CNN, recurrent, and Transformer-based models while using the same model, optimizer, and training hyperparameters as higher-precision baselines.
Abstract
from arXiv · showhide
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for representatio of special values, E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions. Our study covers the main modern neural network architectures - CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters unchanged from the 16-bit baseline training sessions. Our training experiments include large, up to 175B parameter, language models. We also examine FP8 post-training-quantization of language models trained using 16-bit formats that resisted fixed point int8 quantization.
1 Introduction
The paper motivates FP8 as an 8-bit route to reduce deep-learning training and inference costs as models and compute demands grow. It proposes two encodings and evaluates them across architectures and tasks, including models up to 175B parameters.
- Motivation: Growing model sizes and compute demands have made deep-learning training increasingly resource-intensive.Large language models can require weeks of training on thousands of processors.
- Motivation: Reduced-precision representations are central to accelerating deep-learning training and inference, but extreme bit reduction has not consistently preserved practical result quality.The introduction contrasts common floating-point formats with unsuccessful 1-bit networks and challenging int8 inference cases.
- FP8 approach: FP8 reduces training compute requirements and may improve inference relative to int8 through nonlinear sampling of real numbers.Prior work had investigated 5-bit-exponent formats for CNNs, translation networks, and other models.
- Contribution: The paper proposes two FP8 encodings and evaluates training and inference across a wide range of image, speech, and language models.The study covers CNNs, recurrent and Transformer-based models, while retaining baseline hyperparameters.
- Contribution: FP8 training matches FP16 or bfloat16 results across varied tasks, architectures, and model sizes without changing model or optimizer hyperparameters.The evaluation includes language models with up to 175B parameters.
2 Aspects of FP8 Usage in Deep Learning
FP8 usage combines higher-precision arithmetic with FP8 storage and conversion, using scaling factors to align values with each format’s representable range. The approach focuses on the interchange format while accommodating tensor-specific dynamic-range needs.
- Arithmetic and conversion: FP8 inputs are expected to produce higher-precision outputs, with results optionally converted back to FP8 before memory writes.This mirrors common FP16 and bfloat16 practice in which matrix operations produce single-precision outputs.
- Scaling: Scaling factors move higher-precision values into ranges that overlap better with the corresponding FP8 format.The role is analogous to loss scaling in mixed-precision FP16 training.
- Scaling: Some networks require per-tensor scaling because FP8 dynamic range cannot cover the union of important values across all tensors.The passage distinguishes tensor-specific scaling needs from the binary format itself.
- Arithmetic and conversion: Type conversion is orthogonal to the binary format, but special values from wider types are mapped to corresponding FP8 special values.For E4M3, wider-type infinities and NaNs become FP8 NaNs during conversion.
3 FP8 Binary Interchange Format
The FP8 interchange format uses E4M3 and E5M2, assigning different exponent and mantissa layouts to support deep-learning tensors with different dynamic-range needs. It largely follows IEEE-754 conventions while extending E4M3’s range by repurposing special-value encodings.
- 3 FP8 Binary Interchange Format: FP8 consists of E4M3 and E5M2 encodings, named for their exponent and mantissa bit counts; E4M3 is recommended for weights and activations, E5M2 for gradients.Some networks can train with only one of the two formats.
- 3 FP8 Binary Interchange Format: S.E.M notation represents the sign bit, biased exponent field, and mantissa field in FP8 encoding tables.The exponent field has 4 or 5 bits, while the mantissa has 3 or 2 bits.
- 3 FP8 Binary Interchange Format: E4M3 has exponent bias 7, maximum normal magnitude 448, minimum normal magnitude 2^-6, and maximum subnormal magnitude 0.875 ∗ 2^-6.The table passage lists these values for E4M3.
- 3 FP8 Binary Interchange Format: E5M2 follows IEEE-754 conventions for exponents and special values, enabling straightforward conversion between E5M2 and IEEE FP16.Its five exponent bits provide sufficient per-tensor dynamic range according to the empirical studies.
- 3.1 Special value representations: E4M3 extends dynamic range by omitting infinities and retaining one NaN mantissa pattern, increasing the range from 17 to 18 binades.This adds representable magnitudes 256 through 448 while preserving positive and negative zero and NaN representations.
- 3.1 Special value representations: E4M3 preserves IEEE-like sign symmetry because gaining magnitude 480 would complicate implementations that compare and sort values using integer operations.The paper judges the extra maximum value less beneficial for deep learning than retaining this symmetry.
- 3.2 Exponent bias: Both formats retain IEEE-like exponent biases, but experiments motivate per-tensor adjustment rather than one shared exponent bias for every tensor.Scaling factors provide the same range-placement effect as exponent bias.
4 Empirical Results
FP8 training matched higher-precision results across image classification, translation, and language modeling while retaining baseline architectures and optimizer hyperparameters. Inference results depended on tensor coverage and scaling: limited GEMM-input quantization tolerated shared exponent biases, whereas broader quantization required per-tensor calibration.
- Training setup: FP8 training retained the baseline model architectures, initializations, and optimizer hyperparameters while simulating FP8 inputs for GEMM operations.Arithmetic was performed in FP16 or bfloat16 after FP8 conversion and clipping.
- Image classification: ImageNet top-1 accuracy with FP8 training stayed within run-to-run variation of higher-precision training for all listed networks except MobileNet v2.The study included CNNs and the Transformer-based DeiT architecture.
- Language translation: English→German translation scores for Transformer and LSTM-based GNMT models were within run-to-run variation of the higher-precision baselines.Evaluation used sacreBLEU on WMT 2016 English→German models evaluated on newstest2014.
- Language modeling: FP8 language-model training matched 16-bit training results across the evaluated models, including a 175B-parameter model whose perplexity was reported at 75% training.The 175B model’s bfloat16 baseline had not yet completed, while its FP8 loss curve was consistent with successful training.
- Inference and scaling: For GEMM-only FP8 inference, exponent biases from 7 to 10 matched the bfloat16 baseline, but quantizing residual connections made shared bias insufficient.With residuals quantized, exponent bias 7 produced perplexity 12.59 versus 10.19 for the bfloat16 baseline.
- Inference and scaling: Per-channel weight and per-tensor activation calibration achieved perplexities of 10.29 for GEMM-only and 10.44 for GEMM-plus-residual FP8 inference.These calibrated results were close to the bfloat16 baseline of 10.19 reported for the same evaluation.
5 Conclusions
The proposed FP8 format uses E4M3 and E5M2 encodings to accelerate deep learning training and inference while matching 16-bit model accuracy.
- FP8 consists of E4M3 and E5M2 encodings designed to accelerate deep learning training and inference.The format enables smaller, more power-efficient math pipelines and reduces memory-bandwidth pressure.
- FP8 training matches 16-bit training accuracy across diverse image and language models using the same model, optimizer, and training hyperparameters.The experiments cover a wide variety of neural network models and tasks.
- Using FP8 for both training and inference simplifies 8-bit deployment by avoiding the calibration or fine-tuning previously required for int8 models.Those int8 deployment steps added complexity and sometimes failed to maintain accuracy.