Source-linked AI summary
To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability
Joonhyung Lee, Jeongin Bae, Byeongwook Kim, Se Jung Kwon, Dongsoo Lee
TL;DR
LLM pretraining’s computational cost motivates reduced-precision training, but prior FP16 instability raises questions about FP8’s cost-effectiveness. The paper evaluates stability across reduced precision and proposes a sharpness metric; it finds stability decreases as bit widths are reduced and that MS-AMP lacks sufficient robustness for economical use.
Problem
Whether FP8 training can reduce LLM training costs while maintaining stability comparable to higher-precision training remains insufficiently quantified.
Source: §1 · pp. 1–2Method
The authors evaluate reduced-precision stability by emulating lower-precision mantissas and propose a loss-landscape sharpness metric that searches the last-token logit space.
Source: §1 · p. 2, §3.1 · p. 4, §5 · p. 8Results
Takeaways & Limitations
Cost savings from reduced precision depend on keeping training-step counts similar, so stability is a crucial factor in cost-effective LLM training.
Source: §1 · p. 1The experiments use training loss rather than real-world natural-language task scores, although the authors consider loss divergence sufficient to indicate training failure.
Source: §6 · p. 8
Abstract
from arXiv · showhide
The massive computational costs associated with large language model (LLM) pretraining have spurred great interest in reduced-precision floating-point representations to accelerate the process. As a result, the BrainFloat16 (BF16) precision has become the de facto standard for LLM training, with hardware support included in recent generations of accelerators. This trend has gone even further in the latest processors, where FP8 has recently been introduced. However, prior experience with FP16, which was found to be less stable than BF16, raises concerns as to whether FP8, with even fewer bits than FP16, can be a cost-effective option for LLM training. We argue that reduced-precision training schemes must have similar training stability and hyperparameter sensitivities to their higher-precision counterparts in order to be cost-effective. However, we find that currently available methods for FP8 training are not robust enough to allow their use as economical replacements. This prompts us to investigate the stability of reduced-precision LLM training in terms of robustness across random seeds, learning rates, and datasets. To this end, we propose new evaluation techniques and a new metric for quantifying loss landscape sharpness in autoregressive language models. By simulating incremental bit reductions in floating-point representations, we analyze the relationship between representational power and training stability with the intent of aiding future research into the field.
1. Introduction
The introduction argues that reduced precision is attractive for lowering LLM training costs, but its stability must be comparable to higher precision for those savings to hold. It motivates broader robustness tests with BF16 instability findings and contributions that examine precision, hyperparameters, data, and loss-landscape sharpness.
1. Introduction
- Reduced precision is pursued for cost savings, but extra hyperparameter searches or restarts can outweigh faster computation if training stability or step counts worsen.The authors frame the question as whether FP8 is profitable, not whether FP8 pretraining is possible.Source: §1 · pp. 1–2
- Prior FP16 experience raises concern about FP8, while the authors report a narrower stable hyperparameter space at higher learning rates or with “dirtier” data.They motivate this investigation by noting that FP16 is less stable than BF16 despite gradient scaling and other techniques.Source: §1 · p. 2
- 18 of 188 BF16 runs diverged, while none of 70 TF32 runs did, despite matching configurations and training for only 5% of the original run.The authors suspect the approximately 10% BF16 divergence rate underestimates the full-training rate; the figure reports early stopping at 30K steps.Source: §1 · p. 2, Figure 1 · p. 2
- The authors propose a loss-landscape sharpness metric to predict training divergence even before the loss curve itself diverges.The metric is intended to quantify underlying instability, rather than sharpness for its own sake.Source: §1 · p. 2
- The introduction identifies an open economic question: whether proposed FP8 methods reduce costs beyond raw compute speed, given possible restart and convergence costs.The authors say their aim is to assess robustness and trustworthiness, not discredit a particular method.Source: §1 · p. 2
2. Related work
The related work covers FP8 training implementations, numerical-instability studies, reduced-precision hardware, and sharpness metrics. It situates this paper’s stability analysis among those methods and prior findings.
2.1. DeepSeek V3
- DeepSeek V3 uses group-size-128 dynamic FP8 scaling and FP32 accumulation every four WGMMA instructions; the authors treat its customized scheme as evidence that earlier approaches had not achieved economical viability.Earlier cited FP8 work used per-row scaling at finest, and DeepSeek V3 was the first work they knew to use E4M3 in both forward and backward passes.Source: §2.1 · pp. 2–3
2.2. Training stability
- Prior stability work examines smaller-model proxies, loss-landscape sharpness, and FP8 divergence that can emerge late in training.Wortsman et al. (2024) report similar instability patterns in smaller models at larger learning rates; Fishman et al. (2024) report Llama 7B FP8 divergence after 200B tokens.Source: §2.2 · p. 3
2.3. Reduced-precision processors
- Specialized hardware such as tensor cores can raise matrix-multiplication throughput, while smaller floating-point formats reduce circuit demands and mixed-precision methods retain sensitive model portions in FP32.The passage gives H100 BF16 GEMM peak throughput as 989.4 TFLOPS on tensor cores versus 133.8 TFLOPS on CUDA cores.Source: §2.3 · p. 3
2.4. Hybrid FP8
- Hybrid FP8 uses E4M3 for forward-pass resolution and E5M2 for backward-pass range, while software stabilization techniques may erase reduced-precision gains when unsupported natively.Listed techniques include stochastic rounding, chunk-based accumulation, and Kahan summation during optimizer updates.Source: §2.4 · p. 3
2.5. MS-AMP
- MS-AMP O1 performs GEMMs in FP8, reduces weights to half precision, and uses FP8 weight-gradient states and all-reduce communications; the cited landscape visualizations can remain smooth during divergence.The authors use this basic optimization level to examine the effects of less invasive modifications; Figure 2 shows validation loss at each point for Llama 120M E8M3 landscapes at 5K and 10K steps.Source: §2.5 · pp. 3–4, Figure 2 · p. 4
3. Methods
The methods assess reduced-precision training stability through incremental bit reductions, a last-token-logit sharpness proxy, and a fast masking-based simulation. Experiments use TinyLlama pretraining settings and apply reduced precision only to matrix multiplications.
3. Methods
- Experiments use TinyLlama pretraining on SlimPajama and StarCoder, with lr 4e-4, batch 512, and sequence lengths 2048 (120M) or 4096 (7B).The experiments follow the repository's default learning-rate schedule.Source: §3 · p. 4
3.1. Sharpness metric
- The proposed autoregressive sharpness metric searches bounded last-token-logit perturbations for a relative loss spike, providing an instability proxy after standard landscape plots stayed smooth during divergence.Searching the last-token logit requires one forward pass per measurement; the last token receives inputs from all other tokens in decoder-only Transformers.Source: §3.1 · pp. 4–5
3.2. Masking
- Reduced precision is emulated by thresholding the exponent range and masking mantissa bits; only matrix multiplications are masked, while states and other operations retain original precision.The masking approach is an imperfect approximation but causes at most a doubling of training-step time.Source: §3.2 · p. 5
4. Results
Across MS-AMP and simulated bit reductions, the experiments find configuration-dependent convergence and instability. Reduced mantissa precision is associated with increasing sharpness, while some instability appears only under longer or higher-learning-rate tests.
4.2. Bit reduction experiments
- Removing one exponent bit (E7M7) prevented learning; clamping only the outer exponent range reproduced failure, implicating inability to represent large values.The authors therefore investigate mantissa-bit reductions in the remaining experiments.Source: §4.2 · pp. 5–6
4.1. MS-AMP experiments
- On FineWeb Edu, MS-AMP O1 and BF16 converged similarly, but on OpenWebText the FP8 loss gap persisted after sixfold training, especially with LM-head quantization.The authors note that data quality may affect whether MS-AMP reaches the BF16 loss.Source: §4.1 · p. 5, Figure 5 · p. 6, Figure 6 · p. 6
4.3. Loss landscape sharpness
- Sharpness generally rose toward divergence across E8M4 and E8M5 runs of different durations; after divergence, models did not regain their prior training trajectory.The TinyLlama 120M E8M4 and E8M5 models were trained for 20K and 100K steps, respectively.Source: Figure 7 · p. 6, §4.3 · p. 6
- Llama 7B E8M3 and E8M4 diverged early, while E8M5 sharpness rose before visible loss instability; E8M7 sharpness increased gradually.The Llama 7B comparison was stopped at 5,000 steps; E8M5 had not yet shown loss instability then.Source: Figure 8 · p. 7, §4.3 · p. 7
4.4. Robustness to learning rate changes
- E8M5 showed more loss spikes than BF16 at learning rates 4e−4 and 4e−3, especially at 4e−3, although neither had loss divergence.The comparison used 18 random seeds per configuration.Source: Figure 9 · p. 7, §4.4 · p. 7
4.3. Loss landscape sharpness
- Reverting E8M3 training to BF16 reduced sharpness toward BF16 levels, supporting sharpness as an indicator of training instability.The paper reports this as a check that the metric tracks instability when it decreases as well as when it increases.Source: §4.3 · p. 7
5. Discussion
The authors report that MS-AMP FP8 lacks sufficient robustness for cost-effective use, while clarifying that they do not claim FP8 training is impractical. They discuss possible stabilization approaches but leave their investigation to future work.
5. Discussion
- MS-AMP, an open-source FP8 implementation, did not offer sufficient robustness for cost-effective use in the authors’ experiments.The authors explicitly do not argue that FP8 pretraining is impractical.Source: §5 · p. 8
- FP8 may narrow stable hyperparameter choices and require extra tuning; without a BF16 comparison, its competitive performance cannot be known.The authors argue that comparison runs may negate the purpose of using FP8 for training.Source: §5 · p. 8
- Possible stabilization includes higher precision early or in sensitive layers, but these compensations depend on architecture, data, and environment and are left for future work.Source: §5 · p. 8
6. Limitations
The evaluation covers only early pretraining, could not test formats with more exponent bits, and uses training loss instead of downstream-task metrics.
6. Limitations
- The early-pretraining focus may underestimate instability because some failure modes emerge only later in training.The authors note that logits may diverge from zero later, while SwiGLU-layer correlations begin increasing after 200B tokens.Source: §6 · p. 8
- Hardware limitations made experiments with more exponent bits impractical, despite the finding that exponent bits matter more than mantissa bits.FP64 matrix multiplication is over an order of magnitude slower than BF16 on A100 and H100 tensor cores.Source: §6 · p. 8
- The experiments evaluate training loss rather than downstream tasks, though the authors consider loss divergence sufficient to indicate training failure.Lower perplexity does not guarantee better downstream-task performance.Source: §6 · p. 8
7. Conclusion
The paper finds progressively lower LLM training stability as floating-point bit widths shrink, and proposes evaluating robustness under less explored conditions.
7. Conclusion
- Training stability decreases incrementally as floating-point bit widths are reduced, for models up to 7B parameters.The proposed loss-landscape sharpness metric measures the corresponding narrowing of the hyperparameter space in which stable training is possible.Source: §7 · p. 8
- Future reduced-precision training evaluations should test robustness beyond well-established guidelines and include less explored conditions.The authors emphasize that substantial LLM training power and capital consumption make this issue consequential.Source: §7 · p. 8