PaperlayerView exact revision on arXiv

Source-linked AI summary

Training and inference of large language models using 8-bit floating point

Sergio P. Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy-Kramer, Paul Balanca, Carlo Luschi, Stephen Barlow, Andrew William Fitzgibbon

arXiv:2309.17224v1cs.LGcs.ARcs.CLcs.ETcs.PF

TL;DR

FP8’s narrower dynamic range makes scale selection important for avoiding overflow or underflow, and the paper addresses the limited guidance on choosing those scales. It presents dynamic per-tensor scaling for FP8 linear layers and applies it to GPT and Llama models. The authors report matching higher-precision reference results for models up to 70B parameters.

  • Problem

    FP8 scaling choices are not well covered, although careful scaling is needed to keep values within the format’s narrower dynamic range and avoid degradation.

    Source: §1 · pp. 1, 3

  • Method

    The paper presents a methodology for selecting per-tensor scaling biases in FP8 linear layers, including dynamic updates during training and post-training quantization for inference.

    Source: §1 · p. 3

  • Results

    The methodology adapts scaling biases to prevent underflow or overflow and match higher-precision reference results for GPT and Llama models up to 70B parameters.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The methodology provides practitioners with details for selecting FP8 linear-layer scaling biases.

    Source: §1 · p. 3

    The work focuses on FP8 quantization of linear layers; other transformer layers and non-transformer model families remain for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models. Their main challenge is that a careful choice of scaling is needed to prevent degradation due to the reduced dynamic range compared to higher-precision formats. Although there exists ample literature about selecting such scalings for INT formats, this critical aspect has yet to be addressed for FP8. This paper presents a methodology to select the scalings for FP8 linear layers, based on dynamically updating per-tensor scales for the weights, gradients and activations. We apply this methodology to train and validate large language models of the type of GPT and Llama 2 using FP8, for model sizes ranging from 111M to 70B. To facilitate the understanding of the FP8 dynamics, our results are accompanied by plots of the per-tensor scale distribution for weights, activations and gradients during both training and inference.

1 Introduction

Reducing numerical precision can improve efficiency, but FP8’s narrower dynamic range makes scaling choices critical to avoiding overflow or underflow. The paper presents per-tensor scaling-bias methods for FP8 linear layers and demonstrates their use with GPT and Llama models in training and inference.

1 Introduction

  • Representing values with fewer bits can improve memory and communication bandwidth, model-size capacity, and operation throughput, with costs typically scaling linearly in bits per value.These factors affect inference latency, model-size limits, throughput, and training time.Source: §1 · p. 1
  • FP8’s reduced exponent range can cause overflow or underflow, rapidly degrading model accuracy if generated values exceed its representable range.The paper identifies ensuring that training and inference values remain representable as a central scaling challenge.Source: §1 · p. 1
  • Per-tensor scaling biases are computed for FP8 linear layers in training and inference, addressing a design choice that prior FP8 work left underspecified.Prior work either swept scaling values or did not specify how biases were computed.Source: §1 · p. 3
  • FP8 methods are reported to support GPT and Llama model convergence from 111M to 70B parameters in both training and inference.The paper also describes post-training quantization of checkpoints for inference and dynamic bias updates during training.Source: §1 · p. 3

2 The linear layer adapted to FP8

FP8 linear layers use per-tensor scaling to keep weights, activations, and gradients representable despite FP8’s narrower range. The section describes how scales are applied and selected, and when loss scaling is additionally needed.

2 The linear layer adapted to FP8

  • FP8 matrix multiplications require scaling to prevent underflow or overflow; their inputs are FP8 while accumulation may use FP16.FP8 E4 weights and activations need scaling, and the forward and backward passes both use scaled inputs.Source: §2 · p. 3

2.1 Applying a scaling bias before casting to FP8

  • An integer scaling bias shifts weights and activations before FP8 E4 casting, and their product’s output is unscaled using the negative sum of the input biases.For weights, scales may need recomputation during training as weights change; inference weights are not updated.Source: §2.1 · pp. 4–5, §2 · p. 4

2.2 FP8 for gradients during training

  • During training, both backward-pass matrix multiplications use scaled inputs, and gradients are cast to FP8 E5 while weights and activations use FP8 E4.The gradient format is chosen to preserve a wider dynamic range.Source: §2.2 · p. 5

2.3 Choosing the appropriate scaling bias

  • The proposed method computes dynamic per-tensor scaling biases using the absolute maximum and the maximum representable value of the FP8 format.Training computes biases for weights, input activations, and output activation gradients; inference computes them for weights and input activations.Source: §2.3 · p. 5

2.4 Loss scaling in addition to scaling bias when accumulating in FP16

  • With FP16 accumulation, loss scaling is needed to better represent gradients after FP8 matrix multiplication and unscaling; it is not needed with BF16 or FP32 accumulation.The experiments use constant loss scaling with the same values as full FP16 training.Source: §2.4 · pp. 5–6

3 Details to perform training and inference in FP8

The section describes two approaches to choosing FP8 scaling biases: per-layer absolute maxima and a constant bias shared across tensors. It then specifies their inference and training use, including a hardware-dependent cost for absolute-max scaling.

3 Details to perform training and inference in FP8

  • FP8-AMAX computes absolute-maximum scaling per linear layer for each microbatch and replica, while FP8-CSCALE uses one constant bias for weights, activations, and gradients.FP8-CSCALE biases are selected by sweeping values that do not degrade accuracy.Source: §3 · p. 6
  • On hardware with limited SRAM, FP8-AMAX can require a second L2-cache memory round-trip, potentially canceling FP8 matrix-multiplication speedups.The passage notes that using past absolute maxima instead of just-in-time maxima could remedy this; hardware with enough SRAM can calculate biases just in time.Source: §3 · p. 6

3.1 Inference with FP8

  • For higher-precision inference checkpoints, weights are scaled and cast to FP8 once at checkpoint loading; FP8-trained checkpoints need no quantization.The passage describes post-training quantization for higher-precision weights.Source: §3.1 · p. 6

3.2 Training with FP8

  • Training uses different FP8 formats for weights and activations versus gradients, with FP16 weight updates and master weights.For FP16 accumulation, the method reduces the FP8-AMAX bias by a margin to prevent backward-pass overflow; a margin of 3 was empirically sufficient, close to log2(512) = 4.5.Source: §3.2 · p. 6
  • Table 1 organizes the GPT and Llama 2 model sizes used in training and validation experiments.Source: Table 1 · p. 7

4 Experiments

Experiments evaluate GPT-style and Llama 2 decoder models across inference and fine-tuning settings, comparing FP8 approaches with FP16. FP8-AMAX matches FP16 validation accuracy across tested GPT sizes and tasks, while FP8-CSCALE can converge but is less robust as model size grows.

4.1 Model architecture used for the experiments

  • Experiments cover GPT-style models with dense attention and Llama 2 models, using fine-tuned GPT checkpoints for inference and pretrained, non-dialogue-tuned Llama 2 checkpoints.The experiments quantize linear layers across decoder layers; GPT evaluation includes MNLI, SST-2, and QQP, while Llama 2 evaluation covers six benchmarks.Source: §4.1 · p. 7

4.2 FP8 inference for the GPT model

  • FP8-AMAX and FP8-CSCALE match FP16 validation accuracy for GPT models from 111M to 13B.FP8-CSCALE uses a sweep of scaling biases; values outside the convergent intervals can substantially degrade accuracy, and the intervals average five integer values centered around zero.Source: §4.2 · p. 7, Table 2 · p. 8, Table 3 · p. 8

4.3 FP8 few-shot inference for the Llama 2 model

  • FP16 and FP8-AMAX produce comparable results on six Llama 2 benchmarks for 7B and 70B models.The evaluation uses the Eleuther harness; the authors attribute some differences from published HellaSwag results to the evaluation library.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Is FP8-CSCALE enough to train in FP8?

  • FP8-CSCALE reaches at least 99.5% of FP16 validation accuracy at every tested size, but its converging bias range narrows with size and larger models can diverge across seeds.These results suggest FP8-AMAX is more robust for FP8 fine-tuning.Source: §4.4 · p. 9, Table 6 · p. 9

4.5 FP8 fine-tuning results for the GPT model

  • FP8-AMAX fine-tuning converges across GPT sizes from 111M to 13B on MNLI, QQP, and SST-2, with validation accuracy comparable to FP16.The loss-function evolution also converges similarly for FP8-AMAX and FP16.Source: §4.5 · p. 9, Table 5 · p. 9

5 Conclusion

The paper presents FP8 scaling guidance for training and inference, reporting that its method avoids underflow or overflow and matches higher-precision reference results for GPT and Llama models up to 70B. The appendices explain FP8 formats and compare their properties with INT8.

5 Conclusion

  • The method adapts scaling biases to prevent FP8 underflow or overflow and match higher-precision reference results for GPT and Llama models up to 70B.The authors present these technical details for practitioners using FP8 in training and inference.Source: §5 · p. 9

A Floating point formats in deep learning

  • Low-precision formats can improve throughput, reduce model-storage needs, and ease bandwidth constraints, but narrower dynamic range can affect numerical accuracy.The appendix defines low precision as formats with fewer than 32 bits and describes how limited range can cause gradient underflow or overflow.Source: A Floating point formats in deep… · p. 12

B 8-bit floating point formats

  • FP8 proposals use E5 for gradients’ range and E4 for the mantissa precision needed by activations and weights; the paper assumes Noune et al.’s formats for experiments.The two proposals also describe a user-defined bias for scaling tensors.Source: B 8-bit floating point formats · p. 13

C The benefits of FP8 over INT8 for deep learning

  • FP8’s theoretical benefits over INT8 include potential use for inference and 8-bit training, while INT8 inference can require complex scaling techniques to maintain accuracy.These benefits are framed in the context of hardware with accelerated FP8 arithmetic.Source: C The benefits of FP8 over INT8 for… · p. 13

C.1 The distribution of values for integer versus floating-point formats

  • Floating-point formats provide approximately constant SNR within their dynamic range for unit-normal values, while integer formats have only a narrow high-SNR region.Consequently, floating-point tensors can be scaled by arbitrary constants without changing relative representation accuracy, provided they remain in range.Source: §C.1 · pp. 13–14

C.2 Implications for inference

  • For most scenarios, the paper reports FP8 accuracy comparable to INT8 and simpler schemes for attaining accuracy than the cited INT8 methods, while noting some cases of notable FP8 degradation.It also suggests reusing scalings found at the end of FP8 training to simplify inference.Source: §C.2 · p. 14

C.3 Implications for training

Training quantisation must accommodate both forward and backward tensors whose distributions change during learning, so a scale that works at one point may not work later. The section also discusses FP8’s relative robustness to outliers and prior work on reducing them.

C.3 Implications for training

  • Training quantisation must represent forward and backward tensors with potentially different scales, and their distributions change during training, so one scaling choice is not guaranteed to remain suitable.The section contrasts this non-stationarity with inference, where tensor distributions do not change through gradient updates.Source: §C.3 · p. 14

D The challenges of FP8 quantisation at scale

  • Emergent outliers increasingly complicate post-training quantisation as model scale grows, and naive INT8 quantisation can substantially degrade accuracy in their presence.The section notes prior methods to reduce outliers through hyperparameter choices or attention-layer changes, developed for INT8 inference but applicable to FP8 training and inference.Source: D The challenges of FP8 quantisation… · pp. 14–15
  • A modeled tensor with outliers had 635x higher SNR when quantised in FP8 than INT8, although sufficiently large outliers may still exceed FP8’s range.The cited work describes FP8 as more robust than INT8 for emergent outliers that challenge large-model post-training quantisation.Source: D The challenges of FP8 quantisation… · p. 15

E Linear layers in GPT and Llama 2 architectures quantised to FP8

The FP8 scope is the decoder’s linear layers, while dot-product attention remains in higher precision. For a typical Llama 2 70B model at moderate sequence length, those four linear layers account for more than 99.9% of decoder-layer compute.

E Linear layers in GPT and Llama 2 architectures quantised to FP8

  • Dot-product attention and other non-linear decoder layers are outside the study’s FP8 quantisation scope, though the authors identify attention as a possible target for future work.The authors note that attention’s quadratic sequence-length cost can make it dominant for longer sequences.Source: E Linear layers in GPT and Llama 2… · pp. 15–16
  • The four FP8-quantised linear layers are the Q, K, and V projections, the post-attention output projection, and two feed-forward projections.Figure 4 distinguishes the FP8 linear layers from components retained in higher precision.Source: E Linear layers in GPT and Llama 2… · p. 15, Figure 4 · p. 15
  • For a typical Llama 2 70B model with moderate sequence length, the four quantised linear layers constitute more than 99.9% of decoder-layer compute.The compared dot-product attention has quadratic sequence-length cost and can dominate at longer sequence lengths.Source: E Linear layers in GPT and Llama 2… · pp. 15–16

F Results of the FP8-CSCALE method for inference

On MNLI validation across the tested GPT sizes, FP8-AMAX matches FP16 accuracy, while FP8-CSCALE reaches the FP16 target only within a range of scaling biases. Outside that range, CSCALE accuracy degrades.

F Results of the FP8-CSCALE method for inference

  • FP8-CSCALE validation accuracy degrades when the chosen scaling bias falls outside the interval that reaches the FP16 target.Figure 5 shows the comparison; Table 3 reports the interval reaching at least 99.5% of the FP16 value.Source: F Results of the FP8-CSCALE method for… · p. 16, Figure 5 · p. 16
  • FP8-AMAX matches FP16 accuracy on MNLI validation across the different GPT sizes, while FP8-CSCALE reaches the FP16 target only within a scaling-bias interval.Figure 5 compares FP16, FP8-AMAX, and FP8-CSCALE validation accuracy.Source: Figure 5 · p. 16, F Results of the FP8-CSCALE method for… · p. 16
  • The reported comparison covers MNLI validation and varies scaling bias across different GPT sizes.The caption identifies the specific interval reaching at least 99.5% of the FP16 value as a Table 3 result.Source: Figure 5 · p. 16

G Distribution of per-tensor scaling bias for inference with FP8-AMAX

For MNLI inference with FP8-AMAX, weight biases are higher and less varied than activation biases, while activation biases vary by model size and linear-layer type. FP8-CSCALE’s converging ranges are centered around zero, unlike the higher biases selected by FP8-AMAX.

G Distribution of per-tensor scaling bias for inference with FP8-AMAX

H Further details to run the GPT and Llama experiments

The GPT fine-tuning experiments use AdamW with shared batch size and epoch settings, while weight decay varies with model size. They run on IPU machines, which support FP8 in software despite lacking native FP8 hardware.

H.1 Fine-tuning details

  • Fine-tuning uses AdamW with (beta1, beta2) = (0.9, 0.999), epsilon 1e-5, global batch size 512, and three epochs unless otherwise stated.Weight decay is 0.01 for the 111M and 590M models and 0 for larger models; no gradient-norm clipping is specified, and dropout is used.Source: §H.1 · p. 18
  • The learning rate stays constant because warmup plus decay was observed to have little effect on final validation accuracy.Further model-size-specific hyperparameters are listed in Table 8.Source: §H.1 · p. 18, Table 8 · p. 18

H.2 Hardware to run the experiments

  • Experiments run on Bow Pod64, Bow Pod16, IPU-POD64, or IPU-POD16 machines; FP8 is supported in software although the hardware lacks native FP8.The IPUs distribute training and inference across chips for data-replica parallelism or to fit models across multiple chips.Source: §H.2 · p. 18

I Evolution of per-tensor scaling bias during training with FP8-AMAX

During MNLI fine-tuning, weight and activation biases remain fairly stable but settle at different values across decoder layers, whereas gradient biases change more often and vary more with model size. The plots also show distinct patterns by decoder layer and linear-layer type.

I Evolution of per-tensor scaling bias during training with FP8-AMAX

J Comparison of FP16 vs FP8 loss function during training

The appendix compares FP16 and FP8 fine-tuning loss trajectories for five GPT model sizes on MNLI. It reports that QQP and SST2 loss curves are similar to MNLI, but omits them.

J Comparison of FP16 vs FP8 loss function during training

Related papers

Loading 2309.17224v1…