Source-linked AI summary
Microscaling Data Formats for Deep Learning
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, Eric Chung
TL;DR
Deep learning’s scaling demands motivate narrower data formats, but these formats must preserve accuracy while remaining efficient and easy to integrate. This paper evaluates Microscaling formats that pair block-level scales with narrow elements across inference and training. Across diverse benchmarks, MX formats provide practical low-friction alternatives to FP32, including sub-8-bit generative training with minimal accuracy loss and unchanged training recipes.
Problem
Scaling deep learning increases computational and storage costs, motivating formats narrower than FP32 that also preserve accuracy and usability.
Method
The paper empirically evaluates MX formats, which combine a shared block-level scale with narrow floating-point or integer elements, across inference and training benchmarks.
Results
MX formats provide practical low-friction alternatives to FP32 across diverse models; 6-bit MX supports generative training with sub-8-bit weights, activations, and gradients without training-recipe changes.
Takeaways & Limitations
MXINT8 supports direct-cast inference close to FP32, while MXFP6 and mixed-precision MXFP4/MXFP6 enable accurate or near-accurate generative training at lower bit widths.
Takeaways & Limitations
The conversion algorithm sets elements to zero for subnormal Float32 inputs, an implementation choice outside the OCP MX specification.
Abstract
from arXiv · showhide
Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications. This paper evaluates Microscaling (MX) data formats that combine a per-block scaling factor with narrow floating-point and integer types for individual elements. MX formats balance the competing needs of hardware efficiency, model accuracy, and user friction. Empirical results on over two dozen benchmarks demonstrate practicality of MX data formats as a drop-in replacement for baseline FP32 for AI inference and training with low user friction. We also show the first instance of training generative language models at sub-8-bit weights, activations, and gradients with minimal accuracy loss and no modifications to the training recipe.
1 Introduction
MX data formats address the rising computational and storage costs of scaling deep learning by combining narrow bit-width elements with block-level scaling. The paper evaluates whether they can balance hardware efficiency, model accuracy, and user friction across inference and training.
- Scaling deep learning models increases the computing power and storage capacity required for training and deployment.
- Narrow bit-width formats reduce computational and storage costs, but FP8 and INT8 typically require per-tensor scaling factors for tensor dynamic range.
- MX formats target a balance among hardware efficiency, model accuracy relative to FP32, and seamless integration across workloads.
- 8-bit MX formats support direct-cast inference on FP32 pretrained models with minimal accuracy loss and no calibration or finetuning.
- 6-bit MX formats achieve inference close to FP32 after quantization-aware finetuning or post-training quantization, while enabling sub-8-bit generative training without recipe changes.
- A custom CUDA library emulates MX formats on existing GPUs and reproduces the paper’s experiments.
2 Microscaling
An MX block represents k values using one shared scale and k narrow elements, with its semantics determined by the block size, scale format, and element format. Concrete MX formats use E8M0 scales alongside independently specified element formats.
- An MX block encodes a vector of k numbers using one shared scale X and k scalar elements Pi.
- The MX block is defined by block size k, scale data format, and element data format, while its storage layout is implementation-dependent.
- Each represented value is normally vi = XPi, while NaN scales propagate NaN across the block and overflow beyond Float32 is implementation-defined.
- MX formats support NaN through either a block-wide NaN scale or individual NaN elements, and some element formats support individually encoded Inf values.
- Concrete MX formats use E8M0 for the shared scale, whose representable exponents superset those of FP32; element formats determine the named MX variant.
3 Scalar Float to MX Format Conversion
The conversion algorithm derives a shared scale from a vector of scalar floats, quantizes normalized values into the element format, and returns the scale with the encoded elements. Conversion depends on axis selection and includes implementation-specific handling of subnormals.
- Algorithm 1 converts a vector of scalar floats into an MX block by returning a shared scale X and quantized elements Pi.
- Each Pi is quantized from Vi/X into the element format, with overflowing normal values clamped while Infs and NaNs remain unclamped.
- The shared exponent offset maps the maximum input exponent to the largest element-format binade, using the available exponent range fully.
- The algorithm sets Pi to zero for subnormal Float32 inputs, an implementation choice not described in the OCP MX specification.
- For multidimensional tensors, selecting the shared-scale axis matters because MX conversion and transposition are not commutative.
4 Experimental Results
The experiments evaluate MX formats across inference and training workflows, using quantized dot products alongside scalar floating-point vector operations. Results show minimal accuracy loss for direct-cast inference and near- or full-parity training at 6-bit and mixed 4/6-bit precision.
- Compute Flow: MX compute flow quantizes inputs for dot products, while vector operations remain in Bfloat16 or FP32 and dot products return scalar floats.A master FP32 weight copy is updated each training step.
- Methodology: The evaluation covers direct-cast inference, error-diffusion post-training quantization, finetuned inference, and training from scratch.The custom CUDA extension emulates MX formats on existing GPUs.
- Discriminative Inference: MX formats were tested on discriminative tasks spanning translation, text encoding, image classification, speech recognition, and recommendation models.The benchmark suite contains both discriminative and generative tasks.
- Generative Inference: MXINT8 matched baseline FP32 within the standard deviation on all tasks for both GPT3-175B and LLaMA-7B direct-cast inference.The GPT3-175B results report higher-is-better task scores with bootstrap estimated standard deviations.
- Generative Training: MXFP6_e3m2 matched FP32 model quality with lower circuitry footprint and trained generative language models using 6-bit weights, activations, and gradients without recipe modifications.This was reported as the first demonstration of such training at 6 bits.
- Generative Training: MXFP4 weights with MXFP6_e3m2 activations and gradients produced only a minor model-loss penalty without modifying the training recipe.The training hyperparameters were unchanged from the prior experiment.
5 Conclusion
MX formats provide low-friction alternatives to FP32 across diverse deep learning workloads, with especially strong results for inference and sub-8-bit language-model training.
- MX formats offer alternatives to FP32 across generative language, image classification, speech recognition, recommendation, and translation models.
- MXINT8 is a compelling drop-in replacement for FP32 in low-friction direct-cast inference.
- MXFP6 closely matches FP32 for inference after quantization-aware finetuning.
- MXFP6 enables generative language-model training with sub-8-bit weights, activations, and gradients without sacrificing accuracy or changing the training recipe.
- MXFP4 weights with MXFP6 activations and gradients incur only a minor loss penalty for generative language models.