Source-linked AI summary
Training DNNs with Hybrid Block Floating Point
Mario Drumond, Tao Lin, Martin Jaggi, Babak Falsafi
TL;DR
DNN training demands increasingly dense hardware, yet fixed-point arithmetic’s limited range can impede convergence. HBFP performs dot products in BFP and other operations in floating point. Across evaluated models it matches FP32, while 8-bit BFP achieves 8.5× higher throughput than FP16.
Problem
Fixed-point arithmetic offers denser hardware but its limited dynamic range can be insufficient for DNN training convergence.
Source: §1 · p. 1Method
HBFP performs all dot products in BFP and other operations in floating point.
Source: §1 · p. 2Results
HBFP matches FP32 for all evaluated models, and 8-bit BFP achieves 8.5× higher throughput than FP16.
Source: §7 · pp. 8–9Takeaways & Limitations
HBFP is presented as a drop-in replacement for FP32 across tested tasks, with more compact models and arithmetic density similar to fixed-point hardware.
Source: §1 · p. 2, §6 · p. 7BFP may lose data when tensor values span more than the mantissa can capture or the shared exponent poorly represents their distribution.
Source: §4 · p. 4
Abstract
from arXiv · showhide
The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixed-point arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5x higher throughput.
1 Introduction
DNN training pressures hardware toward denser arithmetic, but fixed-point and narrow floating-point representations can lack the range needed for convergence. HBFP uses BFP for dot products and floating point for other operations, seeking dense arithmetic while preserving accuracy.
1 Introduction
- Fixed-point and FP16 training have shown mixed results because their limited numeric ranges can hinder convergence.BFP also presents challenges: general operations may lose density, exponent choice can lose data, and mantissas may not capture wide value distributions.Source: §1 · pp. 1–2
- Selecting exponents before each dot product is proposed as a more aggressive alternative to coarse-grained updates, enabling convergence with narrower mantissas.Coarse-grained approaches adjust exponents only a few times per batch or epoch and can miss drastic changes.Source: §1 · p. 2
- Converting values to BFP only before dot products uses their resilience to input data loss, while floating-point inputs elsewhere preserve arbitrary value distributions.The paper argues that floating-point operations outside dot products add little area overhead because dot products are prevalent.Source: §1 · p. 2
- HBFP combines BFP dot products with floating-point operations elsewhere, and its 12- and 8-bit mantissa configurations match FP32 accuracy.An FPGA prototype shows arithmetic density similar to fixed-point hardware.Source: §1 · p. 2
2 Related Work
Prior work explores hybrid accelerators, reduced-precision inference, and end-to-end low-precision training. HBFP distinguishes itself by applying BFP to training dot products while retaining compatibility with FP32 models.
Hybrid accelerators
- Commodity hardware already separates dot products from other operations, but HBFP uses different numeric representations for those operation classes to enable dense fixed-point training.Volta Tensor Cores and Google’s TPU provide related examples of operation separation.Source: §2 · p. 3
Inference with reduced precision
- Reduced-precision quantization and BFP commonly target inference with weights trained in full precision, whereas this work addresses training with comparable arithmetic density.The paper frames training as the more challenging task.Source: §2 · p. 3
Binarized and ternary neural networks
- Binarized and ternary networks compress models but are trained with floating-point activations and parameters, making them orthogonal to BFP-based training.Other methods use binary operations in forward and backward passes but not for weight-gradient calculation and accumulation; HBFP remains backward-compatible with FP32 models.Source: §2 · p. 3
Training with end-to-end low precision
- End-to-end low-precision methods use fixed point in forward and backward passes, with controls for activation range or quantization points.DoReFa cannot quantize the first and last layers; Flexpoint updates exponents twice per minibatch, while HBFP chooses them before every dot product and accommodates dynamic dataflows in-device.Source: §2 · p. 3
3 Specialized Arithmetic for DNNs
DNN hardware needs dense arithmetic and compact representations without sacrificing accuracy or ease of use. The section contrasts narrow floating point with BFP, which shares exponents to support fixed-point dot products while retaining a wide range.
3 Specialized Arithmetic for DNNs
- Datacenter DNN workloads motivate specialized arithmetic that improves logic density and energy efficiency while reducing memory capacity and bandwidth demands.Representations must also remain easy to use without sacrificing accuracy or requiring algorithmic recovery techniques.Source: §3 · p. 3
- For ResNet-20 on CIFAR-10, 8-bit mantissas converge without precision loss, while reducing exponent width from 8 to 6 bits diminishes validation precision and 2-bit exponents diverge.The cited table reports validation test error across narrow floating-point representations.Source: §3 · p. 4, Table 1 · p. 4
- FP16 uses 11-bit mantissas and 5-bit exponents, but its multiplier is 5.8× larger than an 8-bit fixed-point multiplier and its range can be insufficient for training gradients.An FP16 multiplier is 4.7× smaller than an FP32 multiplier; Google’s 16-bit format uses 8 bits each for mantissas and exponents to improve dynamic range.Source: §3 · p. 4
- BFP shares exponents across tensors, retaining a wide dynamic range while enabling dot products to use fixed-point logic.Because most training and inference arithmetic consists of dot products, this representation can place almost all computation in fixed-point logic.Source: §3 · p. 4
4 DNN Training With BFP Arithmetic
HBFP uses BFP for dot products and floating point for other operations, combining fixed-point computation with flexible handling of other DNN operations. Tiling and wider weight storage address two sources of BFP data loss.
4 DNN Training With BFP Arithmetic
- Each BFP tensor element is represented by a mantissa and shared exponent; BFP dot products compute the mantissa reduction in fixed point without intermediate alignment.Source: §4 · p. 4
- BFP represents a tensor accurately only when its value distribution fits the mantissa and its shared exponent represents that distribution.Source: §4 · p. 4
4.1 Hybrid Block Floating Point (HBFP) DNN Training
- HBFP performs all dot products in BFP and other operations in floating point, converting tensors before each dot product and results back afterward.Source: §4.1 · pp. 4–5
- Dot products tolerate BFP input loss because large values dominate reductions, while general BFP operations can incur costly mantissa realignment and exponent computation.Source: §4.1 · p. 5
4.2 Minimizing BFP Data Loss
- Tiling limits the number of values sharing exponents; for tile size N × N, it adds one floating-point operation every 2 × N operations.Source: §4.2 · p. 5
- Wide weight storage keeps wider mantissas for updates, while forward and backward passes access only the most significant weight bits.Source: §4.2 · p. 5
5 Methodology
The methodology simulates HBFP training on GPUs, evaluates models across image-classification and language-modeling tasks, and implements a proof-of-concept FPGA accelerator.
5.1 HBFP Simulation on GPU
- GPU simulation modified PyTorch linear and convolution layers to emulate BFP forward and backward operations, while keeping wide and narrow BFP weight copies for updates versus passes.Source: §5.1 · pp. 5–6
5.2 Evaluation Setup
- Evaluation covered CIFAR-100, SVHN, ImageNet, and PTB, comparing scratch HBFP training with FP32-tuned models under the same hyperparameters.Source: §5.2 · p. 6
5.3 Hardware Prototype Implementation
- The FPGA prototype implements training operations around MatMul and activation/loss units; Figure 2 shows BFP buffers and dataflow in gray and FP elsewhere.Source: §5.3 · p. 6, Figure 2 · p. 6
6 Evaluation
HBFP matches FP32 performance across the evaluated models and datasets, and its FPGA implementation reaches up to 8.5× the throughput of an FP16 variant. The evaluation identifies an 8- to 12-bit mantissa, 16-bit weight-storage, tile-size-24 configuration as a quality-preserving design point.
BFP design space
- Mantissas wider than 8 bits finish within 1% of FP32 validation error, while 4-bit mantissas have 4.1% larger error.Using 16-bit weight storage improves 8- and 12-bit configurations by 0.21% and 0.43%, respectively, over narrow-storage counterparts.Source: §6 · p. 7
- Tile sizes 24 × 24 and 64 × 64 keep error within 0.5% of FP32; without tiles, error rises 0.8% over FP32.The no-tile configuration often makes large weight matrices share exponents.Source: §6 · p. 7
- The design-space sweet spot is 8- to 12-bit mantissas, 16-bit weight storage, and tile size 24, matching FP32 quality.With 8-bit mantissas, forward- and backward-pass memory bandwidth requirements fall by up to 4× versus FP32.Source: §6 · p. 7
HBFP vs. FP32
- HBFP matches FP32 performance across all evaluated models and datasets.The comparison covers image-classification models and a language-modeling model; Figure 3 illustrates training for three of them.Source: Table 3 · p. 7, §6 · p. 7, Figure 3 · p. 8
HBFP silicon density and performance estimation
- The FPGA prototype reaches 1 TOp/s and 8.5× the throughput of an FP16 multiply-and-add variant.It uses 8-bit multiply-and-add units and floating-point activations on a Stratix V FPGA clocked at 200 MHz.Source: §6 · p. 8
7 Conclusion
HBFP combines block floating point and floating point for DNN training, matching FP32 on all evaluated models while producing more compact models and faster accelerators. The reported implementation uses fixed-point logic for most silicon area and reaches higher throughput than an FP16-based variant.
7 Conclusion
- HBFP training matches FP32 counterparts for all evaluated models while resulting in 2× more compact models.The conclusion also reports 8-bit BFP achieving 8.5× higher throughput than FP16.Source: §7 · pp. 8–9
- HBFP places most silicon real estate in efficient fixed-point logic.The paper describes HBFP as a hybrid BFP-FP representation for DNN training.Source: §7 · p. 8
- Higher throughput is associated with faster and more energy-efficient DNN training and inference, while model compression reduces memory and communication requirements.The stated reductions concern off-chip bandwidth, on-chip memory capacity, and distributed-training communication bandwidth.Source: §7 · p. 9