PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking floating point for deep learning

Jeff Johnson

arXiv:1811.01721v1math.NAcs.LG

TL;DR

Neural-network arithmetic seeks lower hardware overhead, but conventional floating point is less energy efficient than integer math and reduced word sizes lose dynamic range. This paper combines alternative encodings, log-domain multiplication, and Kulisch accumulation to improve floating-point efficiency while retaining useful accuracy and general-purpose arithmetic.

  • Problem

    Neural-network hardware needs lower computational cost, while existing floating-point arithmetic is less energy efficient than integer math and small word sizes can lose dynamic range.

  • Method

    The paper combines dynamic-range-efficient encodings, log-domain multiplication with linear addition, and Kulisch accumulation to produce general-purpose floating-point arithmetic without quantization tinkering or retraining.

  • Results

    The resulting arithmetic achieves reasonably similar CNN precision without retraining, activation sampling, or learned quantization parameters, while retaining general floating-point representations in 8 bits.

  • Takeaways & Limitations

    Alternative representations combining log domains, posits, and Kulisch accumulation can substantially improve floating-point efficiency and applicability for DNNs.

  • Takeaways & Limitations

    A precise Pareto frontier across frequency, latency, energy, area, pipeline depth, implementation, and accuracy remains for future analysis.

Abstract

from arXiv · show

Reducing hardware overhead of neural networks for faster or lower power inference and training is an active area of research. Uniform quantization using integer multiply-add has been thoroughly investigated, which requires learning many quantization parameters, fine-tuning training or other prerequisites. Little effort is made to improve floating point relative to this baseline; it remains energy inefficient, and word size reduction yields drastic loss in needed dynamic range. We improve floating point to be more energy efficient than equivalent bit width integer hardware on a 28 nm ASIC process while retaining accuracy in 8 bits with a novel hybrid log multiply/linear add, Kulisch accumulation and tapered encodings from Gustafson's posit format. With no network retraining, and drop-in replacement of all math and float32 parameters via round-to-nearest-even only, this open-sourced 8-bit log float is within 0.9% top-1 and 0.2% top-5 accuracy of the original float32 ResNet-50 CNN model on ImageNet. Unlike int8 quantization, it is still a general purpose floating point arithmetic, interpretable out-of-the-box. Our 8/38-bit log float multiply-add is synthesized and power profiled at 28 nm at 0.96x the power and 1.12x the area of 8/32-bit integer multiply-add. In 16 bits, our log float multiply-add is 0.59x the power and 0.68x the area of IEEE 754 float16 fused multiply-add, maintaining the same signficand precision and dynamic range, proving useful for training ASICs as well.

1 Introduction

The paper proposes improving floating point itself rather than relying solely on increasingly specialized quantization techniques. It targets lower chip power and area while retaining general-purpose arithmetic and avoiding quantization tinkering or retraining.

  • Floating point can be up to 10× less energy efficient than integer math in hardware implementations.
  • The approach improves floating-point encodings, summation, and multiplication to reduce chip power and area.
  • The resulting arithmetic is intended to work effectively on CNNs without quantization tinkering or retraining.

2 Floating point variants for NNs

Prior neural-network floating-point variants include gradient communication formats, block floating point, and reduced-precision formats for smaller networks. These approaches trade dynamic-range control, management cost, or reduced precision against hardware and data-movement savings.

  • Some 8-bit floating-point work targets gradient communication rather than general computation.
  • Block floating-point variants represent data as significands sharing an exponent, saving data movement and hardware resources but requiring controlled dynamic-range variation and more management.
  • (6, 5)-float was comparable to float32 on MNIST and CIFAR-10 for training smaller networks.
  • bfloat16 preserves float32's normalized exponent range while using reduced precision and smaller multipliers.

3 Space-efficient encodings

The paper uses tapered encodings to allocate word bits between exponent and significand according to magnitude, addressing the inefficiency of fixed-width IEEE-style fields. Posit-style encoding provides variable precision and dynamic range within a fixed word.

  • IEEE 754-style fixed-width fields allocate the same significand precision across magnitudes, while 8-bit implementations can waste substantial space on NaNs and denormals.
  • Posit encoding uses a Golomb-Rice-coded exponent whose remaining word space is assigned to the significand.
  • A posit number system is characterized by word length N and exponent scale s, with minimum and maximum positive finite values fmin = 2^-(N-2)2^s and fmax = 2^(N-2)2^s.
  • The exponent scale s controls dynamic range, while precision tapers from a maximum range to no fraction bits near ±fmax.

4 Accumulator efficiency and precision

The paper replaces repeatedly rounded floating-point accumulation with a wide fixed-point Kulisch accumulator that aligns products by exponent and rounds only once at the end. This provides associative, error-free accumulation apart from the final rounding step.

  • CNN inner products can accumulate up to 4,608 scalar products in a 2d convolution with k = 3 × 3 and cin = 512.
  • Floating-point addition is non-associative, creating reproducibility, parallelization, and rounding-error problems.
  • A Kulisch accumulator is wide enough for the largest and smallest possible products, shifting each significand by exponent before summation.
  • Kulisch accumulation performs one final rounding after all products are summed, unlike FMA's repeated nested rounding.
  • EMA can be more hardware-efficient than FMA because its fixed-point alignment location is known upfront.

5 Multiplier efficiency

The design avoids hardware multipliers by representing values logarithmically, approximating products in the linear domain, and accumulating with Kulisch registers. This combination trades LUT approximation error against efficient, accurate multiply-add hardware.

  • Log-domain arithmetic: LNS avoids hardware multipliers by encoding log2(x), but addition requires the costly σ±(x) function or an error-prone approximation.The approximation log2(1 + x) ≈ x is especially problematic for repeated sums.
  • Log-domain arithmetic: The proposed pipeline multiplies log-domain values, converts fractional parts through a LUT to linear approximations, and expands them for Kulisch accumulation.The LUT maps f ∈[0, 1) to p(f) = 2^f − 1, while the exponent supplies the linear-domain scale.
  • ELMA: ELMA makes the log product and linear sum exact before representation rounding, unlike EMA, whose final result requires rounding.The intermediate log product also avoids overflow or underflow with an extra product bit.
  • ELMA: The log-to-linear mapping plus Kulisch accumulation is efficient and reasonably accurate, but LUT-based conversion becomes impractical for larger types as table parameters scale with 2^fbits.Approximation error from r(p(f)) and r(q(g)) is traded against reduced repeated-summation error and immunity to magnitude differences.
  • Encoding: Tapered posit-style encoding preserves dynamic range in compact log numbers, while increasing γ with guard, round, and sticky bits reduces tapering-rounding error.The same tapering gives posit-based log and linear values identical fmin and fmax.

6 Additional hardware details

The implementation restricts accumulator range to the supported floating-point range and amortizes conversion overhead across many sums, prioritizing energy efficiency over minimal area.

  • Accumulator design: The accumulator range is restricted to [fmin, fmax], with temporary underflow handled instead of overflow.This design choice reflects the authors’ stated experience that temporary underflow is more important to handle.
  • Accumulator design: Conversion from a Kulisch accumulator is amortized because many sums are completed before one final conversion per inner product.The resulting energy for most operations can be lower than MAC/FMA, while increased area is accepted for energy efficiency.

7 FPGA experiments

The experiments evaluate posit and log arithmetic in an FPGA-integrated implementation using an unmodified float32 ResNet-50 model, then compare accuracy and hardware cost.

  • Experimental setup: The implementation uses SystemVerilog for ASIC evaluation, FPGA OpenCL RTL integration, and rudimentary PyTorch integration.It evaluates (N, s) posit and (N, s, α, β, γ) log arithmetic on ResNet-50 with ImageNet validation data.
  • Experimental setup: Float32 parameters and inputs are converted by round-to-nearest-even without retraining or other value adjustment.Batch normalization is fused into preceding affine layers before evaluation.
  • Accuracy results: The (8, 0) linear posit lacks sufficient dynamic range, causing activations to round quickly to zero.The reported (8, 1, 5, 5, 7) log result remains very close to the (8, 1) linear posit result.
  • Accuracy results: The log arithmetic achieves reasonably similar precision to int8/32 quantization without retraining, activation sampling, or learned quantization parameters.The int8/32 comparisons use different trained float32 parameters and therefore are not directly comparable.

8 ASIC evaluation

The ASIC evaluation compares ELMA-based multiply-adds with integer and floating-point baselines under a 28 nm, 500 MHz design study. Results show reduced multiplication power but higher state and add-related costs, while the authors identify a need for broader Pareto analysis.

  • Evaluation setup: The evaluation uses Synopsys Design Compiler and PrimeTime PX with a commercial 28 nm library, targeting 500 MHz and analyzing multiply-add PEs and 32×32 systolic arrays.Power analysis uses TT@25°C at 0.81 V; synthesis uses SS@−40°C.
  • Power results: 90.9 µW lower multiplication power and 68.3 µW higher add power distinguish ELMA from int8/32.ELMA also requires additional state for Kulisch accumulators and decoded log numbers.
  • Hardware behavior: ELMA’s larger Kulisch adder sums effectively 6 bits with carry per cycle, versus up to 16 bits for int8/32.The authors suggest strategies for large Kulisch accumulators may further exploit this small regime.
  • Evaluation scope: A more complete evaluation would need a Pareto frontier spanning frequency, latency, energy, area, pipeline depth, implementation, and accuracy.The paper presents its current analysis as limited and intended to motivate future investigation.

9 Conclusions

The paper concludes that alternative arithmetic designs can improve floating-point efficiency and applicability for DNNs. It presents log-domain representations, posits, Kulisch accumulation, and ELMA as promising directions in a broad, underexplored hardware design space.

  • Conclusion: DNN resilience to numerical tinkering enables re-evaluation of longstanding hardware arithmetic decisions with reduced fear of failure.The conclusion frames DNNs as a useful setting for exploring alternative real-number representations.
  • Conclusion: Log-domain representations, posits, Kulisch accumulation, and ELMA show that floating-point efficiency and applicability can be substantially improved.The authors plan further investigation for DNN training and general numerical algorithms.
Loading 1811.01721v1…