PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning with Limited Numerical Precision

Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan

arXiv:1502.02551v1cs.LGcs.NEstat.ML

TL;DR

Training large neural networks depends on computational resources, motivating study of limited-precision computation. The paper evaluates fixed-point training with stochastic rounding and reports that 16-bit networks achieve nearly the same performance as 32-bit floating-point networks.

  • Problem

    Training large neural networks depends on computational resources, motivating investigation of how limited-precision representation and computation affect training.

    Source: §1 · p. 1

  • Method

    The paper evaluates low-precision fixed-point neural-network training with stochastic rounding on MNIST and CIFAR10, and presents an FPGA matrix-multiplication accelerator.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    On MNIST and CIFAR10, 16-bit fixed-point networks with stochastic rounding achieve nearly the same performance as networks trained using 32-bit floating-point computations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    37 G-ops/s/W power efficiency was reported for the FPGA systolic array, illustrating the hardware-efficiency potential of fixed-point acceleration.

    Source: §5.3 · p. 9

    At 12-bit precision, convergence degrades until SGD stops making progress, and the minimum achievable test error saturates at 28.8%.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

1. Introduction · 2. Related Work

The paper finds that deep networks trained with 16-bit fixed-point arithmetic and stochastic rounding achieve nearly the same performance as 32-bit floating-point training, and presents an FPGA accelerator for this approach. It motivates this work by neural networks’ error resilience and the potential hardware and memory benefits of low-precision fixed-point computation.

1. Introduction

  • Large-scale deep learning depends on hardware that can rapidly train complex networks on labeled data, motivating distributed CPU, GPU, and combined systems (Wu et al., 2015).Such capability supports rapid architecture evaluation and hyperparameter search.Source: §1 · p. 1
  • Neural networks tolerate approximation and training noise, yet large-scale systems largely fail to exploit this resilience and incur computational overhead from general-purpose hardware.The paper notes asynchronous stochastic gradient descent as an exception to this pattern; noise-related performance benefits are discussed by Murray & Edwards (1994), Bishop (1995), and Audhkhasi et al. (2013).Source: §1 · p. 1
  • The work pursues hardware-software co-design by allowing approximate, potentially nondeterministic hardware computations to expose errors to algorithms while preserving the programming model.The stated aim is to enable application-level benefits without significant software redevelopment.Source: §1 · p. 1
  • Fixed-point units can be faster and use less hardware and power than floating-point engines, while low-precision representations reduce memory use and can enable greater data-level parallelism.Reduced memory footprint can allow larger models to fit within available memory.Source: §1 · p. 2

1. Introduction · 2. Related Work

  • 16-bit fixed-point training with stochastic rounding achieves nearly the same performance as 32-bit floating-point training on MNIST and CIFAR10, unlike prior work requiring 32-bit fixed point for convergence.The authors describe this as the first study applying stochastic rounding to low-precision fixed-point training of deep neural networks.Source: §1 · p. 2, §2 · p. 2

1. Introduction

  • The FPGA accelerator combines many fixed-point arithmetic units, a dataflow architecture, and compact stochastic rounding modules to achieve high throughput and low power.The design is presented as a hardware accelerator for deep neural network training.Source: §1 · p. 2

2. Related Work

  • Most prior precision studies target inference rather than training, including vectorized 8-bit operations, FPGA inference, and custom hardware implementations.These approaches assume networks are trained offline at high precision.Source: §2 · p. 2
  • Earlier limited-precision training studies found that 8–16 bits often sufficed for simple networks, but focused on shallow multilayer perceptrons, making their findings difficult to extrapolate to modern deep networks.Prior work included 24-bit floating-point backpropagation, 8–16-bit fixed-point on-chip learning, theoretical analysis, and probabilistic rounding of weight updates below 8 bits (Iwata et al., 1989; Hammerstrom, 1990; Holt & Hwang, 1993; H¨ohfeld & Fahlman, 1992).Source: §2 · p. 2

3. Limited Precision Arithmetic

The section defines fixed-point precision and specifies how rounding and multiply-accumulate operations are handled when computing with limited-precision values.

3. Limited Precision Arithmetic

  • A ⟨IL, FL⟩ fixed-point format uses IL integer bits and FL fractional bits, giving word length IL + FL and smallest positive value ϵ = 2^-FL.Source: §3 · p. 3

3.1. Rounding Modes

  • Stochastic rounding is unbiased: the expected rounded value equals x when converting it to the target fixed-point format.Values outside the format’s range are saturated to its lower or upper limit, regardless of rounding mode.Source: §3.1 · p. 3

3.2. Multiply and accumulate (MACC) operation

  • The MACC procedure multiplies inputs at ⟨2 * IL, 2 * FL⟩ precision, accumulates products in a wide temporary register, then converts the result to its target format.The temporary register requires log2d + 2WL bits in the worst case; rounding after accumulation reduces stochastic-rounding hardware overhead and supports efficient CPU/GPU simulation.Source: §3.2 · p. 3

4. Training Deep Networks

The section tests fixed-point training against floating-point baselines across DNNs and CNNs, finding that 16-bit arithmetic can preserve learning when stochastic rounding retains small updates.

4. Training Deep Networks

  • 16-bit data storage halves memory use versus float, and using the same word length for network variables simplifies hardware implementation.The fixed-point experiments constrain parameters and intermediate back-propagation variables to fixed-point representation.Source: §4 · pp. 4–5

4.1.2. CNN

  • 0.83% and 0.90% test error at 14-bit and 12-bit stochastic precision, respectively, compared with 0.77% for float; round-to-nearest failed to converge.Layer outputs used the ⟨6, 10⟩ format to address difficulty starting training with a low integer range.Source: Figure 2 · p. 5, §4.1.2 · p. 6

4.1.1. Fully connected DNN

  • At 8-bit precision, stochastic rounding let the fully connected MNIST DNN learn without significant performance loss, while round-to-nearest degraded below 14 bits and stochastic rounding failed below 8.At reduced fractional precision, round-to-nearest often rounded parameter updates to zero; stochastic rounding preserved gradient information statistically.Source: §4.1.1 · p. 5

4.2. CIFAR10

  • 25.4% test error with 16-bit stochastic rounding (vs. 24.6% with float) approached float performance after 100 epochs at 12 fractional bits plus 15–20 epochs at 20-bit precision.The authors propose switching to higher-precision fixed-point computation when low-precision learning stagnates.Source: Figure 3 · p. 6, §4.2 · pp. 6–7
  • At 12-bit fractional precision, sparse updates stalled SGD and test error saturated at 28.8%, but increasing word length to 20 bits restored rapid improvement.The authors attribute the stagnation to small gradients combined with diminished learning rates.Source: §4.2 · pp. 6–7, Figure 3 · p. 6

5. Hardware Prototyping

The section describes an FPGA-based fixed-point matrix-multiplication accelerator intended to improve GEMM throughput for neural-network training. Its design uses a systolic array and data reuse to work within on-chip memory and off-chip bandwidth constraints.

5. Hardware Prototyping

  • GEMM throughput is central to training time, and the accelerator uses FPGA fixed-point DSP units suited to the arithmetic while requiring careful management of limited memory bandwidth.The authors cite fast development and lower cost than ASICs as reasons for choosing FPGAs.Source: §5 · p. 7
  • The prototype uses a Kintex325T FPGA with 840 DSP multiply-accumulate units, almost 2 MB of block RAM, and 6.4 GB/s DDR3 bandwidth; matrices reside in DDR3 because they do not fit on-chip.The design reuses on-chip data extensively to make achievable throughput compute-bound despite off-chip bandwidth limits.Source: §5 · p. 7

5.1. System Description

  • The multiplier organizes n^2 FPGA DSP units as a two-dimensional systolic array, with block RAM serving as an L2 cache and logic moving data between DDR3, cache, and array.READ and WRITE logic handle external-memory transfers, while a controller coordinates processing.Source: §5.1 · pp. 7–8, Figure 4 · p. 7
  • Double buffering hides transfer latency as matrix tiles are processed; each loaded A element is reused m times and each B element pn times to use DDR3 bandwidth efficiently.The process loads n columns of B and pn rows of A, then repeats across B's columns and all rows of A.Source: §5.1 · p. 8

FIFOs … DSP MACC FIFO Input A FIFOs

The systolic FPGA design streams matrix elements through local DSP MACC units and uses DSP-based stochastic rounding before output. The reported 28×28 implementation reaches 260 G-ops/s at 37 G-ops/s/W.

FIFOs

  • Input matrices are staged in local block-RAM FIFOs by row of A or column of B, then streamed through DSP units performing multiply and accumulate each cycle.Elements cascade right for A and down for B; rounded outputs enter one FIFO per column before writeback to external memory.Source: DSP DSP DSP FIFO FIFO · p. 8

DSP FIFO MACC

  • Array throughput depends on DSP availability and maximum error-free operating frequency; neighbor-only connections and edge FIFOs limit interconnect delays and improve frequency.The design is a wavefront-type systolic array with local connections between neighboring DSPs and edge FIFOs.Source: DSP DSP DSP FIFO FIFO · p. 8

DSP MACC

  • In a wavefront array, after k cycles (k is the inner dimension), MACC 11 transfers its accumulated partial products to a local register and resets, freeing it for the next multiplication before other elements finish.The matrix multiplication completes after k + 2n −2 cycles, when data from the last DSP unit can be read out.Source: DSP DSP DSP FIFO FIFO · p. 8

DSP ROUND

  • Each output column’s DSP ROUND uses an LFSR-generated random value to round LSBs, detects excess MSBs for saturation, and costs 28 DSP units, under 4% hardware overhead.Pattern detection identifies overflow or underflow, and results saturate to the maximum or minimum 2’s complement values.Source: DSP DSP DSP FIFO FIFO · pp. 8–9

5.3. Results

  • 260 G-ops/s at 37 G-ops/s/W compares favorably with the Intel i7-3720QM, NVIDIA GT650m, and GTX780, reported at 1–5 G-ops/s/W.Vivado estimated 166 MHz operation and 7 W power for the 28×28 KintexK325T FPGA; the authors note newer Xilinx FPGAs could raise throughput.Source: §5.3 · p. 9

DSP ROUND

  • A single DSP unit per output performs stochastic rounding and excess-MSB detection, avoiding the initial full multi-bit comparison of the more direct approach.The compact implementation adds a random number and drops rounded LSBs, while DSP pattern-detect capabilities check excess MSBs.Source: §5.3 · p. 9, DSP DSP DSP FIFO FIFO · p. 9

6. Conclusion

The paper shows that stochastic rounding enables low-precision fixed-point training with results nearly identical to 32-bit floating-point computations. It also demonstrates an energy-efficient matrix-multiplication architecture and proposes hardware-software co-design using relaxed, inexact computing.

6. Conclusion

  • Stochastic rounding lets deep neural networks trained with low-precision fixed-point computations achieve results nearly identical to 32-bit floating-point computations.The paper notes that conventional rounding schemes fail in this setting.Source: §6 · p. 9
  • The authors implement a high-throughput, energy-efficient matrix-multiplication architecture incorporating stochastic rounding with very little overhead.Source: §6 · p. 9
  • The paper’s top-down approach uses neural-network noise tolerance to guide low-level compute-unit design, substituting fixed-point circuits for floating-point units to gain energy efficiency and throughput while risking network performance.Source: §6 · p. 9
  • The authors envision hardware-software co-designed systems for large-scale machine learning using relaxed, inexact models and non-deterministic components throughout the computing stack.Source: §6 · p. 9

Related papers

Loading 1502.02551v1…