PaperlayerView exact revision on arXiv

Source-linked AI summary

Training deep neural networks with low precision multiplications

Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David

arXiv:1412.7024v5cs.LGcs.CVcs.NE

TL;DR

Multipliers are the most space- and power-hungry arithmetic operators in digital neural networks. The paper trains Maxout networks across three datasets and three multiplication formats, assessing how precision affects final training error. It finds very low precision sufficient for training, while identifying the evaluated model, task, and format range as scope boundaries.

  • Problem

    Multipliers are the most space- and power-hungry arithmetic operators in digital neural networks, motivating assessment of whether their precision can be reduced.

    Source: §1 · p. 1

  • Method

    The paper trains Maxout networks on MNIST, CIFAR-10, and SVHN using floating-point, fixed-point, and dynamic fixed-point formats, with low-precision multipliers and high-precision accumulators.

    Source: §1 · p. 1

  • Results

    Very low-precision multiplications suffice for training as well as running trained networks across the three datasets and formats.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dynamic fixed point seems well suited for training deep neural networks, and higher precision for parameter updates helps.

    Source: §11 · p. 8

    The reported scope is image classification with Maxout networks and floating-point, fixed-point, and dynamic fixed-point formats.

    Source: §11 · pp. 8–9

Abstract

from arXiv · show

Multipliers are the most space and power-hungry arithmetic operators of the digital implementation of deep neural networks. We train a set of state-of-the-art neural networks (Maxout networks) on three benchmark datasets: MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those formats, we assess the impact of the precision of the multiplications on the final error after training. We find that very low precision is sufficient not just for running trained networks but also for training them. For example, it is possible to train Maxout networks with 10 bits multiplications.

1 INTRODUCTION

The paper investigates reducing multiplier precision because multipliers are the most space- and power-hungry arithmetic operators in digital neural networks. It evaluates low-precision multiplication with high-precision accumulation across three formats and benchmark datasets.

1 INTRODUCTION

  • The study trains Maxout networks on MNIST, CIFAR-10, and SVHN using floating-point, fixed-point, and dynamic fixed-point formats.The networks have slightly less capacity than those of Goodfellow et al. (2013a).Source: §1 · p. 1
  • Very low-precision multiplications suffice not only to run trained networks but also to train them, across three datasets and three formats.The experiments assess the effect of multiplication precision on final training error.Source: §1 · p. 2

2 MULTIPLIER-ACCUMULATORS

  • Fixed-point multiplier cost grows quadratically with operand precision, while adder and accumulator cost grows linearly.Reducing precision can allow a modern FPGA DSP block to implement three 9 × 9 multipliers instead of one 27 × 27 multiplier.Source: §2 · p. 2, Table 1 · p. 2
  • The training approach uses low-precision multipliers with high-precision accumulators.The paper identifies multiplier precision reduction as its objective because multipliers are the most space- and power-hungry arithmetic operators in digital neural networks.Source: §1 · p. 1, §2 · p. 2

3 FLOATING POINT

Floating-point values use a sign, exponent, and mantissa; the exponent gives range and the mantissa gives precision. The experiments use single precision as their reference and report little to no training impact from half precision.

3 FLOATING POINT

  • Floating-point values consist of a sign, exponent, and mantissa; the exponent provides range and the mantissa provides precision.Figure 1 compares floating-point and fixed-point formats.Source: §3 · p. 2, Figure 1 · p. 3
  • Half-precision floating point has little to no impact on neural-network training in the reported experiments.Single precision is used as the reference format.Source: §3 · p. 3
  • At the time of writing, no standard existed below half precision floating point.Source: §3 · p. 3

4 FIXED POINT

Fixed-point values use a signed mantissa and a shared global scaling factor, which acts as the radix-point position. The factor is typically a power of two, allowing scaling multiplications to be replaced by shifts.

4 FIXED POINT

  • Fixed-point values share one global scaling factor, which acts as the radix-point position.Reducing the scaling factor reduces range and increases precision.Source: §4 · p. 3
  • A power-of-two scaling factor lets fixed-point scaling multiplications be replaced by shifts.Source: §4 · p. 3
  • Fixed-point formats rely on integer operations and are described as hardware-cheaper than floating point because the exponent is shared and fixed.The format is commonly found on embedded systems without an FPU.Source: §4 · p. 3

5 DYNAMIC FIXED POINT

Dynamic fixed point uses multiple periodically updated scaling factors shared by groups of variables, combining aspects of floating- and fixed-point formats. The paper notes that fixed point's single shared exponent is ill-suited to deep learning, where activations, gradients, and parameters have different ranges and gradient ranges diminish during training.

5 DYNAMIC FIXED POINT

  • A single shared fixed exponent is ill-suited to deep learning because activations, gradients, and parameters have different ranges and gradient ranges slowly diminish during training.Source: §5 · p. 3
  • Dynamic fixed point assigns several scaling factors to groups of variables and updates them over time, between floating point's per-scalar factors and fixed point's single global factor.The factors reflect statistics of the values in each group.Source: §5 · p. 4
  • Each layer's weights, bias, weighted sum, outputs, and corresponding gradients receive distinct scaling factors.Factors are initialized globally or found by higher-precision training, then updated at a set frequency using the stated policy.Source: §5 · p. 4

6 UPDATES VS. PROPAGATIONS

Parameter updates use higher precision than forward and backward propagation, allowing small changes to accumulate while saving some memory bandwidth during propagation. The authors relate this precision split to stochastic gradient descent’s implicit averaging.

6 UPDATES VS. PROPAGATIONS

  • Higher precision is used for parameter updates than for forward and backward propagation, preserving small parameter changes while saving memory bandwidth during propagation.The paper attributes the feasibility of this split to stochastic gradient descent’s implicit averaging.Source: §6 · p. 4
  • Each update adds a minibatch cost gradient scaled by the learning rate to the parameter trajectory.C_t(θ_t) is the minibatch cost at iteration t, and ϵ is the learning rate.Source: §6 · p. 4
  • Although successive gradient terms are dependent, minibatch sampling supplies the dominant variation and the slowly moving parameters allow a strong averaging effect.Each contribution is relatively small, making sufficient precision important when adding it to a larger value.Source: §6 · p. 4

7 MAXOUT NETWORKS

Maxout networks use hidden units that select the largest of several weighted-input dot products. The paper relates these units to rectifying units and notes their prior benchmark performance with dropout.

7 MAXOUT NETWORKS

  • A maxout unit outputs the maximum of k dot products between weight vectors and its input.The input can be the output of the preceding layer; the vectors and biases are unit parameters.Source: §7 · p. 4
  • A maxout unit can be seen as a generalization of the rectifying units (Jarrett et al., 2009; Nair and Hinton, 2010; Glorot et al., 2011; Krizhevsky et al., 2012b).Source: §7 · p. 4
  • Combined with dropout, maxout networks had achieved state-of-the-art results on several benchmarks in fully connected and convolutional deep nets.The paper describes dropout as approximating model averaging across subsets of the original network’s units.Source: §7 · p. 4

8 BASELINE RESULTS

The paper establishes baseline Maxout results on MNIST, CIFAR-10, and SVHN, using models with slightly less capacity than Goodfellow et al. (2013a). It describes the dataset overview and reports that the CIFAR-10 and SVHN test errors are higher than that work’s, under the stated procedural differences.

8 BASELINE RESULTS

  • Baseline Maxout models have slightly less capacity than Goodfellow et al. (2013a) and are evaluated on MNIST, CIFAR-10, and SVHN.Table 4 reports test-set error rates across these datasets and precision formats, with single precision serving as the experimental baseline.Source: §8 · p. 5, Table 4 · p. 5

8.1 MNIST

  • MNIST uses permutation-invariant and convolutional Maxout models; neither data augmentation nor unsupervised pre-training is used.Both models use minibatch SGD with momentum, dropout, and a weight-vector norm constraint.Source: §8.1 · p. 5

8.2 CIFAR-10

  • CIFAR-10 uses global contrast normalization and ZCA whitening; its test error is slightly larger than Goodfellow et al. (2013a)’s after reducing hidden units and excluding validation examples from training.The model has three convolutional Maxout layers, a fully connected Maxout layer, and a softmax layer.Source: §8.2 · p. 5

8.3 STREET VIEW HOUSE NUMBERS

  • SVHN uses local contrast normalization; its test error is larger than Goodfellow et al. (2013a)’s after reducing training length.The model has three convolutional Maxout layers, a fully connected Maxout layer, and a softmax layer.Source: §8.3 · p. 6

9 LOW PRECISION RESULTS

Across the tested formats, the section identifies precision settings that keep final test error near the single-precision reference, while dynamic fixed point permits lower bit-widths than fixed point. The reported settings are not universal: the dynamic fixed-point result has an exception for SVHN.

9.1 FLOATING POINT

  • Half-precision floating point has little to no impact on test-set error rate.The authors conjecture that high-precision fine-tuning could recover the small degradation.Source: §9.1 · p. 7

9.2 FIXED POINT

  • 19 (20 with the sign) bits for both propagations and parameter updates in fixed point have little impact on final test error.The radix point is placed after the fifth, or arguably sixth, most significant bit; lower bit-widths sharply raise test error.Source: Figure 2 · p. 6, Figure 3 · p. 6, §9.2 · pp. 7–8

9.3 DYNAMIC FIXED POINT

  • 9 (10 with the sign) propagation bits and 11 (12 with the sign) parameter-update bits in dynamic fixed point have little impact on final test error, except on SVHN.The experiments use a 0.01% maximum overflow rate; the reported settings are significantly better than fixed point.Source: Figure 3 · p. 6, Figure 4 · p. 7, §9.3 · p. 8

10 RELATED WORKS

The work distinguishes training with low-precision arithmetic from earlier uses of quantization on already-trained networks. It highlights dynamic fixed-point training, higher-precision weight updates, and evaluation on recent models and benchmarks as contributions.

10 RELATED WORKS

  • Vanhoucke et al. (2011) used 8-bit linear quantization for already-trained networks, whereas this work trains the networks.Their reported memory-footprint reduction was between 3× and 4×.Source: §10 · p. 8
  • The authors identify training deep networks with dynamic fixed point as a first.Source: §10 · p. 8
  • The work uses higher precision for weights during updates and trains recent models on recent benchmarks.Source: §10 · p. 8

11 CONCLUSION AND FUTURE WORKS

The paper concludes that very low-precision multipliers can suffice for training deep neural networks and that dynamic fixed point seems well suited to the task. It points to hardware applications and identifies broader tasks, models, and formats as future work.

11 CONCLUSION AND FUTURE WORKS

  • Very low-precision multipliers are sufficient for training deep neural networks.Source: §11 · p. 8
  • Dynamic fixed point seems well suited for training deep neural networks.Source: §11 · p. 8
  • Higher-precision parameter updates help, and the work may be exploited to optimize general-purpose hardware memory usage or design power-efficient deep-learning hardware.Source: §11 · p. 8
  • Future extensions include tasks beyond image classification, models beyond Maxout, and formats beyond floating point, fixed point, and dynamic fixed point.The authors also mention combining dynamic fixed point with stochastic rounding as an interesting direction.Source: §11 · pp. 8–9

Related papers

Loading 1412.7024v5…