PaperlayerView exact revision on arXiv

Source-linked AI summary

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, Yuheng Zou

arXiv:1606.06160v3cs.NEcs.LG

TL;DR

DoReFa-Net addresses the challenge of training CNNs with low-bitwidth weights and activations while also reducing gradient precision. It stochastically quantizes parameter gradients and uses low-bitwidth convolutions, achieving comparable accuracy to 32-bit counterparts, including 46.1% top-1 accuracy for a low-bit AlexNet on ImageNet.

  • Problem

    Training and deploying DCNNs is hindered by large parameter counts, high computational complexity, and backward passes that previously retained high-precision gradients.

  • Method

    DoReFa-Net deterministically quantizes weights and activations, stochastically quantizes parameter gradients, and applies bit convolution kernels during forward and backward passes.

  • Results

    46.1% top-1 accuracy was obtained on the ImageNet validation set by a DoReFa-Net AlexNet with 1-bit weights, 2-bit activations, and 6-bit gradients trained from scratch.

  • Takeaways & Limitations

    DoReFa-Net demonstrates comparable prediction accuracy to 32-bit counterparts while supporting bit-convolution acceleration for training and inference.

  • Takeaways & Limitations

    A considerable accuracy gap remains between the best trained-from-scratch model and a model initialized with a 32-bit model, leaving gap closure for future work.

Abstract

from arXiv · show

We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1\% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.

1 INTRODUCTION

DoReFa-Net targets the computational and training costs of DCNNs by quantizing weights, activations, and gradients so bit convolution kernels can operate during both passes. It explores bitwidth configurations and reports comparable accuracy, including a 46.1% ImageNet result for a low-bit AlexNet.

  • Motivation: DCNN parameter counts and computational complexity impede embedded deployment and slow research iteration.Training on large datasets such as ImageNet may take weeks on modern multi-GPU servers.
  • Motivation: Low-bitwidth weights and activations reduce model size and computation, enabling bitwise convolution kernels during the forward pass.BNN and XNOR-Net binarize both convolutional weights and input activations.
  • Research gap: Prior methods had not quantized backward-pass gradients below 8 bits while retaining comparable accuracy; BNN and XNOR-Net kept gradients in full precision.Their backward convolutions therefore still combine 1-bit numbers with 32-bit floating-point values.
  • Approach: DoReFa-Net generalizes binarized neural networks to arbitrary bitwidths for weights, activations, and gradients, enabling bit convolutions in forward and backward passes.This design targets acceleration of both inference and training operations.
  • Hardware implications: DoReFa-Net can use bit convolutions on CPU, FPGA, ASIC, and GPU, potentially reducing energy consumption on FPGA and ASIC hardware.The stated hardware consequence concerns accelerating low-bitwidth neural-network training.
  • Results: 46.1% single-crop top-1 accuracy was achieved by a released DoReFa-Net AlexNet model on the ILSVRC12 validation set.The passage identifies the model as a DoReFa-Net 3 derived from AlexNet and notes a TensorFlow release.

2 DOREFA-NET

DoReFa-Net combines bit convolution kernels with low-bitwidth quantization for weights, activations, and parameter gradients. Its bitwise fixed-point formulation makes low-bitwidth dot products and convolutions computable through binary operations.

  • Quantization: DoReFa-Net quantizes weights and activations deterministically while quantizing parameter gradients stochastically.The formulation is designed for neural networks with low-bitwidth weights and activations and low-bitwidth gradients.
  • Formulation: The method first exploits bit convolution kernels, then quantizes weights, activations, and gradients to low-bitwidth numbers.These are the two main formulation stages described for DoReFa-Net.
  • Bit convolution: A 1-bit dot-product kernel can compute dot products and convolutions for low-bitwidth fixed-point integers.The construction extends the binary kernel beyond strictly 1-bit operands.
  • Bit convolution: For bitwidths M and K, the fixed-point dot product is decomposed into bit-plane pairings weighted by 2^(m+k) and counted with bitwise AND operations.The supplied formulation states that this computation has O(MK) complexity.
  • Bit convolution: For vectors with values in {-1, 1}, the dot product can instead use XNOR and bitcount operations.The corresponding expression is N − 2 × bitcount(xnor(x, y)).

2.2 STRAIGHT-THROUGH ESTIMATOR

The straight-through estimator addresses the zero-gradient problem created by low-bitwidth quantization. DoReFa-Net uses a quantization STE whose forward output is low-bitwidth while its backward operation remains usable for optimization.

  • Motivation: A k-bit representation has only 2^k possible real values, so a continuous function mapping into that finite set has zero input gradient.This motivates using a straight-through estimator to bypass the mismatch between quantization and differentiation.
  • STE principle: An STE is an operator whose forward and backward operations can be defined separately.This flexibility circumvents the nondifferentiability of low-bitwidth sampling or quantization.
  • DoReFa-Net STE: DoReFa-Net’s quantize_k STE maps an input r_i in [0, 1] to a k-bit output r_o in [0, 1].The passage introduces this STE as the one used extensively in the method.
  • Computational consequence: Because the quantized output is representable with k bits, dot products of such values can use fixed-point integer operations after appropriate scaling.This connects the STE’s quantized outputs to efficient bit convolution computation.

2.3 LOW BITWIDTH QUANTIZATION OF WEIGHTS

DoReFa-Net quantizes weights to arbitrary bitwidths using STE-based transformations, including constant scaling for binary weights and tanh range limitation for higher bitwidths.

  • STE formulation: DoReFa-Net uses STEs because low-bitwidth quantization otherwise produces zero gradients for continuous inputs.The STE permits arbitrary forward and backward operations.
  • Binary weights: Binary weights use an STE with a constant scalar scaling all filters instead of XNOR-Net's channel-wise scaling.The constant scaling preserves compatibility with bit convolution kernels during backward propagation.
  • Higher-bit weights: For k > 1, weights are transformed with tanh, normalized into [0, 1], quantized to k-bit fixed-point values, and mapped back to [−1, 1].The maximum is taken over all weights in the layer before applying quantizek.
  • Binary weights: 15: For k = 1, Eqn. 9 differs from Eqn. 7, but the authors find the difference insignificant in experiments.

2.4 LOW BITWIDTH QUANTIZATION OF ACTIVATIONS

DoReFa-Net uses STE-based activation quantization and stochastic gradient quantization so convolutional training can operate on low-bitwidth values.

  • Activations: Activation quantization is designed to replace floating-point convolutions with less computation-intensive bit convolutions.
  • Activations: The authors avoid directly copying BNN and XNOR-Net activation binarization because it failed to reproduce XNOR-Net results and was reported to degrade ImageNet accuracy.
  • Gradients: Gradient quantization is stochastic because deterministic quantization was ineffective for low-bitwidth gradients.The authors state this agrees with experiments using 16-bit weights and gradients.
  • Gradients: Gradient values are affinely mapped into [0, 1], quantized, and mapped back using a scale based on the maximum gradient magnitude per instance.The maximum excludes the mini-batch axis, giving each mini-batch instance its own scaling factor.
  • Gradients: 39: Uniform noise with magnitude matching the quantization error is added to compensate potential bias, and the authors find it critical for good performance.The noise is N(k) = σ/2^(k−1), with σ sampled uniformly from (−0.5, 0.5).
  • Backward pass: Gradient quantization is applied only during the backward pass through an STE on each convolution output.

2.6 THE ALGORITHM FOR DOREFA-NET

Algorithm 1 trains DoReFa-Net with independently specified weight, activation, and gradient bitwidths, while performing expensive operations on low-bitwidth numbers.

  • Training algorithm: Forward, backward-input, and backward-weight operations in convolutional and fully connected layers operate on low-bitwidth numbers.These values have affine mappings to fixed-point integers, enabling acceleration through fixed-point integer dot products.

2.7 FIRST AND THE LAST LAYER

The first and last network layers interface directly with inputs and outputs, so the experiments generally leave them partially or fully unquantized to avoid accuracy degradation.

  • Layer differences: The first and last DCNN layers differ from internal layers because they interface with the network's input and output.The first layer may receive 8-bit image features, while the output is often approximately one-hot.
  • First layer: Most experiments leave the first layer unquantized because it has few input channels, contributes little computation, and may be more accuracy-sensitive.The motivation includes prior evidence that sparsity in the first convolutional layer causes more degradation than in other convolutional layers.
  • Last layer: When the number of classes is small, the last fully connected layer is also left intact unless otherwise noted.Gradients back-propagated from the final fully connected layer are still quantized.
  • Empirical evaluation: The paper states that empirical evidence for these layer-specific choices is presented in Section 3.3.

2.8 REDUCING RUN-TIME MEMORY FOOTPRINT BY FUSING NONLINEAR FUNCTION AND ROUNDING

DoReFa-Net reduces run-time memory and non-bitwise overhead by fusing quantization-related operations. Monotonicity and max-pooling allow these fusions to be implemented through comparisons and commuting quantization.

  • Full-precision activation storage can consume substantial run-time memory and floating-point computation when the nonlinear function uses floating-point arithmetic.
  • Fusing Steps 3, 4, and 6 avoids storing intermediate full-precision results during activation quantization.
  • When h is monotonic, each quantized activation value corresponds to a non-overlapping input range, enabling fixed-point comparisons instead of intermediate-value generation.
  • Fusing later gradient-processing steps avoids generating and storing intermediate g_a^k values.
  • For max-pooling, quantization commutes with the maximum operation: quantize_k(max(a, b)) = max(quantize_k(a), quantize_k(b)).

3 EXPERIMENT RESULTS

Experiments on SVHN and ImageNet examine how weight, activation, and gradient bitwidths affect accuracy, resource use, and training behavior. Low-bit configurations reduce resource requirements, while suitable choices—especially 6-bit gradients with 1-bit weights and 2-bit activations—can retain accuracy close to 32-bit counterparts, though training from scratch remains less accurate than initialized training.

  • Configuration-space exploration: Gradients are generally more bitwidth-sensitive than activations, which are more sensitive than weights; G ≤4 significantly degrades prediction accuracy.
  • ImageNet: 46.1% single-crop top-1 accuracy is achieved by a released AlexNet-derived DoReFa-Net on the ILSVRC12 validation set.
  • ImageNet: Increasing activation bitwidth from 1-bit to 2-bit or 4-bit with 1-bit weights substantially improves ImageNet accuracy toward the 32-bit-weights-and-activations model.
  • ImageNet: 6-bit gradients produce similar accuracies to 32-bit gradients for the “1-1-6” versus “1-1-32”, “1-2-6” versus “1-2-32”, and “1-3-6” versus “1-3-32” comparisons.
  • ImageNet: A considerable accuracy gap remains between models trained from scratch and models initialized from 32-bit models, and closing it is left for future work.
  • Training curves: 6-bit gradient quantization leaves the training curve broadly similar to unquantized gradients, whereas 4-bit gradients in “1-2-4” cause significant accuracy degradation.
  • First and last layers: Quantizing first and last layers causes significant accuracy degradation, especially in models with fewer channels, supporting the practice of leaving those layers unquantized.

4 DISCUSSION AND RELATED WORK

Prior binarized networks accelerate forward convolutions but retain floating-point gradients during backpropagation. DoReFa-Net addresses this gap while experiments indicate comparable accuracy at gradients of 6 bits and below.

  • Related work: Binarized networks accelerate forward convolutions, but backward passes still convolve floating-point gradients with weights.
  • Control experiments: Table 3 examines degradation from additionally quantizing the first convolutional layer and last fully connected layer.The (1, 2, 4) row is the baseline without quantizing those layers; “+ first” and “+ last” add their respective quantization.
  • Related work: Lin et al. reduce some multiplications to bit-shifts, but high-bitwidth additions remain similarly numerous and limit overall speedup.
  • Related work: Gradient quantization before communication reduces distributed-training traffic but leaves full-precision gradients in the backward pass.
  • DoReFa-Net: 6-bit and lower gradients can retain comparable prediction accuracy without increasing channels, including for AlexNet on ImageNet.

5 CONCLUSION AND FUTURE WORK

DoReFa-Net trains networks with low-bitwidth weights, activations, and parameter gradients using different quantization strategies. The method supports bit-convolution acceleration during training and inference while retaining comparable accuracy in the reported SVHN and ImageNet experiments.

  • DoReFa-Net uses deterministic quantization for weights and activations, but stochastic quantization for gradients.
  • DoReFa-Net enables bit-convolution kernels for most forward and backward convolutions, accelerating both training and inference.
  • 46.1% top-1 accuracy was obtained on the ImageNet validation set by an AlexNet-derived model with 1-bit weights, 2-bit activations, and 6-bit gradients.
  • Future work: FPGA training remains future work because FPGA computation units require O(B^2) resources for B-bit arithmetic, favoring low-bitwidth convolutions.
Loading 1606.06160v3…