PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepShift: Towards Multiplication-Less Neural Networks

Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, Ye Henry Tian, Joey Yiwei Li

arXiv:1905.13298v5cs.LGcs.NE

TL;DR

CNN inference and training have high computation, memory, and power requirements, with convolutional and fully connected layers' multiplication use identified as a major contributor. DeepShift replaces multiplication-based convolution and linear operations with bitwise shifts and sign flips, and trains shift values directly or through regular weights constrained to powers of 2. DeepShift networks trained on CIFAR-10 and Imagenet, from scratch or pretrained full-precision weights, achieve or exceed nearly state-of-the-art results.

  • Problem

    CNN inference and training have high computation, memory, and power requirements, with convolutional and fully connected layers' multiplication use identified as a major contributor.

    Source: §1 · pp. 1–2

  • Method

    DeepShift replaces multiplication-based convolution and linear operations with bitwise shifts and sign flips, and trains shift values directly or through regular weights constrained to powers of 2.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    DeepShift networks trained on CIFAR-10 and Imagenet, from scratch or pretrained full-precision weights, achieve or exceed nearly state-of-the-art results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DeepShift's weight representation reduces model storage and memory footprint, with reported energy savings when transferring weights between memory levels.

    Source: §6 · p. 8

    The ResNet18 comparison uses a custom multiplication-based kernel that is slower than NVIDIA's cuDNN kernel, and the authors identify kernel optimization as future work.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

The high computation, memory, and power budgets of inferring convolutional neural networks (CNNs) are major bottlenecks of model deployment to edge computing platforms, e.g., mobile devices and IoT. Moreover, training CNNs is time and energy-intensive even on high-grade servers. Convolution layers and fully connected layers, because of their intense use of multiplications, are the dominant contributor to this computation budget. We propose to alleviate this problem by introducing two new operations: convolutional shifts and fully-connected shifts which replace multiplications with bitwise shift and sign flipping during both training and inference. During inference, both approaches require only 5 bits (or less) to represent the weights. This family of neural network architectures (that use convolutional shifts and fully connected shifts) is referred to as DeepShift models. We propose two methods to train DeepShift models: DeepShift-Q which trains regular weights constrained to powers of 2, and DeepShift-PS that trains the values of the shifts and sign flips directly. Very close accuracy, and in some cases higher accuracy, to baselines are achieved. Converting pre-trained 32-bit floating-point baseline models of ResNet18, ResNet50, VGG16, and GoogleNet to DeepShift and training them for 15 to 30 epochs, resulted in Top-1/Top-5 accuracies higher than that of the original model. Last but not least, we implemented the convolutional shifts and fully connected shift GPU kernels and showed a reduction in latency time of 25% when inferring ResNet18 compared to unoptimized multiplication-based GPU kernels. The code can be found at https://github.com/mostafaelhoushi/DeepShift.

1. Introduction

DeepShift replaces multiplication-based convolution and linear operations with bitwise shifts to address the computation demands of CNN training and inference. The paper introduces two training methods and reports results on CIFAR-10 and ImageNet, alongside a GPU-kernel implementation.

1. Introduction

  • CNN deployment to mobile and IoT devices faces constrained power, price, and memory budgets, while memory communication and model size can also affect power and connectivity costs.The introduction identifies model optimization, size reduction, faster inference, and lower power consumption as key research areas for these applications.Source: §1 · p. 1
  • Reducing training energy and time matters for both cloud/server and edge/mobile settings, including limited scenarios such as online learning and model adaptation.The passage cites substantial training energy use and rapidly increasing computation demands as motivation.Source: §1 · p. 1
  • Architecture search can require substantial training resources, while pruning reduces inference work but does not improve training efficiency because it starts from larger models.Quantization is another approach discussed; the paper notes that some quantization methods replace multiplication with bitwise shifts.Source: §1 · pp. 1–2
  • DeepShift replaces multiplication-based convolution and linear operations with bitwise shifts, and proposes DeepShift-Q and DeepShift-PS to train shift values and sign flips indirectly or directly.A bitwise shift is mathematically equivalent to multiplying by a power of 2.Source: §1 · p. 2
  • DeepShift training on CIFAR-10 and ImageNet achieves or exceeds nearly state-of-the-art results, whether started from scratch or pre-trained full-precision weights.The paper reports experiments for each training method.Source: §1 · p. 2
  • 25% faster inference was reported for ResNet18 using the bitwise-shift GPU kernels versus an unoptimized multiplication-based GPU kernel.The paper developed GPU kernels for bitwise-shift-based matrix multiplication and convolution.Source: §1 · p. 2

2. Related Work

Related work covers efficient architectures, quantization, pruning, and hardware approaches. DeepShift is positioned among power-of-two methods as a way to train models from scratch using such weights.

2. Related Work

  • Efficient architectures include SqueezeNet, MobileNets [32], MobileNetV2 [54], and Octave Convolution [12], while pruning methods reduce redundancy but can add sparse-arithmetic overhead.The related-work section also lists efficient kernel libraries and custom reduced-precision hardware.Source: §2 · pp. 2–3
  • Quantization methods range from binary and ternary weights to configurable or multi-bit schemes; some quantize activations as well as weights.Examples include Binary Connect [16], Binary Weight Networks [53], Ternary Weight Networks [39], and other methods listed in the passage.Source: §2 · pp. 2–3
  • INQ [64] quantizes pre-trained weights to zeros and powers of two for inference, using iterative partitioning and empirically tuned fractions of weights.Unlike INQ, DeepShift does not introduce a new hyperparameter.Source: §2 · pp. 2–3
  • Other power-of-two approaches include LogNN [52], LogQuant, and ShiftCNN [23]; ShiftCNN replaces a multiplication with two or three shifts, while DeepShift uses one.The passage also notes LogNN's manual empirical tuning and that it does not convert the first layer.Source: §2 · p. 3
  • DeepShift is described as the first method to train a model from scratch using powers of 2.The introduction also distinguishes DeepShift from INQ, which requires a pre-trained model.Source: §2 · pp. 2–3

3. DeepShift Networks

DeepShift replaces multiplication in linear and convolution operators with bitwise shifts and sign flips, and introduces DeepShift-Q and DeepShift-PS as distinct training approaches.

3. DeepShift Networks

  • Integer or fixed-point bitwise shifts encode multiplication by powers of 2, while ternary sign flips provide negative, zero, or positive weight signs.LinearShift and ConvShift apply shift and sign matrices in place of ordinary weights.Source: §3 · p. 3

3.1. DeepShift-Q

  • DeepShift-Q rounds regular weights to the nearest power of 2 during forward and backward passes, using a straight-through estimator to enable weight updates.Without the estimator, the rounded-weight derivative is zero except at powers of 2, preventing weight updates.Source: §3.1 · p. 4

3.2. DeepShift-PS

  • DeepShift-PS directly trains shift and sign parameters, with a ternary sign factor that can leave a value unchanged, zero it, or flip its sign.Its backward pass uses straight-through estimators for rounding and sign, and a modified estimator for the flip function.Source: §3.2 · p. 5

4. Implementation

The implementation emulates bitwise-shift precision with fixed-point inputs and biases, and constrains trained shift values to reduce weight-representation requirements.

4. Implementation

  • 32-bit fixed-point inputs and biases, with 16 bits each for integer and fractional parts, are used to emulate bitwise-shift hardware precision.Both LinearShift and ConvShift round their input and bias before the forward pass.Source: §4 · p. 5
  • DeepShift-Q uses Kaiming initialization for Wq, while DeepShift-PS initializes P and S from a uniform random distribution.The PS regularization term is applied to the effective weight WPS rather than separately to P and S, avoiding pressure toward numerically small but effectively large weights.Source: §4 · p. 5
  • Constraining shifts to 0 through -14 or 0 through -15 retains high accuracy in testing and allows weights to use 5 bits, including the sign bit.Although the theoretical shift range is -31 to +31 and needs about 6 bits, trained shifts are rarely positive; out-of-range shifts are clipped.Source: §4 · p. 5

5. Benchmark Results

The experiments test DeepShift variants on MNIST, CIFAR10, and ImageNet, using scratch and pretrained starts. CIFAR10 results are near baseline overall, with MobileNetv2 DeepShift-Q above baseline; pretrained ImageNet methods meet or exceed baseline accuracy for most architectures.

5. Benchmark Results

  • The benchmark covers CIFAR10 and ImageNet, comparing standard operators with shift operators trained from scratch or converted from pretrained 32-bit baselines.Source: §5 · pp. 5–6
  • DeepShift-PS uses RAdam for best validation accuracy, whereas DeepShift-Q and regular 32-bit training use SGD.The authors tested baseline ResNet18 with both optimizers to check that RAdam did not unfairly advantage DeepShift-PS.Source: §5 · p. 6, §5.2 · p. 6

5.1. MNIST Data Set

  • The MNIST evaluation trained two simple models: a three-layer fully connected network and a CNN with two convolutional and two linear layers.Source: §5.1 · p. 6

5.2. CIFAR10 Data Set

  • CIFAR10 shift training reaches almost the multiplication-based baseline accuracy, and MobileNetv2 with DeepShift-Q exceeds it.The numerical validation results are reported for multiple scenarios.Source: §5.2 · p. 6, Table 2 · p. 7

5.2.1 Lower Bit Widths

  • At 3- and 2-bit weight widths, scratch training is more accurate than pretrained initialization, while DeepShift-PS outperforms DeepShift-Q.The authors attribute the scratch advantage to pretrained weights being clipped to the same small values.Source: §5.2.1 · p. 7
  • For 8- or 16-bit activations trained from scratch, DeepShift-Q outperforms DeepShift-PS; from pretrained weights, both are close to baseline accuracy.Source: §5.2.1 · p. 7

5.3.1 Training from Pre-Trained Baseline

  • On ImageNet pretrained models, INQ and DeepShift exceed or nearly meet baseline accuracy for most architectures; DeepShift also beats ABC at the same 5-bit weight width.Source: §5.3.1 · p. 8

5.3.2 Training from Scratch

  • For ImageNet scratch training, the paper compares ResNet18 DeepShift with BWN and TWN, which replace multiplications with sign flips only.Models were trained for 90 epochs; the supplied passages identify Table 5 as the results location but provide no outcome values.Source: §5.3.2 · p. 8

6. Efficiency Analysis

DeepShift replaces multiplications with bitwise shifts, while its GPU kernel reports faster ResNet18 inference than a custom multiplication-based kernel. The section also describes weight-storage benefits and implementation overheads that constrain kernel performance.

6. Efficiency Analysis

  • A bitwise shift replaces each multiplication; in the 32-bit Intel Atom example, shifts take 1 cycle versus 5 to 6 cycles for integer and floating-point multiplication.The paper also cites lower multiplier power and area than bitwise shifters on 16-bit architectures.Source: §6 · p. 8
  • Representing weights with fewer bits compresses model storage, reducing memory footprint and energy for transfers between GPU or CPU memory levels.The paper identifies memory footprint as a primary constraint in mobile and IoT applications and shifts as a candidate for in-memory computing.Source: §6 · pp. 8–9

6.1. GPU Implementation

  • 26% speedup was reported for the ResNet18 ImageNet GPU kernel against a custom multiplication-based kernel, which is slower than NVIDIA’s cuDNN kernel.The comparison is therefore not against the optimized cuDNN implementation.Source: §6.1 · p. 9
  • The authors identify convolution-ReLU fusion, tiling-factor tuning, and runtime JIT compilation as future kernel optimizations.These are proposed for comparing bitwise-shift kernels with NVIDIA’s cuDNN multiplication-based kernels.Source: §6.1 · p. 9
  • The GPU instruction set was not optimized for small-width shift and sign values, while extracting packed values and conditional branches added kernel overhead.The reported implementation extracts 4-bit shifts and 1-bit signs from 32-bit integers.Source: §6.1 · p. 9

7. Future Work

DeepShift trained from pre-trained full-precision models reaches accuracy above or near the originals, while training from scratch incurs a Top-5 accuracy drop. The authors identify hardware realization and optimized vector or instruction support as directions for further work.

7. Future Work

  • Pre-trained full-precision models trained with shifts and sign flips achieved accuracy above or almost the same as the original models; training from scratch reduced Top-5 accuracy by 2% to 3%.The authors state that the pre-trained result can minimize energy and latency for on-device training, while scratch training can save cloud energy.Source: §7 · p. 10
  • Hardware realization remains necessary to evaluate actual speedup; faster execution than multiplication counterparts is presented as an expectation.Source: §7 · p. 10
  • DeepShift proved suitable for ImageNet, while the paper describes BNNs as performing well on smaller datasets but suffering significant degradation on ImageNet.Source: §7 · p. 10
  • The implemented shift-convolution GPU kernel is described as a proof of concept, with improved GPU instruction-set support proposed for small-width data and shifts.One proposed instruction would reverse shift direction when the shift value is negative.Source: §7 · p. 10
  • Future directions include vector shift instructions, FPGA evaluation, dedicated vector shift-and-negate instructions, and training with power-of-two activations or fixed-point gradients.The paper specifically suggests 8-bit or 16-bit fixed-point gradients to speed training.Source: §7 · p. 11

Related papers

Loading 1905.13298v5…