Source-linked AI summary
Neural Networks with Few Multiplications
Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, Yoshua Bengio
TL;DR
Training neural networks is time consuming because floating-point multiplications dominate computation. The paper stochastically binarizes weights and quantizes backpropagated representations to replace most multiplications with sign changes and bit-shifts. Across MNIST, CIFAR10, and SVHN, the approach preserves classification performance and can outperform full-precision or ordinary SGD training, while relying on bounded values for practical quantization.
Problem
Neural-network training is computationally demanding because most training computation consists of floating-point multiplications.
Method
The method stochastically binarizes weights for forward computation and quantizes representations during backpropagation so multiplications become sign changes or bit-shifts.
Results
Across MNIST, CIFAR10, and SVHN, the approach does not hurt classification performance and can yield better performance than standard stochastic gradient descent training.
Takeaways & Limitations
Eliminating most training multiplications could enable dramatically accelerated neural-network training through dedicated hardware implementations.
Takeaways & Limitations
Quantized backpropagation requires bounded values so sampled representations can use a fixed number of bits; otherwise exponent storage and computation increase.
Abstract
from arXiv · showhide
For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.
1 INTRODUCTION
Training deep neural networks is computationally demanding, time consuming, and memory intensive. The paper targets the floating-point multiplications that dominate training computation by addressing both forward hidden-state calculations and backward weight updates.
- Motivation: Training some state-of-the-art architectures can take weeks, while common speech-recognition and machine-translation models may require at least 12 Gigabytes of storage.These demands commonly motivate GPU or CPU clusters and parallelization strategies.
- Approach: The paper focuses on eliminating most floating-point multiplications because they account for most neural-network training computation.
- Approach: The method binarizes weights in the forward pass and uses quantized backpropagation to convert remaining multiplications into bit-shifts.The two components target hidden-state computation and backward error propagation, respectively.
2 RELATED WORK
Prior work reduced neural-network computation through power-of-two weights, Boolean networks, sparse representations, bit-stream connections, or quantized states and gradients. These approaches variously reduced training or testing computation but could incur performance, convergence, or training-phase limitations.
- Power-of-two weights: Power-of-two weight constraints can replace multiplications with binary shifts during training and testing, but may severely reduce performance and eliminate convergence guarantees.
- Boolean and sparse approaches: Completely Boolean networks simplify test-time computation but still require full-precision training, so their computation savings do not extend to training.
- Boolean and sparse approaches: Sparse representation classification and bit-stream networks replace or binarize connections to reduce floating-point multiplication, with the cited sparse approach retaining acceptable accuracy.
- Quantized training: Quantizing states, learning rates, and gradients to powers of two can eliminate multiplications with negligible performance reduction.
3 BINARY AND TERNARY CONNECT
Binary and ternary connect stochastically replace full-precision forward-pass weights with values that make matrix products multiplication-free. Ternary connect additionally permits zero weights, while both methods eliminate forward-pass multiplications.
- Binary connect: A layer with N inputs and M outputs computes y = h(Wx + b), requiring NM floating-point multiplications in Wx when h is ReLU.ReLU contributes no multiplications, so the matrix product contains the layer’s multiplications.
- Binary connect: Binary connect keeps full-precision weights as references and stochastically samples each forward-pass weight as −1 or 1.The sampling probability for 1 increases with how close the reference weight is to 1.
- Binary connect: Constraining reference weights to [-1, 1] keeps sampling probabilities within range, turning floating-point multiplications into sign changes.
- Ternary connect: Ternary connect extends stochastic sampling to zero weights because trained networks commonly contain weights that are zero or close to zero.
- Ternary connect: Both binary and ternary connect eliminate all multiplications in the forward pass.
4 QUANTIZED BACK PROPAGATION
Quantized back propagation eliminates most backward-pass multiplications by quantizing layer inputs to powers of two, while retaining a small number of element-wise multiplications.
- 4 QUANTIZED BACK PROPAGATION: Quantized back propagation targets multiplications in the backward pass, complementing forward-pass elimination.The method addresses weight updates and error-signal propagation after forward-pass multiplications have been eliminated.
- 4 QUANTIZED BACK PROPAGATION: Weight and bias updates form outer products between each layer’s input and its propagating error signal.These outer products are the main multiplication-heavy structures in the backward computation.
- 4 QUANTIZED BACK PROPAGATION: Quantizing layer inputs to integer powers of two converts the outer-product multiplications into binary shifts.The paper chooses x rather than the activation derivative because x is a hidden representation whose distribution is roughly known.
- 4 QUANTIZED BACK PROPAGATION: A maximum shift range of 3 to 4 bits is experimentally sufficient for the network to work well.The paper describes this quantization as tolerable relative to float32’s 24-bit mantissa.
- 4 QUANTIZED BACK PROPAGATION: With ReLU and reused forward-pass values, quantized back propagation leaves only element-wise products, totaling 3 × M multiplications.The remaining products involve the learning rate, activation derivative, and error-signal update for a layer with M outputs.
- 4 QUANTIZED BACK PROPAGATION: Compared with standard back propagation’s at least 2MN +3M multiplications, the remaining amount is negligible.The authors report that experiments show this reduction does not necessarily entail a performance loss.
5 EXPERIMENTS
Experiments across MNIST, CIFAR10, and SVHN evaluate multiplier-light training, showing competitive or improved accuracy while studying computation, convergence, and bit-shift robustness.
- General performance: Experiments cover fully connected and convolutional networks on MNIST, CIFAR10, and SVHN using stochastic gradient descent without momentum.Batch normalization is used for all models, and learning rates are adapted independently for each method.
- MNIST: 1.15% error rate is achieved on MNIST with ternary connect, quantized backpropagation, and batch normalization, versus 1.33% for full-precision training.Without batch normalization, the corresponding error rates are 1.48% and 1.67%; ternary connect at test time yields 1.49%.
- MNIST: Batch normalization adds 3BM + 3M multiplications per layer in the forward pass and roughly twice as many in the backward pass, regardless of binarization.Table 2 reports the remaining multiplication ratio after ternary connect and quantized backpropagation.
- SVHN: 2.99% error rate is obtained on SVHN when ternary connect is extended to test time, improving over ordinary SGD with binary or ternary connect and quantized backpropagation.The passage states that SVHN performance is consistent with the CIFAR10 results.
- Convergence: Ternary connect with quantized backpropagation surpasses ordinary backpropagation, binary connect, and binary connect with quantized backpropagation on CIFAR10.Binarization slows convergence but produces a better optimum after convergence, while quantizing error propagation does not hurt accuracy relative to binary connect.
- The effect of bit clipping: 2 to 10 maximum allowed bit shifts produce roughly unchanged MNIST performance across 10 independent runs, and even 2-bit shifts permit successful learning.This suggests the number of quantization bits need not be redefined for different tasks.
6 CONCLUSION AND FUTURE WORK
The paper proposes eliminating most floating-point multiplications during feedforward neural-network training, with potential acceleration through dedicated hardware. Surprisingly, the approach can improve prediction accuracy, while future work targets implementations, binarization, and recurrent networks.
- The method eliminates most floating-point multiplications used during feedforward neural-network training.
- Prediction accuracy tends to improve rather than degrade, likely because stochastic sampling regularizes models and low precision favors broader minima.
- Noise injection into gradients similarly encourages large-basin regions, lowers training loss, and improves generalization.
- Future work includes FPGA implementations, more efficient binarization, and extending the approach to recurrent neural networks.