Source-linked AI summary
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David
TL;DR
Deep neural networks rely heavily on costly real-valued multiplications, motivating more hardware-efficient computation. BinaryConnect trains with binary propagation weights while retaining precise stored weights, achieving near-state-of-the-art results on MNIST, CIFAR-10, and SVHN.
Problem
Deep neural-network computation relies heavily on expensive real-valued multiplications, creating a hardware-efficiency challenge for training and application.
Method
BinaryConnect uses binary weights during forward and backward propagations while retaining real-valued weights for gradient accumulation.
Results
BinaryConnect achieves near-state-of-the-art results on permutation-invariant MNIST, CIFAR-10, and SVHN while acting as a regularizer.
Takeaways & Limitations
BinaryConnect supports training deep networks with binary propagation weights while replacing many multiply-accumulate operations with additions and subtractions.
Takeaways & Limitations
BinaryConnect requires sufficient precision in stored accumulators to accumulate and average stochastic gradients.
Abstract
from arXiv · showhide
Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.
1 Introduction
BinaryConnect eliminates weight–activation and weight–gradient multiplications by using binary weights during forward and backward propagation while retaining sufficient precision for accumulated gradients. The method acts as a regularizer and achieves state-of-the-art or near-state-of-the-art results on permutation-invariant MNIST, CIFAR-10, and SVHN.
- Introduction: BinaryConnect forces propagation weights to two possible values, eliminating multiplications between real-valued weights and activations or gradients.Binary weights are used during both forward and backward propagation.
- Introduction: Sufficient precision is retained to accumulate and average stochastic gradients, while noisy discretized weights remain compatible with SGD.The passage characterizes stochastic discretization as a form of noise that SGD can average out.
- Introduction: BinaryConnect acts as a regularizer and achieves near-state-of-the-art results on permutation-invariant MNIST, CIFAR-10, and SVHN.The reported evaluations cover all three datasets.
- Introduction: The article introduces BinaryConnect, demonstrates its regularization effect and benchmark performance, and releases the implementation code.The contributions are presented in Sections 2 and 3, with code made available.
2 BinaryConnect
BinaryConnect constrains propagation-time weights to +1 or −1, replacing many multiply-accumulate operations with cheaper additions and subtractions. It preserves real-valued weights for SGD updates, clips them to [−1, 1], and supports deterministic or stochastic binarization and corresponding inference choices.
- Propagation-time binarization: BinaryConnect constrains weights to +1 or −1 during forward and backward propagations, replacing many multiply-accumulate operations with simple additions and subtractions.Fixed-point adders require less area and energy than fixed-point multiply-accumulators.
- Propagation-time binarization: Stochastic binarization samples +1 with probability p = σ(w) and −1 with probability 1 −p, enabling discretization errors across weights to approximately cancel.The method preserves the expected value of the discretized weight.
- Training: BinaryConnect binarizes weights only during forward and backward propagation, while accumulating SGD updates in real-valued parameters.At training time, the binary values are randomly selected for each weight and minibatch in both propagation phases, whereas updates remain real-valued.
- Training: After each update, BinaryConnect clips real-valued weights to [−1, 1] because larger magnitudes do not affect binarization and would otherwise grow without impact on binary weights.Clipping also reflects the practice of bounding weights for regularization.
- Inference: For deterministic BinaryConnect, inference uses binary weights; for stochastic BinaryConnect, the experiments use real-valued weights, while sampling ensembles is an alternative.The stochastic choice focuses on training benefits rather than faster test-time performance.
3 Benchmark results
BinaryConnect achieves near state-of-the-art results on permutation-invariant MNIST, CIFAR-10, and SVHN while acting as a regularizer. Experiments use constrained benchmark protocols, with results reported after validation-based model selection.
- Benchmark results: BinaryConnect obtains near state-of-the-art results on permutation-invariant MNIST, CIFAR-10, and SVHN.The benchmark section frames BinaryConnect as a regularizing method evaluated across these three datasets.
- MNIST: On MNIST, the model uses three hidden layers of 1024 rectifier linear units without convolution, data augmentation, preprocessing, or unsupervised pretraining.Permutation invariance forbids CNNs, and the dataset contains 60,000 training and 10,000 test 28 × 28 grayscale digit images.
- MNIST: The stochastic version of BinaryConnect acts as a regularizer on MNIST, although it is slightly less powerful than Dropout in this context.Each experiment is repeated six times with different initializations, and test error is selected using the best validation error after 1000 epochs.
- CIFAR-10: On CIFAR-10, experiments use a VGG-inspired CNN with global contrast normalization, ZCA whitening, ADAM, Batch Normalization, and no data augmentation.The dataset contains 50,000 training and 10,000 test 32 × 32 color images; the last 5,000 training samples form the validation set.
- SVHN: On SVHN, the CIFAR-10 procedure is adapted by halving the hidden units and training for 200 rather than 500 epochs.SVHN contains 604K training and 26K test 32 × 32 color images, and its results are reported in Table 2.
4 Related works
Prior binary-weight methods use Expectation Backpropagation or ternary-weight retraining, whereas BinaryConnect trains throughout with binary weights during propagations. This enables specialized hardware to avoid propagation multiplications, approximately 2/3 of all multiplications.
- Expectation Backpropagation: Earlier methods [38] train with Expectation Backpropagation, a variant based on Expectation Propagation, rather than standard Backpropagation.Their method optimizes a non-binary posterior distribution while retaining a real-valued weight version.
- Expectation Backpropagation: [37, 38] binarize both neuron outputs and weights, making them more hardware friendly than methods binarizing weights alone.They achieve good classification accuracy for fully connected MNIST networks but not yet for ConvNets.
- Ternary-weight methods: Other methods [39, 40] retrain neural networks with ternary weights during forward and backward propagations.They first train with high-precision weights, ternarize to −H, 0, and +H while adjusting H to minimize output error, then retrain with ternary propagations and high-precision updates.
- BinaryConnect: BinaryConnect trains all the way with binary weights during propagations, avoiding multiplications in forward and backward propagation on specialized hardware.This eliminates about 2/3 of the multiplications.
5 Conclusion and future works
BinaryConnect introduces weight binarization during forward and backward propagation, enabling DNN training with nearly state-of-the-art results on three benchmark datasets. The method could substantially benefit specialized hardware by removing about 2/3 of multiplications.
- Conclusion and future works: BinaryConnect binarizes weights during forward and backward propagations while training deep neural networks.The passage identifies this as the paper’s novel binarization scheme.
- Conclusion and future works: BinaryConnect trains DNNs on permutation-invariant MNIST, CIFAR-10, and SVHN with nearly state-of-the-art results.These are the three datasets explicitly reported in the conclusion.
- Conclusion and future works: About 2/3 of the multiplications could be removed in specialized hardware implementations of deep networks.The passage presents this reduction as a potential major hardware impact of BinaryConnect.