Source-linked AI summary
Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio
TL;DR
The paper addresses the challenge of running deep neural networks efficiently on power- and memory-constrained hardware. It trains networks with binary weights and activations, yielding nearly state-of-the-art results across three datasets and a 7-times-faster MNIST run with a binary GPU kernel without accuracy loss.
Problem
Running deep neural networks on low-power devices is challenging because they are commonly trained on fast, power-hungry GPUs and require costly computation and memory access.
Method
The paper trains BNNs using binary weights and activations for runtime computation and parameter-gradient calculation, with binarization and gradient propagation specified in its training algorithm.
Results
BNNs achieve nearly state-of-the-art results on MNIST, CIFAR-10, and SVHN; a binary matrix multiplication GPU kernel runs the MNIST BNN 7 times faster than an unoptimized GPU kernel without classification-accuracy loss.
Takeaways & Limitations
Binary representations reduce memory size and accesses and replace most arithmetic operations with bit-wise operations, potentially improving power-efficiency.
Takeaways & Limitations
The first layer is an exception to the otherwise binary layer inputs because its input representation is typically non-binary, although it is often the smallest convolution layer.
Abstract
from arXiv · showhide
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs we conduct two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.
Introduction
The paper targets the difficulty of running power-hungry deep neural networks on low-power devices by introducing BNNs with binary weights and activations. Experiments across three datasets and two frameworks support their effectiveness, while binary operations and a specialized GPU kernel improve computational efficiency.
- Motivation: DNNs are difficult to run on low-power devices because training and inference commonly rely on fast, power-hungry GPUs.This motivates research into faster execution on general-purpose and specialized hardware.
- Contribution: The paper introduces BNNs, which use binary weights and activations during runtime and when computing parameter gradients during training.The method constrains neural-network values to binary representations while retaining a training procedure for the network.
- Validation: Experiments implemented in Torch7 and Theano show that BNNs can be trained on MNIST, CIFAR-10, and SVHN with nearly state-of-the-art results.The code for training and running the BNNs is available online in both frameworks.
- Contribution: BNNs drastically reduce memory consumption and replace most arithmetic operations with bit-wise operations during the forward pass.The authors expect these changes to substantially improve power-efficiency.
- Validation: 7 times faster: a binary matrix multiplication GPU kernel runs the MNIST BNN faster than an unoptimized GPU kernel without loss in classification accuracy.This result comes from the authors' specialized binary matrix multiplication implementation.
1. Binarized Neural Networks
BNNs train with binary weights and activations while retaining real-valued gradient accumulators and using straight-through backpropagation. The method combines binarization, clipping, normalization, and shift-based optimization to reduce computation while maintaining training effectiveness.
- Binarization: BNNs constrain weights and activations to +1 or −1, using deterministic sign binarization most often and stochastic binarization in selected train-time activation experiments.Stochastic binarization is harder to implement because it requires random bits during quantization.
- Training: Real-valued weight gradients are accumulated despite binary forward computations because stochastic gradient descent requires sufficient accumulator resolution.The accumulated noise from stochastic gradient contributions supports parameter updates.
- Gradient propagation: The straight-through estimator propagates gradients through discretization while canceling them when the underlying real-valued variable saturates beyond magnitude 1.The method uses deterministic sampling and accounts for the saturation effect.
- Training algorithm: Training uses binarized forward propagation, non-binary backpropagated gradients, weight clipping, batch normalization, and parameter updates through ADAM or shift-based AdaMax.Algorithm 1 supports deterministic or stochastic binarization, standard or shift-based batch normalization, and alternative update rules.
- Efficient training: Shift-based batch normalization approximates batch normalization with few multiplications, while shift-based AdaMax replaces multiplication-heavy ADAM; experiments observed no accuracy loss for either substitution.The batch-normalization motivation is especially strong for convolutional layers, where scaling calculations can be numerous.
- First layer: During BNN calculations, only binarized weights and activations are used, except for the first-layer input, which is typically smaller because input channels are fewer than internal channels.This makes the first layer a limited exception to binary internal representations in computer-vision networks.
2. Benchmark Results
Across Torch7 and Theano, BNNs were trained on MNIST, CIFAR-10, and SVHN with near state-of-the-art results. The experiments used differing activation-binarization and optimization variants across frameworks and datasets.
- BNNs achieved near state-of-the-art classification results on MNIST, CIFAR-10, and SVHN in both framework-based experiment sets.
- The Torch7 experiments stochastically binarized activations at train-time, whereas Theano experiments used deterministic binarization.
- MNIST: The MNIST MLP used three hidden layers of 4096 binary units and an L2-SVM output layer without convolution, augmentation, preprocessing, or unsupervised learning.
- CIFAR-10: The CIFAR-10 ConvNet used no preprocessing or data augmentation, trained for 500 epochs, and selected test error using the best validation error.
- SVHN: The SVHN experiments followed the CIFAR-10 procedure, using half as many convolutional-layer units and 200 training epochs.
3. Very Power Efficient in Forward Pass
BNNs reduce forward-pass storage and arithmetic requirements by using binary weights and activations. The paper reports 32-fold reductions in memory size and accesses relative to 32-bit DNNs, alongside hardware opportunities from repeated binary filters.
- Bit-wise Operations: Binary weights and activations replace most arithmetic operations with bit-wise operations during the forward pass.
- Memory Size and Accesses: BNNs require 32 times smaller memory size and 32 times fewer memory accesses than 32-bit DNNs.The paper expects this reduction to lower energy consumption by more than 32 times.
- Bit-wise Operations: A 32-bit floating-point multiplier costs about 200 Xilinx FPGA slices, whereas a 1-bit XNOR gate costs one slice.
- Exploiting Filter Repetitions: For 3 × 3 binary filters, the maximum number of unique 2D filters is 2^9 = 512, enabling repeated-filter reuse in convolution hardware or software.
- Exploiting Filter Repetitions: Dedicated hardware could reduce convolution time complexity by 60% by exploiting repetitions among binary convolution kernels.
4. Seven Times Faster on GPU at Run-Time
The paper accelerates BNN inference with a SWAR-based XNOR GPU kernel. The kernel substantially speeds binary matrix multiplication and the MNIST MLP without changing classification accuracy.
- SWAR Implementation: SWAR concatenates 32 binary variables into 32-bit registers, producing a 32-times speed-up on bitwise operations such as XNOR.
- SWAR Implementation: 32 connections can be evaluated with accumulation, popcount, and XNOR instructions totaling 6 clock cycles on recent Nvidia GPUs.
- GPU Kernel: 23 times faster than the baseline kernel and 3.4 times faster than cuBLAS, the XNOR kernel accelerates 8192 × 8192 × 8192 binary matrix multiplication.
- MNIST MLP: 7 times faster than the baseline kernel, the MNIST MLP runs without any loss in classification accuracy.
- Kernel Correctness: The baseline and XNOR kernels return identical outputs when their inputs are constrained to −1 or +1.
5. Discussion and Related Work
Earlier work established that binary weights or restricted components could preserve performance, but the paper identifies fully binary deep-network training and inference as an unresolved goal. Its BNNs extend binarization to activations, while retaining full-precision weights as a training bottleneck.
- Earlier methods showed good performance with binary weights and neurons, but EBP used binarized parameters only during inference.
- BinaryConnect binarized CNN weights while retaining full-precision neurons and using real-valued weights as references for binarization.
- Other approaches either retained full-precision weights during testing, adjusted only one weight layer, or compressed already-trained full-precision networks.
- BNNs binarize both weights and neurons during inference and throughout training, extending stochastic-noise reasoning from weights to activations.The authors state this had not previously been achieved for a deep network.
- Binary activations are especially important for ConvNets because they typically contain many more neurons than free weights.
- Full-precision weights must still be saved during training, creating a remaining computational bottleneck that future memory devices might alleviate.
Conclusion
The paper introduces BNNs with binary weights and activations during runtime and gradient computation, then evaluates them across three datasets and two frameworks. BNNs achieve nearly state-of-the-art results while reducing forward-pass memory and arithmetic costs; a binary GPU kernel runs the MNIST MLP 7 times faster without accuracy loss.
- BNNs use binary weights and activations at runtime and when computing parameter gradients during training.
- Experiments on Torch7 and Theano show that BNNs can train on MNIST, CIFAR-10, and SVHN with nearly state-of-the-art results.
- BNNs drastically reduce forward-pass memory size and accesses while replacing most arithmetic operations with bit-wise operations.
- 7 times faster: a binary matrix multiplication GPU kernel runs the MNIST MLP without loss in classification accuracy versus an unoptimized GPU kernel.
- Future work should extend speed-ups to training and broaden benchmarks to models such as RNNs and datasets such as ImageNet.