Source-linked AI summary
Bitwise Neural Networks
Minje Kim, Paris Smaragdis
TL;DR
Neural networks can exceed the resource limits of embedded and always-on systems because floating- and fixed-point computation requires substantial hardware resources. The paper proposes fully bitwise neural networks trained with compression and noisy backpropagation, and reports competitive MNIST performance with computational savings. BNNs are therefore aimed at resource-constrained deployments where arithmetic operations are expensive.
Problem
Growing neural networks demand processing power, memory, and battery resources, while fixed-point implementations still require relatively complex and power-intensive arithmetic.
Method
The paper develops networks in which inputs, weights, biases, hidden signals, and outputs are binary, using XNOR and bit counting for feedforward and weight compression plus noisy backpropagation for training.
Results
BNNs perform well on binarized MNIST inputs with very small additional errors compared with corresponding real-valued networks.
Takeaways & Limitations
BNNs offer a computationally efficient neural-network design for resource-constrained situations where floating-point or fixed-point variables and operations are prohibitively expensive.
Takeaways & Limitations
BNNs may require larger network structures than real-valued networks, and Boolean-function weight learning is an NP-complete problem.
Abstract
from arXiv · showhide
Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.
1. Introduction
Deep neural networks improve task performance but demand substantial computational and memory resources, motivating fully bitwise networks that replace arithmetic with efficient logic operations. The proposed BNN uses binary variables throughout and combines weight compression with noisy backpropagation for training.
- Motivation: Embedded applications face critically constrained processing power, memory, and battery resources as neural networks grow larger.Always-on sensing, vision, and speech applications are especially affected.
- Motivation: Fixed-point networks still require multiplication and addition, which use relatively complex logic and can consume substantial power.
- Proposed approach: BNNs represent inputs, outputs, weights, and biases with single bits, including a 1024 × 1025 binary weight matrix for two 1024-unit hidden layers.
- Proposed approach: The completely bitwise network uses XNOR and bit counting during feedforward instead of multiplication, addition, and nonlinear activation on real-valued variables.
- Training: Training combines weight compression for the real-valued model with noisy backpropagation to train the resulting binary network.
2. Feedforward in Bitwise Neural Networks
Bitwise feedforward represents Boolean inputs and parameters with bipolar binaries, computes agreement using XNOR, and generates outputs with a sign function. This design can match real-valued solutions in some cases but may require larger structures for others.
- Representation: Any Boolean function mapping binary inputs to binary outputs can be represented by a bitwise network with one hidden layer.The construction may simply memorize all possible input-output mappings.
- Forward propagation: Each layer computes XNOR-based binary products and applies a sign activation to generate bipolar outputs.B is the set of bipolar binaries, and the indices identify layers, input units, and output units.
- Training objective: Prediction error can be measured through bitwise agreement between targets and final-layer outputs using XNOR.During training, the error function can instead temporarily involve a softmax layer.
- Forward propagation: XNOR substitutes for binary multiplication, while bit counting implements the accumulation and thresholding needed for activation.Although implemented as 0/1 values in practice, the paper uses ±1 bipolar notation.
- Representational trade-offs: BNNs sometimes solve the same problem without size changes, but generally may require larger network structures than real-valued networks.The paper illustrates this with XOR and a linearly separable problem requiring multiple binary hyperplanes.
3. Training Bitwise Neural Networks
Training proceeds in two stages: a compressed real-valued network is trained first, then a binarized network is retrained with noisy backpropagation while accounting for binary weights and signals.
- 3.1. Real-valued Networks with Weight Compression: The first stage trains a real-valued network with inputs and compressed weights constrained to the range −1 to +1.The relaxed network uses tanh-wrapped weights and can use tanh activation as a relaxed counterpart to the bipolar BNN.
- 3.1. Real-valued Networks with Weight Compression: Weight compression changes backpropagation because hidden-layer errors and gradients use compressed weights and include the compression chain-rule term.The cited training procedure explicitly modifies error and gradient calculations for compressed parameters.
- 3.2. Training BNN with Noisy Backpropagation: The second stage initializes BNN parameters from the trained real-valued parameters and divides them into +1, 0, or −1 groups using a sparsity-controlled threshold.The sparsity parameter determines the binarization boundaries and the number of zero weights.
- 3.2. Training BNN with Noisy Backpropagation: Noisy backpropagation feedforwards with binarized weights and bit operations, then computes errors and gradients using the binarized weights and signals.This makes the training gradients and errors account for weight and signal binarization.
- 3.2. Training BNN with Noisy Backpropagation: Because gradients can be too small to update binary parameters directly, training updates corresponding real-valued parameters before re-binarizing them after each epoch.The update uses a learning-rate parameter and repeats the binarization procedure at every epoch.
4. Experiments
The experiments evaluate three-hidden-layer networks with 1024 units per layer on MNIST using several binary feature representations, comparing real-valued baselines with BNNs. BNNs perform well with very small additional errors relative to the corresponding real-valued networks.
- Experimental setup: The MNIST experiments use three hidden layers with 1024 units per layer and a softmax output layer for multiclass classification.The networks are evaluated on handwritten digit recognition.
- Real-valued baseline: The real-valued baseline uses dropout and tanh for both weight compression and activation to initialize the subsequent bipolar BNN.The baseline uses 64-bit floating-point parameters.
- Feature representations: The study compares real-valued and bitwise networks across bipolar, original 0/1, and two-bit fixed-point feature representations.The two-bit encoding expands the input to 1,568 binary nodes rather than 784.
- BNN results: BNNs are trained with binarized inputs and noisy backpropagation, and they perform well with very small additional errors.The original real-valued dropout network with similar topology has a classification error of 1.35%.
5. Conclusion
The paper proposes fully bitwise neural networks in which all participating values are single-bit variables operated on with simple bitwise logic. It identifies resource-constrained deployment as a target and proposes future investigation of bitwise convolutional networks.
- Conclusion: The proposed BNN represents inputs, weights, biases, hidden units, and outputs with single bits and operates on them using simple bitwise logic.The paper presents this design as computationally efficient for resource-constrained situations.
- Conclusion: The approach targets settings where floating-point or fixed-point variables and operations are prohibitively expensive.The stated future direction is a bitwise version of convolutive neural networks.