Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Itay Hubara; Matthieu Courbariaux; Daniel Soudry; Ran El-Yaniv; Yoshua Bengio

Source-linked AI summary

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio

arXiv:1609.07061v1Sep 22, 2016cs.NEcs.LG

TL;DR

Conventional neural-network hardware requires many multiply-accumulate operations, making deployment on low-power devices challenging. This paper introduces Quantized Neural Networks that use low-precision weights and activations during inference and training, achieving accuracy comparable to 32-bit networks across several tasks while reducing memory access and arithmetic complexity.

Problem
Conventional hardware makes neural-network computation inefficient because of the massive number of multiply-accumulate operations, challenging deployment on low-power devices.
Method
Quantized Neural Networks train and run with low-precision weights and activations, replacing multiply-accumulate operations with XNOR and population-count operations.
Results
Across MNIST, CIFAR-10, SVHN, and ImageNet, quantized networks achieved accuracy comparable to 32-bit counterparts, while ImageNet results remained below the best full-precision results.
Takeaways & Limitations
Binarized networks drastically reduce memory size and accesses and replace most arithmetic with bit-wise operations, with estimated power efficiency improvements exceeding one order of magnitude.
Takeaways & Limitations
The binarized parameters in EBP are used only during inference, unlike the paper’s approach, which binarizes weights and activations throughout training as well.

Abstract

from arXiv · show

We introduce a method to train Quantized Neural Networks (QNNs) --- neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At train-time the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves $51\%$ top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.

1. Introduction

The paper introduces QNNs, which quantize weights and activations during inference and training to reduce computational and memory costs while preserving accuracy across several tasks. Experiments cover vision and language datasets, with additional evidence from low-bit gradients and optimized binary computation.

Approach: XNOR and population-count operations can replace all multiply-accumulate operations in QNNs.This design targets efficient hardware execution by substituting bit-wise operations for arithmetic computation.
Approach: QNNs quantize weights and activations during inference and training, including the extreme 1-bit-per-value Binarized Neural Network case.The method applies quantization when computing parameter gradients as well as during runtime inference.
Experiments: BNNs achieve near state-of-the-art results on MNIST, CIFAR-10, and SVHN, while ImageNet experiments include binary and higher-than-1-bit quantization.The experiments use Torch7 and Theano implementations.
Experiments: 4-bit recurrent QNNs achieve similar accuracy to 32-bit floating-point counterparts on Penn Treebank language modeling tasks.The reported models include vanilla RNNs and LSTMs.
Efficiency: QNNs reduce memory consumption and replace most arithmetic operations with bit-wise operations during the forward pass.These reductions apply at both run-time and train-time; dedicated hardware is argued to reduce time complexity by 60%.
Experiments: A binary matrix multiplication GPU kernel runs the MNIST BNN 7 times faster than an unoptimized GPU kernel without accuracy loss.The paper also reports online code for training and applying the networks.

2. Binarized Neural Networks

Binarized Neural Networks constrain weights and activations to ±1 while using straight-through gradient estimation and hardware-friendly normalization to train low-precision models. The approach preserves useful gradient information within a bounded range and can maintain accuracy while reducing multiplication-heavy computation.

2.1 Deterministic vs Stochastic Binarization: BNNs constrain weights and activations to either +1 or −1, using deterministic or stochastic functions to map real-valued variables into binary values.The deterministic sign function is used most often because stochastic binarization requires random-bit generation in hardware.
2.3 Propagating Gradients Through Discretization: Training accumulates real-valued weight gradients even though binary weights and activations are used to compute parameter gradients.The paper motivates real-valued accumulators because SGD relies on averaging small, noisy gradient contributions.
2.3 Propagating Gradients Through Discretization: Straight-through estimation preserves gradients for inputs within [−1,1] and cancels them when the real-valued input saturates beyond that range.The method addresses the sign function’s zero derivative by using a hard-tanh-like estimator during back-propagation.
2.3 Propagating Gradients Through Discretization: Weights are clipped to [−1,1] before deterministic quantization, preventing real-valued weights from growing without changing their binary values.The clipping is consistent with cancelling gradients when the weight magnitude exceeds 1.
2.4 Shift-based Batch Normalization: Shift-based batch normalization approximates vanilla batch normalization with binary shifts and additions, while the inverse square root remains the main non-shift operation.The standard-deviation vectors used for the inverse-square-root operation comprise only 0.3% of the CIFAR-10 network parameters.
2.4 Shift-based Batch Normalization: No accuracy loss was observed when replacing vanilla batch normalization or ADAM with their shift-based counterparts in the reported experiments.The alternatives are shift-based batch normalization and shift-based AdaMax.
2.6 First Layer: Only binarized weights and activations are used in BNN calculations, except that the first layer can retain continuous-valued inputs represented with fixed-point precision.The first convolution layer is often relatively small because image inputs have fewer channels than internal representations.

3. Qunatized Neural network - More than 1-bit

The paper extends quantization beyond one-bit representations by applying multi-bit and logarithmic schemes to weights, activations, and gradients. Activation bitwidth directly determines how many XnorPopCount kernel executions are required.

3. Qunatized Neural network - More than 1-bit: 2-bit activations double the number of XnorPopCount kernel executions relative to 1-bit activations.The execution count is directly proportional to activation bitwidth.
3. Qunatized Neural network - More than 1-bit: Logarithmic quantization maps values to clipped approximate powers of two within specified minimum and maximum scale ranges.The paper applies the quantization schemes to weights, activations, and gradients in ImageNet experiments.
3. Qunatized Neural network - More than 1-bit: The paper evaluates quantization schemes on the ImageNet dataset for weights, activations, and gradients.The experiments include more-than-1-bit quantization schemes in addition to binary representations.

4. Benchmark Results

Across image and language benchmarks, quantized networks achieve near-state-of-the-art or comparable accuracy to full-precision models, while performance depends on precision, architecture, and binarization choices.

BNNs achieve near state-of-the-art results on MNIST, CIFAR-10, and SVHN.
MNIST, SVHN, and CIFAR-10: Figure 1 shows BNNs are slower to train but nearly as accurate as 32-bit float DNNs on CIFAR-10.
ImageNet: 41.8% top-1 and 67.1% top-5 accuracy are obtained with binarized AlexNet, while GoogleNet reaches 47.1% top-1 and 69.1% top-5 accuracy.
ImageNet: 66.5% top-1 accuracy is achieved on GoogleNet with 4-bit weights and activations, a 5.5% drop versus its 32-bit architecture.

5. High Power Efficiency during the Forward Pass

BNNs improve forward-pass efficiency by reducing memory requirements and replacing most arithmetic with bit-wise operations. Binary filters can also be reused, further reducing computation in suitable hardware.

The reported BNN setup leaves the first and last layers at 32-bit precision rather than binarizing them.
Memory accesses typically consume more energy than arithmetic operations, motivating reductions in both memory size and accesses.
BNNs replace most 32-bit floating-point multiply-accumulations with 1-bit XNOR-count operations.A 32-bit floating-point multiplier is estimated at about 200 Xilinx FPGA slices, versus one slice for a 1-bit XNOR gate.
Binary convolution filters repeat because only 2^(k^2) unique 2D filters exist for filter size k.For CIFAR-10, only 42% of filters are unique; dedicated implementations can apply unique filters once and reuse their results.
Quantized gradients allow the backward pass to use XNOR operations, producing fully fixed-point layers with low bitwidth.QNN complexity scales linearly with the number of bits per weight and activation because the XNOR kernel is applied multiple times.

6. Seven Times Faster on GPU at Run-Time

The GPU implementation packs binary variables into registers with SWAR, accelerating bit-wise matrix multiplication. The resulting binary kernel runs the MNIST MLP substantially faster without changing accuracy.

SWAR concatenates 32 binary variables into 32-bit registers, enabling 32-way parallelism for bit-wise operations.
5.3× theoretical GPU speed-up follows from 32 connections evaluated in 6 clock cycles.The six cycles comprise 1 cycle for accumulation, 4 for popcount, and 1 for XNOR.
The comparison used an unoptimized matrix-multiplication baseline and an XNOR kernel implementing SWAR.
7 times faster MNIST MLP execution was achieved with the binary matrix-multiplication kernel than with an unoptimized GPU kernel, without accuracy loss.

7. Discussion and Related Work

The paper extends prior binary-network methods by using binary weights and activations throughout inference and deep-network training. It also identifies remaining training constraints and compares subsequent extensions and alternatives.

Unlike EBP, which uses binarized parameters only during inference, the proposed approach uses them during training as well.
BinaryConnect retained real-valued weights as references while binarizing weights, but its neurons remained full precision.
Earlier compression and matrix-factorization methods trained with full-precision weights and neurons, requiring MAC operations that QNN training avoids.
The work claims the first deep network to binarize both weights and neurons during inference and the entire training phase.The method extends earlier weight binarization to activations, which are especially important in convolutional networks.
The method still requires storing full-precision weights, creating an energy-consuming computational bottleneck during training.
Xnor-Net added scaling factors and left the first and last layers unbinarized, making it only partially binarized.

Conclusion

The paper concludes that BNNs reduce memory use and arithmetic complexity while preserving strong accuracy across several datasets. GPU experiments show faster execution, whereas ImageNet remains an open challenge for further improvement.

BNNs reduce memory size and accesses and replace most arithmetic operations with bit-wise operations during both run-time and train-time forward passes.
7 times faster MNIST MLP execution was obtained with a binary matrix-multiplication GPU kernel without any loss of accuracy.
Nearly state-of-the-art accuracy was achieved on MNIST, CIFAR-10, and SVHN with BNNs.
Quantizing weights and activations beyond 1-bit produced results comparable to 32-bit floating-point architectures.
ImageNet results were not on par with the best full-precision networks, leaving further ImageNet improvement as an open research avenue.

Appendix A. Implementation Details

This section provides implementation details for experiments on MNIST, SVHN, CIFAR-10, and ImageNet.

Implementation details are given for experiments on MNIST, SVHN, CIFAR-10, and ImageNet.
The implementation details cover four datasets spanning digit and general image classification tasks.
The section focuses on the experimental implementations rather than introducing another dataset.

A.1 MLP on MNIST (Theano)

The MNIST Theano experiment uses a large binary-unit MLP without convolution, augmentation, preprocessing, or unsupervised learning, with an L2-SVM output layer.

MNIST contains 60K training and 10K test 28 × 28 grayscale digit images.
The experiment omits convolution, data augmentation, preprocessing, and unsupervised learning to keep the benchmark challenging.
The MLP has 3 hidden layers of 4096 binary units and an L2-SVM output layer.

A.2 MLP on MNIST (Torch7)

The MNIST Torch7 experiment uses a smaller binary-unit architecture with shift-based optimization and batch normalization to reduce multiplications.

The architecture uses 2048 binary units per layer instead of 4096 and omits dropout.
Shift-based AdaMax and batch normalization replace vanilla implementations to reduce the number of multiplications.
The learning rate decays through a 1-bit right shift every 10 epochs.

A.3 ConvNet on CIFAR-10 (Theano)

The appendix describes low-precision convolutional implementations across CIFAR-10, SVHN, and ImageNet, including dataset-specific architectures, optimization settings, and architecture tables.

CIFAR-10 ConvNet: CIFAR-10 uses 50K training and 10K test 32 × 32 color images across 10 classes, without data augmentation or convolutional changes beyond activation binarization.
CIFAR-10 ConvNet: The CIFAR-10 ConvNet architecture is documented in Table 7, which specifies the use of “same” convolutions.
SVHN: SVHN contains 604K training and 26K test color digit images; its experiments halve convolution-layer units and run for 200 epochs instead of 500.
ImageNet: ImageNet uses 1.2M training and 50K test samples labeled across 1000 categories.
ImageNet: AlexNet has 5 convolutional and 3 fully connected layers, while GoogleNet uses 2 convolutional and 10 inception layers plus auxiliary classifiers.
ImageNet: Table 8 documents the AlexNet ConvNet architecture.