Source-linked AI summary
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi
TL;DR
CNNs deliver strong recognition but require too much memory and computation for many small devices. This paper binarizes weights and inputs to approximate convolutions with binary operations, achieving efficient ImageNet models with competitive accuracy and substantially lower resource requirements.
Problem
CNN-based recognition systems require substantial memory and computation, limiting their suitability for phones and embedded electronics.
Method
The paper approximates CNNs by binarizing weights, and in XNOR-Nets also binarizing inputs so convolutions use XNOR and bitcounting operations.
Results
16.3% top-1 accuracy improvement over a prior binarization method is reported on ImageNet, while binary-weight AlexNet matches full-precision AlexNet accuracy.
Takeaways & Limitations
Binary approximations reduce network size by ∼32× and can speed convolution by ∼58×, enabling the possibility of real-time CPU inference on portable devices.
Takeaways & Limitations
The convolutional filters are assumed to have no bias terms.
Abstract
from arXiv · showhide
We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks. In Binary-Weight-Networks, the filters are approximated with binary values resulting in 32x memory saving. In XNOR-Networks, both the filters and the input to convolutional layers are binary. XNOR-Networks approximate convolutions using primarily binary operations. This results in 58x faster convolutional operations and 32x memory savings. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time. Our binary networks are simple, accurate, efficient, and work on challenging visual tasks. We evaluate our approach on the ImageNet classification task. The classification accuracy with a Binary-Weight-Network version of AlexNet is only 2.9% less than the full-precision AlexNet (in top-1 measure). We compare our method with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than 16% in top-1 accuracy.
1 Introduction
The introduction motivates binary CNNs as a way to bring accurate recognition to resource-limited portable devices, whose storage, battery, and computation constrain full-precision models. It presents Binary-Weight-Networks and XNOR-Networks as efficient approximations that retain accuracy while reducing memory and computation, and positions the work as an ImageNet-scale evaluation of binary neural networks.
- Motivation: 61M parameters and 1.5B high-precision operations make AlexNet costly for smaller devices, while deeper CNNs impose even greater resource demands.AlexNet is reported to require 249MB of memory, and such models can overtax phones’ limited storage, battery power, and compute capabilities.
- Binary approximations: Binary-Weight-Networks approximate all weights with binary values, reducing network size by ∼32× and enabling convolutions using addition and subtraction rather than multiplication.The introduction also reports an approximate ∼2× speed up for binary-weight convolutions.
- Binary approximations: XNOR-Networks binarize both weights and convolutional-layer inputs, allowing convolutions to use XNOR and bitcounting operations.The passage treats fully connected layers as convolutional layers because they can be implemented by convolution.
- XNOR-Networks: ∼58× speed up in CPUs is reported for XNOR-Nets, which provide accurate CNN approximations while improving computational efficiency.Binary weights and binary inputs enable the stated XNOR-based implementation.
- Contributions and evaluation: 16.3% on top-1 image classification in the ImageNet challenge ILSVRC2012 is the reported margin over the state-of-the-art network binarization method of.The introduction identifies this work as the first attempt, to the authors’ knowledge, to evaluate binary neural networks on large-scale datasets such as ImageNet.
2 Related Work
Prior work improves neural-network efficiency through shallower models, pruning, compact layers, parameter quantization, and network binarization. The paper positions its approach most closely against methods that binarize weights and activations.
- Shallow networks: Shallow networks reduce model size, but prior work reports difficulty training shallow networks and limited competitiveness with deep models in vision and speech.The passage also notes theoretical work showing sufficiently large single-hidden-layer sigmoid networks can approximate any decision boundary.
- Compressing pre-trained deep networks: Pruning previously trained networks reduces inference-time size by removing redundant or non-informative weights and connections.Earlier methods include weight decay and Hessian-based approaches such as Optimal Brain Damage and Optimal Brain Surgeon.
- Designing compact layers: Compact-layer designs save memory and computation through global average pooling, bottleneck structures, and convolution decomposition.The passage associates these designs with Network in Network, GoogLeNet, and Residual-Net architectures.
- Quantizing parameters: Parameter-quantization methods reduce numerical precision using vector quantization, thresholding, sparse ternary weights, and fixed-point implementations.The cited work reports that thresholding weights at zero decreases top-1 accuracy on ILSVRC2012 by less than %10.
- Network binarization: Network-binarization studies quantize weights and activations, while differing in binarization methods, training procedures, activation precision, and network structure.The paper identifies these works as most closely related and compares its method with BinaryNet on ImageNet.
3 Binary Convolutional Neural Network
This section formalizes binary convolutional networks by approximating real-valued filters with scaled binary filters and extending binarization to inputs for XNOR-based convolutions. It also specifies training and inference procedures that use binarized weights while retaining real-valued parameters for updates.
- Binary Convolutional Neural Network: The CNN is represented by input tensors, weight filters, and convolutional operations across L layers.Each layer has K_l filters, with inputs and filters represented as tensors.
- Binary-Weight-Networks: 32× memory savings result when real-valued filters are approximated by scaled binary filters, enabling convolutions with additions and subtractions.The approximation uses W ≈ αB, where B contains ±1 values and α is positive.
- Binary-Weight-Networks: B* = sign(W), while the optimal scaling factor is the average of the absolute weight values.The sign assignment sets B_i to +1 for nonnegative weights and −1 for negative weights.
- Training Binary Networks: Training binarizes convolutional weights during forward and backward propagation but updates the underlying high-precision real-valued weights.After training, inference requires only forward propagation with binarized weights.
- XNOR-Networks: XNOR-Networks binarize both weights and inputs so convolutions can use XNOR and bitcounting operations with few non-binary operations.Input scaling factors are computed across sub-tensors, while binary activations provide sign(I) for the binary convolution.
4 Experiments
The experiments evaluate binary networks on efficiency and accuracy, including ImageNet classification across multiple CNN architectures. They report substantial memory and computation benefits while comparing BWN and XNOR-Net with prior binarization methods.
- Experimental scope: The experiments measure binary-convolution efficiency through computational speedup relative to standard convolution and evaluate accuracy on ImageNet.ImageNet is described as the first large-scale dataset used to evaluate binary neural networks in this work.
- Experimental scope: BWN and XNOR-Net are evaluated with AlexNet and deeper CNN architectures, alongside comparisons with BinaryConnect and BinaryNet.The binarization technique is presented as applicable to any CNN architecture.
- Efficiency: Binary convolutions use cNWNI binary operations and NI non-binary operations, but BWN provides no speedup on CPUs that fuse multiplication and addition into one cycle.This limitation motivates retaining full precision in the first and last CNN layers.
- Efficiency: Binary-weight networks can fit easily into portable devices because binary weights substantially reduce memory requirements across AlexNet, VGG-19, and ResNet-18.The experiments compare binary and double-precision weights across the three architectures.
- Accuracy: 9.88% and 10.17% are the CIFAR-10 error rates for BWN and XNOR-Net, respectively, using the same network architecture as BinaryConnect and BinaryNet.The ImageNet experiments extend evaluation to a larger and more challenging dataset.
- Accuracy: ∼17% is the reported margin by which BWN and XNOR-Net outperform BinaryConnect and BinaryNet across ImageNet training and inference epochs.The comparison is presented for top-1 and top-5 accuracy, while final accuracies are reported separately in Table 1.
5 We used the Darknet [44] implementation: http://pjreddie.com/darknet/imagenet/#extraction
The section evaluates two training-design choices for binary networks: computing optimal scaling factors and ordering layers within CNN blocks to reduce quantization loss. It also considers treating the scaling factor α as a learnable network parameter instead of computing it with equation 6.
- Block structure: Layer ordering within a CNN block is evaluated as crucial for training XNOR-Networks.The proposed ordering decreases quantization loss for training XNOR-Net.
- Scaling factors: Optimal scaling factors are evaluated as a key element in training binary-weight networks.The factors are found at each training iteration; alternatively, α can be treated as a network parameter.
- Scaling factors: Treating α as a network parameter is presented as an alternative to computing it with equation 6.A layer after binary convolution multiplies the convolution output by α.
5 Conclusion
The paper introduces simple, efficient, and accurate binary approximations for neural networks. Binary weights reduce network size, while XNOR-Net uses mostly bitwise operations to accelerate convolutional inference.
- The proposed binary approximations are designed to be simple, efficient, and accurate.
- ∼32× network-size reduction enables very deep neural networks to fit on portable devices with limited memory.The reduction comes from learning binary values for weights.
- ∼58× speedup is provided by XNOR-Net, which approximates convolutions using mostly bitwise operations.This enables the possibility of running inference with the proposed architecture.