PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Accurate Binary Convolutional Neural Network

Xiaofan Lin, Cong Zhao, Wei Pan

arXiv:1711.11294v1cs.LGstat.ML

TL;DR

ABC-Net addresses the accuracy degradation of binary CNNs by approximating full-precision weights with multiple binary bases and representing activations with multiple binary activations. It achieves accuracy close to full-precision networks on ImageNet while retaining hardware-friendly binary operations.

  • Problem

    Previous binary CNNs reduce inference cost but typically suffer severe prediction accuracy degradation.

  • Method

    ABC-Net approximates full-precision weights with multiple binary weight bases and real-valued activations with a linear combination of multiple binary activations.

  • Results

    ABC-Net achieves accuracy close to its full-precision counterpart on ImageNet, with five binary activations reducing Top-1 and Top-5 degradation to around 5%.

  • Takeaways & Limitations

    The scheme is parallelizable and hardware friendly, replacing most convolution multiplications with bitwise operations for potentially faster test-time inference.

  • Takeaways & Limitations

    The more elaborate channel-wise weight approximation requires more training computation and is left for future work.

Abstract

from arXiv · show

We introduce a novel scheme to train binary convolutional neural networks (CNNs) -- CNNs with weights and activations constrained to {-1,+1} at run-time. It has been known that using binary weights and activations drastically reduce memory size and accesses, and can replace arithmetic operations with more efficient bitwise operations, leading to much faster test-time inference and lower power consumption. However, previous works on binarizing CNNs usually result in severe prediction accuracy degradation. In this paper, we address this issue with two major innovations: (1) approximating full-precision weights with the linear combination of multiple binary weight bases; (2) employing multiple binary activations to alleviate information loss. The implementation of the resulting binary CNN, denoted as ABC-Net, is shown to achieve much closer performance to its full-precision counterpart, and even reach the comparable prediction accuracy on ImageNet and forest trail datasets, given adequate binary weight bases and activations.

1 Introduction

CNN deployment on mobile devices motivates reducing inference cost, but binarization causes substantial accuracy loss. ABC-Net addresses this gap with multiple binary weight bases and activations, approaching full-precision accuracy.

  • Motivation: Mobile CNN inference requires real-time, low-power, embeddable systems, making test-time performance and hardware cost important constraints.Training is typically performed on servers, whereas inference runs on mobile devices.
  • Accuracy challenge: Binarizing Resnet-18 weights reduces ImageNet Top-1 accuracy from 69.3% to 60.8%, and binarizing activations further reduces it to 51.2%.These results illustrate the gap between binary and full-precision models.
  • Proposed scheme: ABC-Net approximates full-precision weights with a linear combination of multiple binary weight bases.The authors report that 3∼5 binary weight bases adequately approximate full-precision weights, while enabling addition, subtraction, or bitwise operations.
  • Proposed scheme: ABC-Net introduces multiple binary activations to reduce information loss from activation binarization.With five binary activations, Top-1 and Top-5 accuracy degradation on ImageNet is reduced to around 5% relative to full precision.
  • Results: Using more binary weight bases and activations brings binary-network accuracy close to full-precision accuracy on ImageNet.The authors state this is the first binary neural network to achieve comparable prediction accuracy on ImageNet.

2 Related work

Prior quantized and binarized neural networks reduce memory and computation but generally sacrifice accuracy. ABC-Net combines multiple binary operations and activations to narrow the accuracy gap.

  • Quantized Neural Networks: Quantized neural networks reduce memory requirements and computation complexity through low-bitwidth weights and activations.Related work also includes training CNNs with low-bitwidth gradients.
  • Binarized Neural Networks: Binarized neural networks use binary weights and activations at run time, with binary activations supporting energy-efficient event-based computation.The cited BNN work established binary representations for deep models.
  • Gap addressed: Earlier binarization methods significantly reduce accuracy, whereas ABC-Net uses multiple binary operations and activations to allow more information to pass through.The paper reports that its ImageNet results come close to closing the gap with full precision.

3 Binarization methods

ABC-Net approximates full-precision convolutions with multiple binary weight bases and activations, preserving more information while enabling efficient binary operations at inference.

  • 3.1 Binarizing weights: ABC-Net estimates each real-valued convolutional filter with a linear combination of M binary filters.The binary filters are constrained to {-1,+1}; their coefficients are obtained through a linear regression formulation.
  • 3.1 Binarizing weights: The binary weight bases are formed by thresholding shifted, normalized weights, with shifts either evenly spaced or trainable.The construction uses the weight mean and standard deviation and assumes a roughly symmetric, non-sparse weight distribution.
  • 3.1 Binarizing weights: During training, ABC-Net computes binary bases and coefficients while backpropagating through the bases with the straight-through estimator; at test-time, only the binary convolution expression is required.Binary convolutions can use additions and subtractions, or XNOR and bitcount when inputs are binary, and the convolutions for different filters can run in parallel.
  • 3.1.2 Approximate weights channel-wise: The channel-wise weight approximation is more elaborate without extra inference cost, but requires more training computation and is left for future work.The paper therefore focuses on approximating the weights as a whole.
  • 3.2 Multiple binary activations and bitwise convolution: ABC-Net uses multiple binary activations, generated after batch normalization and bounded activation, to approximate real-valued activations with learned coefficients.The resulting activations pass more information than a single binary activation and combine with the weight approximation into M × N binary convolutions that can run in parallel.
  • 3.2 Multiple binary activations and bitwise convolution: Activation binarization is difficult because crude binary activations lose accuracy and real-valued activations vary during inference; ABC-Net addresses this using their statistical structure.Batch normalization stabilizes activation distributions, while multiple binary activations and coefficients approximate the real-valued activation.
  • 3.3 Training algorithm: Increasing the number of binary weight bases can make them more correlated and sometimes cause rank deficiency in the coefficient regression.The paper notes that ℓ2 regularization can address this issue.

4 Experiment results

ABC-Net experiments on ImageNet evaluate weight-basis and activation configurations, showing that increasing these quantities narrows the accuracy gap to full-precision models. The method achieves state-of-the-art binary-model performance while exposing practical accuracy–resource trade-offs.

  • Experimental setup: ABC-Net’s ImageNet evaluation uses ResNet topologies and reports Top-1 and Top-5 classification accuracy.The dataset has about 1.2 million training images across 1000 categories, with 50k validation images resized to 224x224.
  • Effect of weight approximation: A 0.9 percentage-point Top-1 gap remains when M = 5 with full-precision activations, nearly eliminating weight-binarization degradation.Table 1 evaluates different numbers of binary weight bases while keeping activations full-precision.
  • Configuration space exploration: No single M,N configuration is universally preferred because training time, inference time, model size, and accuracy require practical trade-offs.The paper explicitly leaves configuration choice open rather than identifying a definitive combination.
  • Configuration space exploration: Accuracy improves as the numbers of binary weight bases and activations increase; with M = N = 5, the Top-1 gap is around 5%.The configuration study explores combinations of M and N across ResNet-18, ResNet-34, and ResNet-50 settings.
  • Comparison with the state-of-the-art: ABC-Net achieves state-of-the-art performance as a binary model and significantly outperforms prior models using both binary weights and activations.The comparison uses ResNet-18 and includes full-precision ResNet-18, BWN, XNOR-Net, DoReFa-Net, and BNN.
  • Comparison with the state-of-the-art: A five-bit quantization scheme may reach similar accuracy, but ABC-Net’s alternative is described as more efficient and requiring distinctly fewer hardware resources.The comparison concerns five weight bases and five binary activations.

5 Discussion

ABC-Net preserves information through multiple binary weight bases and activations while retaining hardware advantages over fixed-point quantization. Its operations support bitwise computation, lower multiplier resource demands, energy-efficient spiking responses, and runtime integration of shifts and batch normalization.

  • Information preservation: Multiple binary weight bases and activations allow more information to pass through and can approximate full-precision tensors more precisely.The coefficients and shift parameters learn statistical features of full-precision tensors, analogous to batch-normalization scale and shift parameters.
  • Advantage over fixed-point quantization scheme: K binarizations preserve bitwise operations, which are more efficient than fixed-point multiplication.This preserves the computational motivation for binary neural networks while using multiple bases or activations.
  • Advantage over fixed-point quantization scheme: K 1-bit multipliers avoid the overflow, underflow, and precision issues of K-bit multipliers while enabling saved hardware resources to support parallel computing.The paper states that multiplier complexity is proportional to the square of bit-widths, whereas binary outputs remain within {-1,1}.
  • Advantage over fixed-point quantization scheme: Binary activations support spiking event-based computation and communication, consuming energy only when necessary.The paper identifies this as an energy-efficiency advantage unavailable to the fixed K-bit width scheme.
  • Further computation reduction in run-time: Shift parameters can be implemented as comparator thresholds, avoiding extra additions and subtractions during specialized-hardware execution.The threshold is determined by the shift parameter.
  • Further computation reduction in run-time: Batch normalization can be integrated into binarization by transforming its affine parameters into a threshold, producing no extra runtime cost.The paper expresses batch normalization as BN(R) = aR + b and folds its scale and shift into the binarization threshold.

6 Conclusion and future work

The paper concludes that ABC-Net trains binary CNNs with accuracy close to full-precision models while remaining parallelizable and hardware friendly. It identifies specialized hardware implementation and extension to other tasks and model families as future directions.

  • 6 Conclusion and future work: ABC-Net introduces binarization of weights and activations during forward and backward propagation for training binary CNNs.The method is explicitly named ABC-Net.
  • 6 Conclusion and future work: ABC-Net achieves accuracy close to its full-precision counterpart on ImageNet.The conclusion presents this as evidence that the proposed binary CNN can be trained successfully on ImageNet.
  • 6 Conclusion and future work: The scheme is parallelizable and hardware friendly, replacing most convolution multiplications with bitwise operations.The paper connects this design to potentially major effects on specialized CNN hardware and faster test-time inference for real-time embedded systems.
  • 6 Conclusion and future work: Future work includes extending ABC-Net to object detection and RNNs and implementing it on FPGA, ASIC, or customized deep-learning processors.These are proposed directions rather than reported results.

Supplementary Material

Algorithm 1 specifies training for an L-layer ABC-Net by computing forward and backward propagation with multiple binary bases and activations, then updating full-precision parameters and the learning rate.

  • Training inputs and outputs: Training requires a minibatch, targets, M binary weight bases, N binary activations, previous weights, and learning rate η.The algorithm returns updated weights and an updated learning rate.
  • Forward propagation: Forward propagation computes convolution using the binary weight bases, optionally applies max-pooling, batch-normalizes the result, and binarizes activations when l < L.The algorithm loops over layers l = 1 to L and uses N binary activations for nonfinal layers.
  • Backward propagation: Backward propagation computes gradients from the cost while retaining full-precision gradients.The procedure backpropagates through activation, batch-normalization, and convolution operations.
  • Parameter updates: The algorithm accumulates gradients and updates weights, binary-basis coefficients, shift parameters, and learning rate η.The learning rate is decayed by η ← λη after parameter updates.

S2 Weight approximation

The supplementary experiment evaluates how accurately multiple binary bases fit a sampled weight tensor from an ImageNet-pretrained full-precision ResNet-18. The fit improves as the number of bases increases, with M = 3 giving a rough fit and M = 5 nearly perfect.

  • S2 Weight approximation: A randomly sampled slice of an ImageNet-pretrained full-precision ResNet-18 weight tensor is vectorized and approximated with M binary bases using linear regression.The experiment evaluates the resulting estimated weights against the original full-precision weights.
  • S2 Weight approximation: The left subfigure reports RMSE as the number of bases increases, while the right shows five fits for M = 1 through M = 5.Blue denotes full-precision weights and red denotes estimated weights.

S3 Feature map

ABC-Net can support visual perception beyond classification, and its feature maps resemble those of a full-precision counterpart.

  • S3 Feature map: ABC-Net is presented as applicable to CNN-based tasks beyond classification, including faster RCNN object detection.The classification model can serve as a pretrained model for such tasks.
  • S3 Feature map: Feature maps from ABC-Net and its full-precision counterpart are reported to be similar.The comparison uses examples from the first convolution layer.

S5 Relationship between accuracy and number of binary weight bases M

On ImageNet, ABC-Net accuracy appears to increase linearly with the number of binary weight bases M when activations remain full-precision.

  • S5 Relationship between accuracy and number of binary weight bases M: Accuracy appears to have a linear relationship with the number of binary weight bases M.This relationship is reported for the ImageNet experiment shown in Figure S4.
  • S5 Relationship between accuracy and number of binary weight bases M: The experiment holds activations at full precision while varying the number of binary weight bases.This condition limits the comparison to the effect of changing M under full-precision activation.
  • S5 Relationship between accuracy and number of binary weight bases M: The parameter notation defines M as the number of weight bases and N as the number of activations.Table S4 lists settings for Resnet-18, Resnet-34, and Resnet-50 topologies.
  • S5 Relationship between accuracy and number of binary weight bases M: Figure S4 plots Top-1 accuracy on the left and Top-5 accuracy on the right for different choices of M.The experiment uses ABC-Net on ImageNet with full-precision activation.

S6.1 VGG-like Network Topology

The forest-trail visual-perception experiment uses a VGG-like CNN topology and evaluates ABC-Net against its full-precision counterpart on a three-class dataset.

  • S6.1 VGG-like Network Topology: A VGG-like network topology is used for visual perception of forest trails.The topology is illustrated in Figure S5.
  • S6.1 VGG-like Network Topology: The forest-trail dataset contains three classes: turn right, go straight, and turn left.It was built from eight hours of 1920 × 1080, 30fps video recorded with three head-mounted cameras.
  • S6.1 VGG-like Network Topology: ABC-Net is evaluated against its full-precision counterpart on the forest-trail dataset.The classification result is reported in Table S5, whose label FP denotes Full Precision.
  • S6.1 VGG-like Network Topology: Figure S5 illustrates the network topology used for visual perception of forest trails.The passage identifies the topology as VGG-like.
Loading 1711.11294v1…