PaperlayerView exact revision on arXiv

Source-linked AI summary

Binary Neural Networks: A Survey

Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, Nicu Sebe

arXiv:2004.03333v1cs.NEcs.CVcs.LG

TL;DR

Binary neural networks reduce storage and computation for resource-limited deployment but suffer information loss and optimization difficulty. The paper surveys native and optimized binarization approaches, finding that optimization techniques can promise better performance while retaining information across propagation.

  • Problem

    Binary neural networks reduce storage and computation but face performance loss, especially on large networks and datasets.

  • Method

    The paper surveys binary-network algorithms, categorizing them into naive and optimized approaches that focus on preserving information.

  • Results

    Optimizing binary networks with different techniques can promise better performance, and retaining information in forward and backward propagation is a key factor in training high-performance BNNs.

  • Takeaways & Limitations

    Preserving various information during forward and backward propagation is one key factor in training high-performance binary neural networks.

  • Takeaways & Limitations

    Binary networks still face performance loss, especially for large networks and datasets, and the main reasons remain unclear.

Abstract

from arXiv · show

The binary neural network, largely saving the storage and computation, serves as a promising technique for deploying deep models on resource-limited devices. However, the binarization inevitably causes severe information loss, and even worse, its discontinuity brings difficulty to the optimization of the deep network. To address these issues, a variety of algorithms have been proposed, and achieved satisfying progress in recent years. In this paper, we present a comprehensive survey of these algorithms, mainly categorized into the native solutions directly conducting binarization, and the optimized ones using techniques like minimizing the quantization error, improving the network loss function, and reducing the gradient error. We also investigate other practical aspects of binary neural networks such as the hardware-friendly design and the training tricks. Then, we give the evaluation and discussions on different tasks, including image classification, object detection and semantic segmentation. Finally, the challenges that may be faced in future research are prospected.

1. Introduction

Deep neural networks offer strong learning capacity but demand substantial computation and storage, limiting deployment on resource-constrained devices. Binary neural networks address this tension through 1-bit representations and hardware-friendly operations, while also enabling analysis of network structure and robustness.

  • Deployment motivation: Deep CNNs require high-performance hardware, whereas mobile, embedded, and FPGA devices provide limited computational resources.FPGAs may have only a few thousand computing units, far below the millions of floating-point operations required by common deep models.
  • Compression background: Network compression reduces redundancy through pruning, quantization, factorization, compact filters, and knowledge distillation.These approaches remove uncritical parameters, reduce parameter precision, decompose tensors, redesign convolutional filters, or reproduce a larger network's output with a compact model.
  • Hardware benefits: 32× memory saving and 58× speedup on CPUs were achieved by XNOR-Net for a 1-bit convolution layer.Binary neural networks also provide hardware-friendly properties including power efficiency and significant acceleration.
  • Structure and interpretability: Binarization supports structural analysis by testing layer importance, while shortcuts can preserve shallow details and provide direct gradient paths.Bi-Real shortcuts reduce information loss; visualizations indicate that they pass detailed shallow-layer information forward and help avoid gradient vanishing backward.
  • Robustness and network analysis: Studies report that specially designed BNNs can be more robust than full-precision networks because binarization keeps noise magnitude small.Other analyses indicate that the number of neurons may matter more than bit-width, and real-valued neurons may not always be necessary.
  • Survey scope: The survey categorizes binarization methods as naive or optimized, with optimization targeting quantization error, loss functions, and gradient error.It also discusses hardware-friendly methods and training tricks.

2. Preliminary

Full-precision convolution uses floating-point tensor operations, while binarization represents weights and/or activations with 1-bit values. Because sign is nondifferentiable, STE enables gradient-based training, although clipping can prevent updates when full-precision activations exceed 1.

  • Forward Propagation: Full-precision convolution performs floating-point multiplication and addition on weights and activations.These operations constitute a large portion of neural-network inference calculations.
  • Forward Propagation: Network binarization represents floating-point weights and/or activations using 1-bit tensors with associated scaling factors.Binary weights and activations are formed through quantization functions such as sign.
  • Backward Propagation: The sign function is nondifferentiable or has vanishing derivatives, so ordinary backpropagation cannot directly update binary weights.The sign derivative is described as approximately zero almost everywhere.
  • Backward Propagation: The straight-through estimator allows binary neural networks to use the same gradient-descent training method as full-precision networks.STE approximates the gradient during backward propagation.
  • Backward Propagation: When clipping is used in backward propagation, full-precision activations with absolute values greater than 1 cannot be updated.The identity function is also used to approximate the derivative of the sign function in practical scenarios.

3. Binary Neural Networks

Binary neural networks reduce storage and inference computation through 1-bit representations, but quantization causes deviation from full-precision values and complicates optimization. The survey organizes approaches into native binarization and optimization-based methods addressing quantization, loss, and gradient errors.

  • Overview: 1-bit representations replace floating-point multiplication and addition with efficient XNOR-Bitcount operations.This reduces storage space and inference time.
  • Challenges: Binarizing weights and activations causes severe deviation from their full-precision counterparts.The resulting performance degradation remains an open research problem.
  • Challenges: Discrete binarization makes gradient-descent backpropagation struggle to find satisfactory solutions, even with STE.The survey identifies optimization as a central challenge for binary neural networks.
  • Method Taxonomy: The survey categorizes methods into native solutions that directly binarize weights and inputs and optimization-based solutions.Optimization-based techniques minimize quantization error, modify the network loss, or reduce gradient error.
  • Method Taxonomy: Table 1 summarizes the surveyed binary neural-network binarization methods.

3.1. Naive Binary Neural Networks

Naive binary neural networks directly quantize network weights and activations, using stochastic or fixed binarization during forward propagation and STE-based optimization during training. Early methods demonstrated substantial compression and acceleration while retaining strong classification performance on some datasets.

  • Training Strategy: STE-based backward propagation enables optimization of binary networks with standard gradient-based training.
  • BinaryConnect: BinaryConnect pioneered binary neural-network research by converting full-precision weights into 1-bit binary weights.During training, stochastic binarization simulates binary weights in forward propagation while retaining real-valued weights for updates.
  • Training Strategy: Naive binarization uses clipping during backward propagation to limit full-precision weight updates without changing binary weights.
  • Performance: After binarization, model parameters are greatly compressed, and binary models can approach state-of-the-art image-classification performance on some datasets.
  • BNN: BNN established the practicability and acceleration capability of binary neural networks through theoretical and practical analysis.Its inference techniques include Shift-based Batch Normalization and XNOR-Bitcount.
  • Performance: Experiments on image classification report that BNN uses 32× less storage space and 60% less time.

3.2. Optimization Based Binary Neural Networks

Optimization-based binary neural networks address accuracy loss by reducing quantization, loss-function, and gradient errors rather than relying only on direct binarization. The survey describes scaling, residual, loss-aware, distillation, and differentiable approaches that improve binary-network training and accuracy.

  • Motivation: Native binarization saves computational resources but can suffer accuracy loss across diverse tasks because it ignores forward and backward effects.These limitations motivate optimization-based solutions, which have shown improvement over native methods.
  • 3.2.1. Minimize the Quantization Error: Scaling factors approximate full-precision weights with binary values, reducing quantization error while preserving fast computation.BWN and XNOR-Net formulate the approximation as w ≈ αb_w, yielding binary weights in {−α, +α}.
  • 3.2.1. Minimize the Quantization Error: Residual and multi-branch quantization methods recursively or linearly combine binary approximations to reduce information loss in weights and activations.HORQ uses quantized residuals, while ABC-Net combines multiple binary weight matrices and scaling factors.
  • 3.2.2. Improve the Network Loss Function: Teacher-guided distillation trains low-precision students with full-precision teacher information, helping binary networks preserve learning capability.The survey notes that teacher-related loss functions incorporate full-precision guidance into binary-network training.
  • 3.2.2. Improve the Network Loss Function: Quantization-aware loss methods account for global objectives or explicitly regularize activation distributions to address limitations of local approximation.LAB minimizes overall loss for binary weights, while distribution loss supplements cross-entropy to guide binarization.

3.3. Efficient Computing Architectures for Binary Neural Networks

Binary neural networks use hardware-oriented designs to exploit fast binary operations while balancing accuracy, speed, power consumption, and deployment constraints.

  • Binary neural networks offer fast computation, low power consumption, and a low memory footprint.
  • FPGAs support customized data paths and designs around XNOR-Bitcount operations, while ASICs can optimize hardware layouts for network structures.
  • FPGA accelerators use variable-length buffers, multi-stream architectures, computation reuse, and Bitcount, Threshold, and OR operators.
  • 8-24× more efficient than BMXNet is the ARM-assembly-based daBNN inference library for mobile phone platforms.
  • The Xilinx implementation has the most promising speed with low power consumption, while other methods target accuracy or broader deployment.
  • Binary models can achieve comparable accuracy to full-precision models, but few perform well on ImageNet in both speed and accuracy.

3.4. Applications of Binary Neural Networks

Binary neural networks have been applied across classification, detection, segmentation, and robustness studies, with performance depending strongly on task, dataset, and design.

  • Binary neural networks are applied to image classification, object detection, semantic segmentation, and other common tasks.
  • Binary networks achieved comparable performance to full-precision networks on MNIST and infrared datasets, with at least 4× acceleration and three orders of magnitude energy savings over GPUs.
  • Quantized RetinaNet and faster R-CNN detectors showed very encouraging performance on object detection tasks.
  • Group-Net generalizes across classification and semantic segmentation while outperforming previous binary networks in accuracy and computation savings.
  • For small noise magnitudes, quantization can reduce perturbation distance, but vanilla BNNs enlarge it when noise is large.
  • Defensive Quantization suppresses layerwise noise amplification and makes quantized models more robust than full-precision models.

3.5. Tricks for Training Binary Neural Networks

Training binary neural networks requires specialized structural, optimization, gradient, and quantization techniques because binarization makes training discrete and unstable.

  • The survey organizes training techniques around network transformation, optimizer and hyper-parameter selection, gradient approximation, and asymptotic quantization.
  • Network structure transformation: Reordering pooling and batch-normalization layers, or adding structural transformations, can stabilize training and improve accuracy.
  • Network structure transformation: Skip connections, wider blocks, ensembles, circulant filters, and gated residuals adapt network structure to distribution changes and information loss.
  • Optimizer and hyper-parameter selection: Adaptive optimizers such as Adam generally improve training, whereas fixed-rate SGD may require a large batch size.
  • Asymptotic quantization: Asymptotic quantization gradually increases quantization strength or reduces bit-width to limit perturbations and find better solutions.
  • Gradient approximation: Smoother estimators replace rough straight-through approximations to reduce backward-propagation gradient error.

4. Evaluation and Discussions

Evaluation across classification, detection, segmentation, and deployment studies shows that binary networks can approach full-precision performance on smaller tasks but remain challenged by large datasets and practical trade-offs.

  • The evaluation uses CIFAR-10 and ImageNet for classification, and PASCAL VOC and COCO for detection and segmentation.
  • Binary networks can approach full-precision performance on MNIST and CIFAR-10 but suffer severe performance drops on ImageNet.
  • Binary networks are more sensitive to activation binarization than weight binarization, motivating methods that preserve activation information.
  • ImageNet binarization loses about 7% accuracy for weights alone and an additional 20% when activations are binarized in ABC-Net ResNet-18.
  • Skip connections and wider structures can approach full-precision accuracy while retaining acceleration from XNOR-Bitcount operations.
  • Delicate quantizers may preserve more information but can require complicated computation and multistage training that hinder hardware implementation and reproducibility.
  • Object detection methods achieved encouraging progress and significant acceleration in real-world systems, but accuracy and speed must be balanced.

5. Future Trend and Conclusions

The survey categorizes binary-neural-network solutions as naive or optimized, finding that optimization can improve performance while identifying unresolved challenges in information preservation, discrete optimization, and deployment.

  • Conclusions: Existing binary-neural-network solutions are roughly categorized into naive methods and optimized methods.
  • Conclusions: Optimized binary networks can achieve better performance by preserving information during forward propagation and improving backward-propagation optimization.
  • Conclusions: Retaining varied information in forward and backward propagation is identified as a key factor in training high-performance binary neural networks.
  • Future Challenges: Binary neural networks still face performance loss, particularly on large networks and datasets, because suitable architectures and information-preserving binarization remain unclear.
  • Future Challenges: Optimizing binary networks in a discrete space remains difficult even with gradient estimators or approximate binarization functions.
  • Future Trends: Future work includes task-specific customization, hardware-friendly or energy-economic binarization, binarized backward calculation, mix-precision strategies, and architectures supporting information flow.
Loading 2004.03333v1…