Source-linked AI summary
A comprehensive review of Binary Neural Network
Chunyu Yuan, Sos S. Agaian
TL;DR
Resource-constrained devices need deep-learning models with lower storage and computation demands, motivating Binary Neural Networks despite binarization-related performance loss. This paper exclusively reviews 1-bit activation-and-weight networks across their design pipeline, deployment, hardware, and applications. It concludes that BNNs reduce storage, network complexity, and energy consumption while incurring a significant performance drop.
Problem
Deep neural networks are computationally expensive and memory-intensive, limiting deployment on tiny resource-constrained devices.
Method
The paper comprehensively reviews pure BNNs with 1-bit activations and weights across algorithms, optimization, deployment frameworks, hardware, and applications.
Results
The review finds that BNN techniques can significantly reduce storage, network complexity, and energy consumption, but binarization unavoidably causes a significant performance drop.
Takeaways & Limitations
BNNs offer an efficiency-oriented approach for embedding neural networks on resource-limited devices, with deployment and optimization remaining active research areas.
Takeaways & Limitations
BNN training still generally requires conventional GPUs and substantial computation time, while existing frameworks have limited library and hardware support.
Abstract
from arXiv · showhide
Deep learning (DL) has recently changed the development of intelligent systems and is widely adopted in many real-life applications. Despite their various benefits and potentials, there is a high demand for DL processing in different computationally limited and energy-constrained devices. It is natural to study game-changing technologies such as Binary Neural Networks (BNN) to increase deep learning capabilities. Recently remarkable progress has been made in BNN since they can be implemented and embedded on tiny restricted devices and save a significant amount of storage, computation cost, and energy consumption. However, nearly all BNN acts trade with extra memory, computation cost, and higher performance. This article provides a complete overview of recent developments in BNN. This article focuses exclusively on 1-bit activations and weights 1-bit convolution networks, contrary to previous surveys in which low-bit works are mixed in. It conducted a complete investigation of BNN's development -from their predecessors to the latest BNN algorithms/techniques, presenting a broad design pipeline and discussing each module's variants. Along the way, it examines BNN (a) purpose: their early successes and challenges; (b) BNN optimization: selected representative works that contain essential optimization techniques; (c) deployment: open-source frameworks for BNN modeling and development; (d) terminal: efficient computing architectures and devices for BNN and (e) applications: diverse applications with BNN. Moreover, this paper discusses potential directions and future research opportunities in each section.
1. Introduction
Deep learning models are difficult to deploy on resource-constrained devices because they require substantial computation and memory. This motivates BNNs and a focused survey of their development, deployment, hardware, and applications.
- 1.1. Background: Resource-constrained platforms need compact, lightweight, and efficient networks that retain acceptable accuracy.Large models cannot be readily equipped on smartphones and tiny IoT devices because of their computation and storage demands.
- 1.1. Background: Model compression reduces CNN memory and computation costs through quantization, pruning, low-rank factorization, compact filters, and knowledge distillation.
- 1.2. Motivation: BNNs represent activations and weights with 1-bit values, theoretically reducing memory storage by 32 times and increasing inference speed by 58 times over 32-bit CNNs.They are an extreme form of parameter quantization and can be embedded in resource-limited devices.
- 1.3. Related Work: Earlier BNN surveys omitted representative techniques, misreported some benchmark results, and mixed low-bit networks with pure 1-bit activation-and-weight networks.This survey instead covers the BNN design pipeline, including algorithms, deployment, and applications.
- 1.4. Organization: The paper organizes its review around BNN principles and challenges, optimization, open-source modeling frameworks, efficient hardware, applications, and conclusions.
2. Binary Neural Network
BNNs use 1-bit activations and weights in hidden layers to reduce storage and replace expensive convolution arithmetic with bitwise operations, but naive binarization causes substantial accuracy loss. Their forward path adds binarization before convolution, while backward propagation uses specialized optimization such as the straight-through estimator to learn binary weights.
- What is BNN?: BNNs use 1-bit activations and weights in hidden layers, forming an extremely compact version of CNNs.The input and output layers are exceptions to the 1-bit constraint.
- What is BNN?: Binarization reduces model storage and replaces matrix multiplication with XNOR and popcount operations.These operations target the floating-point multiplication and addition that dominate convolutional computation.
- What is BNN?: 32 times lower memory saving and 58 times faster convolution operations were reported for BNNs compared with 32-bit CNNs.These figures are reported for the BNN approach described by Rastegari et al. (2016).
- Forward Propagation: BNN forward-path neural cells binarize activations and weights before convolution, unlike 32-bit CNN cells.The binarization step represents floating-point values with 1-bit values.
- Backward Propagation: Because the sign function has zero derivative, binary weights cannot use traditional gradient descent and require the straight-through estimator during backward propagation.The cited Binarized-Neural-Networks method applies STE to learn binary weights.
- Summary: Naive BNNs provide faster inference and smaller weight sizes but initially achieve much lower accuracy than full-precision CNNs because binarization discards information.The review groups later methods as optimization solutions for this accuracy problem.
3. Binary Neural Network Optimization
BNN optimization is organized into five complementary categories, spanning quantization, loss functions, gradients, topology, and training. The review also identifies unresolved questions about layer-wise information loss, operations, and training speed.
- Optimization taxonomy: BNN enhancement methods are grouped into quantization error minimization, loss improvement, gradient approximation, topology, and training strategies.The review notes that relying on only one enhancement method is difficult and that some problems remain unsolved.
- Quantization error minimization: Scaling factors α and β approximate real-valued activations and weights during binary convolution to reduce information loss.XNOR-Net uses channel-wise scaling factors, yielding Z ≈ (α ∗ β) ∗ (BI ⊛ BW).
- Quantization error minimization: Later methods vary binary representations and scaling: Hadamard transforms, merged factors, adaptive amplitudes, and alternative activation or weight ranges.Examples include HadaNet, XNOR-Net++, DA-BNN, SI-BNN, ReCU, SiMaN, and AdaBin.
- Loss function improvement: Distribution-aware objectives add a regularization term to cross-entropy, with total loss LT = LS + λ LDR.Here, λ balances the added distribution loss or regularization against the cross-entropy loss.
- Gradient approximation: Because the sign derivative is zero, BNN training uses gradient approximations such as STE, while other methods modify gradient functions or activation distributions.The review notes that STE struggles to learn weights near −1 and +1.
- Network topology and training: Architectural and training changes include modified ResNets, ensembles, stochastic quantization, activation substitutions, optimizer choices, learning-rate settings, and pruning.These methods target accuracy through network structure and training procedure rather than only the binarization operation.
- Open problems: Open optimization questions include measuring each layer’s information loss, jointly reducing operations and preserving accuracy, and substantially accelerating BNN training.The review states that BNN training still requires conventional devices and remains costly and time-consuming.
4. Open Source Frameworks of Binary Neural Network
BNN frameworks support model training, conversion, deployment, and binary computation across CPUs, GPUs, ARM devices, and FPGAs. However, framework maintenance and cross-platform interoperability remain limited.
- Framework motivation: Most BNN implementations fail to store weights natively as binary data or exploit XNOR and popcount in standard Python frameworks.This prevents implementations from fully realizing BNN memory and computational benefits.
- Frameworks: BMXNet supports quantized and binarized inputs and weights through custom MXNet operators, with sample code and pretrained binary models.Its operators include QActivation, QConvolution, and QFullyConnected.
- Frameworks: daBNN targets ARM-based inference with bit packing, binary convolution, optimized memory layout, and C++ or ARM assembly implementations.It also provides Java and Android packages and converts PyTorch floating-point models to BNN models.
- Frameworks: FINN supports low-bit and 1-bit BNN development, training, conversion, compilation, and deployment on Xilinx FPGA boards.Its components are Brevitas, the FINN compiler, and PYNQ integration.
- Framework limitations: Only FINN and Larq remain actively maintained; Larq is TensorFlow-only, while FINN exclusively supports Xilinx FPGA boards.The review identifies cross-platform import and maintenance as an open framework problem.
5. Limit-resource Hardware Architecture
Resource-constrained hardware enables timely vision inference for robotics, wearables, and autonomous vehicles. FPGA platforms are especially suitable for BNN because they emphasize bitwise logic and require less memory.
- Hardware motivation: Timely vision inference is needed on computationally limited platforms in robotics, wearable devices, and self-driving cars.The section motivates efficient hardware for embedding trained models outside expensive GPUs.
- FPGA architecture: FPGAs are well suited for BNN because their dominant computations are bitwise logic operations and their memory requirements are reduced.The section introduces FPGA logic cells as an example of the target architecture.
- Platform types: The hardware vocabulary includes FPGA, ARM processor, SBC, SoC, and SoC FPGA, covering programmable logic, processors, single-board computers, and integrated devices.The section defines these terms before discussing representative platforms.
- Efficiency metrics: Efficiency evaluation on resource-constrained platforms considers inference memory, FPS or latency, and FPGA resources such as DSP, LUT, BRAM, and power.These indicators are used to assess trained-model efficiency on representative platforms.
6. Binary Neural Network Applications
The review evaluates BNN applications through image-classification benchmarks on CIFAR-10 and ImageNet, reporting accuracy alongside computational and hardware-efficiency indicators. Results span diverse architectures, bit-width configurations, and optimization methods.
- Benchmark datasets: Image classification is the basic benchmark for BNN performance, using CIFAR-10 and ILSVRC2012 ImageNet datasets.The review records model year, name, topology, accuracy, and for ImageNet also BOPs, FLOPs, and OPs.
- Benchmark datasets: CIFAR-10 contains 60,000 32×32 color images across 10 classes, split into 50,000 training and 10,000 test images.Each class contains 6,000 images.
- Hardware evaluation: The application tables also report hardware-efficiency comparisons on CIFAR-10 and ImageNet using FPGA platforms.Separate tables summarize FPGA efficiency alongside accuracy-performance histories.
- Benchmark datasets: ImageNet results use ILSVRC2012, whose training set contains more than 1.2 million color images across about 1,000 classes.The dataset is organized into training, validation, and test subsets.
- Model comparisons: Reported BNN evaluations compare architectures including AlexNet, ResNet, MobileNet, WRN, and customized networks across accuracy and operation metrics.The tables include binary, multi-bit, pruned, ensemble, and topology-modified variants.
6.2. Point Cloud Classification
BNN point-cloud classification is substantially more challenging than 2D image classification because binarization amplifies information loss during pooling and point-wise feature extraction. BiPointNet introduced the first binarization method for 3D point clouds, with efficiency and accuracy comparisons reported on Arm-based platforms and ModelNet40.
- BNN is more challenging for 3D point clouds because binarization amplifies information loss during pooling and point-wise feature extraction.
- BiPointNet proposed the first binarization method for learning on 3D point clouds.
- The section reports 3D BNN efficiency comparisons using an Arm-based platform.
- The section reports accuracy comparisons for point-cloud classification.
- EMA-avg and EMA-max were proposed in BiPointNet, while OA denotes overall accuracy.
6.3. Object Detection
BNN object detection extends binary models to a task more complex than 2D image classification, with published work evaluated on VOC2007 and MS-COCO benchmarks. The review also corrects a benchmark attribution that treated a 4-bit quantized model as a BNN variant.
- Object detection is more complex and difficult than 2D image classification, but several BNN approaches have been published.
- BinaryDenseNet achieved accuracy comparable to full-precision YOLO in object detection.
- VOC2007 contains 5011 trainval images and 4952 test images across 20 categories.
- MS-COCO2014 contains 80 image categories, with benchmark training using 80000 training images and 35000 sampled validation images.
- The review identifies FQN as a 4-bit quantization network rather than a BNN and separates its results from BNN benchmarks.
- Tables 23–24 report VOC2007 test results, while Tables 25–26 report MS-COCO minival results.
- The benchmark notes define mAP@.5 using IoU = 0.5 and mAP@[.5, .95] across IoU values from 0.5 to 0.95.
6.4. Semantic Segmentation
BNN has been applied to semantic segmentation using Group-Net, which introduces Binary Parallel Atrous Convolution to improve model performance measured by mean intersection-over-union. Evaluation uses PASCAL VOC 2012.
- Group-Net applies BNN to semantic segmentation and proposes Binary Parallel Atrous Convolution to improve mIOU performance.mIOU is measured as averaged pixel intersection-over-union.
- The PASCAL VOC 2012 evaluation dataset contains 20 foreground object classes and one background class.
- Table 27 reports PASCAL VOC 2012 testing results.
6.5. Natural Language Processing
BNN research has expanded into NLP, including text classification and progressively more binarized transformer models. The surveyed evaluations cover diverse GLUE tasks and BERT benchmarks, with metrics spanning accuracy, F1, and correlation measures.
- BNN has been explored for text classification and applied to BERT, including the fully binarized BiBERT model.
- Table 28 summarizes diverse NLP tasks on the GLUE benchmark, while Table 29 reports BNN-BERT benchmark results.
- GLUE evaluations cover paraphrase, sentence similarity, natural language inference, question-answering, sentiment, and linguistic acceptability tasks.
- The reported GLUE metrics include accuracy, F1, Pearson correlation, and Spearman correlation.
- The benchmark notes distinguish embedding, weight, and activation quantization levels and identify teacher-distillation and data-augmentation settings.
6.6. Unsupervised and Semi-supervised Learning
The section describes self-supervised learning as learning from input data without labels before later supervised use, and identifies benchmark settings for evaluating such representations.
- Self-supervised learning trains models to predict information from the input itself before later supervised tasks.
- ImageNet evaluation includes linear evaluation and semi-supervised fine-tuning with 1% or 10% labels.
- Transfer learning evaluates representations on object-centric and scene-centric datasets.Object-centric datasets include CIFAR10, CIFAR100, CUB-200-2011, and Birdsnap; Places205 is scene-centric.
6.7. Other Tasks
BNNs have been applied to pose estimation, face alignment, facial-mask classification, and image retrieval, while architecture design and broader task coverage remain open challenges.
- Other Tasks: BNNs support human pose estimation and face alignment.
- Other Tasks: BinaryCoP classifies correct facial-mask wear and positioning on edge devices.
- Other Tasks: Binary Neural Network Hashing combines BNNs with hashing to accelerate large-scale image retrieval with low storage cost.
- Challenges: Open challenges include selecting or automatically searching BNN architectures while limiting information loss across layers.The review notes that binarization can lose information and that performance degradation accumulates in deeper layers.
- Challenges: Developing transformer-based BNNs for vision is identified as a research opportunity, alongside the limited literature on 3D tasks.
7. Conclusions
This review surveys predominantly 1-bit BNNs, organizing their optimization, deployment, hardware, and applications while identifying research gaps and future directions. It emphasizes efficiency for resource-limited devices but acknowledges performance loss from binarization.
- The review focuses mainly on networks with 1-bit activations and weights that reduce memory usage and computational cost.
- It classifies BNN techniques across quantization, distributions, loss functions, gradient approximation, topology, and training strategies.
- The paper also reviews open-source modeling frameworks, efficient hardware platforms, and application progress.
- The review identifies research gaps and recommends future research directions across the BNN literature.
- Binarization unavoidably causes a significant performance drop despite BNNs’ efficiency benefits.