Source-linked AI summary
Forward and Backward Information Retention for Accurate Binary Neural Networks
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, Jingkuan Song
TL;DR
Binary neural networks reduce model storage and accelerate inference, but binarization loses information in both forward activations and backward gradients, leaving an accuracy gap. IR-Net addresses these losses with Libra-PB and EDE, and experiments on CIFAR-10 and ImageNet show consistent gains over existing state-of-the-art binary networks.
Problem
Binarization causes information loss in forward and backward propagation, contributing to a performance gap between binary and full-precision models.
Method
IR-Net combines Libra-PB, which balances parameter distributions while minimizing quantization error, with EDE, which progressively approximates the sign function during backward propagation.
Results
IR-Net consistently outperforms existing state-of-the-art binary neural networks across various network structures on CIFAR-10 and ImageNet.
Takeaways & Limitations
The unified information-flow perspective preserves forward diversity and backward gradient information without additional activation operations or harmed efficiency.
Takeaways & Limitations
Clip-based gradient approximation clamps gradients outside [-1, +1], preventing further updates for parameters that leave the interval and harming early optimization.
Abstract
from arXiv · showhide
Weight and activation binarization is an effective approach to deep neural network compression and can accelerate the inference by leveraging bitwise operations. Although many binarization methods have improved the accuracy of the model by minimizing the quantization error in forward propagation, there remains a noticeable performance gap between the binarized model and the full-precision one. Our empirical study indicates that the quantization brings information loss in both forward and backward propagation, which is the bottleneck of training accurate binary neural networks. To address these issues, we propose an Information Retention Network (IR-Net) to retain the information that consists in the forward activations and backward gradients. IR-Net mainly relies on two technical contributions: (1) Libra Parameter Binarization (Libra-PB): simultaneously minimizing both quantization error and information loss of parameters by balanced and standardized weights in forward propagation; (2) Error Decay Estimator (EDE): minimizing the information loss of gradients by gradually approximating the sign function in backward propagation, jointly considering the updating ability and accurate gradients. We are the first to investigate both forward and backward processes of binary networks from the unified information perspective, which provides new insight into the mechanism of network binarization. Comprehensive experiments with various network structures on CIFAR-10 and ImageNet datasets manifest that the proposed IR-Net can consistently outperform state-of-the-art quantization methods.
1. Introduction
Binary neural networks reduce storage and inference costs through binarized weights and activations, but information loss in forward and backward propagation limits accuracy. IR-Net addresses both processes through Libra-PB and EDE, achieving strong results across datasets and network structures.
- Motivation: Binary neural networks use 1-bit weights and activations with bitwise operations to reduce storage and inference costs.These benefits support deployment on resource-constrained devices.
- Motivation: Binarization degrades performance because limited representation ability and discreteness cause information loss during forward and backward propagation.Forward diversity decreases, while backward gradients can become inaccurate and direct optimization in the wrong direction.
- Approach: IR-Net retains information in both directions with Libra-PB for balanced, standardized forward quantization and EDE for progressively approximated backward gradients.Libra-PB targets entropy and quantization error, while EDE balances early updating ability with later gradient accuracy.
- Results: IR-Net performs well across CIFAR-10 and ImageNet classification using ResNet-20, VGG-Small, ResNet-18, and ResNet-34.The reported experiments show the method surpasses previous quantization methods by a wide margin.
2. Related Work
Network binarization offers substantial memory and acceleration benefits through bitwise operations, while prior methods trade accuracy against representation flexibility and still retain an accuracy gap. The related methods vary in how they reduce quantization error or expand available representations.
- Prior Methods: XNOR-Net minimizes output quantization error with layerwise scalars, while TWN and TTQ increase representation ability using more quantization points.These methods target accuracy through different parameterization choices.
- Prior Methods: ABC-Net improves accuracy with multiple binary bases, but reduces compression and acceleration ratios as representation flexibility increases.HWGQ instead considers activation-function quantization error, and LQ-Net extends this line of work.
- Network Binarization: Network binarization reduces memory consumption and supports efficient inference through bitwise operations on binary parameters.This distinguishes it from pruning and matrix decomposition in hardware compatibility and acceleration.
- Open Problem: Existing quantization methods still show a significant accuracy drop relative to full-precision models because substantial information loss remains.The passage frames this gap as an unresolved limitation of prior binarization progress.
3. Preliminaries
Binary neural networks replace floating-point weights and activations with 1-bit representations, enabling bitwise vector operations. Training requires a surrogate for the sign function because its derivative is zero almost everywhere.
- Neural-Network Operation: The main neural-network operation is vector multiplication between a weight vector and an input activation vector.The weight vector w and activation vector a are both n-dimensional.
- Binarization: Network binarization represents floating-point parameters with binary values in {−1, +1}, optionally scaled by learned or selected scalars.The notation distinguishes floating-point x from binary Bx and scalar α.
- Binarized Inference: With quantized weights and activations, forward vector multiplication can be implemented using XNOR and Bitcount operations.The inner-product notation ⊙ denotes the bitwise implementation.
- Backward Propagation: The sign function cannot provide useful exact backward gradients because its derivative is zero almost everywhere.Binary-model training therefore commonly uses a Straight-Through Estimator based on Identity or Hardtanh.
4. Information Retention Network
IR-Net treats information loss in both forward binarization and backward gradient approximation as the central bottleneck of accurate binary neural-network training. It combines Libra-PB, which balances and standardizes weights to retain forward information, with EDE, which gradually reduces gradient-approximation error while preserving parameter updates.
- Motivation: Binary networks lose information during both forward quantization and backward gradient approximation, so minimizing quantization error alone is insufficient for accurate models.Binary parameters have only two values, limiting representation ability and changing the solution space relative to full-precision networks.
- Libra-PB: Libra-PB jointly considers quantization error and information entropy, rather than optimizing quantization error alone.Its objective addresses cases where low-bit quantization can produce very low entropy despite small quantization error.
- Libra-PB: Libra-PB balances weights to achieve zero mean and standardizes them to unit norm before binarization.The balanced distribution maximizes binary-weight entropy under the Bernoulli assumption, while standardization supports steadier updates and more stable binary weights.
- Libra-PB: Libra-PB also maintains activation information by producing zero-mean layer outputs, which maximizes activation entropy under the stated analysis.The zero-mean output follows from the balanced binary weights in each layer.
- Libra-PB: An integer bit-shift scalar expands binary-weight representation while reducing the floating-point computation required by prior methods.The scalar is selected to further minimize quantization error without introducing expensive floating-point operations.
- Libra-PB: Balanced Libra-PB weights maximize binary-weight information entropy under the Bernoulli distribution, unlike the less balanced sign-function quantization.The cited comparison reports probabilities of 0.5 for Libra-PB and 0.2 for sign(x) taking value 1.
- EDE: Identity approximation creates large gradient error, whereas clip approximation sets gradients to zero outside [-1, +1] and prevents those parameters from updating.These limitations motivate preserving both accurate gradients and sufficient updating ability during training.
- EDE: EDE resolves the trade-off between identity and clip gradient approximations through a two-stage estimator.It updates all parameters initially, then improves approximation accuracy, reducing the gap between forward binarization and backward approximation.
5. Experiments
The experiments evaluate IR-Net on CIFAR-10 and ImageNet across several network structures, using binarized convolutional and fully connected layers under specified architectural settings.
- IR-Net is evaluated on CIFAR-10 and ImageNet to compare its effectiveness with other state-of-the-art methods.
- The IR-Net implementation replaces original convolutional layers with binary convolutional layers in PyTorch.
- Experiments use VGG-Small, ResNet-18, ResNet-20, and ResNet-34, including normal and Bi-Real ResNet structures.
- All convolutional and fully connected layers except the first and last are binarized, with Hardtanh replacing ReLU for binarized activations.
- IR-Net is trained from scratch with random initialization rather than using a pretrained model.
- SGD is used as the optimization algorithm, while most hyperparameter settings follow the original network papers.
5.1. Ablation Study
The ablation study examines Libra-PB and EDE through information-retention analyses and component comparisons, finding that both components contribute to binary-network performance.
- Effect of Libra-PB: Libra-PB keeps each layer’s binary activations close to maximum Bernoulli-distribution entropy, unlike vanilla binarization’s accumulating information loss.
- Effect of Libra-PB: Libra-PB maximizes binary-weight and binary-activation information entropy by adjusting the network’s weight distribution.
- Effect of EDE: During early training, EDE retains an Identity-like derivative to preserve updating ability while avoiding excessively large derivatives near zero.
- Effect of EDE: As weights approach -1/+1 later in training, EDE increases the derivative toward a standard sign-function approximation to reduce gradient mismatch.
- Ablation Performance: Using Libra-PB or EDE alone improves accuracy, weight standardization is important, and their improvements are superimposable.
5.2. Comparison with SOTA methods
IR-Net is compared with state-of-the-art quantization methods on CIFAR-10 and ImageNet across multiple network structures. The reported results show strong performance, including improved accuracy and a smaller gap to full precision in a 1W/1A setting.
- 2.4% absolute accuracy improvement over the CIFAR-10 ResNet-20 SOTA baseline is reported for 1W/1A, reducing the full-precision gap to 4.3%.
- ImageNet evaluation examines IR-Net with ResNet-18 and ResNet-34 against multiple state-of-the-art quantization methods.
- 58.1% vs. 56.4% for ResNet-18 is reported for IR-Net’s Top-1 accuracy compared with state-of-the-art methods.
5.3. Deployment Efficiency
The paper evaluates IR-Net’s real-world inference efficiency on a Raspberry Pi 3B mobile-device platform. The implementation uses ARM NEON SIMD instructions and makes the daBNN inference framework compatible with IR-Net.
- IR-Net is implemented and tested on a Raspberry Pi 3B with a 1.2 GHz 64-bit quad-core ARM Cortex-A53 processor.
- The deployment uses the ARM NEON SIMD instruction SSHL for the inference implementation.
- The implementation makes the daBNN inference framework compatible with IR-Net.
6. Conclusion
The conclusion presents IR-Net as an information-retention approach for binary neural networks, combining Libra-PB for forward propagation with EDE for backward propagation. The paper reports consistent advantages over existing state-of-the-art binary neural networks.
- Libra-PB keeps forward-propagation diversity through balanced and standardized weight transformation without additional activation operations.
- EDE reduces gradient error in backward propagation while retaining gradient information through sufficient updating ability and accurate gradients.
- The combined IR-Net approach is reported to consistently outperform existing state-of-the-art binary neural networks.