Distribution-sensitive Information Retention for Accurate Binary Neural Network

Haotong Qin; Xiangguo Zhang; Ruihao Gong; Yifu Ding; Yi Xu; Xianglong Liu

Source-linked AI summary

Distribution-sensitive Information Retention for Accurate Binary Neural Network

Haotong Qin, Xiangguo Zhang, Ruihao Gong, Yifu Ding, Yi Xu, Xianglong Liu

arXiv:2109.12338v2Sep 25, 2021cs.CV

TL;DR

Binary neural networks offer compression and faster inference but retain a substantial performance gap because binarization loses information during forward and backward propagation. DIR-Net addresses this with information-retaining binarization, distribution-sensitive gradient estimation, and representation distillation, outperforming state-of-the-art binarization approaches across tasks and architectures while achieving 11.1x storage saving and 5.4x speedup on resource-limited hardware.

Problem
Binary neural networks reduce memory and computation but still have a substantial accuracy gap from 32-bit models because binarization loses information in forward and backward propagation.
Method
DIR-Net combines Information Maximized Binarization, Distribution-sensitive Two-stage Estimation, and Representation-align Binarization-aware Distillation to retain forward, backward, and representation information.
Results
DIR-Net consistently outperformed state-of-the-art binarization approaches across image classification and detection tasks and achieved 11.1x storage saving and 5.4x speedup on Raspberry Pi.
Takeaways & Limitations
The unified information perspective provides a practical and versatile way to improve binary neural networks across mainstream and compact architectures.
Takeaways & Limitations
Extra floating-point operations or parameters can harm deployment on lightweight edge hardware, constraining accuracy-improving BNN designs.

Abstract

from arXiv · show

Model binarization is an effective method of compressing neural networks and accelerating their inference process. However, a significant performance gap still exists between the 1-bit model and the 32-bit one. The empirical study shows that binarization causes a great loss of information in the forward and backward propagation. We present a novel Distribution-sensitive Information Retention Network (DIR-Net) that retains the information in the forward and backward propagation by improving internal propagation and introducing external representations. The DIR-Net mainly relies on three technical contributions: (1) Information Maximized Binarization (IMB): minimizing the information loss and the binarization error of weights/activations simultaneously by weight balance and standardization; (2) Distribution-sensitive Two-stage Estimator (DTE): retaining the information of gradients by distribution-sensitive soft approximation by jointly considering the updating capability and accurate gradient; (3) Representation-align Binarization-aware Distillation (RBD): retaining the representation information by distilling the representations between full-precision and binarized networks. The DIR-Net investigates both forward and backward processes of BNNs from the unified information perspective, thereby providing new insight into the mechanism of network binarization. The three techniques in our DIR-Net are versatile and effective and can be applied in various structures to improve BNNs. Comprehensive experiments on the image classification and objective detection tasks show that our DIR-Net consistently outperforms the state-of-the-art binarization approaches under mainstream and compact architectures, such as ResNet, VGG, EfficientNet, DARTS, and MobileNet. Additionally, we conduct our DIR-Net on real-world resource-limited devices which achieves 11.1x storage saving and 5.4x speedup.

1 Introduction

Binary neural networks reduce memory and computation through 1-bit inference, but binarization loses information in forward and backward propagation, creating an accuracy gap. DIR-Net addresses this with information-retaining binarization, gradient estimation, and representation distillation, and reports strong results across tasks, architectures, and edge hardware.

Problem: Existing BNNs exploit efficient bitwise inference but still suffer a large accuracy drop relative to full-precision models because binarization causes substantial information loss.The loss affects representation diversity in forward propagation and gradient accuracy or update capability in backward propagation.
Method: DIR-Net retains information during training through IMB for forward propagation, DTE for backward propagation, and RBD for aligning full-precision and binarized representations.The method improves internal propagation and introduces external representations to address information loss caused by binarization.
Method: IMB balances and standardizes weights to maximize quantized-parameter information while minimizing binarization error without adding inference-time cost.The approach targets forward-propagation information loss and uses an offline transformation.
Method: DTE adapts its estimator shape to weight and activation distributions, jointly retaining accurate gradients and sufficient parameter-updating capability.This addresses the trade-off between reducing gradient mismatch and preserving optimization updates.
Results: 11.1× storage saving and 5.4× speedup were achieved on Raspberry Pi while DIR-Net exceeded other binarization approaches across classification, detection, and multiple architectures.Evaluations covered CIFAR-10, ImageNet, PASCAL VOC, COCO, ResNet, VGG, EfficientNet, MobileNet, DARTS, Faster R-CNN, and SSD.
Contributions: The study analyzes BNNs from a unified information perspective and reports DIR-Net as versatile across mainstream and compact architectures using the standard training pipeline.The paper also extends its analysis beyond the method to information loss, binarization error, and component ablations.

2 Related Work

Related work compresses CNNs through binarization but leaves a substantial accuracy gap and deployment trade-off. DIR-Net addresses information loss while avoiding resource-consuming floating-point convolution operations.

Binary neural networks: Binary neural networks reduce parameters and accelerate inference through efficient bitwise operations, but their accuracy remains below full-precision counterparts.The initial binarization approach compressed parameters and accelerated computation, while subsequent methods sought to reduce the resulting accuracy gap.
Model binarization: Existing binarization approaches still cause a serious accuracy decrease compared with 32-bit models while often adding resource-consuming floating-point operations.The paper identifies information loss and extra computation as persistent limitations of prior approaches.
DIR-Net: DIR-Net retains information during BNN training while eliminating resource-consuming floating-point operations in convolutional layers.This design targets accuracy and deployment efficiency simultaneously.

3 Preliminaries

The preliminaries describe CNN computation with floating-point weights and activations, then explain how BNNs replace them with scaled binary tensors and bitwise operations. Because the sign function has an almost-everywhere-zero derivative, BNN training uses approximate gradient estimators.

CNN and BNN computation: Floating-point tensor multiplication consumes substantial memory and computation, motivating binary weights and activations for efficient CNN inference.BNNs target embedded deployment by replacing 32-bit tensors with 1-bit representations and exploiting hardware-friendly operations.
Binarization: BNNs represent weights or activations as scaled binary tensors with values in {−1, +1}, typically computed using the sign function.The scalar factors separately account for binary weights and activations.
Bitwise inference: Binary tensor multiplication is approximated with bitwise inner products implemented through XNOR and Bitcount, while Shift can replace multiplication in some inference components.Shift-based batch normalization is cited as a further hardware-oriented acceleration strategy.
Backward propagation: Since the sign function has a derivative that is zero almost everywhere, BNNs use Straight-Through Estimators such as identity or hardtanh for gradient propagation.These approximations address incompatibility between exact sign-function derivatives and backward propagation.

4 Distribution-sensitive Information Retention Network

DIR-Net treats BNN training as an information-retention problem caused by sign-based forward binarization and approximate backward gradients. It proposes a network designed to preserve information during training and improve performance.

Information-retention perspective: DIR-Net identifies information loss from the sign function and gradient approximation as a central cause of limited BNN performance.The paper frames both forward and backward propagation as sources of information loss during training.
DIR-Net: DIR-Net retains information during training by addressing information loss in both forward and backward propagation.The proposed network is presented as the paper’s response to the identified training problem.

4.1 Information Maximized Binarization in the Forward Propagation

Information Maximized Binarization retains forward-propagation information by jointly maximizing binary-parameter information and reducing quantization error through balancing and standardization.

Information Maximized Binarization: IMB jointly addresses information entropy loss and quantization error when binarizing weights and activations.The method is motivated by the numerical difference between full-precision and binarized parameters and by the limits of minimizing quantization error alone.
Information Maximized Binarization: IMB formulates information retention as maximizing mutual information between full-precision and binarized parameters, which reduces to maximizing binary-parameter entropy for deterministic quantization.The conditional entropy term vanishes under the deterministic sign quantizer.
Information Retention via Information Maximized Binarization: Standardization produces zero-mean, unit-norm balanced weights, maximizing binary entropy while preventing overly concentrated full-precision counterparts.The resulting standardized balanced progress supports more stable parameter updates.
Information Retention via Information Maximized Binarization: Balancing before binarization gives IMB-quantized parameters maximum information entropy under a Bernoulli distribution.The balance characteristic makes the two binary outcomes more evenly distributed than with the sign function.
Information Retention via Information Maximized Binarization: IMB also reshapes the data distribution before binarization as an implicit rectifier, distinguishing it from prior empirical redistribution strategies.The paper presents this distribution reshaping from an information-retention perspective.

4.4 Analysis and Discussions

DIR-Net’s analysis examines computational overhead and training stability, emphasizing reduced extra operations and mechanisms that stabilize binarized weights and gradients.

Complexity Analysis: DIR-Net reduces computation costs by using shift-based scalars instead of 32-bit scalars, while deployment speed is evaluated on hardware.The method adds no extra activation-binarization operation during training.
Complexity Analysis: Table 1 reports the additional 32-bit operations consumed by different binarization methods.
Stabilize Training: Weight standardization redistributes concentrated weights to reduce sign changes during optimization and stabilize binary-network training.The full-precision weights otherwise gather around zero, making their signs prone to flipping.
Stabilize Training: DTE stabilizes training by preserving updating capability, preventing excessively steep estimators, and avoiding excessively enlarged gradients.

5 Experiments

The experiments evaluate DIR-Net across classification and detection benchmarks, diverse mainstream and compact architectures, and an information-entropy analysis on ResNet-20.

Experimental Setup: DIR-Net is evaluated on CIFAR-10, ImageNet, PASCAL VOC, and COCO for image classification and object detection.
Experimental Setup: The implementation replaces original convolutional layers with binary convolutional layers binarized using DIR-Net.
Experimental Setup: Experiments cover VGG-Small, ResNet, MobileNetV1, EfficientNet-B0, DARTS, and ReActNet structures to assess versatility.
Information Analysis: ResNet-20 experiments analyze IMB’s effect on activation information entropy across layers.

5.1 Ablation Study

The ablations show that IMB, DTE, and RBD each retain information or improve optimization, with their combination producing substantial accuracy gains on CIFAR-10.

IMB: Information entropy is almost positively related to final accuracy, with maximum information producing the highest BNN accuracy.The result supports entropy as an indicator of retained information in binary networks.
IMB: IMB achieves a 1.1% gain over naive BNN, exceeding separate weight standardization and bit-shift scalar improvements of 0.5% and 0.8%.The ablation uses ResNet-20 on CIFAR-10.
RBD: 86.2% accuracy with RBD using Eq. (24) improves over 83.8% without it on ResNet-20/CIFAR-10.MSE and KL-divergence representation distillation improve results by 1.1% and 1.3%, respectively.
Combined ablation: IMB, DTE, and RBD improvements can be superimposed, enabling higher-accuracy binary neural networks.The authors attribute the combined gains to improved internal propagation and full-precision representation alignment.

5.2 Comparison with SOTA methods

Across CIFAR-10, ImageNet, PASCAL VOC, and COCO, DIR-Net outperforms existing binarization methods across mainstream and compact architectures.

CIFAR-10: DIR-Net has the highest accuracy in all listed CIFAR-10 comparisons and improves over DSQ by 4.9% on 1W/1A ResNet-20.Its ResNet-20 full-precision gap is reduced to 2.7%, and it generally exceeds IR-Net by 1.2% across backbones.
ImageNet: DIR-Net exceeds IR-Net by 0.7% Top-1 and 0.9% Top-5 accuracy on 1W/32A ResNet-34 and also surpasses state-of-the-art binarization methods at 1W/1A.The ImageNet comparison covers ResNet, MobileNetV1, DARTS, and EfficientNet-B0.
Compact architectures: On DARTS, DIR-Net surpasses Bi-Real by 2.8% Top-1 and 3.6% Top-5 without additional computational overhead or training steps.The compact-architecture evaluation also includes EfficientNet-B0 and MobileNet.
PASCAL VOC: DIR-Net improves mAP over existing methods by at least 3.3% for SSD300 and 2.2% for Faster R-CNN on PASCAL VOC.The evaluation uses SSD300 with VGG-16 and Faster R-CNN with ResNet-18.
COCO: On COCO, DIR-Net exceeds BiDet by 0.8% mAP with SSD300 and 0.4% mAP with Faster R-CNN, while seed standard deviation is at most 0.10%.The method also outperforms BNN, XNOR, and Bi-Real.

5.3 Deployment Efficiency on Raspberry Pi 3B

DIR-Net is evaluated on a Raspberry Pi 3B using ARM NEON SIMD instructions to assess practical inference efficiency against existing implementations.

Deployment setup: DIR-Net’s Raspberry Pi 3B deployment is tested with ARM NEON’s SSHL SIMD instruction and the daBNN inference framework.The device uses a 1.2 GHz 64-bit quad-core ARM Cortex-A53.
Efficiency comparison: The evaluation compares single-threaded ResNet-18 inference time across different bit widths and implementations.Table 9 compares DIR-Net with high-performance implementations including NCNN and DSQ.

6 Conclusion

DIR-Net retains information in binary neural networks by improving forward and backward propagation internally and aligning representations externally.

Conclusion: DIR-Net combines IMB for forward-propagation diversity, DTE for backward gradient accuracy, and RBD for external representation retention.The three techniques target complementary information losses in binary neural networks.
Conclusion: IMB maximizes weight and activation information entropy without additional activation operations, while DTE balances updating capability with accurate gradients.The conclusion presents both as information-retention mechanisms for internal propagation.
Conclusion: RBD uses corresponding full-precision networks to improve binary networks orthogonally to the internal-propagation techniques.The scheme relies on external full-precision representations.