Source-linked AI summary
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
Zechun Liu, Zhiqiang Shen, Marios Savvides, Kwang-Ting Cheng
TL;DR
Binary neural networks offer strong compression and computational efficiency but suffer substantial accuracy loss on large-scale datasets. ReActNet combines a compact MobileNetV1-based binary architecture, learnable activation-distribution reshaping, and distributional loss, achieving 69.4% ImageNet top-1 accuracy while reducing computational complexity by more than 22×.
Problem
Binary neural networks provide efficient 1-bit computation but retain a substantial accuracy gap from real-valued networks, especially on large-scale ImageNet.
Method
ReActNet uses a MobileNetV1-based binary baseline with parameter-free shortcuts, RSign and RPReLU activation-distribution adaptation, and distributional loss.
Results
69.4% top-1 accuracy on ImageNet was achieved with more than 22× lower computational complexity than the real-valued benchmark.
Takeaways & Limitations
Activation-distribution reshaping and output-distribution matching substantially improve binary-network accuracy at near-zero or negligible extra computational cost.
Abstract
from arXiv · showhide
In this paper, we propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts, bypassing all the intermediate convolutional layers including the downsampling layers. This baseline network strikes a good trade-off between accuracy and efficiency, achieving superior performance than most of existing binary networks at approximately half of the computational cost. Through extensive experiments and analysis, we observed that the performance of binary networks is sensitive to activation distribution variations. Based on this important observation, we propose to generalize the traditional Sign and PReLU functions, denoted as RSign and RPReLU for the respective generalized functions, to enable explicit learning of the distribution reshape and shift at near-zero extra cost. Lastly, we adopt a distributional loss to further enforce the binary network to learn similar output distributions as those of a real-valued network. We show that after incorporating all these ideas, the proposed ReActNet outperforms all the state-of-the-arts by a large margin. Specifically, it outperforms Real-to-Binary Net and MeliusNet29 by 4.0% and 3.6% respectively for the top-1 accuracy and also reduces the gap to its real-valued counterpart to within 3.0% top-1 accuracy on ImageNet dataset. Code and models are available at: https://github.com/liuzechun/ReActNet.
1 Introduction
ReActNet targets the accuracy gap between binary and real-valued networks by combining an efficient MobileNetV1-based baseline with learnable activation-distribution operations and distributional loss. The resulting model reaches 69.4% ImageNet top-1 accuracy with substantially lower computational cost.
- Motivation: 1-bit CNNs compress weights and activations while enabling efficient logical computation, but their accuracy degradation limits deployment on large datasets.They provide 32× memory compression and up to 58× practical CPU reduction, while XNOR-Net reaches only 51.2% ImageNet accuracy.
- Baseline design: The baseline modifies MobileNetV1 with parameter-free identity shortcuts that bypass binary convolutions, including downsampling layers.The design aims to improve accuracy while retaining the practical value of a compact backbone.
- Activation functions: RSign and RPReLU learn activation-distribution shifts and reshaping, improving baseline accuracy by approximately 7% with negligible extra computational cost.They generalize Sign and PReLU to explicitly adapt distributional parameters.
- Distributional loss: A distributional loss enforces output-distribution similarity between binary and real-valued networks and further boosts accuracy by approximately 1%.The loss replaces the original loss in the proposed training approach.
- Results: 69.4% top-1 accuracy on ImageNet exceeds the 69.3% ResNet-level benchmark while reducing computational complexity by more than 22×.The result also outperforms a state-of-the-art binary network by 4.0% top-1 accuracy at half the OPs.
2 Related Work
Prior binary-network work improved accuracy through architectural, training, optimization, loss, and approximation techniques, but often retained a large efficiency–accuracy trade-off. ReActNet instead emphasizes activation-distribution reshaping and shifting alongside a compact baseline and distributional loss.
- Binary-network compression: Network binarization quantizes both weights and activations to 1 bit as an extreme form of neural-network compression.The approach belongs to the broader compression literature alongside quantization, pruning, distillation, and compact network design.
- Early methods: Early BNN methods performed well on CIFAR10 and MNIST but suffered severe accuracy drops on ImageNet.The cited BNN result is 42.2% top-1 accuracy versus 69.3% for real-valued ResNet-18.
- Architectural advances: XNOR-Net and Bi-Real Net improved ImageNet top-1 accuracy to 51.2% and 56.4%, respectively, through scaling factors and real-valued shortcuts.These gains narrowed the accuracy gap but did not eliminate the efficiency challenge.
- Accuracy–efficiency trade-off: Expanding channel width, increasing depth, or using multiple binary weight bases improved accuracy but added computational cost that weakened binary networks’ compression advantage.This work therefore emphasizes efficiency alongside accuracy.
- ReActNet baseline: A MobileNetV1-based baseline achieved 61.1% top-1 accuracy with approximately twofold lower computational cost than XNOR-Net and Bi-Real Net.The compact backbone was selected because the real-valued starting design should itself be efficient.
- Paper positioning: ReActNet investigates activation-distribution reshaping and shifting as a comparatively underexplored accuracy factor, combining ReAct operations with distributional loss to reach 69.4%.The resulting accuracy gap to the real-valued counterpart is reported as 3.0%.
3 Revisit: 1-bit Convolution
1-bit convolutions binarize weights and activations so floating-point matrix multiplication can be replaced by XNOR and popcount operations. The reviewed network uses binary layers with shortcuts and weight scaling, while ReAct operations remove the need for activation scaling.
- Binary weights and activations replace floating-point matrix multiplication with bitwise XNOR and popcount operations.Both weights and activations are binarized to -1 and +1.
- The baseline replaces MobileNetV1 depth-wise and point-wise convolution pairs with proposed blocks while preserving its channel and layer configuration.Reduction blocks duplicate and concatenate input activations when channel numbers differ.
- Identity shortcuts bypass 1-bit convolutions when input and output channel counts match, while reduction blocks use duplicated inputs and concatenation to address channel differences.The baseline also uses average pooling in shortcuts for spatial downsampling.
- The weight binarization method uses the average absolute real-valued weight as a scaling factor to reduce the difference between binary and real-valued weights.The paper denotes binary and real-valued quantities with subscripts b and r, respectively.
- Introducing ReAct operations makes activation scaling unnecessary and allows it to be eliminated.This change is attributed to the proposed activation operations described later in the paper.
4 Methodology
ReActNet combines a MobileNetV1-based binary architecture with learnable activation-distribution transformations and a distributional loss. These components address the sensitivity of binary feature learning to activation shifts while preserving low computational cost.
- Baseline network: MobileNetV1 is selected as the baseline backbone, with shortcuts bypassing binary convolutions to support compact-network deployment.Shortcuts are added for both ordinary and downsampling layers; channel duplication and average pooling address dimensional mismatches while keeping layers binarized except the first convolution and final fully connected layer.
- Distribution matters: Binary networks are highly sensitive to activation-distribution variation because small shifts before sign binarization can substantially change binary features.Ill-shifted distributions can introduce excessive background noise or too few useful features, harming feature learning.
- ReAct operations: RSign learns channel-wise binarization thresholds, shifting each input distribution to obtain more informative binary activations.The threshold coefficient α can vary across channels.
- ReAct operations: RPReLU learns channel-wise shifts and a negative-region slope to reshape and reposition activation distributions.It shifts inputs by −γ, scales the negative part by β, and shifts outputs by ζ; γ, β, and ζ are learnable coefficients.
- ReAct operations: RSign and RPReLU introduce only 4× the network’s channel count in extra parameters and negligible computational overhead relative to convolutions.Their computation is approximately that of a typical nonlinear layer.
- Distributional loss: The distributional loss uses KL divergence between real-valued and binary networks’ softmax outputs to enforce output-distribution similarity.Unlike block-wise matching methods, it does not require architectural similarity between the real-valued and binary networks.
5 Experiments
Experiments on ImageNet evaluate ReActNet against prior methods and isolate the effects of its architectural and activation-function components. The results show higher accuracy, efficient computation, and improved training stability.
- Comparison with State-of-the-art: ReActNet-A outperforms Real-to-Binary Network by 4.0% top-1 accuracy with approximately half the OPs.
- Comparison with State-of-the-art: ReActNet-B achieves 13.7% higher accuracy than Bi-RealNet-18 with the same number of OPs.
- Ablation Study: Using both RSign and RPReLU raises accuracy by 6.9% over the baseline and narrows the gap to the corresponding real-valued network to within 3.0%.
- Ablation Study: ReActNet learns positive and negative RPReLU slopes, unlike traditional PReLU slopes, which are positive only.
- Ablation Study: ReActNet produces more enriched activation distributions and a more balanced distribution of -1 and +1 after the sign function.
6 Conclusions
The paper develops a more accurate 1-bit CNN by combining parameter-free shortcuts, learnable distribution reshaping, and distributional loss. ReActNet reaches 69.4% top-1 accuracy on ImageNet at 87M OPs, within 3% of its real-valued counterpart.
- The baseline binary network modifies MobileNetV1 with parameter-free shortcuts and achieves 61.1% top-1 accuracy at 87M OPs on ImageNet.
- RSign and RPReLU learn activation-distribution shifts and reshaping, while distributional loss matches binary and real-valued output distributions.
- ReActNet achieves 69.4% top-1 accuracy on ImageNet, only 3% below its real-valued counterpart at substantially lower computational cost.