Source-linked AI summary
Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, Kwang-Ting Cheng
TL;DR
1-bit CNNs offer substantial efficiency benefits but lose classification accuracy on large-scale datasets such as ImageNet. The paper proposes Bi-Real net, which preserves real activations through shortcuts and combines this design with specialized optimization. On ImageNet, it achieves 56.4% and 62.2% top-1 accuracy for 18- and 34-layer networks, respectively, while reporting efficiency gains over full-precision CNNs and higher accuracy than XNOR-Net.
Problem
1-bit CNNs are efficient but have much worse classification accuracy than real-valued CNNs on large-scale datasets such as ImageNet.
Method
Bi-Real net preserves real activations through identity shortcuts and trains 1-bit CNNs with improved sign-gradient approximation, magnitude-aware weight gradients, and clip-based initialization.
Results
56.4% and 62.2% top-1 accuracy are achieved on ImageNet by 18-layer and 34-layer Bi-Real nets, respectively, with up to 16.0× memory saving and 19.0× computational cost reduction versus full-precision CNNs.
Takeaways & Limitations
Bi-Real net and its training algorithm improve 1-bit CNN accuracy while retaining binary weights and activations inside each 1-bit convolution layer.
Takeaways & Limitations
The magnitude-aware-gradient discussion assumes a single weight kernel, with the binary weight represented as a matrix.
Abstract
from arXiv · showhide
In this work, we study the 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While being efficient, the classification accuracy of the current 1-bit CNNs is much worse compared to their counterpart real-valued CNN models on the large-scale dataset, like ImageNet. To minimize the performance gap between the 1-bit and real-valued CNN models, we propose a novel model, dubbed Bi-Real net, which connects the real activations (after the 1-bit convolution and/or BatchNorm layer, before the sign function) to activations of the consecutive block, through an identity shortcut. Consequently, compared to the standard 1-bit CNN, the representational capability of the Bi-Real net is significantly enhanced and the additional cost on computation is negligible. Moreover, we develop a specific training algorithm including three technical novelties for 1- bit CNNs. Firstly, we derive a tight approximation to the derivative of the non-differentiable sign function with respect to activation. Secondly, we propose a magnitude-aware gradient with respect to the weight for updating the weight parameters. Thirdly, we pre-train the real-valued CNN model with a clip function, rather than the ReLU function, to better initialize the Bi-Real net. Experiments on ImageNet show that the Bi-Real net with the proposed training algorithm achieves 56.4% and 62.2% top-1 accuracy with 18 layers and 34 layers, respectively. Compared to the state-of-the-arts (e.g., XNOR Net), Bi-Real net achieves up to 10% higher top-1 accuracy with more memory saving and lower computational cost. Keywords: binary neural network, 1-bit CNNs, 1-layer-per-block
1 Introduction
The paper targets the accuracy gap between efficient 1-bit CNNs and real-valued CNNs by preserving real activations and introducing specialized training techniques. On ImageNet, the resulting Bi-Real net substantially improves accuracy while retaining major efficiency benefits.
- Motivation: 1-bit CNNs reduce memory and computation through binary weights, activations, XNOR operations, and bit-counting, but perform much worse than real-valued models on ImageNet.The paper motivates improving binary CNNs for resource-constrained devices, where computation and memory limit deployment.
- Bi-Real net: Bi-Real net adds an identity shortcut that propagates real activations to the next block, increasing representational capability with negligible computation and no additional memory cost.The shortcut preserves information that conventional 1-bit CNNs discard when real activations are binarized by the sign function.
- Training algorithm: The training algorithm approximates the sign derivative with a piecewise linear function derived from a second-order piecewise polynomial approximation.The paper argues this approximation is closer to the sign function than the clip-function approximation used previously.
- Training algorithm: The method also uses magnitude-aware weight gradients and clip-function pre-training because standard gradients and ReLU initialization are inadequate for binary optimization.Magnitude-aware gradients account for real-weight magnitude, while clip activations are closer to binary activations than ReLU outputs.
- Results: 56.4% and 62.2% top-1 accuracy are achieved by 18-layer and 34-layer Bi-Real nets on ImageNet, respectively.The model also reports up to 16.0× memory saving, 19.0× computational cost reduction, and 10% higher top-1 accuracy than XNOR-Net on the 18-layer network.
2 Related Work
Related work compresses CNNs through parameter reduction, weight quantization, and low-bit weights or activations. The paper positions Bi-Real net as a 1-bit CNN design that uses real-valued shortcuts and customized optimization to address accuracy loss.
- Parameter reduction: Parameter-reduction methods use bottleneck or 1×1 convolution structures to reduce model size and accelerate training.Examples include bottleneck blocks and SqueezeNet, which reports a 50× reduction in parameter count.
- Quantization: Weight quantization can save memory with limited accuracy degradation, but its acceleration is constrained by real-valued activations.This limitation motivates quantizing both weights and activations.
- Binary networks: DoReFa-Net and QNN use 1-bit weights with 2-bit activations, while BinaryNet uses 1-bit weights and activations and performs comparably to full-precision models on MNIST and CIFAR-10.The cited ImageNet accuracy drops for DoReFa-Net and QNN are 6.1% and 4.9%, respectively, relative to real-valued AlexNet.
- Bi-Real net: Bi-Real net extends 1-bit CNNs with a real-valued shortcut and customized optimization to compensate for binarization-related accuracy loss.The proposed optimization addresses gradient mismatch, discrete weight optimization, and initialization.
3 Methodology
Bi-Real Net preserves real-valued activations through identity shortcuts to increase representational capability, and pairs this architecture with specialized training techniques for binary CNNs.
- Standard 1-bit CNNs: 1-bit CNNs use binary weights and intermediate activations obtained through sign functions, while real activations remain during training and inference.Binary convolution outputs can be integer-valued and become real-valued after BatchNorm before subsequent binarization.
- Standard 1-bit CNNs: The sign function can collapse richer integer or real convolution outputs back to binary activations, reducing representational capability on large-scale datasets.For the example described, capability expands after binary convolution but shrinks again after the sign function.
- Bi-Real Net: Bi-Real Net adds an identity shortcut from block input activations to post-BatchNorm activations, then feeds their sum into the next block’s sign function.The block is organized as Sign → 1-bit convolution → BatchNorm → addition operator.
- Bi-Real Net: The shortcut significantly enhances each block’s representational capability with only an additional real-valued addition and no extra memory.The real activations already exist in standard 1-bit CNNs and are computed on the fly.
- Training Bi-Real Net: Training addresses nondifferentiable activation binarization and insufficient weight-gradient magnitude through derivative approximation and magnitude-aware weight updates.The proposed activation derivative uses a piecewise linear function derived from a second-order piecewise polynomial approximation to the sign function.
- Training Bi-Real Net: The initialization method pre-trains a real-valued CNN with clip(−1, x, 1) instead of ReLU because clip activations more closely match binary activations.The training algorithm is intended for Bi-Real Net and other 1-bit CNN structures.
4 Experiments
Experiments on ImageNet evaluate Bi-Real net’s shortcut design, three training techniques, accuracy against binary-network baselines, and efficiency. The results show improved accuracy and substantial memory and computation savings.
- Experimental setup: ImageNet experiments compare Bi-Real net, ResNet, and Plain Net structures, using combinations of the three proposed training techniques.The evaluation reports top-1 and top-5 accuracy on ILSVRC12 ImageNet.
- Ablation study: Bi-Real net and binarized ResNet outperform their plain counterparts, validating the effectiveness of shortcut connections.The plain structure’s thin, deep design limits representational capacity in its last convolution layer.
- Ablation study: About 18% accuracy improvement separates the 18-layer Bi-Real net from the binarized standard ResNet-18.The comparison supports the greater capacity of Bi-Real net’s more frequent shortcuts and its 1-layer-per-block structure.
- Ablation study: About 23%, 12%, and 13% improvements come respectively from the weight, activation, and clip-initialization techniques in the 18-layer Bi-Real net.The three mechanisms are reported to function independently and collaboratively, and improve accuracy across all three network structures.
- Comparison with state-of-the-art methods: Roughly 10% relative top-1 improvement over XNOR-Net is reported for the 18-layer Bi-Real net, with considerable margins over BinaryNet and ABC-Net.The comparison covers both top-1 and top-5 accuracies on Res-18 and Res-34.
- Efficiency analysis: 11.1× and 16.0× memory reductions, plus about 11.1× and 19.0× computation reductions, are reported for the 18-layer and 34-layer networks versus full precision.The efficiency analysis uses parameter bit widths for memory and FLOPs with 1-bit operations counted as 1/64 of real-valued multiplication.
5 Conclusion
The paper proposes Bi-Real net, which uses shortcuts to improve the representational capability of 1-bit CNNs, together with a specialized training algorithm. Experiments report superiority over state-of-the-art binary networks, while future work targets alternative optimization algorithms.
- Conclusion: Bi-Real net enhances 1-bit CNN representational capability through a simple shortcut and introduces specialized training methods for binary weights and activations.The training algorithm uses tighter activation-gradient approximation, magnitude-aware weight gradients, and novel initialization.
- Conclusion: Experimental results demonstrate superiority of Bi-Real net and its training algorithm over state-of-the-art methods.The conclusion identifies integer programming algorithms such as Lp-Box ADMM as future training directions.