Source-linked AI summary
Training Binary Neural Networks with Real-to-Binary Convolutions
Brais Martinez, Jing Yang, Adrian Bulat, Georgios Tzimiropoulos
TL;DR
Binary neural networks promise efficient deployment but face substantial accuracy gaps relative to real-valued networks. The paper builds a strong baseline and adds progressive attention matching plus data-driven channel re-scaling, achieving competitive ResNet-18 results on CIFAR-100 and ImageNet. Its combined methods improve binary-network accuracy while narrowing the full-precision gap.
Problem
Binary neural networks provide compression and computational benefits but typically suffer large accuracy drops, limiting their practical deployment.
Method
The paper combines a strong optimization baseline with progressive real-to-binary attention matching and data-driven channel re-scaling based on pre-binarization activations.
Results
76.2% top-1 accuracy on CIFAR-100 and 65.4% on ImageNet reduce the full-precision gaps below 3% and 5%, respectively, with ResNet-18.
Takeaways & Limitations
The combined strategies make binary networks a compelling alternative to real-valued networks and improve ImageNet state of the art by 5.5 top-1 accuracy.
Takeaways & Limitations
The attention-matching approach assumes that closer agreement between binary and real convolution outputs yields significant accuracy gains.
Abstract
from arXiv · showhide
This paper shows how to train binary networks to within a few percent points ($\sim 3-5 \%$) of the full precision counterpart. We first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet and reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR-100 and ImageNet respectively when using a ResNet-18 architecture. Code available at https://github.com/brais-martinez/real2binary.
1 INTRODUCTION
Binary neural networks offer substantial compression and speed advantages but historically suffer large accuracy gaps. This paper combines a strong baseline with attention matching and data-driven channel re-scaling to narrow that gap.
- Motivation: 32× lower memory usage and CPU speed-ups of up to 58× make binary networks attractive for resource-limited devices.Binary weights replace 32-bit floats, while xnor and pop-count operations replace floating-point multiply-adds.
- Contributions: A strong baseline combines recent binary-network advances with carefully selected optimization techniques and achieves state-of-the-art ImageNet accuracy.The baseline surpasses previously published binary-network results on ImageNet.
- Contributions: Real-to-binary attention matching aligns spatial attention maps from real-valued and binary convolutions, with supervision injected at each binary block.The method also progressively bridges architectural differences through a sequence of teacher-student pairs.
- Contributions: Data-driven channel re-scaling uses pre-binarization real-valued activations to compute scale factors for outputs produced by binary convolutions.The approach is implemented through a gating function that predicts channel-scaling factors.
- Results: 76.2% top-1 accuracy on CIFAR-100 and 65.4% on ImageNet leave gaps below 3% and 5%, respectively, using ResNet-18.These results come from combining the proposed improvements.
2 RELATED WORK
Related work develops binary-network training, scaling, and architectural strategies to improve accuracy while balancing computational cost. The paper positions its attention matching and data-driven re-scaling against these approaches.
- Binary-network training: BNN established back-propagation training with real-valued latent weights, but retained an approximately 28% ImageNet top-1 gap versus full-precision ResNet-18.Weights are binarized during the forward pass and updated through the binarization step during back-propagation.
- Binary-network training: Optimization research improves binary networks through smooth gradients, PReLU, staged binarization, progressive quantization, and learned channel correlations.These methods target the optimization difficulties introduced by binarization.
- Architectural trade-offs: Some methods increase binary operations by factors of 2 to 8, relaxing constraints in exchange for additional computation.Examples include ABC-Net, structure approximation, circulant CNN, and binary ensembles.
- Attention matching: Prior feature-distillation work matches whole feature maps, whereas this paper uses attention matching between real-valued and binary networks.The cited prior approach trains a real-valued network in parallel with the quantized network.
- Channel re-scaling: Unlike discriminatively learned or fixed analytic scaling factors, the proposed channel re-scaling predicts task-oriented factors from data-driven activations.The approach is related to XNOR-Net and later learned scaling methods but differs in how factors are obtained.
3 BACKGROUND
Binary convolution replaces real-valued weights and activations with signs, then uses scaling factors to reduce the resulting quantization error. XNOR-Net provides the standard scaled formulation, with later work learning a unified factor.
- Baselines: The paper treats XNOR-Net and BNN as standard baselines for neural-network binarization.XNOR-Net is described as the method of choice for binarization.
- Binarization: The BNN formulation binarizes weights and activations with the sign function before applying a binary convolution.The binary convolution is implemented using bit-level operations.
- Scaling: Direct binarization introduces high quantization error and low accuracy, motivating real-valued scaling of binary-convolution outputs.XNOR-Net addresses this limitation with separate weight and activation scaling factors.
- Scaling: XNOR-Net computes analytic weight and activation scaling factors, while later work fuses them into one factor learned by back-propagation.The unified factor is denoted Γ in the cited formulation.
4 METHOD
The method builds a strong binary ResNet-18 baseline through architectural and optimization choices, then narrows the real-to-binary gap with attention matching and data-driven channel re-scaling.
- Building a strong baseline: The baseline combines recent binary-network advances with optimization choices, including modified residual blocks, double skip connections, PReLU activations, learned scaling factors, and real-valued downsampling.The block ordering is BatchNorm → Binarization → BinaryConv → Activation, with the skip connection last.
- Building a strong baseline: A 2-stage strategy first trains binary activations with real-valued weights, then initializes training with both weights and activations binarized.Stage 2 uses zero weight decay because binary weights are restricted to 1 or −1.
- Building a strong baseline: The optimization uses Adam with a stepwise scheduler, while augmentation and mix-up are selected differently for CIFAR-100 and ImageNet.Mix-up is described as crucial for CIFAR-100 but slightly harmful for ImageNet.
- Real-to-binary attention matching: The method assumes that matching binary and real-valued convolution outputs can improve accuracy, implemented through attention-map matching and logit matching.Attention matching is applied at multiple transfer points, with standard logit matching also used at the network end.
- Real-to-binary attention matching: A progressive sequence of teacher-student pairs gradually bridges architectural differences from a real-valued ResNet to a fully binary network.The sequence moves through an architecture-matched real-valued student, a binary-activation student, and finally a binary-weight-and-activation student.
- Data-driven channel re-scaling: Data-driven channel re-scaling predicts scale factors from pre-binarization real-valued activations and applies them after binary convolution.The gating function learns these factors by backpropagation for the task rather than using fixed factors shared across examples.
- Computational cost analysis: The method keeps binary operations constant while increasing floating-point operations by only 1% of the total floating-point operation count.This estimate assumes a factor r of 8 and requires a specialized engine to fully exploit binary-convolution efficiency during inference.
5 RESULTS
The experiments evaluate the method on ImageNet and CIFAR-100, combining state-of-the-art comparisons with ablations of teacher supervision, real-valued gaps, downsampling, and gating.
- Experimental setup: ImageNet serves as the benchmark for comparisons with state-of-the-art binary and low-bit quantization methods, while CIFAR-100 supports ablation studies.The experiments use ResNet-18 architectures.
- State-of-the-art comparison: 5.5% top-1 accuracy improvement over the prior state of the art is achieved by the full method on ImageNet.The strong baseline already surpasses the previously best-reported binary-network result by about 1%.
- State-of-the-art comparison: The method surpasses all compared low-bit quantization methods except TTQ, which uses 2-bit weights, full-precision activations, and 1.5× channel width.
- Ablation studies: ∼5% top-1 accuracy gain comes from progressive teacher-student training, compared with ∼1% for teacher-supervised real-valued ResNet-18 training.The authors characterize progressive teacher-student training as particularly important for binary networks.
- Ablation studies: About 2% and 3% gaps to real-valued networks are obtained on CIFAR-100 with full-precision and teacher-supervised comparisons, respectively.Relative degradation is reported as 13% on ImageNet and 10% on CIFAR-100.
- Ablation studies: 5.5% and 6.6% top-1 accuracy gains are observed with binary and real-valued downsample layers, respectively.The ablation results are reported as consistent across entries for both downsample choices.
- Ablation studies: The gating module is ineffective without attention matching, while its added flexibility helps the network mimic the attention map.
6 CONCLUSION
The paper trains binary networks to within a few percentage points of real-valued counterparts by combining progressive attention matching with gating-based output scaling.
- 6 CONCLUSION: The combined strategies improve binary-network ImageNet state-of-the-art performance by 5.5 top-1 accuracy.Attention matching drives optimization, while gating scales binary-convolution outputs and increases convolutional-block representation power.