Source-linked AI summary
XNOR-Net++: Improved Binary Neural Networks
Adrian Bulat, Georgios Tzimiropoulos
TL;DR
Fully binary networks target efficient deployment but suffer from quantization error and difficult training. XNOR-Net++ fuses XNOR-Net’s analytic activation and weight scaling factors into discriminatively learned factors and explores their shapes. On ImageNet, it reports up to 6% improvement over XNOR-Net or the state of the art within the evaluated computational setting.
Problem
Fully binary networks offer efficient computation but introduce large quantization errors, while XNOR-Net’s analytic scaling factors are limited because they do not optimize the task loss.
Method
XNOR-Net++ fuses activation and weight scaling factors into one factor learned discriminatively via backpropagation and explores multiple shapes at fixed test-time complexity.
Results
Up to 6% improvement on ImageNet classification is reported over the state of the art using ResNet-18, with improved training applicable to shallow and deep residual networks.
Takeaways & Limitations
Learning scaling factors for the task loss and varying their shapes improves the representational power and reported accuracy of binary networks.
Takeaways & Limitations
The first and last layers of both networks were kept real.
Abstract
from arXiv · showhide
This paper proposes an improved training algorithm for binary neural networks in which both weights and activations are binary numbers. A key but fairly overlooked feature of the current state-of-the-art method of XNOR-Net is the use of analytically calculated real-valued scaling factors for re-weighting the output of binary convolutions. We argue that analytic calculation of these factors is sub-optimal. Instead, in this work, we make the following contributions: (a) we propose to fuse the activation and weight scaling factors into a single one that is learned discriminatively via backpropagation. (b) More importantly, we explore several ways of constructing the shape of the scale factors while keeping the computational budget fixed. (c) We empirically measure the accuracy of our approximations and show that they are significantly more accurate than the analytically calculated one. (d) We show that our approach significantly outperforms XNOR-Net within the same computational budget when tested on the challenging task of ImageNet classification, offering up to 6\% accuracy gain.
1 Introduction
Binary neural networks offer substantial compression and CPU speed gains, but binarization reduces accuracy. XNOR-Net++ replaces analytically computed scaling with discriminatively learned factors and reports improved accuracy on ImageNet within the same computational budget.
- Motivation: Binary convolutions can achieve ∼32× model compression and ∼58× CPU speed-up, but these gains come at the cost of reduced accuracy.The approach targets deployment beyond desktop machines with high-end GPUs.
- Background: XNOR-Net compensates for binary approximation error by re-scaling binary-convolution outputs with real-valued factors at negligible parameter and complexity cost.These factors preserve the computational advantages of binary convolutions while reducing approximation error.
- Approach: XNOR-Net++ learns fused activation and weight scaling factors through backpropagation, optimizing them for the task loss rather than approximation error.The method replaces separate analytic scaling factors with a single discriminatively learned factor.
- Approach: The method explores multiple scaling-factor shapes, including statistically learned spatial and channel-wise structures, while keeping test-time complexity fixed.These constructions increase the expressivity of the scaling factors without increasing the computational budget.
- Evaluation: XNOR-Net++ approximations are reported as significantly more accurate than analytically calculated factors.The paper evaluates approximation accuracy as a distinct contribution.
- Results: More than 6% absolute ImageNet accuracy improvement is reported over the state-of-the-art, with the training improvement applied to both shallow and deep residual networks.The result is presented under the paper’s binary-network computational setting.
2 Related work
Related work addresses computational constraints through pruning, conditional computation, low-rank approximations, efficient architectures, quantization, and binarization. XNOR-Net++ focuses on learned scaling factors rather than architecture redesign and positions itself as complementary to topology improvements.
- Efficiency methods: Efficiency methods include pruning, conditional computation, low-rank approximations, efficient architectures, and network quantization.These approaches target deep-learning deployment under tight computational constraints.
- Efficient architectures: Efficient convolutional-network design takes holistic approaches that modify overall structure or local approaches that modify convolutional blocks or operations.The distinction organizes architectural methods for mobile and constrained-device settings.
- Efficient architectures: Local architectural optimization includes replacing large filters with smaller filter sequences and decomposing convolutions into asymmetric kernels or bottleneck projections.The cited examples include VGG, Inception, and bottleneck blocks.
- Efficient architectures: Holistic optimization includes residual, dense-connectivity, and real-time object-detection architectures that alter network topology.These methods improve connectivity or optimize the backbone structure for faster inference.
- Network binarization: Network binarization quantizes features and weights to two states, typically {±1}, aligning neural networks with low-precision hardware support.Fully binary networks binarize both weights and activations, whereas other methods binarize only weights or use multi-bit activations.
- Network quantization: Prior quantization methods vary bit allocation or learn codebooks, including 1-bit weights with 2-bit activations and higher-bit feature representations.These methods differ from fully binary networks in their precision allocation.
- Binary networks: XNOR-Net introduced separate real-valued scaling factors for binary weights and features, while XNOR-Net++ fuses them into a single discriminatively learned factor and explores its shape.This work addresses scaling-factor learning rather than proposing a new network topology.
3 Background
Fully binary networks binarize weights and input features, but XNOR-Net reduces the resulting quantization error with analytically computed real-valued scaling factors. The paper reviews this procedure and motivates learning a fused scaling factor instead.
- XNOR-Net remains a state-of-the-art method that uses scaling factors to improve training of binary networks.
- For each CNN layer, weights W and input features I are binarized by applying the sign function.W has dimensions o×c×w×h, while I has dimensions c×w_in×h_in.
- Analytically calculated real-valued factors α and K re-weight binary convolution outputs to mitigate quantization errors.α scales weights, whereas K scales input features.
- The binary convolution uses bitwise operations, while the corresponding real-valued convolution uses ordinary convolution and element-wise multiplication combines scaled outputs.
- Because K is recomputed at every forward pass and can be expensive, the paper fuses α and K into one factor learned by backpropagation.The paper also explores different shapes for this fused factor.
4 Method
XNOR-Net++ replaces analytically computed activation and weight scaling factors with a single discriminatively learned factor and explores several shapes for constructing it. The method preserves a fixed test-time computational budget while improving approximation fidelity and binary-network efficiency.
- Scaling-factor learning: XNOR-Net++ fuses activation and weight scaling factors into one factor Γ learned discriminatively through backpropagation.The learned factor is optimized for the task loss rather than only for reducing binarization approximation error.
- Scaling-factor construction: The method explores four ways to construct Γ while keeping test-time complexity fixed.The proposed constructions increase scaling-factor expressivity through learned spatial and channel-wise structure.
- Scaling-factor construction: Learned channel-wise scaling already outperforms analytically calculated factors, while dense spatial scaling improves performance by 0.6%.Case 1 learns one factor per input channel; Case 2 learns one value for each output pixel.
- Inference cost: At test time, the learned factors are merged into one factor requiring a single element-wise multiplication.This keeps the deployed computation fixed despite using richer factor shapes during training.
- Binary approximation: The method uses binary weights and activations with real-valued scaling to reduce quantization error in binary convolutions.The binary weights in the reconstruction experiment are fixed as sign(W), while scaling factors are trained for the proposed cases.
- Efficiency analysis: Fused scaling requires c × wout × hout additional floating-point operations, compared with separate scaling operations in XNOR-Net.For a representative 256-channel 3 × 3 ResNet layer, the theoretical speed-ups are approximately 63.98× for the proposed method and 63.69× for XNOR-Net, with approximately 64× storage savings.
5 Results
The experiments train binarized AlexNet and ResNet-18 models on ImageNet and compare them with binary networks under the same computational and parameter budget. XNOR-Net++ substantially improves ResNet-18 accuracy, while AlexNet remains harder to train and prone to overfitting.
- Comparison with state of the art: More than 6% absolute improvement is reported over the state of the art on ImageNet classification.The comparison uses methods that binarize both weights and features within the same computational budget.
- Experimental setup: The first and last layers of both networks remain real-valued, following XNOR-Net.The comparison therefore focuses on networks whose intermediate layers binarize weights and features.
- Experimental setup: Experiments evaluate binarized AlexNet and ResNet-18 models on ImageNet using standard binary-network training procedures.ImageNet contains 1.2M training images, 50,000 validation images, and 1,000 non-overlapping classes.
- Comparison with state of the art: 57.1% top-1 accuracy is achieved by the proposed method with ResNet-18, compared with 42.2% for same-parameter ABC-Net.ABC-Net reaches higher accuracy by increasing network size 25×, so the cited comparison uses equal parameter counts.
- Comparison with state of the art: AlexNet shows a smaller improvement and is substantially harder to train and prone to overfitting.The paper reports stronger gains for ResNet-18 than for AlexNet under the evaluated binary-network setting.
6 Conclusion
The paper revisits binary-convolution scaling by learning scale factors discriminatively and exploring alternative shapes. On ImageNet classification with ResNet-18, this approach delivers improvements of up to 6%.
- The authors learn scale factors discriminatively via backpropagation instead of computing them analytically.
- The paper explores different shapes for the learned scale factors.
- Up to 6% improvement is reported on ImageNet classification using ResNet-18.