Source-linked AI summary
Rotated Binary Neural Network
Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, Chia-Wen Lin
TL;DR
BNNs reduce computational and memory costs but suffer quantization error and an accuracy gap because prior methods largely leave angular bias unresolved. RBNN rotates weights toward binary vectors, uses bi-rotation and training adjustments, and outperforms compared methods on CIFAR-10 and ImageNet while achieving around 50% weight flips.
Problem
BNNs face substantial quantization error because scaling methods address the norm gap but leave angular bias between full-precision and binary weight vectors unresolved.
Method
RBNN aligns weights and binary vectors through epoch-wise rotation, bi-rotation with smaller matrices, dynamically adjusted rotated weights, and a training-aware sign-function approximation.
Results
RBNN consistently outperforms compared binary models on CIFAR-10 and ImageNet, with around 50% weight flips maximizing BNN information gain.
Takeaways & Limitations
Reducing angular bias provides a supported route to lowering BNN quantization error and improving binary-model accuracy for resource-constrained deployment.
Takeaways & Limitations
Angular bias between activations and their binarizations remains an open problem, and rotating activations may add inference computation.
Abstract
from arXiv · showhide
Binary Neural Network (BNN) shows its predominance in reducing the complexity of deep neural networks. However, it suffers severe performance degradation. One of the major impediments is the large quantization error between the full-precision weight vector and its binary vector. Previous works focus on compensating for the norm gap while leaving the angular bias hardly touched. In this paper, for the first time, we explore the influence of angular bias on the quantization error and then introduce a Rotated Binary Neural Network (RBNN), which considers the angle alignment between the full-precision weight vector and its binarized version. At the beginning of each training epoch, we propose to rotate the full-precision weight vector to its binary vector to reduce the angular bias. To avoid the high complexity of learning a large rotation matrix, we further introduce a bi-rotation formulation that learns two smaller rotation matrices. In the training stage, we devise an adjustable rotated weight vector for binarization to escape the potential local optimum. Our rotation leads to around 50% weight flips which maximize the information gain. Finally, we propose a training-aware approximation of the sign function for the gradient backward. Experiments on CIFAR-10 and ImageNet demonstrate the superiorities of RBNN over many state-of-the-arts. Our source code, experimental settings, training logs and binary models are available at https://github.com/lmbxmu/RBNN.
1 Introduction
BNNs reduce model size and computation but retain a substantial accuracy gap because binary weights incur quantization error from both norm mismatch and angular bias. RBNN addresses angular bias by rotating weights toward binary-hypercube vertices, using smaller bi-rotation matrices and dynamically adjusted weights.
- BNNs restrict weights and activations to −1 and +1, reducing network size by 32× compared with full-precision networks.
- The remaining accuracy gap is driven in part by quantization error between full-precision weights and their binary vectors.
- A scaling factor reduces the norm gap but cannot remove angular-bias error, whose lower bound is ∥w sin θ∥2.
- RBNN targets information loss from limited weight flips by using rotation to increase flipping during training.
- RBNN rotates full-precision weights toward binary-hypercube vertices at each epoch to reduce angular bias and quantization error.
- Bi-rotation replaces a large rotation matrix with two smaller matrices, while dynamically adjusted rotated weights help avoid potential local optima during binarization.
2 Related Work
Prior BNN research established sign-based binarization and gradient estimation, then pursued accuracy improvements through scaling, variable precision, architectural changes, and differentiable sign approximations.
- Early BNNs binarized weights and activations to −1 or +1 with the sign function and used the straight-through estimator for backpropagation.
- XNOR-Net uses per-channel scaling from weight and activation ℓ1-norms to reduce quantization error.
- DoReFa-Net varies quantization bit-widths for weights, activations, and gradients, using layer-wise scaling for efficient inference.
- Recent methods expand BNN expressive capacity with parallel binary layers, residual shortcuts, and differentiable sign-function approximations.
3 Approach
RBNN reduces BNN quantization error by aligning full-precision weights with their binary vectors through epoch-wise rotation, then uses bi-rotation, adjustable weights, and training-aware gradients for efficient optimization.
- 3.1 Binary Neural Networks: Angular bias between full-precision and binary weights creates irreducible quantization error and reduces information gain through consistent signs.The paper motivates reducing angular bias while increasing the probability of weight flips.
- 3.2 Rotated Binary Neural Networks: RBNN rotates each full-precision weight vector toward its binary hypercube vertex at the beginning of every training epoch.The rotation narrows the angle between the rotated weight vector and its binary vector.
- 3.2 Rotated Binary Neural Networks: Bi-rotation reconstructs a large orthogonal rotation using two smaller orthogonal matrices formed through a Kronecker product.This reduces the large matrix’s memory and computation burden; the matrices are optimized alternately.
- 3.3 Optimization: Alternating optimization updates the binarization and the two rotation matrices, reaching convergence after three update cycles.The rotation matrices are updated using SVD-based rules.
- 3.4 Adjustable Rotated Weight Vector: An adjustable rotated weight vector addresses overshooting or undershooting local optima by moving along the residual direction with α_i constrained to [0, 1].The paper reports that overshooting is dominant and that this constraint further reduces quantization error and improves performance.
- 3.5 Gradient Approximation: RBNN replaces the sign function’s backward gradient with a training-aware approximation whose gradient is broad early and becomes sign-like as training progresses.This enables whole-network updates early while preserving the binary property later.
4 Experiments
Experiments on CIFAR-10 and ImageNet show that RBNN consistently outperforms compared binary-network baselines. Ablations and weight analyses support the contributions of its gradient approximation, rotation, adjustable scheme, and increased weight flipping.
- CIFAR-10 and ImageNet: RBNN outperforms the compared state-of-the-art binary networks across CIFAR-10 models and ImageNet ResNet-18/34.On CIFAR-10, it improves over the best baseline by 0.7%, 1.1%, and 0.9% for the reported ResNet-18, ResNet-20, and VGG-small settings; on ImageNet, it leads in both top-1 and top-5 accuracy.
- ImageNet: 59.9% top-1 and 81.9% top-5 accuracy are achieved by RBNN on ImageNet with ResNet-18, improving over IR-Net by 1.8% and 1.9%.With ResNet-34, RBNN reaches 63.1% top-1 and 84.4% top-5 accuracy, improving over IR-Net by 0.2% and 0.3%.
- Gradient approximation: 1.8% improvement over EDE validates the effectiveness of RBNN's training-aware gradient approximation in the reported CIFAR-10 experiment.The comparison also evaluates straight-through estimation and piecewise polynomial approximation.
- Ablation study: 87.8% accuracy is obtained when binarization, training-aware approximation, weight rotation, and the adjustable scheme are combined.Starting from XNOR-Net binarization, adding rotation or training-aware approximation reaches 86.4% or 86.6%, while their combination reaches 87.1%.
- Weight distribution: RBNN produces two-mode pre-binarization weight distributions centered near −1 and +1, unlike XNOR-Net's tightly zero-centered weights.The distribution produced by RBNN is described as reducing quantization error by placing weights near the binary values.
- Weight flips: Around 50% weight flips occur in each ResNet-20 layer with RBNN, whereas XNOR-Net exhibits a small flipping rate.The paper associates the higher flip rate with maximizing information gain during training.
5 Conclusion
The paper analyzes angular bias in BNN quantization error and proposes RBNN to align rotated weights with binary vectors. It combines bi-rotation, adjustable training weights, and a training-aware sign approximation, with experiments showing reduced quantization error and superior performance over several state-of-the-art methods.
- Conclusion: RBNN reduces angular bias by aligning rotated full-precision weights with binary vectors at the beginning of each training epoch.The method uses two smaller rotation matrices, dynamically adjusts the rotated weight vector, and achieves around 50% weight flips.
- Conclusion: RBNN combines bi-rotation, adjustable rotated weights, and a training-aware sign approximation to reduce quantization error and support gradient propagation.The bi-rotation scheme reduces the complexity of learning a large rotation matrix.
- Conclusion: Experiments on CIFAR-10 and ImageNet demonstrate RBNN's efficacy in reducing quantization error and outperforming several state-of-the-art methods.
Broader Impact
RBNN addresses an overlooked source of quantization error and is intended to support deployment of quantized models on resource-limited devices, while activation angular bias remains unresolved.
- RBNN reduces quantization error by targeting angular bias, which previous work largely ignored.
- The proposed approach is intended to help deploy quantized deep models on resource-constrained devices such as mobile phones.
- Angular bias between activations and their binarized versions remains an open problem, and rotating activations may add inference computation.
- The paper states that RBNN causes fewer accuracy drops than other state-of-the-art methods.
- RBNN is described as unrelated to data selection and therefore not subject to data bias from selection.