Source-linked AI summary
Regularizing Activation Distribution for Training Binarized Deep Networks
Ruizhou Ding, Ting-Wu Chin, Zeye Liu, Diana Marculescu
TL;DR
BNNs offer low-latency, low-energy inference, but degeneration, saturation, and gradient mismatch make their 1-bit training difficult, while prior remedies sacrifice some efficiency. The paper introduces a distribution loss that explicitly regularizes activation flow through a systematic formulation framework. Experiments report consistent accuracy improvements without losing energy benefits and greater robustness to training hyper-parameter selection.
Problem
BNN training is difficult because unregularized activation flow can develop degeneration, saturation, and gradient mismatch, while prior remedies use higher-bit activations or floating-point scaling factors.
Method
The paper embeds an explicitly formulated distribution loss in the objective to regularize activation distributions during BNN training.
Results
36.1% to 41.3% Top-1 accuracy is reported for BNN AlexNet on ImageNet, and distribution-loss training consistently improves BNN accuracy across evaluated settings.
Takeaways & Limitations
The proposed regularization improves binarized-network accuracy while retaining their energy benefits and making training more robust to optimizer and learning-rate choices.
Abstract
from arXiv · showhide
Binarized Neural Networks (BNNs) can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN's energy efficiency. In this paper, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.
1. Introduction
BNNs reduce computation and memory costs through 1-bit weights and activations, but their bounded activation flow makes accurate training difficult. The paper explicitly regularizes activation distributions with a distribution loss, improving accuracy while retaining binarization benefits and robustness to hyper-parameter choices.
- Training challenges: Bounded Sign activations and HardTanh-based gradient approximations create degeneration, saturation, and gradient mismatch challenges during BNN training.Degeneration occurs when nearly all pre-activations in a channel share the same sign, producing nearly constant activations.
- Prior approaches: Prior approaches improve accuracy by widening networks, increasing parameters, or using multiple activation bits and floating-point scaling factors.These strategies trade some of BNNs’ extremely-low energy or pure-logical computation benefits for accuracy.
- Proposed approach: The proposed framework embeds explicit activation-distribution regularization in the loss function to address BNN training difficulties.The authors distinguish this explicit loss-based shaping from implicit regularization through initialization or batch normalization.
- Results: 36.1% to 41.3% Top-1 accuracy is reported for BNN AlexNet on ImageNet, while binarized wide AlexNet improves from 48.3% to 53.8%.The improvements are reported after adding the distribution loss.
- Results: Distribution-loss training is reported to be robust to optimizer selection across Momentum, Nesterov, Adam, and RMSprop.The experiments use CIFAR-10 and report consistent improvement across these optimizers.
2. Related Work
Prior activation-regularization methods were developed mainly for full-precision networks and do not resolve the distinctive difficulties of binarized activations. Binarized-network methods instead commonly use higher-bit activations, scaling factors, or related approximations.
- Full-precision networks: Full-precision DNNs use normalization and reparameterization methods to regularize activations and address gradient vanishing or exploding problems.Examples include batch normalization, weight normalization, normalized propagation, layer-wise normalization, and recurrent-network weight reparameterization.
- Binarized networks: These full-precision activation-regularization approaches do not address the difficulty of training networks with binarized activations.The related-work discussion separates full-precision normalization methods from BNN-specific solutions.
- Binarized networks: BNN research commonly approximates full-precision activations with multiple-bit representations and floating-point scaling factors.Examples include scaling layers, multi-level activation functions, and weighted sums of multiple binary tensors.
3. Activation Regularization
The paper analyzes how bounded activations create degeneration, saturation, and gradient mismatch in BNN training, then proposes explicit distribution regularization. Its framework formulates robust, differentiable losses from these hypotheses and adds them to the training objective.
- Binarized DNNs: BNN inference can use pure-logical operators because binary weights and activations are reduced to logical convolution operations.The BNN block combines binary convolution with batch normalization, pooling, and sign activation while preserving the input-output mapping.
- Binarized DNNs: XNOR-Net and ABC-Net require full-precision multiplications and additions through input-dependent scaling or linear combinations of binary tensors.These operations preserve approximation flexibility but remove the pure-logical advantage described for BNNs.
- Regularizing Activation Distribution: The framework defines activation-flow difficulties using pre-activation tensors and quantiles, including degeneration, saturation, and gradient mismatch.For each batch, A denotes pre-activations before Sign, while A(q) denotes the q quantile of their elements.
- Regularizing Activation Distribution: Direct degeneration regularization based on minimum and maximum pre-activations is vulnerable to outliers and is not differentiable with respect to pre-activations.A small outlier can make the loss zero, producing overly loose regularization.
- Regularizing Activation Distribution: The proposed three-stage framework formulates hypothesis-based losses, replaces high-variance extrema with quantiles, and enables differentiability.It derives LD, LS, and LM for degeneration, saturation, and gradient mismatch, respectively.
- Regularizing Activation Distribution: The distribution loss is added to cross-entropy as Lb total = Lb CE + λLb DL, with λ balancing the two losses.Regularizing activations is intended to maintain effective backpropagation and improve training accuracy without changing the BNN's pure-logical inference mechanism.
4. Experimental Results
Across CIFAR-10, SVHN, CIFAR-100, and ImageNet, distribution loss improves BNN accuracy while preserving binary inference benefits. It regularizes activation and gradient flow, reduces degeneration, and makes training less sensitive to hyper-parameter choices.
- Accuracy improvement: Distribution loss consistently improves BNN accuracy across CIFAR-10, SVHN, CIFAR-100, and ImageNet experiments.The CIFAR-10, SVHN, and CIFAR-100 improvements are reported as statistically significant, while ImageNet results outperform the baseline BNN and several prior methods.
- Accuracy improvement: In most cases, a smaller BNN-DL model achieves the same or higher accuracy than a larger baseline BNN.The comparison is presented jointly in terms of testing accuracy and computation energy, with memory-access energy excluded because it is the same for BNN and BNN-DL during inference.
- Training behavior: Distribution loss improves testing accuracy by regularizing activation and gradient flow during training.Training loss becomes consistently lower than the baseline after a few epochs, and the distribution loss often decays to about 1/10000 of its initial value within the first few epochs.
- Activation analysis: Activation statistics under BNN-DL are more centralized, with fewer channels exhibiting very small standard deviations or extreme positive ratios.These statistics indicate reduced activation degeneration, although the positive ratio also deviates from 0.5 in BNN-DL.
- Robustness to hyper-parameter selection: BNN-DL is robust to optimizer, learning-rate, network-structure, and regularization-level choices.The reported robustness covers SGD with momentum, Nesterov SGD, Adam, RMSprop, VGG-style networks, ResNet-18, and λ values ranging from 0.2 to 2000.
5. Conclusion
The paper addresses training difficulties in 1-bit BNNs by explicitly regularizing activation distributions through a distribution loss. Experiments report improved accuracy, stronger state-of-the-art performance, and robustness to training hyper-parameters without losing energy-efficient inference.
- Distribution loss explicitly regularizes activation distributions associated with degeneration, saturation, and gradient mismatch during BNN training.
- BNNs trained with distribution loss consistently outperform baseline BNNs.
- The approach significantly improves state-of-the-art 1-bit BNN accuracy for AlexNet on ImageNet.
- Distribution-loss training is robust to learning-rate and optimizer selection.
- The method retains latency- and energy-efficient inference with 1-bit weights and activations.
6. Appendix
The appendix describes basic blocks for XNOR-Net, ABC-Net, and ResNet-based BNNs, and specifies architectures and energy-estimation settings used in evaluations.
- Energy estimation: Energy estimation uses a VGG-16 ImageNet convolutional layer with 256 input and output channels, 56x56 feature maps, and 3x3 stride-1 kernels.The calculation compares operation counts across BNN, XNOR-Net, and ABC-Net binarization methods.
- Energy estimation: The energy calculation counts XNORs, counts, fixed-point multiplications, and additions for the compared binarization methods.These operation counts are used to estimate computational energy consumption.
- Basic blocks: Figures 9–11 show the basic blocks for XNOR-Net, ABC-Net, and ResNet-based BNNs.The appendix identifies the corresponding block diagrams for each architecture.
- CIFAR-100 architecture: The CIFAR-100 network uses convolutional layers, basic blocks, global pooling, and a 100-output linear layer in the structure xC-xB-xB-2xB-2xB-4xB-4xB-8xB-8xB-GP-100L.All convolutional layers use 3×3 filters; selected block-entry convolutions use stride 2 for feature-map reduction.
- CIFAR-100 architecture: In the CIFAR-100 architecture, the first convolutional layers of the second, third, and fourth blocks use stride 2, while other convolutional layers use stride 1.This specifies where spatial downsampling occurs in the network.