Source-linked AI summary
Pruning Binarized Neural Networks: A Dedicated Framework and Globally Weighted Algorithms
Roan Rubiales, Jean Pierre David
TL;DR
Edge deployment is constrained by deep-learning computation, and joint pruning and binarization lack adequate framework support. The paper introduces a PyTorch framework and globally weighted pruning methods for BNNs, reporting stronger pruning–accuracy trade-offs while identifying hardware-implementation boundaries.
Problem
Joint pruning and binarization are poorly supported by mainstream frameworks, while existing pruning strategies are challenging to apply to BNNs.
Method
The paper develops a PyTorch framework for BNN training, pruning, and freezing, and uses globally weighted latent parameters to guide binary-aware pruning.
Results
The framework enables systematic benchmarking, and the proposed pruning methods surpass state-of-the-art results for several models with favorable pruning–accuracy trade-offs.
Takeaways & Limitations
The framework provides a foundation for evaluating and prototyping pruning and binarization methods for resource-constrained edge deployments.
Takeaways & Limitations
The framework does not provide inference-ready implementations for microcontrollers or FPGAs, and sparse models make some hardware optimizations difficult to estimate.
Abstract
from arXiv · showhide
Extreme compression of deep neural networks, up to full binarization, dramatically reduces memory footprint and arithmetic complexity, facilitating deployment on constrained edge hardware with field-programmable gate arrays (FPGAs) and microcontrollers. Although combining binarization with pruning promises additional efficiency gains, existing pruning strategies are ill-suited to binarized representations and rarely translate into meaningful hardware savings. We introduce a PyTorch-based, research-oriented framework that incorporates freezing and pruning mechanisms for designing and optimizing binarized neural networks. The framework enables rapid and reproducible evaluation of state-of-the-art approaches and the fast prototyping of new ones. Leveraging this framework, we propose a novel pruning method that accounts for the relative importance of learned parameters across abstraction levels. Such a global weighting mechanism consistently achieves a superior trade-off between model accuracy and pruning rate, achieving a 70% pruning rate on VGG11 with constant accuracy, while state-of-the-art results reach only 41% in the binarized setting.
1 Introduction
Deep-learning models demand substantial computation and power, limiting edge deployment; binarization and pruning address these constraints, but their combination remains difficult to support effectively.
- High floating-point operation counts increase inference power consumption and limit deployment on edge platforms.The paper highlights robotics, medical implants, and AIoT devices as affected domains.
- Quantization, pruning, and lightweight architectures reduce model size and computation, but complex models can still exceed efficient FPGA synthesis limits.Floating-point MACs are especially costly, while even integer and fixed-point arithmetic may leave too many operations.
- 1bit binarization constrains weights and activations to binary precision, minimizing arithmetic hardware cost.The paper presents binarization as an extreme form of quantization for efficient implementations.
- Pruning BNNs is challenging because all binarized weights have identical magnitude, yet adapted strategies can improve compression without reducing accuracy.Prior work also reports a potential regularization effect that can reduce generalization error.
- Joint pruning and binarization remain poorly supported by mainstream deep-learning frameworks.The paper identifies this lack of support as a substantial research and development challenge.
- The paper contributes a PyTorch framework, a global weighting mechanism, and three binary-aware pruning algorithms.The algorithms target conversion and optimization of full-precision pretrained networks into binary models.
2 Related works
Related work establishes BNNs and pruning as complementary efficiency techniques, while emphasizing that binary pruning must operate on latent weights and account for hardware execution patterns.
- BNNs quantize weights to 1 bit, typically using −1 and 1, while latent real-valued weights support training.Binary-aware training dynamically binarizes latent weights during optimization.
- BNN research commonly modifies established architectures, activations, building blocks, and layer ordering to accommodate binary training.These architectures target highly resource-constrained edge devices such as FPGAs and microcontrollers.
- Pruning removes weights to reduce model size and computational cost, with structured pruning generally producing more tangible deployment savings than unstructured pruning.Hardware is often not optimized for sparse computation, limiting the practical impact of individually removed weights.
- BNN pruning methods mainly evaluate latent weights by magnitude or sign-oscillation frequency because binarized weights have identical amplitudes.This shifts pruning decisions from the binary weights to their real-valued latent counterparts.
- Magnitude pruning removes small absolute weights using thresholds, but one global threshold can be ineffective when channel scales differ substantially.Weights across channels may vary by several orders of magnitude.
- Freezing algorithms dynamically select subnetworks during training using metrics such as cosine similarity between channel outputs at successive times.In this setting, freezing also serves as regularization and early stopping until only biases remain updated.
3 Proposed Binary Neural Network framework
The proposed torch-based framework automates experiments and supports customizable BNN architectures, pruning, and freezing, enabling systematic validation and exploration of pruning variants.
- The framework automates cross-experiments and comparisons of architectures, pruning methods, and freezing algorithms across parameter settings.It is implemented in the torch Python library.
- BNN architectures, pruning algorithms, and freezing algorithms are directly customizable in code, while other experiment parameters are configured through TOML files.Configuration files specify items such as pretrained weights and architecture hyperparameters.
- A foundational base class supports BNN modules built from binary layers and sign activations, while custom modules can extend the same interface.Standard PyTorch modules may be integrated but are not binarized by the framework.
- Existing pruning methods were reimplemented with near-original results, validating the framework as a reasonable benchmarking tool.The framework was then used to explore algorithm variants affecting accuracy and compression rate.
4 Proposed pruning algorithms
The pruning algorithms use globally weighted latent weights and pre-binarization pruning to select thresholds across layers, channels, or the full network while targeting favorable accuracy–compression trade-offs.
- Global Weighting for pruning: Global weighting transforms copied weights so channels share a similar scale, leaving within-channel or within-layer differences to guide threshold selection.The mechanism addresses unequal parameter magnitudes across abstraction levels.
- Global Weighting for pruning: Batch Normalization Folding integrates normalization parameters into the preceding layer before pruning-related transformations.The folded quantities include weights, bias, expected value, and standard deviation.
- Global Weighting for pruning: Weight Normalization scales tensors by a p-norm along layers or channels, with layer-wise normalization used after batch-normalization folding.Channel-wise normalization would cancel the folding effect on weights.
- Global Weighting for pruning: Global weights determine thresholds, while corresponding original weights are pruned, enabling threshold orderings different from naive magnitude orderings.This mechanism is intended to support higher pruning rates with similar post-pruning accuracy.
- Pre-binarization pruning: Pruning occurs after activation-only pretraining and before full binarization, followed by binarization and fine-tuning.The setup is expected to allow more accuracy recovery than pruning after binarization.
- Single threshold magnitude pruning: The single-threshold search algorithm finds one network-wide threshold using sorted global weights and validation accuracy.Its threshold directly corresponds to the model’s pruning ratio when global weights are used.
- Single threshold magnitude pruning: The sweep variant evaluates validation metrics across evenly spaced pruning ratios and leaves final threshold selection to the resulting accuracy or loss landscape.It samples ratios across [0%, 100%].
- Accuracy-gradient pruning: Accuracy-gradient pruning estimates threshold effects with finite differences and first increases thresholds with positive accuracy derivatives.It later raises the threshold with the least negative derivative until the target accuracy is reached.
5 Experiments
Experiments evaluate pruning and global weighting across BinaryNet, VGG11, and NiN using CIFAR-10 and a two-stage binarization protocol. Global weighting improves pruning–accuracy trade-offs, while model-size reductions do not directly predict operational speedups.
- Architectures and dataset: Experiments compare pruning methods on BinaryNet, VGG11, and NiN, with CIFAR-10 as the primary dataset.The framework also supports other datasets, while validation uses a held-out subset of the training data.
- Dataset and training procedure: Training uses real-valued weights with binarized activations during 300-epoch pretraining, followed by pruning and 100 epochs of fine-tuning.Channel freezing is applied when binary fine-tuning diverges or when more than 95% of weights are frozen.
- Comparison of global weighting transformations: Global weighting lowers validation loss relative to magnitude-only pruning, with the clearest improvement on BinaryNet and thinner gains on VGG11 and NiN.The comparison uses single-threshold sweeps with transformations including BatchNorm folding.
- Comparison of global weighting transformations: 65% pruning produces a 0.37% accuracy decrease with Batch Normalization Folding, while global weighting outperforms magnitude-only pruning at every tested rate.The reported result persists through binarization and fine-tuning.
- Comparison of different pruning methods: 45.0% pruning preserves 89.7% BinaryNet accuracy, and Algorithm 2 reaches 70.0% pruning on VGG11 with almost constant accuracy.The BinaryNet pruning ratio is reported as 1.66 times that of Chen et al.’s layer-sensitivity method.
- Estimated hardware gains: VGG11 becomes 3.3× smaller but achieves only 1.4× maximum speedup because pruning is concentrated in fully connected layers.The framework reports model size and binary multiply-accumulate counts but does not provide inference-ready implementations for target devices.
6 Discussion
The framework supports pruning and binarization while benchmarking methods that achieve favorable pruning–accuracy trade-offs. However, unstructured sparsity and layerwise pruning distributions limit how directly model-size reductions translate into hardware speedups.
- The framework supports both pruning and binarization, enabling systematic benchmarking and results surpassing the state of the art for several models.Its relevance is especially emphasized for edge deployments on microcontrollers and FPGAs.
- The proposed methods, especially single-threshold algorithms, achieve better pruning results than other methods.The authors attribute these outcomes to global weighting and pre-binarization pruning.
- Binary MACs are easier to optimize on many architectures, including CPUs using XNOR-Count operations and FPGAs using pipelining.These optimizations are relevant to the hardware-oriented evaluation of binarized models.
- Highly sparse models can make hardware optimizations less efficient in ways that are difficult to estimate.The resulting hardware benefit therefore cannot be inferred directly from binary operation counts.
7 Conclusions and future work
The paper presents a preliminary framework for binarization, pruning, and freezing, alongside pruning methods that improve reported pruning rates without compromising precision. It identifies structured pruning, deeper ablations, and automatic code generation as future directions.
- The framework provides a foundation for implementing binarized architectures and pruning or freezing algorithms.The authors validate it by reproducing compact binarized CNNs and popular BNN pruning methods.
- Pruning rates reach up to twice those reported in the literature without compromising model precision.The proposed methods are based on an accuracy-gradient ascent.
- The pruning methods would benefit from conversion to structured pruning and a more in-depth ablation study.The ablation study is intended to assess the effects of global weighting and pre-binarization pruning.
- Automatic code generation for on-device implementations is left for future work.This would complete the deployment pipeline for models pruned by the proposed methods.