Source-linked AI summary
Local Binary Convolutional Neural Networks
Felix Juefei-Xu, Vishnu Naresh Boddeti, Marios Savvides
TL;DR
Standard CNNs are computationally expensive, memory-intensive, and vulnerable to over-fitting with limited data, creating a need for efficient models on resource-constrained platforms. The paper introduces LBC layers using fixed sparse binary filters and learnable linear weights, and reports comparable performance to regular CNNs across visual datasets with substantial parameter and computational savings.
Problem
Standard CNNs are costly to train and deploy, while fully binarized networks can lose performance relative to real-valued networks.
Method
LBC layers approximate standard convolutional activations using fixed sparse binary filters, a nonlinear activation, and learnable linear weights.
Results
LBCNNs perform comparably to regular CNNs across MNIST, SVHN, CIFAR-10, and ImageNet while reducing parameters and computational and memory requirements.
Takeaways & Limitations
LBCNNs are suited to low-sample-complexity learning and learning or inference on resource-constrained environments.
Takeaways & Limitations
The paper defines sparsity level as the percentage of non-zero elements, with 100% sparsity corresponding to a dense weight tensor.
Abstract
from arXiv · showhide
We propose local binary convolution (LBC), an efficient alternative to convolutional layers in standard convolutional neural networks (CNN). The design principles of LBC are motivated by local binary patterns (LBP). The LBC layer comprises of a set of fixed sparse pre-defined binary convolutional filters that are not updated during the training process, a non-linear activation function and a set of learnable linear weights. The linear weights combine the activated filter responses to approximate the corresponding activated filter responses of a standard convolutional layer. The LBC layer affords significant parameter savings, 9x to 169x in the number of learnable parameters compared to a standard convolutional layer. Furthermore, the sparse and binary nature of the weights also results in up to 9x to 169x savings in model size compared to a standard convolutional layer. We demonstrate both theoretically and experimentally that our local binary convolution layer is a good approximation of a standard convolutional layer. Empirically, CNNs with LBC layers, called local binary convolutional neural networks (LBCNN), achieves performance parity with regular CNNs on a range of visual datasets (MNIST, SVHN, CIFAR-10, and ImageNet) while enjoying significant computational savings.
1. Introduction
Standard CNNs are effective but costly to train and deploy, motivating LBCNNs, which replace learned convolution filters with fixed sparse binary filters plus learnable linear weights. The paper reports comparable performance to regular CNNs across visual datasets while reducing parameters and computational or memory demands.
- Standard CNNs require expensive training, large memory and disk footprints, and may over-fit when data are limited.
- Complete CNN binarization improves computational efficiency but can lose performance relative to networks with real-valued weights.
- LBC layers use fixed sparse binary filters, nonlinear activation, and learnable linear weights instead of fully learnable convolutional filters.The learnable weights combine activated filter-response maps.
- 9× to 169× parameter savings are achieved for convolution filters ranging from 3 × 3 to 13 × 13.
- LBCNNs are theoretically good approximations of standard convolutional activations and perform comparably to regular CNNs on MNIST, SVHN, CIFAR-10, and ImageNet.The reported comparison also includes significant parameter savings during training.
- LBCNNs combine fixed and learnable weights, distinguishing them from fully binarized networks while retaining sparsity and binarization efficiencies.
2. Forming LBP with Convolutional Filters
The paper reformulates local binary patterns as convolutional operations: sparse difference filters produce binarized response maps, which are then linearly combined. Varying weights, pivots, and neighbor ordering makes the formulation more flexible than a fixed LBP descriptor.
- Traditional LBP compares neighboring pixels with a central pivot, thresholds the differences into bits, and combines them into a descriptor.Neighborhoods such as 3 × 3 and 5 × 5 patches produce binary encodings.
- LBP descriptors can vary through the choice of base, pivot, and ordering of neighboring pixels.These choices change the resulting local-texture representation.
- Eight 2-sparse 3 × 3 difference filters followed by binarization reproduce the LBP bit maps using convolution.
- A weighted sum of the activated filter responses reconstructs the LBP image, with weights controlling the encoding base and ordering.The formulation is y = sum_i σ(b_i ∗ x) · v_i, where σ is the Heaviside binarization operator.
- Changing the nonzero +1 and -1 support in the filters changes the pivot used by the convolutional LBP formulation.
3. LBCNN
LBCNN replaces standard convolution with fixed sparse binary filters, nonlinear activation, and learnable linear combinations, motivated by LBP. Theoretical analysis supports approximation under stated conditions, while the design reduces parameters and supports efficient training and computation.
- Local Binary Convolution Module: LBP reformulation uses convolutional filters, binary thresholding, and weighted pooling, motivating the LBC layer’s CNN-compatible structure.The reformulated pipeline filters inputs, applies a nonlinear operation, and linearly combines resulting bit maps.
- Local Binary Convolution Module: Each LBC layer filters its input with m fixed anchor filters, applies nonlinear activation, and combines the resulting maps with learnable 1×1 convolution weights.The fixed filters are non-learnable, whereas the second convolutional layer is learnable.
- Local Binary Convolution Module: p · h · w · q learnable parameters are used by a standard layer, whereas LBC uses fixed spatial filters and learnable linear weights.The parameter ratio simplifies to h · w when p = m; the analysis assumes filters have no bias terms.
- Local Binary Convolution Module: 9×, 25×, 49×, 81×, 121×, and 169× parameter savings arise for 3×3, 5×5, 7×7, 9×9, 11×11, and 13×13 filters.These savings are reported during learning relative to standard convolutional layers.
- Learning with LBC Layers: Only the learnable 1×1 filters are updated; fixed anchor weights remain unchanged and can be generated deterministically or stochastically.The experiments use randomly assigned ±1 nonzero anchor weights at a selected sparsity level.
- Theoretical Analysis: Theorem 3.5 states that Bernoulli filters yield positive elements in Bx with high probability, supporting ReLU approximation when d > 0.For d = 0, a vector v can always achieve d′ = d; for d > 0, approximation requires positive activated responses.
- Theoretical Analysis: For τ image patches, a shared vector v combines intermediate bit maps, and using more intermediate filters m can improve approximation.When τ ≤ m, the analysis states that v can be solved so that the LBC output matches the target under the described construction.
4. Experimental Results
Experiments compare LBCNN with matched CNN baselines across visual classification datasets and ImageNet settings. LBCNN achieves comparable accuracy while using substantially fewer learnable parameters and supporting shared weights.
- 512 randomly generated anchor weights with sparsity values of 0.1, 0.5, or 0.9 are used across the experiments.
- LBCNN and CNN use the same architecture, data, filter counts, and learning-rate settings, differing in fixed sparse binary versus dense learnable weights.
- Figure 5 reports CIFAR-10 accuracy for LBCNN variants and accuracy and loss for full ImageNet classification.
- 10× fewer learnable parameters are used by LBCNN with 3 × 3 kernels in the matched baseline experiments.
- LBCNN models are compared with CNN baselines and other state-of-the-art methods on MNIST, SVHN, CIFAR-10, and ImageNet classification.
- Shared convolutional weights reduce model size by roughly a factor of D for D LBC layers while retaining comparable performance to unshared LBCNN.
- 6.622× fewer learnable convolutional parameters are achieved by AlexNet with LBCNN modules while performing comparably to AlexNet.
5. Discussion
The discussion identifies computational, memory, and statistical benefits of LBC layers. Fixed sparse binary filters reduce complexity and parameters, while lower model complexity is associated with faster convergence and less overfitting in limited-data settings.
- Computational: 9× to 169× fewer learnable parameters are required during training and inference with LBC layers.
- Computational: Sparse binary filters further reduce computational complexity and memory and space requirements during both training and inference.
- Computational: Sharing convolutional weights across LBC layers further reduces memory requirements for deep CNNs on resource-constrained embedded systems.
- Statistical: LBCNN's fewer learnable parameters can regularize learning and prevent overfitting relative to higher-capacity CNNs.
- Statistical: Randomly generated sparse Bernoulli filters provide activation decorrelation, with smaller normalized correlation indicating greater decorrelation.
- Sample Complexity: On reduced-sample CIFAR-10 and FRGC experiments, LBCNN trains or converges faster, is less prone to overfitting, and outperforms CNN on the FRGC task.
6. Conclusions
The paper concludes that LBC layers approximate standard convolutional layers while substantially reducing learnable parameters and computational complexity. LBCNNs perform as well as standard CNNs across multiple small- and large-scale visual datasets.
- LBC layers use fixed sparse binary convolutional weights plus learnable linear weights as an alternative to standard CNN convolutional layers.
- 9× to 169× fewer parameters are learned for 3 × 3 and 13 × 13 filters, respectively.
- LBC modules are shown theoretically and empirically to approximate standard convolutional layers.
- LBCNNs perform as well as standard CNNs across multiple small- and large-scale visual datasets and network architectures.