PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Connection Between Binary and Spiking Neural Networks

Sen Lu, Abhronil Sengupta

arXiv:2002.10064v3cs.LGstat.ML

TL;DR

Edge intelligence needs accurate models with reduced compute demands, while the connection between binary and spiking networks remained underexplored. The paper converts standard trained networks to spiking networks and analyzes binary-weight variants and latency optimizations. It reports near-full-precision accuracy on large-scale image-recognition datasets and lower inference latency through simple optimizations.

  • Problem

    The paper examines the underexplored potential of binary-weight SNNs to combine binary-network hardware with high accuracy, given BNN accuracy loss and SNN training challenges.

    Source: §I · p. 2

  • Method

    The authors train non-spiking networks with binary weights and convert them to SNNs, then evaluate design-time and run-time latency optimizations.

    Source: §II · p. 2, A. Datasets and Implementation · p. 4

  • Results

    Binary SNNs achieve near-full-precision accuracy on large-scale image-recognition datasets; on ImageNet, top-1 SNN (ANN) accuracy is 62.71% (65.4%) for the binarized model and 66.56% (69.05%) for the full-precision model.

    Source: §V · p. 8, E. ImageNet Results · p. 8

  • Takeaways & Limitations

    B-SNNs can use a hardware backbone designed for BNN-catered in-memory computing, with inference time-steps adjustable to application accuracy requirements.

    Source: §II · p. 2, §V · p. 8

    SNN energy efficiency involves a trade-off between power benefits and latency overhead, which depends on the architectural options.

    Source: 2) Neural Node Options · p. 6

Abstract

from arXiv · show

On-chip edge intelligence has necessitated the exploration of algorithmic techniques to reduce the compute requirements of current machine learning frameworks. This work aims to bridge the recent algorithmic progress in training Binary Neural Networks and Spiking Neural Networks - both of which are driven by the same motivation and yet synergies between the two have not been fully explored. We show that training Spiking Neural Networks in the extreme quantization regime results in near full precision accuracies on large-scale datasets like CIFAR-$100$ and ImageNet. An important implication of this work is that Binary Spiking Neural Networks can be enabled by "In-Memory" hardware accelerators catered for Binary Neural Networks without suffering any accuracy degradation due to binarization. We utilize standard training techniques for non-spiking networks to generate our spiking networks by conversion process and also perform an extensive empirical analysis and explore simple design-time and run-time optimization techniques for reducing inference latency of spiking networks (both for binary and full-precision models) by an order of magnitude over prior work.

I. INTRODUCTION

Edge intelligence motivates models that reduce storage and computation while preserving accuracy on constrained devices. This work examines whether binary-output spiking networks can combine the hardware advantages of binary networks with accuracy supported by temporal information integration.

I. INTRODUCTION

  • Edge devices need accurate machine-learning platforms with low storage and compute demands for real-time, privacy-sensitive on-device processing.The motivating settings include mobile phones, wearables, smart sensors, and robotic devices with limited connectivity and battery life.Source: §I · p. 1
  • Binary Neural Networks (BNNs) use binary neuron activations and synaptic weights, enabling simpler XNOR and Pop-Count operations and specialized in-memory hardware.A cited example reports 58× less computation time and 32× smaller model size than a corresponding full-precision model.Source: §I · p. 1
  • Spiking Neural Networks (SNNs) use binary spike outputs and event-driven computation, while ANN-to-SNN conversion has enabled competitive ImageNet recognition.The cited hardware examples include IBM TrueNorth and Intel Loihi; the conversion work is cited as,.Source: §I · p. 1
  • Whether SNNs can use binary weights on large-scale tasks had not been empirically explored, despite the near-equivalent binary state representations of BNNs and SNNs.The SNN additionally operates over multiple time-steps; neuron states are −1, +1 in BNNs and 0, 1 in SNNs [p10].Source: §I · pp. 1–2
  • The paper investigates binary-weight SNNs as a way to bridge BNN accuracy loss, drawing on the approximation of ReLU-like integration over time.The authors propose that this connection could support BNN-catered in-memory accelerators without compromising recognition accuracy.Source: §I · p. 2

II. RELATED WORK & MAIN CONTRIBUTIONS

The paper positions binary-weight spiking networks as complementary to mixed-precision approaches and as distinct from earlier shallow binary-weight SNN work. It proposes a B-SNN approach and describes the binary-network training elements it uses.

II. RELATED WORK & MAIN CONTRIBUTIONS

  • Mixed-precision networks can require hardware that handles variable layer precision, whereas this work targets BNN-customized hardware and time-domain encoding.The authors contrast encoding information over time in SNNs with encoding it across space in mixed-precision networks; the passage also discusses potential power and hardware trade-offs.Source: §II · p. 2
  • Earlier binary-weight SNN work focused on unsupervised learning or analog bistable synapses and was typically limited to shallow networks and simple digit recognition.The authors distinguish those efforts – from their focus on supervised deep BNNs/SNNs.Source: §II · p. 2
  • The proposed B-SNN uses standard non-spiking training followed by conversion, with design-time and run-time optimizations intended to cut inference latency by an order of magnitude without compromising accuracy.Depending on the application’s accuracy requirement, the network can run for multiple time-steps on BNN-catered in-memory hardware.Source: §II · p. 2

A. Binary Networks

  • The BNN implementation follows XNOR-Net, using binary values for feedforward dot products while retaining proxy full-precision weights for gradient calculation.Training uses a Straight-Through Estimator with gradient clipping to (−1, +1); α is a non-binary scaling factor determined by the proxies’ L1 norm.Source: A. Binary Networks · p. 2

B. Spiking Networks

ANN-to-SNN conversion leverages the functional correspondence between ReLU units and integrate-fire neurons. The neuron accumulates input spikes and emits binary spikes when its membrane potential crosses a threshold.

B. Spiking Networks

  • ANN-to-SNN conversion is one of three broad SNN training approaches and scales to large problems by relying on standard backpropagation-trained networks.The other approaches named are backpropagation through time from scratch and unsupervised learning through Spike-Timing Dependent Plasticity.Source: B. Spiking Networks · p. 2
  • An integrate-fire neuron accumulates incoming spikes in a membrane state and fires when that state crosses a threshold.The paper describes the membrane state as vmem and the threshold as vth.Source: B. Spiking Networks · p. 3
  • The expected output spiking rate is given as the input firing rate multiplied by the synaptic weight, E[Y(t)] = w·E[X(t)].Here E[X(t)] denotes the input firing rate, described as the total spike count over a given number of time-steps.Source: B. Spiking Networks · p. 3
  • The paper connects IF spiking neurons to ReLU functionality and states that a ReLU-trained ANN can be converted with minimal accuracy loss.For a negative synaptic weight in the described single-input case, the neuron’s firing rate is zero; sparse spiking can support event-driven inference.Source: B. Spiking Networks · p. 3, Figure 1 · p. 3

C. Connecting Binary and Spiking Networks

The proposed B-SNN combines binary weights with analog ReLU activations during ANN training, then converts the network to IF neurons. The section describes architectural and threshold choices, the evaluation setup, and latency-reduction approaches for converted networks.

C. Connecting Binary and Spiking Networks

  • The B-SNN uses binary-weight training with analog ReLU neurons, while omitting bias and batch-normalization layers because they can cause substantial conversion accuracy loss.The architecture is analogous to Binary-Weight Networks, with pooling and neuron-reset choices also treated as design constraints.Source: C. Connecting Binary and Spiking… · p. 3
  • SNN thresholds trade off latency and accuracy, and the authors normalize them to each layer’s maximum ANN activation recorded on the training set.The passage notes that other thresholding schemes may further reduce conversion accuracy loss.Source: C. Connecting Binary and Spiking… · p. 3
  • As the number of time-steps grows, a B-SNN approaches a Binary-Weight Network; for finite time-steps, its activations represent multiple bits over time while remaining event-driven.The proposed RRAM mapping uses one row per bit-cell for B-SNN spikes rather than two differential rows per BNN input, reducing array area by 50%; the B-SNN still accumulates dot products over time.Source: C. Connecting Binary and Spiking… · p. 3, Figure 2 · p. 4

A. Datasets and Implementation

  • Evaluation uses CIFAR-100 and ImageNet, with empirical analysis and optimization performed on CIFAR-100 before settings are used for ImageNet simulation.The datasets are static images; the paper notes that SNNs are suited to spatio-temporal event-sensor data, while sensor-algorithm co-design remains active research.Source: A. Datasets and Implementation · pp. 3–4
  • The study uses a modified VGG-15 and ANN-to-SNN conversion, while targeting conversion latency through control choices rather than computationally exhaustive fine-tuning.The modified architecture removes one linear layer after VGG-16 XNOR-Nets could not be trained successfully with three fully connected layers; the reported accuracies are top-1 only. Conversion replaces ReLUs with IF nodes, with weights normalized from maximum ANN activations on sampled training images.Source: A. Datasets and Implementation · p. 4

B. Training B-SNNs

Hybrid training first fits a constrained full-precision model, then binarizes its weights; the resulting constrained-BWNs approach the original full-precision accuracy, while their improvement over XNOR-Net is minimal.

B. Training B-SNNs

  • The final validation accuracy improvement of constrained-BWN over an iso-architecture XNOR-Net is minimal.The authors attribute this to conversion-suitable architectural constraints coupled with weight binarization.Source: B. Training B-SNNs · p. 4
  • Fig. 3 compares CIFAR-100 validation convergence for full-precision training, constrained-BWN training after epoch 200, scratch-trained BWN, and XNOR-Net.Full-precision training is plotted from epochs 0–200, with constrained-BWN training following it.Source: Figure 3 · p. 5
  • Constrained-BWNs exhibited accuracies close to the original full-precision accuracies after hybrid training.The approach first trains a constrained full-precision model, then binarizes its weights.Source: B. Training B-SNNs · p. 5

C. Design-Time SNN Optimizations

Pooling type and placement affect conversion accuracy and latency. Average pooling before the neuron layer offers the best reported tradeoff, while max pooling performs slightly better in constrained-BWN models but can degrade after conversion.

1) Architectural Options:

  • Constrained-BWN accuracy was 64.9%, 65.8%, 67.7%, and 67.6% for average-before, average-after, max-before, and max-after pooling, respectively.All four architectures performed similarly to full-precision ANNs, with max-pooling architectures slightly better.Source: 1) Architectural Options · p. 5
  • Max-pooling architectures can lose accuracy during SNN conversion because max pooling is not distributed linearly over time, unlike average pooling.The authors note that linear average pooling avoids this conversion issue.Source: 1) Architectural Options · p. 5
  • The max-pooling-before-ReLU option adds an IF layer after convolution; without it, SNN accuracy fell to 33%.The authors hypothesize that max-pooling convolution outputs directly over time at each timestep explains the low accuracy.Source: 1) Architectural Options · p. 5

C. Design-Time SNN Optimizations

  • For average pooling before the ReLU/IF neuron layer, full-precision (binary) SNN accuracy is 63.2% (63.7%), versus full-precision (binary) ANN accuracy of 64.9% (64.8%).The authors selected this option as the best tradeoff between inference latency and final accuracy.Source: 1) Architectural Options · p. 5

1) Architectural Options:

2) Neural Node Options:

Subtractive-IF (SIF) neurons subtract the threshold at firing, reducing conversion-related accuracy degradation but increasing spiking activity. At a 62% accuracy target, SIF had lower delay than Reset-IF (RIF), with normalized-operation counts varying by model type.

Neural Node Options

  • SIF subtracts the threshold voltage at firing, removing the reset discontinuity and reducing conversion accuracy degradation, but at the cost of higher spiking activity.Source: 2) Neural Node Options · pp. 5–6
  • SNN energy optimization trades power benefits against latency overhead, which depends on the architectural options.Source: 2) Neural Node Options · p. 6
  • Lower normalized operations indicate higher SNN energy efficiency, with 1 representing iso-energy; the count excludes first and last layers because they are not binarized.The metric uses IF spiking rate and convolutional and linear layer operation counts.Source: 2) Neural Node Options · p. 6
  • At 62% accuracy, SIF had smaller delay than RIF; normalized operations were 4.40 (SIF) and 4.38 (RIF) for B-SNNs, and 2.35 (SIF) and 6.40 (RIF) for full-precision SNNs, where RIF missed the target.The comparison is shown in Fig. 6; the full-precision RIF model did not reach 62% within 400 timesteps.Source: Figure 6 · p. 6, 2) Neural Node Options · p. 6

1) Threshold Balancing Factor:

The section examines percentile-based threshold balancing and early exit as ways to reduce SNN inference costs. Early exit lowered computation without compromising the reported 62% accuracy, while threshold balancing exposes a latency–accuracy trade-off.

1) Threshold Balancing Factor:

  • A long-tailed ANN activation distribution makes maximum-activation firing thresholds unnecessarily high, motivating percentile-based threshold balancing.The maximum-activation histogram is shown in Fig. 7; the authors report that most SNN activations at inference are lower than the resulting threshold.Source: 1) Threshold Balancing Factor · p. 6, Figure 7 · p. 7
  • Threshold balancing can reduce B-SNN computation with a small accuracy relaxation while retaining a hardware framework similar to XNOR-Net.Source: 1) Threshold Balancing Factor · p. 7

2) Early Inference:

  • Early exit reduced computation from 4.30 to 3.55 without compromising 62% accuracy, with mean inference time of 62.4 rather than 105 timesteps.The timestep distribution is shown in Fig. 11; the accuracy and computation results are shown in Fig. 10.Source: 2) Early Inference · p. 7, Figure 10 · p. 8, Figure 11 · p. 8

E. ImageNet Results

On ImageNet, the binary SNN reached accuracy near the full-precision SNN while using 5.09 normalized operations. The section reports the full-precision ANN baseline and describes binarizing its pretrained model.

  • The full-precision VGG-15 ANN reached 69.05% top-1 accuracy after 100 ImageNet training epochs.Training used batch size 128, learning rate 1e-2, weight decay 1e-4, and SGD with momentum 0.9.Source: E. ImageNet Results · p. 8
  • The constrained BWN model reached 65.4% top-1 accuracy after 40 binarization-phase epochs following full-precision training.The network was binarized from the pretrained ANN using the previously described hybrid methodology.Source: E. ImageNet Results · p. 8
  • The binary SNN reached 62.71% top-1 accuracy versus 66.56% for the full-precision SNN, with 5.09 normalized #OPS.The corresponding ANN accuracies were 65.4% for the binary model and 69.05% for the full-precision model; early exit was excluded for comparison with prior work.Source: Figure 12 · p. 8, E. ImageNet Results · p. 8

V. CONCLUSIONS AND FUTURE WORK

The paper connects binary and spiking networks through time-domain encoding and ANN–SNN conversion. It reports near-full-precision accuracy for binary SNNs and identifies further accuracy, architecture-generalizability, and hardware-comparison work.

  • ANN–SNN conversion expresses multi-bit ANN activations as binary values over time, and the resulting binary SNNs achieve near-full-precision accuracy on large-scale image-recognition datasets.Source: §V · p. 8
  • The proposal uses a hardware backbone suited to BNN-catered in-memory computing platforms while exploring design-time and run-time optimizations for SNNs.Source: §V · p. 8
  • Future work includes reducing the accuracy gap, testing generalizability to architectures with residual connections, and evaluating hardware benefits against mixed- or reduced-precision implementations.Source: §V · p. 9

Related papers

Loading 2002.10064v3…