Source-linked AI summary
Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networks
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer
TL;DR
Converting large CNNs to spiking networks can incur substantial accuracy loss and exclude common CNN features. The paper develops a theory and conversion tools to address these issues, reporting 91.35% SNN accuracy on CIFAR10 versus 91.94% for the ANN.
Problem
Applying prior conversion algorithms to larger networks such as CIFAR10 can cause accuracy drops of 10% or more and restrict the CNN features that can be converted.
Source: §1 · p. 2Method
The paper analyzes how SNN firing rates approximate ANN ReLU activations and derives conversion mechanisms and tools, including a reset modification and robust normalization.
Source: §2 · p. 2, §3.2.2 · p. 5Results
On CIFAR10, the SNN achieved 91.35% accuracy versus 91.94% for the ANN, which the authors describe as the best SNN result reported to date.
Source: §5 · p. 8Takeaways & Limitations
The authors report that the accuracy gap between ANNs and SNNs can be almost completely closed using conversion methods that cover most standard CNN features.
Source: §6 · p. 8The typical accuracy-latency tradeoff remains, and the networks were not specifically trained to converge fast.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Deep convolutional neural networks (CNNs) have shown great potential for numerous real-world machine learning applications, but performing inference in large CNNs in real-time remains a challenge. We have previously demonstrated that traditional CNNs can be converted into deep spiking neural networks (SNNs), which exhibit similar accuracy while reducing both latency and computational load as a consequence of their data-driven, event-based style of computing. Here we provide a novel theory that explains why this conversion is successful, and derive from it several new tools to convert a larger and more powerful class of deep networks into SNNs. We identify the main sources of approximation errors in previous conversion methods, and propose simple mechanisms to fix these issues. Furthermore, we develop spiking implementations of common CNN operations such as max-pooling, softmax, and batch-normalization, which allow almost loss-less conversion of arbitrary CNN architectures into the spiking domain. Empirical evaluation of different network architectures on the MNIST and CIFAR10 benchmarks leads to the best SNN results reported to date.
1 Introduction
Spiking networks offer event-based computation that may reduce latency and computational load, but converting large CNNs has posed accuracy and architecture challenges. This study analyzes those challenges and develops theory and mechanisms to improve conversion.
1 Introduction
- SNNs update when relevant information arrives, unlike ANNs that process the full input and compute activations layer by layer before producing an output.This event-based operation can reduce latency and computational load, and a deep SNN can be queried after its first output spike.Source: §1 · p. 1
- The study investigates why ANN-to-SNN conversion succeeds and proposes mechanisms to address its challenges and improve deep-SNN performance.It analytically examines conversion, introduces tools for a wider class of networks, and evaluates the proposed methods on CIFAR10.Source: §1 · p. 2
- Prior conversion studies showed near-state-of-the-art accuracy alongside faster classification, with approaches spanning weight normalization, noise injection, and neuromorphic-platform constraints.Diehl et al. (2015) reported less than 1% accuracy loss on MNIST, while earlier CNN conversions had restrictions such as zero biases and average pooling.Source: §1 · p. 2
- Large-network conversion can lose 10% or more in accuracy, partly because overly pessimistic weight normalization reduces firing rates in higher layers.Earlier methods also restricted conversion to networks without common features such as max-pooling, softmax, batch normalization, or biases.Source: §1 · p. 2
2 Theory of Conversion of ANNs into SNNs
The theory analyzes how SNN firing rates approximate ANN ReLU activations, assuming a one-to-one correspondence between ANN units and SNN neurons. It derives how reset choices shape approximation error and its propagation through deeper layers.
2 Theory of Conversion of ANNs into SNNs
- Reset to zero creates a persistent firing-rate approximation error that does not disappear with longer simulation.Smaller simulation time steps can reduce the error but increase computational effort.Source: §2 · p. 3
- Larger thresholds and smaller inputs improve the reset-to-zero approximation, but can require longer integration times.Reducing the simulation time step is another accuracy option, at increased computational effort.Source: §2 · p. 3
- Reset by subtraction makes firing-rate estimates converge to their target, with discrete sampling as the only remaining approximation error.The resulting approximations are more accurate than those of Cao et al. (2015) and Diehl et al. (2015).Source: §2 · pp. 3–4
- When input current exceeds the threshold, a neuron cannot reach its target frequency because its firing rate is capped at rmax.Weight normalization can prevent this saturation.Source: §2 · p. 4
- In deeper layers, approximation errors from earlier layers propagate and are multiplied by the weights of subsequent layers.Accumulated errors reduce input spike rates and help explain deteriorating firing rates in higher layers.Source: §2 · p. 4
3 New Methods for ANN-to-SNN Conversion
The section introduces methods to extend ANN-to-SNN conversion to more networks and reduce approximation errors. It also explains how biases can be represented in spiking networks.
3 New Methods for ANN-to-SNN Conversion
- The proposed methods aim to broaden the range of convertible ANNs and reduce approximation errors in SNNs.Source: §3 · p. 4
3.1 Converting biases
- An ANN bias can be implemented in an SNN as a constant input current proportional to that bias.Source: §3.1 · p. 4
- The conversion theory applies to neurons with biases, and parameter normalization can be extended to biases as well.Source: §3.1 · p. 4
3.2 Parameter normalization
Parameter normalization addresses the SNN’s restricted firing-rate range and extends data-based weight normalization to biases. A percentile-based scale offers a more robust alternative to maximum-based normalization, trading occasional saturation for higher firing rates across more samples.
3.2 Parameter normalization
- SNN firing rates are restricted to [0, rmax], unlike ANN activations, motivating normalization to avoid firing-rate approximation errors.Source: §3.2 · p. 4
3.2.1 Normalization with biases
- To preserve layer information, weights and biases are scaled jointly using the layer’s maximum ReLU activation across training examples.Source: §3.2.1 · p. 5
3.2.2 Robust mormalization
- Setting λ to the p-th percentile discards extreme outliers and raises firing rates for more samples, with typical effective p values from 99.0 to 99.999.The trade-off is that some neurons may saturate; the passage states this tends to harm performance less than too-low spike rates.Source: §3.2.2 · p. 5
- The maximum observed activation was 23.16, over three times the 99.9th-percentile scale of 6.83 in the first convolution layer.Maximum-based normalization can leave most samples far below the scale, producing insufficient firing to drive higher layers.Source: Figure 1 · p. 5, §3.2.2 · p. 5
3.3 Conversion of Batch-normalization layers
Batch-normalization transformations can be integrated into network weights after training, so converted SNNs need no separate batch-normalization conversion. The passage reports loss-less conversion with this integration.
3.3 Conversion of Batch-normalization layers
- Batch-normalization applies an affine transformation intended to produce zero-mean, unit-variance activations, using learned β and γ.Source: §3.3 · p. 6
- After training, batch-normalization transformations can be integrated into weights, preserving their effect while eliminating their computations.Source: §3.3 · p. 6
- Integrating batch-normalization parameters into preceding-layer weights yielded loss-less conversion and required no additional conversion for batch-normalization layers.Source: §3.3 · p. 6
3.4 Analog input to first hidden layer
For frame-based image benchmarks, analog input values can be used in the first hidden layer, with spiking computation thereafter. The passage reports this is particularly effective when ANN units have low activations.
3.4 Analog input to first hidden layer
- Because truly event-based benchmark datasets are rare, conversion accuracy is evaluated on conventional frame-based databases such as MNIST and CIFAR.Source: §3.4 · p. 6
- Converting analog pixel intensities or RGB values into Poisson firing rates adds variability and impairs performance without notable benefits.Source: §3.4 · p. 6
- Using analog inputs in the first hidden layer and spikes thereafter is particularly effective for low-activation ANN units, where spiking undersampling challenges conversion.Source: §3.4 · p. 6
3.5 Spiking softmax
The paper replaces winner-by-spike-count classification with a spiking implementation of softmax. It uses externally triggered spikes to compare accumulated inputs, including when final-layer neurons otherwise would not spike.
3.5 Spiking softmax
- Earlier conversion methods could fail when every final-layer neuron received negative inputs, leaving all neurons silent.Source: §3.5 · p. 6
- An external Poisson generator with a variable firing rate determines when an output spike is produced.Source: §3.5 · p. 6
- When triggered, a softmax competition uses accumulated membrane potentials to select the output.Source: §3.5 · p. 6
3.6 Spiking max-pooling layers
The paper introduces spiking max-pooling in which gating passes spikes from the neuron with the highest estimated firing rate and discards the others.
3.6 Spiking max-pooling layers
- The gating mechanism passes spikes from the maximally firing neuron and discards spikes from the others.Source: §3.6 · p. 6
- The gating function is controlled by estimates of presynaptic firing rates, such as online or exponentially weighted averages.Source: §3.6 · p. 6
- The demonstrated implementation uses exponentially weighted firing-rate averages to control the gating function.Source: §3.6 · p. 6
4 Results
Results distinguish gains from improved ANN architectures from gains through improved SNN conversion. The proposed methods yield near-lossless CIFAR10 conversion and a high-accuracy MNIST result.
4 Results
- Results assess two routes to improving SNN accuracy: training a better ANN and reducing approximation errors during conversion.Source: §4 · p. 7
4.1 Contribution of improved ANN architectures
- 87.86% CIFAR10 accuracy was achieved by the CNN with biases, softmax, batch-normalization, and max-pooling; removing softmax reduced accuracy to 69.44%.Source: §4.1 · p. 7
4.2 Contribution of improved SNN conversion methods
- 87.62% CIFAR10 accuracy followed conversion of the 87.86% ANN, while the MNIST network reached 99.44%.Source: §4.2 · p. 7
- Normalization factors corresponding to the 99th or 99.9th activation percentiles produced accurate classifications quickly and errors similar to the underlying ANN.Source: §4.2 · p. 7
5 Discussion
The discussion reports strong CIFAR10 SNN results and near-lossless conversion, while noting that the networks were not specifically trained for fast convergence. Figure 2 illustrates conversion-setting effects and the accuracy-latency tradeoff.
5 Discussion
- The discussion reports 87.82% CIFAR10 accuracy for the paper's best SNN and compares it with prior SNN results using smaller or specialized networks.Source: §5 · p. 7
- Figure 2 shows accuracy approaching the ANN level over time and indicates that robust weight normalization can tune the latency-final-accuracy tradeoff.Source: Figure 2 · p. 8
- Starting from an ANN with 91.94% accuracy, the method achieved 91.35% accuracy on CIFAR10, which the authors describe as by far the best SNN result reported to date.Source: §5 · p. 8
- The paper reports a typical conversion drop of less than 1%, but its networks were not specifically trained to converge fast.Source: §5 · p. 8
6 Conclusions
The paper’s conversion theory identifies a reset-mechanism change that improves SNN classification accuracy and supports tools for converting a broad class of CNNs. The authors report state-of-the-art SNN results and an almost completely closed ANN–SNN accuracy gap.
6 Conclusions
- A simple switch of reset mechanisms improves the classification accuracy of the resulting SNN.The authors say this mechanism was revealed by their theory of ANN-to-SNN conversion.Source: §6 · p. 8
- New conversion tools cover most standard CNN features and support almost loss-less ANN-to-SNN conversion.The authors report that these tools helped achieve state-of-the-art SNN results.Source: §6 · p. 8
- The authors report that the accuracy gap between ANNs and SNNs can be almost completely closed.Source: §6 · p. 8
- Future work includes applying the methods to datasets requiring large networks, such as ImageNet, and reducing spikes that redundantly transmit static-input information.Source: §6 · p. 8