Source-linked AI summary
Training Spiking Deep Networks for Neuromorphic Hardware
Eric Hunsberger, Chris Eliasmith
TL;DR
Deep ANNs are successful in image categorization, but transferring them into spiking networks that retain classification accuracy is a central challenge. This paper trains networks using LIF neurons and noise to reduce conversion error. The resulting spiking networks perform nearly as well as their non-spiking counterparts across five datasets, including ImageNet, and may be more energy-efficient on neuromorphic hardware.
Problem
A central challenge is training an ANN so it can be transferred to a spiking network while minimizing the resulting network’s classification error.
Source: §2 · p. 2Method
The method trains deep networks with LIF neurons and adds noise to neuron outputs to simulate variability from filtered spike trains.
Source: §2.2 · p. 2, §4 · p. 8Results
Takeaways & Limitations
On specialized neuromorphic hardware, these spiking networks can be significantly more energy-efficient than traditional ANNs.
Source: §4 · p. 7Implementing max-pooling and local contrast normalization in spiking networks remains future work.
Source: §4 · p. 9
Abstract
from arXiv · showhide
We describe a method to train spiking deep networks that can be run using leaky integrate-and-fire (LIF) neurons, achieving state-of-the-art results for spiking LIF networks on five datasets, including the large ImageNet ILSVRC-2012 benchmark. Our method for transforming deep artificial neural networks into spiking networks is scalable and works with a wide range of neural nonlinearities. We achieve these results by softening the neural response function, such that its derivative remains bounded, and by training the network with noise to provide robustness against the variability introduced by spikes. Our analysis shows that implementations of these networks on neuromorphic hardware will be many times more power-efficient than the equivalent non-spiking networks on traditional hardware.
1 Introduction
Deep ANNs have advanced image categorization from smaller datasets to ImageNet, while spiking networks have been explored for brain-inspired modeling and energy-efficient neuromorphic systems. This paper presents a method for translating deep ANNs into spiking networks that supports varied neuron nonlinearities, scales to ImageNet, and is analyzed for energy efficiency.
1 Introduction
- Deep ANNs have progressed from MNIST, CIFAR-10, and Street-view house numbers to large-scale image categorization on ImageNet.ImageNet is described as containing 1.2 million images across 1000 categories.Source: §1 · p. 1
- Spiking deep networks transmit information through instantaneous single-bit spikes rather than real-valued activities, with energy-efficient neuromorphic systems among their motivations.Other work also aims to understand the brain by reverse engineering it.Source: §1 · p. 1
- The method translates deep ANNs into spiking networks, supports a broad range of neural nonlinearities, and scales to ImageNet.The paper also analyzes the potential power efficiency of neuromorphic implementations relative to non-spiking networks on traditional hardware.Source: §1 · p. 1
2 Methods
The method trains an ANN on static images and transfers its parameters to a spiking network, with changes to the neuron model and training procedure to reduce conversion error. It uses a smoothed LIF rate response and noise that simulates variability in filtered spikes, then replaces rate neurons with spiking LIF neurons for deployment.
2 Methods
- The ANN is trained on static images, then its weights and biases are used to form an SNN; the central challenge is minimizing the transferred network’s classification error.The network is based on a convolutional architecture with convolutional, locally connected, and/or fully connected layers.Source: §2 · p. 2, §2.1 · p. 2
2.1 Convolutional ANN
- The convolutional ANN removes local response normalization and replaces max pooling with average pooling because the former operations are difficult to implement in the feedforward spiking framework.Average pooling is straightforward in spiking neurons because it is a weighted sum.Source: §2.1 · p. 2
2.2 Leaky integrate-and-fire neurons
- LIF neurons integrate input current into membrane voltage, fire at threshold Vth = 1, and hold voltage at zero during the refractory period.With constant input current, the model yields a steady-state firing rate.Source: §2.2 · p. 2
- Smoothing the LIF rate response bounds its derivative, addressing the infinite derivative near zero input that causes backpropagation problems.The soft response is adjustable and approaches max(x, 0) as γ approaches zero; Figure 1 contrasts the hard and soft responses and their derivatives.Source: §2.2 · p. 3, Figure 1 · p. 3
2.3 Training with noise
- An α-function synaptic filter reduces spike variability, and training noise added to neuron outputs simulates variability in filtered spike trains.The α-filter is a second-order lowpass filter chosen for better noise reduction than a first-order lowpass filter.Source: §2.3 · p. 3
- Filtered-spike distributions vary with input current, but training uses Gaussian noise with σ = 10 for positive inputs and none when j ≤ 0.Near threshold the distribution is skewed toward higher rates; at higher currents it is skewed toward lower rates. Figure 2 shows filtered-spike variability versus input current.Source: §2.3 · pp. 3–4, Figure 2 · p. 4
2.4 Conversion to a spiking network
- During ANN-to-SNN conversion, weights and biases remain identical, while the network replaces soft LIF rate neurons with spiking LIF neurons and removes training noise.Post-synaptic filters are added to remove a significant portion of the high-frequency variation produced by spikes.Source: §2.4 · p. 4
3 Results
Across five datasets, the spiking LIF networks performed nearly as well as rate-based counterparts, and comparisons with prior spiking networks highlight the method’s scale and neuron-count tradeoffs. The efficiency analysis finds dataset-dependent accuracy–efficiency tradeoffs and identifies settings where neuromorphic implementations can be efficient.
3 Results
- Across MNIST, SVHN, CIFAR-10, CIFAR-100, and ImageNet, spiking LIF networks showed minimal accuracy loss compared with rate-based counterparts.Table 1 compares LIF SNNs with ReLU and LIF ANNs using the same network structure; ImageNet reports top-1 error and top-5 error in brackets.Source: §3 · p. 4, Table 1 · p. 5
- The method scales to ImageNet, which no other SNN implementation had done, and surpasses some one-chip TrueNorth results with about an order of magnitude fewer neurons.The compared TrueNorth networks used about 20× more neurons for one chip and 160× more for eight chips.Source: §3 · p. 5, Table 2 · p. 5
- On CIFAR-10, successive network modifications added error, while moderate training noise improved spiking-network performance; too much noise worsened it.Using σ = 20 rather than σ = 10 reduced the error added by spiking conversion but increased ANN error, resulting in worse SNN performance.Source: §3 · p. 5, Table 3 · p. 6
3.1 Efficiency
- 0.76 was the CIFAR-10 network’s relative efficiency versus standard hardware when each image was presented for 200 ms, making it somewhat less efficient.The estimate uses average rates of 2,693,315,174 synops/s and 49,536,000 neuron updates/s.Source: §3.1 · p. 6
- Estimated neuromorphic efficiency involves an accuracy tradeoff, with configurations that are significantly more efficient while sacrificing little accuracy.The estimates assume neuron updates cost 0.25 standard-hardware flops and synaptic events cost 0.08 flops.Source: §3.1 · p. 6, Table 4 · p. 7
- CIFAR-10 and ImageNet could run without synapses with little accuracy loss, whereas MNIST needed synapses; lower MNIST firing rates were proposed as the explanation.Average rates were 9.67 Hz for MNIST, 148 Hz for CIFAR-10, and 93.3 Hz for ImageNet.Source: §3.1 · p. 7
- Efficiency depends on classification timing and network properties; shortening presentation time requires a smaller synapse time constant, and synaptic events dominate energy use.The optimal classification start and end times depend on layer count, firing rates, and synapse time constants.Source: §3.1 · pp. 6–7
- Without training noise and synapses, the CIFAR-10 network did not exceed 18.06% accuracy, supporting noise training’s benefit.This comparison tested a network trained without noise and run without synapses.Source: §3.1 · p. 7
4 Discussion
The discussion presents accurate deep spiking networks using LIF neurons, with minimal ANN-to-SNN accuracy loss and potential energy-efficiency gains on neuromorphic hardware. It also identifies smoothing and noise training as key techniques, while noting that the efficiency analysis covers static image classification.
4 Discussion
- State-of-the-art LIF spiking networks maintain high classification accuracy, include a large-scale ImageNet system, and can be significantly more energy-efficient on neuromorphic hardware.The discussion reports minimal accuracy drop when converting from ANN to SNN.Source: §4 · p. 7
- Figure 3 examines classification-time effects on accuracy across starting times c0 and ending times c1; networks can be made efficient with minimal accuracy loss.The discussion says classification time affects accuracy and energy efficiency.Source: §4 · p. 7, Figure 3 · p. 8
- Smoothing the LIF response keeps its derivative bounded and introduces very little error when switching from the soft response to the hard response.For neurons requiring more smoothing, the authors suggest gradually relaxing smoothing during training.Source: §4 · p. 8
- Noise training decreases ANN-to-SNN transition error by 0.6% overall on CIFAR-10, but too little noise hurts SNN accuracy and too much hurts ANN accuracy.The noise simulates variability in filtered spike trains.Source: §4 · p. 8
- The efficiency analysis covers static image classification; efficiency gains for dynamic sequences are expected because similar adjacent frames reduce the need to reset after input changes.The authors describe dynamic-input efficiency as an expectation, not an evaluated result.Source: §4 · p. 8
- Future work includes lowering firing rates through refractory-period changes, implementing max-pooling and local contrast normalization, and fine-tuning offline-trained networks online with STDP.Other proposed directions include adapting neurons and lateral inhibition in convolutional layers.Source: §4 · pp. 8–9