Source-linked AI summary
Spiking Deep Networks with LIF Neurons
Eric Hunsberger, Chris Eliasmith
TL;DR
Deep spiking networks need neuron models that are biologically plausible yet trainable and transferable from rate-based networks. This paper smooths the LIF response for backpropagation and trains with output noise to handle spike variability, achieving strong CIFAR-10 and MNIST spiking-network results.
Problem
The paper addresses how to train deep networks that transfer accurately from static rate-based models to spiking networks using biologically plausible LIF neurons.
Method
It smooths the LIF response to bound its derivative, trains the static network with output noise, and converts its parameters to a filtered spiking network.
Results
The approach achieves the best published spiking-network result on CIFAR-10 and the best LIF-neuron spiking-network result on MNIST.
Takeaways & Limitations
The method supports accurate deep convolutional spiking networks with more biologically plausible LIF neurons and can be applied to other neuron types.
Abstract
from arXiv · showhide
We train spiking deep networks using leaky integrate-and-fire (LIF) neurons, and achieve state-of-the-art results for spiking networks on the CIFAR-10 and MNIST datasets. This demonstrates that biologically-plausible spiking LIF neurons can be integrated into deep networks can perform as well as other spiking models (e.g. integrate-and-fire). We achieved this result by softening the LIF response function, such that its derivative remains bounded, and by training the network with noise to provide robustness against the variability introduced by spikes. Our method is general and could be applied to other neuron types, including those used on modern neuromorphic hardware. Our work brings more biological realism into modern image classification models, with the hope that these models can inform how the brain performs this difficult task. It also provides new methods for training deep networks to run on neuromorphic hardware, with the aim of fast, power-efficient image classification for robotics applications.
1 Introduction
Deep artificial neural networks, especially convolutional networks, have achieved strong image-categorization results and can approach or exceed human performance on some datasets. Spiking research seeks to make these models more biologically plausible while supporting brain understanding and energy-efficient neuromorphic image categorization.
- Motivation: Deep ANNs and convolutional networks have achieved strong image-categorization results across MNIST, CIFAR-10, Street-view house numbers, and ImageNet.Progress extended from small datasets to the large and challenging ImageNet benchmark.
- Motivation: Their success has placed deep ANNs ahead of other machine methods for image categorization.
- Motivation: Near-human or better performance on some datasets suggests these models may illuminate how the human visual system solves similar tasks.
- Spiking networks: Spiking networks transmit information through instantaneous single-bit spikes rather than real-valued activities.
- Spiking networks: Spiking research targets both reverse-engineering brain computation and building energy-efficient neuromorphic systems for image categorization.
2 Methods
The method trains a static deep network that can transfer to spiking neurons, using modified LIF responses and output noise to address backpropagation and spike variability. The converted network preserves learned parameters while replacing rate units with filtered spiking LIF neurons.
- 2 Methods: The static network is trained on images with traditional deep-learning techniques before its parameters are transferred to a dynamic spiking network.The transfer objective is to make the spiking network's classification error as close as possible to the static network's error.
- 2.1 Static convolutional network: The convolutional architecture is based on Krizhevsky et al.'s five-layer network, replacing ReLU nonlinearities with a modified LIF neuron nonlinearity.The architecture includes convolutional, locally connected, and fully connected softmax layers.
- 2.1 Static convolutional network: Local response normalization and max-pooling are removed or replaced because their lateral-connection implementations are difficult in the feedforward spiking framework.Average pooling is easy to implement in spiking neurons as a weighted sum.
- 2.2 Leaky integrate-and-fire neurons: Smoothing the LIF rate equation removes its hard threshold and bounds the derivative, addressing the infinite derivative near j → 0+ that disrupts backpropagation.The soft maximum ρ1(x) = log(1 + e^x) replaces the hard maximum; a further substitution controls smoothing strength.
- 2.3 Training with noise: Output noise is added during static-network training to simulate variability in filtered spike trains received by postsynaptic neurons.The filtered spike train estimates neuron activity, while its statistics vary substantially with input current.
- 2.4 Conversion to a spiking network: The converted spiking network keeps the static network's weights and biases, replaces the soft LIF rate model with LIF dynamics, and removes training noise.Incoming spikes are filtered with an α-filter before reaching the LIF neuron; the selected synaptic decay constant is τs = 5 ms.
3 Results
On CIFAR-10 and MNIST, the authors evaluate spiking LIF networks under dataset-specific training and testing protocols, reporting competitive spiking-network results.
- The CIFAR-10 network used random 24 × 24 training patches and center patches from test images, without the original paper’s multiview testing.
- The CIFAR-10 static network achieved 14.63% error, compared with 11% in the original paper under different testing and training conditions.The authors attribute the difference to not using multiview testing and training for 160 rather than 520 epochs.
- 17.05% error on the full CIFAR-10 test set was the best published result for a spiking network on this dataset.
- Each successive modification for spiking implementation added about 1–2% classification error.
- Training with σ = 10 reduced the error introduced by switching to spiking neurons, whereas σ = 20 increased static-network error enough to worsen final spiking performance.The error introduced by switching was 2% with σ = 10 versus 1% with σ = 20, while σ = 20 added 2.5% error to the static network.
- The network’s average firing rate across all CIFAR-10 neurons was 148 spikes/s, with layer averages ranging from 7.6 to 172 spikes/s.The first two convolutional layers averaged 172 and 104 spikes/s; the locally connected layers averaged 10.6 and 7.6 spikes/s.
- On MNIST, the LIF spiking network performed on par with state-of-the-art published spiking-network results.The MNIST model used two hidden layers of 500 and 200 nodes and was trained with noise to address filtered spike-train variation.
4 Discussion
The paper shows that smoothed LIF neurons and noise-trained networks support accurate deep spiking classification while reducing conversion error. It also identifies broader applicability and practical constraints for comparisons and future implementations.
- Empirical results: The approach provides a way to train accurate deep convolutional networks with more biologically accurate LIF neurons instead of rectified-linear or sigmoid neurons.The resulting network can run in spiking neurons, with noise training reducing error introduced during the transition from rate-based computation.
- LIF response smoothing: Smoothing the LIF response function keeps its derivative bounded, enabling standard backpropagation with biologically plausible LIF neurons.The method is applicable to other neuron types with discontinuous response functions, though harsher discontinuities may require gradually relaxing the smoothing.
- Noise training: Training with noise on neuron outputs decreased spiking-network error by 3.4% by simulating variability introduced when filtering spike trains.Too little noise reduces robustness, while too much noise lowers static-network accuracy; the Gaussian approximation is only rough because spiking variability is non-Gaussian.
- Future directions: Future work includes more representative noise models, lower firing rates, max-pooling and local contrast normalization, and online STDP fine-tuning.The proposed noise should accommodate non-Gaussian variability and statistics that change across input currents.
- Empirical results: The network achieved the best published spiking-network result on CIFAR-10 and the best LIF spiking-network result on MNIST.Direct comparison remains difficult because conversion error depends heavily on neuronal firing rates.