Source-linked AI summary
S4NN: temporal backpropagation for spiking neural networks with one spike per neuron
Saeed Reza Kheradpisheh, Timothée Masquelier
TL;DR
Supervised learning for single-spike, latency-coded SNNs is challenging because firing-time thresholding is non-differentiable. The paper proposes S4NN, which backpropagates approximate errors through multilayer networks using firing latencies and non-leaky integrate-and-fire neurons. It reports 97.4% MNIST accuracy and 99.2% Caltech Face/Motorbike accuracy, with decisions made early and sparsely.
Problem
SNNs need supervised learning methods that handle information encoded in spike times despite the non-differentiability of thresholding at firing times.
Method
S4NN adapts backpropagation to multilayer single-spike SNNs by computing and propagating approximate errors using firing latencies.
Results
97.4% accuracy on MNIST and 99.2% accuracy on Caltech Face/Motorbike were achieved using non-leaky integrate-and-fire neurons.
Takeaways & Limitations
S4NN supports fast, accurate, sparse object recognition with a simpler neuron model and can be implemented with arbitrary hidden-layer depth.
Takeaways & Limitations
The paper evaluates S4NN on image categorization and leaves other data modalities and hardware implementations for future work or outside its scope.
Abstract
from arXiv · showhide
We propose a new supervised learning rule for multilayer spiking neural networks (SNNs) that use a form of temporal coding known as rank-order-coding. With this coding scheme, all neurons fire exactly one spike per stimulus, but the firing order carries information. In particular, in the readout layer, the first neuron to fire determines the class of the stimulus. We derive a new learning rule for this sort of network, named S4NN, akin to traditional error backpropagation, yet based on latencies. We show how approximated error gradients can be computed backward in a feedforward network with any number of layers. This approach reaches state-of-the-art performance with supervised multi fully-connected layer SNNs: test accuracy of 97.4% for the MNIST dataset, and 99.2% for the Caltech Face/Motorbike dataset. Yet, the neuron model that we use, non-leaky integrate-and-fire, is much simpler than the one used in all previous works. The source codes of the proposed S4NN are publicly available at https://github.com/SRKH/S4NN.
1 Introduction
S4NN addresses supervised learning in single-spike SNNs by adapting backpropagation to firing latencies and rank-order coding. The approach uses simple non-leaky integrate-and-fire neurons while preserving multilayer credit assignment.
- The main challenge in supervised SNN learning is the non-differentiability of spiking neurons’ thresholding activation at firing times.
- Existing approaches use rate-based activations, smoothed spike functions, surrogate gradients, or latency learning to address this challenge.
- S4NN classifies static inputs using rank-order coding, where neurons fire at most once and earlier spikes carry information.
- The paper adapts backpropagation to single-spike SNNs by propagating errors through layers using latencies instead of firing rates.
- S4NN uses a simpler non-leaky integrate-and-fire neuron with instantaneous synapses than the neuron models emphasized in prior approaches.
2 Methods
S4NN converts inputs into one-spike latency codes, propagates spikes through non-leaky integrate-and-fire layers, and trains weights by backpropagating firing-time errors. Classification uses the earliest output spike, while relative targets encourage the correct neuron to fire first without unnecessarily delaying already-early responses.
- 2.2 Forward path: The network contains an input layer, one or more hidden layers of non-leaky integrate-and-fire neurons, and one output neuron per category.
- 2.4 Temporal error backpropagation: Training compares actual and target output firing times, then backpropagates temporal errors and updates synaptic weights with stochastic gradient descent.
- 2.1 Time-to-first-spike coding: S4NN uses time-to-first-spike coding to convert each input value into at most one input spike, with larger values producing earlier spikes.
- 2.2 Forward path: Each integrate-and-fire neuron emits a spike the first time its membrane potential reaches threshold and does not fire again.
- 2.2 Forward path: The first output neuron to fire determines the predicted category, so earliest spikes throughout the network influence the decision.
- 2.5 Relative target firing time: Strict fixed targets can make an already-early correct neuron fire later during learning, contrary to the goal of fast responses.
- 2.5 Relative target firing time: Relative targets set the correct output’s target to the earliest observed firing time and penalize competing neurons only when they fire too close to it.
3 Results
S4NN was evaluated on Caltech face/motorbike and MNIST, showing high recognition accuracy with fast, sparse spike-based decisions. Training reduced MSSE below 0.1, while learned hidden neurons became selective to faces and motorbike variants.
- Caltech face/motorbike dataset: MSSE fluctuated during early training but stabilized after 15 epochs and remained below 0.1.Early jumps were attributed to large initial weight changes that could temporarily silence output neurons.
- Caltech face/motorbike dataset: The four hidden neurons learned category-selective patterns: one responded to faces, while three selected different motorbike shapes.The motorbike-selective neurons covered variation among motorbike shapes.
- MNIST Dataset: 97.4% ± 0.2% categorization accuracy was reported on MNIST using the proposed S4NN.The comparison included recent SNNs with spike-time-based supervised learning rules.
- Caltech face/motorbike dataset: 97.4% precision was obtained with decisions in 89.7 time steps and 218.3 spikes on average.These corresponded to 35.17% of maximum simulation time and 18.22% of 784+400+10 possible spikes.
4 Discussion
The discussion highlights S4NN’s sparse, latency-based computation with simple non-leaky integrate-and-fire neurons, while identifying hardware implementation and broader modality evaluation as open boundaries.
- S4NN uses multiple layers of non-leaky integrate-and-fire neurons with time-to-first-spike coding and temporal error backpropagation.
- 97.4% MNIST accuracy and 99.2% Caltech Face/Motorbike accuracy demonstrate accurate object recognition with a simple non-leaky neuron model.
- O(l*n^2*t) feedforward computation depends on the number of layers, largest-layer width, and time until the first output spike.The required time steps may be smaller than the maximum simulation time because learning encourages early accurate responses.
- At most one spike per neuron and early decisions may reduce latency, energy, and memory requirements relative to longer spike-rate processing.The paper states that hardware implementations themselves are outside its scope.
- Instantaneous synapses simplify processing because each input spike causes a punctual potential increment without the state required by current- or conductance-based synapses.
- The evaluation covers image categorization only, while future work is proposed for other modalities and convolutional spiking architectures.Extending S4NN to convolutional layers must address vanishing or exploding gradients and weight sharing.