Source-linked AI summary
SLAYER: Spike Layer Error Reassignment in Time
Sumit Bam Shrestha, Garrick Orchard
TL;DR
Deep SNN training is hindered because spike generation is non-differentiable and temporally dependent, preventing direct standard backpropagation. The paper introduces SLAYER, which reassigns error across layers and time while learning weights and axonal delays, and reports state-of-the-art SNN accuracy across visual, action, and spoken-digit recognition tasks.
Problem
SNNs offer efficient spike-based computation, but their non-differentiable spike function prevents direct use of standard error backpropagation.
Method
SLAYER backpropagates error across SNN layers and time, handles the spike-function derivative through a stochastic approximation, and learns synaptic weights and axonal delays.
Results
SLAYER achieves state-of-the-art SNN accuracy across spoken-digit, visual-digit, and visual-action recognition tasks, including 98.78% testing accuracy on NMNIST.
Takeaways & Limitations
The CUDA-accelerated framework supports training larger SNN benchmarks and can configure a network offline before deployment to a chip.
Takeaways & Limitations
The paper considers axonal delays while omitting synaptic delays for simplicity.
Abstract
from arXiv · showhide
Configuring deep Spiking Neural Networks (SNNs) is an exciting research avenue for low power spike event based computation. However, the spike generation function is non-differentiable and therefore not directly compatible with the standard error backpropagation algorithm. In this paper, we introduce a new general backpropagation mechanism for learning synaptic weights and axonal delays which overcomes the problem of non-differentiability of the spike function and uses a temporal credit assignment policy for backpropagating error to preceding layers. We describe and release a GPU accelerated software implementation of our method which allows training both fully connected and convolutional neural network (CNN) architectures. Using our software, we compare our method against existing SNN based learning approaches and standard ANN to SNN conversion techniques and show that our method achieves state of the art performance for an SNN on the MNIST, NMNIST, DVS Gesture, and TIDIGITS datasets.
1 Introduction
SNNs offer a potentially efficient alternative to power-intensive ANNs, but their non-differentiable spike functions prevent direct use of standard backpropagation. SLAYER addresses this gap with temporal error reassignment, supports learning weights and delays, and is evaluated through a CUDA-accelerated framework.
- Motivation: Power-constrained deployments may favor SNNs because conventional ANNs generally require powerful GPUs and computing clusters.Biologically inspired spiking neurons and advances in SNN hardware motivate renewed research in efficient computation.
- Challenge: SNNs retain network topologies similar to ANNs but use memory-dependent, non-differentiable spiking neurons, blocking direct application of ANN backpropagation.Prior supervised methods largely target single neurons, while fewer approaches backpropagate through multiple SNN layers.
- Prior approaches: Earlier multilayer methods either define derivatives only around firing times or ignore the temporal effect of spike signals.These limitations motivate a more general temporal credit-assignment mechanism.
- Contribution: SLAYER distributes error credit across SNN layers and backward through time, while simultaneously learning synaptic weights and axonal delays.Its temporal reassignment reflects that a spiking neuron's current state depends on previous states.
- Evaluation: A released CUDA-accelerated framework trains SNNs and demonstrates state-of-the-art accuracy on neuromorphic tasks spanning visual digit, action, and spoken digit recognition.The paper organizes these demonstrations across benchmark experiments and compares SLAYER with prior learning methods and conversion approaches.
2 Spiking Neural Network: Background
SNNs process spike trains with neuron states that evolve over time, using spike-response and refractory kernels to compute membrane potentials. Their thresholded spike function is non-differentiable, while axonal delays extend the temporal response model to multilayer architectures.
- SNN background: An SNN uses biologically realistic spiking neurons as computational units, with inputs and outputs represented as spikes.The neurons maintain an internal state over time.
- Spiking neuron model: In the Spike Response Model, incoming spikes are convolved with a response kernel, while output spikes generate a refractory response.The incoming response is written as ai(t) = (ε ∗ si)(t), and the refractory response as (ν ∗ s)(t).
- Spiking neuron model: Synaptic weights scale spike-response signals, and the membrane potential is the sum of postsynaptic potentials and refractory responses.The resulting neuron state is compared with a predefined threshold to determine output spikes.
- Learning challenge: The undefined derivative of the spike function obstructs error backpropagation, while spike-response kernels distribute input effects into the future.This temporal distribution creates the dependency that learning methods must account for.
- Network extensions: Axonal delays are modeled by shifting the spike-response kernel, with d ≥ 0 specifying the delay.The formulation considers axonal delay while omitting synaptic delay for simplicity.
- Implementation: The CUDA implementation makes the SLAYER learning framework publicly available for training SNNs.The paper provides a public code link for the framework.
- Multilayer SNN: The feedforward formulation applies to fully connected, convolutional, and pooling layers, with layer weights and axonal delays defining forward propagation.The network operates on spike trains rather than ordinary numeric input and output values.
3 Backpropagation in SNN
SLAYER extends error backpropagation to multilayer SNNs by assigning credit across both layers and time, while approximating the non-differentiable spike derivative probabilistically. The method accounts for temporal dependencies and supports gradients for synaptic weights and axonal delays.
- Prior work: Prior deep-SNN training methods either convert trained ANNs, risk dead neurons, or approximate spike derivatives at a single time step while ignoring temporal dependencies.These approaches differ in whether they train an ANN shadow network or directly train the SNN.
- Loss function: The loss can target either a desired spike train or the number of output spikes within a classification interval.For interval-based classification, the target spike train itself need not be specified.
- Temporal credit assignment: SLAYER backpropagates error through multiple SNN layers and backward through time because neuron states depend on previous states.Signals at time n contribute to network losses at samples m = n through Ns.
- Spike derivative: SLAYER estimates the spike-function derivative using the probability density of a spiking-state change under small perturbations.The derivative is interpreted as a probability density function and approximated using a spike escape rate that decreases with distance from threshold.
- Parameter learning: The resulting gradients support learning both synaptic weights and axonal delays using standard or adaptive optimization methods.The paper identifies gradient descent, RmsProp, ADAM, and NADAM as applicable optimization choices.
4 Experiments and Results
Experiments evaluate SLAYER on a Poisson spike-train task and classification benchmarks using a CUDA-accelerated framework, covering spiking and converted non-spiking datasets. SLAYER reaches strong results across MNIST, NMNIST, DVS Gesture, and TIDIGITS, with temporal credit assignment visible during learning.
- Experimental Setup: The CUDA-accelerated framework simulates SNNs for all reported experiments, averaging SLAYER accuracy over five independent trials.The framework uses specified spike-response and refractory kernels, although SLAYER is independent of kernel choice.
- Poisson Spike Train: In the Poisson task, SLAYER suppresses unwanted spikes, reinforces desired spikes, and converges to the target train at epoch 739.The experiment uses 250 inputs over 50 ms, a Poisson target, and 25 hidden neurons.
- MNIST Digit Classification: 99.36% testing accuracy on MNIST is reported as the best result for completely SNN-based learning.Images are converted into 25 ms spike trains, and classification uses output spike counts.
- NMNIST Digit Classification: SLAYER surpasses the reported NMNIST state of the art with a comparable neuron count, while a no-delay network reaches 98.78% testing accuracy.The no-delay result is an increase of approximately 0.1%.
- DVS Gesture Classification: 93.64% average testing accuracy is achieved on DVS Gesture using only the first 1.5 s of approximately 6 s videos and fewer neurons and layers than TrueNorth’s approach.The reported result surpasses TrueNorth trained with EEDN, although output filtering raises TrueNorth accuracy to 94.59%.
- TIDIGITS Classification: 99.09% accuracy on TIDIGITS improves on the cited SNN approach while remaining competitive with the 99.7% non-spiking MFCC and HMM-GMM result.Audio is converted to spikes using MFCC followed by a Self Organizing Map.
5 Discussion
SLAYER provides a temporal error-backpropagation method for learning SNN weights and axonal delays, supported by a CUDA framework and state-of-the-art accuracy across several recognition tasks.
- SLAYER learns both synaptic weights and axonal delays while accounting for temporal dependencies, non-differentiable spikes, and dead neurons.
- SLAYER achieves state-of-the-art SNN accuracy on spoken digit, visual digit, and visual action recognition.
- Requiring both true and false neurons to fire, with a higher target for the true class, prevents neurons from becoming dormant.
- The CUDA-accelerated framework helps address larger SNN datasets, although these datasets remain smaller than those used in conventional deep learning.
- Neuromorphic chips such as TrueNorth, SpiNNaker, and Loihi have limited or absent learning mechanisms, making offline SLAYER training suitable for configuring deployed networks.