PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks

Wenrui Zhang, Peng Li

arXiv:2002.10085v4cs.NEcs.LG

TL;DR

Existing SNN backpropagation methods struggle with discontinuous spikes and often require many time steps, limiting performance and scalability. TSSL-BP separates inter- and intra-neuron temporal dependencies to train deep SNNs more precisely. It achieves strong image-classification accuracy with short temporal windows, including five steps in reported settings.

  • Problem

    SNN BP training must handle non-differentiable spike events and long simulation windows that increase latency and hinder scaling to deep architectures.

  • Method

    TSSL-BP trains SNNs by decomposing error propagation across inter-neuron firing-time dependencies and intra-neuron state evolution.

  • Results

    TSSL-BP improves accuracy and runtime efficiency across several image datasets while training over short windows, including 5 time steps in reported experiments.

  • Takeaways & Limitations

    Short-window TSSL-BP supports low-latency SNN training and inference while preserving high accuracy across image-classification tasks.

  • Takeaways & Limitations

    The method requires bounding updates when membrane voltage barely reaches threshold because the resulting derivative can produce an undesirable large weight update.

Abstract

from arXiv · show

Spiking neural networks (SNNs) are well suited for spatio-temporal learning and implementations on energy-efficient event-driven neuromorphic processors. However, existing SNN error backpropagation (BP) methods lack proper handling of spiking discontinuities and suffer from low performance compared with the BP methods for traditional artificial neural networks. In addition, a large number of time steps are typically required to achieve decent performance, leading to high latency and rendering spike-based computation unscalable to deep architectures. We present a novel Temporal Spike Sequence Learning Backpropagation (TSSL-BP) method for training deep SNNs, which breaks down error backpropagation across two types of inter-neuron and intra-neuron dependencies and leads to improved temporal learning precision. It captures inter-neuron dependencies through presynaptic firing times by considering the all-or-none characteristics of firing activities and captures intra-neuron dependencies by handling the internal evolution of each neuronal state in time. TSSL-BP efficiently trains deep SNNs within a much shortened temporal window of a few steps while improving the accuracy for various image classification datasets including CIFAR10.

1 Introduction

SNN backpropagation aims to match conventional deep-network training, but discontinuous spikes, complex spatiotemporal dynamics, and long simulations remain obstacles. TSSL-BP addresses these challenges by modeling temporal dependencies and enables accurate training over short windows across image datasets.

  • SNN BP methods seek performance comparable to conventional DNN backpropagation, but developing equally mature training tools remains difficult.
  • Non-differentiable spike events make errors difficult to backpropagate precisely through complex spatial and temporal neural dynamics.
  • Long simulations increase latency and make spike-based computation difficult to scale to deep architectures.
  • TSSL-BP decomposes error propagation across inter-neuron and intra-neuron dependencies, using presynaptic firing times and neuronal-state evolution.
  • TSSL-BP trains SNNs over 5-10 time steps and improves accuracy and runtime efficiency on MNIST, NMNIST, FashionMNIST, and CIFAR10.

2 Background

Existing SNN training approaches either restrict spike outputs, ignore temporal correlations, or approximate spikes with continuous surrogate gradients. The paper uses LIF neurons and synaptic dynamics to represent firing, currents, membrane voltage, and reset behavior.

  • Existing Backpropagation methods for SNNs: SpikeProp variants are limited to one output spike per neuron, while ANN-to-SNN conversion introduces approximation errors and misses temporal learning.
  • Existing Backpropagation methods for SNNs: Surrogate-gradient BP methods approximate non-differentiable spike events, while some rate-coded methods treat spike timing as noise.
  • Existing Backpropagation methods for SNNs: TSSL-BP is designed as a universal training method for employed spike codes and supports inference and training over five time steps.
  • Spiking Neuron Model: The model adopts leaky integrate-and-fire neurons with synaptic dynamics to represent biologically plausible spiking behavior.
  • Spiking Neuron Model: Presynaptic spikes are converted through a synaptic model into an unweighted postsynaptic current that contributes to the postsynaptic neuron.
  • Spiking Neuron Model: Membrane voltage depends on leak, weighted synaptic input, and reset dynamics; discrete simulation uses a first-order Euler discretization.
  • Spiking Neuron Model: The firing output is generated by applying a Heaviside step function when membrane voltage reaches the firing threshold.

3 Methods

TSSL-BP trains SNNs by defining spike-sequence targets and propagating errors through inter-neuron and intra-neuron temporal dependencies. It avoids activation smoothing by computing inter-neuron effects at presynaptic firing times and accounting for incremental spike-time shifts.

  • 3 Methods: TSSL-BP trains output neurons to produce user-specified desired firing sequences using a loss that compares desired and actual spike trains over time.The method can use a suitable distance between the sequences; the paper defines total squared error across output neurons and time steps, with a Van Rossum-distance kernel described for spike trains.
  • 3 Methods: Backpropagated error at each time depends on output spikes and all preceding neuron states, making ∂a(l)[tk]/∂u(l)[tm] the key quantity for TSSL-BP.Once these dependencies are known for tk ≥ tm, errors can be propagated and layer gradients calculated.
  • 3 Methods: Activation smoothing can replace one discrete spike with multiple fictitious spikes and PSCs, altering firing times and degrading gradient accuracy for temporal learning.TSSL-BP is designed to avoid this alteration while handling nondifferentiable all-or-none firing behavior.
  • 3 Methods: TSSL-BP decomposes spatio-temporal error propagation into inter-neuron dependencies between neurons and intra-neuron dependencies across successive spikes of one neuron.A postsynaptic neuron depends on presynaptic neurons through spikes generating PSCs, while earlier spikes can affect later spikes from the same presynaptic neuron.
  • 3 Methods: Inter-neuron dependencies are evaluated through presynaptic firing times, because a PSC is triggered only when the presynaptic neuron fires.A membrane-potential perturbation can shift firing time, thereby shifting PSC onset and perturbing the postsynaptic input.
  • 3 Methods: Unlike activation smoothing, TSSL-BP applies the chain rule through presynaptic firing times rather than treating every prior time point as an inter-neuron dependency.The method also includes incremental changes in spiking times, reflecting the all-or-none firing behavior of spiking neurons.

4 Experiments and Results

Across MNIST, N-MNIST, FashionMNIST, and CIFAR10, TSSL-BP achieves strong or leading accuracy with substantially shorter temporal windows. The trained networks also exhibit sparse firing activity.

  • MNIST: 99.53% accuracy on MNIST is achieved with TSSL-BP using only 5 time steps, an 80× reduction in step count versus ST-RSBP.ST-RSBP is slightly higher by 0.09%, but its accuracy drops noticeably under short time windows.
  • N-MNIST: 99.40% accuracy is achieved on N-MNIST, while 99.28% outperforms the best previously reported result using 10 times fewer time steps.N-MNIST inputs are spikes rather than real-valued currents.
  • FashionMNIST: 89.80% test accuracy is achieved on FashionMNIST with a two-hidden-layer feedforward SNN, and 92.83% with a spiking CNN.The feedforward result exceeds the previously best HM2BP method, while the spiking CNN outperforms a same-size non-spiking CNN trained by standard BP.
  • CIFAR10: 89.22% accuracy, with 88.98% mean and 0.27% standard deviation over five trials, is achieved on one CIFAR10 CNN, while a second architecture reaches 91.41%.TSSL-BP delivers the best reported result among the compared ANN, converted-SNN, and STBP models, with up to 3.98% improvement over STBP and fewer time steps.
  • Firing sparsity: More than 84% of neurons are silent in the five-step CIFAR10 network, while about 4% fire at every time step.About 12% of neurons fire more than once.
  • Firing sparsity: More than 75% of neurons remain silent in the 100-step N-MNIST network, while about 5% have firing rates greater than 10%.Firing rates are averaged over 100 testing samples.

5 Conclusion

The conclusion presents TSSL-BP as a method for training deep SNNs that handles spiking discontinuities and preserves dependencies across space and time. It supports accurate learning with low temporal latency and reduced computational cost.

  • TSSL-BP trains deep SNNs while circumventing the non-differentiability of spiking activation functions and reflecting all-or-none firing across spatial and temporal dependencies.
  • TSSL-BP learns arbitrarily specified target firing sequences with high accuracy while training and inferring over only a few time steps.The conclusion contrasts this with existing SNN BP methods that commonly require hundreds of time steps for decent accuracy.
  • Training and inference over a few time steps reduce the computational cost, decision time, and energy dissipation required by large or deep SNNs.

Broader Impact

TSSL-BP is presented as improving temporal precision while enabling deep SNN training with low latency and reduced computational cost. The paper also describes potential benefits for scalability, energy efficiency, and community reuse.

  • TSSL-BP captures inter-neuron and intra-neuron dependencies to improve temporal learning precision.It uses presynaptic firing times for inter-neuron dependencies and models each neuron’s internal state evolution over time.
  • SNNs may provide ultra-low-power inference and training on large-scale neuromorphic computing hardware.The paper identifies training difficulties related to spiking neurons as a major challenge limiting wider adoption.
  • TSSL-BP delivers high-precision training over a short temporal window of a few time steps.The paper contrasts this with methods requiring hundreds of time steps for decent accuracy, linking shorter windows to faster decision making.
  • More than one order of magnitude reduction in training cost supports improved scalability and energy efficiency.The paper attributes these reductions to TSSL-BP’s low-latency training capability and reports lower energy dissipation during training and inference.
  • The implementation is prototyped in PyTorch and planned for public release.The authors state that the code is intended to benefit the brain-inspired computing community and neuromorphic hardware development.

Supplementary Materials for: Temporal Spike Sequence Learning via Backpropagation for Deep

The supplementary material identifies Wenrui Zhang as affiliated with the University of California, Santa Barbara.

  • Wenrui Zhang is affiliated with the University of California, Santa Barbara.

1 Full Derivation of TSSL-BP

The derivation describes TSSL-BP’s variables, loss construction, temporal error propagation, and decomposition of dependencies across neurons and time.

  • The derivation defines layer-specific weights, membrane potentials, and output spike trains for the neurons in each layer.Variables associated with layer l use l as a superscript.
  • The loss compares desired and actual output spike trains across considered time steps.The experiments define the loss using square error at each output neuron and time step, with a kernel-based alternative also described.
  • The weight-gradient derivation uses backpropagated errors and presynaptic activity to obtain weight updates.The supplementary text relates membrane potentials to postsynaptic currents and propagates error from later to earlier layers.
  • TSSL-BP backpropagates error through inter-neuron and intra-neuron dependencies.Inter-neuron effects are represented through presynaptic firing times, while intra-neuron effects include the firing-and-resetting mechanism in the LIF model.
  • The derivation assumes that no other presynaptic spike occurs between selected firing times in one stated step.Under this condition, a later intra-neuron dependency is already available from the backpropagation process.

2 Experiments and Results

The experiments use LIF-based SNNs implemented in PyTorch and run on an NVIDIA Titan XP GPU. They use short simulations, direct real-valued inputs for static images, and tabulated parameter settings.

  • Experiments use LIF-based SNNs implemented in PyTorch and run on an NVIDIA Titan XP GPU.The simulation step size is 1 ms, and thresholds and learning rates are empirically tuned.
  • Only a few time steps are used to demonstrate low-latency spiking neural computation.
  • Static-image inputs are converted directly from raw pixel densities into real-valued spike currents within a short time window.Neuromorphic datasets retain their original spikes as inputs.

2.3 Handling of Practical Issues

TSSL-BP addresses two practical issues in spike-time backpropagation: unstable updates near threshold and blocked gradients when neurons do not fire.

  • Near-threshold spikes can make the time derivative of membrane potential very small, producing excessively large weight updates.The method mitigates this numerical instability by setting a bound on the derivative.
  • Networks with no firing activity can block backpropagation through affected neurons.A warm-up mechanism increases firing activity before applying TSSL-BP when no spike occurs within a specified window.
  • The method uses fixed target spike sequences for five-step experiments, with one sequence assigned to the class-specific output neuron and another representing no spikes.The selected sequences are [0, 1, 0, 1, 1] and [0, 0, 0, 0, 0].

2.5 Datasets

The experiments evaluate TSSL-BP on conventional image datasets and neuromorphic event data, using dataset-specific preprocessing and temporal encoding.

  • MNIST: MNIST provides 60,000 training and 10,000 testing 28 × 28 grayscale images, with pixel values converted into real-valued input currents.Fully connected networks encode each sample into a 784 × Nt matrix over the simulation time steps.
  • N-MNIST: N-MNIST contains 34 × 34 × 2 ON- and OFF-event spike sequences originally lasting 300ms at 1us resolution.The experiments reduce temporal resolution 3000-fold, producing preprocessed samples of about 100 time steps.
  • Fashion-MNIST: Fashion-MNIST supplies 28 × 28 grayscale clothing images and uses the same preprocessing steps as MNIST.
  • CIFAR-10: CIFAR-10 contains 60,000 32 × 32 color images across 10 object classes, with normalization, augmentation, and dropout during training.The training set uses random cropping, horizontal flipping, and dropout at a rate of 0.2.
Loading 2002.10085v4…