PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Spiking Neural Networks Using Lessons From Deep Learning

Jason K. Eshraghian, Max Ward, Emre Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, Wei D. Lu

arXiv:2109.12894v6cs.NEcs.ETcs.LG

TL;DR

The paper addresses how to train efficient, biologically plausible SNNs while retaining lessons from deep learning and neuroscience. It presents a tutorial and perspective spanning spike encoding, gradient-based learning, and online learning, and identifies connections between backpropagation and STDP alongside hardware efficiency gains. The paper also marks limitations and open boundaries, including changing practices and constraints in specific learning approaches.

  • Problem

    The paper addresses how to train brain-inspired SNNs using deep-learning principles while handling spike encoding, gradient optimization, and biological plausibility.

  • Method

    The paper synthesizes neuron models, spike encoding, gradient-based optimization, biologically motivated learning rules, and online recurrent-learning approaches in a tutorial and perspective.

  • Results

    Backpropagation through time produces weight updates that closely resemble STDP learning curves, while SNN implementations have reported energy improvements of up to 2–3 orders of magnitude over conventional ANN acceleration on embedded hardware.

  • Takeaways & Limitations

    SNN training can connect modern deep-learning optimization with biologically motivated local learning while supporting energy-efficient execution on tailored hardware.

  • Takeaways & Limitations

    The paper cautions that reported complexity improvements for RTRL variants often rely on narrow architectures or approximations, and its encoding list is non-exhaustive.

Abstract

from arXiv · show

The brain is the perfect place to look for inspiration to develop more efficient neural networks. The inner workings of our synapses and neurons provide a glimpse at what the future of deep learning might look like. This paper serves as a tutorial and perspective showing how to apply the lessons learnt from several decades of research in deep learning, gradient descent, backpropagation and neuroscience to biologically plausible spiking neural neural networks. We also explore the delicate interplay between encoding data as spikes and the learning process; the challenges and solutions of applying gradient-based learning to spiking neural networks (SNNs); the subtle link between temporal backpropagation and spike timing dependent plasticity, and how deep learning might move towards biologically plausible online learning. Some ideas are well accepted and commonly used amongst the neuromorphic engineering community, while others are presented or justified for the first time here. The fields of deep learning and spiking neural networks evolve very rapidly. We endeavour to treat this document as a 'dynamic' manuscript that will continue to be updated as the common practices in training SNNs also change. A series of companion interactive tutorials complementary to this paper using our Python package, snnTorch, are also made available. See https://snntorch.readthedocs.io/en/latest/tutorials/index.html .

1 Introduction

The paper introduces SNNs as brain-inspired models for improving neural-network efficiency through spikes, sparsity, and event-driven processing. It then frames training SNNs with deep-learning methods while connecting spike encoding, gradient-based optimization, and biologically motivated learning.

  • Motivation: SNNs target the efficiency gap between powerful deep-learning models and the brain’s low-power computation.The paper contrasts rapidly increasing deep-learning computational demands with the brain’s approximately 12–20 W operation.
  • Scientific uses: SNNs also provide models for investigating neural dynamics, learning rules, memory formation, and how natural intelligence may arise.Their utility extends beyond engineering applications to modeling individual neurons and higher-level neural systems.
  • Neuromorphic computing: Neuromorphic systems combine event-driven sensors, spike-based SNN algorithms, and specialized hardware to reduce data movement and power consumption.Sensors generate signals when changes occur; SNNs use binary activations encoded over time, while sparse activations support efficient hardware execution.
  • Applications: The paper aims to combine the broad capabilities of artificial neural networks with the potential efficiency of SNNs.SNN applications include low-power systems in medical, robotics, mixed-reality, edge-computing, and other domains.
  • Paper scope: The tutorial develops SNN training from neuron models and spike encoding through gradient-based optimization, STDP connections, and biologically plausible online learning.Its structure covers spike assignments, training challenges, approximate derivatives, and links between backpropagation and STDP.

2 From Artificial to Spiking Neural Networks

Spiking neural networks represent information with temporally timed, sparse binary spikes and suppress static input, aiming to improve efficiency while preserving useful neural computation. The section derives the leaky integrate-and-fire neuron and connects its recurrent formulation to practical modeling choices.

  • Neural code: SNNs encode information through single-bit spikes whose timing carries information, rather than through floating-point activation intensity.This representation supports low-precision parameters and spatial and temporal sparsity.
  • Neural code: Sparse spike activity reduces storage and computation because only non-zero events need to be represented.The passage illustrates this with sparse vectors and run-length-style indexing.
  • Neural code: Event-driven processing suppresses static input, while silicon-retina pixels independently report changes in brightness instead of waiting for a global frame.Compared with conventional frame-based cameras, this avoids activity-independent sampling and can reduce motion blur and active pixels.
  • Spiking neurons: An LIF neuron integrates weighted inputs into membrane potential U(t), emits a spike when U(t) reaches threshold θ, and resets after emission.The model is motivated by an RC circuit with time constant τ = RC and exponential membrane relaxation.
  • Spiking neurons: Discrete-time LIF dynamics use β as the membrane-potential decay rate, with weighted input and threshold-reset terms forming a recursive neuron model.The input coefficient can be absorbed into a learnable weight W, simplifying the input to I_in[t] = WX[t].
  • Alternative spiking neuron models: Neuron variants differ in threshold timing, reset mechanism, input scaling, leakage, synaptic filtering, and recurrence, so the appropriate choice depends on the goal.The paper notes that soft-reset variants can yield higher performance, although the reason remains an open question.

3 The Neural Code

SNNs assign meaning to spikes through separate input-encoding and output-decoding choices. The paper surveys rate, latency, and delta-modulated codes, their biological and data-dependent trade-offs, and links learning objectives to biologically motivated training alternatives.

  • SNNs separate input encoding—converting or passing data into the network—from output decoding, which trains informative firing behavior.
  • Rate coding represents input intensity through firing rate or spike count, whereas latency coding represents it through spike timing.Time-to-first-spike schemes assign greater meaning to individual spike times than rate codes do.
  • Delta modulation emits spikes for temporal changes in input intensity and remains silent otherwise, motivating event-based sensing and preencoded neuromorphic datasets.DVS cameras and silicon cochleas natively record changes rather than regularly sampled values.
  • The listed encoding mechanisms are non-exhaustive and not necessarily independent, while converting non-spiking datasets into spikes can degrade accuracy or performance through information loss.
  • Gradient-based SNN training can replace global error signals with localized errors or stochastic approximations, while rate-coded outputs are more robust to sparsity regularization than time-to-first-spike outputs.Local errors can reduce on-chip communication distance, whereas temporal outputs have had less success with sparsity-promoting regularization.

4 Training Spiking Neural Networks

Training SNNs requires addressing non-differentiable spikes, inactive neurons, and long temporal dependencies. The paper surveys shadow training, BPTT, surrogate gradients, and biologically inspired links to STDP, while highlighting quantization robustness and practical training choices.

  • SNN training methods include shadow training, native backpropagation through time, and local learning rules.Shadow training converts a trained ANN; BPTT trains the SNN directly; local rules use signals local to each weight.
  • Zero gradients at non-differentiable spikes can prevent error backpropagation from updating SNN weights.The paper identifies this non-differentiability as a central obstacle to gradient-based SNN training.
  • BPTT unrolls the SNN over time and sums gradient contributions from parameter applications across present and future losses.Recurrent weight sharing makes each time-indexed application an instance of the same learnable parameter.
  • Surrogate gradients enable dead neurons to transmit error signals, but introduce biased gradient estimates compared with spike-time gradients.Spike-time gradients are unbiased but cannot train dead neurons; surrogate gradients trade unbiasedness for broader trainability.
  • Quantization-aware training makes SNNs highly robust to low-precision weights, with binarized weights often affecting classification less than in equivalent non-spiking networks.The paper suggests sub-threshold neuron dynamics may absorb approximation and truncation errors.
  • BPTT updates decay as β^n across n time steps, producing weight-update curves that closely resemble STDP learning curves.This correspondence motivates replacing portions of BPTT with cheaper, local, gradient-modulated STDP rules.

5 Online Learning

Online learning constrains gradient calculations to present and immediately preceding values, motivating RTRL and its SNN adaptations. These methods enable temporally local updates but face substantial spatial and computational costs, with approximations trading exactness for efficiency.

  • Brain-inspired learning requires gradients to be temporally local, depending only on values at time t or t −1.
  • RTRL estimates BPTT gradients while propagating gradient-related information forward through time instead of storing the full temporal history.The influence term is recursively updated from present inputs and prior influence.
  • The influence calculation uses only present inputs and the influence at t −1, enabling online loss calculation with surrogate-gradient credit assignment.The input spike modulates both membrane potential and influence, and the resulting gradient flow is illustrated in Figure 12.
  • Full-network RTRL requires an influence matrix whose memory scales as O(n^3), despite BPTT memory scaling as O(nT).The matrix has dimensions n × P, with approximately O(n^2) parameters in the stated setting.
  • 5.2.1 RTRL Variants in SNNs: RTRL alternatives reduce demands by removing terms, adding stochastic noise, or tailoring approximations to specific architectures.Unbiased online recurrent optimization is reported to achieve quadratic rather than cubic computational complexity with network size.
  • 5.2.1 RTRL Variants in SNNs: Claims that RTRL variants outperform BPTT require caution because deferred updates make the two methods effectively identical.
  • 5.3 Spatial Locality: Surrogate-gradient BPTT subsumes STDP-like effects, with spike-time differences producing exponentially decaying weight-update magnitudes.The authors connect this result to error-triggered plasticity and argue that no neural-code-specific modification is required.

6 Outlook

The outlook connects SNN research to efficient computation, biological learning, and the broader question of which problems benefit from spikes. It emphasizes mixed-mode differentiation and cautions that brain-like learning likely requires heterogeneous methods and collaboration across disciplines.

  • Deep learning and neuroscience increasingly intersect because brain-derived models can support energy-efficient neural networks.
  • SNN hardware has reported energy-consumption improvements of up to 2–3 orders of magnitude over conventional ANN acceleration on embedded hardware.The reported benefits are associated with spikes, sparsity, and static data suppression.
  • Whether spikes are broadly advantageous for computation beyond energy efficiency remains an open question.The paper frames this as a question about which problems spikes are best suited to solve.
  • SNNs support mixed-mode differentiation by propagating temporal gradient terms forward while using reverse-mode differentiation for spatial terms.Forward-propagated terms may be approximated by eligibility traces that decay over time.
  • Brain-like learning is unlikely to reduce to one unified principle because neuronal diversity implies heterogeneous objectives and synaptic update rules.
  • Putting spikes at the center of deep learning requires cooperation among machine-learning engineers, neuroscientists, and circuit designers.

Additional Materials

The paper is accompanied by interactive tutorials for gradient-based learning with SNNs using the snnTorch Python package.

  • Interactive snnTorch tutorials provide complementary material for gradient-based learning with spiking neural networks.

A.1 Forward Euler Method to Solving Spiking Neuron Models

The forward Euler appendix discretizes spiking-neuron dynamics into update equations and compares reset mechanisms. It presents simplifying assumptions that favor computational ease, while noting trade-offs between performance and efficiency.

  • The forward Euler method approximates continuous-time integration by discretizing time into finite steps.The derivation uses small ∆t and isolates the membrane potential at the next time step.
  • The membrane decay rate β is related to the inverse time constant through β = (1 −∆t/τ), though this approximation is precise only when ∆t << τ.The appendix prefers calculating β from the continuous-time solution when possible.
  • The discrete neuron update weights input current by (1 −β) and shifts it one time step to contribute instantaneously to membrane potential.The paper states that this is not physiologically precise but makes the model resemble an RNN.
  • The derivation absorbs (1 −β) into a learnable weight and decouples W from β, favoring simplicity over biological precision.
  • Reset-by-subtraction retains residual superthreshold information, whereas reset-to-zero forces U[t] to zero after a spike.The two approaches converge for a sufficiently small time window, and the appendix notes that reset-by-subtraction is generally better for performance while reset-to-zero is more efficient.
  • The appendix’s spike-encoding and loss-function descriptions are given for one data sample and can be generalized to batches.

B.1 Rate Coded Input Conversion

Rate coding converts each input pixel into a spike train whose firing probability reflects normalized intensity; higher-intensity pixels fire at higher rates.

  • Each normalized pixel intensity becomes the probability of a spike at each time step in a 3-D rate-coded tensor.The conversion treats each feature independently and samples spikes as Bernoulli trials.
  • White pixels spike with certainty, black pixels never spike, and gray pixels spike probabilistically.For X_ij = 0.5, the expected spike proportion approaches 0.5 as the number of time steps increases.
  • Greater pixel intensity corresponds to a higher firing rate in the rate-coded representation.

B.2 Latency Coded Input Conversion

Latency coding represents input intensity through spike timing: stronger inputs reach threshold sooner, producing earlier spikes.

  • An RC circuit model derives a logarithmic relationship between input feature intensity and spiking timing.
  • A spike is emitted when the membrane potential reaches threshold under constant current injection.The membrane potential relaxes exponentially toward the steady-state value I_inR.
  • Larger input currents charge the membrane potential to threshold faster, causing earlier spikes.The steady-state potential I_inR is set to the input feature x.
  • In latency coding, a greater input intensity corresponds to an earlier spike time.

B.3 Rate Coded Outputs

Rate-coded output trains are classified by counting spikes from each output neuron and selecting the neuron with the largest count.

  • Output spike trains are represented as time-varying vectors, and each neuron’s spike count is obtained by summing across T time steps.
  • The output neuron with the maximum spike count determines the predicted class.
  • In the example, the first output neuron fires 8 times and represents the correct predicted class.

B.4 Cross Entropy Spike Rate

The cross-entropy spike-rate formulation treats output spike counts as logits and compares their softmax probabilities with a one-hot target vector.

  • The output-layer spike counts are treated as logits in the softmax function.Each c_i is an element of the output spike-count vector c.
  • Cross entropy is computed between the softmax probabilities and a one-hot target vector.The target components y_i take values in {0, 1}.
  • The target vector specifies the correct class as a one-hot encoded vector.

B.5 Mean Square Spike Rate

Mean square spike rate loss compares each output neuron's spike count with a target vector specifying the desired count for each class.

  • The output spike-count vector c is compared against target counts y_i over a period T.
  • The target count for the correct class should exceed those of the incorrect classes.
  • The target vector specifies the total desired number of spikes for each class.

B.6 Maximum Membrane

Maximum membrane loss uses each output neuron's peak membrane potential as a logit for cross-entropy training, while summed membrane potentials provide an alternative.

  • Maximum membrane loss takes each output neuron's peak membrane potential over time as its logit.
  • The membrane-derived logits replace spike counts in softmax, with cross entropy measured against the target label.
  • Increasing the correct class's peak and suppressing incorrect peaks promotes more correct-class firing and less incorrect-class firing.
  • Summing membrane potential over time provides an alternative way to obtain the logits.

B.7 Mean Square Membrane

Mean square membrane loss matches membrane potentials to targets across time, while related latency, spike-time, and activity objectives specify firing timing or regulate spike activity.

  • B.7 Mean Square Membrane: Mean square membrane error sums losses across all time steps and output neurons against time-varying membrane targets.
  • B.7 Mean Square Membrane: Time-static membrane targets can efficiently implement rate coding by setting the correct class above threshold and others below threshold.
  • B.8 Cross Entropy Latency Code: Latency cross entropy applies a decreasing transform to spike times so the correct class, which should fire first, receives the larger logit.
  • B.8 Cross Entropy Latency Code: Inverse spike time must exclude spikes at t = 0 because f_i approaches infinity there.
  • B.9 Mean Square Spike Time: Mean square spike-time losses support targets for first spikes or multiple spikes by summing errors between actual and desired timings.
  • B.10 Mean Square Relative Spike Time: Relative spike-time losses define a latency window γ between correct-class and incorrect-class firing.
  • B.11 Activity Regularization: Activity regularization penalizes excessive output firing or activity beyond upper and lower thresholds to encourage controlled or sparse responses.

C Appendix C: Training Spiking Neural Networks

The appendix connects spike-based gradient training with synaptic timing rules: membrane responses and spike times support gradient pathways, while STDP updates weights according to pre- and post-synaptic timing.

  • C.1 SpikeProp: SpikeProp models membrane potential using weighted presynaptic spike responses and an alpha-function kernel with time constant τ.
  • C.2 Spike-Time Gradients: Spike-time loss gradients can backpropagate through each neuron's spike time rather than through the entire simulation history.
  • C.2 Spike-Time Gradients: The membrane-potential derivative with respect to spike time contributes to the gradient calculation.
  • C.3 Spike Timing Dependent Plasticity: STDP increases synaptic strength when presynaptic spikes precede postsynaptic spikes and decreases it when their order is reversed.
  • C.3 Spike Timing Dependent Plasticity: The STDP update magnitude is parameterized by maximum modulation values and time constants over the interspike interval.
  • C.3 Spike Timing Dependent Plasticity: Correlated presynaptic spike trains increase weights faster than uncorrelated trains through causal spiking.
  • C.3 Spike Timing Dependent Plasticity: Unbounded potentiation can cause indefinite weight growth, motivating upper limits or homeostatic adaptive thresholds.
Loading 2109.12894v6…