Source-linked AI summary
Training Deep Spiking Neural Networks using Backpropagation
Jun Haeng Lee, Tobi Delbruck, Michael Pfeiffer
TL;DR
Deep SNNs promise efficient event-based computation, but asynchronous spikes are non-differentiable and complicate training. The paper enables direct backpropagation by differentiating membrane-potential signals while treating spike-time discontinuities as noise, and reports improved results on PI MNIST and N-MNIST.
Problem
Non-differentiable spike signals make adequate training algorithms for deep SNNs difficult, limiting their accuracy relative to conventional deep networks.
Method
The method constructs continuous differentiable signals from low-pass-filtered spikes and membrane potentials, treating abrupt membrane-potential changes as noise during backpropagation.
Results
The technique improves the previous state of the art on both PI MNIST and N-MNIST and reaches accuracy levels matching conventional deep networks.
Takeaways & Limitations
The reported performance makes deep SNNs attractive for redundant-information tasks and energy-constrained applications that can benefit from event-based computation.
Takeaways & Limitations
The derivative treatment omits refractory periods and neglects higher-order effects through multiple lateral inhibitory connections.
Abstract
from arXiv · showhide
Deep spiking neural networks (SNNs) hold great potential for improving the latency and energy efficiency of deep neural networks through event-based computation. However, training such networks is difficult due to the non-differentiable nature of asynchronous spike events. In this paper, we introduce a novel technique, which treats the membrane potentials of spiking neurons as differentiable signals, where discontinuities at spike times are only considered as noise. This enables an error backpropagation mechanism for deep SNNs, which works directly on spike signals and membrane potentials. Thus, compared with previous methods relying on indirect training and conversion, our technique has the potential to capture the statics of spikes more precisely. Our novel framework outperforms all previously reported results for SNNs on the permutation invariant MNIST benchmark, as well as the N-MNIST benchmark recorded with event-based vision sensors.
1 Introduction
Deep SNNs use event-based updates to improve efficiency, but their non-differentiable spike signals make deep training difficult. The proposed approach treats membrane-potential signals as differentiable, enabling direct error backpropagation through spike-based networks.
- SNNs update units asynchronously in response to events, reducing redundant computation and potentially improving efficiency for real-time applications.
- SNNs have not matched conventional deep networks on traditional tasks partly because spike signals are non-differentiable.
- The method creates a continuous differentiable signal by adding low-pass-filtered spikes to membrane potentials and treating abrupt changes as noise during backpropagation.
- Regularization and normalization techniques address dead neurons and support stable, balanced learning in deep SNNs.
2 Related Work
Prior gradient-descent approaches for SNNs were limited by non-differentiable spikes, so successful methods primarily trained continuous rate models and converted them into spiking networks. Earlier reported MNIST performance remained around 94.09% for one such approach.
- The non-differentiability of spikes has limited direct investigation of gradient-descent methods for SNNs.
- Most successful prior approaches trained networks in continuous rate domains and converted them into spiking versions.
- 94.09% accuracy was reached by O’Connor et al. on MNIST using a spiking deep belief network based on the Siegert event-rate approximation model.
3 Spiking Neural Networks
The paper models fully connected SNNs with asynchronously updated leaky integrate-and-fire neurons, dynamic refractory effects, bounded membrane potentials, and optional winner-takes-all inhibition.
- The framework studies fully connected SNNs and distinguishes total synapses and neurons from their active subsets receiving or sending spike inputs.
- Leaky integrate-and-fire neurons asynchronously update membrane potentials from input-spike timing, providing computationally efficient event-based dynamics.
- The membrane potential incorporates synaptic weights and a dynamic refractory weight that suppresses input effects briefly after an output spike.
- The membrane potential is clipped to −Vth when it falls below that bound, helping balance neuron participation during training.
- Winner-takes-all circuits use lateral inhibition so that one spiking neuron suppresses the others in its group, with equal connection strengths reducing implementation costs.
4 Using Backpropagation in SNNs
The method derives a differentiable transfer function for spiking neurons and backpropagation equations, then stabilizes deep SNN training through approximations, initialization, error normalization, and regularization.
- 4.1 Transfer function and derivatives: The transfer-function derivation expresses membrane potential using input effects, resets, and lateral inhibition, providing the basis for backpropagation across layers.
- 4.1 Transfer function and derivatives: Spike-induced step jumps are treated as noise while continuous membrane-potential signals are differentiated to obtain backpropagation derivatives.
- 4.1 Transfer function and derivatives: Refractory periods are omitted from the derivative derivation because neuron activity is rarely dominated by them in the stated normal operating regime.
- 4.1 Transfer function and derivatives: Higher-order gradient effects through multiple lateral inhibitory connections are neglected, using a first-order approximation to reduce computational cost.
- 4.2 Initialization and Error Normalization: Initialization aims to balance forward activations and backward errors across layers, while regularization keeps weights near the initialization condition.
- 4.2 Initialization and Error Normalization: Backpropagation error normalization balances update magnitudes among layers using effective input and output activities.
5 Regularization
The paper introduces regularization methods to improve SNN stability, generalization, and neuron participation during training. Weight regularization controls parameter growth, while threshold regularization balances activity across neurons.
- Regularization addresses SNN generalization and the tendency of large initial thresholds to leave many neurons silent.These issues are especially significant in winner-takes-all circuits.
- Weight Regularization: Exponential regularization improves stability and generalization by replacing inadequate conventional L2 regularization.The paper reports that L2 regularization causes initial fast weight growth followed by continued decrease.
- Weight Regularization: Exponential regularization combines continuous weight decay with strong suppression of weight increases.Its decay rate is exponentially proportional to the squared sum of weights, combining properties of L2 and max-norm regularization.
- Threshold Regularization: Threshold regularization balances activity by increasing thresholds for active neurons and decreasing thresholds for all neurons according to the number that fired.Highly active neurons become less sensitive, while rarely active neurons respond more easily to later stimuli.
6 Results and Discussion
The experiments evaluate the proposed SNNs on permutation-invariant MNIST and event-based N-MNIST under specified training conditions. The method achieves strong benchmark performance, including a reported 98.53% N-MNIST accuracy and improvements over prior SNN results on both tasks.
- PI MNIST: The objective uses normalized output spike counts, with each output defined relative to the maximum output spike count.This normalization avoids the need for weight regularization in the output layer.
- PI MNIST: PI MNIST evaluation uses 1-second input streams for stable accuracy measurements, while training uses 50-ms presentations per digit.Networks with two hidden layers receive 200-ms stimuli during the first training epoch to address initially declining activity in higher layers.
- PI MNIST: The PI MNIST experiments prohibit data augmentation and convolutions to exclude techniques exploiting spatial input correlations.
- N-MNIST: N-MNIST is generated by moving a Dynamic Vision Sensor across projected digit images, producing event streams through three-phase saccadic movement.The movement shifts the digit position in pixel space.
- N-MNIST: 98.53% accuracy with 500 hidden units is reported as the best N-MNIST result with SNNs at the time.
- Overall Results: The proposed spike-based backpropagation improves prior SNN state-of-the-art results on both PI MNIST and N-MNIST.The paper reports accuracy levels matching conventional deep networks.