Source-linked AI summary
Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi
TL;DR
SNN training is limited by non-differentiable spikes and methods that use only spatial information, motivating a unified spatio-temporal approach. The paper introduces iterative LIF dynamics, STBP, and an approximated spike derivative, achieving the best reported accuracy across the evaluated static and dynamic datasets and architectures.
Problem
Existing SNN training methods either incompletely exploit spatial and temporal dynamics or struggle with non-differentiable spike activity, limiting supervised training.
Method
The paper uses an iterative LIF model, spatio-temporal backpropagation across spatial and temporal domains, and an approximated derivative for spike activity.
Results
The method achieves the best reported accuracy across the evaluated static MNIST, dynamic N-MNIST, and custom object-detection datasets, including 98.89% testing accuracy on MNIST.
Takeaways & Limitations
The framework provides a unified way to train SNNs with rich spatio-temporal dynamics without many complicated training skills.
Abstract
from arXiv · showhide
Compared with artificial neural networks (ANNs), spiking neural networks (SNNs) are promising to explore the brain-like behaviors since the spikes could encode more spatio-temporal information. Although pre-training from ANN or direct training based on backpropagation (BP) makes the supervised training of SNNs possible, these methods only exploit the networks' spatial domain information which leads to the performance bottleneck and requires many complicated training skills. Another fundamental issue is that the spike activity is naturally non-differentiable which causes great difficulties in training SNNs. To this end, we build an iterative LIF model that is more friendly for gradient descent training. By simultaneously considering the layer-by-layer spatial domain (SD) and the timing-dependent temporal domain (TD) in the training phase, as well as an approximated derivative for the spike activity, we propose a spatio-temporal backpropagation (STBP) training framework without using any complicated technology. We achieve the best performance of multi-layered perceptron (MLP) compared with existing state-of-the-art algorithms over the static MNIST and the dynamic N-MNIST dataset as well as a custom object detection dataset. This work provides a new perspective to explore the high-performance SNNs for future brain-like computing paradigm with rich spatio-temporal dynamics.
I. INTRODUCTION
SNN training is difficult because spikes are non-differentiable and existing approaches incompletely exploit spatial or temporal dynamics. The paper proposes iterative LIF dynamics and STBP to jointly train across both domains.
- Motivation: SNNs encode spatial and temporal information through layer-wise dataflow and neuronal dynamics, unlike most feedforward DNNs.Their event-driven operation is also described as hardware friendly.
- Motivation: Existing unsupervised, indirect supervised, and direct supervised methods each face limitations in achieving high-performance SNN training.STDP uses local activities, ANN-to-SNN conversion is not bio-plausible, and direct BP ignores temporal dynamics.
- Motivation: Spike activity is non-differentiable, creating gradient vanishing or exploding problems that obstruct error propagation.The paper addresses this with an approximated derivative for the spike activity.
- Contribution: STBP jointly propagates errors through layers and time while avoiding many specialized training skills used by prior schemes.The framework is presented as a direct supervised approach for exploiting spatio-temporal dynamics.
- A. Iterative Leaky Integrate-and-Fire Model in Spiking Neural Networks: The iterative LIF model makes spatial and temporal recurrences explicit, improving compatibility with gradient-descent training.Its state combines spatial input accumulation with leaky temporal memory.
B. Spatio-Temporal Backpropagation Training
STBP computes gradients by combining layer-wise spatial error propagation with time-unfolded temporal propagation. The resulting gradients can train SNN parameters with standard gradient-descent optimization.
- Loss and gradient computation: The loss minimizes mean squared error across samples and time steps within a specified time window.The output and label vectors define the supervised target for the final layer.
- Loss and gradient computation: STBP derives gradients of the loss with respect to weights and biases from output and membrane-potential derivatives at every layer and time.These derivatives are the essential intermediate quantities for obtaining parameter gradients.
- Spatial and temporal propagation: Spatial error propagation follows typical BP by accumulating weighted error signals from upper layers across network layers.This corresponds to the vertical propagation path in the spatio-temporal diagram.
- Spatial and temporal propagation: Temporal error propagation unfolds neuronal states across time so the chain rule can propagate errors through recurrent membrane dynamics.The temporal direction shares neuronal states across time steps and resembles BPTT conceptually.
- Spatial and temporal propagation: At earlier times and hidden layers, STBP combines spatial accumulation from upper layers with temporal error from self-feedback dynamics.This is the joint spatio-temporal case in the four-case derivation.
C. Derivative Approximation of the Non-differentiable Spike Activity
The paper addresses spike-activity non-differentiability by replacing its ideal derivative with approximating curves whose shapes and steepness can be varied.
- The ideal spike derivative is Dirac-like, causing gradient vanishing or exploding that disables error propagation.
- Four candidate curves, h1 through h4, approximate the derivative of spike activity for STBP training.The candidates are introduced to address the non-differentiable output gate derivative.
- The approximation curves correspond to derivatives of rectangular, polynomial, sigmoid, and Gaussian cumulative distribution functions.
- Experiments analyze how different curve shapes and values of ai influence SNN testing accuracy.
A. Parameter Initialization
The parameter-initialization procedure aims to stabilize firing activity by balancing timely responses against excessive spiking, while avoiding additional complex training techniques.
- Parameter initialization must stabilize network firing while preserving timely responses and neuronal selectivity.
- The forward pass depends on multiply-accumulate operations over prespikes and weights plus threshold comparison.
- The method normalizes weights, thresholds, and other parameters before simulation.
- The simulations avoid fixed-amount-proportional reset, error normalization, weight or threshold regularization, and other complex training skills.
B. Dataset Experiments
The experiments evaluate STBP on static image datasets and dynamic event data, converting static samples into spike trains before SNN processing.
- STBP is tested on static MNIST, a custom object-detection dataset, and dynamic N-MNIST.
- Static-dataset samples are converted from pixel intensities to spike events using Bernoulli sampling.
1) Spatio-temporal fully connected neural network:
The fully connected SNN experiments cover static classification, object detection, and dynamic N-MNIST event streams. STBP reports strong results while using the networks’ spatio-temporal information without complex training skills.
- Static Dataset: The static experiments use MNIST and a two-category pedestrian-detection dataset, with spike patterns generated from image inputs.
- Static Dataset: 98.89% testing accuracy is achieved on MNIST with the proposed STBP method using a similar MLP architecture.
- Static Dataset: The object-detection experiment compares the proposed spiking model with a typical non-spiking MLP and reports better performance for the proposed model.
- Dynamic Dataset: N-MNIST represents dynamic vision-sensor data with on-events and off-events, motivating a two-channel input representation.
- Dynamic Dataset: The N-MNIST network uses the structure 34×34×2-400-400-10.
- Static Dataset: On the object-detection dataset, STBP achieves 98.78% accuracy compared with reported ANN and SNN methods.
2) Spatio-temporal convolution neural network:
The framework extends to convolutional SNNs by replacing weighted summation with convolution and using average pooling, enabling deeper networks with richer spatial information. It is evaluated on MNIST and an object detection dataset against other spiking and conventional CNNs.
- The spatio-temporal framework is extended to convolutional networks, allowing deeper architectures and more powerful spatial-domain information.
- Convolutional neurons update their membrane states from convolved inputs according to the LIF model.
- Average pooling replaces standard max pooling because binary SNN coding is inappropriate for max pooling.
- The spiking CNN is tested on MNIST and an object detection dataset, with MNIST preprocessing using elastic distortion.
- The evaluation compares the model with state-of-the-art spiking CNNs on MNIST and typical CNNs on the object detection dataset.
1) The Impact of Derivative Approximation Curves:
The study examines how approximating the spike derivative affects SNN accuracy and how temporal-domain backpropagation contributes to performance. Nonlinear curve shape has limited influence, while approximation width and inclusion of temporal information matter more.
- The Impact of Derivative Approximation Curves: Different nonlinear derivative approximations produce only small variations in performance.
- The Impact of Derivative Approximation Curves: Both excessively large and small rectangular widths worsen performance, while a1 = 2.5 achieves the highest testing accuracy.
- The Impact of Derivative Approximation Curves: The approximation should capture the spike derivative’s nonlinear nature; its specific curve shape is less critical.
- The Impact of Temporal Domain: STBP incorporates temporal-domain information alongside spatial-domain backpropagation, whereas the spatial-only baseline is termed SDBP.
- The Impact of Temporal Domain: STBP achieves 98.48% testing accuracy on MNIST and 98.32% on the object detection dataset without complex training skills.
- The Impact of Temporal Domain: Compared with STBP, SDBP loses 1.21% accuracy on the objective tracking dataset, five times the loss on MNIST.
IV. CONCLUSION
The paper presents STBP as a unified supervised-training framework for SNNs that exploits spatial and temporal information and uses an approximated spike derivative. Experiments support its performance on small-scale MLP and CNN networks without extra training techniques, while future work targets larger and more dynamic applications.
- An iterative LIF-based framework enables spatio-temporal backpropagation for supervised SNN training.
- STBP combines spatial and temporal features and is implemented on both MLP and CNN architectures across static and dynamic datasets.
- The model outperforms state-of-the-art SNNs on relatively small-scale spiking MLPs and CNNs and outperforms same-size DNNs on dynamic N-MNIST.
- STBP achieves high performance without the extra training techniques commonly required by existing schemes.
- Approximation curves should capture nonlinear behavior, while their specific shape is not critical.
- Future work includes dynamic data processing and accelerating supervised training for large-scale SNNs on conventional or neuromorphic hardware.