PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi

arXiv:1706.02609v3cs.NEq-bio.NCstat.ML

TL;DR

SNN training is limited by non-differentiable spikes and methods that use only spatial information, motivating a unified spatio-temporal approach. The paper introduces iterative LIF dynamics, STBP, and an approximated spike derivative, achieving the best reported accuracy across the evaluated static and dynamic datasets and architectures.

Abstract

from arXiv · show

Compared with artificial neural networks (ANNs), spiking neural networks (SNNs) are promising to explore the brain-like behaviors since the spikes could encode more spatio-temporal information. Although pre-training from ANN or direct training based on backpropagation (BP) makes the supervised training of SNNs possible, these methods only exploit the networks' spatial domain information which leads to the performance bottleneck and requires many complicated training skills. Another fundamental issue is that the spike activity is naturally non-differentiable which causes great difficulties in training SNNs. To this end, we build an iterative LIF model that is more friendly for gradient descent training. By simultaneously considering the layer-by-layer spatial domain (SD) and the timing-dependent temporal domain (TD) in the training phase, as well as an approximated derivative for the spike activity, we propose a spatio-temporal backpropagation (STBP) training framework without using any complicated technology. We achieve the best performance of multi-layered perceptron (MLP) compared with existing state-of-the-art algorithms over the static MNIST and the dynamic N-MNIST dataset as well as a custom object detection dataset. This work provides a new perspective to explore the high-performance SNNs for future brain-like computing paradigm with rich spatio-temporal dynamics.

I. INTRODUCTION

SNN training is difficult because spikes are non-differentiable and existing approaches incompletely exploit spatial or temporal dynamics. The paper proposes iterative LIF dynamics and STBP to jointly train across both domains.

Motivation

  • SNNs encode spatial and temporal information through layer-wise dataflow and neuronal dynamics, unlike most feedforward DNNs.Their event-driven operation is also described as hardware friendly.Source: §I · p. 1, A. Iterative Leaky Integrate-and-Fire… · p. 2
  • Existing unsupervised, indirect supervised, and direct supervised methods each face limitations in achieving high-performance SNN training.STDP uses local activities, ANN-to-SNN conversion is not bio-plausible, and direct BP ignores temporal dynamics.Source: §I · p. 1
  • Spike activity is non-differentiable, creating gradient vanishing or exploding problems that obstruct error propagation.The paper addresses this with an approximated derivative for the spike activity.Source: §I · p. 1

Contribution

A. Iterative Leaky Integrate-and-Fire Model in Spiking Neural Networks

B. Spatio-Temporal Backpropagation Training

STBP computes gradients by combining layer-wise spatial error propagation with time-unfolded temporal propagation. The resulting gradients can train SNN parameters with standard gradient-descent optimization.

Loss and gradient computation

  • The loss minimizes mean squared error across samples and time steps within a specified time window.The output and label vectors define the supervised target for the final layer.Source: B. Spatio-Temporal Backpropagation… · p. 3
  • STBP derives gradients of the loss with respect to weights and biases from output and membrane-potential derivatives at every layer and time.These derivatives are the essential intermediate quantities for obtaining parameter gradients.Source: B. Spatio-Temporal Backpropagation… · p. 3

Spatial and temporal propagation

  • Spatial error propagation follows typical BP by accumulating weighted error signals from upper layers across network layers.This corresponds to the vertical propagation path in the spatio-temporal diagram.Source: B. Spatio-Temporal Backpropagation… · pp. 3–4, Figure 2 · p. 4
  • Temporal error propagation unfolds neuronal states across time so the chain rule can propagate errors through recurrent membrane dynamics.The temporal direction shares neuronal states across time steps and resembles BPTT conceptually.Source: B. Spatio-Temporal Backpropagation… · pp. 3–4
  • At earlier times and hidden layers, STBP combines spatial accumulation from upper layers with temporal error from self-feedback dynamics.This is the joint spatio-temporal case in the four-case derivation.Source: B. Spatio-Temporal Backpropagation… · pp. 3–4

C. Derivative Approximation of the Non-differentiable Spike Activity

The paper addresses spike-activity non-differentiability by replacing its ideal derivative with approximating curves whose shapes and steepness can be varied.

A. Parameter Initialization

The parameter-initialization procedure aims to stabilize firing activity by balancing timely responses against excessive spiking, while avoiding additional complex training techniques.

B. Dataset Experiments

The experiments evaluate STBP on static image datasets and dynamic event data, converting static samples into spike trains before SNN processing.

1) Spatio-temporal fully connected neural network:

The fully connected SNN experiments cover static classification, object detection, and dynamic N-MNIST event streams. STBP reports strong results while using the networks’ spatio-temporal information without complex training skills.

Static Dataset

Dynamic Dataset

Static Dataset

2) Spatio-temporal convolution neural network:

The framework extends to convolutional SNNs by replacing weighted summation with convolution and using average pooling, enabling deeper networks with richer spatial information. It is evaluated on MNIST and an object detection dataset against other spiking and conventional CNNs.

1) The Impact of Derivative Approximation Curves:

The study examines how approximating the spike derivative affects SNN accuracy and how temporal-domain backpropagation contributes to performance. Nonlinear curve shape has limited influence, while approximation width and inclusion of temporal information matter more.

The Impact of Derivative Approximation Curves

The Impact of Temporal Domain

IV. CONCLUSION

The paper presents STBP as a unified supervised-training framework for SNNs that exploits spatial and temporal information and uses an approximated spike derivative. Experiments support its performance on small-scale MLP and CNN networks without extra training techniques, while future work targets larger and more dynamic applications.

  • An iterative LIF-based framework enables spatio-temporal backpropagation for supervised SNN training.Source: §IV · p. 9
  • STBP combines spatial and temporal features and is implemented on both MLP and CNN architectures across static and dynamic datasets.Source: §IV · p. 9
  • The model outperforms state-of-the-art SNNs on relatively small-scale spiking MLPs and CNNs and outperforms same-size DNNs on dynamic N-MNIST.Source: §IV · p. 9
  • STBP achieves high performance without the extra training techniques commonly required by existing schemes.Source: §IV · p. 9
  • Approximation curves should capture nonlinear behavior, while their specific shape is not critical.Source: §IV · p. 9
  • Future work includes dynamic data processing and accelerating supervised training for large-scale SNNs on conventional or neuromorphic hardware.Source: §IV · p. 9

Related papers

Loading 1706.02609v3…