PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning in Spiking Neural Networks

Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothee Masquelier, Anthony S. Maida

arXiv:1804.08150v4cs.NEcs.AI

TL;DR

Training deep SNNs is difficult because their spike-based transfer functions are non-differentiable, despite their biological and hardware motivations. This review surveys supervised and unsupervised deep-SNN methods and compares architectures, learning approaches, performance, computational cost, and hardware friendliness. SNN accuracy still generally trails ANNs, but the gap is decreasing and can vanish on some tasks while SNNs typically require fewer operations.

  • Problem

    Deep SNN training remains challenging because nondifferentiable spike-based computations limit conventional backpropagation, despite the importance of multilayer learning.

  • Method

    The paper reviews recent supervised and unsupervised methods across feedforward, convolutional, probabilistic, recurrent, and other deep SNN architectures.

  • Results

    97.93% accuracy was reported for a spiking multilayer perceptron on MNIST, comparable to conventional deep neural networks.

  • Takeaways & Limitations

    Deep SNNs provide a biologically inspired, power-efficient platform whose accuracy is approaching that of conventional deep learning methods.

Abstract

from arXiv · show

In recent years, deep learning has been a revolution in the field of machine learning, for computer vision in particular. In this approach, a deep (multilayer) artificial neural network (ANN) is trained in a supervised manner using backpropagation. Huge amounts of labeled examples are required, but the resulting classification accuracy is truly impressive, sometimes outperforming humans. Neurons in an ANN are characterized by a single, static, continuous-valued activation. Yet biological neurons use discrete spikes to compute and transmit information, and the spike times, in addition to the spike rates, matter. Spiking neural networks (SNNs) are thus more biologically realistic than ANNs, and arguably the only viable option if one wants to understand how the brain computes. SNNs are also more hardware friendly and energy-efficient than ANNs, and are thus appealing for technology, especially for portable devices. However, training deep SNNs remains a challenge. Spiking neurons' transfer function is usually non-differentiable, which prevents using backpropagation. Here we review recent supervised and unsupervised methods to train deep SNNs, and compare them in terms of accuracy, but also computational cost and hardware friendliness. The emerging picture is that SNNs still lag behind ANNs in terms of accuracy, but the gap is decreasing, and can even vanish on some tasks, while the SNNs typically require much fewer operations.

I. INTRODUCTION

Deep SNNs offer biologically inspired, temporally sensitive and potentially energy-efficient computation, but their training remains difficult because spike-based transfer functions are non-differentiable. This review surveys architectures and learning approaches for improving deep SNN performance.

  • Motivation: Deep neural networks achieve strong performance across applications, but conventional neurons use static continuous-valued activations unlike biological spike-based communication.Information in SNNs is conveyed through spike timing and spike rates.
  • Motivation: SNNs are intrinsically sensitive to temporal information and can reduce energy consumption because their output spike trains are sparse in time.Sparse spike events support low-energy implementations in both biological systems and hardware.
  • Motivation: Deep SNNs are scientifically motivated as models for studying multilayer brain computation and coding questions such as rate versus temporal coding.Effective training could support hypotheses and experimental ideas before physiological experiments.
  • Training challenge: Training deep SNNs remains challenging because spike trains are represented by sums of Dirac delta functions without derivatives, limiting derivative-based optimization.This difficulty is especially significant for multilayer learning.
  • Review scope: The review covers feedforward, convolutional, probabilistic, recurrent, and other deep SNN architectures and compares their learning approaches and performance.Its stated goal is to help develop efficient, high-performance deep SNNs and foster neuroscience crossfertilization.

II. SPIKING NEURAL NETWORK: A BIOLOGICALLY INSPIRED APPROACH TO INFORMATION PROCESSING

SNNs process information through discrete spike trains, with synaptic weights and learning rules shaping network responses. STDP provides a biologically inspired mechanism that adjusts weights according to the relative timing of pre- and postsynaptic spikes.

  • SNN fundamentals: SNNs encode analog inputs into spike trains using rate-based, temporal, or population coding before propagating them through weighted synaptic connections.Synapses may be excitatory or inhibitory, respectively increasing or decreasing membrane potential.
  • SNN fundamentals: The nondifferentiability of spike trains makes learning rules the most challenging component of developing multilayer deep SNNs.This limits direct use of the popular backpropagation algorithm.
  • Spike-timing-dependent plasticity: STDP changes synaptic efficacy according to the relative spike times of presynaptic and postsynaptic neurons within a temporal window of roughly tens of milliseconds.The rule is a bio-plausible form of local weight adaptation.
  • Spike-timing-dependent plasticity: When presynaptic spikes precede postsynaptic spikes, STDP strengthens the connection; when they follow, it weakens the connection.This timing-based interpretation is commonly used for unsupervised learning in SNNs.
  • Spike-timing-dependent plasticity: The standard STDP rule uses exponentially decaying effects, with A > 0 and B < 0 representing learning-rate parameters and τ setting the temporal learning window.The two cases correspond to long-term potentiation and long-term depression, although artificial SNNs commonly use variants.
  • Network-level effects: Repeated spatiotemporal patterns learned through STDP can produce stimulus selectivity, reduce postsynaptic response latency, and support pattern detection by a single neuron.STDP focuses on early spikes that contain much of the information needed for pattern recognition.

2) Probabilistic Characterization of Unsupervised STDP:

Probabilistic STDP models connect local spike-timing rules with inference over hidden causes and temporal structure. These approaches use stochastic circuits and reservoirs to approximate established probabilistic learning procedures.

  • Probabilistic motivation: Bayesian inference estimates hidden causes from prior knowledge and the likelihood of new observations to obtain posterior probabilities.The paper presents this as a motivation for probabilistic computation in neural systems.
  • Probabilistic STDP: A biologically plausible STDP rule with Poisson inputs and a stochastic winner-take-all circuit can approximately implement online expectation maximization for a multinomial mixture distribution.An inhibitory neuron was later used to implement the winner-take-all circuit in a cortical microcircuit-compatible extension.
  • Reservoir models: Liquid state machines built from stochastic winner-take-all reservoirs and linear readouts can use STDP to represent statistical structure in spatiotemporal input patterns.The generated spike trains can be viewed as samples from a hidden Markov model state space.
  • Probabilistic STDP: A modified unsupervised rule addressed negative excitatory synaptic weights and supported spatiotemporal pattern recognition and spoken-word classification.The acquired expectation maximization was shown to approximately implement EM for a Gaussian mixture model embedded in HMM states.

3) Supervised Learning:

Supervised learning in SNNs uses labeled outputs to reduce readout error, but deep implementations face nondifferentiable spike trains and biologically difficult feedback requirements. Proposed methods address these challenges through surrogate gradients, spike-timing objectives, STDP-based rules, and related likelihood or competitive approaches.

  • Supervised SNN learning adjusts weights using labeled desired outputs to reduce the error between desired and observed spike trains.The error is often defined over output spike trains, or readout error, in response to inputs.
  • Backpropagation is difficult for SNNs because spike-generating activation functions lack ordinary derivatives and require feedback connections using feedforward weights.The latter requirement is the weight transport problem, involving symmetric, accurately targeted feedback weights.
  • SpikeProp backpropagated spike-timing errors through a three-layer SRM network and classified a temporally encoded, nonlinearly separable XOR problem.Later Multi-SpikeProp variants extended the approach to multiple spike coding, while newer formulations refined temporal spike errors.
  • ReSuMe adapts the Widrow-Hoff rule to SNNs, expressing excitatory weight changes as the sum of STDP driven by desired spikes and anti-STDP driven by observed spikes.The desired-output signal is supplied by a teacher neuron without a direct physical connection, motivating the term remote supervised learning.
  • BP-STDP converts backpropagation updates into temporally local STDP rules for multilayer SNNs and achieves accuracies comparable to equal-sized conventional and spiking networks on MNIST.Other supervised approaches optimize postsynaptic spike likelihood or use target-neuron competition with STDP and non-target anti-STDP.

III. DEEP LEARNING IN SNNS

Deep learning in SNNs extends multilayer architectures and learning methods toward biologically plausible, power-efficient computation. The field seeks performance comparable to conventional deep networks while addressing the limited trainability and performance of directly trained spiking systems.

  • Deep learning uses many layers of trainable parameters and is typically trained end-to-end with backpropagation.Deep architectures emphasize stacked nonlinear layers, with theoretical and experimental evidence favoring depth over equally wide structures.
  • Deep networks commonly use convolutional, belief-network, recurrent, and fully connected architectures, with classification outputs often produced by softmax.Softmax maps output-unit values to mutually exclusive probabilities in the range (0, 1) that sum to 1.
  • Directly trained spiking deep networks generally perform worse than traditional deep networks, making comparable performance an important challenge for SNN hardware implementations.The review covers deep fully connected SNNs, spiking CNNs, spiking DBNs, and spiking RNNs.
  • Earlier hierarchical SNNs learned visual features with STDP, but many used only one trainable layer despite having multiple processing layers.These systems therefore provided multilayer processing without multilayer learning.
  • Deep SNN development targets networks that combine the power of DNNs with the efficiency and biological plausibility of SNNs.Initial implementation routes include converting trained neural networks to spiking platforms or using spike-based online learning.

A. Deep, Fully Connected SNNs

Deep fully connected SNNs use supervised and unsupervised learning strategies, including STDP, stochastic gradient descent, backpropagation, and ANN-to-SNN conversion. Reported results show competitive MNIST accuracy, while conversion methods target lower computational cost.

  • 95% MNIST accuracy was achieved by a two-layer SNN trained with unsupervised spike-based learning rules.
  • 97.93% MNIST accuracy approached the 98.37% performance of conventional ReLU-based deep networks using a spiking multilayer perceptron.
  • 98.88% MNIST accuracy was reported using membrane potential as a differentiable signal for backpropagation in a deep SNN.
  • Event-driven random backpropagation simplifies the backpropagation chain path using error-modulated synaptic plasticity with locally available learning information.
  • ANN-to-SNN conversion reuses trained DNN synaptic weights to exploit spiking hardware while avoiding direct training of spiking CNNs.
  • Figure 3 compares accuracy against total operations for 522 converted deep SNN configurations and a non-spiking DNN reference.

B. Spiking CNNs

Spiking CNNs combine convolutional feature extraction with spike-based learning or conversion from conventional CNNs. The reviewed approaches use biologically motivated filters, STDP, backpropagation, and inherited CNN weights to achieve strong image-classification performance.

  • Spiking CNNs apply convolutional and pooling hierarchies to extract increasingly complex visual features for classification.
  • STDP-trained spiking CNNs using Difference-of-Gaussian filters achieved 98.4% accuracy on MNIST with an SVM classifier.
  • SAILnet trained orientation-selective kernels for a spiking CNN, followed by an STDP feature-discovery layer for classification.
  • 99.05% MNIST accuracy was obtained with stacked convolutional autoencoders, comparable to traditional CNNs.
  • Backpropagation was used to build hierarchical spiking convolutional autoencoders as modules for deep spiking CNNs.
  • CNN-to-SNN conversion uses pretrained synaptic weights and has produced near-conventional CNN performance with fewer operations and lower energy consumption.

C. Spiking Deep Belief Networks

Spiking deep belief networks extend layer-wise unsupervised learning by replacing conventional stochastic units with spiking neurons or by converting pretrained DBNs. Spiking RBM learning can approximate contrastive divergence and preserve learned statistical properties.

  • DBNs stack pretrained RBMs using greedy layer-wise unsupervised learning, with contrastive divergence updates that do not require derivatives.
  • A spiking RBM replaces memoryless stochastic RBM units with stochastic integrate-and-fire neurons.
  • A variant of STDP can approximate contrastive divergence in a particular spiking network, yielding equivalent learned distributions’ statistical properties.
  • Figure 7 contrasts a three-RBM DBN with 784 input and 10 output neurons against a spiking RBM with 784 inputs, 500 hidden neurons, and 40 class neurons.
  • Pretrained DBNs can be converted to LIF-neuron networks for MNIST classification and extended for noise robustness and hardware constraints.
  • Hybrid Boltzmann machines use continuous hidden units and binary visible units, and are thermodynamically equivalent to Hopfield networks after marginalizing hidden units.

D. Recurrent SNNs

The paper defines recurrent SNNs by cycles in their directed graph and notes that winner-take-all, softmax, and backpropagation mechanisms can be implicitly recurrent.

  • A recurrent neural network has a cycle in its directed graph representation.
  • Winner-take-all and softmax modules are implicitly recurrent because equivalent brain functions use mutually recurrent inhibitory connections.
  • Backpropagation is also described as implicitly recurrent because its training algorithm presupposes recurrent connections.

1) Gated SNNs:

Recurrent SNN research includes direct spiking adaptations of gated networks and event-driven alternatives, but recurrent SNNs remain relatively few. These approaches target temporal processing while addressing the training and implementation challenges of spiking systems.

  • Gated recurrent architectures: BPTT training of recurrent networks can suffer from vanishing gradients on long sequences, motivating gated architectures such as LSTMs and GRUs.Gated recurrent networks use state and gates to control information flow through input sequences.
  • Recurrent SNN status: Only a few recurrent SNNs exist, and many are conventional recurrent networks converted into spiking frameworks.This limits the current breadth of recurrent SNN approaches.
  • Spiking LSTM implementations: A spiking LSTM implemented on IBM TrueNorth represented positive and negative values with two spike-train channels as a bio-plausible design choice.The work addressed both construction of a spiking LSTM and deployment on a neuromorphic chip.
  • Event-driven gating: Phased LSTM adds a time gate for asynchronously sampled, event-driven inputs arriving at multiple time scales.In experiments with varied event-driven sampling, it trained faster than a regular LSTM while matching its accuracy.

2) Liquid State Machines and Reservoirs:

Liquid-state and reservoir approaches use recurrent spiking dynamics to process temporal information, while newer architectures combine reservoir structure with trainable readouts or gated-memory components. Reported results include LSTM-comparable performance on sequential benchmarks.

  • Liquid State Machines: A liquid state machine uses a sparsely connected recurrent SNN reservoir designed to provide universal analog fading memory.The reservoir contains excitatory and inhibitory neurons and is part of the broader reservoir-computing framework.
  • Reservoir structure: Reservoir models combine time-varying spike inputs, a recurrent SNN reservoir, and a readout that extracts information from the reservoir state.Reservoir synapses may be trainable, and spatially dependent connection probabilities can structure the recurrent network.
  • Long short-term memory SNNs: The LSNN architecture combines input streams, an excitatory-inhibitory reservoir, adaptive-threshold neurons, and readout neurons.It was trained with BPTT using pseudo-derivatives based on membrane potential.
  • Benchmark results: LSNN achieved accuracy comparable to LSTMs on sequential MNIST and the TIMIT Acoustic-Phonetic Continuous Speech Corpus.The sequential MNIST task presents digit pixels as a sequence rather than as a static image.
  • Bio-inspired gated models: SubLSTM replaced multiplicative LSTM gates with subtractive operations intended for implementation by lateral inhibitory circuits.The model connected cortical-microcircuit considerations with learning through standard deep-learning frameworks.
  • Overall direction: Recurrent SNN studies combine conventional recurrent-network ideas with the bio-inspired framework of reservoir computing.The review characterizes this combination as a promising direction for recurrent SNNs.

E. Performance Comparisons of Contemporary Models

The review compares deep SNN architectures across datasets and distinguishes online learning from offline deployment through conversion. Offline approaches report higher performance, while online learning directly trains multilayer SNNs but generally achieves lower accuracy.

  • Learning tracks: Offline deployment methods report higher performance by converting pretrained neural networks rather than training multilayer SNNs directly.Online learning supports multilayer SNN training but reports lower accuracy rates.
  • Architecture comparison: Spiking CNNs achieve higher accuracy rates than spiking DBNs and fully connected SNNs on image-classification tasks.The comparison is intended to guide architecture and learning-mechanism choices for future investigations.

IV. SUMMARY

The review examines deep SNNs as biologically inspired, power-efficient alternatives to computationally expensive deep networks, focusing on learning methods that address nondifferentiable spike communication. Its surveyed results indicate that several spike-based approaches can match traditional DNN performance while retaining a brain-inspired framework.

  • Motivation: Deep networks offer strong hierarchical feature extraction and classification but are computationally expensive and difficult to implement on portable-device hardware.This motivates interest in alternative architectures with more efficient implementations.
  • Motivation: SNNs use sparse, spike-based communication and are described as power-efficient models for multilayer neural architectures.The review seeks to combine deep-learning performance with bio-inspired, power-efficient platforms.
  • Training challenge: Spike trains lack obvious derivatives, preventing SNNs from directly using derivative-based optimization for training.The paper reviews learning approaches for different SNN layers to address this open problem.
  • Scope: The review covers deep fully connected SNNs, spiking CNNs, and spiking DBNs as major deep-learning approaches.These architectures require specialized neuron models and learning techniques.
  • Conclusion: Recent reported results indicate that spike-based deep-learning methods perform as well as traditional DNNs.The paper frames this result as evidence for the potential of brain-inspired, power-efficient deep SNN platforms.
Loading 1804.08150v4…