Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks

Yingyezhe Jin; Wenrui Zhang; Peng Li

Source-linked AI summary

Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks

Yingyezhe Jin, Wenrui Zhang, Peng Li

arXiv:1805.07866v6May 21, 2018cs.NEcs.LG

TL;DR

SNNs offer spatio-temporal processing and low-power event-driven hardware, but training remains difficult because spike events are discrete and existing BP methods have scalability and gradient-mismatch limitations. The paper introduces HM2-BP, which couples macro-level rate errors with micro-level spike-train temporal effects, achieving strong results across static, dynamic, and speech benchmarks.

Problem
SNN training is challenging because firing counts depend on discretely adjusted spike events, while existing BP methods face scalability, discontinuity-handling, and rate-coded gradient limitations.
Method
HM2-BP backpropagates across firing rates and spike trains, using S-PSPs to capture temporal effects and directly computing rate-coded loss gradients with respect to tunable parameters.
Results
HM2-BP achieves strong performance across MNIST, N-MNIST, EMNIST, and TI46, including 99.49% accuracy on MNIST and 90.98% recognition accuracy on TI46.
Takeaways & Limitations
Backpropagation across macro and micro levels provides a practical approach to combining temporal effects and firing-rate behavior when training deep SNNs.
Takeaways & Limitations
The micro-level gradient computation approximates exact spike-timing effects because the timing-dependent factor is assumed not to change substantially during one small-update iteration.

Abstract

from arXiv · show

Spiking neural networks (SNNs) are positioned to enable spatio-temporal information processing and ultra-low power event-driven neuromorphic hardware. However, SNNs are yet to reach the same performances of conventional deep artificial neural networks (ANNs), a long-standing challenge due to complex dynamics and non-differentiable spike events encountered in training. The existing SNN error backpropagation (BP) methods are limited in terms of scalability, lack of proper handling of spiking discontinuities, and/or mismatch between the rate-coded loss function and computed gradient. We present a hybrid macro/micro level backpropagation (HM2-BP) algorithm for training multi-layer SNNs. The temporal effects are precisely captured by the proposed spike-train level post-synaptic potential (S-PSP) at the microscopic level. The rate-coded errors are defined at the macroscopic level, computed and back-propagated across both macroscopic and microscopic levels. Different from existing BP methods, HM2-BP directly computes the gradient of the rate-coded loss function w.r.t tunable parameters. We evaluate the proposed HM2-BP algorithm by training deep fully connected and convolutional SNNs based on the static MNIST [14] and dynamic neuromorphic N-MNIST [26]. HM2-BP achieves an accuracy level of 99.49% and 98.88% for MNIST and N-MNIST, respectively, outperforming the best reported performances obtained from the existing SNN BP algorithms. Furthermore, the HM2-BP produces the highest accuracies based on SNNs for the EMNIST [3] dataset, and leads to high recognition accuracy for the 16-speaker spoken English letters of TI46 Corpus [16], a challenging patio-temporal speech recognition benchmark for which no prior success based on SNNs was reported. It also achieves competitive performances surpassing those of conventional deep learning models when dealing with asynchronous spiking streams.

1 Introduction

SNNs offer spatio-temporal processing and event-driven hardware advantages, but their complex dynamics and discrete spikes make direct backpropagation difficult. HM2-BP addresses these challenges by propagating rate-coded errors across macro- and micro-level representations, achieving strong benchmark results.

SNNs are designed to process spatio-temporal patterns and support ultra-low-power event-driven neuromorphic hardware.
Discrete firing events make gradients of rate-coded loss functions with respect to tunable weights difficult to compute.
Existing direct-training approaches either omit temporal correlations, smooth away spike discontinuities, or compute gradients inconsistent with the rate-coded loss.
HM2-BP decomposes rate-coded error derivatives and backpropagates across firing rates, spike trains, and their interactions.
90.98% recognition accuracy on TI46 spoken English letters demonstrates performance on a challenging spatio-temporal speech benchmark without prior reported SNN success.

2 Hybrid Macro-Micro Backpropagation

HM2-BP addresses SNN training bottlenecks by combining macro-level rate-error propagation with micro-level spike-train temporal effects. Its S-PSP and decoupled modeling connect discrete firing counts to temporally precise spike interactions while enabling gradients with respect to tunable weights.

HM2-BP addresses complex spiking dynamics and non-differentiable spike impulses at both macro and micro levels.
2.1 Micro-level Computation of Spiking Temporal Effects: The LIF model describes membrane-voltage dynamics, threshold-triggered spikes, reset behavior, and synaptic currents driven by weighted presynaptic firing events.
2.1 Micro-level Computation of Spiking Temporal Effects: S-PSP aggregates the temporal influence of a presynaptic spike train on postsynaptic firings across exact spike times.
2.2 Error Backpropagation at Macro and Micro Levels: The total weighted S-PSP is related to firing counts, providing a bridge for rate-coded error backpropagation across discrete spikes and temporal effects.The paper describes the total post-synaptic potential as directly correlated with firing count and uses it to rewrite the rate-coded loss.
2.2 Error Backpropagation at Macro and Micro Levels: HM2-BP decomposes error propagation into macro-level rate backpropagation and micro-level temporal backpropagation, including their interactions across output and hidden layers.The method follows a forward simulation, output firing-count error computation, bidirectional-level backpropagation, and parameter update.
2.3 Decoupled Micro-Level Model for S-PSP: The decoupled S-PSP model separates rate and temporal effects, while a timing-based correction factor is approximated as stable under small incremental weight updates.The approximation relies on the correction factor changing little during one training iteration.

3 Experiments and Results

Experiments evaluate HM2-BP on static, dynamic, and extended handwritten-character benchmarks, plus speech, using fully connected and convolutional SNNs. The method reports strong accuracy, convergence behavior, and computational efficiency across these settings.

Fully Connected SNNs for the Static MNIST: 98.93% test accuracy is achieved by the fully connected HM2-BP SNN on static MNIST, outperforming STBP and reaching its best accuracy after 100 rather than 200 epochs.MNIST images are Poisson-encoded into 400 ms spike sequences without preprocessing or data augmentation.
Fully Connected SNNs for N-MNIST: 98.88% accuracy is achieved on N-MNIST, outperforming the previous best ANN at 97.38% and SNN at 98.78% with significantly fewer training epochs.N-MNIST provides asynchronous spatio-temporal spike patterns from ON- and OFF-events, simulated with a 0.6 ms time step.
Spiking Convolution Network for the Static MNIST: 99.49% accuracy is achieved by the spiking convolution network on static MNIST, surpassing the best previously reported performance at the same model complexity after 190 epochs.The network uses two 5 × 5 convolutional layers, pooling layers, and one fully connected hidden layer with elastic-distortion augmentation.
Fully Connected SNNs for EMNIST: HM2-BP significantly outperforms the non-spiking ANN and eRBP on EMNIST with fewer training epochs.EMNIST contains 47 classes, with 112,800 training samples and 18,800 testing samples.
Fully Connected SNNs for TI46 Speech: HM2-BP produces excellent results on the 16-speaker spoken English letters of the TI46 Speech corpus, a challenging spatio-temporal benchmark with no prior SNN success reported.Figure 5 also shows convergence for the first three experiments and limited change in a randomly selected synapse’s estimated α over training epochs.
Training Complexity Comparison and Implementation: The hybrid method is roughly O(N_T) times more efficient than [32] because it does not unwrap gradient computation in the time domain.The authors also report a publicly available CUDA implementation for direct training of deep SNNs.

4 Conclusion and Discussions

The paper presents HM2-BP as a direct-training method for deep SNNs that combines exact spike-timing information with rate-coded error propagation. Across several datasets and asynchronous spiking streams, it reports leading or competitive performance.

Conclusion and Discussions: HM2-BP trains deep SNNs directly from spiking activities using hybrid macro-level and micro-level error backpropagation.The method defines rate-coded errors at the macro level while capturing spike timings microscopically through S-PSPs.
Conclusion and Discussions: The approach directly computes gradients of the rate-coded loss with respect to tunable parameters while accounting for exact temporal effects.Its S-PSP representation captures spike timing, and the rate-coded error is propagated across both levels.
Conclusion and Discussions: HM2-BP achieves the best reported performances for fully connected and convolutional SNNs on MNIST, N-MNIST, EMNIST, and TI46 datasets.The paper also reports competitive performance exceeding conventional deep learning models on asynchronous spiking streams.
Conclusion and Discussions: Combining temporal effects with firing-rate behavior provides a practical training perspective for handling SNN discontinuities while exploiting both rate and temporal information.The authors attribute the method’s performance to addressing scalability, temporal effects, and discontinuous-loss gradient computation.

A Gradient Computation for the Output Layer with Lateral Inhibition

The output-layer derivation incorporates fixed lateral inhibition between output neurons into gradient computation. The resulting expression adds an inhibition factor while leaving the macro-level output error term unchanged.

Output Layer with Lateral Inhibition: The derivation assumes lateral inhibition between every pair of output neurons, with all inhibition weights fixed at w0.The total post-synaptic potential includes the lateral inhibition contribution.
Output Layer with Lateral Inhibition: The lateral-inhibition contribution is represented by the second term of the output neuron’s total post-synaptic potential.This term describes interactions between neurons in the same output layer.
Output Layer with Lateral Inhibition: The lateral-inhibition gradient expression matches the corresponding non-inhibited expression except for an added factor γ.The factor γ captures the effect of lateral inhibition.
Output Layer with Lateral Inhibition: Lateral inhibition does not change the macro-level derivative of the output-layer error with respect to the aggregated membrane potential.The macro-level backpropagation term remains the same as in the case without lateral inhibition.