PaperlayerView exact revision on arXiv

Source-linked AI summary

Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation

Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy

arXiv:2005.01807v1cs.LGcs.CVstat.ML

TL;DR

Deep SNNs face a trade-off between expensive, slow backpropagation training and high-latency ANN-to-SNN conversion. The paper combines conversion-based initialization with incremental STDB, achieving competitive accuracy with fewer time steps and reduced training effort. The method is evaluated across CIFAR and ImageNet using VGG and ResNet architectures.

  • Problem

    Direct spike-based backpropagation is computationally expensive and slow, while ANN-to-SNN conversion can require high inference latency and does not exploit spike timing.

  • Method

    The method initializes an SNN with ANN-converted weights and thresholds, then trains it incrementally using STDB with a spike-time-based surrogate gradient.

  • Results

    The trained SNN achieves higher accuracy and fewer spikes per inference than purely converted SNNs at reduced time steps, with near iso-accuracy to similar ANNs and converted SNNs at lower compute cost and energy.

  • Takeaways & Limitations

    Hybrid initialization enables large-scale SNN training to converge with fewer epochs and supports lower-latency inference than conversion alone.

  • Takeaways & Limitations

    The ANN-to-SNN conversion setup excludes bias terms and therefore cannot use Batch Normalization as a regularizer, using Dropout instead.

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) which can potentially lead to higher energy-efficiency in neuromorphic hardware implementations. Many works have shown that an SNN for inference can be formed by copying the weights from a trained Artificial Neural Network (ANN) and setting the firing threshold for each layer as the maximum input received in that layer. These type of converted SNNs require a large number of time steps to achieve competitive accuracy which diminishes the energy savings. The number of time steps can be reduced by training SNNs with spike-based backpropagation from scratch, but that is computationally expensive and slow. To address these challenges, we present a computationally-efficient training technique for deep SNNs. We propose a hybrid training methodology: 1) take a converted SNN and use its weights and thresholds as an initialization step for spike-based backpropagation, and 2) perform incremental spike-timing dependent backpropagation (STDB) on this carefully initialized network to obtain an SNN that converges within few epochs and requires fewer time steps for input processing. STDB is performed with a novel surrogate gradient function defined using neuron's spike time. The proposed training methodology converges in less than 20 epochs of spike-based backpropagation for most standard image classification datasets, thereby greatly reducing the training complexity compared to training SNNs from scratch. We perform experiments on CIFAR-10, CIFAR-100, and ImageNet datasets for both VGG and ResNet architectures. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time steps, which is 10X faster compared to converted SNNs with similar accuracy.

1 INTRODUCTION

SNNs offer sparse, event-driven processing for potentially low-power neuromorphic intelligence, but conventional training and ANN-to-SNN conversion impose accuracy, latency, or computational costs. The paper proposes hybrid initialization and spike-timing dependent backpropagation to reduce training effort and inference time while maintaining accuracy.

  • Motivation: SNNs communicate through sparse binary spikes, making them a potential low-power alternative to conventional ANNs on neuromorphic hardware.ANNs process real-valued signals, whereas SNNs use discrete events and temporal processing.
  • Motivation: ANN-to-SNN conversion can match deep ANN accuracy on complex tasks but typically requires long inference latency.Converted SNNs exploit analog ANN training but do not optimize parameters based on spiking activity.
  • Hybrid training: The proposed hybrid method initializes an SNN with converted ANN weights and thresholds, then applies incremental spike-based backpropagation for few epochs.This initialization is intended to reduce convergence effort and enable inference with fewer time steps.
  • Spike-timing dependent backpropagation: STDB computes a surrogate gradient from neuron spike timing, triggering parameter updates when spikes occur and using timing differences in the gradient.The rule is motivated by Hebbian plasticity, which links synaptic changes to connected-neuron spiking activity.
  • Evaluation: The hybrid approach targets large VGG- and ResNet-like SNNs while avoiding exploding memory during spike-based backpropagation.Experiments cover ImageNet and CIFAR datasets, with near iso-accuracy to similar ANNs and converted SNNs at lower compute cost and energy.

2 SPIKE TIMING DEPENDENT BACKPROPAGATION (STDB)

STDB trains SNNs through temporal and spatial credit assignment using a surrogate gradient based on each neuron's most recent spike time. The method uses LIF dynamics, soft reset, BPTT, and memory-conscious gradient computation.

  • Leaky Integrate and Fire (LIF) Neuron Model: The LIF neuron leaks, integrates weighted presynaptic spikes, and emits a spike when membrane potential reaches threshold.Its discrete dynamics track membrane potential across time and trigger threshold-based output spikes.
  • Leaky Integrate and Fire (LIF) Neuron Model: Soft reset subtracts the firing threshold after a spike, preserving excess membrane potential for later time steps.This reduces information loss compared with resetting directly to a reset potential.
  • Spike Timing Dependent Backpropagation: BPTT performs spatial credit assignment across layers and temporal credit assignment across the network's recurrent time steps.The network is unrolled over time, with outputs accumulated across time steps for the final output.
  • Spike Timing Dependent Backpropagation: The output layer removes leak and does not fire, allowing cross-entropy loss to be defined on accumulated membrane potential.Hidden-layer updates use the thresholding nonlinearity's surrogate derivative, while output-layer neurons avoid that nondifferentiability.
  • Spike Timing Dependent Backpropagation: STDB computes surrogate gradients from the time since each post-neuron's most recent spike.The gradient depends on Δt, the difference between the current time step and the latest spike time.
  • Spike Timing Dependent Backpropagation: The proposed surrogate gradient is αe^-β∆t, with α and β adjusted according to the total simulation time T.For large T, β is lowered to slow decay and α is reduced to limit exploding gradients; values can be stored in a lookup table.

3 SNN WEIGHT INITIALIZATION

ANN-SNN conversion initializes SNN weights and thresholds before STDB training. The conversion requires threshold adjustment and imposes ANN-training constraints, including removal of bias.

  • ANN-SNN Conversion: ANN-SNN conversion copies trained ANN parameters into an SNN and uses conversion as initialization for STDB.The converted weights and thresholds provide an optimizer initialization for temporal and spatial credit assignment.
  • ANN-SNN Conversion: ∼2500 time steps is the reported inference latency drawback of conventional ANN-SNN conversion.Conversion does not optimize parameters afterward using spiking activity or temporal information.
  • ANN-SNN Conversion: Weight-normalization sets the threshold to 1 after scaling weights, whereas threshold-balancing leaves weights unchanged and sets the threshold to the normalization factor.The two procedures are described as having a similar effect.
  • Conversion Constraints: Conversion training omits bias because bias indirectly affects threshold voltage and makes threshold balancing more prone to conversion loss.The absence of bias also eliminates Batch Normalization as an ANN regularizer.

4 NETWORK ARCHITECTURES

The architectures use layer-wise threshold balancing for VGG and modify residual networks to reduce ANN-SNN conversion loss. Thresholds are computed from maximum neuronal inputs, while residual preprocessing replaces the original wide convolution.

  • VGG architecture: VGG hidden convolutional and linear layers receive thresholds equal to their maximum input over all time steps.Threshold assignment excludes the input and output layers and proceeds sequentially because each layer’s threshold is needed for subsequent computation.
  • Residual architecture: The residual SNN architecture includes shortcut connections between nonadjacent layers.These connections motivate architectural considerations intended to minimize ANN-SNN conversion loss.
  • Evaluation comparison: Table 1 compares ANN accuracy, long-run converted-SNN accuracy, reduced-time-step converted performance, and STDB-trained performance.The final column reports the Column-4 network after fewer than 20 STDB training epochs.

5 OVERALL TRAINING ALGORITHM

The training pipeline initializes SNN parameters through ANN-SNN conversion and then applies STDB-based forward and backward propagation. It preserves layer thresholds during training and uses fixed dropout masks across each input sequence.

  • Initialization: Algorithm 1 initializes SNN weights and thresholds from ANN-SNN conversion before training.The converted parameters provide the starting point for subsequent STDB optimization.
  • STDB training: Algorithms 2 and 3 perform mini-batch neuron initialization, forward propagation, backward propagation, and credit assignment for STDB training.Neuron parameters are initialized per mini-batch, while forward and backward propagation compute the updates.
  • Threshold handling: All neurons in a layer share one threshold voltage, and that threshold remains unchanged during training.This preserves the threshold assignment established during initialization.
  • Input and dropout: Dropout masks are sampled per mini-batch and reused across the entire input duration.The masked spike inputs are generated by element-wise multiplication with the fixed mask.
  • Input and dropout: The Poisson generator produces spike trains whose rates are proportional to input pixel values.A new random number is generated and compared at every time step for each pixel.

6 EXPERIMENTS

The experiments evaluate the proposed training mechanism on CIFAR10, CIFAR100, and ImageNet, including a VGG16 spike-activity comparison under matched 100-time-step conditions.

  • VGG16 comparison: 91.87% accuracy is achieved by the hybrid VGG16 SNN versus 89.20% for the purely converted SNN on 1,500 randomly selected test samples.Both networks are evaluated for 100 time steps.
  • Datasets: CIFAR10 contains 60,000 labeled 32×32 RGB images across 10 categories, split into 50,000 training and 10,000 testing images.The dataset provides the smallest-category benchmark described in this section.
  • Datasets: CIFAR100 is described as similar to CIFAR10 but with 100 categories.Its passage does not specify additional dataset dimensions or split counts.
  • Datasets: ImageNet contains 1.2 million labeled training images and 50,000 validation images across 1,000 categories.The dataset consists of high-resolution images.

7 ENERGY-DELAY PRODUCT ANALYSIS OF SNNS

Energy-delay analysis treats spike count and simulation duration as the primary first-order factors. Under matched conditions, the hybrid SNN produces fewer spikes and higher accuracy than the converted SNN.

  • Spike activity: 1.5× less average spiking is reported for the conversion-and-STDB SNN than the converted SNN across all layers under iso conditions.The comparison uses matched time steps, threshold voltages, and inputs, while the hybrid model also achieves higher accuracy.
  • Energy-delay factors: The first-order energy-delay product depends on the number of spikes and total time steps.A single spike consumes a constant amount of energy, so both activity and duration contribute to the analysis.
  • Measurement: 5.8 average spikes per neuron are measured in the tenth layer for both networks over 100 time steps across the evaluated inputs.This average sums layer spikes over the time window and divides by the number of neurons.
  • Interpretation: Higher spiking activity corresponds to lower energy-efficiency.The analysis therefore compares layer-wise average spike counts between converted and hybrid SNNs.

8 RELATED WORK

Prior direct-training methods face sparse-activity limitations or rely on membrane-potential surrogate gradients, while this work uses spike timing for deep SNN training.

  • Direct SNN backpropagation can fail on sparse activity because learning does not occur when neurons remain silent.
  • Earlier surrogate gradients were computed from membrane potential, whereas this work uses differences in spike timing.
  • Prior approaches evaluated shallow two-convolution-layer networks on MNIST, while this work trains deep stacked SNNs on complex classification tasks.

9 CONCLUSIONS

The proposed hybrid method initializes SNN backpropagation from ANN conversion, then uses spike-time surrogate gradients to reduce training epochs, latency, and spikes while retaining accuracy.

  • The hybrid method initializes SNN backpropagation with weights and thresholds from an ANN-converted SNN.
  • Spike-time surrogate gradients capture temporal information and help reduce the number of processing time steps.
  • The trained SNN achieved higher accuracy and fewer spikes per inference than purely converted SNNs at reduced time steps.
  • The method was tested on CIFAR and ImageNet datasets and achieved state-of-the-art performance with fewer time steps.

A COMPARISONS WITH OTHER SURROGATE GRADIENTS

The paper compares spike-timing surrogate gradients with linear and exponential membrane-potential approximations. The alternatives show similar convergence behavior, while spike timing enables bounded, precomputable gradients.

  • Spiking-neuron derivatives are approximated because the step-function derivative is undefined at spikes and zero elsewhere.
  • Linear and exponential surrogate gradients are functions of membrane potential relative to threshold voltage.
  • All tested approximations produced similar accuracy and convergence epochs for hybrid training on VGG9 with CIFAR10.
  • Spike-time differences behave similarly to membrane-potential differences while remaining bounded by the total number of time steps.
  • Because spike-time gradients have a bounded domain, they can be precomputed and stored in a lookup table.

B COMPARISONS OF SIMULATION TIME AND MEMORY REQUIREMENTS

For VGG16 on CIFAR10, SNN simulation requires substantially more time and memory than ANN training, while hybrid training reduces the SNN training-time burden.

  • One SNN training epoch took 78 minutes and 9.36 GB, compared with 0.57 minutes and 1.47 GB for ANN training.
  • One SNN inference epoch took 11.39 minutes and 1.37 GB, compared with 0.05 minutes and 1.15 GB for ANN inference.
  • 250 epochs were required for ANN and scratch-trained SNN convergence, whereas hybrid training used 250 ANN epochs followed by 20 spike-backpropagation epochs.
  • Hybrid training was one order of magnitude faster than training the SNN from scratch.
  • Hybrid training retained the SNN memory requirement because it still performs spike-based-backpropagation fine-tuning.
Loading 2005.01807v1…