Source-linked AI summary
Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, Kaushik Roy
TL;DR
Deep SNNs are difficult to train directly from input spikes because spike generation is discontinuous, while shallow networks have limited representation capacity. The paper addresses this with spike-based backpropagation using a LIF-aware approximate derivative, and reports strong classification results across standard datasets while analyzing sparse spiking computation.
Problem
Directly training deep SNNs from input spikes remains difficult because spike generation is discontinuous and non-differentiable, while shallow architectures have limited expressive power.
Method
The paper trains deep convolutional SNNs end-to-end with spike-based backpropagation using a pseudo-derivative that combines straight-through estimation with a leak correction for LIF neurons.
Results
The method achieves the best reported spike-based-learning classification accuracies on MNIST, SVHN, and CIFAR-10, including 99.59% on MNIST.
Takeaways & Limitations
Deep VGG and ResNet SNNs can be trained directly with spike inputs while retaining competitive image-recognition accuracy and enabling analysis of sparse event-based inference.
Takeaways & Limitations
Overall SNN efficiency remains difficult to estimate because memory traffic and unpredictable asynchronous dataflows can dominate energy consumption on von-Neumann hardware.
Abstract
from arXiv · showhide
Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.
1 Introduction
SNNs offer event-based, potentially energy-efficient computation, but shallow architectures limit representation capacity and direct training of deep SNNs remains difficult. This work introduces spike-based backpropagation with a leaky-neuron pseudo-derivative and evaluates deep VGG and ResNet architectures.
- Motivation: SNNs process discrete spatiotemporal events and may reduce computation and energy through sparse activity, especially in deeper layers.Specialized hardware exploits asynchronous event-based signaling for streaming data.
- Motivation: Shallow SNNs have limited expressive power, motivating spike-based versions of backpropagation that use differentiable surrogates for spiking activations.Existing layer-wise learning has demonstrated feature extraction, but standard ANN backpropagation still achieves better classification performance.
- Contributions: The paper develops supervised spike-based gradient-descent backpropagation using an approximate pseudo-derivative for LIF neurons.It also adapts deep ANN architectures and dropout for deep SNN training.
- Evaluation: The method is evaluated on MNIST, SVHN, CIFAR-10, and N-MNIST using deep convolutional SNN architectures.The experiments include VGG- and ResNet-inspired networks.
- Results: The authors report best classification accuracy on MNIST, SVHN, and CIFAR-10 among spike-based learning methodologies and analyze inference time and energy consumption.The paper also compares spike-based backpropagation with ANN-SNN conversion techniques.
2 The Component and Architecture of Spiking Neural Networks
The paper builds convolutional SNNs from LIF neurons, convolutional and pooling operations, and deep VGG- and ResNet-inspired feature extractors. Spike accumulation, leakage, thresholding, resetting, and pooling determine how activity propagates across time and layers.
- Spiking Neural Network Component: LIF neurons integrate weighted presynaptic spikes into a membrane potential that decays exponentially and resets after threshold crossing.The membrane potential is the neuron's internal state and produces an output spike when it exceeds Vth.
- Building Blocks: Convolutional layers convolve input spikes with trainable kernels, accumulate current in Vmem, and propagate spikes when Vmem exceeds Vth.Subthreshold membrane potential remains as residue for later time steps while leaking.
- Building Blocks: Spatial-pooling layers use fixed kernel weights and no membrane-potential leak, with average-pooling implemented using a fixed 2×2 kernel.The pooling threshold must balance excessive spikes against insufficient propagation.
- Architecture: The architectures alternate convolutional and spatial-pooling layers as intermediate feature extractors before a fully connected classification layer.Input images are encoded as Poisson-distributed spike trains with spike probability proportional to pixel intensity.
- Architecture: Deep SNN architectures leverage VGG and ResNet designs to stack convolutional layers and construct hierarchical feature extractors.The paper uses small 3×3 kernels in its VGG-inspired design and includes spiking VGG and ResNet blocks.
3 Supervised Training of Deep Spiking Neural Network
The proposed algorithm trains deep SNNs directly from spike inputs by combining backpropagation with an approximate derivative for leaky LIF neurons. Forward propagation accumulates temporal spike inputs, while backward propagation estimates errors and updates weights through hidden layers.
- Motivation: Standard backpropagation is incompatible with SNNs because spike outputs are discrete over time and the spike-generation function is non-differentiable.The proposed method addresses this discontinuity by formulating an approximate derivative for LIF activation.
- Forward Propagation: During forward propagation, weighted pre-spike trains accumulate in decaying membrane potentials; hidden neurons spike and reset after crossing threshold.The final layer suppresses spiking, accumulates weighted inputs through the final time step, and uses the resulting membrane potential to quantify outputs.
- Weight Update: Weight gradients are computed from incoming spike trains and propagated error gradients, then weights are updated with a learning rate to reduce loss.Iterative mini-batch updates are described as driving the network state toward a local minimum.
- Backward Propagation: The loss compares final outputs with target labels, then gradients propagate backward through hidden layers using the recursive chain rule.At the final layer, leak statistics are treated as noise so accumulated membrane potential can be approximated by total input current over T.
- LIF Derivative: The LIF pseudo-derivative combines a straight-through estimate for IF activation with a leak correctional term derived from low-pass-filtered output spikes.The correction accounts for membrane leakage, and the combined derivative is expected to be less than 1.
- Regularization: Dropout is applied during training to regularize deep SNNs, while inference uses the network without dropout.Because each SNN iteration contains multiple forward steps, error backpropagation and parameter modification occur only at the last time step.
4.1 Experimental Setup
The experiments evaluate end-to-end spike-based BP on deep SNNs across standard and neuromorphic vision datasets using architectures matched to dataset complexity.
- Training: Weights are trained end-to-end with a mini-batch spike-based BP algorithm using Poisson-distributed spike-train inputs.Static datasets use stochastic gradient descent for 150 epochs, with learning-rate reductions at epochs 70, 100, and 125.
- Datasets: The evaluation covers MNIST, SVHN, CIFAR-10, and N-MNIST, reporting top-1 test classification accuracy.N-MNIST contains ON and OFF event streams generated from static MNIST using a Dynamic Vision Sensor.
- Architectures: MNIST and N-MNIST use LeNet5-derived networks with alternating convolutional and spatial-pooling layers followed by two fully connected layers.The architecture is selected for the relatively simpler MNIST and N-MNIST benchmarks.
- Baselines: The experiments compare direct-spike-trained models with ANN-SNN conversion models on accuracy, inference speed, and energy efficiency.The compared networks differ in some architectural choices, including pooling operations.
- Input encoding: Static image pixels are converted into Poisson spike streams by comparing each pixel intensity with a uniformly distributed random number at every time step.A spike is generated when the pixel intensity exceeds the random number.
Optimal #time-steps for Training
Training performance depends on the spike-train time window: very short windows prevent learning, whereas accuracy saturates near 100 time steps.
- Training time window: 10 time-steps provide too little input information for VGG9 on CIFAR-10, leaving final-layer outputs spike-free and gradients at zero.With no output spikes, the network is not updated.
- Training time window: 35–50 time-steps allow the network to learn and converge to a reasonable point.
- Training time window: Accuracy begins saturating from 70 time-steps and training improvement completely saturates at about 100 time-steps.The same trend is observed for VGG7, ResNet7, ResNet9, and ResNet11 on SVHN and CIFAR-10.
- Training time window: The authors select 100 time-steps as the optimal training window because longer windows provide only marginal improvement.This choice is discussed in relation to the temporal precision of spike inputs.
Optimal #time-steps for Inference
A network trained with 100 time-steps retains strong inference accuracy with only 30 time-steps, peaking around 100 and degrading slightly thereafter.
- Inference time window: The CIFAR-10 VGG9 network performs very well with only 30 inference time-steps, while peak performance occurs around 100 time-steps.The network was trained using 100 time-steps before testing across inference windows from 10 to 4000.
- Inference time window: Beyond 100 inference time-steps, accuracy degrades slightly from the peak.This differs from ANN-SNN converted networks, whose accuracy keeps improving as inference time increases.
4.2 Results
The proposed direct spike-based training achieves strong classification across datasets and deep architectures, while depth benefits depend on network design.
- Evaluation protocol: Reported table accuracies are averages of maximum inference accuracies from three independent runs with different random seeds.
- Classification performance: The proposed method achieves 99.59% MNIST accuracy, matching ANN-SNN conversion and exceeding other reported SNN training schemes.
- Classification performance: The method achieves approximately 96% SVHN inference accuracy for both trained non-residual and residual SNNs.The paper notes that prior SVHN SNN inference performance had not been reported.
- Classification performance: On CIFAR-10, the proposed VGG9 reaches 90.45% accuracy, while ResNet9 remains within approximately 1.5% of its baseline ANN.Conversion models are near-lossless for VGG9 and within 0.5–1% of the baseline ANN for ResNet9.
- Accuracy and depth: SVHN accuracy improves as network depth increases, although gains become small after progressively deepening VGG architectures.The shallow LeNet-derived network achieves 92.38% inference accuracy before deeper models are evaluated.
- Accuracy and depth: Non-residual CIFAR-10 networks saturate and then degrade with additional depth, whereas residual networks maintain peak accuracy in ResNet9 and ResNet11.The paper attributes the non-residual degradation to the degradation problem and credits residual connections with maintaining accuracy.
5 Discussion
The discussion compares direct spike-based training with ANN-SNN conversion across efficiency, latency, spike counts, and accuracy. Directly trained SNNs are especially advantageous for deeper networks, although overall hardware efficiency and spike-count comparisons remain condition-dependent.
- Comparison with Relevant works: The proposed pseudo-derivative accounts for LIF leak and supports spike-based backpropagation in deep VGG and ResNet architectures.The method combines straight-through estimation with a leak-correction term and was demonstrated in networks exceeding 10 trainable layers.
- #Spikes/Inference: Direct-spike-trained SNNs have higher average layer-wise spike activity than ANN-SNN conversions, although activity decreases with network depth in both.ANN-SNN conversions reduce activity through higher neuron thresholds produced by layer-wise threshold modulation.
- #Spikes/Inference: At maximum accuracy, direct-spike-trained ResNet11 reaches peak accuracy in 100 time-steps, whereas ANN-SNN conversions require 3000-3500 time-steps.Under this condition, direct-spike-trained ResNet11 uses far fewer spikes per inference while achieving comparable accuracy.
- Inference Speedup: The proposed method achieves 8x (5x) VGG9 speedup at iso-accuracy and up to 36x (25x) at maximum accuracy versus the two ANN-SNN conversion schemes.For ResNet networks, the corresponding speedups are 6x at iso-accuracy and up to 35x at maximum accuracy.
- Complexity Reduction: For maximum accuracy, direct-spike-trained SNN AC operations are 3.61:1 for VGG9, 5.06:1 for ResNet9, and 2.09:1 for ResNet11 relative to ANN MAC operations.The reported ratios compare ANN-SNN conversion AC operations, direct-spike-trained SNN AC operations, and ANN MAC operations.
- Complexity Reduction: Estimated inference energy efficiency reaches 8.87x over ANN for VGG9 and 15.32x over ANN for ResNet11 using the proposed method.These estimates rely on MAC and AC energy differences and compare computational energy, not total system energy.
- Iso-spike Comparison for Optimal Condition: The iso-accuracy comparison can disadvantage the proposed method because direct-spike-trained SNNs are evaluated at maximum-accuracy latency while conversions often use shorter latency.The authors identify the need to determine when each approach is preferable for spike count, time-steps, and accuracy.
- Iso-spike Comparison for Optimal Condition: At equal spike counts, conversion networks can match or slightly exceed accuracy in shallower models but require 3x-10x higher inference latency.For deeper VGG9 and ResNet11, conversions achieve significantly lower accuracy even with much higher latency.
6 Conclusion
The paper presents spike-based backpropagation for deep SNN architectures and reports competitive classification on standard image-recognition tasks. It also reports expected computational energy savings on suitable neuromorphic hardware.
- Conclusion: The proposed spike-based backpropagation methodology enables deep SNNs to achieve competitive classification accuracies on standard image-recognition tasks.Experiments cover 7-11 layer VGG and ResNet architectures on MNIST, SVHN, and CIFAR-10.
- Conclusion: The method achieves the best reported classification accuracies among spike-based-learning networks on MNIST, SVHN, and CIFAR-10.The authors state that the methodology substantially reduces the performance gap between ANN and SNN models.
- Conclusion: Deep SNNs trained with the proposed method are expected to provide computational energy savings for inference on suitable neuromorphic hardware.The conclusion frames this as an expected benefit rather than a direct measurement of total system energy.