Source-linked AI summary
Going Deeper in Spiking Neural Networks: VGG and Residual Architectures
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, Kaushik Roy
TL;DR
SNNs have been limited mainly to shallow architectures and simple problems, despite their potential for low-power event-driven hardware. The paper proposes an ANN-to-SNN conversion technique for deep VGG and residual networks, achieving strong results on CIFAR-10 and ImageNet while examining sparse event-driven computation. The work supports deep SNNs as a pathway toward large-scale visual recognition on neuromorphic hardware.
Problem
SNN applications have largely remained limited to shallow architectures and simple problems, leaving deep SNN performance on complex visual recognition tasks insufficiently demonstrated.
Method
The paper proposes SPIKE-NORM, an ANN-to-SNN conversion technique that normalizes weights using maximum spike-based inputs sequentially across network layers.
Results
The technique statistically outperforms state-of-the-art conversion methods and achieves competitive classification results across CIFAR-10 and ImageNet for VGG and ResNet architectures.
Takeaways & Limitations
Deep SNNs can provide competitive accuracies on complex datasets and may support large-scale visual recognition using low-power neuromorphic hardware.
Takeaways & Limitations
The conversion incurs time-step delay overhead, and exact delay comparisons with ANN feedforward passes remain outside the article's scope.
Abstract
from arXiv · showhide
Over the past few years, Spiking Neural Networks (SNNs) have become popular as a possible pathway to enable low-power event-driven neuromorphic hardware. However, their application in machine learning have largely been limited to very shallow neural network architectures for simple problems. In this paper, we propose a novel algorithmic technique for generating an SNN with a deep architecture, and demonstrate its effectiveness on complex visual recognition problems such as CIFAR-10 and ImageNet. Our technique applies to both VGG and Residual network architectures, with significantly better accuracy than the state-of-the-art. Finally, we present analysis of the sparse event-driven computations to demonstrate reduced hardware overhead when operating in the spiking domain.
1 INTRODUCTION
SNNs replace analog neural processing with binary spike-based information and temporal operation. Their limited performance on complex datasets such as CIFAR-10 and ImageNet has required new training mechanisms.
- SNNs process and transmit binary spike-based information rather than analog values used by standard ANNs.
- Limited SNN performance on CIFAR-10 and ImageNet is attributed to their temporal information processing and difference from ANN operation.
2 RELATED WORK
SNN training approaches include supervised and unsupervised mechanisms, with conversion-based supervised methods leveraging standard ANN training to address performance limitations.
- SNN training is broadly divided into supervised and unsupervised learning categories.
- Unsupervised mechanisms such as STDP remain attractive for low-power local learning but are outperformed by supervised networks on MNIST.
- Some supervised approaches train ANNs with backpropagation before converting them to SNNs to leverage stronger ANN performance.
3 MAIN CONTRIBUTIONS
The paper investigates deep SNNs for complex visual recognition using ANN-to-SNN conversion across VGG and residual architectures. It reports competitive or strong classification results and increasing sparsity with network depth.
- Deep SNN proof-of-concept experiments extend from 16 to 34 layers and cover CIFAR-10 and ImageNet.
- 8.45% classification error on CIFAR-10 is reported as the best-performing result for an SNN at the time.
- 30.04% top-1 error and 10.99% top-5 error are reported for VGG-16 on the ImageNet 2012 validation set.
- 12.54% CIFAR-10 classification error and 34.53% top-1 and 13.67% top-5 ImageNet validation errors are reported for ResNet architectures.
- SNN network sparsity significantly increases as network depth increases, motivating event-driven operation to reduce compute overhead.
4 PRELIMINARIES
The paper represents ANN inputs as time-dependent spike trains and relates ReLU activations to integrate-and-fire neurons. Its conversion preliminaries address thresholds, biases, normalization, and pooling for SNN operation.
- 4.1 Input and Output Representation: Rate encoding makes the average number of input spikes over a time window approximately proportional to the original ANN input magnitude.A Poisson process generates spikes by comparing random values with pixel intensities.
- 4.2 ANN and SNN Neural Operation: ReLU neurons are used for ANN-to-SNN conversion because they are functionally equivalent to non-leaky, non-refractory integrate-and-fire neurons.
- 4.2 ANN and SNN Neural Operation: The integrate-and-fire neuron integrates incoming spikes, fires when membrane potential crosses a threshold, and resets its membrane potential to zero.
- 4.2 ANN and SNN Neural Operation: For positive synaptic weights, output spiking rate is proportional to input spiking rate, with the factor determined by weight-to-threshold ratio.
- 4.2 ANN and SNN Neural Operation: Choosing the threshold-to-weight ratio appropriately is essential to limit classification-accuracy loss during ANN-to-SNN conversion.
- 4.2 ANN and SNN Neural Operation: Bias-free neural units simplify conversion but prevent Batch Normalization, so dropout is used as the regularizer.
- 4.2 ANN and SNN Neural Operation: Spatial averaging is preferred to max-pooling because binary SNN activations make max-pooling cause significant information loss.
5 DEEP CONVOLUTIONAL SNN ARCHITECTURES: VGG
The paper proposes SPIKE-NORM, which accounts for actual SNN activity during ANN-SNN conversion by sequentially balancing layer weights and thresholds. The method addresses information loss arising when ANN activations do not represent binary spike integration.
- 5.1 Proposed Algorithm: SPIKE-NORM: Prior data-based normalization sets a neuron threshold to the maximum corresponding ANN ReLU activation after one pass through the training set.This approach was previously evaluated on three-layer fully connected and convolutional MNIST architectures.
- 5.1 Proposed Algorithm: SPIKE-NORM: SPIKE-NORM balances each layer’s threshold using the actual operation of the SNN during ANN-SNN conversion.The technique sequentially normalizes network weights rather than relying only on ANN activation maxima.
- 5.1 Proposed Algorithm: SPIKE-NORM: The conversion begins with Poisson spike trains and records each layer’s maximum weighted spike-input activation over a sufficiently long time window.This maximum SNN activation determines the first layer’s normalization and threshold-balancing factor.
- 5.1 Proposed Algorithm: SPIKE-NORM: Representative output spikes from each normalized layer become the input spike stream for sequential processing of the next layer.The procedure continues layer by layer through the network.
2 // Set input of 1st layer equal to spike train
Algorithm 1 implements SPIKE-NORM by forwarding spike trains through each layer and adjusting thresholds from the maximum weighted spike-based activation. It then passes each layer’s output spike train to the next layer.
- 2 // Set input of 1st layer equal to spike train: The algorithm initializes the first layer with the input Poisson spike train and iterates over all network layers and time steps.Each layer is evaluated using its membrane potential and spiking threshold.
- 2 // Set input of 1st layer equal to spike train: For each layer, SPIKE-NORM computes the maximum weighted spike-input activation using the dot product between weights and layer inputs.This quantity determines the threshold-balancing factor.
- 2 // Set input of 1st layer equal to spike train: The algorithm records the current layer’s output spike train as the input to the next layer.This preserves the sequential, layerwise nature of the conversion procedure.
- 2 // Set input of 1st layer equal to spike train: Algorithm 1 is identified as SPIKE-NORM and outputs weight-normalization or threshold-balancing factors for each neural layer.The factors are represented as vth,norm[i].
6 EXTENSION TO RESIDUAL ARCHITECTURES
The paper extends ANN-SNN conversion to residual architectures by analyzing shortcut-path constraints and threshold balancing. It identifies junction ReLUs, path alignment, and early-layer activation scaling as important design considerations.
- 6 EXTENSION TO RESIDUAL ARCHITECTURES: Residual units contain parallel identity and non-identity paths, with the latter using two spatial convolutions and an intermediate ReLU.The paper focuses on identity shortcut connections.
- 6 EXTENSION TO RESIDUAL ARCHITECTURES: Figure 2 presents the basic ResNet unit, conversion constraints, and typical maximum SNN activations for a junction-ReLU residual network.The plotted case uses unequal spiking thresholds between identity and non-identity paths and is not representative of equal-threshold operation.
- 6 EXTENSION TO RESIDUAL ARCHITECTURES: Removing ReLUs at residual junctions slightly improves CIFAR-10 classification accuracy but complicates near-lossless ANN-SNN conversion.The paper attributes observed degradation mainly to the absence of junction ReLUs because converted integrate-and-fire neurons introduce temporal delays.
- 6 EXTENSION TO RESIDUAL ARCHITECTURES: All fan-in layers to a junction must be threshold-balanced consistently so that the next layer receives appropriately rate-encoded spike information.The non-identity-path threshold depends on activity at the preceding junction.
- 6 EXTENSION TO RESIDUAL ARCHITECTURES: After a few initial layers, maximum SNN activations decay toward unity because of identity mapping, while initial junction balancing factors can be substantially above one.The paper identifies these high initial factors as a possible source of converted-SNN classification degradation.
7 EXPERIMENTS
Experiments evaluate VGG and residual SNN conversions on CIFAR-10 and ImageNet, reporting conversion accuracy, architectural constraints, convergence, and sparse-event computation.
- Experimental setup: Experiments use CIFAR-10 testing data and the 50,000-image ImageNet 2012 validation set, with top-1 and top-5 error rates reported for ImageNet.ImageNet uses 224×224 input crops and single-crop testing results.
- VGG architectures: 0.15% error increment was achieved for VGG-16 on CIFAR-10 with SPIKE-NORM, compared with 0.24% using ANN-model-based weight normalization.The baseline ANN error rate was 8.3%.
- VGG architectures: 0.57% was the margin by which SNN-model-based weight normalization minimized ImageNet ANN-to-SNN accuracy loss on the more complex benchmark.The comparison uses single-crop ImageNet validation results.
- Residual architectures: 34.53% top-1 error was reported for the converted ResNet-34 SNN on ImageNet at 2500 timesteps, versus 29.31% for its baseline ANN.The deeper residual model replaces the initial 7×7 convolution with three 3×3 layers, two of them plain non-residual units.
- Sparse-event computation: Neuron spiking activity becomes sparser with increasing network depth, while the VGG and ResNet SNN-to-ANN operation ratios are 1.975 and 2.4, respectively.The ratios count only ReLU/IF spiking-neuron activations; energy benefits require fewer than one average spike per neuron over the inference window.
8 CONCLUSIONS AND FUTURE WORK
The work provides evidence that SNNs can exhibit similar computing power to ANNs and may support large-scale visual recognition on low-power neuromorphic hardware, while several performance gaps remain open.
- The work provides evidence that SNNs exhibit similar computing power as their ANN counterparts.
- The findings potentially support using SNNs for large-scale visual recognition on low-power neuromorphic hardware.
- Batch-Normalization remains difficult to use because bias-less neural units constrain ANN training without it.
- Future work includes bias-enabled spiking neurons, alternative neural functionalities, and lower-loss ANN-SNN conversion for deeper ResNet architectures.