PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Residual Learning in Spiking Neural Networks

Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, Yonghong Tian

arXiv:2102.04159v6cs.NE

TL;DR

Deep SNNs are difficult to optimize, and prior Spiking ResNet designs struggle with identity mapping and gradient stability. The paper proposes SEW ResNet, proves its residual-learning properties, and evaluates it across image and neuromorphic datasets. SEW ResNet resolves the degradation problem, outperforms directly trained SNNs in accuracy and time-steps, and enables directly trained SNNs deeper than 100 layers.

  • Problem

    Deep SNNs face optimization difficulties, while Spiking ResNet can hardly implement identity mapping and suffers from vanishing or exploding gradients.

  • Method

    The paper proposes SEW residual blocks for SNNs and proves that they can implement identity mapping while overcoming vanishing and exploding gradients.

  • Results

    SEW ResNet solves the degradation problem, achieves higher performance with greater depth, and outperforms directly trained SNNs in accuracy and time-steps across ImageNet, DVS Gesture, and CIFAR10-DVS.

  • Takeaways & Limitations

    SEW ResNet provides a method for training very deep SNNs, including directly trained networks with more than 100 layers.

  • Takeaways & Limitations

    Spiking ResNet can implement identity mapping under specific IF-neuron conditions, but complex neuronal dynamics make this difficult.

Abstract

from arXiv · show

Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet. We evaluate our SEW ResNet on ImageNet, DVS Gesture, and CIFAR10-DVS datasets, and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps. Moreover, SEW ResNet can achieve higher performance by simply adding more layers, providing a simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at https://github.com/fangwei123456/Spike-Element-Wise-ResNet.

1 Introduction

Deep learning has benefited ANNs through deeper representations, while deep SNNs remain less effective on complex tasks. This motivates exploring deeper SNN structures such as ResNet.

  • Deep learning uses multiple layers to learn data representations with multiple levels of abstraction.
  • Deeper networks have advantages over shallower networks in computation cost and generalization ability.
  • SNNs offer biological plausibility, event-driven operation, and low power consumption, but remain worse than ANNs on complex tasks such as ImageNet classification.
  • Spiking ResNet mimics ANN residual blocks by replacing ReLU activation layers with spiking neurons.
  • Directly trained Spiking ResNet had not been validated to solve the degradation problem, motivating deeper SNN architectures.

2 Related Work

Deep SNNs have mainly been obtained through ANN-to-SNN conversion or surrogate-gradient backpropagation. Existing methods adapt residual structures and normalization, but converted SNNs generally require longer inference times to match ANN precision.

  • ANN-to-SNN conversion trains a ReLU ANN first, then replaces ReLU with spiking neurons and adds scaling operations.
  • Some conversion methods achieve near loss-less accuracy with VGG-16 and ResNet, but converted SNNs need longer times to rival the original ANN in precision.
  • Residual ANN structures require specific normalization during ANN-to-SNN conversion to match their activation behavior.
  • Backpropagation-based methods generally retain ResNet-like structures, with custom surrogate methods evaluated on shallow ResNets.

3 Methods

The paper analyzes why standard Spiking ResNet blocks struggle with identity mapping and gradients, then introduces SEW blocks that use element-wise spike operations to address both issues.

  • Spiking ResNet drawbacks: Spiking ResNet replaces ANN ReLU layers with spiking neurons, but its residual block can restrict identity mapping across neuron models.For LIF neurons with changing membrane time constants, selecting a threshold that guarantees spike transmission is difficult.
  • Spiking ResNet drawbacks: Even when identity mapping holds, layerwise surrogate-gradient products can vanish or explode in deep Spiking ResNet.The analysis identifies zero, finite, or unbounded gradient factors depending on the surrogate derivative.
  • SEW ResNet: SEW ResNet replaces direct addition with an element-wise function g operating on the residual spikes A_l[t] and shortcut spikes S_l[t].The residual mapping is defined as A_l[t] = SN(F_l(S_l[t])).
  • SEW ResNet: ADD and IAND implement identity mapping by setting A_l[t] ≡ 0, while AND does so by setting A_l[t] ≡ 1.These conditions can be obtained by zeroing the final batch-normalization weights and choosing the bias appropriately.
  • Downsample blocks: When block dimensions differ, SEW downsample shortcuts add a spiking neuron to the convolutional shortcut rather than using only convolution and batch normalization.Downsample shortcuts are not identity mappings because they use convolutional layers with stride greater than one.
  • SEW ResNet: SEW blocks avoid vanishing or exploding gradients because the identity-mapping gradient is constant.With AND and IAND, binary spikes also prevent the infinite-output issue associated with repeated additive growth in ANN RBA blocks.

4 Experiments

Experiments show that SEW ResNet supports deeper directly trained SNNs without the degradation and gradient-instability problems observed in Spiking ResNet, while achieving strong results across static and neuromorphic datasets.

  • ImageNet Classification: Validation accuracy decreases with depth in Spiking ResNet, whereas SEW ResNet improves training loss and accuracy as layers are added.The authors interpret this contrast as evidence that SEW ResNet addresses the degradation problem.
  • ImageNet Classification: SEW ResNet outperforms directly trained Spiking ResNet at matched ImageNet structures, sometimes using fewer simulation time-steps.ANN2SNN methods achieve higher accuracy but use 64 and 87.5 times as many time-steps as SEW ResNet.
  • ImageNet Classification: 67.04% v.s. 67.05%: SEW ResNet-34 is slightly below Spiking ResNet-34 (large), but uses 4 versus 6 time-steps and 21.8M versus 85.5M parameters.The comparison uses the same reported ImageNet accuracy metric while highlighting lower simulation and parameter costs.
  • Spiking Response Analysis: SEW firing-rate analyses examine identity-mapping behavior across networks from 18 to 152 layers and compare downsample blocks with identity-mapping regions.With ADD, lower firing rates indicate that SEW blocks are closer to implementing identity mapping, except in downsample blocks.
  • Gradient Analysis: Gradient analyses find decay in Spiking ResNet and show that SEW variants mitigate vanishing or exploding gradients under tested threshold, surrogate-gradient, and initialization settings.SEW ADD and IAND gradients can increase slowly with depth when firing rates remain adequate, while AND can suffer silence-related vanishing gradients.
  • Neuromorphic Dataset Evaluation: On neuromorphic benchmarks, SEW ADD reaches 97.92% versus 90.97% for Spiking ResNet on DVS Gesture, while CIFAR10-DVS reports 70.2% versus 67.8% with fewer time-steps.The CIFAR10-DVS comparison uses 8 versus 10 time-steps; the DVS Gesture result identifies SEW ADD as the highest-accuracy network among tested alternatives.

5 Conclusion

The paper analyzes why previous Spiking ResNet fails to realize residual learning and proposes SEW ResNet to address these issues. Experiments show that SEW ResNet supports deeper SNNs with higher accuracy.

  • SEW ResNet addresses Spiking ResNet’s difficulty implementing identity mapping and its vanishing/exploding gradient problems.
  • Experiments on ImageNet, DVS Gesture, and CIFAR10-DVS show that SEW residual blocks solve the degradation problem.
  • SEW ResNet achieves higher accuracy by simply increasing network depth, supporting the study of very deep SNNs.

A.1 Hyper-Parameters

The appendix specifies training, preprocessing, and hyper-parameter settings for the SNN experiments. It also references figures and tables documenting training behavior and configuration choices.

  • All datasets use the stated surrogate-gradient implementation, neuron parameters, SGD with momentum 0.9, reset detachment, and mixed-precision training.The surrogate gradient uses an arctangent function with α = 2; neurons use Vreset = 0 and Vth = 1.
  • ImageNet: ImageNet training samples use random 224×224 crops or horizontal flips, while testing uses resizing and central cropping.
  • Figure 10 compares training loss and training/test accuracy with and without random temporal delete.
  • Event-based datasets: DVS Gesture uses the specified AER preprocessing and random temporal delete, whereas CIFAR10-DVS uses AER preprocessing without random temporal delete.
  • Tables 7 and 8 report SNN hyper-parameters and DVS Gesture learning rates, respectively.

A.2 Random Temporal Delete

Random temporal delete is introduced as augmentation for sequential data to reduce overfitting. On DVS Gesture, training uses 12 of 16 temporal slices while inference uses the full sequence.

  • Random temporal delete randomly removes T − Ttrain temporal slices and trains on the remaining Ttrain slices.For DVS Gesture, Ttrain = 12 and T = 16; inference uses Ttest = T.
  • Figure 11 reports firing rates of the last spiking neuron and output in each 7B-Net block on DVS Gesture.
  • Random temporal delete increases training difficulty, producing higher training loss and lower training accuracy than training without it.
  • Random temporal delete produces higher test accuracy than no deletion, indicating reduced overfitting.

A.3 Firing rates on DVS Gesture

Firing-rate analysis on DVS Gesture examines when SEW blocks approach identity mapping and how block connectivity affects spike activity. Replacing AND with IAND alleviates silence and substantially improves accuracy.

  • SEW AND blocks approach identity mapping as firing rates approach 1, while other SEW functions approach identity mapping as firing rates approach 0.
  • Making every SEW block an identity would reduce 7B-Net to a simpler network that may underfit, so identity mapping is not always necessary.
  • Max pooling can increase firing rates between blocks by squeezing sparse spikes, so firing rates need not decrease with block depth.
  • 95.49% versus 70.49%: SEW IAND achieves much higher accuracy than SEW AND and alleviates the silence problem.

A.4 Gradients in Spiking ResNet with Firing Rates

The analysis links gradient behavior in Spiking ResNet to firing rates and surrogate-gradient choices. Experiments show that SEW ResNet is more robust than Spiking ResNet across surrogate functions, despite surrogate functions substantially affecting optimization and accuracy.

  • Gradient analysis: Identity mapping in Spiking ResNet can transmit activations across sequential blocks when the specified neuron and threshold conditions hold.For IF neurons with 0 < Vth ≤ 1, the passage states that successive block states remain equal to the later block output.
  • Gradient analysis: Vanishing or exploding gradients arise in Spiking ResNet through accumulative multiplication between blocks.The supplied analysis motivates examining firing rates before gradients and identifies accumulative multiplication as the source of these problems.
  • Surrogate-gradient experiments: Setting surrogate derivatives to 0/1 values can avoid gradient explosion or vanishing in the analyzed equation, but surrogate-function choice still strongly affects optimization.Rectangular and Constant 1 derivatives avoid the stated gradient problem, yet can produce lower accuracy or prevent convergence.
  • Surrogate-gradient experiments: SEW ResNet consistently achieves higher accuracy than Spiking ResNet when both use the same surrogate function.Table 9 compares the two architectures on CIFAR-10 with different surrogate functions; learning rates were fine-tuned for each function.

A.6 Reproducibility

The experiments use SpikingJelly, an open-source PyTorch-based SNN framework, and provide source code with identical random seeds across implementations.

  • Implementation: All experiments are implemented with SpikingJelly, an open-source SNN framework based on PyTorch.The framework is identified as the implementation platform for all experiments.
  • Implementation: Source code is publicly available, and identical seeds are used in all codes to support reproducibility.The supplied passage gives the repository URL and states that identical seeds maximize reproducibility.
Loading 2102.04159v6…