PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Spiking Neural Networks towards Deep Residual Learning

Yifan Hu, Lei Deng, Yujie Wu, Man Yao, Guoqi Li

arXiv:2112.08954v3cs.NEcs.LG

TL;DR

Directly trained SNNs have limited model scale and representation power, while residual designs can suffer degradation from spike-dynamics mismatches. The paper proposes MS-ResNet with membrane-based shortcuts and demonstrates scalable deep SNNs, including 76.02% ImageNet accuracy for MS-ResNet104.

  • Problem

    Directly trained SNNs have limited model scale and representation power, restricting their capabilities for complex tasks and limiting applicability to spatiotemporal data.

  • Method

    MS-ResNet removes the interblock LIF(·) to construct clean membrane-based identity shortcuts and analyzes forward propagation and gradient norms for deep SNN training.

  • Results

    76.02% accuracy on ImageNet is achieved by MS-ResNet104, alongside directly trained models reaching 482 layers on CIFAR-10 and 104 layers on ImageNet.

  • Takeaways & Limitations

    The resulting models combine deep representation capacity with sparse spiking activity and short simulation, supporting energy-efficient processing of spatiotemporal information.

Abstract

from arXiv · show

Despite the rapid progress of neuromorphic computing, inadequate capacity and insufficient representation power of spiking neural networks (SNNs) severely restrict their application scope in practice. Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks, but rarely did previous work assess their applicability to the characteristics of spike-based communication and spatiotemporal dynamics. In this paper, we first identify that this negligence leads to impeded information flow and the accompanying degradation problem in previous residual SNNs. To address this issue, we propose a novel SNN-oriented residual architecture termed MS-ResNet, which establishes membrane-based shortcut pathways, and further prove that the gradient norm equality can be achieved in MS-ResNet by introducing block dynamical isometry theory, which ensures the network can be well-behaved in a depth-insensitive way. Thus we are able to significantly extend the depth of directly trained SNNs, e.g., up to 482 layers on CIFAR-10 and 104 layers on ImageNet, without observing any slight degradation problem. To validate the effectiveness of MS-ResNet, experiments on both frame-based and neuromorphic datasets are conducted. MS-ResNet104 achieves a superior result of 76.02% accuracy on ImageNet, which is the highest to our best knowledge in the domain of directly trained SNNs. Great energy efficiency is also observed, with an average of only one spike per neuron needed to classify an input sample. We believe our powerful and scalable models will provide a strong support for further exploration of SNNs.

I. INTRODUCTION

SNNs offer spatiotemporal, event-driven communication and energy-efficiency potential, but limited model scale restricts complex-task capabilities. The paper identifies degradation in transplanted residual architectures and introduces MS-ResNet to support deeper directly trained SNNs.

  • SNNs encode information through spatiotemporal dynamics and asynchronous binary spikes, supporting event-driven communication and energy-efficient neuromorphic computing.
  • Directly trained SNNs are attractive for power-efficient neuromorphic hardware and AER-based spatiotemporal data, but their limited scale restricts complex-task performance.
  • Previous work rarely evaluates whether residual learning fits spike-based communication and SNN spatiotemporal dynamics, while direct ResNet transplantation exhibits degradation as depth increases.
  • MS-ResNet removes the interblock LIF(·) to create a clean identity shortcut, while retaining LIF(·) within residual paths.
  • MS-ResNet is extended to 482 layers on CIFAR-10 and 104 layers on ImageNet without observed degradation.
  • The SNN architecture uses membrane potentials and spike trains, with a fully connected classifier counting final feature-map spikes across the simulation period.

B. The Degradation Problem in SNNs

Depth analysis on CIFAR-10 shows that directly deepening PlainNet or using vanilla spiking ResNet does not prevent degradation. Removing interblock LIF(·) alleviates this problem, motivating the proposed architecture.

  • Directly deepening SNNs is not a trustworthy route to higher accuracy, so the experiment varies depth while holding other architectural factors controlled.
  • PlainNet accuracy begins dropping at 14 layers, while shortcut connections shift the peak only to 20 layers and severe decline occurs by 56 layers in spiking ResNet.
  • Removing interblock LIF(·) alleviates degradation within 56 layers while retaining spiking activations in residual paths.

III. SPIKING RESIDUAL BLOCKS

The proposed MS-ResNet removes the interblock LIF from the shortcut, enabling unimpeded information flow and addressing unavailing residual representations in spiking residual blocks. Its design supports block dynamical isometry and more balanced firing-pattern changes across layers.

  • Theoretical advantage: Block dynamical isometry provides gradient norm equality and helps avoid gradient vanishing or explosion.The paper analyzes this property as a theoretical advantage of MS-ResNet.
  • Architecture: Removing the interblock LIF constructs a shortcut that carries information throughout the network without gating residual-path confluences.An additional LIF remains at the residual-path top to convert messages into sparse spikes.
  • Residual representation: 84% of neurons are estimated not to change firing state, making their residual representations ineffective at the current timestep.For firing neurons, reset can also erase the residual increment, rendering the representation futile across spatial and temporal dimensions.
  • Architecture: MS-ResNet’s clean shortcut prevents unavailing residual representations regardless of whether a neuron’s firing state changes.This preserves small residual representations while avoiding the interblock gating decision.
  • Measurement: SSIM quantifies firing-pattern similarity using feature-map firing rates as pixel values, with identical images scoring +1.More circular radar curves indicate more even variation between layers.
  • Residual representation: Five layers in vanilla ResNet-56 have SSIM +1, indicating unchanged firing patterns and ineffective feature extraction at those layers.The resulting workload imbalance is followed by a dramatic information change at the 6-th layer and is mitigated in MS-ResNet.

B. Gradient Evolvement at Backpropagation

The analysis applies block dynamical isometry to residual SNNs and finds that MS-ResNet maintains gradient stability, unlike vanilla spiking ResNet.

  • Dynamical isometry evaluates whether input-output Jacobian singular values are equilibrated, providing a framework for analyzing gradient behavior.
  • Block dynamical isometry requires each block’s Jacobian to have approximately unit spectrum-moment and near-zero variance.
  • MS-ResNet can achieve gradient norm equality, whereas vanilla spiking ResNet does not achieve block dynamical isometry.
  • The LIF(·) gate is unsuitable as a serial function between blocks because its fixed hyper-parameter must match a dynamically changing firing probability.
  • MS-ResNet limits the influence of a non-optimal block to λ layers and is therefore as stable as a much shallower λ-layer network.
  • Vanilla spiking ResNet has unstable gradient flow that may manifest as degradation, while MS-ResNet avoids this drawback through its shortcut design.

C. Spike-based Convolution and an Extra LIF(·) at the Top

Removing interblock LIF(·) supports a continuous shortcut but undermines spike-based convolution; an additional top LIF(·) restores spiking inputs for classification.

  • Without interblock LIF(·), convolution layers receive continuous rather than binary-spike inputs, reducing access to spike-driven operations and input/output sparsity.
  • Removing interblock LIF(·) connects adjacent residual-path convolutions into a {CONV-BN-CONV-BN-LIF} structure that is equivalent to a single CONV5x5 in effect.
  • An extra LIF(·) at the end of the convolutional part ensures the fully connected classifier receives spiking signals, including shortcut-only information.

D. Depth Analysis on CIFAR-10

Depth experiments on CIFAR-10 show that MS-ResNet scales to very deep models without the degradation observed in vanilla spiking ResNet.

  • MS-ResNet expands to larger depths without vanilla spiking ResNet’s degradation problem while retaining better shallow-network accuracy than the W/O LIF(·) variant.
  • 91.7% test accuracy is achieved by MS-ResNet110 and 91.9% by MS-ResNet482 on CIFAR-10.
  • The limited regularization method causes overfitting, so deeper-model accuracy improves only modestly despite evidence of scalability and degradation avoidance.

A. Benchmark Results

MS-ResNet scales directly trained SNNs to deeper networks while maintaining strong accuracy across ImageNet and CIFAR10-DVS, with sparse firing activity.

  • ImageNet: MS-ResNet104 reaches 74.21% accuracy on ImageNet and 76.02% when inference images are enlarged to 288×288.MS-ResNet34 achieves 69.42%, surpassing previous directly trained SNNs at the same depth.
  • CIFAR10-DVS: MS-ResNet20 achieves a new record on CIFAR10-DVS while using about one-sixth the parameters of two prior models.
  • Firing activity: 0.225 and 0.192 are the firing rates of MS-ResNet34 and MS-ResNet104, respectively, corresponding to about one spike per neuron over five to six timesteps.The firing rate is defined as each neuron's firing probability per timestep.
  • Firing activity: MS-ResNet firing rates remain relatively stable within stages, decrease across the first three stages, and may increase in the last stage.

C. Energy Efficiency Estimation

The paper estimates MS-ResNet energy use from spike-driven operations and examines training behavior, initialization, and loss landscapes in deep models.

  • Energy estimation: MS-ResNet estimates energy from residual-path spike-based CONV layers using operation counts and 45nm technology data.Encoding and shortcut downsampling costs are excluded, accounting for 4% and 1% of FLOPs in ResNet-34 and ResNet-104, respectively.
  • Energy estimation: One ImageNet sample costs one-third to one-fifth as much energy in MS-ResNet as in an ANN with the same structure.Sparse firing and short simulation make synaptic operations comparable in count to ANN FLOPs.
  • Energy estimation: One CIFAR10-DVS frame costs one twenty-fourth as much energy in an SNN as in an ANN.MS-ResNet20 has a 22.19% average firing rate on CIFAR10-DVS.
  • Training behavior: As MS-ResNet deepens, its loss landscape develops a larger well-behaved basin, whereas vanilla spiking ResNet develops more local minima and maxima.The authors associate MS-ResNet's structural modification with avoiding ill-conditioned loss cases and degradation.
  • Initialization: Initializing γ2 to zero accelerates convergence under standard BN and TDBN, including one previously non-convergent 104-layer case.
  • Initialization: Initializing γ1 near zero is inappropriate because it can zero the first LIF output and disrupt the second BN's normalization.

C. Alternative Structures

Alternative shortcut structures can scale deeply without degradation when interblock LIF(·) is removed, while MS-ResNet is selected for computing efficiency and implementation advantages.

  • Alternative structures: Removing interblock LIF(·) lets all three alternative residual structures expand to large scale without degradation.The result emphasizes maintaining a thorough identity mapping throughout the network.
  • Performance impact: Deep MS-ResNet models achieve significant improvements in network scalability and task performance compared with the slight accuracy gains seen on ANNs.The comparison is stated at the level of design impact rather than a single benchmark value.
  • Structure selection: MS-ResNet is selected as the final structure because SR-C violates the spike-based CONV criterion, while SR-B has lower accuracy and complicates BN fusion.The choice primarily reflects computing efficiency and implementation considerations.
  • Motivation: The paper reports degradation in vanilla ResNet training and attributes it to implicit unavailing residual representation and unstable gradient norms caused by interblock LIF(·).This motivates removing the interblock membrane operation in the proposed architecture.
  • Deep scaling: 482-layer and 104-layer directly trained models are enabled on CIFAR-10 and ImageNet, respectively, without the degradation problem.The greater depth is reported to provide superior representation power.

APPENDIX A GRADIENT EVOLVEMENT AT BACKPROPAGATION

The appendix states assumptions and lemmas for analyzing gradient behavior in serial neural networks, including products of independent Jacobian matrices and conditions involving central matrices.

  • Assumption: The gradient analysis assumes that Jacobian matrices from different network blocks are independent.This assumption underlies the subsequent lemmas for general serial neural networks.
  • Gradient analysis: The multiplication lemma considers a product of independent random Jacobian matrices with compatible dimensions.The supplied passage states the matrix-product setup but does not provide the full displayed expression.
  • Gradient analysis: A second lemma applies when at most one matrix in the Jacobian sequence is not central.The condition is presented as sufficient for the stated result in the appendix.

B. ImageNet Training Details

ImageNet training follows a canonical ResNet-based setup, with stronger augmentation and regularization for MS-ResNet104 and a two-phase procedure to reduce direct-training cost.

  • Architecture: The ImageNet architecture replaces the post-Conv1 max-pooling layer with a stride-2 convolution at the beginning of Conv2.This follows the canonical ResNet architecture with the stated modification.
  • Training setup: MS-ResNet18 and MS-ResNet34 use 224x224 random crops, horizontal flips, SGD, weight decay of 1e-4, momentum of 0.9, and 125 training epochs.The initial learning rate increases linearly by 0.1 for every 256 samples of batch size.
  • MS-ResNet104: MS-ResNet104 adds AutoAugment, label smoothing, dropout, and average-pooling downsampling before shortcut stride-2 convolutions.The shortcut convolution stride is changed to 1 to avoid discarding information.
  • Two-phase training: The two-phase procedure pretrains with T=1 for 200 epochs before formal temporal training, reducing the time needed for the initial phase.The pretraining uses a large batch size and learning rate set to 0.1 per 256 batch size.
  • Training results: 74.21% accuracy is achieved after fine-tuning, compared with 71.72% for the T=1 model and 72.8% for direct temporal extension to T=6.The T=1 pretrained model occupies one-tenth of the total training time.
Loading 2112.08954v3…