PaperlayerView exact revision on arXiv

Source-linked AI summary

Going Deeper With Directly-Trained Larger Spiking Neural Networks

Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, Guoqi Li

arXiv:2011.05280v2cs.NEcs.AI

TL;DR

Directly trained SNNs are limited by unstable gradients, firing-rate imbalance, and the shallow depth of existing methods, while ANN conversion requires long coding windows. The paper introduces STBP-tdBN with modified shortcut connections and reports deep, directly trained SNNs with high performance across static and neuromorphic datasets, including ImageNet.

  • Problem

    Direct SNN training is constrained by gradient vanishing or explosion and inappropriate input-threshold balance, while ANN-SNN conversion usually requires hundreds or thousands of timesteps.

  • Method

    The paper combines spatio-temporal backpropagation with threshold-dependent batch normalization and modified shortcut connections to train deep SNNs directly.

  • Results

    The method extends directly trained SNNs from fewer than 10 to more than 50 layers and achieves high accuracy on CIFAR-10 and ImageNet, with state-of-the-art performance on neuromorphic datasets.

  • Takeaways & Limitations

    The scheme provides a feasible directly trained approach for complex large-scale classification while retaining spike-based communication efficiency and supporting neuromorphic-hardware implementation.

  • Takeaways & Limitations

    ANN-SNN conversion methods usually require hundreds or thousands of timesteps, while directly trained SNN neurons can otherwise receive too few or continuously firing spikes.

Abstract

from arXiv · show

Spiking neural networks (SNNs) are promising in a bio-plausible coding for spatio-temporal information and event-driven signal processing, which is very suited for energy-efficient implementation in neuromorphic hardware. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. Currently, there are two main routes to explore the training of deep SNNs with high performance. The first is to convert a pre-trained ANN model to its SNN version, which usually requires a long coding window for convergence and cannot exploit the spatio-temporal features during training for solving temporal tasks. The other is to directly train SNNs in the spatio-temporal domain. But due to the binary spike activity of the firing function and the problem of gradient vanishing or explosion, current methods are restricted to shallow architectures and thereby difficult in harnessing large-scale datasets (e.g. ImageNet). To this end, we propose a threshold-dependent batch normalization (tdBN) method based on the emerging spatio-temporal backpropagation, termed "STBP-tdBN", enabling direct training of a very deep SNN and the efficient implementation of its inference on neuromorphic hardware. With the proposed method and elaborated shortcut connection, we significantly extend directly-trained SNNs from a shallow structure ( < 10 layer) to a very deep structure (50 layers). Furthermore, we theoretically analyze the effectiveness of our method based on "Block Dynamical Isometry" theory. Finally, we report superior accuracy results including 93.15 % on CIFAR-10, 67.8 % on DVS-CIFAR10, and 67.05% on ImageNet with very few timesteps. To our best knowledge, it's the first time to explore the directly-trained deep SNNs with high performance on ImageNet.

Introduction

SNNs offer energy-efficient spatio-temporal processing but are difficult to train deeply because of unstable gradients and firing-rate imbalance. The paper proposes STBP-tdBN with modified shortcuts, extending directly trained SNNs to deep architectures and evaluating them across static and neuromorphic datasets.

  • SNNs support continuous spatio-temporal processing through intrinsic neuronal dynamics and event-driven communication, with potential for lower energy consumption.
  • ANN-to-SNN conversion usually requires hundreds of timesteps, whereas direct training can reduce timesteps but faces unstable gradient propagation.
  • Directly trained SNNs must balance neuronal thresholds and inputs because too few spikes suppress information, while continuous firing reduces sensitivity to input changes.
  • The paper proposes threshold-dependent batch normalization to address gradient vanishing or explosion and adjust firing rates, together with modified residual shortcuts for deep SNNs.
  • 50-layer directly trained SNNs are evaluated on CIFAR-10, ImageNet, DVS-Gesture, and DVS-CIFAR10.
  • The reported models achieve competitive CIFAR-10 and ImageNet results with no more than 6 timesteps and state-of-the-art performance on DVS-Gesture and DVS-CIFAR10.

Related Work

Prior work either converts ANNs into SNNs or trains SNNs with gradient descent. Conversion methods scale well but require long temporal windows, while direct methods use fewer timesteps yet have faced depth and optimization challenges.

  • ANN-SNN conversion transforms ReLU outputs into binary spikes and achieves competitive large-scale-dataset accuracy relative to ANNs.
  • Conversion methods ignore rich temporal dynamics and usually require hundreds or thousands of timesteps to approach pretrained-ANN accuracy.
  • Gradient-descent methods train SNNs through error backpropagation and have achieved high performance on CIFAR-10 and neuromorphic datasets.
  • STBP uses an iterative LIF model and supports direct SNN training in ANN-oriented programming frameworks while accelerating training.
  • Block Dynamical Isometry analyzes blockwise first and second moments to explain how network factors affect gradient distributions.
  • Normalization methods such as batch, group, and layer normalization are established tools for stabilizing and deepening neural-network training.

Materials and Methods

STBP-tdBN combines spatio-temporal backpropagation with threshold-dependent normalization and residual architecture choices for directly training deep SNNs. The method normalizes inputs across relevant dimensions, fuses normalization into inference weights, and propagates gradients through iterative LIF dynamics.

  • Iterative LIF model: The iterative LIF model represents membrane decay, accumulated presynaptic input, threshold firing, and reset dynamics for SNN computation.
  • Iterative LIF model: The iterative LIF formulation makes forward and backward propagation over spatial and temporal dimensions compatible with general machine-learning frameworks.
  • Threshold-dependent batch normalization: tdBN normalizes pre-synaptic feature inputs along the channel dimension using statistics estimated over the mini-batch across T timesteps.
  • Inference implementation: During inference, moving averages estimate dataset-level statistics, and batchnorm-scale-fusion removes BN operations while preserving full-spiking layer-by-layer propagation.
  • Threshold-dependent batch normalization: Unlike standard BN, tdBN normalizes across temporal and spatial dimensions and sets normalized variance according to the firing threshold through αVth.
  • Overall training algorithm: STBP-tdBN trains deep SNNs from scratch using iterative LIF dynamics, decoding-layer outputs, softmax, and cross-entropy loss.
  • Overall training algorithm: Spatio-temporal backpropagation computes gradients across both spatial network structure and temporal steps, using an approximate derivative for nondifferentiable spikes.

Theoretical Analysis

The analysis uses Block Dynamical Isometry to explain how tdBN stabilizes gradients and scaling factors regulate membrane-potential and firing-rate distributions in deep SNNs.

  • Gradient stability: tdBN is analyzed as a means to alleviate gradient vanishing or explosion during directly trained SNN optimization.The analysis combines gradient-norm theory with spiking-neuron properties.
  • Gradient stability: Block Dynamical Isometry evaluates blockwise Jacobian moments to characterize whether gradients avoid vanishing or explosion.A network achieves this condition when the relevant block Jacobian statistics remain appropriately controlled.
  • Gradient stability: When τdecay = 0 and input and output second moments are fixed to V_th^2, the theory predicts gradient propagation without vanishing or explosion.The proposition simplifies temporal gradient analysis by setting membrane decay to zero.
  • Gradient stability: In a 20-layer plain SNN on CIFAR-10, τdecay = 0 produced a steadier gradient-norm curve during early training, supporting the analysis.The experiment measured mean gradient norms across layers during the first 1/6 epoch.
  • Firing-rate regulation: Scaling factors α and V_th normalize pre-activations toward N(0, V_th^2), balancing membrane potential against threshold.Theorem 2 relates pre-activation distributions to membrane-potential distributions, and Fig. 3 reports their high similarity.
  • Firing-rate regulation: Firing rates depend on pre-activation variance: insufficient variance can make spikes disappear, whereas excessive variance can make neurons fire continuously and increase computation.The scaling factors are used to control these regimes in deep SNNs.

Deep Spiking Residual Network

The proposed deep spiking residual network replaces ANN components with LIF neurons and tdBN, while modifying shortcut connections for deep SNN training.

  • Basic block: The shortcut connection is modified to improve the suitability of residual learning for deep SNNs.The design adds a tdBN layer before the final residual addition.
  • Basic block: The architecture replaces conventional BN and ReLU components with tdBN and LIF models in residual basic blocks.The basic block uses two 3 × 3 convolution layers and adds tdBN before the final addition.
  • Network architecture: The architecture begins with a stride-2 convolutional encoding layer, propagates spikes through residual blocks, and ends with pooling or a fully connected softmax decoder.The channel count doubles when feature maps are halved, following the ResNet-ANN pattern.

Experiment

The experiments evaluate deep residual SNNs on static image and neuromorphic benchmarks, reporting high accuracy with few timesteps. Results include competitive ImageNet performance and state-of-the-art results on DVS datasets.

  • Experimental scope: The directly-trained SNNs are evaluated on CIFAR-10, ImageNet, DVS-Gesture, and DVS-CIFAR10 using residual architectures and few timesteps.The experiments cover both static non-spiking and neuromorphic datasets, with comparisons summarized in Tables 1–3.
  • Static datasets: 93.15% top-1 accuracy is achieved on CIFAR-10 with only 6 timesteps.The result is reported as state-of-the-art and is associated with reduced latency and computation cost compared with other SNN models.
  • Static datasets: 67.05% top-1 accuracy is achieved on ImageNet with a large ResNet-34 using just 6 timesteps.A ResNet-50 also reaches 64.88% top-1 accuracy.
  • Neuromorphic datasets: 96.87% accuracy is achieved on DVS-Gesture with ResNet-17 and 40 timesteps.The paper reports this as state-of-the-art for directly-trained SNNs on DVS-Gesture.
  • Neuromorphic datasets: 67.8% accuracy is achieved on DVS-CIFAR10 with ResNet-19 in 10 timesteps.The paper reports this as the best performance in its comparison.

Conclusion

The paper introduces a normalization method and modified shortcut connections for directly training very deep SNNs, then evaluates them on static-image and neuromorphic datasets. It reports high accuracy with low inference latency and state-of-the-art neuromorphic performance.

  • The proposed normalization method enables directly-trained deep SNNs with high performance.The method is presented as balancing input stimulus and neuronal threshold during training to facilitate convergence.
  • Modified shortcut connections extend directly-trained SNNs from fewer than ten layers to more than fifty layers.
  • The model achieves high accuracy on CIFAR-10 and ImageNet with significantly small inference latency.
  • The work reports a directly-trained, very deep SNN on ImageNet for the first time to the authors’ knowledge.
  • On neuromorphic datasets, the model efficiently processes temporal-spatial information and achieves state-of-the-art performance.
  • The scheme preserves spike-based communication efficiency and supports more complex large-scale classification tasks for neuromorphic hardware implementations.

Supplementary Material

The supplementary material specifies threshold-dependent batch normalization for training and inference. It identifies the tensor inputs, normalized outputs, trainable parameters, and inference-time convolutional quantities.

  • During training, tdBN maps weighted outputs from the previous layer to normalized pre-synaptic inputs for the next layer.The tensors use dimensions [Timestep, Batch size, Cout, H, W].
  • The training procedure uses trainable channel parameters λ and β, threshold Vth, and hyper-parameter α.The parameters are initialized as λ ← 1 and β ← 0.
  • Training computes channel-wise means and variances from the weighted outputs before normalization.
  • During inference, tdBN receives convolution kernels, biases, and spiking outputs, and produces pre-synaptic inputs for the next layer.The inference tensors retain timestep, batch, channel, height, and width dimensions.

B Proofs of Theorems

The proofs analyze tdBN-equipped SNNs through block Jacobians, timestep-wise gradient propagation, and iterative LIF dynamics. The stated result is Block Dynamic Isometry, under which gradient magnitudes avoid sharp growth or decay with depth.

  • B Proofs of Theorems: The analysis represents an SNN as blocks between tdBN layers and studies each block’s input-output Jacobian across timesteps.
  • B Proofs of Theorems: Under fixed second moments for block inputs and outputs, the analysis derives conditions associated with the threshold Vth.
  • B Proofs of Theorems: For each timestep, tdBN-equipped SNNs can achieve Block Dynamic Isometry, so gradient norms do not sharply increase or diminish as depth grows.
  • B Proofs of Theorems: The resulting Jacobian condition means the SNN can avoid gradient vanishing or explosion.
  • B Proofs of Theorems: The proof considers the iterative LIF model, whose membrane potential combines decayed previous potential with current input and produces binary spikes at threshold.
  • B Proofs of Theorems: The reported similarity between pre-activation and membrane-potential distributions supports the theorem’s assumptions and analysis.

C Details of Experiments

The experiments use ResNet-based SNN architectures across static-image and neuromorphic datasets with specified preprocessing, optimization, acceleration, and batch-size settings.

  • Network architectures: The experiments evaluate ResNet-19 on CIFAR-10 and DVS-CIFAR10, ResNet-34 and ResNet-50 on ImageNet, and ResNet-17 on DVS-Gesture.
  • Dataset introduction and pre-processing: CIFAR-10 contains 50,000 training images and 10,000 testing images across ten classes, with random cropping, flipping, and channel normalization.
  • Dataset introduction and pre-processing: ImageNet contains 1.28 million training images and 50,000 validation images, processed with random crops, flips, and general normalization.
  • Dataset introduction and pre-processing: DVS-Gesture contains 11 classes of event streams, downsized to 32 × 32 and sampled every 30 ms with timestep T = 40.
  • Dataset introduction and pre-processing: DVS-CIFAR10 contains 10 classes with 9,000 training images and 1,000 testing images, downsampled to 42 × 42 with spikes accumulated every 5 ms.
  • Optimization and acceleration: All experiments use SGD with initial learning rate 0.1 and momentum 0.9, alongside mixed-precision training and, where needed, multi-GPU synchronized batch normalization.

D Analysis of Computation Reduction

SNN inference reduces computation relative to ANNs by replacing most multiply-accumulate work with accumulate operations and exploiting sparse, event-driven spikes. The analyzed models maintain sparse spikes across few timesteps, supporting computation and runtime advantages despite encoding and decoding multiplications.

  • SNNs mainly use accumulate operations, whereas ANNs rely on multiply-accumulate operations.SNN synaptic computation is dominated by additions because weights are accumulated rather than repeatedly multiplied.
  • Event-driven processing avoids synaptic computation when no spike is received, reducing operations through spike sparsity.These savings can offset the cost of evaluating SNNs across T timesteps.
  • ResNet-19 and ResNet-34(large) exhibit surprisingly sparse spikes with few timesteps, while firing rates decrease slowly in deeper layers.These measurements are used to estimate the AC operations required during inference.
  • Sparse spikes produce significant computation reduction and energy efficiency, and event-driven neuromorphic execution may also improve runtime over same-architecture ANNs.The comparison concerns single feed-forward inference, with fewer timesteps contributing to the runtime advantage.
Loading 2011.05280v2…