PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Training for Spiking Neural Networks: Faster, Larger, Better

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi

arXiv:1809.05793v2cs.NE

TL;DR

Spiking neural networks lacked effective learning algorithms and efficient programming frameworks needed to match artificial neural networks. This paper combines NeuNorm, direct training, faster coding, and an explicitly iterative LIF implementation, achieving best reported neuromorphic-dataset accuracy and comparable CIFAR10 performance with 90.53% accuracy.

  • Problem

    Spiking neural networks had not demonstrated performance comparable to artificial neural networks because effective learning algorithms and efficient programming frameworks were lacking.

  • Method

    The paper combines NeuNorm-based neuronal normalization, direct training, optimized rate coding, and an explicitly iterative LIF model in a PyTorch-compatible framework.

  • Results

    The method achieved best accuracy on neuromorphic datasets and comparable accuracy with ANNs on non-spiking CIFAR10, reaching 90.53% versus 75.42% for prior direct training.

  • Takeaways & Limitations

    Implementing directly trained SNNs in a mainstream machine-learning framework could facilitate further SNN development.

Abstract

from arXiv · show

Spiking neural networks (SNNs) that enables energy efficient implementation on emerging neuromorphic hardware are gaining more attention. Yet now, SNNs have not shown competitive performance compared with artificial neural networks (ANNs), due to the lack of effective learning algorithms and efficient programming frameworks. We address this issue from two aspects: (1) We propose a neuron normalization technique to adjust the neural selectivity and develop a direct learning algorithm for deep SNNs. (2) Via narrowing the rate coding window and converting the leaky integrate-and-fire (LIF) model into an explicitly iterative version, we present a Pytorch-based implementation method towards the training of large-scale SNNs. In this way, we are able to train deep SNNs with tens of times speedup. As a result, we achieve significantly better accuracy than the reported works on neuromorphic datasets (N-MNIST and DVS-CIFAR10), and comparable accuracy as existing ANNs and pre-trained SNNs on non-spiking datasets (CIFAR10). {To our best knowledge, this is the first work that demonstrates direct training of deep SNNs with high performance on CIFAR10, and the efficient implementation provides a new way to explore the potential of SNNs.

Introduction

SNNs offer power-efficient neuromorphic processing through event-driven spike communication, but their complex dynamics, nondifferentiability, and limited training frameworks have hindered performance and scalability. This paper proposes algorithmic and programming improvements for faster direct training of deeper, larger SNNs with stronger accuracy.

  • Motivation: SNNs use spatio-temporal dynamics and binary spike signals, enabling event-driven computation on specialized neuromorphic hardware for power-efficient processing.Examples include SpiNNaker, TrueNorth, and Loihi.
  • Motivation: SNNs have not demonstrated performance comparable to ANNs because effective learning algorithms and efficient programming frameworks remain lacking.These limitations constrain network scale and application spectrum.
  • Challenges: Training large-scale SNNs is challenged by complex spatial and temporal neural dynamics and discrete, non-differentiable spiking activity that impede backpropagation.Neural activities propagate across layers while also affecting states over time.
  • Challenges: Existing programming platforms provide limited support for deep SNN training, forcing researchers to build models from scratch while training speed is usually slow.Several platforms focus on simulating biological features rather than training deep SNNs.
  • Contributions: The paper proposes NeuNorm, improved rate coding, and an explicitly iterative LIF model to enable faster direct training of deeper SNNs in Pytorch.Compared with Matlab running time, the implementation achieves tens of times speedup and reports stronger neuromorphic-dataset accuracy plus comparable non-spiking-dataset accuracy.

Related work

Prior SNN learning approaches include unsupervised, indirect supervised, and direct supervised methods, but most remain limited to shallow networks or toy datasets. Existing normalization techniques and programming frameworks also face SNN-specific dynamics, hardware, or scalability constraints.

  • Learning algorithm for deep SNNs: Most SNN learning methods remain limited to networks with fewer than 4 layers or toy datasets such as MNIST and Iris.
  • Learning algorithm for deep SNNs: STDP is biologically plausible, but its lack of global information hinders convergence on large models and complex datasets.
  • Learning algorithm for deep SNNs: ANN-to-SNN conversion is currently the most successful approach for large-scale SNNs, but trains ANNs rather than revealing how SNNs learn directly.
  • Learning algorithm for deep SNNs: Direct supervised learning trains SNNs without conversion, and spatiotemporal backpropagation achieved state-of-the-art accuracy on MNIST and N-MNIST.
  • Normalization: Normalization methods successful in ANNs are unsuitable for SNNs because of complex neural dynamics, binary spiking representations, and hardware-incompatible lateral operations.
  • SNN programming frameworks: NEURON and Genesis prioritize biologically realistic simulation, whereas BRIAN2 and NEST target larger-scale SNN simulation with biological features.

Approach

The approach converts the LIF neuron into an explicitly iterative form for tractable implementation in mainstream ML frameworks, introduces NeuNorm to stabilize neuronal selectivity, and optimizes rate coding for faster response. It also provides a complete PyTorch training procedure with improved input and output representation through shorter coding windows.

  • Iterative LIF: The LIF model is converted from implicit continuous-domain equations into an explicitly iterative update suitable for mainstream ML frameworks.The formulation preserves updating, firing, and resetting, with membrane potential reset after threshold crossing.
  • Training implementation: A concise PyTorch pseudocode procedure specifies state updates from previous potential, previous spike output, current input spikes, and synaptic weights.The procedure returns the next membrane potential and spike output after applying decay, input accumulation, threshold firing, and reset dynamics.
  • NeuNorm: NeuNorm normalizes input strength across feature maps using auxiliary neurons and moving-average firing-rate statistics.Auxiliary neurons receive lateral inputs and use trainable weights to control stimulus strength emitted to the next layer.
  • Rate coding: Rate coding is optimized on both encoding and decoding sides to reduce the need for long simulation windows.The method addresses input sampling error and output representation precision through modified coding and voting-based decoding.
  • Rate coding: The coding design extends SNN representation capability by maintaining higher precision at the first and last layers despite lower internal precision.This combines input-side encoding improvements with output-side voting to support convergence and performance.

Experiment

Experiments evaluate training acceleration and application accuracy on neuromorphic N-MNIST and DVS-CIFAR10 datasets and non-spiking CIFAR10. The results show faster large-scale SNN training, short simulation windows, and strong accuracy, including 99.53% on N-MNIST and 60.5% on DVS-CIFAR10.

  • Experiment: Experiments assess training acceleration and application accuracy on N-MNIST, DVS-CIFAR10, and CIFAR10.Dataset preparation, preprocessing, training details, and parameter configurations are provided in the Appendix.
  • Network scale: 8 layers demonstrate direct and effective learning for larger-scale SNNs, beyond prior direct-training structures usually limited to 2–4 layers.The paper identifies this as the first implementation of direct effective learning for larger-scale SNNs.
  • Runtime: Tens of times acceleration is achieved by the Pytorch implementation on all three datasets.The improvement is attributed to specialized Pytorch convolution optimizations that are not well supported in most existing SNN platforms.
  • Simulation length: 4–8 simulation steps achieve satisfactory CIFAR10 performance without much accuracy degradation.Even one-step simulation achieves not bad performance with faster response and lower energy.
  • Accuracy evaluation: 99.53% accuracy is achieved on N-MNIST with NeuNorm, while DVS-CIFAR10 reaches 60.5% and NeuNorm reduces best-accuracy training from 157 to 103 epochs.Without NeuNorm, the best accuracy is 58.1% after 157 epochs; NeuNorm also speeds convergence.

Conclusion

The paper presents direct training for deeper and larger SNNs, introducing NeuNorm, optimized rate coding, and an explicitly iterative LIF model for PyTorch implementation.

  • Conclusion: The proposed direct training algorithm targets deeper and larger SNNs with high performance.
  • Conclusion: NeuNorm normalizes neuronal activities to improve SNN performance.
  • Conclusion: The method optimizes rate coding from both encoding and decoding aspects.
  • Conclusion: The original continuous LIF model is converted into an explicitly iterative version for PyTorch implementation.
  • Conclusion: The approach enables tens of times training accelerations and larger network scale.

A Dataset Introduction · Neuromorphic dataset

The paper introduces N-MNIST and DVS-CIFAR10 as neuromorphic datasets represented by event-based spike trains derived from static handwritten-digit and natural-image data. These datasets encode intensity changes and motion-dependent spike patterns with distinct spatial, temporal, and sampling characteristics.

  • Neuromorphic dataset: N-MNIST converts frame-based MNIST handwritten digits into DVS event streams by scanning static images and detecting pixel-intensity changes.Each sample is generated through directional scanning of the static image.
  • Neuromorphic dataset: N-MNIST records two spike-event types, On-event and Off-event, corresponding to increasing and decreasing pixel intensity.The two event polarities arise because intensity changes have two directions.
  • Neuromorphic dataset: N-MNIST samples have size 34 × 34 × 2 × T, where T is the temporal length of the spatio-temporal spike pattern.The pixel dimension expands to 34×34 because images shift during the moving process.
  • Neuromorphic dataset: DVS-CIFAR10 converts 10000 static CIFAR10 images into spike trains, with 1000 images per class and 128 × 128 image resolution.The dataset uses different DVS types and movement paths to generate its spike trains.
  • Neuromorphic dataset: DVS-CIFAR10 contains imbalanced spike events and larger image resolution because it uses different DVS types and movement paths.The paper adopts different parameter configurations for this dataset.
  • Neuromorphic dataset: DVS-CIFAR10 has 6 times less samples than the original CIFAR10 dataset, with 9000 samples randomly chosen for use.The passage states that the dataset contains 10000 images before describing the random choice of 9000.
  • Neuromorphic dataset: Figure 7 illustrates neuromorphic datasets using 32 × 32 and 128 × 128 samples from N-MNIST and DVS-CIFAR10, respectively.Each sub-picture shows a 5 ms-width spike train.

Non-spiking dataset

The non-spiking dataset is CIFAR10, comprising 60,000 color images across 10 classes, with 50,000 used for training and 10,000 for testing.

  • Dataset composition: CIFAR10 contains 60,000 color images from 10 classes, divided into 50,000 training images and 10,000 test images.Each image has size 32 × 32.

B Training setting … Parameter configuration

The training setting applies dataset-specific preprocessing and optimization choices, while implementation uses standard convolution operations and reports simulation parameters for AlexNet and CIFARNet. Neuromorphic inputs are temporally accumulated, DVS-CIFAR10 is spatially down-sampled, and CIFAR10 images receive augmentation and normalization.

  • Data pre-processing: N-MNIST and DVS-CIFAR10 spike trains are accumulated within every 5 ms.DVS-CIFAR10 is additionally down-sampled from 128 ×128 to 42×42 using stride = 3 and padding = 0.
  • Data pre-processing: CIFAR10 preprocessing crops and flips images along each RGB channel before rescaling with the global mean pixel intensity.The supplied passage ends during the rescaling description.
  • Optimizer: Adam is used for neuromorphic datasets, whereas CIFAR10 uses stochastic gradient descent with initial learning rate r = 0.1 and momentum 0.9.For CIFAR10, r decays to 0.1r over each 40 epochs.
  • Optimizer: All adopted convolution operations use officially provided conv functions to improve flexibility for customizing deep SNNs.The passage contrasts this approach with encapsulated function modules described as unfriendly to users.
  • Parameter configuration: Simulation parameters for each dataset are provided in Table 7 for model evaluation.The supplied passage identifies Table 7 as containing the configuration but does not provide its parameter values.
  • Parameter configuration: The paper illustrates the structures of the AlexNet and CIFARNet models in Figure 8.The figure denotes the left structure as AlexNet and the right structure as CIFARNet.

C Details for Acceleration experiment

The acceleration experiments averaged 10 runs using 10 simulation steps and batch size 20 per epoch. PyTorch and Matlab were evaluated with GPU execution but without parallelization on the same CPU-GPU server.

  • Experimental setup: 10 experiments were averaged, with 10 simulation steps and batch size 20 per epoch used because Matlab runs slowly.The experiments ran on an i7-6700K CPU and GTX1060 GPU server.
  • Experimental setup: Both PyTorch 3.5 and Matlab 2018b enabled GPU execution without parallelization.The reported hardware was an i7-6700K CPU and GTX1060 GPU.
Loading 1809.05793v2…