PaperlayerView exact revision on arXiv

Source-linked AI summary

Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation

Qi Xu, Yaxin Li, Jiangrong Shen, Jian K Liu, Huajin Tang, Gang Pan

arXiv:2304.05627v2cs.NEcs.AI

TL;DR

Typical SNNs are hard to train deeply because spikes are non-differentiable, limiting their structures and training methods. This paper trains an SNN student with knowledge from an ANN teacher, reporting strong classification results, shorter training latency, and noise immunity. The reported KDSNN performance does not surpass the ANN teacher.

  • Problem

    Typical SNNs are difficult to train deeply because their non-differentiable spikes prevent direct gradient-based parameter adjustment and limit network structures.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The method uses knowledge distillation and joint ANN-SNN training to transfer teacher ANN features to an SNN student without requiring matching network structures.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The authors report state-of-the-art performance on experimental datasets with less training latency and better anti-noise ability, while KDSNN does not outperform ANN performance.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The authors conclude that KDSNN performs well on image-classification tasks and shows noise immunity across different noise types and intensities.

    Source: §5 · p. 8

    The proposed SNN model does not perform better than the ANN model.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Spiking neural networks (SNNs) are well known as the brain-inspired models with high computing efficiency, due to a key component that they utilize spikes as information units, close to the biological neural systems. Although spiking based models are energy efficient by taking advantage of discrete spike signals, their performance is limited by current network structures and their training methods. As discrete signals, typical SNNs cannot apply the gradient descent rules directly into parameters adjustment as artificial neural networks (ANNs). Aiming at this limitation, here we propose a novel method of constructing deep SNN models with knowledge distillation (KD) that uses ANN as teacher model and SNN as student model. Through ANN-SNN joint training algorithm, the student SNN model can learn rich feature information from the teacher ANN model through the KD method, yet it avoids training SNN from scratch when communicating with non-differentiable spikes. Our method can not only build a more efficient deep spiking structure feasibly and reasonably, but use few time steps to train whole model compared to direct training or ANN to SNN methods. More importantly, it has a superb ability of noise immunity for various types of artificial noises and natural signals. The proposed novel method provides efficient ways to improve the performance of SNN through constructing deeper structures in a high-throughput fashion, with potential usage for light and efficient brain-inspired computing of practical scenarios.

1. Introduction

Typical SNNs struggle to match ANNs on common tasks and are difficult to train deeply because spikes are non-differentiable. This paper uses ANN-guided knowledge distillation to construct flexible deep SNNs, reporting strong performance, lower training latency, and noise immunity.

1. Introduction

  • Typical SNNs underperform ANNs on some common tasks and are difficult to train deeply because non-differentiable spikes constrain training rules and network structures.The introduction notes that many current SNNs use shallow fully connected structures.Source: §1 · p. 1
  • Knowledge distillation lets ANN teachers transfer information to SNN students, including across heterogeneous network structures.The student may be smaller and shallower than a larger, deeper teacher ANN.Source: §1 · p. 2
  • The proposed method combines a unified ANN-SNN loss with surrogate-gradient student training to construct deep SNNs from a trained ANN.The authors evaluate image classification on MNIST, CIFAR10, CIFAR100, and noisy variations.Source: §1 · p. 2
  • Evaluations across several datasets, including noisy ones, support the method's reported efficiency and effectiveness.The introduction characterizes the resulting SNN as more reasonable, without specifying a metric in this passage.Source: §1 · p. 2
  • The proposed SNN achieves state-of-the-art performance on experimental datasets with less training latency and better anti-noise ability.The authors also report improved student performance over offline ANN-to-SNN conversion through joint ANN-SNN training.Source: §1 · p. 2

2. Related Work and Motivation

Deep SNN training is challenged by non-differentiable spikes; existing approaches include ANN-to-SNN conversion and surrogate gradients. The paper motivates a knowledge-distillation approach that combines their benefits and transfers intermediate ANN information to a student SNN.

2. Related Work and Motivation

  • Non-differentiable spikes make direct loss-based training of deep SNNs difficult; existing approaches include ANN-to-SNN conversion and surrogate-gradient training.Typical SNNs also lack the deep hierarchical structures common in ANNs.Source: §2 · p. 2

2.1. ANN-to-SNN Conversion Methods

  • ANN-to-SNN conversion reuses trained ANN parameters, but can lose real-time spike updates and waste resources during ANN training and conversion.The cited approach keeps weights stable and transfers them to an SNN with nearly the same structure.Source: §2.1 · p. 2

2.2. Surrogate Gradient Training Methods

  • Surrogate-gradient methods address spike non-differentiability, but remain gradient-based and are described as less biologically plausible.Prior work includes deep SNN backbones and methods that reduce inference timesteps.Source: §2.2 · p. 2

2.3. Motivation

  • The proposed KD approach combines ANN-to-SNN conversion with surrogate-gradient training, using teacher outputs and intermediate layers to train a potentially heterogeneous SNN student.The paper proposes two joint losses: one matching ANN and SNN output layers, and another spanning intermediate and output layers.Source: §2.3 · p. 3

3. Proposed Method

The method uses a pretrained ANN teacher to guide an SNN student without requiring matching structures. It combines output-based or intermediate-feature knowledge transfer with surrogate-gradient training for the student.

3.1. Spiking Neuron Model

  • The student uses an IF neuron: weighted incoming spikes update membrane potential, threshold crossing fires a spike, and the neuron resets.The paper describes the membrane-potential update as the charging process and the threshold function as the spike-firing process.Source: §3.1 · pp. 3–4

3.2. Joint ANN-to-SNN Knowledge Distillation Method

  • Response-based KD transfers teacher output soft labels, with temperature T flattening class probabilities and a combined loss incorporating true-label supervision.The total loss weights the distillation and true-label objectives with α.Source: §3.2 · p. 4
  • Feature-based KD transfers intermediate ANN representations, using transformed student features and an L2 distance to guide student training.Intermediate feature locations may follow groups of layers; the student features are transformed with a 1×1 convolution, and teacher features with margin ReLU.Source: §3.2 · p. 4
  • During backpropagation, a differentiable function approximates the Heaviside spike step so gradients can update the student SNN's weights.The student converts its output into spike frequency features when calculating the distillation loss.Source: §3.2 · p. 5

3. Proposed Method

  • The method pretrains and freezes an ANN teacher, then trains an SNN student whose structure can differ from the teacher's.The same samples are input to both models during student training.Source: §3 · p. 3, §3.2 · p. 5

4. Experimental Evaluation

Evaluation on MNIST, CIFAR10, and noisy variants tests KDSNN across architectures, robustness, resource use, and comparison with other spiking models. Reported results include high accuracy at four time steps, gains over original SNNs, and smaller operation and parameter footprints than ANN teachers.

4.1. Experimental Settings

  • Experiments used MNIST, CIFAR10, and noisy variations, with student SNNs simulated for only 4 time steps.The settings include background, rotation-normalized, and combined background-rotation MNIST variants, plus CIFAR10 at different noise intensities.Source: §4 · p. 5, §4.1 · p. 5, Figure 2 · p. 6

4.2. Evaluation under different knowledge levels and architectures

  • Response-based KDSNN improved accuracy over same-architecture SNNs, reaching 93.41% with a ResNet18 student and Pyramidnet18 teacher, about 0.73% higher than without KD.Accuracy curves also rose across training epochs with each of the three tested ANN teachers.Source: §4.2 · pp. 6–7, Figure 3 · p. 7
  • Feature-based KDSNN reached 92.10% with a WRN16-2 student and Pyramidnet18 teacher, a 1.76% improvement and higher than response-based KDSNN for that architecture.The reported comparison also finds stronger ANN teachers yield better student performance, with WRN16-2 results of 90.98%, 91.14%, and 91.11% for three teachers.Source: §4.2 · pp. 6–7
  • KDSNN improved noisy-data accuracy over original SNNs, including by 1.31% on background-image MNIST and by 1.27%, 1.54%, and 0.83% across three CIFAR10 Gaussian-noise levels.The authors state that the proposed SNN does not outperform ANN, while outperforming original SNNs in noisy environments.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Parameter Comparison between ANN and SNN

  • With ResNet18 as teacher, KDSNN-trained SNNs used 0.05M Synaptic Operations versus 457.72M ANN FLOPs, and CIFAR10 student models had fewer parameters than ANN teachers.The passage characterizes the SNN structures as simpler, with fewer convolutional layers and parameters.Source: §4.3 · p. 7, Table 3 · p. 8

4.4. Performance Comparison with Other Methods

  • 99.37% test accuracy on MNIST and 93.41% on CIFAR10 were achieved with 4 time steps; the CIFAR10 result exceeded several listed SNN methods.The comparison names SPIKE-NORM, Hybrid Training, RMP, and Opt as methods exceeded on CIFAR10.Source: §4.4 · pp. 7–8, Table 4 · p. 8

5. Conclusion

The paper combines knowledge distillation with spiking-based surrogate-gradient methods and ANN-to-SNN conversion to construct efficient SNNs. It reports image-classification performance, noise immunity, and resource efficiency, while identifying teacher-model weaknesses as future work.

5. Conclusion

  • The proposed methods combine ANN knowledge distillation with spiking-based surrogate gradients and ANN-to-SNN conversion to address the non-differentiability of binary spikes.They use high-dimensional, accurate teacher ANN features.Source: §5 · p. 8
  • The KDSNN model performs well on image-classification tasks and shows noise immunity across different noise types and intensities, with rapid training convergence.The paper also compares memory occupation and synaptic operations between typical ANNs and constructed SNNs.Source: §5 · p. 8
  • KDSNN is described as deep and efficient, performing better than some other spiking models with few resource consumption.The authors identify resource-constrained devices, including neuromorphic hardware platforms, as a potential application setting.Source: §5 · p. 8
  • Future work will expand ANN and SNN structures and address cases where the teacher is weaker than the student or lacks a probability distribution..Source: §5 · p. 8

Related papers

Loading 2304.05627v2…