PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks

Jianhao Ding, Zhaofei Yu, Yonghong Tian, Tiejun Huang

arXiv:2105.11654v1cs.NEcs.AIcs.CVcs.LG

TL;DR

Deep SNN deployment is hindered by conversion accuracy loss and long inference times, despite ANN-SNN conversion’s efficiency for training deep models. The paper develops theoretically grounded normalization and fit-optimization methods for direct, fast conversion, achieving near loss-less deep-network conversion with 8.6× faster reasoning at 0.265× the typical method’s energy consumption.

  • Problem

    ANN-SNN conversion is efficient for training deep SNNs but commonly suffers accuracy loss and long inference time, impeding practical application.

  • Method

    The paper derives sufficient conditions for optimal conversion, replaces ReLU with a trainable Rate Norm Layer, and optimizes an upper bound of an ANN-activation/SNN-firing-rate fit curve.

  • Results

    8.6× faster reasoning under 0.265× energy consumption is reported alongside near loss-less conversion in deep network architectures and multiple datasets.

  • Takeaways & Limitations

    The method provides a theoretically supported route to accurate and low-latency conversion for deep SNNs.

Abstract

from arXiv · show

Spiking Neural Networks (SNNs), as bio-inspired energy-efficient neural networks, have attracted great attentions from researchers and industry. The most efficient way to train deep SNNs is through ANN-SNN conversion. However, the conversion usually suffers from accuracy loss and long inference time, which impede the practical application of SNN. In this paper, we theoretically analyze ANN-SNN conversion and derive sufficient conditions of the optimal conversion. To better correlate ANN-SNN and get greater accuracy, we propose Rate Norm Layer to replace the ReLU activation function in source ANN training, enabling direct conversion from a trained ANN to an SNN. Moreover, we propose an optimal fit curve to quantify the fit between the activation value of source ANN and the actual firing rate of target SNN. We show that the inference time can be reduced by optimizing the upper bound of the fit curve in the revised ANN to achieve fast inference. Our theory can explain the existing work on fast reasoning and get better results. The experimental results show that the proposed method achieves near loss less conversion with VGG-16, PreActResNet-18, and deeper structures. Moreover, it can reach 8.6x faster reasoning performance under 0.265x energy consumption of the typical method. The code is available at https://github.com/DingJianhao/OptSNNConvertion-RNL-RIL.

1 Introduction

ANN-SNN conversion offers an efficient route to training deep SNNs, but accuracy loss and long simulation times limit practical use. The paper proposes theoretically grounded conversion and fit-optimization methods, reporting near loss-less conversion and faster, lower-energy inference.

  • Motivation: ANN-SNN conversion requires less GPU computation than surrogate-gradient training and has yielded strong performance on large networks and datasets.These advantages make it the paper’s focus for deep SNN training.
  • Motivation: Converted SNNs incur long simulation times on deeper networks and more complicated datasets, restricting applications such as real-time tracking and detection.Prior methods provide limited theoretical guidance for reducing inference latency.
  • Contributions: The paper derives sufficient conditions for optimal conversion and introduces Rate Norm Layer to replace ReLU, enabling direct conversion from a trained ANN to an SNN.The layer is intended to reduce potential information loss caused by normalization.
  • Contributions: An optimal fit curve quantifies agreement between source-ANN activations and target-SNN firing rates, while optimizing its upper bound reduces inference time.The analysis also provides a theoretical basis for explaining threshold scaling methods for fast inference.

2 Methods

The method develops a theoretical ANN-SNN conversion framework, introduces Rate Norm Layers for direct conversion, and optimizes firing-rate fitting to reduce inference delay.

  • Theory for Conversion from ANN to SNN: The conversion theory matches ReLU activations in an ANN with firing rates from reset-by-subtraction spiking neurons over time.The spiking model uses soft reset, reducing membrane potential by the firing threshold after each spike.
  • Theory for Conversion from ANN to SNN: Theorem 1 states sufficient layer-wise conditions for ANN-to-SNN equivalence as inference time approaches infinity.The conditions relate ANN weights, SNN weights, ANN activation maxima, and SNN firing thresholds.
  • Theory for Conversion from ANN to SNN: Weight normalization and threshold balancing are equivalent scaling operations required for ANN-SNN conversion.The theorem’s scaling relation shows that conversion requires either scaling weights or setting firing thresholds.
  • Rate Norm Layer: Rate Norm Layer replaces ReLU with a clip function whose trainable upper bound outputs a simulated firing rate.Its threshold is trainable, and setting the trainable scalar p_l to 1 recovers the Max Norm algorithm.
  • Optimization for Fast Inference: The method quantifies ANN-SNN fitting with a scale-invariant K curve and accelerates inference by targeting faster convergence across layers.Layer stacking increases delay, while K compares simulated ANN firing rates with real SNN rates and ideally converges to 0.

3 Experiments

Experiments evaluate Rate Norm Layers for conversion accuracy and Rate Inference Loss for fast, energy-aware inference across image-recognition benchmarks. The proposed method achieves strong accuracy, rapid convergence, and lower energy to reach reliable accuracy.

  • Experimental Setup: Experiments cover MNIST, CIFAR-10, and CIFAR-100 using CNN, AlexNet, VGG-16, PreActResNet-18, and PreActResNet-34 architectures.CIFAR-10 uses VGG-16 and PreActResNet-18; CIFAR-100 additionally uses PreActResNet-34.
  • Accuracy Performance: Rate Norm Layers achieve state-of-the-art performance on MNIST and CIFAR-100 and similar performance on CIFAR-10.For CIFAR-100 VGG-16, the proposed method reaches 75.02% top-1 accuracy versus 70.93% for the state-of-the-art ANN-SNN algorithm.
  • Accuracy Performance: 75.02% top-1 accuracy is achieved for VGG-16 trained on CIFAR-100, compared with 70.93% for the state-of-the-art ANN-SNN algorithm.The reported conversion loss is low or negative, while other methods usually have positive conversion loss.
  • Fast Inference Performance: 85.40% accuracy is reached in 32 time-steps, while Max Norm, Robust Norm, and RMP-SNN reach 10.00%, 43.03%, and 63.30%.The proposed method exceeds 90% accuracy in 52 time-steps, compared with 446 time-steps for Max Norm, an 8.6 times speedup.
  • Fast Inference Performance: Rate Inference Loss keeps Ω relatively low and average, supporting faster inference than Max Norm.The comparison uses threshold and Ω distributions visualized for VGG-16.
  • Energy Estimation: 0.265 times the Max Norm energy consumption is required to reach 90% accuracy, despite the proposed model's relatively high power from its lower threshold.The power histogram integrates energy until accuracy reaches 90%.

4 Conclusions

The paper combines Rate Norm Layers, an optimal fit curve, and Rate Inference Loss to convert conventional ANNs into accurate, fast deep SNNs. Experiments report low accuracy loss and fast reasoning in deep architectures.

  • 4 Conclusions: The method replaces ReLU with Rate Norm Layers and quantifies ANN activation–SNN firing-rate correspondence using an optimal fit curve.Rate Norm Layers enable direct conversion from a trained ANN to an SNN.
  • 4 Conclusions: Rate Inference Loss reduces inference time by optimizing the upper-bound coefficient of the fit curve.The paper uses a two-staged learning scheme to obtain fast and accurate deep SNNs.
  • 4 Conclusions: Experiments demonstrate low accuracy loss and fast reasoning for deep structures including VGG and PreActResNet.

Appendix A: Proofs

The appendix develops the theoretical basis for ANN-SNN conversion by relating bounded ANN activations to SNN firing rates and analyzing convergence of simulated firing rates.

  • Conversion Theory: The proofs assume reset-by-subtraction neurons and analyze the relationship between firing rates in consecutive SNN layers.As time tends to infinity, firing rates are related through the layer parameters and thresholds.
  • Neuron Constraints: Membrane potentials remain within [0, v_th,l], while binary spikes restrict firing rates to [0, 1].
  • Conversion Theory: Theorem 1 states sufficient layer-wise equations under which the ANN and converted SNN representations are equivalent.The conditions compare ANN weights, SNN parameters, activation maxima, and firing thresholds.
  • Conversion Theory: ReLU activations are normalized by their layer-wise maxima so normalized activations lie in [0, 1], matching the firing-rate range.The proof introduces z_i = a_i / max_i and uses bounded activations on limited datasets.
  • Convergence Analysis: Theorem 2 analyzes the discrepancy K between simulated and real firing rates for constant coding.The proof models cumulative rounded firing and uses clipping to bound firing-rate behavior.

Appendix B: Supplementary of Methods and Experiments

The supplementary methods describe a two-stage training and conversion algorithm for Rate Norm Layers, followed by direct transfer of trained ANN parameters and thresholds to the SNN.

  • Training Algorithm: All layers share the same trainable p_i during training to reduce threshold-training instability in deep models.A sigmoid parameterization constrains Rate Norm Layer values to [0, 1].
  • Implementation: Experiments use PyTorch training on an NVIDIA GeForce RTX 2080 Ti GPU.
  • Training Algorithm: Stage 1 trains the Rate Norm Layer ANN for accuracy, while Stage 2 trains it for fast inference.Both stages use mini-batch training over the dataset.
  • Training Algorithm: The algorithm initializes p_k and running maxima to 1.0, with momentum m = 0.1 and λ = 0.5.
  • ANN-SNN Conversion: Conversion copies trained ANN weights and biases into the SNN and sets each SNN threshold v_th,k to the corresponding ANN θ_k.
Loading 2105.11654v1…