PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks

Shikuang Deng, Shi Gu

arXiv:2103.00476v1cs.NEstat.ML

TL;DR

Converting ANNs to SNNs avoids the training difficulty caused by discrete spikes, but existing conversion methods lack a whole-network loss approximation. The paper decomposes conversion error layer-wise and proposes a threshold-balancing pipeline with soft-reset mechanisms. The resulting converted SNNs retain nearly the ANN accuracy while requiring about one-tenth of the typical simulation time.

  • Problem

    Existing ANN-to-SNN conversion methods lack a theoretical analysis of whole-network conversion error and an efficient loss approximation.

  • Method

    The method recursively decomposes conversion error into layer-wise activation errors and modifies source-ANN ReLU functions to approximate target-SNN spiking frequencies.

  • Results

    The converted SNNs achieve almost no accuracy loss relative to conventional ANNs with approximately one-tenth of the typical SNN simulation time.

  • Takeaways & Limitations

    The shorter simulation length preserves more of the SNN energy-efficiency advantage, especially for large networks.

  • Takeaways & Limitations

    When layers are strongly coupled, the interaction term may not be negligible, making layer-wise recursion suboptimal and motivating blockwise approaches.

Abstract

from arXiv · show

Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) that comprise of spiking neurons to process asynchronous discrete signals. While more efficient in power consumption and inference speed on the neuromorphic hardware, SNNs are usually difficult to train directly from scratch with spikes due to the discreteness. As an alternative, many efforts have been devoted to converting conventional ANNs into SNNs by copying the weights from ANNs and adjusting the spiking threshold potential of neurons in SNNs. Researchers have designed new SNN architectures and conversion algorithms to diminish the conversion error. However, an effective conversion should address the difference between the SNN and ANN architectures with an efficient approximation \DSK{of} the loss function, which is missing in the field. In this work, we analyze the conversion error by recursive reduction to layer-wise summation and propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. This pipeline enables almost no accuracy loss between the converted SNNs and conventional ANNs with only $\sim1/10$ of the typical SNN simulation time. Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory.

1 INTRODUCTION

SNNs offer event-driven efficiency but are difficult to train directly because spikes are discontinuous. This work analyzes conversion error and proposes a conversion strategy that controls activation differences with shorter simulation.

  • SNNs model biological neural activity with spiking neurons, including widely used Integrate-and-Fire neurons that emit spikes when accumulated input exceeds a threshold.
  • ANNs achieve strong performance but rely on high-precision digital information transmission and substantial energy and memory, limiting embedded deployment.
  • SNNs use event-driven spiking signals and can reduce energy and memory consumption on embedded platforms.
  • Direct supervised SNN training can approach ANN performance on smaller datasets but requires substantial time and memory and struggles to converge for large networks.
  • The proposed approach theoretically analyzes conversion error and modifies source-ANN ReLU activations to approximate target-SNN spiking frequencies.
  • The conversion algorithm controls source-target activation differences while using a much shorter simulation length than existing approaches.

2 PRELIMINARIES

The conversion pipeline aligns ANN and SNN representations through threshold balancing and a modified ReLU, while the paper establishes notation and the layer-wise conversion framework. The threshold ReLU suppresses excessive activations so neurons can activate within shorter simulations.

  • The pipeline combines threshold balancing with a modified ReLU to reduce ANN-to-SNN conversion error.The modified ReLU thresholds maximum activation values and shifts its turning point.
  • The source ANN and target SNN share the same infrastructure, with Wl denoting the l-th layer's weight matrix and Vth the SNN threshold voltage.The SNN is simulated over T time points, with membrane potentials and spike outputs tracked across layers.
  • The source ANN uses a threshold ReLU that outputs 0 for nonpositive inputs, x below yth, and yth for x at least yth.yth is the threshold added to the regular ReLU function.
  • Figure 1 contrasts ANN activation functions with the SNN step function and illustrates their conversion error under a Vth/2T shift.The figure presents the conversion pipeline and the discrepancy between continuous ANN activations and discrete SNN outputs.
  • The paper derives weight transforms, decomposes total conversion error into layer-wise activation-to-frequency errors, and estimates an optimal shift for each layer.These components form the stated analytical basis for the conversion pipeline.

3 CONVERSION EQUATION FROM ANN TO SNN

The SNN conversion equations describe temporal membrane-potential accumulation, spike generation, and the resulting layer-to-layer PSP propagation. Because SNN outputs are discrete while ANN outputs are continuous, an intrinsic conversion difference remains even when average inputs are equalized.

  • Each incoming SNN spike contribution adds Wl·x′l(t+1) to the layer's membrane potential.The accumulated quantity forms the temporal potential at time t+1.
  • When the temporal potential exceeds Vth, the neuron emits a spike, subtracts Vth, and updates its membrane potential.The emitted spike carries potential Vth to the next layer.
  • Accumulating the membrane-potential update over T time points yields the next layer's input as the l-th layer's expected PSP.The expected PSP is the time-averaged spike output, a′l+1 = Σt=1^T θl(t)/T.
  • When Vth exceeds the source ANN's maximum activation, the remaining membrane potential stays below threshold and is not output.The clip operation expresses the resulting output under this condition.
  • The clipped SNN output is a step function, whereas ANN output is continuous and therefore differs intrinsically through floor rounding.This discrepancy can persist even when a′l and al are equalized, motivating later error minimization.

4 DECOMPOSITION OF CONVERSION ERROR

The paper decomposes conversion error by expanding the loss around the source ANN output and tracing layer-wise activation discrepancies to the final output.

  • Layer-wise error: Layer output error combines propagated upstream error with error from differing ANN and SNN activation functions.The activation-function discrepancy is denoted ∆a′_l.
  • Loss decomposition: The loss is expanded around the source ANN’s last-layer activation using a second-order Taylor approximation.Because the source ANN is optimized, the first-order term is ignored, leaving the second-order term as the conversion-error target.
  • Layer-wise error: Substituting the layer-wise error into the loss expansion reduces the objective to terms involving activation discrepancies across layers.The Hessian and weight matrices connect errors between adjacent layers.
  • Optimization target: Assuming the Hessian term is constant reduces conversion-error minimization to minimizing activation-value differences at each layer.The Hessian may alternatively be approximated with the Fisher Information Matrix.

5 LAYER-WISE AND TOTAL CONVERSION ERROR

The layer-wise analysis motivates modifying the ANN activation function to reduce discrepancies with the target SNN, while threshold and simulation choices govern residual error.

  • Layer-wise error: Minimizing conversion error is analyzed as minimizing layer-wise output errors caused by different ANN and SNN activation functions.The next step is to modify the source activation function to reduce these errors.
  • Threshold selection: Extreme threshold choices prevent reliable conversion when neurons either fail to spike or spike at every time point.In both cases, the remaining membrane potential retains most ANN information, making conversion nearly impossible.
  • Activation alignment: Threshold ReLU and the SNN activation curve exhibit a systematic bias even when their inputs are equal.The bias can be optimized by shifting either activation function.
  • Activation alignment: A shift δ is introduced to reduce the expected squared difference between source ANN and target SNN activations.The analysis considers shifting the source activation while fixing the SNN activation.
  • Activation alignment: Uniformly distributed layer inputs permit approximate reduction of the layer-wise loss optimization.The distribution is defined over intervals determined by Vth/T.
  • Error controls: Lower threshold values and longer simulation times decrease conversion error, but the optimal shift also depends on activation distributions and overfitting.The shift optimizes ANN–SNN output agreement rather than SNN accuracy directly.
  • Conversion pipeline: Algorithm 1 pre-trains the source ANN with threshold ReLU and assigns layer thresholds during conversion.The threshold voltage can also be calculated during training.

6 RELATED WORK

Prior conversion work progressed from simple ReLU-based conversion to normalization, soft reset, and deeper SNN architectures, motivating the paper’s combined strategy.

  • Early conversion methods: Early ReLU-based ANN-to-SNN conversion performed well on simple datasets but did not scale to large networks and complex datasets.Subsequent work introduced weight normalization for a three-layer CNN without bias.
  • Reset mechanisms: Spike subtraction, also called soft reset, was introduced to reduce information loss caused by resetting membrane potential.Later work used soft reset as an alternative to hard reset.
  • Deep SNNs: Sengupta et al. used SPiKE-NORM to obtain deep SNNs such as VGG-16 and ResNet-20.Kim et al. applied SNNs to object recognition.
  • Threshold balancing: Threshold balancing is equivalent to weight normalization and copies ANN weights and biases into the target SNN.Each SNN layer’s threshold is set to the maximum output of the corresponding ANN layer.
  • Reset mechanisms: Soft reset preserves informative remaining potential by replacing hard-reset rest potential with membrane potential minus threshold voltage.This mechanism is described as reducing information loss.
  • Simulation efficiency: Prior methods also addressed varied neuron activation ranges using percentile-based normalization or channel-level threshold balancing.These approaches target the long simulation times needed to activate neurons with high thresholds.

7 EXPERIMENTS

Experiments across CIFAR-10, CIFAR-100, and ImageNet evaluate threshold and shift modifications for ANN-to-SNN conversion. The proposed method achieves competitive accuracy with substantially shorter simulation lengths, especially for VGG-16 and ResNet-20.

  • ANN performance with threshold ReLU: Thresholding significantly reduces maximum-activation variation across VGG-16 layers, while shifting has little additional effect on that distribution.This suggests a common spiking threshold can activate different layers more effectively after modifying ReLU.
  • ANN performance with threshold ReLU: On CIFAR-100, threshold ReLU decreases VGG-16 accuracy by 0.16% but increases ResNet-20 accuracy by 2.78%.These results support threshold ReLU as a reasonable ANN source for conversion.
  • Simulation length and shift: Threshold and shift shorten convergence below 100 simulation steps, compared with approximately 200 steps for regular ReLU when accuracy loss is small.For T < 50, their combination converges fastest, while threshold is more efficient than shift; differences become minor for T > 100.
  • Comparison with related work: On CIFAR-Net, the model exceeds 90% accuracy within 16 simulation steps, while direct-training methods have high training costs and do not extend to VGG-16 or ResNet-20.For ResNet-20, the model outperforms compared methods in both accuracy and simulation length.
  • Comparison with related work: For ResNet-20, conversion improves accuracy by 1.26% on CIFAR-10 and 2.74% on CIFAR-100 relative to the source ANN trained with regular ReLU.The authors suggest thresholding and discretization may reduce an overfitting deficit associated with batch normalization and max pooling.
  • Comparison with related work: For VGG-16, the method reaches comparable performance to RMP in 16 steps on CIFAR-10 and 32 steps on CIFAR-100, versus 1536 and 2048 RMP steps.On ImageNet, 400–600 steps produce lower conversion error than RMP, TSC, and SPIKE-NORM using 4096, 4096, and 2500 steps, respectively.
  • Comparison with related work: The proposed algorithm potentially extends to recurrent neural network conversion, with an illustrative appendix example comparing favorably against direct training.The reported extension is presented as a potential application rather than a demonstrated general result.

8 DISCUSSION

The discussion identifies theoretical and practical implications of the conversion strategy, while noting limits involving layer coupling, simulation length, and supported ANN architectures.

  • Earlier conversion methods optimized procedures or structures but did not theoretically analyze whole-network conversion error or which ANNs are easier to transform.
  • The error decomposition can also frame network quantization because perturbations in weights, activation functions, and values propagate through later-layer outputs.
  • Strong coupling between layers can make layer-wise recursion suboptimal, motivating blockwise approaches for further optimization.
  • The method greatly reduces the simulation length required by converted SNNs, especially for large networks, and can combine with Spiking-YOLO, SPIKE-NORM, and RMP.
  • Future extensions include converting ANNs trained with batch normalization or activation functions that take both positive and negative values.

A.1 NETWORK STRUCTURE AND OPTIMIZATION SETUP

The experiments use CIFARNet, VGG-16, and ResNet-20 with average pooling, alongside specified dropout, optimization, and training-schedule settings.

  • The study adopts CIFARNet, VGG-16, and ResNet-20, using average pooling in all pooling layers to adapt the networks to SNNs.
  • Dropout is set to p = 0.2 between convolutional layers and p = 0.5 between fully connected layers.
  • For CIFAR-10 and CIFAR-100, training uses learning rate 0.01, batch size 128, 300 epochs, SGD with momentum 0.9, and weight decay 5e −4.
  • The learning rate decays by 0.1 at epochs 180, 240, and 270, while threshold ReLU is introduced after 20 epochs of regular-ReLU training.

A.2 SNN PERFORMANCE WITH SIMULATION LENGTH 400-600

Across longer simulations, threshold-ReLU and shift variants are compared with regular ReLU and prior conversion methods. Thresholding can achieve nearly loss-free conversion, while combining threshold and shift reduces error fastest at short simulation lengths.

  • A.2 SNN PERFORMANCE WITH SIMULATION LENGTH 400-600: Simulation lengths 400–600 are used to average converted-SNN accuracy and variance on CIFAR-10 and CIFAR-100.The compared variants include regular ReLU, threshold ReLU, and threshold plus shift operation.
  • A.3 SNN PERFORMANCE WITH SHORT SIMULATION LENGTH: The conversion loss is defined as the accuracy difference accANN − accSNN between the source ANN and converted SNN.Comparisons include regular ReLU and threshold ReLU, with threshold-ReLU results reported in brackets.
  • A.3 SNN PERFORMANCE WITH SHORT SIMULATION LENGTH: Using threshold and shift together reduces conversion error fastest at short simulation lengths, while threshold alone reaches the minimum error for relatively long simulations.The shift-plus-regular-ReLU configuration is similar to TSC on CIFAR-10 and better than TSC and RMP on CIFAR-100.
  • A.3 SNN PERFORMANCE WITH SHORT SIMULATION LENGTH: For VGG-16 on ImageNet, the proposed model achieves nearly zero conversion loss with simulation length 256.This result is reported in the short-simulation comparison.
  • RNN conversion example: In a sentiment-analysis RNN example, the converted SNN reaches 0.5430 validation accuracy, close to the source RNN’s 0.5428 and above directly trained or regular-ReLU conversions.The directly trained SNN reaches 0.5106, while using regular ReLU in the source RNN yields 0.5100.
  • A.5 COMPARE THE PERFORMANCE ON LONG SIMULATION: Over extended simulation lengths, regular-ReLU conversion converges to about 0.75% accuracy loss, whereas threshold-ReLU conversion is almost loss-free.The result indicates thresholding improves both convergence efficiency and final performance relative to regular ReLU.
Loading 2103.00476v1…