PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks

Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, Tiejun Huang

arXiv:2303.04347v1cs.NE

TL;DR

ANN-SNN conversion can match ANN performance but usually requires long time-steps, limiting low-latency deployment. The paper analyzes conversion errors and replaces ReLU with a quantization clip-floor-shift activation; across CIFAR-10/100 and ImageNet, it reports state-of-the-art accuracy with fewer time-steps, including 4-step operation.

  • Problem

    ANN-SNN conversion requires long time-steps to match ANN activations, while short simulations cause performance degradation that hampers practical SNN application.

  • Method

    The paper analyzes clipping, quantization, and unevenness errors, then trains source ANNs with a quantization clip-floor-shift activation replacing ReLU.

  • Results

    The method achieves state-of-the-art accuracy with fewer time-steps on CIFAR-10, CIFAR-100, and ImageNet, including 93.96% accuracy with 4 time-steps on CIFAR-10.

  • Takeaways & Limitations

    The reported conversion method supports high-accuracy, ultra-low-latency SNNs and may benefit neuromorphic-hardware implementation and large-scale SNN application.

  • Takeaways & Limitations

    Choosing quantization steps L requires trading low-latency accuracy against model capacity and best accuracy at relatively large time-steps.

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) have gained great attraction due to their distinctive properties of low power consumption and fast inference on neuromorphic hardware. As the most effective method to get deep SNNs, ANN-SNN conversion has achieved comparable performance as ANNs on large-scale datasets. Despite this, it requires long time-steps to match the firing rates of SNNs to the activation of ANNs. As a result, the converted SNN suffers severe performance degradation problems with short time-steps, which hamper the practical application of SNNs. In this paper, we theoretically analyze ANN-SNN conversion error and derive the estimated activation function of SNNs. Then we propose the quantization clip-floor-shift activation function to replace the ReLU activation function in source ANNs, which can better approximate the activation function of SNNs. We prove that the expected conversion error between SNNs and ANNs is zero, enabling us to achieve high-accuracy and ultra-low-latency SNNs. We evaluate our method on CIFAR-10/100 and ImageNet datasets, and show that it outperforms the state-of-the-art ANN-SNN and directly trained SNNs in both accuracy and time-steps. To the best of our knowledge, this is the first time to explore high-performance ANN-SNN conversion with ultra-low latency (4 time-steps). Code is available at https://github.com/putshua/SNN\_conversion\_QCFS

1 INTRODUCTION

ANN-SNN conversion can produce deep, accurate SNNs, but conventional approaches require many time-steps and suffer conversion errors at ultra-low latency. This paper analyzes those errors and introduces an activation-function replacement targeting high accuracy with few time-steps.

  • Motivation: ANN-SNN conversion has achieved performance comparable to ANNs on large-scale datasets, but typically depends on long inference time-steps.Longer time-steps also increase resource consumption in directly trained SNNs.
  • Motivation: Tens to hundreds of time-steps remain impractical for ultra-low-latency SNNs despite normalization, threshold, reset, and shift-based improvements.The paper identifies this remaining conversion-theory gap as difficult to eliminate in a few time-steps.
  • Contributions: The proposed analysis attributes conversion error to clipping, quantization, and previously neglected unevenness errors caused by changes in spike-arrival timing.Unevenness can produce more or fewer spikes than expected.
  • Contributions: The quantization clip-floor-shift activation replaces ReLU, better approximates SNN activation, and yields zero expected conversion error theoretically.The authors evaluate it on CIFAR-10, CIFAR-100, and ImageNet against conversion and backpropagation methods.
  • Contributions: 91.18% top-1 accuracy on CIFAR-10 is achieved with 2 time-steps, demonstrating the method’s reported ultra-low-latency performance.This is presented as an example of exceeding state-of-the-art accuracy with fewer time-steps.

2 PRELIMINARIES

The paper formulates ANN-SNN conversion using analog activations and integrate-and-fire neurons with reset-by-subtraction. Averaging the SNN dynamics over time establishes the relationship between firing-based outputs and ANN activations, while long simulations reduce their mismatch.

  • Neuron models: ANN neurons compute a linear transformation followed by a nonlinear mapping, with h(·) specified as the ReLU activation function.The vector a_l denotes outputs in layer l, and W_l is the weight matrix from layer l−1.
  • Neuron models: The SNN model uses integrate-and-fire neurons whose membrane potentials accumulate inputs and trigger spikes when they exceed layer-specific thresholds.The spike output s_l(t) is binary, and the threshold-triggered update is expressed through the Heaviside step function.
  • Neuron models: Reset-by-subtraction subtracts the firing threshold after a spike instead of resetting the membrane potential to zero, reducing information loss.This mechanism is used in the ANN-SNN conversion framework.
  • ANN-SNN conversion: ANN-SNN conversion maps an ANN activation value to an SNN firing rate or average postsynaptic potential.Summing the membrane-potential update over T time-steps and dividing by T produces the averaged relationship between adjacent layers.
  • ANN-SNN conversion: When T is long enough, the converted SNN nearly matches the source ANN activation function, but large T causes inference latency that limits practical use.The paper therefore targets high-performance conversion at extremely low latency.

3 CONVERSION ERROR ANALYSIS

The conversion mismatch arises from clipping, quantization, and uneven spike timing. The analysis shows how these mechanisms distort SNN outputs relative to ANN outputs, motivating an activation function that more closely matches SNN quantization behavior.

  • Error formulation: The layerwise conversion error is defined by subtracting the ANN output from the converted SNN output under the same weighted input.The analysis assumes equal outputs from the preceding ANN and SNN layers and uses z_l as the shared weighted input.
  • Error formulation: The estimated error is nonzero when residual membrane potential remains and the weighted input z_l is positive.The paper identifies three contributing factors to this error.
  • Error types: Clipping error occurs because SNN outputs are bounded by θ_l while ANN activations can extend to a larger maximum value.Values above the effective mapping range are mapped to the same SNN maximum.
  • Error types: Quantization error arises because discrete output spikes make φ_l(T) discrete with resolution θ_l/T, so continuous ANN activations cannot be represented exactly.Flooring is the resulting quantization mechanism in the conversion mapping.
  • Error types: Unevenness error is caused by spike-timing changes: with the same presynaptic firing rates and weights, the postsynaptic neuron can produce more or fewer spikes than the ANN output implies.The example produces φ_l(T)=0.4, 0.8, or 0.2 for an ANN output of 0.4, depending on arrival timing.
  • Error interactions: When the final membrane potential lies in [0, θ_l], unevenness error reduces to quantization error, enabling an estimated SNN activation function that ignores timing unevenness.This assumption supports the subsequent conversion-error derivation.

4 OPTIMAL ANN-SNN CONVERSION

The paper replaces ReLU with quantization clip-floor and clip-floor-shift activations to reduce ANN-SNN conversion error at short time-steps. The shift formulation yields zero expected conversion error under stated assumptions, while quantization level L trades low-latency accuracy against model capacity.

  • 4.1 QUANTIZATION CLIP-FLOOR ACTIVATION FUNCTION: The quantization clip-floor activation replaces ReLU and is designed to eliminate conversion error when simulation time-steps T equal quantization steps L.The authors use this activation to train source ANNs before conversion.
  • 4.1 QUANTIZATION CLIP-FLOOR ACTIVATION FUNCTION: When T = L, θl = λl, and vl(0) = 0, the estimated conversion error is zero.Theorem 1 states this result for an ANN converted to an SNN with the same weights.
  • 4.1 QUANTIZATION CLIP-FLOOR ACTIVATION FUNCTION: When T differs from L, conversion error can exceed zero and transmit across layers, degrading converted-SNN accuracy.Training separate source ANNs for different quantization steps and conversion time-steps would impose considerable cost.
  • 4.2 QUANTIZATION CLIP-FLOOR-SHIFT ACTIVATION FUNCTION: The clip-floor-shift activation adds a shift vector ϕ, allowing the expected conversion error to reach zero for arbitrary T and L under the theorem’s assumptions.The theorem sets θl = λl and vl(0) = θlϕ, with the shift term specified as 1/2.
  • 4.2 QUANTIZATION CLIP-FLOOR-SHIFT ACTIVATION FUNCTION: The shift term ϕ = 0.5 makes the mean conversion error zero even when L ≠ T, supporting high-performance conversion at ultra-low time-steps.This comparison is reported using Figures 2b and 2c.
  • 4.2 QUANTIZATION CLIP-FLOOR-SHIFT ACTIVATION FUNCTION: Choosing L is a trade-off: smaller quantization steps favor low-latency performance, whereas overly low quantization reduces model capacity and can hurt accuracy at larger time-steps.L is the only undetermined hyperparameter of the clip-floor-shift activation.
  • 4.3 ALGORITHM FOR TRAINING QUANTIZATION CLIP-FLOOR-SHIFT ACTIVATION FUNCTION: The floor function is trained with a straight-through estimator, using derivative d⌊x⌋/dx = 1, and the ANN is optimized with stochastic gradient descent.This supplies the training rule for the quantization clip-floor-shift activation.

5 RELATED WORK

Prior ANN-SNN conversion work improved accuracy through normalization and threshold-scaling methods, while this paper targets conversion errors and extreme time-step reduction. The related-work discussion positions the proposed approach as fitting ANNs to SNNs for both ultra-low and larger time-steps.

  • 5 RELATED WORK: ANN-SNN conversion developed from early CNN conversion with normalization toward scaling methods for deeper networks and complex datasets.The cited approaches include data-based and model-based normalization, weight normalization, and threshold scaling.
  • 5 RELATED WORK: Weight normalization and threshold rescaling were later shown to be equivalent conversion strategies.The equivalence is attributed to Ding et al. (2021).
  • 5 RELATED WORK: This work differs by targeting conversion-error elimination, end-to-end quantization-layer training, and one SNN that performs well at ultra-low and large time-steps.The authors contrast this aim with prior work lacking trials at more extreme time-steps.

6 EXPERIMENTS

Experiments on CIFAR-10, CIFAR-100, and ImageNet show that the proposed conversion method maintains high accuracy at very short time-steps and performs strongly against prior methods. Ablations further show that the shift term and quantization-step choice are important for balancing latency and accuracy.

  • Comparison of quantization clip-floor and quantization clip-floor-shift: The clip-floor-shift activation yields better ANN performance than clip-floor activation across VGG-16 and ResNet-20 experiments on CIFAR-10 and CIFAR-100.The comparison includes original ReLU ANNs, clip-floor ANNs, and clip-floor-shift ANNs.
  • Comparison with the state-of-the-art: 93.96% accuracy is achieved with only 4 time-steps on CIFAR-10, making inference 8 times faster than SNNC-AP at 32 time-steps.For ResNet-20, the method achieves 83.75% accuracy with 4 time-steps.
  • Comparison with the state-of-the-art: At T = 32 on CIFAR-10, the method reaches 95.54% accuracy versus 95.52% for the ANN, while several baselines incur larger accuracy losses.RMP, RTS, RNL, and SNNC-AP report accuracy losses of 33.3%, 19.48%, 7.42%, and 2.01%, respectively.
  • Comparison with the state-of-the-art: On ImageNet at T = 32, the proposed method is 4.83% more accurate than SNNC-AP and 69.28% more accurate than RTS for ResNet-34.At 16 time-steps, ResNet-34 still achieves 59.35% accuracy.
  • Comparison of quantization clip-floor and quantization clip-floor-shift: With L = 4, converted-SNN accuracy increases with T and reaches source-ANN accuracy when T exceeds 16, unlike the unshifted activation.The unshifted converted SNN first improves and then declines rapidly as T increases.
  • Effect of quantization steps L: The recommended quantization step is L = 4 or 8 because larger L reduces ultra-low-latency accuracy, whereas very small L reduces model capacity and can lower accuracy.When L = 2, a clear gap remains between the best SNN and source-ANN accuracy; for L > 4, the best SNN accuracy approaches the source ANN.

7 DISCUSSION AND CONCLUSION

The method replaces ReLU with quantization clip-floor-shift activation and establishes zero expected conversion error, targeting accurate SNNs with very few time-steps. Remaining unevenness and one-step limitations leave directions for further research.

  • The quantization clip-floor-shift activation replaces ReLU and better matches SNN activation while minimally affecting source-ANN performance.The method is designed for ANN-SNN conversion with high accuracy and ultra-low latency.
  • The expected conversion error is zero regardless of whether SNN time-steps and ANN quantization steps match.
  • The method achieves state-of-the-art accuracy with fewer time-steps on CIFAR-10, CIFAR-100, and ImageNet.
  • Unevenness error can leave a gap between ANN and SNN accuracy even when L = T, while high-performance conversion remains difficult at T = 1.
  • Conversion methods may reduce overall computing cost while preserving comparable source-ANN performance, and model compression may further reduce neuron activity.

A.1 NETWORK STRUCTURE AND TRAINING CONFIGURATIONS

The conversion pipeline modifies the source ANN before training, then transfers its weights and aligns SNN thresholds and initial membrane potentials with learned activation statistics. Training uses dataset-specific optimization settings and fixed quantization steps.

  • The source ANN replaces max-pooling with average-pooling and ReLU with quantization clip-floor-shift activation before training.After training, weights are copied to the converted SNN.
  • Converted-SNN thresholds θl are set to the source ANN maximum activations λl, while initial membrane potentials are set to θl/2.
  • Images are resized to 32 × 32 for CIFAR and 224 × 224 for ImageNet, with random crops, Cutout, and AutoAugment applied across datasets.
  • Models are trained for 300 epochs using SGD with momentum 0.9, cosine learning-rate decay, and dataset-specific learning rates and weight decay.The quantization step L is set to 4 for the reported network training configurations.

A.2 INTRODUCTION OF DATASETS

The experiments use CIFAR-10, CIFAR-100, and ImageNet, spanning ten-class, 100-class, and large-scale image classification datasets with specified train-test sizes.

  • CIFAR-10 contains 60000 32 × 32 images across 10 classes, divided into 50000 training and 10000 test images.
  • CIFAR-100 contains 60000 32 × 32 images across 100 classes, divided into 50000 training and 10000 test images.
  • ImageNet uses ILSVRC 2012 with 1,281,167 training images and 50000 testing images.

A.3 DERIVATION OF EQUATION 12 AND PROOF OF THEOREM 2

The appendix derives conversion-error behavior from neuron and spike dynamics, proves zero expected error under stated conditions, and reports low-time-step accuracy, energy, and implementation results. It also documents limitations of maximum-activation thresholding and mismatched quantization.

  • Derivation of Equation 12 and proof of Theorem 2: Theorem 2 states that with θl = λl and vl(0) = θlϕ, expected conversion error is zero for arbitrary T and L when ϕ = 1.
  • Derivation of Equation 12 and proof of Theorem 2: Uneven input-spike timing can produce more or fewer output spikes than expected, creating conversion error.
  • Derivation of Equation 12 and proof of Theorem 2: In the illustrated T = 5 example, the SNN output firing rate is 0.6 versus ANN activation 0.4, showing excess spikes from timing unevenness.
  • Comparison of methods with or without dynamic threshold: Setting the threshold to maximum ANN activation causes long firing delays and larger quantization error, while dynamic thresholds perform better on CIFAR-100.
  • Influence of different quantization steps: 86.53% CIFAR-10 accuracy and 61.41% CIFAR-100 accuracy are achieved with VGG-16, L = 2, and T = 1.
  • Comparison with supervised training methods: 94.73% CIFAR-10 accuracy is achieved with 4 time-steps, whereas listed backpropagation methods need at least 5 time-steps for approximately 91% accuracy.
  • Comparison on CIFAR-100: On CIFAR-100, the method achieves 69.62% accuracy with 4 time-steps and exceeds SNNC-AP by 3.46% at T = 32 for VGG-16.
Loading 2303.04347v1…