PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimized Potential Initialization for Low-latency Spiking Neural Networks

Tong Bu, Jianhao Ding, Zhaofei Yu, Tiejun Huang

arXiv:2202.01440v1cs.NEcs.CVcs.LG

TL;DR

ANN-to-SNN conversion enables high-performing deep SNNs, but achieving high precision requires long simulations and creates an accuracy–latency trade-off. The paper analyzes conversion and optimizes initial membrane potentials, reporting state-of-the-art comparisons across CIFAR-10, CIFAR-100, and ImageNet with reduced inference time.

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) have been attached great importance due to the distinctive properties of low power consumption, biological plausibility, and adversarial robustness. The most effective way to train deep SNNs is through ANN-to-SNN conversion, which have yielded the best performance in deep network structure and large-scale datasets. However, there is a trade-off between accuracy and latency. In order to achieve high precision as original ANNs, a long simulation time is needed to match the firing rate of a spiking neuron with the activation value of an analog neuron, which impedes the practical application of SNN. In this paper, we aim to achieve high-performance converted SNNs with extremely low latency (fewer than 32 time-steps). We start by theoretically analyzing ANN-to-SNN conversion and show that scaling the thresholds does play a similar role as weight normalization. Instead of introducing constraints that facilitate ANN-to-SNN conversion at the cost of model capacity, we applied a more direct way by optimizing the initial membrane potential to reduce the conversion loss in each layer. Besides, we demonstrate that optimal initialization of membrane potentials can implement expected error-free ANN-to-SNN conversion. We evaluate our algorithm on the CIFAR-10, CIFAR-100 and ImageNet datasets and achieve state-of-the-art accuracy, using fewer time-steps. For example, we reach top-1 accuracy of 93.38\% on CIFAR-10 with 16 time-steps. Moreover, our method can be applied to other ANN-SNN conversion methodologies and remarkably promote performance when the time-steps is small.

Introduction

ANN-to-SNN conversion supports high-performing deep SNNs, but matching firing rates to ANN activations creates an accuracy–latency trade-off. The paper analyzes threshold scaling and optimizes initial membrane potentials to improve conversion at low latency.

Introduction

  • ANN-to-SNN conversion has yielded strong performance in deep networks and large-scale datasets, but high precision requires long simulation times to match spiking and analog neuron outputs.Source: Introduction · p. 2
  • Optimizing initial membrane potentials can yield expected error-free conversion while reducing inference time and improving accuracy at low latency.The analysis identifies half the firing threshold as the initial potential that minimizes expected squared conversion error and makes expected conversion error zero.Source: Introduction · p. 2
  • Threshold scaling plays a role similar to weight normalization, helping explain how threshold balancing can reduce conversion loss and inference latency.Source: Introduction · p. 2
  • The method achieves state-of-the-art accuracy on nearly all tested datasets and network structures while using fewer time-steps.Source: Introduction · p. 2
  • The method can be applied to other ANN-SNN conversion methodologies and promote performance when time-steps are small.Source: Introduction · p. 2

Related Work

Prior work trains SNNs through gradient-based optimization or ANN-to-SNN conversion. The paper distinguishes its approach by directly optimizing initial membrane potentials for low-inference-time performance.

Related Work

  • Gradient-based methods include activation-based approaches using backpropagation through time and timing-based approaches, with depth limitations for each.Activation-based gradients become unstable beyond 50 layers, while timing-based methods are usually limited to networks under 10 layers.Source: Related Work · p. 2
  • ANN-to-SNN conversion can produce deep SNNs with performance comparable to deep ANNs, but prior conversion methods still face accuracy loss at relatively small time-steps.Prior work includes weight normalization, threshold balancing, soft reset, dynamic thresholds, and methods for fitting activation and firing-rate curves.Source: Related Work · p. 2
  • Unlike the reviewed approaches, this paper directly optimizes initial membrane potentials to increase performance at low inference time.Source: Related Work · p. 3

Methods

The methods relate ANN activations to SNN firing rates, then optimize membrane-potential initialization to reduce conversion error. They also derive a threshold-scaling connection to weight normalization.

ANNs and SNNs

  • The IF model integrates weighted inputs and bias, fires when membrane potential exceeds its threshold, and uses reset-by-subtraction after firing.The potential is reduced by the firing threshold after a spike.Source: ANNs and SNNs · p. 3

Theory for ANN-SNN conversion

  • At sufficiently large latency, ANN activations and SNN firing rates can be related by copying weights and biases and setting each threshold to the ANN activation upper bound.Threshold scaling plays a role similar to weight normalization and can reduce conversion loss while improving inference latency.Source: Theory for ANN-SNN conversion · pp. 3–4

Optimal initialization of membrane potentials

  • Finite-time conversion has quantization error because firing rates are discrete, motivating direct optimization of membrane potentials to reduce error.The paper identifies low-latency differences between ANN activations and SNN firing rates as a source of layer-by-layer accuracy degradation.Source: Optimal initialization of membrane… · p. 4
  • The conversion algorithm trains a clipped ANN with trainable upper bounds, replaces max pooling with average pooling, then transfers parameters and initializes membrane potentials.The algorithm updates weights, biases, and clipping bounds by stochastic gradient descent before conversion.Source: Optimal initialization of membrane… · p. 4
  • Setting the initial membrane potential to half the firing threshold minimizes expected squared conversion error and makes the expected conversion error zero.The derivation assumes activations lie between zero and the threshold and uses a distributional assumption over that interval.Source: Optimal initialization of membrane… · pp. 4–5

Experiments

The experiments evaluate membrane-potential initialization and converted-SNN accuracy across CIFAR-10, CIFAR-100, and ImageNet, including tests with other conversion methods. They report strong accuracy at low time-steps and improvements over zero initialization.

Implementation details

  • The conversion copies source-ANN weights and biases, sets each firing threshold to the analog activation upper bound, and initializes each layer’s neurons to half that threshold.The evaluation uses VGG-16, ResNet-20, and ResNet-18; source max-pooling layers are replaced with average-pooling layers and trainable clipping layers are added.Source: Implementation details · p. 5

The effect of membrane potential initialization

Comparison with the State-of-the-Art

The effect of membrane potential initialization

  • Non-zero initial potentials outperform zero initialization, and optimal initialization performs best; zero initialization cannot work below 10 time-steps.At more than 256 time-steps, all tested initialization methods reach the source ANN accuracy; the authors relate zero initialization’s early inactivity to delayed first spikes.Source: The effect of membrane potential… · p. 6
  • With thresholds scaled to 1, an initial potential of 0.5 is theoretically optimal and gives optimal or near-optimal performance across the tested networks and datasets.The derivation assumes a piecewise uniform distribution; the reported optimal values fall between 0.4 and 0.6.Source: Figure 3 · p. 5, The effect of membrane potential… · pp. 6–7

Comparison with the State-of-the-Art

  • 74.82% VGG-16 accuracy at 32 time-steps on CIFAR-100 exceeds the reported 7.64% for Opt. and 73.55% for Calibration at the same duration.The proposed method reaches 52.34% and 72.85% top-1 accuracy for ResNet-20 and ResNet-18, respectively, at 16 time-steps.Source: Table 2 · p. 7, Comparison with the State-of-the-Art · p. 7
  • 64.70% and 72.47% top-1 accuracy are reported for ImageNet VGG-16 at 32 and 64 time-steps, respectively.The authors characterize the method as effective on this large dataset and state-of-the-art across the evaluated datasets.Source: Comparison with the State-of-the-Art · p. 7, Table 3 · p. 8

Apply optimal initial potentials to other models

Conclusion

The paper derives a relationship between ANN forwarding and SNN dynamics, and presents optimized membrane-potential initialization as a route to expected error-free conversion. It reports state-of-the-art comparisons across CIFAR-10, CIFAR-100, and ImageNet, with compatibility across conversion methods and low inference time.

Conclusion

  • The paper derives a relationship between ANN forwarding and SNN dynamics.The derivation frames its analysis of ANN-to-SNN conversion.Source: Conclusion · p. 8
  • Optimal membrane-potential initialization can implement expected error-free ANN-to-SNN conversion and reduce time to first spike, shortening inference time.Source: Conclusion · p. 8
  • The converted SNN is reported to outperform state-of-the-art comparisons on CIFAR-10, CIFAR-100, and ImageNet, while the algorithm is compatible with other conversion methods and promotes performance at low inference time.Source: Conclusion · p. 8

Proofs of Theorem 1

The proof identifies the threshold midpoint as the initialization that minimizes expected squared conversion error and yields zero expected error under its distributional assumption.

Proofs of Theorem 1

  • The expected squared conversion error is minimized when the initial membrane potential is V_l^th/2.The derivation assumes z_i is uniformly distributed within each small interval [mt, mt+1].Source: Proofs of Theorem 1 · p. 10
  • At this initialization, the expected conversion error is zero for each neuron i.The proof states the zero-error condition for i = 1, 2, ..., N.Source: Proofs of Theorem 1 · pp. 10–11
  • The proof reduces the minimization to minimizing the contribution from each element of z.N is the number of elements in z, or neurons in layer l.Source: Proofs of Theorem 1 · p. 10

The effect of membrane potential initialization

Across the tested network structures and datasets, initial potentials from 0.4 to 0.6 are optimal, with 0.5 achieving optimal or near-optimal performance.

The effect of membrane potential initialization

  • Initial potentials from 0.4 to 0.6 are optimal, and 0.5 achieves optimal or near-optimal performance.The comparison varies time steps from 1 to 75 and initial potential from 0.1 to 0.9.Source: The effect of membrane potential… · p. 11

Energy Estimation on Neuromorphic Hardware

The energy analysis estimates operation counts using hardware power efficiencies and reports substantial energy-efficiency gains for converted SNNs at 32 time steps.

Energy Estimation on Neuromorphic Hardware

Related papers

Loading 2202.01440v1…