PaperlayerView exact revision on arXiv

Source-linked AI summary

Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation

Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, Zhi-Quan Luo

arXiv:2205.00459v2cs.NEcs.LG

TL;DR

SNN training is difficult because non-differentiable spike trains hinder efficient gradient-based learning, while existing methods can require high latency or fall short of ANN-level performance. DSR trains through gradients of sub-differentiable mappings over weighted firing-rate spike representations, achieving competitive or state-of-the-art performance with low latency across static and neuromorphic datasets.

  • Problem

    Non-differentiable spike trains make SNNs difficult to train efficiently, and existing methods may involve high latency or lower performance than ANNs.

  • Method

    DSR represents spike trains with weighted firing rates and backpropagates through sub-differentiable mappings, while training thresholds and adding a neural-model hyperparameter to reduce representation error.

  • Results

    DSR achieves competitive or state-of-the-art SNN performance with low latency on CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10.

  • Takeaways & Limitations

    DSR supports low-latency SNN training without backpropagation through the temporal domain and remains effective under ultra-low latency or deep network structures.

  • Takeaways & Limitations

    DSR may suffer performance drops at extremely low latency, such as only 2 or 3 time steps, because accurate spike representation is required.

Abstract

from arXiv · show

Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. However, it is a challenge to efficiently train SNNs due to their non-differentiability. Most existing methods either suffer from high latency (i.e., long simulation time steps), or cannot achieve as high performance as Artificial Neural Networks (ANNs). In this paper, we propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance that is competitive to ANNs yet with low latency. First, we encode the spike trains into spike representation using (weighted) firing rate coding. Based on the spike representation, we systematically derive that the spiking dynamics with common neural models can be represented as some sub-differentiable mapping. With this viewpoint, our proposed DSR method trains SNNs through gradients of the mapping and avoids the common non-differentiability problem in SNN training. Then we analyze the error when representing the specific mapping with the forward computation of the SNN. To reduce such error, we propose to train the spike threshold in each layer, and to introduce a new hyperparameter for the neural models. With these components, the DSR method can achieve state-of-the-art SNN performance with low latency on both static and neuromorphic datasets, including CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10.

1. Introduction

SNNs promise energy-efficient neuromorphic AI, but their non-differentiable spike trains make effective training difficult. DSR uses spike representations and sub-differentiable mappings to target high performance with low latency.

  • SNNs offer energy efficiency on neuromorphic hardware, motivating their use as an alternative to energy-intensive ANNs.
  • Non-differentiable spike trains hamper gradient-based SNN training, while existing approaches face performance, latency, or training-cost limitations.Surrogate-gradient methods can use short simulations but require costly spatial and temporal backpropagation; ANN-to-SNN conversion is another major approach.
  • Table 1 compares ANN-to-SNN conversion, surrogate gradients, and DSR by latency, low-latency performance, and neuromorphic-data applicability.
  • DSR encodes spike trains with weighted firing-rate representations and trains through sub-differentiable mappings instead of differentiating at each time step.The method analyzes representation error and proposes trainable thresholds plus a new neural-model hyperparameter to reduce it.
  • DSR achieves competitive or state-of-the-art SNN performance with low latency on CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10.Experiments also evaluate ultra-low latency and deep network structures.

2. Related Work

Prior SNN training work includes biologically inspired learning, ANN-to-SNN conversion, and direct training. These approaches differ in how they represent spike information and manage temporal credit assignment.

  • Biologically inspired Hebbian-learning approaches seek biological plausibility but do not achieve competitive performance on complicated datasets.
  • SNN learning methods are mainly categorized as ANN-to-SNN conversion and direct training.
  • ANN-to-SNN conversion transfers parameters from an ANN, relying on firing-rate estimates to preserve the source network’s performance.Weight normalization, temporal switch coding, rate norm layers, and bias shifts reduce the performance gap.
  • Direct-training methods treat SNNs as RNNs and use BPTT with surrogate gradients or spike-time gradients.These approaches backpropagate through time steps and therefore suffer high computational costs.
  • SNN information is represented mainly through temporal coding based on firing times or rate coding based on firing rates.

3. Proposed Differentiation on Spike Representation (DSR) Method

DSR represents spike trains with firing-rate-based spike representations, recasts common SNN forward dynamics as sub-differentiable mappings, and backpropagates through those mappings. Trainable thresholds and an explicit Δt hyperparameter reduce representation error while avoiding temporal backpropagation and spike-function non-differentiability.

  • Spiking neural models: IF and LIF neurons integrate input into membrane potentials, emit spikes when the threshold is exceeded, and reset their potentials.The discretized dynamics use a membrane update function f and a reduce-by-subtraction reset rule.
  • Spike representation: The SNN forward pass is approximated layer by layer by a sub-differentiable mapping from the previous layer’s spike representation to the next.The mapping is parameterized by each layer’s trainable weights.
  • Spike representation: Weighted firing-rate coding converts spike trains into representations that carry the information used to derive the mapping for LIF and IF models.For LIF neurons, the representation is related to a weighted average input current and a sub-differentiable mapping.
  • Differentiation on spike representation: The DSR training algorithm backpropagates through spike representations rather than through individual time steps or discontinuous spike functions.This avoids the temporal-domain error propagation required by BPTT and the usual spike-function non-differentiability problem.
  • Error reduction: Trainable layerwise spike thresholds and a new Δt hyperparameter reduce finite-time-step representation error and support high performance with low latency.The explicit Δt creates a larger feasible region for the LIF time constant τ, requiring Δt < τ.

4. Reducing Representation Error

Finite simulation steps introduce representation error, so the paper separates quantization and deviation errors and proposes threshold training and a firing-control hyperparameter to reduce them.

  • Error sources: Representation error er consists of quantization error eq from finite firing-rate precision and deviation error ed from varying input currents.The deviation error is expected to be zero under i.i.d. input currents and therefore is less problematic for stochastic optimization.
  • Quantization error: The original IF firing-rate approximation uses floor rounding, producing quantized outputs determined by the latency N.For unchanged input currents, the scaled average firing rate applies a clamped floor-rounded quantity.
  • Training the Spike Threshold: Small spike thresholds reduce quantization error but narrow the scaled firing-rate range, so each layer’s threshold is trained with an L2 regularizer.The regularizer balances quantization error against approximation capacity.
  • Introducing a new hyperparameter: A new hyperparameter α ∈[0, 1] controls neuron firing and changes the IF firing mechanism to reduce quantization error.For the IF model, the modified firing mechanism uses α = 0.5.
  • Introducing a new hyperparameter: The modified firing mechanism halves the maximum absolute quantization error, while α = 0.5 minimizes average absolute quantization error for the IF model.For the LIF model, the best α depends on latency N, so experiments use different values.

5. Experiments

The experiments evaluate DSR across static and neuromorphic recognition benchmarks, latency settings, network depths, and representation-error ablations. DSR generally achieves strong performance with low latency, including under reduced time steps.

  • 5. Experiments: The benchmark study covers CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10, with results summarized in Table 2.The evaluation compares ANN, ANN-to-SNN conversion, direct training, and the proposed method for the first three datasets.
  • 5.1. Comparison to the State-of-the-Art: DSR outperforms other methods on CIFAR-10 and CIFAR-100 at 20 time steps for both IF and LIF models.On CIFAR-100, its accuracy is 5%-10% higher than other methods, while matching or exceeding ANNs with the same architectures.
  • 5.1. Comparison to the State-of-the-Art: DSR also outperforms direct-training methods on ImageNet and achieves better performance than some ANN-to-SNN methods while using fewer time steps.The ImageNet experiments use pre-activation ResNet-18 and hybrid training; some ANN-to-SNN methods achieve higher accuracy but require much longer simulations.
  • 5.1. Comparison to the State-of-the-Art: DSR outperforms other state-of-the-art methods on DVS-CIFAR10 with low latency for both IF and LIF models.The neuromorphic experiments use VGG-11 and three runs for each neural model.
  • Effectiveness of the Proposed Method with Low Latency: Less than 1% accuracy drop occurs when reducing DSR simulations from 20 to 5 time steps.The 5-time-step results also outperform other state-of-the-art methods reported in Table 2.
  • Effectiveness of the Proposed Method with Deep Network Structure: The experiments additionally test DSR with deeper pre-activation ResNets and ablate threshold training and firing-mechanism modification for reducing representation error.The ablation uses CIFAR-10, PreAct-ResNet-18, the IF model, and 20 time steps.

6. Conclusion and Discussions

The paper concludes that DSR trains low-latency SNNs through sub-differentiable spike-representation mappings and performs strongly across vision tasks. Its main limitation is possible performance degradation at extremely low latency.

  • 6. Conclusion and Discussions: DSR represents SNN forward computation as a sub-differentiable mapping and avoids temporal-domain backpropagation.The method also trains thresholds and adds a neural-model hyperparameter to reduce representation error.
  • 6. Conclusion and Discussions: Experiments report state-of-the-art performance on mainstream vision tasks and effectiveness with ultra-low latency or very deep network structures.These conclusions follow the paper’s evaluations across the stated task and architecture settings.
  • 6. Conclusion and Discussions: DSR may suffer a performance drop at extremely low latency, such as only 2 or 3 time steps.The stated reason is that DSR requires relatively accurate spike representation for backpropagation.

A.1. Derivation for Eq.

The supplementary derivations establish limiting spike-rate mappings for LIF and IF neurons under mild assumptions. Both analyses control the contribution of the remaining membrane potential and yield clamp-based approximations.

  • LIF derivation: For the LIF model, the derivation separates accumulated membrane potential into exceeded and remaining components.The exceeded component does not contribute to spike firing, while the remaining component is the residual potential.
  • LIF assumptions: The LIF derivation assumes appropriate limiting firing behavior and V+[N] ∈ [0, Vth].It also requires the relevant right-hand-side limit to exist.
  • LIF derivation: The LIF spike-rate approximation has error bounded by Vth/τ as N → ∞.This bound follows after approximating the remaining membrane potential with a clamp expression.
  • IF derivation: For the IF model, the scaled firing rate is related to the average input current through a clamp expression as N → ∞.The derivation defines the scaled firing rate and average input current before taking the limit.
  • IF assumptions: The IF derivation assumes the relevant input-current limit exists, the neuron’s limiting firing behavior is appropriate, and V+[N] ∈ [0, Vth].The bounded-residual assumption excludes extreme input distributions that make the remaining potential unbounded.

B. Pseudocode of the Proposed DSR Method

Algorithm 1 describes one DSR training iteration by simulating IF or LIF neurons across time and layers, computing the loss, backpropagating through the mappings, and updating weights and thresholds.

  • Inputs and outputs: The algorithm takes time steps, network depth, network parameters, thresholds, input data, labels, and other hyperparameters as inputs.Its outputs are trained network parameters and thresholds.
  • Forward pass: For each time step and layer, the forward pass computes neuron states using the IF or LIF model.At the final time step, the algorithm computes the layer output according to the selected model.
  • Backward pass: The backward pass computes the loss from the final output and label, then propagates derivatives through the network layers.The pseudocode calculates derivatives with respect to the output and recursively applies the chain rule.
  • Parameter update: The iteration updates both network weights and neuron thresholds after backpropagation.Thresholds are included among the trained network parameters in the algorithm specification.

C.1. Dataset Description and Preprocessing

The experiments use CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10 with dataset-specific preprocessing. Batch normalization is adapted to temporal SNN data by combining batch and time dimensions.

  • Dataset preprocessing: CIFAR-10 contains 60,000 32×32 color images across 10 classes, while CIFAR-100 contains 100 classes and uses the same preprocessing.Both datasets use normalization, random cropping, and horizontal flipping.
  • Dataset preprocessing: ImageNet-1K contains 1,281,167 training images, 50,000 validation images, and 100,000 test images across 1000 classes.Preprocessing includes normalization, random resized cropping, and horizontal flipping.
  • Dataset preprocessing: DVS-CIFAR10 contains 10,000 event-based 128×128 images in 10 classes, with each spike train divided into 20 integrated slices.The dataset is converted from CIFAR-10 using an event-based sensor.
  • Batch normalization: Batch normalization reshapes temporal data by combining batch and time dimensions before computing its mean and variance.The transformed data uses learnable γ and β parameters with ε stabilizing division.

C.3. Network Architectures

The experiments use different network architectures for static and neuromorphic datasets, with SNN-specific architectural changes for implementation and generalization.

  • Static datasets: CIFAR-10, CIFAR-100, and ImageNet use PreAct-ResNet-18 with spiking neurons after pooling operations and the final classifier.Max pooling is replaced by average pooling, and an additional batch-normalization operation stabilizes the output layer’s weighted firing rates.
  • Neuromorphic dataset: DVS-CIFAR10 uses VGG-11 with dropout after spiking neurons and only one fully connected layer.The dropout probability is 0.1, reducing the number of neurons while supporting generalization.

C.4. Training Hyperparameters

Training uses model-specific threshold and neural-dynamics hyperparameters together with cosine-annealed optimization settings, with several LIF values varying by simulation length.

  • IF model: The IF model initializes each layer’s threshold V_i_th at 6, constrains it to at least 0.01 during training, and sets α to 0.5.These settings control the IF model’s threshold and hyperparameter configuration.
  • LIF model: The LIF model fixes each layer’s time constant τ_i to 1, while its threshold-related settings, Δt, and α vary with the number of time steps.The corresponding values are listed in Table 5.
  • Optimization: Cosine annealing is used as the learning-rate schedule across CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10.When using 5 time steps, the initial learning rate changes from 0.1 to 0.05.

D. Firing Sparsity

DSR-trained SNNs exhibit sparse firing across layers while maintaining similar firing rates across tested time-step settings; the reported implementation also evaluates reduced weight precision.

  • Firing rates: All layers have firing rates below 20%, with many layers at no more than 5%.The measurements are from CIFAR-10 models using PreAct-ResNet-18 and different time steps.
  • Firing rates: 7.5%-9.5% total firing rates are observed across different time steps and both IF and LIF neuron models.The firing rate does not increase as the number of time steps decreases.
  • Latency and sparsity: The method is described as achieving low latency and high firing sparsity because firing rates remain stable when time steps decrease.This connects temporal efficiency with sparse spike generation in the reported experiments.
  • Weight precision: The trained SNNs use 32-bit weights, and CIFAR-10 experiments additionally quantize them to 8-bit and 4-bit precision.Quantization uses straight-through estimation.
Loading 2205.00459v2…