Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks

Wei Fang; Zhaofei Yu; Yanqi Chen; Timothee Masquelier; Tiejun Huang; Yonghong Tian

Source-linked AI summary

Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks

Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, Yonghong Tian

arXiv:2007.05785v5Jul 11, 2020cs.NEcs.CVcs.LG

TL;DR

SNN learning commonly fixes shared membrane dynamics while optimizing synaptic weights, limiting neuron heterogeneity. This paper introduces PLIF neurons that learn membrane time constants with weights and reevaluates pooling; the resulting SNNs outperform state-of-the-art comparisons across static and neuromorphic datasets, while showing greater robustness to initialization and faster learning.

Problem
Existing SNN learning methods generally optimize synaptic weights while manually fixing a shared membrane time constant, despite membrane dynamics affecting neuron behavior and expressiveness.
Method
The paper introduces Parametric Leaky Integrate-and-Fire neurons and a backpropagation-based algorithm that learns membrane time constants alongside synaptic weights, while evaluating max- and average-pooling.
Results
The proposed PLIF SNNs outperform state-of-the-art comparison methods on static and neuromorphic datasets, are more robust to initial values, and learn faster than LIF-based SNNs.
Takeaways & Limitations
Learnable membrane parameters improve neuron heterogeneity, robustness to initialization, and learning speed, while max-pooling offers lower computation cost and comparable or better performance than average-pooling.
Takeaways & Limitations
The membrane time constant is shared among neurons within the same layer, an assumption motivated by neighboring neurons having similar properties.

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) have attracted enormous research interest due to temporal information processing capability, low power consumption, and high biological plausibility. However, the formulation of efficient and high-performance learning algorithms for SNNs is still challenging. Most existing learning methods learn weights only, and require manual tuning of the membrane-related parameters that determine the dynamics of a single spiking neuron. These parameters are typically chosen to be the same for all neurons, which limits the diversity of neurons and thus the expressiveness of the resulting SNNs. In this paper, we take inspiration from the observation that membrane-related parameters are different across brain regions, and propose a training algorithm that is capable of learning not only the synaptic weights but also the membrane time constants of SNNs. We show that incorporating learnable membrane time constants can make the network less sensitive to initial values and can speed up learning. In addition, we reevaluate the pooling methods in SNNs and find that max-pooling will not lead to significant information loss and have the advantage of low computation cost and binary compatibility. We evaluate the proposed method for image classification tasks on both traditional static MNIST, Fashion-MNIST, CIFAR-10 datasets, and neuromorphic N-MNIST, CIFAR10-DVS, DVS128 Gesture datasets. The experiment results show that the proposed method outperforms the state-of-the-art accuracy on nearly all datasets, using fewer time-steps. Our codes are available at https://github.com/fangwei123456/Parametric-Leaky-Integrate-and-Fire-Spiking-Neuron.

1. Introduction

SNN learning remains challenging because existing methods typically learn synaptic weights while fixing shared membrane parameters, limiting neuronal heterogeneity and expressiveness. The paper proposes learning membrane time constants alongside weights and reevaluates pooling choices.

SNNs attract interest for temporal information processing, low power consumption, and biological plausibility, but efficient high-performance learning remains challenging.
Most existing SNN learning methods optimize synaptic parameters while treating membrane-related parameters as manually selected hyperparameters.
Using the same membrane time constant for all neurons limits neuronal heterogeneity and the expressiveness of resulting SNNs.
The proposed training algorithm learns both synaptic weights and membrane time constants, with different adjustments affecting neuronal dynamics.
The paper reevaluates pooling methods and reports comparable performance for max-pooling and average-pooling while emphasizing max-pooling’s lower computation cost and binary compatibility.

2. Related Works

Prior SNN research includes unsupervised, reward-based, conversion-based, and membrane-parameter methods, each with distinct limitations. The paper positions systematic learning of membrane time constants as an unresolved research direction.

Unsupervised learning of SNNs: Unsupervised SNN methods use local rules such as Hebbian learning and STDP, but are mainly suitable for shallow networks and remain below state-of-the-art ANN performance.
Reward-based learning of SNNs: Reward-based SNN methods use reinforcement-learning or STDP-inspired mechanisms driven by reward or punishment signals.
ANN to SNN conversion: ANN-to-SNN conversion can achieve near-lossless ANN inference, but trades accuracy against latency and is restricted to rate coding for temporal tasks.
Earlier work learned membrane leak and firing threshold when fine-tuning converted SNNs, but systematic research on learning membrane time constants had not been conducted.

3. Methods

The methods learn membrane time constants alongside synaptic weights through PLIF neurons, while using a discrete spiking-neuron formulation and spike max-pooling. Layer-specific time constants increase neuronal diversity, and spike max-pooling dynamically selects transmitting neurons.

Parametric Leaky Integrate-and-Fire model: PLIF neurons learn synaptic weights and membrane time constants, rather than treating the time constant as a fixed hyperparameter.The trainable time constant is optimized during training, with shared values within layers and distinct values across layers.
Parametric Leaky Integrate-and-Fire model: PLIF rules increase neuron heterogeneity and network expressiveness while controlling computation costs.Time constants are shared within layers for neighboring neurons but differ across layers to produce diverse phase-frequency responsiveness.
Discrete spiking-neuron formulation: The discrete spiking-neuron model separates neuronal dynamics, spike generation, and membrane reset.The general formulation uses a dynamics function, a threshold-triggered spike, and a hard reset to Vreset after firing.
Parametric Leaky Integrate-and-Fire model: The PLIF reformulation trains a bounded parameter k(a) instead of directly optimizing τ, ensuring τ > 1 and avoiding numerical instability.The experiments use the sigmoid activation as the clamp function, giving k(a) ∈ (0, 1).
Parametric Leaky Integrate-and-Fire model: The membrane time constant controls the balance between remembering current input and forgetting past membrane information.In the recurrent interpretation of LIF and PLIF neurons, this balance is analogous to gating in LSTM networks.
Network formulation: The network combines convolutional-spiking encoder modules, pooling layers, and fully connected spiking classifier layers, with parameters shared across time-steps.Inputs are fed directly to the network, whose first convolutional-spiking module performs learnable image-to-spike encoding.
Spike max-pooling: Spike max-pooling applies winner-take-all transmission at each time-step: only a firing neuron in the pooling window connects forward.If multiple neurons fire simultaneously, one connection is randomly selected; asynchronous firing lets neurons connect in turn.

4. Experiments

Experiments evaluate PLIF neurons and pooling choices across static and neuromorphic classification datasets. PLIF improves robustness to initialization and learning speed, while max-pooling achieves comparable accuracy with lower computational cost.

The evaluation covers MNIST, Fashion-MNIST, CIFAR-10, N-MNIST, CIFAR10-DVS, and DVS128 Gesture classification tasks.
The proposed PLIF method achieves the highest accuracy on all evaluated datasets except CIFAR-10, where it is slightly below an ANN-to-SNN conversion method.
The proposed method uses fewer inference time-steps than prior best-performing methods, reaching up to 256× fewer steps than ANN-to-SNN conversion.
PLIF ablation: PLIF neurons maintain higher accuracy and faster convergence than LIF neurons when the initial membrane time constant is poorly chosen.
PLIF ablation: Different initial membrane time constants tend to converge during training, indicating that PLIF neurons can learn suitable time constants from varied starting values.
Pooling ablation: Max-pooling performs similarly to average-pooling and achieves slightly higher accuracy on CIFAR-10, CIFAR10-DVS, and DVS128 Gesture.

5. Conclusion

The conclusion presents PLIF neurons as a way to learn membrane time parameters in SNNs while improving performance across static and neuromorphic datasets. It also supports max-pooling for its accuracy, computational, temporal, and binary-spike advantages.

PLIF neurons incorporate learnable membrane time parameters and outperform state-of-the-art comparison methods on static and neuromorphic datasets.
PLIF-based SNNs are more robust to initial values and learn faster than SNNs using LIF neurons.
Max-pooling is recommended because it has lower computation cost, higher temporal fitting capability, and binary spike input-output compatibility.

B. Reproducibility

The implementation uses SpikingJelly with identical random seeds and publishes source code and training logs. Network structures are specified for each dataset using convolutional, pooling, PLIF, dropout, fully connected, and voting layers.

All experiments use SpikingJelly, identical seeds, and publicly available source code and training logs.
Dataset-specific architectures combine convolutional layers, batch normalization, PLIF neurons, max-pooling, dropout, fully connected layers, and average-pooling voting.

D. Training Algorithm to Fit Target Output

The training algorithm fits SNN outputs to target class-coded spike sequences by iteratively simulating the network, computing loss, and updating parameters with gradient descent.

The SNN output is a C × T tensor, and the target assigns one to the labeled class and zero to all other classes at each time-step.
Training iterates over input time-steps, collects output spikes, computes a loss against the target, and updates network parameters using the loss gradient.
The loss is a distance between target outputs and network outputs, with mean squared error given as an example.

E. Introduction of the Datasets

The paper evaluates static image datasets and neuromorphic event datasets, converting asynchronous events into fixed time-slice frames for SNN processing.

Datasets: MNIST, Fashion-MNIST, and CIFAR-10 provide static image classification benchmarks.MNIST and Fashion-MNIST contain 60,000 training and 10,000 test examples; CIFAR-10 contains 50,000 training and 10,000 test images.
Datasets: N-MNIST is a neuromorphic version of MNIST recorded with an ATIS sensor and contains 60,000 training and 10,000 test examples.
Datasets: CIFAR10-DVS is a neuromorphic CIFAR-10 dataset with 10,000 examples, using the first 9,000 samples per class for training and the remaining 1,000 for testing.
Datasets: DVS128 Gesture contains 11 hand-gesture classes recorded from 29 subjects under three illumination conditions.
Preprocessing: For neuromorphic data, asynchronous events are divided into T nearly equal-count slices and integrated into frames, with T also serving as the experiment’s number of time-steps.The representation sums events in each slice; the slicing uses minimal and maximal timestamp indexes for each slice.

G. Hyper-Parameters

The experiments use fixed neuron voltage settings and standard optimization choices, while reporting both conventional test accuracy and validation-based accuracy estimates.

Optimization: Training uses Adam with learning rate 0.001, cosine annealing over 64 schedule steps, batch size 16, and dropout probability 0.5.
Neuron settings: All neurons use Vreset = 0 and Vth = 1, with the threshold fixed rather than fine-tuned for individual tasks.The paper describes Vth = 1 as implementing an implicit weight normalization that can mitigate exploding and vanishing gradients.
Evaluation: The proposed method is evaluated with and without a validation split across the different datasets.Table 7 is specifically described as comparing accuracy under these two evaluation settings.
Evaluation: The main-text protocol alternates training and test evaluation while recording maximum test accuracy, which the authors note overestimates accuracy.The validation protocol uses 85% of each class for training and 15% for validation, then tests once using the model with maximum validation accuracy.

I. Distribution of the First Wfc

During training, the learned first fully connected layer’s membrane time constants converge despite different initial values, including convergence toward non-leaky behavior after that layer.

Distribution of the First Wfc: PLIF neurons after the first fully connected layer learn to become Non-Leaky-Integrate-and-Fire neurons as 1/τ converges.
Distribution of the First Wfc: Figure 8 presents the distribution of the first fully connected layer’s learned parameter during CIFAR10-DVS training.

J. Visualization of Spiking Encoder

The visualizations show that the spiking encoder extracts structured spatial and semantic features, with firing-rate representations becoming more distinct over time and output spikes selectively responding to gestures.

CIFAR10-DVS: As Ts increases, firing-rate textures from channels 40 and 103 become more distinct, resembling the use of a Poisson encoder.
Visualization scope: The visualizations focus on the first spiking-neuron layer because deeper-layer spikes contain more semantic features but are harder to read and understand.
Fashion-MNIST: Fashion-MNIST firing rates from channels 45, 75, and 76 detect upper, left, and right edges of input images.The visualization uses ten input images and the corresponding firing rates at Ts=8 in the first PLIF layer.
CIFAR10-DVS: For CIFAR10-DVS, accumulated firing rates can reconstruct images filtered by the convolutional layer and produce clearer textures than binary output spikes.The comparison covers firing rates across all 128 channels at Ts=19 and binary output spikes.
DVS128 Gesture: DVS128 Gesture output spikes from channel 59 almost exclusively contain the gesture response spikes.The visualization compares three gesture samples with corresponding spikes from the first conventional PLIF layer.

K. Relations between different Encoders

The paper relates rate, latency, and learnable encoders by comparing how image intensities are converted into spike patterns. Learnable encoders can produce richer, semantically informative patterns than simplified latency encoding.

Rate encoding: Poisson rate encoding fires a spike at each time-step with probability equal to the image pixel intensity.Over T time-steps, its expected spike count is pT.
Learnable encoding: The proposed network directly feeds images into a Conv2d-Spiking Neurons module, whose first layer acts as a learnable encoder.This avoids first converting images into spikes with a separate encoder.
Latency encoding: A latency encoder represents each pixel with one spike whose firing time is inversely proportional to pixel intensity.Larger input intensity produces an earlier spike.
Empirical visualization: Figures 10–12 visualize channel spikes, firing rates, and input-output activity across time for CIFAR10-DVS and DVS128 Gesture samples.The visualizations include all 128 first-layer channels and selected channels across time-steps.
Learnable encoding: The proposed learnable encoders use learnable weights and multiple channels to generate complex spike patterns with more semantic information.For DVS128 Gesture samples, the encoding can preserve the gesture while discarding the player identity.