PaperlayerView exact revision on arXiv

Source-linked AI summary

GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks

Xingting Yao, Fanrong Li, Zitao Mo, Jian Cheng

arXiv:2210.13768v4cs.NEcs.AIcs.CV

TL;DR

Existing LIF-based spiking neurons generally provide only single biological features across neuronal behaviors, limiting neuronal dynamic diversity. GLIF unifies dual bio-features with learnable gating and channel-wise membrane parameters, and experiments report superior performance across datasets, including 75.48% top-1 accuracy on CIFAR-10 with two time steps.

  • Problem

    Most LIF-based neurons support only a single biological feature in each neuronal behavior, limiting their expressiveness and neuronal dynamic diversity.

  • Method

    GLIF fuses dual bio-features across membrane leakage, integration accumulation, and spike initiation using learnable gating factors and channel-wise learnable membrane parameters.

  • Results

    75.48% top-1 accuracy was achieved on CIFAR-10 with two time steps, while experiments across static and neuromorphic datasets demonstrated GLIF effectiveness.

  • Takeaways & Limitations

    GLIF enlarges the representation space and increases neuronal dynamic diversity and adaptivity by allowing heterogeneous, changing spiking neurons.

  • Takeaways & Limitations

    Applying GLIF in other training frameworks, including ANN2SNN conversion, remains a problem for future study.

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) have been studied over decades to incorporate their biological plausibility and leverage their promising energy efficiency. Throughout existing SNNs, the leaky integrate-and-fire (LIF) model is commonly adopted to formulate the spiking neuron and evolves into numerous variants with different biological features. However, most LIF-based neurons support only single biological feature in different neuronal behaviors, limiting their expressiveness and neuronal dynamic diversity. In this paper, we propose GLIF, a unified spiking neuron, to fuse different bio-features in different neuronal behaviors, enlarging the representation space of spiking neurons. In GLIF, gating factors, which are exploited to determine the proportion of the fused bio-features, are learnable during training. Combining all learnable membrane-related parameters, our method can make spiking neurons different and constantly changing, thus increasing the heterogeneity and adaptivity of spiking neurons. Extensive experiments on a variety of datasets demonstrate that our method obtains superior performance compared with other SNNs by simply changing their neuronal formulations to GLIF. In particular, we train a spiking ResNet-19 with GLIF and achieve $77.35\%$ top-1 accuracy with six time steps on CIFAR-100, which has advanced the state-of-the-art. Codes are available at \url{https://github.com/Ikarosy/Gated-LIF}.

1 Introduction

GLIF addresses the limited diversity of simplex LIF neurons by fusing biological features across leakage, integration, and spike initiation. Learnable gating and channel-wise parameters enlarge neuronal representation and support heterogeneous, adaptive spiking dynamics, with strong benchmark results.

  • Motivation: SNNs use spike-based communication for energy-efficient processing and can capture temporal information in neuromorphic data.The stated benefits arise from high sparsity and multiplication-free information processing.
  • Motivation: Existing vanilla and variant LIF neurons typically support only a single biological feature in each neuronal behavior.Examples include exponential versus linear membrane-potential decay, motivating hybrid bio-features.
  • GLIF: GLIF fuses dual bio-features for membrane leakage, integration accumulation, and spike initiation through learnable gating factors.Sigmoid-computed factors determine each feature’s contribution and can reduce to single-feature behavior at 0 or 1.
  • GLIF: Channel-wise learnable membrane parameters give different SNN channels distinct, changing spiking neurons, increasing dynamic diversity, heterogeneity, expressive ability, and adaptivity.The method trains these parameters during learning, paralleling the paper’s motivation from neuronal maturation during development.
  • Results: 75.48% top-1 accuracy was achieved on CIFAR-10 with only two time steps, demonstrating GLIF effectiveness on a static benchmark.The result is reported as state of the art in the introduction.

2 Related Work

Prior work develops LIF variants with different biological features, learnable membrane parameters, and surrogate-gradient training, but these approaches address separate aspects of spiking-neuron design.

  • Variant LIF: Variant LIF models modify membrane-potential leakage, integration accumulation, or spike initiation to represent different biological features.The related-work discussion distinguishes these three neuronal behaviors and describes exponential decay as one leakage mechanism.
  • Parametric spiking neurons: Parametric spiking neurons make one or more membrane-related parameters learnable instead of manually tuned or universally shared.Prior examples learn the membrane leak or potential threshold during SNN training.
  • Supervised direct learning: Supervised direct SNN learning uses surrogate gradients to approximate the non-differentiable spiking term during backpropagation.The related-work passage attributes this approach to approximating the local derivative of membrane potential.

3 Methodology

GLIF unifies LIF variants by gating dual bio-features across leakage, integration, and spike initiation. Its learnable membrane parameters and channel-wise sharing increase neuronal diversity, adaptivity, and heterogeneity.

  • Gated LIF Model: GLIF fuses dual bio-features in membrane leakage, integration accumulation, and spike initiation through gating units Gα, Gβ, and Gγ.These units compute gating factors with Sigmoid functions over learnable parameters to control each feature’s proportion.
  • Membrane Potential Leakage: The gating factor α shifts leakage from linear decay toward exponential decay as it increases from 0 to 1.The primitives τlin and τexp are fused so α balances their amplitudes.
  • Integration Accumulation: The gating factor β balances uniform coding with flexible coding based on the learnable, time-dependent conductance gt.β approaching 0 favors uniform coding, whereas β approaching 1 favors flexible coding.
  • Spike Initiation: The gating factor γ combines soft and hard reset primitives, with γ approaching 0 favoring soft reset and γ approaching 1 favoring hard reset.GLIF performs hard reset on the exponential-decay-related membrane-potential component.
  • Unified Formulation: GLIF can represent simplex LIF formulations when gating factors are binary, including vanilla LIF for α, β, γ = 1, 0, 1.The model therefore unifies different simplex LIFs while balancing dual bio-features.
  • Channel-wise Parametric Method: All GLIF membrane-related parameters are trainable, including primitives and gating factors, and are shared channelwise rather than globally.This produces distinct spiking neurons across channels and increases heterogeneity after training.

4 Experiments

Experiments evaluate GLIF across static image and event-based datasets using established SNN architectures, comparisons with prior methods, and ablations of its design choices. GLIF improves reported performance across these settings, while learnable gating and channel-wise sharing contribute to its effectiveness.

  • Experimental Settings: Experiments cover CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS with ResNet-family, 7B-wideNet, and CIFARNet architectures.The study applies tdBN and reports three runs with mean ± std for CIFAR experiments.
  • CIFAR: GLIF outperforms prior methods on CIFAR datasets, including STBP-tdBN, TET, and Dspike across reported comparisons.On CIFAR-10 and CIFAR-100, GLIF improves over the cited baselines; it exceeds Dspike by at least 0.63% and 2.92% absolute accuracy, respectively.
  • ImageNet: 69.09% v.s. 63.72% accuracy is reported for GLIF versus STBP-tdBN on ImageNet ResNet-34.GLIF also exceeds PLIF on ResNet-34 and improves MS-ResNet-18 when replacing its neurons.
  • CIFAR10-DVS: GLIF outperforms SEW-PLIF, PLIF, STBP-tdBN, and LIAF on the event-based CIFAR10-DVS dataset.The comparison uses the same 7B-wideNet architecture as SEW+PLIF.
  • Ablation Study: Ablations show that learnable gating factors and channel-wise parameter sharing outperform static-gating variants and layer-wise sharing.GLIF_f performs worse than GLIF_s despite its larger representation space, indicating that balancing dual primitives matters.

5 Conclusion

GLIF fuses bio-features within each neuronal behavior and uses learnable membrane-related parameters to increase spiking-neuron adaptivity and dynamic diversity. Experiments across benchmark datasets report state-of-the-art performance, while learned-parameter distributions motivate further study of biological plausibility and ANN2SNN application.

  • GLIF fuses bio-features in each neuronal behavior, enlarging the representation space of spiking neurons.
  • Channel-wise learnable membrane-related parameters increase neuronal adaptivity and dynamic diversity through training with BPTT.
  • GLIF SNNs achieve state-of-the-art performance compared with other methods across various benchmark datasets.
  • Learned parameters form layer-wise distributions that the authors identify as interesting for further study of SNN biological plausibility.
  • Applying GLIF in other training frameworks, including ANN2SNN conversion, remains a problem for future study.

Checklist

The checklist records that the paper describes its contributions, limitations, experimental settings, compute resources, and reproducibility materials. It also reports that error bars are provided for CIFAR experiments but not for ImageNet and CIFAR10-DVS because of time constraints.

  • The authors report that the paper describes its contributions and limitations.
  • The authors report that code, data, instructions, training details, and experimental settings are available or described.
  • The authors report mean ± std error bars for CIFAR experiments.
  • ImageNet and CIFAR10-DVS experiments were not repeated for error-bar reporting because of time considerations.
  • The experiments use CIFAR, ImageNet, and CIFAR10-DVS, described as public datasets, without a stated asset license.

A.1 Visualization of Learned Parameters

The appendix visualizes learned gating factors and primitives in GLIF-based ResNet-18 on CIFAR-100. Training preserves near-uniform gating while producing layer-wise, heterogeneous primitive distributions that the authors associate with biological properties.

  • The visualization examines learned parameters from ResNet-18 trained on CIFAR-100.
  • Figure 3 shows learned gating factors α, β, and γ distributed across channels in each layer.
  • Figure 4 shows learned primitives Vth, reVth, τexp, and τlin forming layer-wise bell-shaped distributions with shared displacement directions.
  • After training, gating-factor distributions remain uniform but narrow around their initial ranges, suggesting similar weighting of dual bio-features across the SNN.
  • The authors report increased heterogeneity in GLIF-based ResNet-18 and identify learned parameters as embodying some biological properties of brains.

A.2 Implementation Details

The implementation uses SGD with specified momentum, weight decay, learning-rate scheduling, and GLIF initialization settings. Architectures and preprocessing are adapted for CIFAR and CIFAR10-DVS while otherwise following prior work.

  • SGD uses momentum 0.9 and weight decay 5e−5, with no weight decay on GLIF parameters.
  • GLIF and model learning rates use cosine annealing, while CIFAR gating-factor learning rates are one-tenth of other parameter rates.
  • ImageNet sets gt to 0.5 at four time steps and 0.9 at six time steps.
  • For CIFAR, ResNet-18 removes max pooling, replaces the first convolution with 3 × 3, and changes two stride-2 convolutions to stride 1.
  • The implemented 7B-wideNet uses a rescheduled down-sampling strategy for 48 × 48 inputs and has fewer operations because of the smaller input size.
  • Except for the specified networks, the experiments use the same architectures as prior works.
  • CIFAR10-DVS preprocessing splits each event stream into 16 slices, integrates each slice into a frame, and resizes frames to 48 × 48 pixels.

A.3 Coarsely Fused LIF

GLIF_f removes gating factors and directly stacks primitives, enlarging representation space but weakening performance relative to GLIF_s. The ablation shows that balancing the dual primitives is essential for a valid representation space.

  • GLIF_f directly stacks primitives without gating factors as a coarsely fused LIF comparison model.
  • GLIF_f has a larger representation space than GLIF because its gating mechanism is removed.
  • GLIF_f performs worse than GLIF_s, indicating that balancing the dual primitives is essential for regulating representation space within a valid range.
Loading 2210.13768v4…