PaperlayerView exact revision on arXiv

Source-linked AI summary

The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks

Ruichen Ma, Yicai Chen, Pujun Zhou, Yue Zuo, Chongyang Li, Shuang Liu, Shaogang Hu, Guanchao Qiao

arXiv:2610.21748v1cs.CV

TL;DR

SNNs still update deeply inhibited neurons even when their near-term firing probability is negligible. The DSN adds state-driven neuronal dormancy and trains it end-to-end with surrogate-gradient BPTT. Across static and neuromorphic datasets, it reduces energy, and on ImageNet it improves baseline SNN accuracy while outperforming static pruning at comparable reported costs.

  • Problem

    In an ImageNet-trained ResNet-18 SNN, 44% of neurons were deeply inhibited yet continued subthreshold updates despite negligible near-term firing probability.

    Source: §1 · p. 2

  • Method

    The DSN uses a negative-threshold state-driven gate for instance-level dormancy and trains the mechanism end-to-end with BPTT using surrogate gradients.

    Source: §1 · p. 2

  • Results

    Across static and neuromorphic datasets, DSN reduces computational energy by 40% to over 70%; on ImageNet, it improves baseline SNN accuracy by 1.34% and outperforms static pruning at comparable reported costs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    State-driven neuronal dormancy complements presynaptic input-driven sparsity, enabling multiplicative reductions in synaptic operations.

    Source: §1 · p. 2

    The dormancy mechanism is particularly compatible with rate coding and multi-spike architectures, but may be less suitable for low-redundancy TTFS coding.

    Source: §5 · p. 9

Abstract

from arXiv · show

Spiking neural networks (SNNs) offer a compelling pathway toward energy-efficient artificial intelligence, fundamentally anchored in presynaptic input-driven sparsity. However, a substantial yet chronically overlooked source of computational overhead persists: the continuous subthreshold dynamics of postsynaptic neurons. Quantitative analysis reveals that up to 44% of neurons in an ImageNet-trained ResNet-18 model reside in a deeply inhibited state, imposing a persistent computational burden despite having a negligible probability of firing. To address this inefficiency, the Dormant Spiking Neuron (DSN) model is introduced. By incorporating a state-driven dynamic gate, the DSN allows a neuron to enter a dormant state when its membrane potential falls below a predefined negative threshold, dynamically pruning redundant computations at the instance level. Crucially, this dormancy mechanism is directly integrated into the neuronal dynamics and trained end-to-end via backpropagation through time using surrogate gradients, enabling the network to actively learn a robust dormancy policy that adaptively preserves critical information flow. Extensive experiments across static and neuromorphic datasets demonstrate that the DSN framework consistently reduces computational energy by 40% to over 70%. Notably, on the large-scale ImageNet dataset, the DSN outperforms state-of-the-art static pruning techniques at comparable reported cost reductions and simultaneously improves the corresponding baseline SNN accuracy by 1.34%. By establishing a trainable, state-driven sparsity paradigm orthogonal to conventional input-driven methods, the DSN unlocks synergistic efficiency gains, presenting an effective approach for advancing neuromorphic computing.

1 Introduction

SNNs save computation through input-driven sparsity, but deeply inhibited postsynaptic neurons continue costly subthreshold updates. The DSN adds trainable, state-driven dormancy and reports energy reductions across datasets alongside improved ImageNet accuracy.

1 Introduction

  • 44% of neurons in an ImageNet-trained ResNet-18 SNN were deeply inhibited, with negligible near-term firing probability despite continued state updates.The analysis also found 26% in the active spiking region and 30% in the subthreshold active region.Source: §1 · p. 2
  • The DSN uses a negative-threshold state-driven gate to make neurons dormant per instance, complementing presynaptic input-driven sparsity.Unlike static pruning of chronic dataset-level redundancy, this targets transient instance-level redundancy.Source: §1 · p. 2
  • End-to-end BPTT with surrogate gradients trains the dormancy mechanism so the network can learn a robust policy that preserves critical information flow.The training approach addresses potential performance degradation from static or post-hoc thresholding.Source: §1 · p. 2
  • 40% to over 70% energy reductions across static and neuromorphic datasets accompany a 1.34% improvement in baseline SNN accuracy on ImageNet.The DSN also outperforms state-of-the-art static pruning on ImageNet at comparable reported cost reductions.Source: §1 · p. 2

2 Related Work

Conventional pruning targets chronic redundancy using dataset-level statistics, which may not adapt when neuron relevance varies across dynamic inputs. The DSN instead bypasses computations dynamically according to postsynaptic state, complementing input-driven sparsity.

2 Related Work

  • Conventional SNN pruning often uses firing-rate thresholds to remove components, targeting chronic dataset-level redundancy and adapting poorly when neuron relevance varies across dynamic inputs.Such statistical decisions also leave real-time subthreshold updates of temporarily inhibited neurons unaddressed.Source: §2 · pp. 2–3
  • Figure 1 sequences a dormancy gate check, potential integration, gated spike firing, and membrane-potential update with reset.A neuron crossing the dormancy threshold bypasses subsequent stages for current and future timesteps.Source: Figure 1 · p. 3, §2 · p. 3
  • The DSN bypasses computations when a neuron's membrane potential falls below a negative threshold, targeting transient instance-level redundancy rather than permanently pruning structures.This state-driven mechanism operates orthogonally to presynaptic input-driven sparsity, allowing multiplicative cost reductions.Source: §2 · p. 3

3 Method

The method makes dormancy an irreversible, state-triggered gate, trains it with surrogate-gradient BPTT, and combines postsynaptic state-driven sparsity with presynaptic input-driven sparsity. It describes both post-training inference-time optimization and direct training, with energy reductions reported across benchmarks.

3.1 Dormant Spiking Neuron Dynamics

  • A dormancy gate irreversibly closes when the preceding membrane potential falls below a threshold, making the neuron permanently inactive and bypassing later dynamics.For active neurons, the dynamics include potential integration, gated spike firing, and potential update with reset; the standard LIF decay function is also specified.Source: §3.1 · pp. 3–4

3.2 Direct Training via Surrogate Gradients

  • Surrogate gradients replace the Heaviside derivative only during backpropagation, enabling BPTT to compute loss gradients for the network weights across timesteps.The forward pass retains the exact Heaviside function.Source: §3.2 · p. 4

3.3 Analysis of Computational Efficiency

  • Input-driven sparsity skips inactive input rows, while state-driven sparsity skips inactive output columns; their orthogonal combination compounds savings multiplicatively.The computational-load analysis quantifies cost using synaptic operations.Source: §3.3 · pp. 4–5

3.4 Inference-Time Optimization vs. Direct Training

  • Inference-time optimization applies a fixed dormancy rule after training, which may silence neurons needed for late features in nonstationary streams; BPTT adapts representations to dormancy.Both paradigms use the same dormancy mechanism at inference, implemented with a 1-bit mask per neuron to bypass later weight-memory accesses and membrane-potential updates.Source: §3.4 · p. 5
  • 40% to over 70% estimated energy reductions across static and neuromorphic benchmarks accompany accuracy comparable to or better than baseline SNNs.On ImageNet, DSN outperforms state-of-the-art static pruning at comparable reported cost reductions and improves accuracy.Source: Table 1 · p. 6

4 Experiments

Experiments span static and neuromorphic benchmarks with two spike-driven SNN architectures; reported energy values are analytical estimates, and device-level energy and latency depend on target hardware. The evaluations compare DSN performance and overhead, then examine threshold choice, training, timestep effects, representations, and neuronal utilization.

4.1 Experimental Setup

  • The evaluation covers CIFAR10, CIFAR100, ImageNet, DVS-CIFAR10, and DVS-Gesture using MS-ResNet and Spike-Driven Transformer; reported energy is analytically estimated.Actual device-level energy and latency depend on the target hardware.Source: §4.1 · p. 6

4.2 Performance Evaluation

  • 73%: DSN (BPTT) on ResNet-18 achieves 64.44% ImageNet accuracy at 1.17 mJ, versus 4.29 mJ for the baseline, with a 1.34% accuracy improvement.At comparable reported cost reductions, DSN also outperforms the listed ResNet-18 static pruning methods on ImageNet.Source: §4.2 · p. 7
  • On DVS-Gesture, DSN matches baseline accuracy at 96.88% with 42% less estimated energy; on DVS-CIFAR10 it saves over 52% with a 0.2% accuracy drop.At T = 24 on DVS-CIFAR10, accuracy deviates from baseline by 0.4 percentage points.Source: §4.2 · p. 7
  • DSN-BPTT training on ImageNet reduces throughput by 19% and increases GPU memory use by 34% versus the baseline SNN.The measurement used ResNet-18, one GPU, and batch size 128; the paper describes the inference-energy reductions as a favorable trade-off for deployment.Source: §4.2 · p. 7

4.3 Ablation Studies and Analysis

  • Across three CIFAR100 seeds, accuracy stays relatively stable as Dth varies, while energy rises sharply beyond the aggressive-pruning region near Dth ≈ −0.1.The study selects Dth = −0.05 for static datasets and −0.5 for neuromorphic datasets to prioritize stability on dynamic event streams.Source: §4.3 · p. 7
  • Energy savings increase with simulation length on CIFAR100, from approximately 30% at T = 2 to over 63% at T = 16 and above 73% at T = 96.At T = 4, savings are 45%.Source: §4.3 · p. 8
  • Layer-wise ImageNet analysis reports NCU above 70% in most layers, with savings linked to increased dormancy rather than indiscriminate firing suppression; ImageNet dormancy masks average 0.40 overlap across input pairs.Static pruning has mask overlap fixed at 1 across inputs, supporting the distinction between instance-level DSN redundancy and dataset-level pruning.Source: §4.3 · p. 9
  • Smooth surrogate gradients outperform a straight-through estimator for the dormancy gate, including a 1.57% ImageNet accuracy advantage; three-seed results show small standard deviations across evaluated datasets.The SG approach also improves accuracy by 0.30% on CIFAR10 and 0.63% on CIFAR100.Source: §4.3 · pp. 7, 9

5 Limitations and Discussion

DSN's usefulness depends on the coding scheme: its dormancy mechanism is especially compatible with rate coding and multi-spike architectures, while TTFS offers less redundancy to exploit. The discussion also points to further optimization of active subthreshold neurons.

5 Limitations and Discussion

  • DSN may be less suitable for TTFS coding, where precise single-spike timing leaves little redundancy for dormancy without risking performance degradation.The mechanism is described as particularly compatible with rate coding and multi-spike architectures.Source: §5 · p. 9
  • In highly sparse TTFS networks, both the available margin and practical need for further computational reduction may be limited.The passage notes that TTFS networks generally have high sparsity and low baseline computational overhead.Source: §5 · p. 9
  • Fine-grained optimization of remaining active subthreshold neurons could improve computational utilization and help explore energy-accuracy trade-offs.This is presented as an avenue for future research beyond bypassing computations for deeply inhibited neurons.Source: §5 · p. 9

6 Conclusion

The conclusion presents DSN as a trainable, state-driven approach to reducing redundant computation from deeply inhibited neurons. Across static and neuromorphic evaluations, it reports energy reductions while maintaining competitive accuracy, with an ImageNet comparison against static pruning methods.

6 Conclusion

  • 40% to over 70% estimated energy reduction was reported across static and neuromorphic datasets and network architectures, with competitive task accuracy relative to baselines.On ImageNet, DSN outperformed the evaluated static pruning methods at comparable reported cost reductions.Source: §6 · p. 10
  • DSN puts neurons into a permanent dormant state based on membrane potential, bypassing redundant computations at the instance level.The mechanism is trained end-to-end via BPTT using surrogate gradients to improve robustness on temporal streams.Source: §6 · p. 10
  • The conclusion frames DSN as state-driven sparsity orthogonal to presynaptic input-driven sparsity for improving neuromorphic computing efficiency.It targets continuous subthreshold updates of deeply inhibited neurons.Source: §6 · p. 10

A Detailed BPTT Derivation for the DSN

The DSN’s forward dynamics update a dormancy gate, integrate inputs, gate spike firing, and reset membrane potential. Its BPTT derivation uses surrogate gradients for the non-differentiable step functions.

A.1 Forward Dynamics and Gradient Notations

  • The DSN updates dormancy from the preceding membrane potential and threshold, integrates weighted incoming spikes with decayed potential, gates firing by threshold and active state, then updates and resets potential.The passage describes these four stages as the dormancy gate update, potential integration, gated spike firing, and potential update and reset.Source: §A.1 · p. 13
  • The BPTT setup defines loss gradients with respect to spike output, updated membrane potential, dormancy state, and integrated potential.The listed gradients are δt S,n, δt V,n, δt D,n, and δt H,n, respectively.Source: §A.1 · p. 13
  • For a standard LIF neuron, the membrane-potential decay function can be f(V) = τ^-1 · V.The passage gives this as an example of f(V).Source: §A.1 · p. 13
  • During the backward pass, continuous surrogate gradients replace exact derivatives of the non-differentiable Heaviside step functions used for dormancy and spike generation.The passage identifies d(x) for the dormancy gate and s(x) for spike generation.Source: §A.1 · p. 13

A.2 Backward Pass Gradient Derivation

The backward pass accumulates weight gradients across timesteps by propagating error signals through network depth and time. Its recurrences account for firing, reset, temporal-state, and dormancy paths, then form an implementable end-to-end training workflow.

A.2 Backward Pass Gradient Derivation

  • The weight gradient is accumulated over simulation timesteps using the multivariate chain rule, with error signals propagated backward through both network depth and time.The derivation unrolls recurrence relations for the defined error terms.Source: §A.2 · p. 13

A.2.1 Step 1: Deriving δt

  • The integrated-potential error receives contributions from firing and reset paths; detaching the spike in the reset equation treats it as constant and simplifies the reset-path gradient.The text describes spike detachment as a standard SNN approximation for training stability and mitigating gradient anomalies.Source: §A.2.1 · p. 14

A.2.2 Step 2: Deriving δt

  • With the reset dependency detached, the spike-output error flows exclusively backward through the spatial path from the subsequent layer.The spike output affects the next layer’s integrated potential, while the local reset path is detached.Source: §A.2.2 · p. 14

A.2.3 Step 3: Deriving δt

  • The membrane potential carries temporal state forward, influencing both the next timestep’s dormancy gate and integrated potential and thereby contributing to its temporal gradient.The passage states that the temporal relation uses partial derivatives and notes the LIF case.Source: §A.2.3 · p. 14, §A.2.4 · p. 14

A.2.4 Step 4: Deriving δt

  • The dormancy-state gradient accounts for both its continuation to the next timestep and its effect on immediate spike generation, formalizing the dormancy mechanism’s gradient.The derivation distinguishes temporal and firing paths.Source: §A.2.4 · p. 14

A.3 The Complete BPTT Algorithm

  • The complete BPTT workflow unrolls gradients backward through timesteps and layers, accumulates weight gradients, and supports end-to-end training of the DSN with dynamic state-driven sparsity gating.The aggregated weight gradient after the loops is defined as ∂L/∂Wn = ∆Wn.Source: §A.3 · p. 15

B.1 FLOPs Calculation for Standard Architectures

The section estimates standard-architecture computation by counting multiply-accumulate operations, treating one MAC as one FLOP. It gives separate formulas for convolutional and fully connected layers and sums layer counts for the network total.

B.1 FLOPs Calculation for Standard Architectures

  • One MAC is counted as one FLOP for consistency with the paper’s energy accounting.Source: §B.1 · p. 15
  • Convolutional-layer MACs equal Hout · Wout · K · K · Cin · Cout.The formula uses output-map dimensions, kernel size, and input and output channel counts.Source: §B.1 · p. 15
  • A fully connected layer’s MAC count is Nin · Nout, and the network total sums the MAC counts across its layers.Source: §B.1 · p. 15

B.2 SOPs Calculation for Spiking Architectures

The section estimates spiking-network operations using temporal spike activity, then accounts for DSN’s additional postsynaptic dormancy filtering. Its energy model combines first-layer dense MAC energy with later-layer sparse AC energy, including transformer components.

B.2 SOPs Calculation for Spiking Architectures

  • A baseline spiking layer’s SOP count scales its dense MAC count by the average temporal firing rate of the preceding layer’s input spikes.Source: §B.2 · pp. 15–16
  • DSN layer operations require both an arriving input spike and a non-dormant receiving neuron, reducing SOPs multiplicatively by presynaptic activity and postsynaptic activity factors.The presynaptic factor is the mean binary spike activity; the postsynaptic factor is the fraction of non-dormant neurons.Source: §B.2 · p. 16

B.3 Energy Consumption Model

  • SNN inference energy combines first-layer dense MAC energy with the SOP-based AC energy of subsequent spiking layers.The model assigns 4.6 pJ to a 32-bit MAC and 0.9 pJ to a 32-bit AC under a 45nm CMOS framework.Source: §B.3 · p. 16
  • The DSN energy estimate substitutes DSN-layer SOPs for baseline SNN SOPs, while the baseline calculation uses vanilla SNN SOPs.Source: §B.3 · p. 17

B.3.1 Spike-Driven Transformer Energy Analysis

  • The Spike-Driven Transformer energy example identifies where DSN applies postsynaptic activity factors across self-attention modules, with derivations based on sequence length, embedding dimension, and MLP expansion ratio.Its listed spike-rate terms include Rin, RKV, RQKV, and RMLPn.Source: §B.3.1 · p. 17

C.1 Supplementary Visualizations and Analysis

The supplementary analysis examines inhibited-neuron recovery, threshold sensitivity, and the layer- and sample-specific structure of DSN dormancy. It also specifies the neuron and surrogate-gradient settings used in the experiments.

Motivation: Membrane Potential Distribution

  • 44% of neurons in the ImageNet-trained ResNet-18 SNN were deeply inhibited (Vm ≤0), and approximately 1% of these neurons fired in subsequent timesteps.The analysis uses this low immediate-future firing probability to motivate bypassing redundant instance-level computations.Source: §C.1 · p. 17

Threshold Sensitivity Across Datasets and Architectures

  • For neuromorphic datasets, Dth = −0.5 provides a favorable performance-efficiency balance, while Dth = −0.05 remains near the low-energy plateau for static datasets.The threshold sweeps support the settings used in the main experiments.Source: §C.1 · pp. 17–18, Table 4 · p. 18

Neuron Model and Surrogate Gradient Mechanism

  • Experiments use LIF neurons with τ = 2.0, Vth = 1.0, and Vreset = 0.0, and replace the Heaviside derivative during backpropagation with a surrogate gradient.The BPTT surrogate sharpness parameter is α = 4.0.Source: §C.1 · p. 18

Deep Dive into DSN

  • More than 12 ResNet-34 layers have mean dormancy-mask Jaccard overlap below 0.2 across inputs, unlike static pruning’s fixed overlap of 1.The analysis interprets this heterogeneity as evidence of sample-specific dynamic pruning; ResNet-18 layer-wise firing, non-dormant rates, and NCU are also visualized.Source: §C.1 · p. 18, Figure 5 · p. 18

C.2 Implementation Details and Training Protocols

The implementation uses consistent augmentation and closely aligned training hyperparameters across baselines and DSN models. Training protocols vary by dataset, and baseline results are reproduced internally except for ImageNet.

C.2 Implementation Details and Training Protocols

  • Standard augmentation was applied equivalently to baselines and DSN models, with training hyperparameters aligned to the original Spike-Driven Transformer settings.Key configurations are summarized in Table 5.Source: §C.2 · p. 19
  • DSN models were trained from scratch on CIFAR10, CIFAR100, ImageNet, and DVS-CIFAR10, while DVS-Gesture used fine-tuning from a pretrained SNN checkpoint.Automatic mixed precision was used across experiments; the main experiments and single-seed ablations used seed 2025.Source: §C.2 · p. 19
  • Baselines for CIFAR10, CIFAR100, DVS-Gesture, and DVS-CIFAR10 were reproduced internally, while ImageNet baseline results came from published literature.Experiments used two NVIDIA RTX 5090 GPUs and the stated Ubuntu, Python, PyTorch, and SpikingJelly environment.Source: §C.2 · p. 19

Related papers

Loading 2610.21748v1…