Source-linked AI summary
Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks
Bojian Yin, Federico Corradi, Sander M. Bohte
TL;DR
SNN performance has lagged behind classical ANNs despite potential biological and energy-efficiency advantages. The paper combines recurrent spiking networks, adaptive neurons, and a Multi-Gaussian surrogate gradient, achieving strong temporal-task accuracy and a theoretical one-to-three-order computational-efficiency advantage over comparable RNNs.
Problem
SNNs have struggled to demonstrate a clear advantage over classical ANNs, despite their potential biological plausibility, temporal coding, and hardware energy benefits.
Method
The paper trains recurrent spiking networks with BPTT, a Multi-Gaussian surrogate gradient, and adaptive spiking neurons whose time constants are co-trained with network weights.
Results
SRNNs achieve state-of-the-art SNN performance across sequential and temporal tasks, approach or equal conventional RNN accuracy, and show a theoretical one-to-three-order energy advantage.
Takeaways & Limitations
Sparse spiking and recurrent adaptive dynamics position SRNNs as a computationally efficient approach for temporal AI tasks and hardware implementations.
Takeaways & Limitations
The reported energy advantage is theoretical because the analysis omits memory access, local-memory, spike-routing, sensor, pruning, and quantization costs.
Abstract
from arXiv · showhide
Inspired by more detailed modeling of biological neurons, Spiking neural networks (SNNs) have been investigated both as more biologically plausible and potentially more powerful models of neural computation, and also with the aim of extracting biological neurons' energy efficiency; the performance of such networks however has remained lacking compared to classical artificial neural networks (ANNs). Here, we demonstrate how a novel surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields state-of-the-art for SNNs on challenging benchmarks in the time-domain, like speech and gesture recognition. This also exceeds the performance of standard classical recurrent neural networks (RNNs) and approaches that of the best modern ANNs. As these SNNs exhibit sparse spiking, we show that they theoretically are one to three orders of magnitude more computationally efficient compared to RNNs with comparable performance. Together, this positions SNNs as an attractive solution for AI hardware implementations.
Introduction
The paper addresses the limited comparative advantage of SNNs by combining recurrent spiking networks, adaptive neurons, and a novel surrogate gradient. It evaluates these models on challenging temporal benchmarks while emphasizing sparse activity and computational efficiency.
- Motivation: SNNs encode information through sparse, binary spike timing, potentially improving biological plausibility, computational power, and hardware energy efficiency.Spiking neurons integrate weighted input spikes over time, emit a spike at threshold crossing, and reset their membrane potential.
- Motivation: Surrogate gradients address the discontinuous spiking mechanism that complicates supervised weight-gradient calculation.Recent surrogate-gradient approaches have narrowed the performance gap with conventional neural networks on selected benchmarks.
- Contribution: SRNNs combine recurrently connected spiking layers with BPTT, a Multi-Gaussian surrogate gradient, and co-trained adaptive neuron time constants.The Multi-Gaussian gradient includes negative slopes, and adaptive neurons learn internal temporal dynamics with network weights.
- Evaluation: SRNNs are evaluated on ECG, speech, radar gesture, sequential MNIST, and permuted sequential MNIST benchmarks with inherent temporal structure.The benchmark set includes Google Speech Commands, TIMIT, and SoLi gesture recognition.
Spiking Recurrent Neural Networks
The proposed SRNNs use recurrent spiking layers with LIF or adaptive LIF neurons trained through time using surrogate gradients. Their operation replaces dense ANN communication with sparse spike-triggered accumulation while retaining trainable temporal dynamics.
- Architecture: SRNNs use multi-layer recurrently connected spiking neurons, with recurrent layers projecting to output neurons and optional feedforward connections between recurrent layers.The architecture is designed for temporal processing and can also receive bidirectional input in bi-SRNN variants.
- Neuron models: LIF neurons integrate current with leakage and emit a spike when membrane potential crosses threshold, whereas ALIF neurons add a decaying adaptive threshold.The LIF decay is governed by τm, while ALIF adaptation decays with τadp.
- Training: BPTT updates synaptic weights and neural time constants, while surrogate gradients replace the discontinuous spike derivative with a smooth function.This enables recurrent spiking networks to be trained in optimized deep-learning frameworks.
- Training: The Multi-Gaussian surrogate gradient is a weighted sum of Gaussian functions designed to include negative slopes inspired by ELU and LeakyReLU.Its parameters h and s control the constructed gradient shape.
- Computational cost: SRNN computational cost is compared with LSTM-style recurrent ANNs using accumulation and multiply-and-accumulate operations.Spikes trigger accumulation at target neurons, whereas dense ANN communication requires multiply-and-accumulate operations at every timestep.
Results
SRNNs were evaluated across temporal benchmarks and architectural variants, achieving strong accuracy with sparse activity, online decision-making, and theoretically lower energy costs than comparable RNNs.
- Benchmarks: The study evaluates SRNNs on ECG, speech, radar gesture, sequential MNIST, and permuted sequential MNIST benchmarks with inherent temporal structure.The experiments compare architectures, surrogate gradients, hyperparameters, neuron models, and classical or state-of-the-art RNNs.
- Accuracy: SRNNs achieve state-of-the-art performance among SNNs, exceed conventional LSTM models, and approach or exceed modern RNNs.They also substantially close the accuracy gap with CNNs and attention-based networks on SHD, SSC, and GSC.
- Training choices: The Multi-Gaussian surrogate gradient consistently outperforms Gaussian, Linear, and SLayer gradients, while training adaptive-neuron time constants improves performance.Not training τm or τadp, or neither, reduces performance.
- Activity: Most trained networks have sparsity below 0.1, while larger networks improve accuracy and reduce average sparsity but increase total spike counts.The ECG task uses more spikes because it was tuned to the smallest SRNN network, containing 46 neurons.
- Architecture: Adaptive LIF neurons improve both performance and activity sparsity, and splitting one recurrent layer into two improves performance and sparsity on SHD.Similar improvements from multiple recurrent layers were observed on other tasks.
- Efficiency: 59x greater theoretical energy efficiency is reported for SRNNs than RNNs on more complex tasks at equivalent performance levels.The comparison uses inference MAC and AC operation costs; LSTMs are often estimated to be 1000x less efficient.
Discussion
SRNNs achieve state-of-the-art SNN performance on temporal tasks while approaching or equaling conventional RNN accuracy and offering a theoretical energy advantage. The discussion also emphasizes scope, implementation, and hardware caveats, alongside possible neuroscience implications.
- Energy efficiency: One to three orders of magnitude lower theoretical computational energy than conventional RNNs is reported for SRNNs, with the advantage increasing on more complex tasks.The estimate is based on computational operations and favors larger networks solving harder tasks.
- Efficiency comparison: At equivalent Google Speech Commands performance, the SRNN requires 19.6x fewer MACs than TinySpeech while operating online and streamingly.Neither model was optimized beyond accuracy, or sparsity for SRNNs, before this comparison.
- Scope: The study focuses on temporal or sequential problems with relatively limited input dimensionality, emphasizing streaming tasks that avoid segmentation and buffering.Most classification decisions could be made early with near-optimal accuracy.
- Implementation limits: BPTT through fully unrolled computational graphs is costly in memory and training time because existing frameworks do not exploit spatial or temporal sparsity.The authors note that larger networks may require dedicated hardware or sparsity-optimized frameworks, while eProp and RTRL could alleviate the limitation.
- Hardware caveats: The reported energy advantage is theoretical because MAC-based estimates omit memory availability, memory access, spike routing, and sensor energy costs.The networks were also unoptimized by pruning and standard quantization techniques, leaving room for further improvements.
- Neuroscience implications: Training task-specific time constants may have neuroscience implications because effective time constants in biological spiking neurons are variable and dynamic.The authors present this as a suggestion that such properties may be subject to biological learning processes.
Methods
The SRNNs use recurrent spiking neurons trained with BPTT, differentiable surrogate gradients, adaptive dynamics, and task-specific spike encoding and decoding. Training and losses are accumulated across sequence timesteps, with output interpretation adapted to spiking and non-spiking layers.
- Neuron models: SRNNs model LIF and ALIF neurons, with ALIF adding an adaptive threshold whose decay is controlled by τadp.The membrane potential decays with τm, while adaptation contributes βηt to the dynamical threshold.
- Input and task setup: Inputs are encoded as spikes by level-crossing for ECG, while other listed tasks use injected current or dataset-provided spike representations.Level-crossing ECG encoding creates separate positive and negative event streams using L+ = 0.3 and L− = 0.3.
- Output decoding: Classification is decoded either from summed output spikes or from output membrane potentials passed through softmax.For ECG, spiking ALIF output neurons performed best, consistent with classification switching within a sample.
- Optimization: BPTT unrolls input sequences, accumulates gradients across timesteps, and updates recurrent weights and neural time-constants with Adam.The trainable parameters include synaptic weights and neural time-constants.
- Losses: Sequence losses use cross-entropy or negative log-likelihood, with whole-sequence losses for standard classification and timestep-summed losses for ECG and SoLi.The target probability for class c is compared with the predicted probability after sequence processing or at each streaming timestep.
- Optimization: Surrogate gradients replace the discontinuous spike derivative, with Multi-Gaussian gradients evaluated alongside Linear, Gaussian, and SLayer alternatives.The reported Multi-Gaussian settings are h = 0.15, s = 6, and σ = 0.5; the comparison gradients use grid-searched hyperparameters.
Supplementary Information
The supplementary information documents initialization schemes, task-level optimization parameters, a bidirectional SRNN architecture, and comparative performance and energy-consumption reporting. These materials define how time constants and computational efficiency are compared across models.
- Initialization: Figure S1 compares constant, uniformly distributed, and normally distributed time-constant initialization schemes on SoLi training accuracy, loss, and mean firing rate.MGuni samples time constants near MGconstant, whereas MGstd5 uses a normal distribution with standard deviation 5.0 around the same value.
- Experimental parameters: Table S1 specifies task-specific initialization and optimization parameters, including layerwise time-constant distributions, losses, learning rates, decay schedules, and minibatch sizes.Gaussian initialization is described by per-layer mean and standard deviation; LIF non-spiking output layers use only τm.
- Architecture and cost: Figure S2 presents a bidirectional SRNN architecture and compares computational costs across layers in regular RNNs and GRU units.The computational-complexity calculation follows the cited reference.
- Performance and efficiency: Table S2 reports model performance and relative energy consumption using Adaptive SRNN as the unit reference for energy per timestep and error rate.Efficiency is defined as the product of the relative energy and error ratios.