Source-linked AI summary
Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
Bojian Yin, Federico Corradi, Sander M. Bohté
TL;DR
Spiking neural networks have yet to match classical networks consistently, while energy consumption limits always-on applications. This paper trains adaptive spiking recurrent networks for sequential and streaming tasks, reporting strong benchmark performance and theoretical energy savings over classical RNNs on harder tasks.
Problem
Spiking neural networks have not consistently achieved competitive performance, and energy consumption limits deep-network use in always-on applications.
Source: §1 · p. 1Method
The paper trains compact SRNNs with surrogate-gradient BPTT and auto-differentiation, learning spiking-neuron parameters to adapt to task dynamics.
Source: §1 · p. 2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
The emergence of brain-inspired neuromorphic computing as a paradigm for edge AI is motivating the search for high-performance and efficient spiking neural networks to run on this hardware. However, compared to classical neural networks in deep learning, current spiking neural networks lack competitive performance in compelling areas. Here, for sequential and streaming tasks, we demonstrate how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance compared to other spiking neural networks and almost reach or exceed the performance of classical recurrent neural networks (RNNs) while exhibiting sparse activity. From this, we calculate a $>$100x energy improvement for our SRNNs over classical RNNs on the harder tasks. To achieve this, we model standard and adaptive multiple-timescale spiking neurons as self-recurrent neural units, and leverage surrogate gradients and auto-differentiation in the PyTorch Deep Learning framework to efficiently implement backpropagation-through-time, including learning of the important spiking neuron parameters to adapt our spiking neurons to the tasks.
1 Introduction
The introduction frames SNN training and competitive performance on sequential tasks as open challenges for energy-efficient computing. It presents trainable adaptive SRNNs and reports strong benchmark performance with sparse activity and lower theoretical energy use.
1 Introduction
- SNNs communicate asynchronously through sparse binary spikes, but their non-differentiable activations, statefulness, and mismatch with frame-based benchmarks complicate training and evaluation.Conventional efficiency approaches still use synchronous, frame-based activation exchange.Source: §1 · p. 1
- Surrogate gradients, BPTT, and PyTorch auto-differentiation let the authors train compact SRNNs and their spiking-neuron parameters to adapt to task dynamics.Source: §1 · p. 2
- SRNNs exceed SNN state of the art and approach or exceed classical ANN performance, with >100x theoretical energy improvement over high-performing ANNs on the harder tasks.The reported tasks are sequential and streaming classification benchmarks with limited input dimensionality.Source: §1 · p. 2
2 Related work
Related work contrasts temporal models and event-based SNNs with conventional frame-based networks. It describes progress in SNN learning while noting unresolved performance and efficiency questions on temporal tasks.
2 Related work
- RNNs model temporal inputs with hidden states and are commonly trained using BPTT, while LSTMs, IndRNNs, and causal convolutions offer alternative sequence-modeling approaches.BPTT is computationally expensive and can have gradient-stability problems; causal convolutions require substantial network size and data-history memory.Source: §2 · p. 2
- SNNs use stateful, event-driven spikes, and despite progress in learning and image classification, their efficiency relative to conventional CNNs remains unclear.Source: §2 · p. 2
- Temporal tasks may suit compact SNNs on low-power neuromorphic hardware, but prior LIDAR and speech work still had a performance gap from deep-learning solutions.Source: §2 · p. 2
3 Spiking Recurrent Neural Networks
The section models spiking neurons as recurrent units and trains SRNNs with BPTT and surrogate gradients. It describes adaptive timescales, output-decoding options, and task-dependent losses.
3 Spiking Recurrent Neural Networks
- LIF neurons integrate input current with leakage and emit a spike when membrane potential crosses a fixed threshold, while the paper uses LIF and adaptive neurons in recurrent layers.The paper also describes converting an SRNN to an ANN by communicating membrane potential at each timestep instead of occasional spikes.Source: §3 · pp. 2–4
- Adaptive neurons raise their threshold after each spike and let it decay; modeling threshold and membrane decay as self-recurrence makes their time constants trainable memory parameters.The threshold combines a fixed minimum b0 with an adaptive contribution βηt, whose decay uses τadp; the membrane time constant is τm.Source: §3 · pp. 3–4, Figure 2 · p. 3
- BPTT unrolls the SRNN across input timesteps, and surrogate gradients approximate the missing gradient through the discontinuous spike generator.The authors use a Gaussian surrogate derivative and implement training in PyTorch, including optimization of spiking-neuron parameters.Source: §3 · pp. 4–5
- Spike-count decoding can misclassify tied counts, be affected by neuron resets, and cannot provide real-time readout from single output neurons.For streaming tasks, the paper says single-neuron outputs require direct measurements at each timestep.Source: §3 · pp. 4–5
- Membrane-potential decoding uses the last timestep, the maximum over time, or a non-spiking readout integrator that avoids reset effects on classification.The readout integrator computes a non-spiking membrane potential from input spikes and uses its average over time.Source: §3 · pp. 4–5
- The loss depends on whether classification is sequence-level or streaming, with streaming readouts compared against targets at each timestep and sequence outputs compared after the full input.The paper uses cross-entropy as the error function.Source: §3 · pp. 4–5
4 Experiments
Experiments evaluate SRNNs on QTDB ECG waveform streaming and S-MNIST, PS-MNIST, and SHD sequential classification, using task-specific decoding. Adaptive SRNNs perform strongly across these tasks, with sparse activity and performance benefits from trainable adaptive-neuron time constants.
Encoding and decoding
- Task-specific decoding worked best: membrane potentials for ECG, spike counts for both MNIST tasks, and an average readout integrator for SHD.ECG level-crossing encoding used a 0.3 threshold and compressed the signal by about 75%.Source: §4 · p. 5
ECG
- 84.4% accuracy came from the smallest ECG network, a 46-neuron adaptive SRNN; the same-sized LIF SRNN achieved 73.7%.The RELU SRNN reached 86.4%.Source: §4 · p. 5, Table 1 · p. 6
S-MNIST and PS-MNIST
- 97.82% on S-MNIST and 91.0% on PS-MNIST were achieved by adaptive SRNNs, while LIF SRNNs failed to learn both tasks.The S-MNIST model used two recurrent layers and the same encoding scheme as the cited SNN comparison; PS-MNIST had no known SNN benchmark data in the paper.Source: §4 · p. 6, Table 1 · p. 6
SHD
- 84.4% accuracy on SHD approached the 128-unit LSTM result; a non-spiking RELU SRNN reached 88.93%.The adaptive SRNN used two layers of 128 neurons and 4 ms bins; a single-layer 256-neuron adaptive SRNN achieved 81.71%.Source: Table 1 · p. 6, §4 · p. 7
Sparsity
- Average firing sparsity was 0.07–0.1 on MNIST and 0.02–0.13 on SHD, while ECG neurons fired about once every three timesteps (FR=0.32).The paper associates ECG's relatively lower sparsity with the need to read out labels at every timestep.Source: §4 · p. 7
Complexity of spiking neurons
- Adaptive spiking neurons with trainable time constants substantially outperformed LIF neurons, and successively ablating time-constant training reduced SHD performance.Training also substantially modified the adaptive time constants on SHD.Source: §4 · p. 7, Figure 3 · p. 7, Figure 4 · p. 8
5 Efficiency
The paper estimates recurrent-network energy by counting MAC and AC operations and accounting for sparse firing. Under this theoretical accounting, adaptive SRNNs approach RELU SRNN performance with lower energy, and are over 100x more efficient than classical RNNs on harder tasks.
5 Efficiency
- Table 2 covers per-layer energy for different neuron types, including a non-spiking adaptive neuron.Its caption defines m and n as layer input and output sizes and distinguishes AC and MAC energy costs.Source: Table 2 · p. 8
- Energy estimates count MACs for standard neurons and ACs for input spikes plus internal MACs for spiking neurons.The estimates use RTL-level power numbers for 45nm CMOS from [10].Source: §5 · pp. 8–9
- Network energy is aggregated across layers and timesteps, with SRNN estimates scaled by average firing rate Fr.The per-layer accounting uses input size m, output size n, and the operation costs E_MAC and E_AC.Source: §5 · p. 9
- 28–243x more energy efficiency is reported for adaptive SRNNs versus RELU SRNNs; on complex SHD and (P)S-MNIST tasks, SRNNs are over 100x more efficient than classical RNNs.The comparison is theoretical; the paper also places SRNN solutions on the accuracy–energy Pareto front.Source: §5 · p. 9, Table 3 · p. 9
6 Conclusions
The paper concludes that adaptive SRNNs achieve competitive task performance with substantially lower theoretical energy use than analog recurrent networks. It attributes their results to adaptive spiking neurons with trainable timescales, while conversion to a non-spiking RELU RNN consistently increased performance.
6 Conclusions
- Up to 243x greater theoretical energy efficiency than a slightly better-performing analog RELU SRNN, and up to 1900x versus similarly performing classical RNNs such as LSTMs, is reported.The paper also reports state-of-the-art performance among SNNs and approaching or exceeding state-of-the-art for classical RNNs on the evaluated tasks.Source: §6 · p. 9
- Adaptive spiking neurons with trainable time-constants are identified as key to the results, providing multiple-timescale memory that may adapt to task dynamics.The proposed explanation that this memory adaptation makes the approach effective is stated as a hypothesis.Source: §6 · p. 9
- Converting the SRNN to a non-spiking RELU RNN consistently increased performance, which the authors take to suggest the nested hierarchical recurrent architecture is effective.Source: §6 · p. 10
- Surrogate gradients made it feasible to train complex SRNNs and their neuron parameters through PyTorch automatic differentiation.The authors see this approach as opening opportunities to improve and scale SNNs further.Source: §6 · p. 10