Source-linked AI summary
A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardware
Philipp Plank, Arjun Rao, Andreas Wild, Wolfgang Maass
TL;DR
Whether working memory in spiking neurons can support large relational DNNs is an open question. The paper uses AHP-current neurons to replace LSTM units, reporting much less than one spike per neuron during relational-network computation.
Problem
The paper asks whether working memory in spiking neurons can also support large DNNs for relational reasoning.
Source: Design of a very sparsely active large… · p. 6, Discussion · p. 11Method
The paper replaces LSTM units with AHP-current spiking neurons and uses them in large spike-based relational networks.
Source: Discussion · p. 11Results
AHP-neurons can replace LSTM units, while relational networks achieve much less than one spike per neuron during the whole computation.
Source: Discussion · p. 11Takeaways & Limitations
The approach provides an in-memory computing solution without separate units for computing and working memory.
Source: Discussion · p. 11Task 17 positional reasoning has comparatively high error at shorter compute times, a trade-off for faster training and energy efficiency; more time steps can solve it.
Source: S4.1 Output accuracy of the Spiking… · p. 25
Abstract
from arXiv · showhide
Spike-based neuromorphic hardware holds the promise to provide more energy efficient implementations of Deep Neural Networks (DNNs) than standard hardware such as GPUs. But this requires to understand how DNNs can be emulated in an event-based sparse firing regime, since otherwise the energy-advantage gets lost. In particular, DNNs that solve sequence processing tasks typically employ Long Short-Term Memory (LSTM) units that are hard to emulate with few spikes. We show that a facet of many biological neurons, slow after-hyperpolarizing (AHP) currents after each spike, provides an efficient solution. AHP-currents can easily be implemented in neuromorphic hardware that supports multi-compartment neuron models, such as Intel's Loihi chip. Filter approximation theory explains why AHP-neurons can emulate the function of LSTM units. This yields a highly energy-efficient approach to time series classification. Furthermore it provides the basis for implementing with very sparse firing an important class of large DNNs that extract relations between words and sentences in a text in order to answer questions about the text.
Long short-term memory for spike-based neuromorphic hardware
This section examines how slow AHP currents provide memory for sparse spiking networks implementing LSTM-like functions, including time-series classification and relational reasoning. It describes training and optimizing these networks for Loihi, including placement under hardware memory, connectivity, and spike-transport constraints.
Long short-term memory for spike-based neuromorphic hardware
- AHP currents support long-term temporal classification through the pointwise separation property (PSPR), rather than by storing continuous-valued variables in LSTM memory cells.Slow AHP dynamics also provide gradient-propagation pathways through time, improving BPTT gradient transmission.Source: Long short-term memory for spike-based… · pp. 3, 5
- Four orders of magnitude lower EDP was measured for the Loihi AHP-SNN than for a conventional LSTM on CPU or GPU at batch size 1.Loihi also outperformed by over 2x on latency and over 1000x on energy per inference.Source: Figure 2 · p. 4, Long short-term memory for spike-based… · p. 6
- 89.8% accuracy for distinguishing digits 6 and 8 was achieved with AHP-neurons, versus 69.6% without them, using random recurrent weights and trained linear readouts.AHP-neurons also substantially enhanced the separation of network states for those digit pairs.Source: Figure 2 · p. 4, Long short-term memory for spike-based… · p. 5
Design of a very sparsely active large DNN for energy-efficient spike-based relational reasoning
- Less than one spike per neuron for the whole computation enabled sparse RelNet feed-forward processing through spike-rate and voltage regularization, fixed-time output, and short-time-constant neurons without a refractory period.Voltage regularization countered spike-rate regularization’s tendency to push membrane potentials too negative for network participation.Source: Design of a very sparsely active large… · pp. 6–7, Discussion · p. 11
Discussion
- The 20-sentence RelNet required 2308 neuro-cores across 22 Loihi chips because of connectivity constraints, increasing energy and latency despite optimized placement.The authors expect less constrained future hardware to yield larger energy savings for this sparse network.Source: Design of a very sparsely active large… · p. 7, Figure 5 · p. 10, Discussion · p. 11
Design of a very sparsely active large DNN for energy-efficient spike-based relational reasoning
- Between 4 and 16 times less energy was consumed by the Loihi Spiking RelNet than by the GPU ANN RelNet from Santoro et al.15.Loihi computation time was slightly larger, but its EDP remained lower; the longest stories averaged 6.54 ms per sample.Source: Table 1 · p. 9, Design of a very sparsely active large… · p. 9
- 16 out of 17 trained bAbI tasks were solved to under 5% error by the Spiking RelNet; longer time steps and larger time constants solved all 17 with higher energy consumption and latency.Source: Design of a very sparsely active large… · p. 9
Discussion
- On-chip training with e-prop and richer cortical neuron models such as GLIF3 are proposed as future directions for AHP-based neuromorphic systems.The paper also suggests these systems could support more energy-efficient simulations of large brain-network models.Source: Discussion · p. 11
Methods
- Each AHP-neuron spike strengthens a slowly decaying inhibitory current, suppressing later spikes and providing longer-term memory implemented in a separate Loihi compartment.The AHP time constant is typically over 100 ms, compared with roughly 20 ms for voltage and synaptic currents.Source: Methods · p. 12
- AHP currents provide a slow-decay path for informative gradients to travel far back in time, unlike standard LIF gradients that decay through voltage, synaptic currents, and surrogate derivatives.For standard LIF neurons, voltage and synaptic gradient decay factors correspond to 5–20 ms time constants, while the surrogate-gradient dampening factor is kept low.Source: Methods · p. 14
- 29 Hz per neuron was the average firing rate of the sparsely connected AHP-SNN classifying the MNIST test set on Loihi.Training constrained overall connectivity to 20% of possible synapses to fit Loihi neuro-core axonal and synaptic resources.Source: Methods · p. 15
- The Spiking RelNet embeds sentences and questions with AHP-SNNs, applies a feed-forward relational function to each sentence pair, and uses shared weights in its aggregation layer.The element-wise aggregation layer is an addition to the proposed architecture that enables Spiking RelNet implementation on neuromorphic hardware.Source: Methods · pp. 15–16
- 15 hours on one GPU for embedding pretraining enabled Spiking RelNet training in 80 epochs, which took over a week on two GPUs without that pretraining approach.The sentence and question AHP-SNN embeddings were pretrained to reproduce LSTM outputs, then fixed while the relational and readout functions were trained.Source: Methods · p. 16
- Using neurons without a refractory period and an AHP time constant of 7 ms reduced required simulation steps with only a mild accuracy decrease; stretched dynamics solved all 17 tasks at higher energy and latency.The number of time steps affects Loihi compute time, energy, delay, and training speed.Source: Methods · p. 16
- Although relational-function modules grow quadratically with sentence count, relevant relations grow linearly, so global spike-rate regularization concentrates activity on relevant sentence pairs.The resulting low spike rate supported power- and delay-efficient feed-forward spiking hardware implementations.Source: Methods · pp. 16–17
- Voltage regularization keeps scaled membrane voltages near the surrogate-gradient support, preserving gradient propagation and fitting voltage values within Loihi’s fixed-precision registers.The loss penalizes scaled voltages outside [−2.0, 0.4], while the surrogate-gradient support is [−1.0, 0.5].Source: Methods · p. 17
- The sMNIST AHP-SNN uses 240 neurons with 20% recurrent connections and fits on one neuro-core; the Spiking RelNet reaches 238,604 neurons for up to 20 sentences.Source: Methods · pp. 17–18
- With an empirical limit of about 40,000 synapses per neuro-core, dense-input and recurrent layers must span multiple neuro-cores.Source: Methods · p. 18
- Relay layers reproduce AHP-SNN spikes to reduce fanout, while sparse one-to-one connections to an aggregation layer address Loihi's fanin limit.Neuro-cores have limits of 2,048 output neuro-core connections (4,096 for intra-chip connections) and 4,096 input neurons.Source: Methods · p. 18
- Placement rules keep relay-to-relational traffic on-chip, group later relational layers, and reduce distinct sentence inputs per chip; Fig. 5E shows improved delay and energy-delay product.Source: Methods · p. 18
Data availability
- The MNIST and bAbI datasets are freely available.Source: Data availability · p. 18
Code availability
- Loihi source code is freely available on GitHub.Source: Code availability · p. 19
Author contributions statement
The statement credits the authors’ roles in the research, discloses Intel-related competing interests, and notes supplementary information.
Author contributions statement
- A.R., P.P., A.W. and W.M. analyzed the experimental data and wrote the manuscript; A.R., P.P. and W.M. designed and planned the experiments, which A.R. and P.P. carried out.Source: Author contributions statement · p. 20
Competing interests
- P.P. and A.W. are employed by Intel Labs; W.M. and A.R. belong to the Intel Neuromorphic Research Community, and W.M. received Intel research funding for related work.Source: Competing interests · p. 20
Additional information
- Supplementary information is available.Source: Additional information · p. 20
S2 Energy and Time benchmarking S2
- The supplementary contents list sMNIST energy and latency benchmarking and Spiking RelNet results, architecture, training, Loihi connectivity constraints, and network placement.Source: S2 Energy and Time benchmarking S2 · p. 21
S1 Neuron models
Across the listed AHP-SNN examples, AHP-neuron parameters match LIF-neuron parameters except for the additional β and τAHP parameters. The table covers sMNIST and Spiking RelNet variants on bAbI, including a slowed simulation with 3× temporal resolution.
S1.1 Neuron Parameters
- AHP-neuron parameters match LIF-neuron parameters except for the additional β and τAHP parameters.The listed configurations are for the sMNIST task and Spiking RelNet examples on bAbI.Source: Table S1 · p. 22
- The parameter examples cover a single AHP-SNN for sMNIST and the original Spiking RelNet benchmarked on bAbI.Source: Table S1 · p. 22
- A separate Spiking RelNet configuration slows the simulation and increases temporal resolution by a factor of 3, with results reported in Table S3.Source: Table S1 · p. 22
S2 Energy and Time benchmarking
The section benchmarks the energy and latency of spiking networks on Intel Loihi [Davies et al., 2018] against artificial neural network implementations on CPUs and GPUs. It describes hardware power measurement and steady-state, wall-clock timing procedures.
S2 Energy and Time benchmarking
- Energy and latency measurements compare spiking networks on Loihi with artificial neural network implementations on CPUs and GPUs.Loihi performance was read from hardware sensors.Source: S2 Energy and Time benchmarking · p. 22
- Loihi power telemetry over I2C supports estimates of static and dynamic consumption and contributions from neuro-cores and synchronous on-chip x86 cores.Power usage was collected through the system’s on-board voltage regulators.Source: S2 Energy and Time benchmarking · p. 22
- CPU power was estimated with Intel Power Gadget 3.5, while nvidia-smi measured whole-board GPU power; idle baseline energy was subtracted to calculate dynamic energy.The GPU tool does not provide a detailed breakdown of where power is consumed.Source: S2 Energy and Time benchmarking · pp. 22–23
- Workloads ran long enough to reach steady-state power, and Python timeit measured wall-clock execution time independently of power measurements for per-sample latency.GPU tests with batch sizes 50 and 100 repeated the test set to reach steady state.Source: S2 Energy and Time benchmarking · p. 23
S3 Details for AHP-SNN on the sMNIST task
The section describes threshold-crossing spike encoding for sMNIST and outlines the energy, latency, and energy-delay metrics used to benchmark the network against LSTM implementations.
S3.1 Input encoding for sMNIST
- Adjacent-pixel crossings trigger direction-specific spikes: neuron 2t for an upward crossing and 2t+1 for a downward crossing, using half as many linearly spaced thresholds as input neurons.Thresholds span pixel values 0 to 255, and the current and next pixel values are presented.Source: S3.1 Input encoding for sMNIST · p. 23
S3.2 Energy and latency benchmarking results
- The sMNIST benchmark compares the AHP-SNN on Loihi with LSTM implementations on GPU and CPU by energy, output latency, and energy-delay product.The supplied passages do not report benchmark values or identify a winner.Source: S3.2 Energy and latency benchmarking… · p. 23, Table S2 · p. 24
S4 Details for the Spiking RelNet on the bAbI task · S4.1 Output accuracy of the Spiking RelNet
The Spiking RelNet solves 16 of 17 trained bAbI tasks within 5% error; tripling simulation times allows all 17 to meet that threshold but increases energy-delay product 4.5-fold.
S4.1 Output accuracy of the Spiking RelNet
- The Spiking RelNet solves 16/17 trained tasks within the 5% error threshold; three tasks were excluded because the non-spiking RelNet did not converge on them.The excluded tasks are Task 2: Two Supporting Facts, Task 3: Three Supporting Facts, and Task 16: Basic Induction.Source: S4.1 Output accuracy of the Spiking… · p. 25
- Task 17: Positional Reasoning has comparatively high error, which the authors attribute to complex sentences requiring longer computation time.They frame this as a trade-off for faster training and energy efficiency; Task 17 can be solved with more time steps.Source: S4.1 Output accuracy of the Spiking… · p. 25
- Extending feed-forward simulation from Tsim = 37 ms to 45 ms significantly reduces Task 17 error, but it remains above the 5% solved threshold.This extension pads the embeddings and runs relational-network modules C–E for the longer simulation.Source: S4.1 Output accuracy of the Spiking… · p. 25
- Tripling temporal parameters lets the Spiking RelNet solve all 17 tasks within 5% error, but increases energy-delay product 4.5-fold (from 148.47 to 663.80).The tripled simulation uses Tinp = 40 ms, Tsim = 110 ms, and Tword = 30 ms, with roughly the same total number of spikes in both cases.Source: S4.1 Output accuracy of the Spiking… · p. 25
- The reported trade-off is greater energy efficiency with a slight decrease in overall performance, with only one task losing performance.With the standard simulation setting, Task 17 is the task with high error; longer time steps can solve it.Source: S4.1 Output accuracy of the Spiking… · p. 25
S4.2 Energy and latency benchmarking results for the Spiking RelNet
The section benchmarks the Spiking RelNet on Loihi against a non-spiking RelNet on a GPU, comparing energy, output latency, and energy-delay product across task sizes.
S4.2 Energy and latency benchmarking results for the Spiking RelNet
- The benchmark compares energy consumed, output latency, and energy-delay product for the Spiking RelNet on Loihi versus a non-spiking RelNet on a GPU, varying task size.The caption identifies the dataset as bAbI and describes the analysis as a scaling comparison.Source: S4.2 Energy and latency benchmarking… · p. 25, Table S4 · p. 27
- Samples were grouped by sentence count, which determines the number of AHP-SNNs and cores per sample; measurements used 250 inputs, except network size 16 used 100.The caption attributes the smaller sample count at size 16 to a shortage of test samples containing 16 sentences.Source: Table S4 · p. 27
S4.3 Details of the Spiking RelNet architecture
The architecture encodes sentences and questions as spike sequences, computes relations for sentence pairs, aggregates relation outputs, and applies a spiking readout.
The embedding networks
- Separate AHP-SNNs embed sentences and questions using different weight sets but otherwise identical parameters.Each network has 100 LIF and 100 AHP neurons. Dedicated input neurons emit for 10 ms per word; the longest input is 110 ms. The final 14 ms of activity is padded to 37 ms so spikes can propagate through later feed-forward layers.Source: The embedding networks · p. 28
The relational function
- A four-layer LIF network without AHP currents computes an output spike sequence for each sentence pair i ≤ j, including self-pairs.Each layer has 256 neurons, connects all-to-all to the next, and uses synaptic delays of 1–3 ms.Source: The relational function · p. 28
The aggregation layer
- A single LIF aggregation layer sums relation-instance output spikes element-wise, an addition to Santoro et al. [Santoro et al., 2017] required by neuromorphic hardware constraints.Each aggregation neuron receives the summed output spikes across all relational-function instances and produces a spike sequence.Source: The aggregation layer · pp. 28–29
The readout function
- The readout uses a three-layer feed-forward LIF network with layer sizes 256, 512, and 160, followed by a specially designed linear readout.The layers, including the readout, are connected all-to-all, with synaptic delays of 1–3 ms.Source: The readout function · p. 29
S4.4 Details of Spiking RelNet training
Spiking RelNet training pre-trains AHP-SNN sentence and question encoders to reproduce pretrained LSTM outputs, then freezes them and uses their spike embeddings to train the feed-forward network.
Pretraining the AHP-SNN networks
- The AHP-SNN sentence and question encoders are trained to reproduce outputs from an end-to-end non-spiking RelNet that uses LSTMs and has reached optimal performance.This pretraining is chosen to reduce the epochs required to train the Spiking RelNet.Source: Pretraining the AHP-SNN networks · p. 29
- A 32-unit linear readout is trained to match LSTM outputs with mean squared error, which serves as the loss for AHP-SNN weight training using BPTT.Source: Pretraining the AHP-SNN networks · p. 29
- After pretraining, AHP-SNN weights are frozen, and their spikes connect directly to gθ through randomly initialized feed-forward weights; the pretraining readout weights are not used.The authors find that the sentence information is encoded in the spike embeddings and can be processed by the feed-forward part.Source: Pretraining the AHP-SNN networks · p. 29
Rate regularization for gθ instances
The method applies more aggressive spike-rate regularization to gθ instances than to other layers, targeting each neuron's summed rate across instances for each story.
Rate regularization for gθ instances
- Unlike the rate regularization for other layers, which moves each neuron's batch-mean rate toward a target, gθ instances receive more aggressive rate regularization.The trained spike rates are referenced in main-text Fig. 3B.Source: Rate regularization for gθ instances · pp. 29–30
- For each gθ layer, the loss pushes each neuron's summed spike rate across instances toward Rtarget = 300 Hz.This corresponds to 3.7 spikes per neuron for a two-sentence task and 0.05 spikes per neuron for a 20-sentence task; the layer losses are summed into the final loss.Source: Rate regularization for gθ instances · pp. 29–30
S4.5 Constraints on connectivity on Loihi
For placing a Spiking RelNet that processes stories up to M = 20 sentences on Loihi, the section explains how connectivity constraints shape layer placement and motivate relay and aggregation layers.
Synaptic memory limit
- Dense input connections exceed a neuro-core’s synaptic-parameter SRAM capacity, so postsynaptic neurons must be split across cores to fit their incoming synapses.The aggregation layer is excepted from the dense-input pattern described here.Source: Synaptic memory limit · p. 30
Limits pertaining to fanouts – AHP-SNN relay layer
- Because a question-AHP-SNN neuron fans out to 840 neuro-cores, violating the 512-core limit, relay layers reduce AHP-SNN fanout and can be split without added latency.The output-axon limit also constrains AHP-SNN placement, while spreading recurrent neurons across too many cores adds latency.Source: Limits pertaining to fanouts – AHP-SNN… · p. 31
Input axon limit – the aggregation layer
- 53,760 presynaptic neurons per readout neuron exceed Loihi’s 4096-input-axon limit; aggregation reduces this to 210 per neuron, allowing 256 neurons across 14 neuro-cores.The original Relational Network connects each relational-function instance to the final readout all-to-all; the added spiking LIF aggregation layer receives one-to-one connections from those instances with shared weights.Source: Input axon limit – the aggregation layer · p. 32
S4.6 Optimized network placement to minimize spike congestion
The placement scheme targets the main source of spike congestion by co-locating relay layers with the first layers of relational-function instances and minimizing cross-chip connections. Grouping instances by shared sentence inputs reduces relay-network use, while keeping subsequent layers together limits later cross-chip communication.
Optimized network placement to minimize spike congestion
- Because most congestion occurs between AHP-SNNs, relay layers, and gθ inputs, each relay layer is placed on the same chip as its gθ instance’s first layer.This makes the AHP-SNN-to-relay connections the cross-chip communication to optimize.Source: S4.6 Optimized network placement to… · pp. 32–33
- With a limit of 128 neuro-cores per chip, gθ instances are grouped to minimize their distinct input sentences and thus the number of relay layers; Figure S2 shows this arrangement.The instances are organized by their sentence indices, with squares grouping those that need fewer distinct inputs.Source: S4.6 Optimized network placement to… · p. 33, Figure S2 · p. 34
- Each gθ instance’s subsequent layers are placed in sequence, with its remaining three layers kept on one chip; the resulting cross-chip communication is minimal and causes no significant delay.Source: S4.6 Optimized network placement to… · p. 33
- The layout reduces cross-chip spike congestion by keeping most connections within a chip and raises the relay-network output axon limit from 2048 to 4096, requiring fewer relay networks.Each relay network fans out only to neuro-cores on the same chip.Source: S4.6 Optimized network placement to… · pp. 33–34