Source-linked AI summary
Long short-term memory and learning-to-learn in networks of spiking neurons
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, Wolfgang Maass
TL;DR
RSNN models have lagged behind ANNs despite the brain’s optimized connectivity, prior learning, and richer neuronal dynamics. The paper combines BPTT, DEEP R rewiring, and neuronal adaptation to create LSNNs, which approach LSTM performance and learn new related tasks efficiently through L2L.
Problem
RSNN models have remained suboptimal because they omit optimization histories and biological dynamics present in brain networks.
Method
The paper combines BPTT and DEEP R with adapting spiking neurons, then applies the resulting LSNNs in learning-to-learn settings.
Results
LSNNs approach LSTM performance on Sequential MNIST and TIMIT, and L2L enables RSNNs to learn new related tasks from few examples without synaptic plasticity.
Takeaways & Limitations
Sparse, adapting RSNNs can compute with spikes while supporting enhanced computation, fast task learning, and reinforcement-learning capabilities.
Takeaways & Limitations
The LSNN architecture and meta-parameters were not searched as rigorously as those for the compared LSTM models.
Abstract
from arXiv · showhide
Recurrent networks of spiking neurons (RSNNs) underlie the astounding computing and learning capabilities of the brain. But computing and learning capabilities of RSNN models have remained poor, at least in comparison with artificial neural networks (ANNs). We address two possible reasons for that. One is that RSNNs in the brain are not randomly connected or designed according to simple rules, and they do not start learning as a tabula rasa network. Rather, RSNNs in the brain were optimized for their tasks through evolution, development, and prior experience. Details of these optimization processes are largely unknown. But their functional contribution can be approximated through powerful optimization methods, such as backpropagation through time (BPTT). A second major mismatch between RSNNs in the brain and models is that the latter only show a small fraction of the dynamics of neurons and synapses in the brain. We include neurons in our RSNN model that reproduce one prominent dynamical process of biological neurons that takes place at the behaviourally relevant time scale of seconds: neuronal adaptation. We denote these networks as LSNNs because of their Long short-term memory. The inclusion of adapting neurons drastically increases the computing and learning capability of RSNNs if they are trained and configured by deep learning (BPTT combined with a rewiring algorithm that optimizes the network architecture). In fact, the computational performance of these RSNNs approaches for the first time that of LSTM networks. In addition RSNNs with adapting neurons can acquire abstract knowledge from prior learning in a Learning-to-Learn (L2L) scheme, and transfer that knowledge in order to learn new but related tasks from very few examples. We demonstrate this for supervised learning and reinforcement learning.
1 Introduction
The paper targets RSNNs’ limited temporal-processing performance by combining deep optimization with biologically motivated neuronal adaptation. The resulting LSNNs support stronger computation, learning, and sparse spike-based activity.
- Motivation: RSNN models may underperform because their connectivity and learning history differ from biologically optimized brain networks.The paper asks whether deep learning can approximate evolutionary, developmental, and prior-learning effects whose biological details remain unknown.
- Motivation: The approach combines BPTT with DEEP R synaptic rewiring to optimize RSNN weights and connectivity.This is intended to approximate functional contributions of evolutionary and developmental optimization.
- Contributions: LSNNs reach new RSNN performance levels on Sequential MNIST and TIMIT, while also enabling learning-to-learn and meta-reinforcement learning.The paper reports new motor-control capabilities and supervised and reinforcement-learning applications.
- Contributions: The resulting RSNNs use sparse connectivity and 10-20 Hz sparse firing activity, computing with spikes rather than firing rates.The paper links these capabilities to possible implementation on spike-based neuromorphic chips.
- Related work: The paper positions LSNNs as an alternative to prior RSNN work focused mainly on generating dynamic patterns.It distinguishes the present focus on computation and learning from that earlier task emphasis.
- LSNN model: Neuronal adaptation adds longer-timescale dynamics missing from common RSNN models and provides the basis for Long short-term memory Spiking Neural Networks.Adaptive neurons temporarily reduce excitability after firing through increased, decaying firing thresholds.
3 Applying BPTT with DEEP R to RSNNs and LSNNs
The paper uses BPTT to optimize LSNN synaptic weights and combines it with DEEP R to optimize connectivity. BPTT is treated as an effective task-level optimization method rather than a biologically realistic learning rule.
- Optimization: BPTT optimizes LSNN synaptic weights for specific ranges of tasks, although the paper does not claim biological realism for the algorithm.The authors compare its functional role to optimization through evolution and development.
- Optimization: DEEP R extends the optimization to the connectivity matrix by continuously updating the set of active synaptic connections.The paper describes DEEP R as a biologically inspired rewiring method that theoretically converges to an optimal network configuration.
4 Computational performance of LSNNs
LSNNs are evaluated on sequential handwritten-digit classification and speech recognition, with adaptation supporting long-timescale processing. Their performance approaches LSTM benchmarks, while architecture search remains less extensive for LSNNs.
- Sequential MNIST: Sequential MNIST presents each handwritten digit pixel by pixel over 784 steps, requiring continuous short-term-memory updates.The experiments use 1 ms and 2 ms pixel presentation steps, with adaptation time constants adjusted accordingly.
- Sequential MNIST: 94.7% and 96.4% LSNN test accuracy were achieved for 1 ms and 2 ms pixels, compared with 98.5% and 98.0% for LSTM networks.The LIF and RNN baselines did not reach this accuracy without enhanced short-term-memory capabilities.
- Sequential MNIST: The DEEP R LSNN uses a sparse architecture whose readout connectivity is denser than connectivity elsewhere in the network.Figure 1C presents connection probabilities among the network’s three subpopulations.
- TIMIT: A standard LSNN achieved 33.2% classification error on TIMIT, below the approximately 40% mean error of the best-performing complex LSTM version.Its error remained above the 29.7% mean of the 20 best LSTM hyperparameter choices but improved on the no-forget-gate LSTM result of 34.2%.
- TIMIT: Replacing all adapting neurons with regularly firing excitatory neurons increased TIMIT error to 37%.This comparison isolates the reported performance difference associated with the adapting-neuron configuration.
5 LSNNs learn-to-learn from a teacher
LSNNs use outer-loop learning-to-learn to acquire priors that support rapid learning of new nonlinear functions from teacher-provided examples, without changing synaptic weights during the task.
- Learning-to-learn framework: L2L optimizes LSNN hyperparameters across a family of tasks so the network can learn individual tasks rapidly in an inner loop.The outer loop uses BPTT and integrates performance across many randomly drawn tasks.
- Learning-to-learn framework: The supervised task family contains nonlinear functions generated by two-layer target networks with 40 randomly selected weights and biases.Each target network has 30 weights and 10 biases drawn from [-1,1].
- Results: After outer-loop training, LSNNs approximated new target networks after only 5 to 20 labelled trials, outperforming an optimal linear predictor.The learned internal model is stored in the LSNN’s short-term memory.
- Results: The LSNN also learned sinusoidal functions with arbitrary phase and amplitudes, retaining its prior knowledge under adversarially selected examples.The adversarial examples happened to lie in a straight line.
- Significance: This L2L-induced learning provides structural priors for online learning in recurrent spiking networks without modifying synaptic weights during a task.The authors describe this as a previously undocumented capability for SNNs.
6 LSNNs learn-to-learn from reward
Meta-RL enables LSNN agents to learn navigation strategies from rewards and use abstract task regularities to improve later behavior within episodes.
- Task: The LSNN agent was trained across navigation tasks in a circular arena with continuous states and actions, receiving rewards for reaching randomly placed goals.Each episode lasted 2 seconds, and goals were placed randomly on the arena border.
- Learning dynamics: Before meta-RL training, the agent performed a random walk and could fail to find the goal within an episode.The pretraining behavior is shown in Figure 3C.
- Learning dynamics: After 300k training episodes, the agent used two abstract facts: the goal lies on the border and its position remains fixed throughout an episode.These facts supported more efficient exploration and subsequent navigation.
- Conclusion: Meta-RL produced new reward-based control capabilities in sparsely firing recurrent spiking networks.The result extends learning-to-learn from teacher inputs to reward-driven behavior.
7 Discussion
The discussion presents adapting neurons, BPTT, rewiring, and L2L as a route to stronger, sparse-firing RSNNs for computation and rapid task learning.
- Computational capabilities: BPTT combined with DEEP R produced sparsely connected RSNNs approaching LSTM performance on sequential MNIST and TIMIT.The paper attributes the qualitative performance jump to introducing adapting neurons with longer time constants.
- Learning capabilities: L2L enabled RSNNs to learn nonlinear mappings from new tasks without backpropagation or synaptic-weight changes during task learning.Outer-loop training installed abstract knowledge about shared task properties.
- Learning capabilities: Meta-RL allowed an RSNN agent to search for a goal and later exploit learned knowledge to navigate efficiently from random starting positions.The agent required no additional outer control or clock.
- Implications: The resulting networks used sparse firing activity and short-term memory for functions beyond simply remembering a preceding input.The authors relate these behaviors to possible forms of activity-silent memory.
- Implications: These findings suggest application paradigms for spike-based hardware based on energy-efficient sparse firing and non-firing periods.The discussion names neuromorphic implementation as an application area.
Supplementary information for: Long short-term memory and learning-to-learn in networks of spiking
The supplement provides detailed model and simulation information corresponding to the main text’s sections.
- Publication details: The paper is authored by Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass.The listed affiliation is the Institute for Theoretical Computer Science at Graz University of Technology.
- Publication details: The work is identified as arXiv:1803.09574v4, dated 25 December 2018.The record lists the computer science neural and evolutionary computation category.
- Supplementary information: The supplementary information is organized according to the corresponding sections of the main text.It is presented as a detailed account of the models and simulations.
2 LSNN model
The LSNN model uses adaptive leaky integrate-and-fire neurons whose spike-driven thresholds provide longer-timescale dynamics, with recurrent spike inputs and leaky readouts.
- Neuron model: LSNN neurons follow an adaptive leaky integrate-and-fire model in which spikes reset membrane voltage and raise a threshold that decays toward baseline.The adaptation time constant controls the threshold dynamics, while neurons also observe a strict refractory period.
- Synaptic input and readout: The input current is the weighted sum of spikes from external inputs and recurrent neurons, with corresponding synaptic delays and weights.Network neurons project to readout neurons through output synapses.
- Synaptic input and readout: Readout neurons receive added voltage from network spikes and integrate it with leaky dynamics governed by the membrane time constant.Each network neuron connects to the readout population through output weights.
- Discrete-time dynamics: The membrane potential and adaptive threshold evolve in discrete time, with leak, weighted input current, spike-triggered reset, and threshold adaptation terms.Simulations use a 1 ms time step and binary spike trains.
3 Applying BPTT with DEEP R to RSNNs and LSNNs
The training procedure combines BPTT with surrogate gradients and DEEP R rewiring to optimize both LSNN weights and sparse network connectivity while stabilizing recurrent learning.
- Gradient propagation: BPTT addresses recurrent training, while pseudo-derivatives provide a gradient surrogate for the non-differentiable spikes.The method must propagate gradients through continuous or discretized recurrent dynamics.
- Gradient propagation: A pseudo-derivative amplitude γ < 1, typically γ = 0.3, dampens back-propagated error growth and stabilizes deep unrolled recurrent networks.The dampening is applied to gradient propagation through spikes.
- Gradient propagation: Adaptive neurons retain an additional gradient pathway through their dynamic thresholds across many time steps, and this pathway is not dampened.This distinguishes threshold-state propagation from propagation through spikes.
- Initialization and stability: Weight scaling by 1/√nin and spectral normalization are used to initialize trainable networks with realistic firing rates and avoid exploding gradients.The largest eigenvalue is scaled to be at most 1 when applicable.
- Connectivity optimization: When excitatory and inhibitory signs are constrained, DEEP R preserves each synapse's sign, inherited from the initial neuron-sign assignment.Weight initialization is adapted to the prescribed fractions of inhibitory and excitatory neurons.
- Connectivity optimization: DEEP R is integrated with BPTT to update active connections and optimize the LSNN connectivity matrix as well as synaptic weights.The rewiring method continuously disconnects and reconnects synapses while preserving the connectivity level.
4 Computational performance of LSNNs
LSNNs are evaluated on sequential MNIST and TIMIT using temporally encoded inputs and sparse rewired architectures, with adaptation and task-specific training settings.
- Sequential MNIST: Sequential MNIST presents the 784 image pixels one after another, using either population-rate or threshold-crossing spike encodings.The task evaluates digit classification after the full sequential presentation.
- Sequential MNIST: The network averages readout activity after digit presentation and trains with cross-entropy, while the best models use 12% global connectivity during rewiring.Results are reported for both 1 ms and 2 ms pixel durations.
- TIMIT: TIMIT tests frame-wise classification of audio into 61 phoneme classes after preprocessing speech into 39 MFCC-derived input channels.The LSNN receives each 10 ms audio frame over ten 1 ms simulation steps.
- Model configurations: LSNNs use mixed populations of regular and adaptive neurons, with adaptation time constants selected separately for sequential MNIST and TIMIT.The TIMIT network contains 300 regular and 100 adaptive neurons.
5 LSNNs learn-to-learn from a teacher
The LSNN is trained across families of target functions so that it can infer new functions from sequential examples, outperforming a linear trace-based baseline on both tested families.
- Task families: The learning-to-learn setup trains LSNNs to regress functions sampled from either random target networks or parameterized sinusoidal functions.Target-network functions use two inputs and one output; sinusoidal functions vary in phase and amplitude.
- Meta-training procedure: Each episode samples a new target function and presents one input example per step, together with the previous step's target value.Episodes contain 500 steps, enabling the LSNN to learn from sequential examples.
- Meta-training procedure: BPTT updates the LSNN once per batch of 10 episodes using mean squared prediction error accumulated across all episode steps.The method trains the network to perform the regression procedure rather than fit one fixed function.
- Results: For target-network functions, test MSE is 0.0056 ± 0.0039 versus 0.0217 ± 0.0046 for the linear baseline.The baseline uses L2-regularized linear regression on the neurons' mean spiking trace.
- Results: For sinusoidal functions, test MSE is 0.3134 ± 0.2293 versus 1.4592 ± 1.2958 for the linear baseline.The same comparison evaluates whether the learned procedure generalizes across sampled sinusoidal targets.
6 LSNNs learn-to-learn from reward
The LSNN agent learns navigation by encoding position and reward into spikes, producing stochastic velocity actions, and optimizing its policy over repeated episodes. The task rewards reaching circular goals while penalizing attempts to leave the arena.
- Task family: The agent navigates a two-dimensional circular arena with radius 1 toward goals of radius 0.3 placed around radius 0.85.Goal centers are uniformly distributed on the circle of radius 0.85.
- Task family: Reaching a goal yields reward 1, whereas attempting to move outside the arena yields reward −0.02.The agent’s position is reset uniformly within the arena after reaching a goal.
- Input encoding: At each timestep, the agent receives its position and reward as inputs, with each position coordinate encoded by 40 neurons using a Gaussian population rate code.
- Output decoding: The five readout neurons define a stochastic action distribution whose sampled two-dimensional velocity updates the agent’s position.The first four readouts parameterize Gaussian action means and variances, while the fifth predicts the value function.
- Network training: Training uses PPO over K = 10 episodes of T = 2000 timesteps, with a loss combining the policy objective, value error, entropy, and firing-rate regularization.The trajectories are conditioned on fixed current noise during each update, and the loss is minimized with one ADAM step.
- Network training: The task LSNN contains 400 hidden units, 20% global connectivity, adaptive neurons with τa = 1200 ms, and adaptation amplitude β = 1.7.The network includes 200 excitatory, 80 inhibitory, and 120 adaptive neurons.