Source-linked AI summary
Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity
Thomas Miconi, Aditya Rawal, Jeff Clune, Kenneth O. Stanley
TL;DR
Brains use neuromodulated plasticity to control long-term synaptic change, but prior differentiable plasticity methods supported only passive changes. This paper makes neuromodulated plasticity trainable with gradient descent, and reports better performance across reinforcement-learning and supervised tasks, including language modeling.
Problem
Differentiable plasticity allowed gradient-based optimization of passive weight changes, but the paper identifies a need to train networks whose plasticity is actively controlled by neuromodulation.
Source: §1 · p. 2Method
The paper extends differentiable plasticity with network-computed neuromodulatory signals that control plastic changes, including retroactive gating through eligibility traces.
Source: §1 · p. 2, §3.2.2 · p. 4Results
Neuromodulated plastic networks outperform non-plastic and non-modulated networks on simple reinforcement-learning and language-modeling tasks; retroactive neuromodulation improves perplexity by about 1.7 vs. the Baseline LSTM.
Source: §1 · p. 2, §4.3 · p. 7Takeaways & Limitations
Differentiable neuromodulation offers a framework for training large-scale self-modifying neural networks with gradient descent.
Source: §5 · p. 8The framework needs evaluation on more complex tasks and improvements including multiple neuromodulatory signals and addressing pitfalls in reward-modulated Hebbian learning.
Source: §5 · p. 8
Abstract
from arXiv · showhide
The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The resulting self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning. Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent. Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity. We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks. In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters). We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks.
1 INTRODUCTION
Brains store long-term information through synaptic plasticity, whose moment-to-moment neuromodulation helps control learning and adaptation. This paper introduces differentiable neuromodulated plasticity, allowing gradient descent to train networks that outperform non-plastic and non-modulated networks on the reported tasks.
1 INTRODUCTION
- Neuromodulation lets the brain control where and when plasticity occurs, filtering irrelevant events, selectively incorporating important information, and supporting reward-dependent reinforcement learning.The passage also links this control to combating catastrophic forgetting.Source: §1 · p. 1
- Gradient-based training of neuromodulated plastic networks would extend deep learning with biologically inspired self-modifying abilities; prior evolved networks were small.The paper frames gradient descent as an alternative to evolution for optimizing these networks.Source: §1 · pp. 1–2
- Backpropamine extends differentiable plasticity to neuromodulation, enabling gradient-descent training and outperforming non-plastic and non-modulated networks on simple reinforcement-learning and complex language-modeling tasks.The language-modeling network has millions of parameters.Source: §1 · p. 2
2 RELATED WORK
Prior work studied neuromodulated plasticity in evolved networks and differentiable plasticity with passive weight changes. This paper instead directly optimizes neuromodulation of plasticity within a network using gradient descent.
2 RELATED WORK
- Evolved networks with neuromodulated plasticity outperformed non-neuromodulated and non-plastic networks in various tasks, but were historically small and used low-dimensional problem spaces.A focus of this work was mitigating catastrophic forgetting by activating plasticity only in weights relevant to the current task.Source: §2 · p. 2
- Differentiable plasticity made individual connections' plasticity trainable by gradient descent, but supported only passive changes driven by pre- and post-synaptic activity.Prior work reported that this approach could improve performance over recurrence without plasticity in some tasks.Source: §2 · p. 2
- Unlike more complex approaches that compute weight modifications with a neural network, this work directly optimizes plasticity's neuromodulation within one network using gradient descent instead of evolution.The passage presents this direct optimization as an approach not taken by the cited alternatives.Source: §2 · p. 2
3 METHODS
The methods build on differentiable Hebbian plasticity, where trainable structural parameters coexist with activity-dependent traces. Backpropamine lets a network-controlled signal modulate those traces, either immediately or retroactively through eligibility traces.
3.1 BACKGROUND: DIFFERENTIABLE HEBBIAN PLASTICITY
- Each connection combines a fixed weight with a trainable plastic component whose magnitude is scaled by its plasticity coefficient.The plastic component is represented by a Hebbian trace that accumulates pre- and post-synaptic activity.Source: §3.1 · p. 3
- Structural components w_i,j, α_i,j, and η are optimized by gradient descent between episodes to minimize expected episode loss, while Hebb_i,j is initialized to zero each episode and remains purely intra-life.Hebb_i,j is constrained to [−1, 1] by a hard clip; η sets how fast new information enters the plastic component, whereas α_i,j sets its maximum magnitude.Source: §3.1 · p. 3
3.2 BACKPROPAMINE: DIFFERENTIABLE NEUROMODULATION OF PLASTICITY
- Backpropamine modulates plasticity moment to moment with a network-controlled signal M(t), computed as a scalar output and optionally passed through trainable connection-specific weights.The described experiments use the direct scalar signal for simple reinforcement-learning tasks and a trainable weight vector for language modeling.Source: §3.2 · p. 3
3.2.1 SIMPLE NEUROMODULATION
- Simple neuromodulation replaces the global plasticity learning-rate parameter η with the network-computed, time-varying signal M(t).This gives the network control over how plastic its connections should be at each moment.Source: §3.2.1 · p. 4
3.2.2 RETROACTIVE NEUROMODULATION AND ELIGIBILITY TRACES
- Retroactive neuromodulation uses an eligibility trace—an exponential average of pre- and post-synaptic activity with trainable decay η—that accumulates into the plastic component when gated by current M(t).M(t) can be positive or negative, approximating rises or dips in baseline dopamine levels.Source: §3.2.2 · p. 4
4 EXPERIMENTS
Across three tasks, the experiments test neuromodulated plastic networks in reinforcement learning and language modeling. The results show learning on the cue-association task, an advantage over non-modulated plasticity in maze navigation, and lower language-model perplexity with neuromodulation.
4.1 TASK 1: CUE-REWARD ASSOCIATION
- The cue task used 20-bit cues and variable trial timing in 200-step episodes; only recurrent connections in a 200-neuron recurrent layer were plastic.Inputs also included elapsed episode time, the previous response, and the previous reward; outputs included a response, a future-reward estimate, and a neuromodulatory signal.Source: §4.1 · pp. 4–5
- Modulated plastic networks learned the cue-reward task, while non-modulated plastic and non-plastic networks failed.The authors hypothesize that neuromodulation helps memorize reward associations with arbitrary high-dimensional stimuli; the neuromodulator responded to reward in a complex, time-dependent manner.Source: §4.1 · p. 5, Figure 1 · p. 5
4.2 TASK 2: MAZE NAVIGATION TASK
- The maze agent used 100 recurrent neurons, four action outputs, and inputs encoding its 3 × 3 wall neighborhood, prior action, and prior reward.Only recurrent weights were plastic.Source: §4.2 · pp. 5–6
- In maze exploration, modulatory approaches outperformed non-modulated plasticity.The reward location was invisible, fixed within an episode, and randomized across episodes.Source: §4.2 · pp. 5–6, Figure 2 · p. 6
4.3 TASK 3: LANGUAGE MODELING
- The language experiments compared baseline, differentiably plastic, simply neuromodulated, and retroactively neuromodulated LSTMs using parameter-matched smaller models and a larger model.The larger-model comparison used simple neuromodulation only; its plastic version reduced two layer sizes to keep the parameter count from exceeding baseline.Source: §4.3 · pp. 6–7
- Retroactive neuromodulation lowered Penn Tree Bank test perplexity by about 1.7 versus Baseline LSTM; the large model's modulated version showed a small improvement over its non-plastic counterpart.The retroactive-versus-simple neuromodulation improvement was barely not statistically significant (p = 0.066).Source: §4.3 · pp. 7–8, Table 1 · p. 7
- For the smaller language model, each variant received an equally powered hyperparameter search and was evaluated over 16 runs.The reported metric was mean test perplexity with a 95% confidence interval.Source: §4.3 · p. 7, Table 1 · p. 7
5 DISCUSSION AND FUTURE WORK
The paper introduces gradient-trained neuromodulated plastic networks and reports that plastic and neuromodulated LSTMs outperform standard LSTMs on benchmark language modeling. It also relates the approach to L2RL and identifies several improvements needed to investigate its full potential.
5 DISCUSSION AND FUTURE WORK
- The framework makes neuromodulated plastic networks trainable with gradient descent, opening research into optimizing large-scale self-modifying networks.The authors describe this as the first such gradient-trained approach.Source: §5 · p. 8
- Plastic and neuromodulated LSTMs outperform standard LSTMs on a benchmark language-modeling task.The authors identify testing these LSTMs on other common LSTM problems, such as forecasting, as future work.Source: §5 · p. 8
- Compared with L2RL, the approach allows learning through changes to connectivity as well as hidden-state changes.The authors suggest that an outer-loop meta-training algorithm could learn the weight-modification scheme itself, a possibility they connect to evolution and meta-meta-learning.Source: §5 · p. 8
- Further work includes multiple modulatory signals, more complex tasks, addressing interference in reward-modulated Hebbian learning, and allowing meta-training to design the architecture.These are presented as improvements needed to investigate the framework’s full potential.Source: §5 · p. 8
A.1.1 ADDING PLASTICITY TO LSTMS
The LSTM implementation adds plasticity to the data path through i_t, while leaving the control paths unmodified. Each plastic connection has a learned plasticity coefficient and an individual learned η.
A.1.1 ADDING PLASTICITY TO LSTMS
- Each LSTM node has four weighted recurrent paths: i_t, j_t, f_t, and o_t.The latter three paths control data flow, while i_t carries the actual data.Source: §A.1.1 · pp. 11–12
- Plasticity is introduced on the LSTM path through i_t, which carries the actual data, rather than on the control paths.The corresponding pre- and post-synaptic activations are h_{t−1} and i_t.Source: §A.1.1 · pp. 11–12
- A layer of size 200 has 40k plastic connections, each with its own η learned through backpropagation.The plasticity coefficients α_{i,j} are used in equation 1.Source: §A.1.1 · p. 12
A.1.2 ADDING NEUROMODULATION TO LSTMS
For neuromodulated LSTMs, each layer uses a dedicated neuron whose output replaces the individual η values for its plastic connections.
A.1.2 ADDING NEUROMODULATION TO LSTMS
- Each neuromodulated LSTM layer has a dedicated modulatory neuron whose output fans out to all plastic connections in that layer.Its input is the previous time-step’s layer activations, h_{t−1}.Source: §A.1.2 · p. 12
- Simple neuromodulation replaces η with the output M(t) of a network-computed neuron.The per-layer LSTM setup uses M(t) for the layer’s plastic connections.Source: §A.1.2 · p. 12
- Alternatives with one modulatory neuron per node or one for the whole network performed worse in preliminary experiments and were not evaluated further.Source: §A.1.2 · p. 12
A.2 MORE DETAILS FOR LANGUAGE MODELING EXPERIMENT
The language-modeling models were trained with SGD for 13 epochs, with minibatch hidden states carried forward. Grid search varied four learning and regularization hyperparameters.
A.2 MORE DETAILS FOR LANGUAGE MODELING EXPERIMENT
- All four models were trained with SGD for 13 epochs, starting at learning rate 1.0.LSTM hidden states started at zero, then each minibatch’s final states initialized the next minibatch.Source: §A.2 · p. 12
- Grid search varied the learning-rate decay factor from 0.25 to 0.4 in steps of 0.01 and the decay start epoch across 4, 5, and 6.Source: §A.2 · p. 12
- Grid search also varied the initial weight scale across 0.09–0.12 and the L2 penalty across 1e−2 to 1e−6.The weight-scale values were 0.09, 0.1, 0.11, and 0.12; the L2 values were 1e−2, 1e−3, 1e−4, 1e−5, and 1e−6.Source: §A.2 · p. 12
A.3 LARGE WORD-MODELLING NETWORK
The fixed-cue task distinguishes cases where non-modulated plasticity can learn from those where neuromodulated plasticity is reported to learn a task it cannot. The large word-model setup and plotted modulator dynamics describe implementation choices and run-to-run variation, respectively.
A.3 LARGE WORD-MODELLING NETWORK
- The large word-model setup used three stacked LSTMs and parameter-matched non-plastic and neuromodulated-plastic networks (24,221,600 vs 24,198,893 parameters).Plasticity coefficients were per-neuron, and a single modulator neuron generated neuron-specific rates; hyperparameters were not tuned because of computational constraints.Source: §A.3 · pp. 12–13
A.4 DYNAMICS OF NEUROMODULATION
- Modulator outputs showed complex, run-varying responses to perceived reward, including both positive and negative modulation patterns.The authors identify understanding how these dynamics support efficient within-episode learning as future work.Source: §A.4 · p. 13, Figure 3 · p. 14
A.5 CUE-REWARD ASSOCIATION TASK
- The cue-task advantage is suggested to be stronger when arbitrary high-dimensional cues must be memorized alongside their reward association.The paper calls for further investigation into which problems benefit most from neuromodulated plasticity.Source: §A.5 · p. 13
- With fixed four-bit cues, non-modulated plasticity learned the association task, though more slowly than neuromodulated plasticity.The figure reports medians and interquartile ranges over 10 runs; non-modulated plasticity succeeds in this simplified setting.Source: §A.5 · p. 13, Figure 4 · p. 15