Source-linked AI summary
Continual Reinforcement Learning with Complex Synapses
Christos Kaplanis, Murray Shanahan, Claudia Clopath
TL;DR
Continual learning requires acquiring new memories while preserving old ones, and reinforcement-learning agents also face nonstationary experience within a task. This paper equips tabular and deep agents with a biologically inspired synaptic model and reports improved continual learning across sequential tasks and within a single task.
Problem
Continual learning requires acquiring new memories while preserving old ones, while reinforcement-learning agents also encounter nonstationary experience within a task.
Source: §1 · p. 1, §3.3 · p. 6Method
Results
The model helps agents learn across two sequential tasks and within a single task, reducing the need for an experience replay database.
Source: §1 · pp. 1–2Takeaways & Limitations
Different timescales of plasticity can correspond to improved behavioural memory over distinct timescales, even without prior knowledge of when data distributions change.
Source: §1 · p. 2This is a proof of concept; the model remains to be tested on more numerous and complex tasks and other architectures.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Unlike humans, who are capable of continual learning over their lifetimes, artificial neural networks have long been known to suffer from a phenomenon known as catastrophic forgetting, whereby new learning can lead to abrupt erasure of previously acquired knowledge. Whereas in a neural network the parameters are typically modelled as scalar values, an individual synapse in the brain comprises a complex network of interacting biochemical components that evolve at different timescales. In this paper, we show that by equipping tabular and deep reinforcement learning agents with a synaptic model that incorporates this biological complexity (Benna & Fusi, 2016), catastrophic forgetting can be mitigated at multiple timescales. In particular, we find that as well as enabling continual learning across sequential training of two simple tasks, it can also be used to overcome within-task forgetting by reducing the need for an experience replay database.
1. Introduction
Artificial neural networks can forget earlier learning, while replay-based mitigation has scaling and data-availability limits and does not explain lifelong memory. This paper tests whether synapses with plasticity across timescales can mitigate forgetting in reinforcement learning.
1. Introduction
- Because memories overlap across synapses, learning requires plasticity that is both responsive to new experience and stable enough to retain old memories—the stability-plasticity dilemma.The paper points to plasticity across timescales as a possible way to balance fast adaptation and retention.Source: §1 · p. 1
- Artificial networks can quickly forget earlier learning under nonstationary data, while replay databases scale poorly with task count and may become unavailable.In reinforcement learning, experience distributions can shift within a task as well as between tasks.Source: §1 · p. 1
- Benna-Fusi synapses support continual learning across two simple tasks and within one task, reducing replay need despite consolidation lacking advance knowledge of distribution-change timing.The authors report improved behavioral memory over distinct timescales.Source: §1 · pp. 1–2
- The paper tests whether a biologically plausible synaptic model can mitigate catastrophic forgetting in tabular and deep reinforcement learning.The work is framed as a proof of principle for adding biological complexity to agent parameters.Source: §1 · p. 1
2. Background
The background introduces a synaptic model that approximates memory retention across timescales with a finite chain of interacting variables, then reviews the reinforcement-learning methods used in the paper. It also explains why replay helps deep networks and how soft Q-learning modifies their objective.
2.1. The Benna-Fusi Model
- The Benna-Fusi model was derived to maximize memory signal-to-noise over time, and a finite chain of communicating variables approximates its power-law decay.The expected doubly logarithmic area under the SNR-versus-time curve is typically maximized by a kernel that decays as t^-1/2.Source: §2.1 · p. 2
- The visible synaptic weight is read from u1, while hidden chain variables regularize it using the history of modifications; larger beakers and narrower tubes correspond to longer timescales.The model uses exponentially increasing beaker areas and decreasing tube widths down the chain.Source: §2.1 · p. 2
2.2. Reinforcement Learning
- Reinforcement learning formalizes an agent's interaction with an environment as an MDP, with a policy chosen to maximize expected discounted future rewards.At each step, the agent observes a state, takes an action, receives a reward, and transitions to another state.Source: §2.2 · p. 2
2.2.1. Q-LEARNING
- Q-learning estimates state-action values from sampled experiences, updating them toward reward plus discounted next-state value to converge to the optimal value function.The paper uses naive Q(λ), which maintains eligibility traces for state-action pairs and can use them to modulate synaptic consolidation.Source: §2.2.1 · p. 3
2.2.2. DEEP Q NETWORKS
- Deep Q Networks approximate state-to-Q-value mappings, while an experience replay database randomizes stored experiences to decorrelate updates and limit forgetting of older experiences.The paper trains DQN agents with soft Q-learning, which maximizes policy entropy alongside reward; the authors report that it helped stabilize performance over time.Source: §2.2.2 · p. 3
3. Experiments
The experiments test whether Benna–Fusi synapses help tabular and deep RL agents retain useful information across task switches and within a single task. Across these settings, the model is associated with faster relearning or more stable performance, with outcomes differing by task.
3. Experiments
- The experiments were a proof of principle: tabular Q-learning tested the model’s mechanics, while deep RL tested continual learning across tasks and within one task.The varied model parameters jointly determined the range of timescales captured.Source: §3 · pp. 3–4
3.1.1. EXPERIMENTAL SETUP
- In the tabular setup, a grid-world agent alternated between opposite-corner rewards; Benna-Fusi chains added hidden Q-values that retain and regularize visible values over longer timescales.The modified variant scaled consolidation flow by the eligibility trace, focusing it on parameters being used and modified.Source: §3.1.1 · p. 4
3.1.2. RESULTS
- In the tabular grid world, Benna-Fusi agents switched between good policies faster than controls, and deeper hidden values retained earlier reward locations for later recall.The modified agent was quickest to relocate the reward after epoch changes.Source: §3.1.2 · p. 4, Figure 2 · p. 5
3.2.1. EXPERIMENTAL SETUP
- The deep-RL comparison alternated Cart-Pole and Catcher on a shared network, using task-specific gains and biases and clearing replay between epochs.The Benna-Fusi network used 30 variables per parameter; its updates were approximated with one Euler step per replay batch.Source: §3.2 · p. 5, §3.2.1 · p. 5
3.2.2. RESULTS
- Across alternating Cart-Pole and Catcher training, Benna-Fusi agents relearned tasks faster; controls often failed to regain Cart-Pole, while Benna-Fusi’s early Catcher convergence could be slower.The Benna-Fusi advantage persisted across tested epoch lengths and a three-task rotation.Source: §3.2.2 · p. 6, Figure 4 · p. 7
3.3.1. EXPERIMENTAL SETUP
- For the single-task test, smaller deep agents trained online on Cart-Pole and Catcher after every time step, without an experience replay database.The Benna-Fusi model used a larger first tube width to capture shorter timescales.Source: §3.3 · p. 6, §3.3.1 · pp. 6–7, Figure 4 · p. 7
3.3.2. RESULTS
- In online learning without replay, the Benna-Fusi agent usually mastered CartPole while controls failed to maintain a consistently good policy; all agents learned Catcher, with controls somewhat faster.The authors suggest that CartPole’s more nonstationary training distribution may make control agents more prone to forgetting, while Catcher’s broad state coverage and weaker temporal correlation may help controls learn successfully.Source: §3.3.2 · p. 7, Figure 5 · p. 7
- In CartPole, updates may overwrite Q-values for rarely visited tilted-pole states; a resulting poor action can expose the agent to unfamiliar states, while low exploration can hinder relearning.The authors report that simply keeping exploration from decaying to zero did not solve the problem and could make learning less stable; Benna-Fusi retained memory of actions in suboptimal situations while adapting to recent good-policy experience.Source: §3.3.2 · p. 7
4. Related Work
The paper situates Benna-Fusi consolidation among approaches that protect parameters using importance factors, and among models that represent memory across timescales. It also notes other continual-learning strategies that are orthogonal and potentially combinable.
4. Related Work
- Earlier consolidation methods penalize parameter changes using importance factors based on Fisher diagonals [Kirkpatrick et al., 2017], trajectory-wide loss contribution [Zenke et al., 2017], or output-norm sensitivity [Aljundi et al., 2017].These approaches are presented as quadratic-cost methods for protecting recall of previous tasks [Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2017].Source: §4 · pp. 7–8
- Benna-Fusi consolidation operates over multiple timescales, without derived importance factors or task-boundary knowledge, unlike earlier parameter-consolidation approaches.The authors note these properties may help when the timing and timescale of training-data changes are unknown.Source: §4 · p. 8
- Fast and slow weight components were previously used for rapid retrieval of old memories, while Adam momentum and soft target updates retain parameter values with exponentially declining memory [Kingma & Ba, 2014; Lillicrap et al., 2015].The paper contrasts that exponential decline with the Benna-Fusi model’s power-law decay and notes the earlier fast/slow model was tested only on simple random binary input-output matching.Source: §4 · p. 8
- Growing networks, multi-model networks, and generative replay are described as orthogonal continual-learning approaches that could be combined with this method [Rusu et al., 2016; Goodfellow et al., 2013; Fernando et al., 2017; Shin et al., 2017].Generative replay mimics old datasets that are no longer available.Source: §4 · p. 8
5. Conclusion
The paper presents a Benna-Fusi synaptic model for tabular and deep reinforcement-learning parameters as a proof of concept for mitigating forgetting. It identifies limited evaluation and weaker initial results on larger DQN tasks as important boundaries, and suggests neuromodulator-informed adaptation.
5. Conclusion
- Representing each tabular or deep reinforcement-learning parameter as interacting dynamical variables can help mitigate catastrophic forgetting over multiple timescales.The authors frame this as a proof of concept inspired by a computational model of biological synapses.Source: §5 · p. 8
- The work is a proof of concept, and initial larger-DQN experiments on Arcade Learning Environment tasks struggled to match control-agent performance.The reasons for that result remain to be investigated; the authors also call for evaluation with more and more complex tasks and other architectures.Source: §5 · p. 8
- The authors propose modulating flow between hidden variables using factors such as dopamine or parameter-importance measures to consolidate memory more selectively and efficiently.Dopamine is described as associated with reward prediction error and exposure to novel stimuli.Source: §5 · p. 8
A. Experimental details
The appendix identifies parameter tables for the tabular and deep Q-learning experiments and supplies notes about eligibility-trace scaling and update order.
A. Experimental details
- The appendix provides parameter-value tables for the tabular Q-learning and deep reinforcement-learning experiments.The supplied table captions identify their scope but do not provide individual parameter values.Source: Table S1 · p. 11, Table S2 · p. 11, A. Experimental details · p. 11
- The modified Benna-Fusi model scales the flow between beakers by a multiple of the eligibility trace.This is stated in an implementation footnote.Source: A. Experimental details · p. 11
- Updates were applied sequentially as in stochastic gradient descent rather than all at once as a minibatch.This is stated in an implementation footnote.Source: A. Experimental details · p. 11
B. Additional Experiments
Across the tested epoch lengths, the Benna-Fusi agent relearns tasks faster than or as quickly as the control agent. The comparison used a learning rate of 0.001, with fewer epochs for longer runs.
B.1. Varying Epoch Lengths
- For every tested epoch length, the Benna-Fusi agent was quicker or equally quick to relearn each task than the control agent.The figure compares time to relearn by agent and epoch length.Source: Figure S1 · p. 12
- Both agents used a learning rate of 0.001 in the epoch-length comparison.Runs with longer epochs were run for fewer epochs.Source: Figure S1 · p. 12
- The results indicate that the Benna-Fusi model improves memory across a range of timescales.This is the stated implication of the relearning-time comparison.Source: Figure S1 · p. 12
B.2. Three-task experiments
Across three rotated tasks, the Benna-Fusi agent showed the same remembering-time benefits as in the two-task setting. Cart-PoleLong was introduced as a new task because sequential control-agent training on it after Cart-Pole caused forgetting of the Cart-Pole policy.
B.2. Three-task experiments
- Cart-PoleLong doubled the pole length, and sequentially training a control agent on it after Cart-Pole caused catastrophic forgetting of the Cart-Pole policy.That forgetting was the criterion for treating Cart-PoleLong as a distinct task.Source: §B.2 · p. 12
- Training rotated through Cart-PoleLong, Catcher, and Cart-Pole over a total of 24 epochs.Figure S2 compares control and Benna-Fusi remembering times; each epoch comprised 20,000 episodes and both agents used a learning rate of 0.001.Source: §B.2 · p. 12, Figure S2 · p. 12
- By the end of 24 rotated epochs, the Benna-Fusi agent could recall each task faster than control, although it took a little longer to learn Catcher.The reported remembering-time pattern matched the benefits in the two-task setting.Source: Figure S2 · p. 12, §B.2 · p. 12
B.3. Varying size of replay database
The replay-database experiments compared control agents with different database sizes against an online Benna-Fusi agent on Cart-Pole. When the database was too small for control to achieve stable performance, the Benna-Fusi agent could do so.
B.3. Varying size of replay database
- With a replay database too small for stable control-agent performance, the online Benna-Fusi agent could attain stable Cart-Pole performance.The figure reports a 100-test-episode moving average; the control agents used η = 0.001.Source: Figure S3 · p. 13
- The comparison tested the Benna-Fusi agent online against control agents using different-sized experience replay databases on Cart-Pole.One experience was sampled from the database for training after every time step.Source: Figure S3 · p. 13
- When the replay database was too small, control agents could not attain stable performance on the task.The figure compares their 100-test-episode moving-average rewards with the online Benna-Fusi agent.Source: Figure S3 · p. 13
B.4. Catcher single task
On Catcher, the control agent learned faster, but both it and the Benna-Fusi agent ultimately learned a good policy. In the separate Cart-Pole exploration-value tests, none of the runs achieved stable performance.
B.4. Catcher single task
- The control agent learned Catcher faster, but both it and the Benna-Fusi agent ended with a good policy.Figure S4 compares their 100-test-episode moving-average rewards.Source: Figure S4 · p. 13
- Figure S4 compares the 100-test-episode moving-average reward per episode for the Benna-Fusi agent and the best control agent on Catcher.Both agents eventually learned a good policy, although control learned faster.Source: Figure S4 · p. 13
B.5. Varying final exploration value
- None of the Cart-Pole runs achieved good stable performance when epsilon was prevented from decaying below different minimum values.Figure S5 plots the 100-test-episode moving average of reward per episode.Source: Figure S5 · p. 14