Source-linked AI summary
Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, Gerald Tesauro
TL;DR
Continual learning must adapt neural networks to non-stationary streams without losing prior knowledge. The paper introduces Meta-Experience Replay, combining experience replay with optimization-based meta-learning to promote gradient alignment. MER outperforms strong baselines across supervised and non-stationary reinforcement-learning settings, including with small memory buffers.
Problem
Continual learning requires neural networks to learn from non-stationary streams while retaining previously acquired knowledge.
Method
Meta-Experience Replay combines experience replay with optimization-based meta-learning to make future gradients more likely to transfer and less likely to interfere.
Results
MER outperforms strong baselines on supervised continual-learning benchmarks and non-stationary reinforcement-learning environments.
Takeaways & Limitations
The transfer-interference perspective yields a general-purpose continual-learning solution that balances transfer and interference across past and future data.
Takeaways & Limitations
The experiments focus on achieving higher performance than baselines when each approach has only a small memory buffer.
Abstract
from arXiv · showhide
Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples. We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning. This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely. We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning. Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller.
1 SOLVING THE CONTINUAL LEARNING PROBLEM
Continual learning targets agents that adapt incrementally across changing task distributions while retaining prior knowledge. The paper reframes this challenge around weight sharing and gradient interactions in both temporal directions.
- Motivation: Continual learning exposes neural networks to sequential task experiences rather than an i.i.d. stationary distribution.The goal is incremental learning without supervised task discovery while adapting after every experience.
- Existing perspectives: Catastrophic-forgetting approaches prioritize stability by limiting weight sharing across experiences to preserve past knowledge.This reflects a focus on backward interference from current learning onto old memories.
- Existing perspectives: Stability-plasticity methods balance preserving past knowledge with rapidly learning current experiences, but often treat the temporal problem asymmetrically.Their simplified setting separates old memories from current learning and emphasizes reducing backward interference.
- Proposed perspective: The paper extends stability-plasticity to unbounded distributions by considering weight sharing and its trade-off in both forward and backward directions.This motivates a temporally symmetric transfer-interference perspective.
- Proposed perspective: Transfer-interference analysis treats future transfer as a central concern, not merely current learning and interference with past examples.The perspective links gradient alignment, weight sharing, and learning dynamics moving forward through time.
2 THE TRANSFER-INTERFERENCE TRADE-OFF FOR CONTINUAL LEARNING
The transfer-interference trade-off extends continual-learning analysis beyond protecting the past: parameter sharing must support future learning while avoiding interference. The paper therefore promotes gradient alignment across examples.
- Operational definitions: Transfer occurs when learning one example improves performance on another without repetition, whereas interference causes the other example to be unlearned.These operational notions are defined through interactions between gradients for distinct examples.
- Operational definitions: Overlapping parameter use increases potential transfer, while separating parameters reduces potential interference between examples.The trade-off connects the desired gradient relationship to the degree of weight sharing.
- Backward-focused formulations: Past stability-plasticity solutions primarily minimize interference from current learning onto aggregated past experiences.They generally do so by explicitly or implicitly reducing weight sharing.
- Temporally symmetric trade-off: Continual learning must preserve future learnability because future data and distributions are uncertain.This extends the temporal horizon beyond the usual past-versus-current framing.
- Temporally symmetric trade-off: The transfer-interference trade-off treats gradient alignment and weight sharing as interactions occurring both backward and forward in time.The paper proposes learning dynamics that promote alignment at each point in time.
3 A SYSTEM FOR LEARNING TO LEARN WITHOUT FORGETTING
The system combines experience replay with optimization-based meta-learning to approximate learning over all seen examples while shaping parameter updates toward transfer and away from interference. MER integrates replay sampling, Reptile-style updates, and current-example prioritization in an efficient SGD extension.
- Combining replay with meta-learning: MER’s objective encourages aligned gradients across examples, thereby favoring parameter sharing for transfer and separation where opposite gradients cause interference.The formulation combines ordinary losses with gradient-based regularization, while the relative weighting is controlled by α.
- Experience replay: Experience replay approximates optimization over the stationary distribution of all examples seen so far in a non-stationary stream.Reservoir sampling makes the memory resemble a stationary distribution over past examples, which supports random batch training.
- Experience replay: The current example is interleaved with replay samples so learning remains effective even when that example is not stored in memory.This preserves priority for the incoming experience while replay batches provide stabilization and past-example coverage.
- Combining replay with meta-learning: MER modifies Reptile to integrate experience replay, using sampling strategies that make the online objective approximately achievable.The method applies inner and outer Reptile-style meta-updates across replay batches and current examples.
- Controlling regularization: Sample-level batches provide fine-grained second-derivative information and allow more Reptile batches for a given number of replay samples.The design also supports stronger regularization while current-example prioritization prevents replay from dominating learning.
- Unique properties: The approach is an easy-to-implement, computationally efficient SGD extension whose regularization changes transfer and interference dynamics rather than only stabilizing replay.It combines the fast-learning machinery of optimization-based meta-learning with the past-data access of experience replay.
4 EVALUATION FOR SUPERVISED CONTINUAL LIFELONG LEARNING
MER is evaluated against continual-learning baselines on standard and highly non-stationary supervised benchmarks, including settings with limited episodic storage. It consistently improves retained learning performance, with larger advantages as non-stationarity increases.
- Evaluation setup: The evaluation compares online learning, independent predictors, task-specific inputs, EWC, GEM, and MER using retained accuracy as the main metric.Learning accuracy and backward transfer and interference provide additional views of adaptation and forgetting.
- Standard supervised benchmarks: MER adds significant retained-accuracy value over GEM on both MNIST Permutations and MNIST Rotations.The paper attributes this to balancing transfer and interference with respect to past and future data.
- Many-task non-stationarity: 100 tasks and 200 training examples per task make Many Permutations substantially more non-stationary than standard benchmarks, while Omniglot treats 50 alphabets as tasks.Omniglot contains 1623 characters and uses task-specific output layers.
- Many-task non-stationarity: MER and GEM achieve nearly double the performance of EWC and online learning on Many Permutations and Omniglot.The advantage of MER over GEM grows in increasingly non-stationary settings, with especially large gains on Omniglot.
- Many-task non-stationarity: Even with up to 100 gradient-descent steps per incoming example, no tested variant exceeded 5.5% retained accuracy in the extremely non-stationary setting.The passage also reports that MER handles an Omniglot memory size of 500, whereas GEM did not consistently converge there.
- Omniglot analysis: MER provides a stronger balance of Omniglot performance and computation than the baselines, with efficiency closer to EWC than GEM.It is reported to fit training data better than baseline models while maintaining this computational profile.
5 EVALUATION FOR CONTINUAL REINFORCEMENT LEARNING
MER is tested in non-stationary reinforcement-learning environments built from Catcher and Flappy Bird by changing game parameters across tasks. In Catcher, DQN-MER consistently outperforms standard DQN with replay and shows minimal or no forgetting of earlier tasks.
- Environment design: Catcher and Flappy Bird provide non-stationary environments by changing fixed game-dependent parameters across tasks.Agents are evaluated over time on all tasks using a standard DQN model.
- Environment design: Catcher tasks increase pellet velocity five times, while Flappy Bird tasks reduce pipe separation five times during training.These parameter changes create six sequential tasks in the Catcher experiment.
- Results: DQN-MER performs consistently better than standard DQN with an experience-replay buffer under non-stationary Catcher conditions.The Catcher experiment trains sequentially on six tasks for 25k frames each, reaching 150k frames.
- Results: After later Catcher tasks, standard DQN significantly forgets how to handle slowly moving pellets, whereas DQN-MER exhibits minimal or no forgetting.Later tasks subsume the earlier task because they use faster-moving pellets.
6 FURTHER ANALYSIS OF THE APPROACH
The further analysis tests whether MER changes gradient relationships and which components account for its gains across MNIST continual-learning benchmarks. MER consistently improves over experience replay, with meta-learning becoming more valuable as buffers shrink.
- MER is evaluated for shifts in gradient dot products between incoming examples and randomly sampled past examples across three MNIST-based benchmarks.The analysis tracks mean dot products over learning and compares MER with experience replay.
- Experience replay variants consistently outperform GEM on their own, while meta-learning added to experience replay consistently produces further gains.The comparison also finds similar performance when combining reservoir sampling and GEM-style buffering.
- Meta-learning provides increasing value for smaller memory buffers.
- All MER variants significantly improve on experience replay, while variants increasing the current example’s importance perform further better and comparably to each other.Algorithm 7 offers the best computational-efficiency/performance tradeoff in the MNIST experiments.
- Adaptive optimizers such as Adam and RMSProp do not close the gap between experience replay and MER, and can hurt generalization with smaller buffers.The analysis attributes this pattern to greater overfitting on the buffer compared with SGD.
7 CONCLUSION
The paper reframes continual learning as a transfer-interference trade-off and introduces MER to shape gradient interactions during experience replay. MER outperforms strong baselines across supervised continual-learning benchmarks and non-stationary reinforcement-learning environments.
- MER regularizes experience replay so incoming-example gradients are more likely to transfer and less likely to interfere with past examples.
- The paper reports MER as a general-purpose continual-learning solution that outperforms strong baselines in supervised benchmarks and non-stationary reinforcement-learning environments.
- Continual learning is defined as online learning from a non-stationary stream composed of successive stationary task distributions.
- The experiments primarily use a single-headed setting without requiring explicit task-change announcements.
B RELATION TO PAST WORK
The paper distinguishes its transfer-interference perspective from prior stability-plasticity, replay, meta-learning, and architectural approaches. Its central extension is to optimize weight sharing with respect to future as well as past and present gradients.
- Unlike prior stability-plasticity approaches focused on backward interference, this work extends the temporal horizon toward uncertain future learning.
- The transfer-interference trade-off seeks weight sharing that improves stability and learning efficacy on unseen data, not only performance on past examples.
- Earlier weight-sharing strategies address catastrophic forgetting or stability-plasticity through mechanisms including semi-distributed representations and dual-network models.
- The proposed perspective differs from few-shot meta-learning by targeting preservation of past knowledge and generalization to future data alongside faster learning.
- MER implicitly optimizes weight sharing and improves experience replay, while remaining conceptually compatible with options-based reinforcement-learning methods.
- GEM protects past-task performance by constraining incoming gradients against interference with stored task memories, making it a strong baseline for comparison.
G EXPERIENCE REPLAY ALGORITHMS
The paper develops experience-replay and MER variants that combine replayed examples with SGD and Reptile-style meta-updates. The derivation links MER’s effective gradient to gradient-alignment terms across replay batches and motivates reduced outer updates in practice.
- Experience Replay: Experience replay samples k −1 memory examples with the current example to form a minibatch, then updates parameters with SGD and reservoir sampling.
- Experience Replay: Task-aware replay balances sampled task losses in multi-headed settings, where each task has a distinct output space.
- Experience Replay: Both experience-replay variants are effective, but each performs significantly better than the other on some datasets and settings.
- Meta-Experience Replay: MER variants differ in how they form replay batches and increase the importance of the current example during Reptile-style updates.
- Meta-Experience Replay: MER’s effective gradient contains first-order gradient terms and higher-order Hessian-gradient interactions across replayed examples.
- Meta-Experience Replay: With β = 1, MER matches one Reptile loop over the total examples in its replay batches when the current example has the same proportion.
- Meta-Experience Replay: The experiments use β below 1 because higher-order terms from parameter mismatch across replay batches can disturb learning.
- Meta-Experience Replay: Reservoir sampling addresses the non-random order of non-stationary online data needed for the Reptile-based analysis.
J.1 HYPERPARAMETER SEARCH
The experiments searched hyperparameter grids for multiple continual-learning baselines and Meta-Experience Replay variants across benchmark tasks and buffer sizes.
- The grids varied learning rates, batch sizes, and meta-learning rates across benchmark-specific settings.
- Hyperparameter searches covered Experience Replay, Meta-Experience Replay, Independent Model Per Task, and Task Specific Input Layer.
- MER searches additionally varied the number of batches per example and, in one variant, the current example learning rate multiplier.
- Forward transfer was reported only for MNIST Rotations because the metric was considered inappropriate for MNIST Permutations and Omniglot.
L ABLATION EXPERIMENTS
The ablation experiments compare buffer handling and gradient-direction choices for GEM, finding no improvement from task-level gradient directions.
- The GEM ablation used reservoir sampling and treated all buffer contents as old data in its quadratic program.
- Task-level gradient directions did not improve the ablation results.
M REPRODUCIBILITY OF RESULTS
The reproducibility experiments tested whether MER’s reported gains depended on its larger hyperparameter search space, across additional seeds and machines. Results differed quantitatively but preserved the same qualitative model comparisons.
- The reproducibility study addressed concern that MER’s larger hyperparameter search space might artificially inflate improvements.
- Quantitative differences generalized across seeds and machines, while the qualitative conclusions about model performance remained approximately unchanged.
N.4 FURTHER COMMENTS ON CONTINUAL REINFORCEMENT LEARNING EVALUATION
The Flappy Bird evaluation tracks training performance across task switches in a non-stationary environment. DQN-MER approaches single-task asymptotic performance and transfers knowledge across tasks.
- Figure 5 averages performance over three validation episodes and three seed initializations, with vertical grid lines marking task switches.
- DQN-MER achieves approximately single-task asymptotic performance by the end of training for most tasks.
- DQN-MER shows positive forward transfer on remaining tasks and often matches a single-task DQN trained for 1 million steps.
- On final tasks, DQN-MER achieves better performance than single-task DQN trained on those tasks alone for 150k steps.