Learning to Continually Learn

Shawn Beaulieu; Lapo Frati; Thomas Miconi; Joel Lehman; Kenneth O. Stanley; Jeff Clune; Nick Cheney

Source-linked AI summary

Learning to Continually Learn

Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, Nick Cheney

arXiv:2002.09571v2Feb 21, 2020cs.LGcs.CVcs.NEstat.ML

TL;DR

Continual learning needs models to learn sequential tasks without catastrophically forgetting, but prior solutions are predominantly manually designed. This paper meta-learns input-conditioned gating for a prediction network, achieving strong continual-learning performance across 600 sequential classes while identifying remaining scope limits.

Problem
Artificial neural networks catastrophically forget sequentially learned tasks, and most existing remedies are manually designed.
Method
ANML meta-learns a neuromodulatory network that gates a prediction learning network’s activations, creating selective activation and indirectly selective plasticity.
Results
ANML achieves 63.8% accuracy on held-out images from 600 sequentially presented Omniglot classes over nearly 9,000 SGD updates.
Takeaways & Limitations
The results demonstrate that meta-learned selective activation can reduce catastrophic forgetting at an unprecedented continual-learning scale.
Takeaways & Limitations
ANML has so far been demonstrated only on simple Omniglot character-class tasks, leaving its scalability to more complex tasks unresolved.

Abstract

from arXiv · show

Continual lifelong learning requires an agent or model to learn many sequentially ordered tasks, building on previous knowledge without catastrophically forgetting it. Much work has gone towards preventing the default tendency of machine learning models to catastrophically forget, yet virtually all such work involves manually-designed solutions to the problem. We instead advocate meta-learning a solution to catastrophic forgetting, allowing AI to learn to continually learn. Inspired by neuromodulatory processes in the brain, we propose A Neuromodulated Meta-Learning Algorithm (ANML). It differentiates through a sequential learning process to meta-learn an activation-gating function that enables context-dependent selective activation within a deep neural network. Specifically, a neuromodulatory (NM) neural network gates the forward pass of another (otherwise normal) neural network called the prediction learning network (PLN). The NM network also thus indirectly controls selective plasticity (i.e. the backward pass of) the PLN. ANML enables continual learning without catastrophic forgetting at scale: it produces state-of-the-art continual learning performance, sequentially learning as many as 600 classes (over 9,000 SGD updates).

1 Introduction

Continual learning systems must acquire sequential tasks without catastrophic forgetting, yet existing remedies are largely manually designed and often scale poorly. ANML instead meta-learns context-dependent gating that produces selective activation and indirectly selective plasticity in a prediction network.

The challenge: Catastrophic forgetting rapidly degrades previously acquired capabilities when neural networks learn new sequential tasks.This difficulty is less prominent in conventional training with independently and identically distributed, interleaved data.
Limits of existing solutions: Replay reduces forgetting by mixing stored prior experiences with new data, but storage and computation become expensive and do not scale to many tasks.
Limits of existing solutions: Selective-plasticity methods limit parameter changes, but typically rely on manually designed heuristics.
Limits of existing solutions: Freezing old weights and adding capacity scales poorly, while frozen pathways cannot improve using information from other tasks.
Meta-learning perspective: Meta-learning directly optimizes the ability to continually learn, replacing manually specified solutions with an empirically learned algorithm.The approach follows a broader shift toward learning features, architectures, and hyperparameters rather than hand-designing them.
ANML: ANML meta-learns a neuromodulatory network that gates a prediction network’s activations according to input context.Gating creates selective activation and indirectly controls backward gradients, producing selective plasticity.
ANML: Unlike prior neuromodulation methods that directly adjust learning rates, ANML directly modulates activations to control learning indirectly.

2 The Problem Formulation

The problem is to learn many tasks sequentially while retaining high average performance across them. Because differentiating through the full learning sequence is infeasible at large task counts, the framework uses an approximate remember-set meta-loss during meta-training.

Objective: Continual learning seeks high average performance after sequentially learning a large number of tasks from a common domain.
Meta-learning setup: Meta-learning separates outer-loop meta-training, which improves inner-loop learning, from meta-testing, which evaluates the learned learner.The terminology distinguishes meta-train training/testing from meta-test training/testing.
Computational challenge: Differentiating through the entire sequence is infeasible for large n because of unstable gradients and hardware memory limits.The passage gives n = 600 as an example of the scale at which the naive approach fails.
Approximate meta-loss: The OML procedure approximates the desired meta-loss by evaluating newly learned classes together with a random remember set of previously seen classes.
Approximate meta-loss: The remember set is used only during meta-training, making it tractable to meta-train learning over many classes while each inner loop sees one class.
Experimental domain: Experiments use Omniglot, with 1,623 character classes; 963 support meta-training and 660 are held out for meta-testing.Meta-train training uses 20 labeled instances per class, while meta-testing uses 15 for training and 5 held out for evaluation.

3 A Neuromodulated Meta-Learning Algorithm (ANML)

ANML meta-learns a neuromodulatory network that gates a prediction network, enabling input-dependent selective activation and selective plasticity during continual learning. Its nested inner- and outer-loop procedure trains and evaluates this architecture across sequential tasks while retaining prior-task performance.

Architecture and mechanism: ANML meta-learns a context-dependent neuromodulatory network that gates a prediction network’s activations and indirectly controls its plasticity.The neuromodulatory network is meta-learned in the outer loop, while the prediction network is updated during inner-loop learning.
Inner-loop learning: During each inner loop, the prediction network undergoes 20 SGD updates on one Omniglot class while neuromodulatory gating modifies its forward activations.The gating also reduces gradients reaching subsets of prediction-network weights, producing selective plasticity.
Outer-loop meta-learning: The outer loop backpropagates a meta-loss through the 20 inner-loop updates and updates both networks’ initial weights with Adam.The meta-loss evaluates the newly learned class together with remembered instances from other meta-training classes.
Baseline controls: ANML is compared with OML and OML-OLFT controls, and OML-OLFT never outperforms ANML in the reported comparisons.The controls use approximately the same total parameter count, while OML-OLFT fine-tunes only one layer.

4 Results

ANML scales continual learning to 600 sequential classes while retaining performance on previously seen and held-out classes. Its neuromodulatory gating produces sparse, separable representations associated with reduced catastrophic forgetting and strong generalization.

Continual Learning at Scale: 600 sequential classes are learned across more than 9,000 SGD updates, extending the longest previously reported robust trajectory of 200 tasks.The paper tests whether continual learning can remain robust at substantially longer sequence lengths.
Continual Learning at Scale: ANML significantly outperforms OML on meta-test training trajectories of every tested length, while Scratch and Pretrain achieve below 3% on sufficiently long trajectories.Scratch falls below 3% at 50 or more classes, and Pretrain falls below 3% at 400 or more classes.
Generalization to Held-Out Classes: After training on 600 classes, ANML reaches 63.8% accuracy on held-out instances, versus 18.2% for OML and 44.2% for OML-OLFT.ANML significantly outperforms every other treatment for trajectories of any tested length.
Generalization to Held-Out Classes: ANML significantly outperforms chance on 99.3% of classes across the 600-task sequence, including classes unseen for hundreds of tasks.This result supports learning without catastrophic forgetting over long intervals between exposures.
Comparison with Oracles: At 600 classes, sequential ANML reaches 63.8%, exceeding the oracle versions of OML and other algorithms; only ANML-Oracle performs better, at 71%.The sequential ANML result is compared with treatments trained i.i.d. rather than sequentially.

5 DISCUSSION AND FUTURE WORK

ANML demonstrates strong continual-learning performance at 600 sequentially learned classes, while the authors identify remaining questions about transfer, harder tasks, and broader meta-learning combinations.

Results: ANML outperformed OML and even surpassed i.i.d.-trained OML-Oracle, suggesting meta-learned neuromodulation may extend beyond catastrophic-forgetting reduction.The authors propose studying forward and backward transfer in future work.
Results: 63.8% accuracy was achieved on held-out images from 600 Omniglot classes after nearly 9,000 sequential SGD updates, with one presentation of each image and 15 training instances per class.
Future work: A 10% performance drop between ANML-Oracle and sequentially trained ANML remains, leaving room for improvement.The authors also note that gating additional layers and reducing the parameter cost of fine-grained synapse-level gating could improve the method.
Future work: ANML has so far been demonstrated only on simple Omniglot character-class tasks, so its scaling to more complex tasks, reinforcement learning, and shifted task distributions remains open.
Future work: Future directions include combining ANML with recurrent meta-learning algorithms and differentiable Hebbian learning.
Broader implications: The authors frame ANML as evidence supporting meta-learning solutions to difficult machine-learning problems and the broader AI-generating-algorithms paradigm.

6 CONCLUSION

The paper introduces ANML, which meta-learns neuromodulatory gating to reduce catastrophic forgetting during continual learning. It demonstrates sequential learning across 600 classes and over 9,000 SGD updates, while identifying harder challenges for future evaluation.

Method: ANML directly meta-learns neuromodulatory parameters that gate a prediction network’s activations, producing selective activation and indirectly selective plasticity.
Results: ANML reduced catastrophic forgetting relative to traditional and state-of-the-art methods while learning up to 600 sequential classes over 9,000 SGD updates.
Conclusion: The authors describe ANML as a promising stepping stone toward improved continual-learning algorithms, while noting that harder challenges remain to be tested.

7 SUPPLEMENTARY INFORMATION

The supplementary experiments examine ANML and OML under additional meta-test epochs, increased plasticity, and 600-task evaluation. ANML retains substantially better accuracy and robustness than OML when layers are unfrozen, while its multi-epoch performance reaches 75.37% on held-out instances from 600 classes.

Multi-epoch meta-test evaluation: The meta-test protocol uses one epoch over only 15 images per class, leaving the effect of additional epochs on ANML’s advantage an open question; supplementary testing evaluates accuracy across multiple epochs.This setup was chosen to mimic online continual learning rather than traditional deep-learning training.
Multi-epoch meta-test evaluation: 75.37% accuracy was achieved by ANML on held-out meta-test instances from all 600 classes after sequential training on those classes.This was the highest accuracy reported for any treatment in the paper.
Layer plasticity: At 600 classes, OML-FT:PLN+RLN final and OML-Unlimited fell below 1% mean meta-test test accuracy, a 95% drop relative to original OML.Both variants allowed additional fine-tuning in OML’s representation-learning components.
Layer plasticity: At 600 classes, ANML-FT:PLN reached 31.5% accuracy and ANML-FT:PLN+NM out reached 24.7%, exceeding OML’s 18.19% accuracy.Their differences from OML were statistically significant, with p = 1.08 × 10^-23 and 1.43 × 10^-20, respectively.
Layer plasticity: The two modified ANML treatments dropped 50.6% and 61.2% relative to unmodified ANML’s 63.8% accuracy at 600 classes.These results quantify degradation when additional ANML components are made plastic during meta-test training.
Layer plasticity: ANML-Unlimited protected against forgetting through sequences of up to 100 classes before becoming statistically inferior to unmodified OML.OML variants permitting additional representation-network fine-tuning performed significantly worse than ANML-Unlimited for every tested trajectory length.