PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning Representations for Continual Learning

Khurram Javed, Martha White

arXiv:1905.12588v2cs.LGcs.AIstat.ML

TL;DR

Continual learning systems must learn quickly from new data without forgetting prior knowledge, yet neural networks are vulnerable to both sample inefficiency and catastrophic interference. The paper introduces OML, which meta-learns representations by optimizing through online updates, and reports improved robustness, natural sparsity, and complementarity with existing continual-learning methods. It also identifies the need to learn these representations online without a separate meta-training phase.

  • Problem

    Neural networks struggle to learn efficiently from correlated online data while preserving previously learned knowledge.

  • Method

    OML meta-learns representations by directly optimizing through online updates, with the representation fixed while the prediction network adapts.

  • Results

    OML representations improve online updating, remain robust to forgetting, become naturally sparse, and combine effectively with existing continual-learning strategies.

  • Takeaways & Limitations

    A basic online update using OML representations is competitive with rehearsal-based continual-learning methods.

  • Takeaways & Limitations

    The method currently requires a separate offline meta-training phase; learning the representations online remains an open next step.

Abstract

from arXiv · show

A continual learning agent should be able to build on top of existing knowledge to learn on new data quickly while minimizing forgetting. Current intelligent systems based on neural network function approximators arguably do the opposite---they are highly prone to forgetting and rarely trained to facilitate future learning. One reason for this poor behavior is that they learn from a representation that is not explicitly trained for these two goals. In this paper, we propose OML, an objective that directly minimizes catastrophic interference by learning representations that accelerate future learning and are robust to forgetting under online updates in continual learning. We show that it is possible to learn naturally sparse representations that are more effective for online updating. Moreover, our algorithm is complementary to existing continual learning strategies, such as MER and GEM. Finally, we demonstrate that a basic online updating strategy on representations learned by OML is competitive with rehearsal based methods for continual learning. We release an implementation of our method at https://github.com/khurramjaved96/mrcl .

1 Introduction

Continual learning requires rapid learning from a continuous data stream while preserving prior knowledge, but neural systems struggle with both goals. OML addresses this by explicitly learning representations that reduce interference and support future learning.

  • Motivation: Continual learning agents must quickly learn predictions from new data while avoiding overwriting older knowledge.The setting is also called cumulative or lifelong learning.
  • Prior approaches: Existing approaches modify online updates, replay or generate samples, or use semi-distributed representations to address catastrophic interference.These strategies include parameter-importance regularization, rehearsal, and sparse representations.
  • OML: OML explicitly trains representations to avoid interference and promote future learning by optimizing through an online update.The meta-objective uses catastrophic interference as a training signal.
  • Contributions: OML-learned representations improve sequential regression and classification online, become naturally sparse, and enhance existing continual learning strategies.The paper reports complementarity with methods such as Meta Experience Replay.

2 Problem Formulation

The paper formulates continual learning as prediction from a correlated, potentially unending stream of input-target samples. Because online learning uses a single trajectory rather than iid samples, algorithms must account for sequence correlation.

  • Problem setting: A Continual Learning Prediction problem presents an agent with an unending stream of samples and requires continual prediction and learning.Samples are input-target pairs drawn over time.
  • Assumptions: Inputs may follow correlated sequences, while each target depends on the current input rather than past inputs.The input process can depend on previous observations, but targets are conditionally tied to the current input.
  • Objective: The objective minimizes expected prediction loss under the input density µ and target distribution p(Y|X).The model parameters W and θ are updated to minimize the continual-learning objective.
  • Online regime: Online learning is restricted to a single length-k correlated trajectory instead of direct iid sampling from p(x,y)=p(y|x)µ(x).This mismatch can make standard iid algorithms perform poorly.
  • Scope: The formulation covers online regression, incremental classification, history-dependent targets, and reinforcement-learning value prediction.History dependence can be represented by defining each input as the most recent m observations.

3 Meta-learning Representations for Continual Learning

OML meta-learns representations for continual learning by differentiating through sequential online updates. It separates representation learning from prediction learning so the representation can be fixed while the predictor adapts online.

  • Motivation: End-to-end neural networks are sample-inefficient on single correlated trajectories and suffer catastrophic interference during online learning.Prior meta-learning improves sample efficiency or adaptation but does not address catastrophic interference directly.
  • Architecture: The model composes a representation learning network φθ(X) with a prediction learning network gW, producing fW,θ(X)=gW(φθ(X)).θ is meta-learned and fixed at meta-test time, while gW is learned online from a trajectory.
  • OML objective: OML meta-learns the representation to maximize fast adaptation while minimizing interference under online updates.It is contrasted with MAML-Rep, which learns a representation using a few-shot-style objective without the same online-aware design.
  • Update function: The update function U maps parameters through k stochastic-gradient steps, producing Wt+k while retaining the representation parameters θ.Each update uses the current predictor parameters and one sequential sample.
  • Algorithm: Unlike MAML-Rep, OML uses one data point from the trajectory per update, allowing the objective to capture online effects such as forgetting.MAML-Rep instead uses the complete batch Sk for its inner updates.
  • Representation geometry: OML learns representations in which different solution manifolds can support appropriate generalization or avoid interference through geometric separation.Parallel manifolds can generalize appropriately, while orthogonal manifolds can avoid interference.

4 Evaluation

The evaluation tests whether OML-learned representations support rapid online learning while limiting interference across regression and classification streams. Across these experiments, OML remains robust to sequential updates, performs competitively against representation baselines, and learns sparse representations without dead neurons.

  • Evaluation setup: OML is evaluated by meta-training representations offline and measuring online prediction error on new continual-learning problems.Experiments include simulated incremental regression and sequential classification using real data.
  • Incremental Sine Waves: OML learns new sine-wave functions with a negligible increase in average MSE, whereas Pre-training and SR-NN show more interference.Pre-training error increases as more functions are learned, while SR-NN also suffers noticeably more interference than OML.
  • Split-Omniglot: On Split-Omniglot, OML learns the training trajectory almost perfectly with minimal forgetting and achieves better test generalization than the baselines.Only 15 samples per class are used for training and 5 for testing, contributing to the train–test performance difference.
  • Split-Omniglot: Under IID sampling, OML and MAML-Rep perform equally well, indicating comparable representation quality and an online-learning advantage for OML.The reported difference is attributed to OML representations being more suitable for incremental learning.
  • Representation analysis: OML produces highly sparse, well-distributed representations without dead neurons, using the full representation space.Pre-training has some dead neurons, while SR-NN can achieve sparsity by leaving a large part of representation space unused.

5 Improvements by Combining with Knowledge Retention Approaches

The experiments combine OML representations with existing continual-learning methods and compare them with standard online, approximate IID, and rehearsal-based updates. OML improves all tested algorithms, while OML with basic online updating is already competitive with rehearsal methods.

  • Experimental setup and findings: OML improves EWC, MER, and ER-Reservoir when these methods learn from OML representations.The comparison includes standard whole-network online learning and variants using pretrained fixed representations.
  • Experimental setup and findings: OML with basic Online updating outperforms continual-learning methods without OML.The reported Split-Omniglot evaluation covers 50 one-class tasks and 20 five-tasks-per-class settings.
  • Experimental setup and findings: OML representations outperform approximate IID sampling, indicating gains beyond merely reducing update correlation.Approximate IID training uses SGD on a random shuffling of each trajectory.
  • Experimental setup and findings: With OML representations, the performance gap between online and experience-replay algorithms is less pronounced than with other representations.Memory-based methods use a buffer of 200, and reported error margins are one standard deviation over 10 runs.

6 Conclusion and Discussion

The paper concludes that OML learns representations robust to interference during online updates and supportive of future learning. It also identifies online representation learning without a separate meta-training phase as an important next step.

  • Conclusion: OML learns representations robust to interference under online updates while promoting future learning.The authors report improved robustness to forgetting on highly correlated data streams.
  • Conclusion: Sparsity emerges in OML representations without explicit sparsity training.The conclusion presents sparsity as a property of the learned representations.
  • Conclusion: OML is complementary to existing continual-learning methods and can be combined with them for improvements over each approach alone.This conclusion concerns combinations with existing state-of-the-art continual-learning methods.
  • Discussion and future work: Learning OML representations online without a separate meta-training phase remains an important next step.The paper suggests periodically optimizing the representation on a recent data buffer as a plausible strategy.

A Discussion on the Connection to Few-Shot Meta-Learning

OML differs from gradient-based few-shot meta-learning by learning a fixed representation alongside an initialization and by optimizing correlated continual-learning trajectories for low interference and high transfer. The reported results also show that an RLN is important for effective continual learning.

  • Connection to few-shot meta-learning: OML updates only PLN during inner updates, whereas MAML updates all parameters.This changes the objective from finding an initialization with desired properties to learning an initialization and fixed representation.
  • Connection to few-shot meta-learning: OML samples trajectories for correlated inner updates and computes meta-loss on data representing the continual-learning problem.The optimization targets minimizing interference and maximizing transfer rather than only enabling quick adaptation.
  • Connection to few-shot meta-learning: OML learns the RLN and PLN initialization end-to-end with a single objective.The paper contrasts this with gradient-based meta-learning formulations focused on model initialization.
  • Connection to few-shot meta-learning: An RLN is extremely important for effective continual learning, while vanilla MAML with correlated trajectories performed poorly for online learning.This empirical observation motivates separating representation learning from the inner online updates.

B Reproducing Results

The paper releases code and pretrained OML models for the Split-Omniglot and Incremental Sine Waves experiments. Online-learning results are selected using a learning-rate sweep over validation trajectories.

  • Reproduction materials: Code and pretrained OML models are provided for Split-Omniglot and Incremental Sine Waves.The release also includes hyperparameters for both representation-learning experiments.
  • Reproduction procedure: Online-learning experiments sweep 10 learning rates for each method and report the best rate over 50 random trajectories.The best rate is selected using five validation trajectories.

B.1 Computing Infrastructure

OML representations were learned on a single V100 GPU, including deep-network meta-updates with roll-outs up to 400 steps.

  • OML learned representations in less than five hours for both regression and Omniglot experiments on a single V100 GPU.For smaller Omniglot roll-outs, good representations could be learned within an hour.

C Representations

The paper visualizes learned representations across methods and specifies an approximate inner-loop meta-training procedure for OML.

  • The learned representations are illustrated with additional samples in Figure 8 and an averaged best-performing SR-NN representation in Figure 10.The averaged SR-NN representation was excluded from Figure 6 because of space constraints.
  • Figure 8 compares representations for random input images across methods, including SR-NN models with 4.9% and 15% sparsity.The 4.9% SR-NN model matches OML's sparsity, while the 15% model achieves the best Split-Omniglot performance.
  • Figure 9 compares OML variants that update or fix θ at meta-test time and reports training error during meta-testing.The figure caption states that model initialization is not an effective bias for incremental learning.
  • Figure 10 shows the average activation map for the best-performing SR-NN with 15% sparsity, using light-to-dark green for activation magnitude.The scale ranges from 0 to the maximum activation.
  • Algorithm 3 samples tasks and training data, performs m inner gradient steps on W, then updates θ using test-set loss after truncated roll-outs.The procedure uses step sizes α and β and stops gradients through the predictor after the meta-update.

C.1 Why Learn an Encoder Instead of an Initialization

Learning an encoder performs significantly better than learning only an initialization because sparse representations make online updates less global and reduce interference with past knowledge.

  • Learning an encoder performs significantly better than learning only an initialization.The meta-learning optimization is also less sensitive to hyperparameters and converges faster when learning an encoder.
  • Gradient descent can greedily change initial-layer weights on highly correlated streams, causing those changes to interfere with past knowledge.This makes an initialization an ineffective inductive bias for incremental learning.
  • Sparse encoder representations make updates less global because weights connected to zero-valued features remain unchanged.This sparsity provides a mechanism for reducing interference during online updating.
Loading 1905.12588v2…