PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradient Episodic Memory for Continual Learning

David Lopez-Paz, Marc'Aurelio Ranzato

arXiv:1706.08840v6cs.LGcs.AI

TL;DR

Continual-learning models must learn from sequential, mostly one-time examples without forgetting earlier tasks. This paper introduces evaluation metrics and Gradient Episodic Memory (GEM), which performs competitively while minimizing forgetting and enabling positive backward transfer across MNIST and CIFAR-100 variants.

  • Problem

    Sequential task learning with one-time examples causes catastrophic forgetting, limiting models’ retention of solutions to previously learned tasks.

  • Method

    The paper formalizes continual learning with accuracy and forward/backward-transfer metrics and proposes GEM, which uses episodic memory to avoid forgetting and favor positive backward transfer.

  • Results

    Across all datasets and methods, GEM performs similarly or better than the multimodal model, minimizes backward transfer, and exhibits negligible or positive forward transfer.

  • Takeaways & Limitations

    GEM provides competitive continual-learning performance while retaining knowledge of previous tasks and supporting beneficial backward transfer.

  • Takeaways & Limitations

    GEM does not exploit structured task descriptors, lacks advanced memory management, and requires one backward pass per task at each iteration.

Abstract

from arXiv · show

One major obstacle towards AI is the poor ability of models to solve new problems quicker, and without forgetting previously acquired knowledge. To better understand this issue, we study the problem of continual learning, where the model observes, once and one by one, examples concerning a sequence of tasks. First, we propose a set of metrics to evaluate models learning over a continuum of data. These metrics characterize models not only by their test accuracy, but also in terms of their ability to transfer knowledge across tasks. Second, we propose a model for continual learning, called Gradient Episodic Memory (GEM) that alleviates forgetting, while allowing beneficial transfer of knowledge to previous tasks. Our experiments on variants of the MNIST and CIFAR-100 datasets demonstrate the strong performance of GEM when compared to the state-of-the-art.

1 Introduction

The paper frames continual learning as learning from an ordered, non-iid stream of task-specific examples, unlike conventional iid supervised learning. It identifies catastrophic forgetting and transfer learning as central challenges and introduces metrics and GEM to address them.

  • Problem setting: Conventional ERM assumes iid data and often requires multiple passes over the training set, unlike the paper’s human-like learning setting.Humans observe ordered data, seldom see the same example twice, and encounter different learning tasks.
  • Challenges: Catastrophic forgetting occurs when learning new tasks harms performance on previously solved tasks.The paper identifies this as a defining challenge of continual learning.
  • Problem setting: Continual learning presents examples sequentially across tasks rather than as iid samples from one fixed distribution.A whole sequence of examples from the current task may appear before switching to the next task.
  • Challenges: Related tasks create transfer-learning opportunities that can accelerate learning new tasks and improve performance on old tasks.The potential benefits of transfer motivate evaluating learners beyond performance on an isolated task.
  • Contributions: The paper introduces metrics for evaluating continual learners and proposes GEM to alleviate forgetting while transferring beneficial knowledge to past tasks.It compares GEM with the state of the art in later experiments.

2 A Framework for Continual Learning

The framework models continual learning as prediction from a locally iid stream of task-labeled examples, with task descriptors enabling predictions across past, current, and future tasks. It evaluates both accuracy and knowledge transfer through a task-by-task performance matrix and fine-grained learning curves.

  • Data and prediction: The data continuum consists of triplets (x_i, t_i, y_i) containing a feature vector, task descriptor, and target vector, with the continuum assumed locally iid.Each triplet is associated with a task-specific feature and target distribution.
  • Data and prediction: The learner seeks a predictor f : X × T → Y that can predict targets for test pairs from past, current, or future tasks.The predictor may be queried at any time while observing examples sequentially.
  • Task descriptors: Task descriptors may be integer task identifiers or structured objects such as natural-language descriptions, creating an opportunity for zero-shot learning.Structured descriptors can expose relationships among tasks.
  • Training protocol: Training presents one example at a time or a small mini-batch, never repeats examples, and streams tasks sequentially without imposing a task order.A future task may coincide with a task observed in the past.
  • Evaluation metrics: Backward transfer measures effects on previous tasks, forward transfer measures effects on future tasks, and large negative backward transfer corresponds to catastrophic forgetting.Positive transfer improves performance on preceding or future tasks, respectively; forward transfer can support zero-shot learning.
  • Evaluation metrics: The matrix R records test accuracy on every task after each task is learned, while evaluating more often produces learning curves that capture learning speed.R_i,j is the test accuracy on task t_j after observing the last sample from task t_i, or after the i-th continuum example in the fine-grained setting.

3 Gradient of Episodic Memory (GEM)

GEM is a continual-learning model that stores episodic memories for past tasks and constrains gradient updates to prevent increases in previous-task losses. Unlike distillation-based approaches, GEM permits positive backward transfer and efficiently computes updates through a task-sized dual quadratic program.

  • Model and memory: GEM stores a subset of observed examples from each task in an episodic memory M_t to support continual learning.With a total memory budget M and known task count T, the method allocates m = M/T memories per task; otherwise, m can decrease as new tasks appear.
  • Gradient constraints: GEM treats previous-task losses as inequality constraints, preventing their increase while allowing them to decrease and thereby enabling positive backward transfer.This contrasts with distillation, which keeps past predictions invariant and makes positive backward transfer impossible.
  • Gradient constraints: GEM diagnoses potential loss increases using angles between past-task loss gradients and the proposed update, then projects violating gradients to the closest feasible update.The diagnosis assumes local linearity and representative episodic memories; satisfying all constraints makes previous-task loss increases unlikely.
  • Efficient optimization: The dual GEM quadratic program uses t − 1 variables rather than p network parameters, and recovers the projected update as ˜g = G⊤v⋆ + g.Adding a small constant γ ≥ 0 to v⋆ was found to bias projections toward beneficial backward transfer.

4 Experiments

Experiments evaluate GEM on three 20-task continual-learning benchmarks against five alternatives using accuracy and transfer metrics. GEM generally matches or exceeds strong baselines, limits forgetting, and benefits from episodic memory while using less computation.

  • Datasets and setup: Experiments use MNIST Permutations, MNIST Rotations, and Incremental CIFAR100, each with T = 20 sequential tasks and one pass through each example.MNIST tasks contain 1000 examples from 10 classes; CIFAR100 tasks contain 2500 examples from 5 classes.
  • Baselines: GEM is compared with single, independent per-task, multimodal, EWC, and iCARL predictors, with iCARL applicable only to CIFAR100.GEM, EWC, and iCARL use the single architecture plus episodic memory.
  • Main results: GEM performs similarly or better than the multimodal model, minimizes backward transfer, and exhibits negligible or positive forward transfer across datasets.Average accuracy, backward transfer, and forward transfer are summarized in Figure 1.
  • Main results: GEM exhibits minimal forgetting and positive backward transfer on CIFAR100 as test accuracy on the first task evolves throughout continual learning.This result is reported in Figure 1’s right panel.
  • Efficiency: GEM performs significantly better than EWC while using less computation by optimizing T = 20 variables instead of p = 1109240 parameters for CIFAR100.Its bottleneck is computing previous-task gradients at every learning iteration.
  • Memory and robustness: GEM’s final CIFAR100 accuracy increases with episodic-memory size, outperforms iCARL across a wide range, and matches the iid-learning oracle accuracy while minimizing negative BWT.On MNIST Rotations, repeated passes worsen forgetting for memory-less methods, whereas GEM remains aligned with the iid upper bound.

5 Related work

Continual learning studies sequential task learning while retaining and leveraging prior knowledge, building on implementations and theoretical work. Related approaches address catastrophic forgetting through task descriptors, modular networks, parameter regularization, or episodic memory, while multitask, transfer, and domain-adaptation settings assume simultaneous task access.

  • Continual learning: Continual learning requires retaining knowledge of past tasks while leveraging it to acquire new skills quickly.The literature includes both implementations and theoretical investigations of this setting.
  • Task descriptors: Task descriptors are related to reinforcement-learning systems that use task or goal descriptors as inputs, whereas CommAI emphasizes highly structured textual descriptors.The paper instead focuses on catastrophic forgetting.
  • Avoiding catastrophic forgetting: Catastrophic forgetting has been addressed by freezing early layers and cloning or fine-tuning later layers, as well as by modular networks sharing primitives across tasks.The freezing strategy was considered as the paper’s “independent” baseline.
  • Regularization approaches: The paper’s approach most closely relates to single-model regularization methods, including synaptic memory and episodic memory approaches that modify learning to protect prior-task knowledge.Synaptic-memory methods adjust learning rates to minimize changes to parameters important for previous tasks.
  • Adjacent learning settings: Multitask learning assumes simultaneous access to all tasks, while transfer learning and domain adaptation likewise assume multiple tasks are simultaneously available.These setups are related to continual learning but differ in task availability assumptions.

6 Conclusion

The paper formalizes continual learning with protocols measuring accuracy and forward/backward knowledge transfer, and introduces GEM to mitigate forgetting while promoting positive backward transfer. It also identifies structured task descriptors, advanced memory management, and computational cost as directions for improvement.

  • Contributions: The authors define training and evaluation protocols that assess accuracy and knowledge transfer forward and backward between tasks.These protocols are intended to evaluate models beyond test accuracy alone.
  • Contributions: GEM uses episodic memory to avoid forgetting and favor positive backward transfer.The conclusion describes GEM as a simple model with competitive performance against the state-of-the-art.
  • Limitations and future directions: GEM does not leverage structured task descriptors that could enable positive forward transfer, including zero-shot learning.Using task descriptors is identified as a potential improvement.
  • Limitations and future directions: The paper leaves advanced memory management, such as building coresets of tasks, unexplored.Coresets are cited as an example of an advanced memory-management approach.
  • Limitations and future directions: Each GEM iteration requires one backward pass per task, increasing computation time.Reducing this computational cost is identified as a research direction.

A Hyper-parameter Selection

The experiments used predefined hyper-parameter grids, with task-specific best values reported for MNIST rotations, MNIST permutations, and CIFAR-100 incremental learning.

  • A Hyper-parameter Selection: The best single-model learning rates were 0.003 for rotations, 0.03 for permutations, and 1.0 for CIFAR-100 incremental learning.These values are marked in parentheses within the single-method learning-rate grid.
  • A Hyper-parameter Selection: The hyper-parameter search covered single, independent, multimodal, EWC, and iCARL methods across the reported experiments.The supplied grid lists learning-rate choices and, for independent and EWC, finetuning or regularization settings.
  • A Hyper-parameter Selection: The independent-model grid selected learning rates of 0.1 for rotations, 0.03 for permutations, and 0.3 for CIFAR-100, with finetuning enabled across all three.The grid marks finetune as yes for rotations, permutations, and CIFAR-100.
  • A Hyper-parameter Selection: For EWC, the selected learning rates were 0.01 for rotations, 0.1 for permutations, and 1.0 for CIFAR-100, with regularization values of 1000, 3, and 1, respectively.The EWC grid marks the task-specific values in parentheses.

B Full experiments … B.3.5 Model GEM

This section defines the evaluation matrices used to report model performance across tasks and datasets. The matrices distinguish baseline accuracy before training from post-training task accuracies.

  • B Full experiments: Evaluation matrices R report each model’s performance on each dataset, with baseline accuracy shown separately before training begins.The first row contains baseline test accuracy b̄, while remaining entries record task accuracies after training successive tasks.
Loading 1706.08840v6…