Using Hindsight to Anchor Past Knowledge in Continual Learning

Arslan Chaudhry; Albert Gordo; Puneet K. Dokania; Philip Torr; David Lopez-Paz

Source-linked AI summary

Using Hindsight to Anchor Past Knowledge in Continual Learning

Arslan Chaudhry, Albert Gordo, Puneet K. Dokania, Philip Torr, David Lopez-Paz

arXiv:2002.08165v2Feb 19, 2020cs.LGstat.ML

TL;DR

Continual learning must handle sequential, changing tasks while retaining prior knowledge under a small episodic-memory budget. HAL adds hindsight-learned anchors and bilevel optimization to experience replay, and experiments report improved accuracy and reduced forgetting across benchmarks and memory sizes.

Problem
Continual learning requires models to learn sequential tasks in a single-pass, limited-memory setting while avoiding catastrophic forgetting of prior knowledge.
Method
HAL complements experience replay with one gradient-optimized anchor per class per task and bilevel updates that preserve predictions on past-task anchors.
Results
Up to 7.5% higher accuracy and almost 23% less forgetting than experience replay were reported across four supervised benchmarks, with gains for 1–5 examples per class per task.
Takeaways & Limitations
Anchoring complements and improves experience-replay methods across the evaluated benchmarks and episodic-memory sizes.
Takeaways & Limitations
The ideal anchor objective would require entire past-task distributions and future-task access, so HAL approximates it using episodic memory and hindsight updates.

Abstract

from arXiv · show

In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a new objective that we call anchoring, where the learner uses bilevel optimization to update its knowledge on the current task, while keeping intact the predictions on some anchor points of past tasks. These anchor points are learned using gradient-based optimization to maximize forgetting, which is approximated by fine-tuning the currently trained model on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

1 Introduction

Continual learning addresses models learning sequential tasks under limited memory while remaining responsive to evaluations on past tasks. HAL complements experience replay with hindsight-learned anchors and bilevel updates, improving accuracy and forgetting across benchmarks.

Problem: Continual learning presents sequential tasks in a single-pass setting where only a few examples can be stored in episodic memory.Predictions may be requested on examples from any task throughout the model’s lifetime.
Problem: Catastrophic forgetting causes models to quickly lose previously acquired knowledge as they learn new tasks.The challenge is especially pronounced for deep models whose parameters are updated after each example.
Hindsight Anchor Learning: HAL uses bilevel optimization to regularize memory-based continual learning with one gradient-optimized anchor per class per task.Anchors maximize an approximation of forgetting by evaluating a temporary predictor fine-tuned on episodic memories of past tasks.
Hindsight Anchor Learning: HAL keeps predictions invariant at anchors near the classifier’s decision boundary while minimizing current mini-batch loss.The method targets points described as easiest to forget during future updates.
Results: Across four supervised continual-learning benchmarks, HAL improves accuracy by up to 7.5% and reduces forgetting by almost 23% over experience replay.The reported gains hold for episodic memories containing 1 to 5 examples per class per task.

2 Continual learning setup

The continual-learning setup models an ordered, unknown-length stream of task-labeled iid examples and restricts the learner to a small episodic memory. Evaluation uses final average accuracy and final maximum forgetting, with initial tasks reserved for hyperparameter validation.

Learning stream: Each observation is a triplet (x_i, y_i, t_i) containing an input, target, and task identifier drawn iid from a task distribution.Tasks are experienced in order, and the total number of tasks is unknown in advance.
Learning objective: The learner estimates a task-aware predictor f_θ = (w ◦ φ) composed of a feature extractor and classifier to minimize multi-task error.The predictor maps inputs and task identifiers to outputs.
Memory constraint: The episodic memory stores the last m observed examples per class for each experienced task, with m ∈ {1, 3, 5}.This yields an O(t) memory footprint as the number of tasks grows.
Evaluation: Final average accuracy measures test performance across tasks after the continual-learning experience has finished.The metric averages a task’s accuracy after the model has completed all tasks.
Evaluation: Final maximum forgetting measures each task’s performance decrease from its peak accuracy to its final accuracy.The metric tracks the largest such decrease across the task history.
Evaluation: The first k < T tasks are used to cross-validate hyperparameters and excluded from final accuracy and forgetting calculations.This follows the stated evaluation protocol for the considered algorithms.

3 Hindsight Anchor Learning (HAL)

HAL complements experience replay with anchors learned to expose future forgetting and a bilevel update that preserves predictions on those anchors. It approximates future forgetting by fine-tuning on episodic memory, while constraining anchors toward mean task embeddings.

Experience replay: Experience replay updates the model using current-task and randomly sampled episodic-memory minibatches.The replay minibatch is combined with the current minibatch through the average loss.
Bilevel anchoring: HAL’s nested optimization trades off minimizing replay loss against preserving predictions at anchors for all past tasks.The update first computes a temporary replay-based parameter vector, then performs the actual conservative update.
Anchor construction: Anchors are constructed by gradient ascent in image space to maximize an approximation of forgetting for a chosen class label.This replaces direct optimization over the entire task distribution with k gradient-ascent updates from random initialization.
Anchor regularization: A mean embedding regularizer pushes each anchor embedding toward the running mean embedding of its task.The feature extractor’s task mean embedding is updated as a running average after each minibatch.
HAL procedure: HAL learns one anchor per class per task and combines these anchors with experience replay throughout continual learning.After each task, the algorithm fine-tunes on memory, constructs anchors, and discards the task mean embedding.
Hindsight approximation: Future forgetting is approximated by fine-tuning the current model for one epoch on episodic memory, producing temporary parameters used to update anchors.This hindsight estimate substitutes past-task fine-tuning for access to the final model after future tasks.

4 Experiments

Experiments across four supervised continual-learning benchmarks evaluate HAL against replay, regularization, and other baselines under small episodic memories. HAL achieves strong accuracy and forgetting results, remains effective as memory grows, and trades additional memory and computation for improved performance.

Experimental setup: Experiments use Permuted MNIST, Rotated MNIST, split CIFAR-100, and split miniImageNet, with common architectures and supervised continual-learning evaluation.The first three tasks are used for hyperparameter optimization and later discarded from evaluation.
Main results: With one example per class per task, HAL achieves maximum Accuracy and minimal Forgetting across all benchmarks, excluding the two oracle baselines.Results are averaged over five runs with different random seeds.
Main results: Memory-based methods substantially outperform regularization approaches under the single-epoch setup, while ICARL performs poorly relative to direct experience replay.The paper attributes EWC’s weakness to its need for multiple passes and VCL’s weakness to noisy posterior estimation.
Memory-size study: HAL consistently improves accuracy over ER-Ring and other baselines as episodic memory increases from 1 to 5 examples per class per task.Total memory ranges from 200 to 1000 examples for MNIST and from 85 to 425 for CIFAR and ImageNet.
Efficiency: HAL adds overhead over experience replay but is substantially faster than MER and is reported to provide the best efficiency–performance tradeoff.The additional memory stores task anchors, which the authors report as more effective than additional replay samples.
Memory-size study: For equal memory in bytes, HAL outperforms experience replay using twice as many stored examples, despite requiring extra memory for anchors.ER-Ring and HAL store one example per class per task, whereas ER-Ring-2|M| stores two.
Ablation study: HAL anchors lie near classifier decision boundaries, and HAL performs close to a noncontinual oracle when forgetting is quantified with access to future tasks.This supports the effectiveness of the replay-based approximation used to learn anchors.

5 Related work

Continual learning methods use regularization, modular architectures, or episodic memory to manage changing task streams without retaining complete datasets. HAL extends experience replay with hindsight-learned anchors that target forgotten, decision-boundary-near examples.

Continual learning addresses changing task sequences while storing only a small episodic memory rather than complete task datasets.
Regularization methods constrain updates to parameters important for past tasks but become brittle under feature drift and typically require multiple passes over each task.
Modular methods allocate different prediction-function components to tasks but do not scale well because module architectures require combinatorial search.
In feature space, HAL anchors lie near class-cluster edges or decision boundaries, while in image space they remain within class data clusters.
Episodic-memory methods replay stored examples, preserve predictions through distillation or loss constraints, and include experience replay that directly optimizes memory loss.
The paper focuses on task-based continual learning and assumes a task structure, distinguishing it from task-free learning on undivided streams.
HAL learns one anchor per class and past task by hindsight optimization of forgetting, unlike dataset distillation, which requires access to an entire task dataset.

6 Conclusion

HAL introduces hindsight anchoring as a bilevel objective for continual learning. It learns task-class anchors that maximize anticipated forgetting and keeps their predictions invariant, improving experience-replay methods across standard benchmarks.

HAL learns one anchor point per class per task whose prediction remains invariant through nested optimization.
The method uses temporary updates on episodic memories to simulate future-task forgetting in hindsight and optimize anchors toward that forgetting.
HAL complements experience replay and achieves state-of-the-art performance on four standard continual-learning benchmarks.

Appendix

The appendix documents the approximate anchoring update, additional experiments, the hyper-parameter grid, and HAL pseudocode.

Appendix A describes the approximate update performed by the anchoring objective.
The appendix reports additional experimental results and provides the hyper-parameter grid used in the study.
The appendix includes pseudocode for HAL.

A Approximate Update of Anchoring Objective

The appendix approximates the nested anchoring update with a Taylor expansion. The resulting gradient combines the temporary-update gradients with Hessian-gradient terms and resembles second-order meta-learning formulations.

The approximation focuses on the regularization component of the nested anchoring objective and denotes its gradient as g_anc.
The derivation defines θ_0 before the temporary update and identifies ℓ_ce and ℓ_L2 as cross-entropy and L2 losses.
The temporary update is represented by U_0 = θ_0 − αg_0, with θ_1 defined as the resulting temporary parameter vector.
The chain-rule derivation and first-order Taylor expansion produce the approximate anchoring gradient.
g_anc = g_1 − α(H_1 · g_0 + H_0 · g_1) + O(α^2), a form described as similar to the second-order MAML gradient.

B More Results

HAL preserves predictor performance more effectively than other baselines as new tasks are learned. Episodic-memory experiments also vary memory size from 3 to 5 examples per class per task.

B More Results: HAL preserves predictor performance more effectively than other baselines on Permuted MNIST and Split CIFAR.Figure 3 provides a fine-grained analysis of average accuracy as new tasks are learned.
B More Results: Memory sizes range from 600 to 1000 examples for MNIST and from 255 to 425 for CIFAR and ImageNet.These sizes result from storing 3 to 5 examples per class per task.

C Hyper-parameter Selection

The experiments use benchmark-specific hyperparameter grids and compare methods with episodic memories across accuracy and forgetting results. The supplied selection passages also identify the memory-size settings used for these comparisons.

C Hyper-parameter Selection: Hyperparameter selection reports the best values for different benchmarks in parentheses.The section introduces a grid of considered hyperparameters rather than a single shared setting.
C Hyper-parameter Selection: Accuracy and forgetting are evaluated for methods using episodic memory at large memory sizes.Tables 5 and 6 average results over five runs with different random seeds.
C Hyper-parameter Selection: HAL’s tested learning-rate grid includes 0.003, 0.01, 0.03, 0.1, 0.3, and 1.0, with benchmark-specific selections.The listed 0.03 setting applies to CIFAR and miniImageNet, while 0.1 applies to MNIST permutations and rotations.

D Hyperparameter Sensitivity

HAL is reported as insensitive to the choice of hyperparameters. Its training procedure combines current-task batches, episodic-memory replay, temporary updates, running mean embeddings, and anchor generation.

D Hyperparameter Sensitivity: HAL is not sensitive to the choice of hyperparameters in the reported Table 7 evaluation.Each tested hyperparameter is varied while all others remain at their optimal values from the appendix.
D Hyperparameter Sensitivity: HAL samples current-task and episodic-memory batches before applying a temporary parameter update on their union.The pseudocode denotes this update as θ̃ ← θ − α · ∇_θ ℓ(B ∪ B_M).
D Hyperparameter Sensitivity: The algorithm updates a running average of mean embeddings and adds current batches to a ring-buffer memory.These operations precede anchor generation for the current task.
D Hyperparameter Sensitivity: Anchor generation fine-tunes the current parameters on episodic memory, stores the updated parameters, and performs gradient-ascent steps on initialized task anchors.The pseudocode labels the ascent objective as maximizing forgetting.