PaperlayerView exact revision on arXiv

Source-linked AI summary

Dark Experience for General Continual Learning: a Strong, Simple Baseline

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, Simone Calderara

arXiv:2004.07211v2stat.MLcs.LG

TL;DR

Continual Learning needs methods that handle streams without task boundaries, offline retraining, or unbounded resources. The paper proposes Dark Experience Replay, which rehearses stored examples while matching logits sampled during training. DER satisfies General Continual Learning requirements and achieves strong results across standard and MNIST-360 settings, with observed benefits in flatness and calibration.

  • Problem

    Continual Learning methods often overlook practical streams with blurred task boundaries, shifting distributions, bounded memory, and unavailable previous data.

  • Method

    Dark Experience Replay combines rehearsal and knowledge distillation by storing past inputs with logits sampled throughout the optimization trajectory, using reservoir sampling without task boundaries.

  • Results

    DER achieves state-of-the-art performance in almost all evaluated settings and qualifies as a state-of-the-art baseline for MNIST-360 General Continual Learning.

  • Takeaways & Limitations

    The paper recommends DER as a starting point for future studies on Continual Learning and General Continual Learning.

  • Takeaways & Limitations

    The analysis provides competitors requiring task boundaries with those boundaries during training for fair comparison, so the comparison includes an evaluation accommodation.

Abstract

from arXiv · show

Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through mixing rehearsal with knowledge distillation and regularization; our simple baseline, Dark Experience Replay, matches the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on both standard benchmarks and a novel GCL evaluation setting (MNIST-360), we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. We further explore the generalization capabilities of our objective, showing its regularization being beneficial beyond mere performance.

1 Introduction

Continual Learning addresses catastrophic forgetting in non-i.i.d. streams, but practical settings require bounded memory, no task boundaries, and no test-time task oracle. The paper develops General Continual Learning around these constraints and finds simple replay baselines surprisingly strong, with DER improving on ER.

  • Continual Learning trains on non-i.i.d. streams while limiting catastrophic forgetting, computational cost, and memory footprint.
  • Experience Replay consistently outperforms cutting-edge methods across the considered evaluation settings.
  • Practical General Continual Learning requires no task boundaries, no test-time oracle, and constant memory.
  • Experience Replay is among the few methods fully compliant with General Continual Learning, while MER and GSS suffer from very long running times.
  • Dark Experience Replay improves on ER with a simple formulation, using distillation, and achieves strong results in standard CL and the MNIST-360 GCL setting.

2 Related Work

Continual Learning methods address forgetting through rehearsal, knowledge distillation, regularization, or architectural changes. These approaches differ in how they replay data, preserve responses, constrain weights, or allocate parameters across tasks.

  • Experience Replay interleaves stored old samples with current data in training batches to address catastrophic forgetting.
  • MER reframes replay as meta-learning, while GSS selects examples for the memory buffer using gradients.
  • Knowledge-distillation methods preserve past behavior through teacher responses, with LwF using current-task examples and iCaRL replaying past exemplars.
  • Regularization methods constrain network-weight changes, whereas architectural methods assign distinct parameter sets to different tasks.
  • Progressive Neural Networks add new networks incrementally, producing a linearly growing memory requirement.

3 Dark Experience Replay

Dark Experience Replay combines rehearsal with knowledge distillation by storing past inputs and their logits, sampled throughout training without task boundaries. Its reservoir-based replay matches current outputs to stored logits, while DER++ adds labels to address sudden distribution shifts.

  • The DER objective fits the current task while encouraging the network to mimic its original responses on past samples.
  • DER stores past inputs with network logits rather than ground-truth labels, using these logits as the distillation targets.
  • Reservoir sampling selects random stream samples with equal storage probability, avoiding reliance on task boundaries or known stream length.
  • Logits sampled during optimization may be sub-optimal, yet the paper reports no performance harm and benefits for minimum flatness and calibration.
  • DER matches logits directly because probability-space squashing can lose information, and the replay expectation is approximated with buffer batches.
  • DER++ adds a label-based loss because reservoir replay may retain logits biased by earlier tasks after sudden distribution shifts.
  • Unlike LwF and task-boundary methods such as FDR, DER replays past exemplars with logits sampled throughout the optimization trajectory.

4 Experiments

The experiments compare DER and DER++ across standard continual-learning settings and introduce MNIST-360 to evaluate General Continual Learning without reliable task boundaries. DER methods achieve strong results across benchmarks, while DER++ combines complementary strengths in Task-IL.

  • Evaluation Protocol: The evaluation covers Task-IL, Class-IL, and Domain-IL, using standard protocols on CIFAR-10, Tiny ImageNet, and MNIST variants.Task-IL and Class-IL partition classes, whereas Domain-IL applies task-dependent input transformations without test-time task identities.
  • Evaluation Protocol: The comparison reports final average accuracy for DER, DER++, regularization, distillation, architectural, and rehearsal-based methods, alongside SGD and JOINT bounds.Experiments use SGD, validation-based hyperparameter selection, and data augmentation applied consistently to competitors.
  • Standard CL Results: DER and DER++ achieve state-of-the-art performance in almost all settings, with especially strong results in Domain-IL and Class-IL.The authors attribute Domain-IL performance to preserving similarity structure through soft targets, while Class-IL benefits from learning relations among classes across tasks.
  • Standard CL Results: DER performs on par with ER on average in Task-IL, whereas DER++ generally improves accuracy by combining the strengths of both methods.Task-IL compares classes within exclusive subsets, making an overall class vision less rewarding.
  • MNIST-360: MNIST-360 presents batches of two consecutive digits with increasing rotations, combining smooth rotation shifts and sharp class changes while showing each example once.The stream visits nine possible digit couples three times and excludes digit 9 because rotation makes 6 and 9 indistinguishable.

5 Model Analysis

The model analysis examines why DER benefits from logits sampled throughout training, comparing its optimization geometry, calibration, buffer utility, and processing time with related replay methods. DER and DER++ show flatter minima and lower calibration error than key baselines, while DER’s buffer also supports specialization to previously seen tasks.

  • Analysis Setup: The analysis compares DER and DER++ with FDR and ER to assess logits sampled throughout optimization against task-boundary logits and hard labels.The comparison targets the role of dark knowledge stored across the training trajectory.
  • Optimization Geometry: DER and DER++ produce the lowest empirical Fisher eigenvalue sums on S-CIFAR-10, indicating flatter minima than ER and FDR.The empirical Fisher Information Matrix is used as an approximation to the intractable Hessian; larger eigenvalues correspond to sharper minima.
  • Calibration: DER and DER++ achieve lower Expected Calibration Error than ER and FDR on TinyImageNet without posterior calibration methods.The reported reliability analysis indicates less overconfident predictions for models trained with Dark Experience.
  • Buffer Informativeness: DER’s buffer yields better accuracy than ER and FDR when a model is fine-tuned on samples from an already seen task and evaluated on the remaining test data.The analysis simulates new examples from an old distribution becoming available during the stream.
  • Training Time: The study also evaluates DER, DER++, and other rehearsal methods by wall-clock time at the end of the final task under matched computational conditions.This measures whether training can keep pace with the rate at which stream data arrive.

6 Conclusions

The paper introduces Dark Experience Replay as a simple knowledge-distillation baseline for continual learning and proposes MNIST-360 for General Continual Learning. Its experiments support DER as a starting point for future CL and GCL studies.

  • Conclusions: Dark Experience Replay leverages knowledge distillation to retain past experience and avoid catastrophic forgetting.The method is presented as a simple baseline for Continual Learning.
  • Conclusions: MNIST-360 provides an experimental protocol for General Continual Learning, where task boundaries are not cleanly available.The protocol is intended to support evaluation of streams with shifting domains and classes.
  • Conclusions: The authors recommend DER as a starting point for future studies because of its strong results across the evaluated CL and GCL settings and its observed properties.The conclusion connects the recommendation to the paper’s extensive experimental analysis.

Broader Impact

The paper presents reproducible comparisons and Dark Experience Replay as a simple, memory-limited baseline, while highlighting calibration and MNIST-360 as practical contributions.

  • The work includes a clear and extensive comparison of continual-learning methods across multiple datasets.
  • Dark Experience Replay is presented as a simple baseline that outperforms other methods while maintaining a limited memory footprint.
  • DER is better calibrated than Experience Replay, supporting its use as a starting point for continual-learning decision-making applications where overconfidence is detrimental.
  • MNIST-360 is introduced as an evaluation protocol adhering to the General Continual Learning scenario.
  • The paper discusses memory constraints and task-boundary requirements across distillation, rehearsal, architectural, and regularization methods.

A.2 No Task Boundaries

Several continual-learning methods rely on task boundaries or task identities, which conflicts with settings where such information is unavailable during training or inference.

  • Distillation, architectural, and regularization methods depend on task boundaries for teacher updates, model changes, or parameter snapshots.
  • GEM uses task identities to associate memorized inputs with their original tasks and construct separate constraints, while A-GEM uses one generic constraint.
  • Architectural methods and LWF require task labels before prediction, whereas regularization, rehearsal, and iCaRL can infer without task information.

B Details on the Implementation of MNIST-360

MNIST-360 creates a task-free stream of rotating MNIST digits by pairing consecutive classes, balancing their exemplars, and gradually changing rotations throughout training.

  • MNIST-360 presents digits 0 through 8 at increasing angles, with no task boundaries signaled to the evaluated method.
  • The stream uses nine consecutive-digit pairs, shown in order for R rounds, and forms batches from exemplars of two paired classes.
  • Each digit’s examples are split into groups that are assigned across the pseudo-tasks in which that digit appears.
  • Batches maintain equal proportions of the two active digits, and each pseudo-task ends after its designated exemplars have been shown rather than after a fixed batch count.
  • Digit-specific counters track exemplar exposure, while rotations increase across training and span an entire 2π angle for every digit.
  • Evaluation occurs after training because task boundaries are unavailable; test digits retain increasing rotations, and the order of digits does not require special batching.

C Accuracy vs. Memory Occupation

The paper relates method accuracy to total memory usage across standard and General Continual Learning experiments, emphasizing reduced memory footprints and reservoir-based rehearsal implementation.

  • Reduced memory footprint is treated as especially important for continual learning, with memory usage straightforward to assess for rehearsal methods through buffer size.
  • Figure 3 compares performance with memory allocation across experiments in Sections 4 and F.1.
  • Successive points for a method represent increasing buffer sizes, while methods with lower accuracy or excessive memory may be omitted.
  • The appendix provides the reservoir-sampling insertion algorithm, using the memory buffer, seen-example count, current example, and label as inputs.

E Details on the Implementation of iCaRL

The iCaRL implementation represents each class by the mean feature vector of its exemplars and classifies inputs by feature-distance comparisons. Its network-response formulation reproduces the same prediction without masking, while weight decay supports competitive performance.

  • iCaRL computes each class prototype µ_y as the average feature vector of its exemplars and represents an input with feature vector φ(x).
  • The classification rule selects the class whose prototype is closest to the input feature vector.
  • iCaRL’s network response h(x) is defined as the negative distance between φ(x) and the tensor of class prototypes µ.
  • Taking the argmax of h(x) without masking yields the same prediction as the distance-based rule in the Class-IL setting.
  • A weight-decay regularization term is included to make iCaRL competitive with the other evaluated approaches.

F Additional Results

The additional experiments examine protocol details, replay variants, single-epoch training, transfer metrics, and hyperparameter settings. They show why evaluation choices affect interpretation, particularly when limited optimization can confound forgetting with underfitting.

  • F.1 Sequential-MNIST: Sequential MNIST divides the MNIST Digits training set into five tasks, each introducing two new digits.
  • F.2 Additional Comparisons with Experience Replay: Figure 4 compares DER++ with Experience Replay variants while varying the number of samples retained for later replay.
  • F.2 Additional Comparisons with Experience Replay: In the low-memory regime, reservoir sampling weakens ER, whereas DER++ does not exhibit the same flaw.
  • F.3 Single-Epoch Setting: One-epoch training makes catastrophic forgetting difficult to disentangle from underfitting because the model receives few gradient steps.
  • F.3 Single-Epoch Setting: Single-epoch joint training underfits dramatically, limiting the interpretability of comparisons among continual-learning methods in that setting.
  • F.4 Forward and Backward Transfer: Forward Transfer, Backward Transfer, and Forgetting measure different aspects of transfer and performance degradation, but their interpretation depends on the continual-learning setting and metric assumptions.
  • F.4 Forward and Backward Transfer: Increasing regularization can preserve past knowledge while preventing adequate learning of the current task, producing high backward transfer but low final accuracy.
  • Hyperparameter Selection: The appendix reports selected hyperparameters and grid-search spaces for the evaluated methods, with batch size and minibatch size treated differently for MNIST-360.
Loading 2004.07211v2…