Source-linked AI summary
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, Zsolt Kira
TL;DR
Continual-learning evaluations use diverse scenarios and inconsistent comparison conditions, making it difficult to assess methods fairly. This paper categorizes the scenarios and evaluates methods with a uniform framework and strong baselines, finding that simple methods can perform similarly to mainstream approaches. It also identifies scenario difficulty patterns and argues for harder, longer task sequences.
Problem
Continual-learning evaluations vary in scenarios, methodologies, baseline tuning, and task-sequence length, limiting meaningful comparison.
Method
The paper categorizes task differences into canonical scenarios and systematically evaluates state-of-the-art methods alongside Adagrad, L2 regularization, and naive rehearsal baselines.
Results
Adagrad and L2 outperform online EWC and match SI, while naive rehearsal performs similarly to state-of-the-art methods under the same space overhead.
Takeaways & Limitations
Simple baselines can be surprisingly effective, while scenario design should better distinguish methods and include harder evaluation settings.
Takeaways & Limitations
Strong baseline performance does not mean continual learning is solved, because harder datasets and longer task sequences can increase difficulty.
Abstract
from arXiv · showhide
Continual learning has received a great deal of attention recently with several approaches being proposed. However, evaluations involve a diverse set of scenarios making meaningful comparison difficult. This work provides a systematic categorization of the scenarios and evaluates them within a consistent framework including strong baselines and state-of-the-art methods. The results provide an understanding of the relative difficulty of the scenarios and that simple baselines (Adagrad, L2 regularization, and naive rehearsal strategies) can surprisingly achieve similar performance to current mainstream methods. We conclude with several suggestions for creating harder evaluation scenarios and future research directions. The code is available at https://github.com/GT-RIPL/Continual-Learning-Benchmark
1 Introduction
Continual learning addresses adaptation to changing data while retaining performance on earlier tasks, but evaluations vary substantially in methodology and difficulty. The paper categorizes these scenarios, evaluates them uniformly, and argues that carefully tuned simple baselines can match or exceed state-of-the-art methods.
- Motivation: Continual learning updates models incrementally as data streams in, but gradient-based updates can cause catastrophic forgetting of earlier tasks.Parameter interference after learning a new task can make the model forget how to solve old tasks.
- Evaluation gap: Existing evaluations lack uniformity, comparable baseline tuning, and sufficiently difficult task sequences.The paper specifically identifies short task queues as a limitation of current evaluations.
- Contributions: The paper categorizes experimental methodologies into canonical settings and compares their relative difficulty within a uniform scenario-generation framework.The framework is designed to support systematic evaluation of current state-of-the-art methods.
- Contributions: The paper demonstrates that Adagrad, L2 regularization, and naive rehearsal can achieve comparable or better performance than current state-of-the-art methods when used properly.The released PyTorch framework is intended to enable fair and uniform evaluation.
2 Generating Task Sequences for Evaluating Continual Learning
Continual-learning task sequences are commonly generated by permuting or splitting datasets, but these constructions differ in task identity, input and label distributions, and output representations. The paper organizes these differences into three canonical scenarios for consistent comparison.
- Existing methodologies: Permuted MNIST changes pixel permutations across sequential ten-digit classification tasks, while Split MNIST divides digits into five two-class tasks.Split MNIST can use multiple output heads with known task identity or a single head predicting all digits.
- Categorization: The categorization describes task differences through shifting input/output distributions and whether tasks share representation spaces.Tasks are represented as T1 = (X1, Y1, t1) and T2 = (X2, Y2, t2), with a sequence {T1, T2, ..., Tk}.
- Incremental Domain Learning: Incremental domain learning changes the marginal input distribution P(X) while continual learning retains old-task performance as well as learning the new task.This distinguishes continual learning from domain adaptation, which considers performance on the new task.
- Incremental Class Learning: Incremental class learning assigns each task an exclusive subset of classes while keeping all labels in one naming space and using outputs for the full class set.A ten-permutation Permuted MNIST construction can produce 10 × 10 total classes when each permuted digit becomes a new class.
- Incremental Task Learning: Incremental task learning uses disjoint output spaces whose dimensions or semantic meanings may differ, requiring task-specific output components selected by task identity.The old and new tasks may even be classification and regression problems, respectively.
3 Experiments
Experiments use a common MNIST-based setup and matched comparison conditions, then evaluate sequential-training, regularization, rehearsal, and generative-rehearsal approaches under controlled memory budgets.
- Configuration: The experiments generate three continual-learning scenarios from MNIST using splitting, with 60k training images and 10k test images after zero-padding and normalization.No data augmentation beyond the stated preprocessing is applied.
- Configuration: All methods use the same two-hidden-layer perceptron with 400 nodes per layer, ReLU activations, and a softmax output layer.Models are trained for four epochs per task with mini-batch size 128 and Adam by default.
- Baselines: The baselines include sequential training with Adam, SGD, or Adagrad, L2 regularization, and naive rehearsal.L2 regularization tunes one coefficient by grid search, while sequential baselines use specified optimizer settings.
- Baselines: Naive rehearsal constructs each mini-batch from equal amounts of new and replayed data using a fixed buffer matched to the memory overhead of online EWC and SI.The uncompressed buffer stores approximately 1.1k images, while 8-bit compression increases capacity to 4.4k images.
- Comparison methods: Popular regularization, gradient-based, and generative-rehearsal methods are compared with tuned hyperparameters under controlled static memory overhead.RtF is the exception, using half the overhead of DGR.
4 Results and Discussion
Across the evaluated scenarios, simple baselines often match mainstream continual-learning methods, while scenario construction and task-boundary assumptions strongly affect difficulty and applicability.
- Adagrad and L2 outperform online EWC and perform similarly to SI, suggesting Adagrad is a strong baseline for these experiments.The authors attribute Adagrad’s performance possibly to smaller updates for parameters frequently used by past tasks.
- Naive rehearsal matches state-of-the-art methods at the same space overhead and substantially outperforms online EWC and SI, especially for incremental class learning.
- The scenarios differ in difficulty: incremental task learning is easiest, incremental class learning is harder than incremental domain learning.
- Permutation-based scenarios generally yield comparable or higher baseline performance than splitting-based scenarios despite having more classes and tasks, indicating simpler evaluations.
- EWC’s performance is usually not significantly different from Adagrad, but EWC, SI, and MAS require task boundaries that are often unavailable in real-world settings.
- Strong baseline performance does not solve continual learning: longer task sequences cause regularization methods to degrade, motivating harder datasets and task-identity-free scenarios.
A Permuted MNIST Experiments
Permuted MNIST generates continual-learning scenarios through pixel permutations and supports ten-task sequences, with experiments comparing initialization choices and methods under controlled evaluation settings.
- Permuted MNIST preserves ten-digit classification while assigning each task a different pixel permutation; the task-sequence length equals the number of permutations.
- The experiment uses a neural network with 1000 nodes in each hidden layer, increasing the space overhead of online EWC and SI.The reported parameter overhead is approximately 4,048,000 parameters, equivalent to buffers of 4k images in naive rehearsal or 16k compressed images.
- Regularization coefficients for L2, EWC, online EWC, SI, and MAS are selected by grid search, while reported results average ten runs with random initialization.Results for LwF, DGR, and RtF come from prior work using the same architecture and training procedure.
- Pre-allocating all output nodes makes incremental class learning easier because every node receives classification loss from the beginning.
- Table 3 reports average accuracy on all seen tasks for Permuted MNIST, with memory mechanisms separated and static memory overhead controlled across listed methods.Each value is averaged over ten runs.
- The Permuted MNIST incremental-class setting is much easier when the total number of classes is known, unlike the comparable Split MNIST variants.
B Lengthened Task Queue Experiments
The experiments lengthen continual-learning task queues across multiple datasets and compare MLP and CNN models with regularization methods and optimization baselines. Results show that SI and Online EWC are sensitive to regularization and initialization choices, while Adagrad performs similarly without hyperparameter search.
- Model and method comparison: Figure 3 compares SI and Online EWC under best and worst hyperparameter selections against two additional optimization methods for both MLP and CNN models.Solid lines denote SI and Online EWC, while dashed lines denote the additional optimization methods.
- Longer task queues: Task queues are extended from 5 to 78 tasks across MNIST, Fashion MNIST, EMNIST letter, SVHN, and CIFAR100, with two classes per task appearing once.The extended setting is designed to expose continual adaptation under varied domain shifts.
- Experimental setup: The evaluation compares CNN and MLP architectures with approximately 300K parameters each, using Adam for training and a 0.001 learning rate for Adagrad.Each task is trained for 10 epochs with batch size 128.
- Results: Regularization-based methods can underperform Adam when their regularization coefficient is poorly tuned, whereas Adagrad achieves similar performance without hyperparameter search.The comparison is reported for long task queues with both MLP and CNN models.
- Sensitivity analysis: Sensitivity analyses vary regularization weights and initialization methods, showing that SI and Online EWC depend on both choices, including initialization of head parameters.Figure 4 varies regularization weights across columns, while Figure 5 varies initialization methods across columns; both separate SI and Online EWC by row.