Source-linked AI summary
Don't forget, there is more than forgetting: new metrics for Continual Learning
Natalia Díaz-Rodríguez, Vincenzo Lomonaco, David Filliat, Davide Maltoni
TL;DR
The paper addresses the limited consensus around continual-learning evaluation and the near-exclusive focus on forgetting. It proposes implementation-independent metrics spanning performance, transfer, memory, and computation, combines them into a weighted CLscore, and evaluates five strategies on iCIFAR-100. The experiments illustrate metric variability across criteria and support comprehensive evaluation practices.
Problem
Continual-learning evaluation lacks consensus and focuses almost exclusively on forgetting, although practical assessment also involves performance, transfer, memory, and computational factors.
Method
The paper maps seven desiderata to [0, 1] metrics and aggregates them into a flexible CLscore that supports application-specific weighting.
Results
The metrics were evaluated on five continual-learning strategies using iCIFAR-100, with results illustrating variability across criteria under equal metric weights.
Takeaways & Limitations
Comprehensive evaluation can expose differences across continual-learning criteria beyond accuracy and forgetting, while CLscore provides an optional ranking mechanism.
Takeaways & Limitations
The authors plan to refine the metrics and assess more strategies in more exhaustive evaluation settings.
Abstract
from arXiv · showhide
Continual learning consists of algorithms that learn from a stream of data/tasks continuously and adaptively thought time, enabling the incremental development of ever more complex knowledge and skills. The lack of consensus in evaluating continual learning algorithms and the almost exclusive focus on forgetting motivate us to propose a more comprehensive set of implementation independent metrics accounting for several factors we believe have practical implications worth considering in the deployment of real AI systems that learn continually: accuracy or performance over time, backward and forward knowledge transfer, memory overhead as well as computational efficiency. Drawing inspiration from the standard Multi-Attribute Value Theory (MAVT) we further propose to fuse these metrics into a single score for ranking purposes and we evaluate our proposal with five continual learning strategies on the iCIFAR-100 continual learning benchmark.
1 Introduction and Related Work
Continual learning evaluation has largely centered on catastrophic forgetting, despite broader practical concerns. The paper motivates more comprehensive evaluation across performance, transfer, and computational factors.
- Catastrophic forgetting is the rapid overriding of previously learned knowledge when a neural network trains sequentially on new data.
- The proposed evaluation perspective considers performance over time, backward and forward knowledge transfer, memory overhead, and computational efficiency.
- Evaluation granularity can vary at the task level, including use of the same test batch for multiple training batches.
- Existing continual-learning metrics include overall performance, forgetting, Average Accuracy, Backward Transfer, and Forward Transfer.These metrics evaluate performance relative to offline learning, lower-bound accuracy, or performance across past and future test batches.
2 Proposed Metrics for Continual Learning
The paper defines implementation-independent continual-learning metrics spanning performance dynamics, knowledge transfer, memory, and computation, then maps them into a weighted CLscore for ranking. The proposed measures extend evaluation beyond forgetting by assessing accuracy and transfer throughout learning.
- Framework: Continual learning is modeled as sequential updates that transform a model and external memory using each task’s training set and task label.The framework represents each update as <h_i−1, Tr_i, M_i−1, t> → <h_i, M_i>.
- Aggregation: Seven criteria are normalized to [0,1], assigned weights summing to one, averaged over runs, and aggregated into a maximized CLscore.Criteria intended for minimization are transformed using 1−c_i so all components retain increasing monotonicity.
- Performance and transfer: Accuracy averages train-test matrix entries on and below the principal diagonal, while the revised metric evaluates performance at every timestep.This dynamic formulation is intended to better characterize continual learning over time than evaluating only after the final task.
- Performance and transfer: Backward Transfer measures how learning affects previous tasks; the expanded metric averages transfer after each task and separates remembering from positive improvement.Remembering is defined from negative BWT, whereas positive BWT captures improvement over time.
- Performance and transfer: Forward Transfer measures how learning affects future tasks through average above-diagonal accuracies, including possible zero-shot performance.The train-test matrix entry R_i,j records performance on task j after training through task i.
- Efficiency: Efficiency criteria cover model size, stored samples, and computation, with computational efficiency based on multiplication-addition operations and an optional epoch scaling factor.Model-size efficiency compares Mem(θ_i) with the first-task model, while sample storage is bounded relative to the lifetime dataset.
3 Experiments and Conclusions
The experiments evaluate five continual-learning strategies with proposed metrics on iCIFAR-100, using equal criterion weights for the reported CLscore. The results emphasize variability across criteria and the need for comprehensive evaluation beyond accuracy and forgetting, while positioning the CLscore as optional and context-dependent.
- Results: Figure 1 visualizes metric variability across strategies and includes accuracy computed over a fixed test set.The spider chart uses larger area to indicate better metric values.
- Experiments: Five strategies—cumulative, naïve, EWC, SI, and LwF—are evaluated on iCIFAR-100, where each task contains 10 disjoint classes.The benchmark evaluation compares cumulative and naïve baselines with three continual-learning strategies.
- Experiments: Each criterion is reported as an average over three runs, and the experiments use homogeneous weights that value every CL metric equally.The appendix also reports other possible weight configurations.
- Conclusions: The metric variability supports evaluating aspects beyond accuracy and forgetting when different application contexts make other criteria important.The paper presents comprehensive evaluation as the motivation for its metric set and results.
- Conclusions: The CLscore is optional and context dependent; future work will refine the metrics and evaluate more strategies in more exhaustive settings.The stated aim is to stimulate comprehensive evaluation practices rather than require the aggregate score.
A.1 Metrics and benchmark illustration
The benchmark illustration defines how accuracy, backward transfer, and forward transfer are extracted from the train-test accuracy matrix, while incorporating intermediate time steps. It also situates strategies against naïve and cumulative baselines and shows how weighting configurations affect aggregate CLscore comparisons.
- Metrics: Matrix R contains the test accuracy on task j after the model observes the last sample from task i, with R_ii denoting accuracy immediately after learning task i.The matrix has N tasks and is used to select entries for the proposed metrics.
- Metrics: Accuracy averages diagonal and below-diagonal entries of R, including intermediate training steps rather than only the final row.This modification captures degradation and improvement throughout continual learning.
- Metrics: The FWT definition removes the random-initialization subtraction term to guarantee non-negative values while retaining potential positive transfer.The paper notes that some strategies can perform worse than random accuracy on future tasks.
- Benchmark: The experiment uses CIFAR-100 split into ten incremental batches of 10 classes and evaluates a CNN with four convolutional and two fully connected layers.The model’s hyperparameters are chosen to maximize the accuracy metric for each strategy.
- Baselines: Naïve learning is the lower-bound strategy, whereas cumulative learning retrains on all observed data and serves as an upper-bound reference.Only cumulative learning assumes that all previous data can be stored and reused.
- Visualization: Spider charts compare EWC, LWF, and SI with naïve and cumulative reference bounds under equal criterion weighting, where larger area indicates a higher CLscore.Table 3 additionally reports CLscore and CLstability under multiple weighting configurations.