Source-linked AI summary
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Vinay V. Ramasesh, Ethan Dyer, Maithra Raghu
TL;DR
Catastrophic forgetting causes earlier-task performance to decline during sequential training, but its underlying representational process remains incompletely understood. The paper investigates hidden representations, mitigation methods, task similarity, and benchmark settings using representational analyses, analytic modeling, and CIFAR-based experiments. It finds that forgetting concentrates in deeper layers, mitigation methods stabilize those layers, and forgetting is most severe for tasks with intermediate similarity.
Problem
Catastrophic forgetting causes significant accuracy losses on earlier tasks, while the fundamental properties governing it remain insufficiently understood across settings.
Method
The paper combines representational similarity, layer freezing and resetting, studies of replay buffers and EWC, analytic modeling, and experiments on split CIFAR-10 and a CIFAR-100 distribution-shift task.
Results
Forgetting is concentrated in higher layers, replay buffers and EWC stabilize deeper representations, and forgetting is most severe for tasks with intermediate similarity.
Takeaways & Limitations
Forgetting depends on hidden representations, task semantics, and the optimization process, with deeper-layer stability central to the mitigation methods studied.
Takeaways & Limitations
The analytic model approximates features as frozen after first-task training and allows only final-layer weights to evolve.
Abstract
from arXiv · showhide
A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks. Despite the ubiquity of catastrophic forgetting, there is limited understanding of the underlying process and its causes. In this paper, we address this important knowledge gap, investigating how forgetting affects representations in neural network models. Through representational analysis techniques, we find that deeper layers are disproportionately the source of forgetting. Supporting this, a study of methods to mitigate forgetting illustrates that they act to stabilize deeper layers. These insights enable the development of an analytic argument and empirical picture relating the degree of forgetting to representational similarity between tasks. Consistent with this picture, we observe maximal forgetting occurs for task sequences with intermediate similarity. We perform empirical studies on the standard split CIFAR-10 setup and also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.
1 Introduction
Catastrophic forgetting remains a central challenge because sequentially trained models lose accuracy on earlier tasks, while its underlying causes remain insufficiently understood. This paper investigates hidden representations, mitigation methods, task similarity, and realistic benchmarks.
- Catastrophic forgetting causes significant accuracy losses on earlier tasks after sequential training.The problem appears across continual learning, multi-task learning, transfer, input distribution shift, and data augmentation.
- The paper investigates how forgetting affects hidden representations, whether layers forget unequally, how mitigation methods operate, and how task similarity shapes forgetting.It also asks which benchmarks capture catastrophic forgetting as it naturally arises in practice.
- Deeper layers are the primary source of forgetting, changing most during sequential training.This finding comes from representational similarity, layer freezing, and layer reset experiments.
- The empirical studies use standard split CIFAR-10 and a novel CIFAR-100 based task approximating input distribution shift.
- Replay buffers and elastic weight consolidation mitigate forgetting by stabilizing deeper representations.The two methods are different approaches, but both stabilize deeper layers.
- Forgetting follows semantically consistent patterns and is related to task similarity.Accurate task-similarity measures depend on both the data and the optimization process.
- An analytic framework and experiments show that forgetting is most severe for tasks with intermediate similarity.
2 Related Work
Prior work has largely emphasized methods for mitigating catastrophic forgetting, with less analysis of how forgetting arises across hidden representations. The paper also seeks more generalizable insights than benchmarks that may support setting-specific mitigation.
- Prior mitigation research includes parameter partitioning, regularization, Bayesian approaches, and replay buffers.
- Analysis of how different hidden representations cause forgetting has been limited, motivating this paper’s focus.
- Permuted-MNIST is useful for initial study but may lead to mitigation methods that work only in limited settings.
- The paper uses split CIFAR-10 and introduces a CIFAR-100 task approximating realistic input distribution shift to gain more generalizable insights.
3 Datasets and Tasks for Catastrophic Forgetting
The paper studies catastrophic forgetting on standard split CIFAR-10 and a novel CIFAR-100 task designed to approximate input distribution shift, across multiple neural network architectures.
- Experiments use standard split CIFAR-10 and a novel CIFAR-100 task approximating input distribution shift.
- Split CIFAR-10 Task: Split CIFAR-10 divides ten-class image classification into m sequential tasks of n classes each.In this work, m = 2 and each task contains a disjoint set of five classes, with task-specific classification heads and a shared body.
- Split CIFAR-100 Distribution Shift Task: The CIFAR-100 distribution-shift task assigns each task n superclasses whose inputs come from different subsets of constituent classes.Different constituent-class subsets create an input distribution shift across tasks.
- Neural Network Architectures: The experiments span VGG, ResNet, and DenseNet architectures.The latter two include increasing numbers of skip connections, testing generalizability across architectures.
4 Catastrophic Forgetting and Neural Network Hidden Representations
The paper tests whether forgetting is distributed equally across hidden layers and finds that higher layers are its primary source across both benchmark tasks. Representational similarity, freezing, and reset experiments provide converging evidence.
- The study asks whether specific layers forget more than others during sequential training.
- Higher layers are the primary source of forgetting, while many lower layers remain stable across both tasks.The evidence combines representational similarity, layer freezing, and layer reset experiments.
- Representational similarity compares hidden-layer representations before and after second-task training using CKA scores.The similarity score ranges from 0 to 1.
- Freezing the lowest few layers has little effect on second-task accuracy, indicating that these representations change little during sequential training.
- Resetting blocks beginning at the highest layer significantly increases first-task accuracy, further implicating the highest layers in forgetting.The reset excludes the task output head.
5 Forgetting Mitigation Methods and Representations
The paper finds that common forgetting-mitigation methods primarily stabilize deeper representations, consistent with higher layers being disproportionately responsible for forgetting across task settings.
- EWC constrains parameter changes with a quadratic penalty, whereas replay buffers periodically reuse stored data from earlier tasks.The methods represent distinct approaches to mitigating forgetting.
- EWC and replay buffers mitigate forgetting by stabilizing higher-layer representations during sequential training.Increasing mitigation strength reduces representational change, particularly in deeper layers.
- Higher layers change more than lower layers during Task 2 training on split CIFAR-10 and distribution-shift CIFAR-100 tasks.CKA comparisons show lower-layer representations remain relatively stable while higher-layer representations change significantly.
- Freezing lower layers has little effect on learning Task 2 in both CIFAR task settings.This supports the conclusion that lower layers are less central to the observed forgetting process.
6 Semantics
Forgetting varies systematically with task semantics, but similarity is representation-dependent: the most severe forgetting occurs at intermediate task similarity rather than at either extreme.
- 6.1 Forgetting, Task Semantics and a Similarity Puzzle: Task representational similarity depends on both the underlying data and the optimization procedure.This dependence explains why semantic similarity can correspond to different forgetting outcomes across setups.
- 6.1 Forgetting, Task Semantics and a Similarity Puzzle: Setup 1 shows semantically similar sequential tasks produce less forgetting, whereas Setup 2 shows dissimilar categories are hurt less.The contrasting outcomes create a similarity puzzle across experimental setups.
- 6.1 Forgetting, Task Semantics and a Similarity Puzzle: Forgetting is most severe for task representations with intermediate similarity.This pattern is supported by the analytic framework and by experiments varying task similarity.
- 6.2 An analytic model of forgetting semantics: In the frozen feature model, sufficiently similar and sufficiently divergent tasks have minimal forgetting, while intermediate tasks forget.The model represents logits as a linear mapping over frozen nonlinear features and studies how second-task training changes original-task logits.
- 6.2 An analytic model of forgetting semantics: Small feature overlap between new-task training data and an original-task test point implies small logit changes and minimal forgetting.Completely orthogonal representations leave original-task predictions constant during second-task training.
- 6.2 An analytic model of forgetting semantics: Adding an unrelated category encourages dissimilar representations, and tuning task similarity by interpolation produces maximal forgetting at intermediate similarity.These experiments empirically test the analytic model’s prediction.
7 Conclusion
The paper studies catastrophic forgetting across hidden representations, mitigation strategies, task semantics, and realistic distribution shifts. Its experiments use standard CIFAR-10 and a novel CIFAR-100-based setting, with broader architectural comparisons.
- Task settings: The CIFAR-100 setting approximates input distribution shift without task-specific components because task identity is unavailable at inference.The setting also permits continuously varying the data distribution, although that possibility is not explored here.
- Scope: The distribution-shift experiments evaluate whether conclusions extend beyond the standard split CIFAR-10 benchmark.The paper introduces the CIFAR-100-based task specifically to approximate realistic input distribution shift.
- Models: VGG, ResNet, and DenseNet architectures provide varied structural tests for the reported behaviors.The architectures differ in properties such as skip connections.
B.2 Training
The training and width experiments test whether catastrophic forgetting depends on model capacity and training procedures. Wider networks improve task performance but show little change in forgetting, while headfirst training reduces forgetting without harming the new task.
- Training setup: The experiments use multiple CIFAR-10 splits to test whether the reported behaviors depend on one particular class partition.The additional splits vary the animal and object classes assigned to each task.
- Representational analysis: The study uses representational-similarity methods including CKA to compare hidden representations across models or parameter settings.CKA is invariant to orthogonal transformation and isotropic scaling, but not arbitrary linear transformations.
- B.4 Elastic Weight Consolidation: EWC regularizes new-task training by penalizing parameter changes weighted by diagonal Fisher information.The penalty strength is controlled by λ, and the reference parameters are those learned on the earlier task.
- C Forgetting versus network width: Forgetting shows minimal dependence on network width across VGG, DenseNet, and ResNet models.The amount of forgetting is measured as the percent drop in task-1 accuracy after training on task 2.
- D Headfirst training: Training the new task head alone for up to five epochs improves original-task performance without sacrificing new-task performance.This procedure holds the rest of the model fixed before full-network training.
- D Headfirst training: The headfirst result suggests that coadaptation between a freshly initialized head and the shared model contributes to forgetting.The conclusion is based on the observed reduction in forgetting when head-only training precedes full-network training.
- Task-specific stages: Making deeper stages task-specific substantially improves retained performance, with the deepest two stages recovering a significant fraction of forgotten-task performance.This pattern holds across the evaluated architectures.
F Additional replay buffer and EWC results
Additional experiments test whether representational findings generalize across architectures and datasets, and examine head-only training at task switches. The results support a broad pattern of stabilizing or protecting deeper representations.
- Representational similarity: CKA results extend the main representational findings to VGG and DenseNet, as well as replay-buffer experiments on split CIFAR-100.The accompanying plots indicate applicability across the studied architectures and datasets.
- Headfirst training: Training only the newly initialized readout layer before the full network consistently improves task-0 accuracy across VGG, ResNet, and DenseNet.The head-only phase uses 0, 1, 2, or 5 epochs, while the remaining parameters are held fixed.
G Layer reset and retrain experiments
Resetting or retraining only the upper part of a forgotten network can recover much of its earlier performance, while linear readouts reveal usable information in earlier layers after forgetting. Some experiments also show that later training can improve the original task.
- Task-specific stages: Making deeper stages task-specific produces a dramatic performance increase, with two such stages sufficient to recover significant performance.The result holds across all evaluated architectures.
- G Layer reset and retrain experiments: Retraining with all but the last couple of layers frozen nearly recovers pre-forgetting performance.This reset-and-retrain result is consistent with the broader finding that forgetting is concentrated in higher layers.
- Retraining effects: Training on task 2 can improve task-1 performance when the full network is subsequently retrained, especially for ResNet and to some degree VGG.Some reset-and-retrain conditions also outperform the pre-forgetting network.
- H Linear regression on forgotten network activations: A linear model trained on post-forgetting activations recovers a substantial fraction of lost performance, nearly matching the pre-forgetting model for ResNet and VGG.The forgetting-induced drop reaches up to forty percent, while the post-forgetting linear readout is only about one percent less accurate for those architectures.
- H Linear regression on forgotten network activations: After forgetting, linear readouts rely more on earlier-layer activations rather than the final layers.Before forgetting, the linear model places most importance on final-layer representations.
- CIFAR-100 distribution shift: The CIFAR-100 representation analyses compare changes in VGG and DenseNet with corresponding ResNet results under distribution shift.These experiments extend the representation analysis to additional architectures.
J Additional semantic experiments
Additional semantic experiments show that forgetting depends consistently on task semantics, but whether similar or dissimilar tasks forget less depends on the task setup. Across architectures and category choices, the experiments support setup-dependent semantic patterns.
- Across VGG, ResNet, and DenseNet, semantic experiments produce consistent results across different choices of objects and animals.
- Sequential binary classification tasks: In single-category binary tasks, similar categories lead to less forgetting because the model is not pressured to distinguish objects from animals.
- Four-class and CIFAR-100 tasks: In four-class-to-two-class tasks and the CIFAR-100 setup, dissimilar categories lead to less forgetting.
- Sequential binary classification tasks: For sequential binary CIFAR-10 tasks, less forgetting occurs when the initial task is more similar to the second task.
K Analytic model
The analytic models relate forgetting to feature overlap and show that dissimilar task representations can preserve earlier-task predictions. Experiments using rotated representations support this relationship, while changes in final-layer weights alone do not necessarily cause forgetting.
- Analytic framework: The analytic framework links changes in predictions during second-task training to the overlap between features evaluated on the two tasks.The feature-overlap matrix is also connected to kernels used in linear models and wide neural networks.
- Analytic framework: In the multi-head model, prediction changes depend on both representation overlap and similarity between the model heads.When feature overlap is zero, predictions remain constant; otherwise, head similarity further scales the change.
- Analytic framework: Adding a ReLU non-linearity before the readout head does not alter the essential dependence of output changes on the overlap matrix.The stated conclusion applies to the analytic model's output evolution.
- Rotating representations: Explicitly rotating second-task features to reduce similarity minimizes forgetting in sequential binary CIFAR-10 tasks.The experiment uses car-versus-plane followed by cat-versus-horse classification with frozen features.
- Weight changes and forgetting: Task 1 predictions remain constant despite significant changes to final-layer weights when the two task representations are orthogonal.This setup uses a π/2 rotation to make the second-task features orthogonal to the first-task features.