Source-linked AI summary
An Empirical Study of Example Forgetting during Deep Neural Network Learning
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, Geoffrey J. Gordon
TL;DR
The paper asks whether example forgetting occurs within a single classification task, without a clear distributional shift. It defines and measures transitions from correct to incorrect classification during training, finding stable forgetting patterns and competitive generalization after removing many examples.
Problem
The paper investigates whether a forgetting process related to catastrophic forgetting occurs when neural networks learn examples within a single task.
Method
The paper defines forgetting events as correct-to-incorrect transitions and analyzes forgetting statistics across examples, datasets, seeds, architectures, and training settings.
Results
Examples differ substantially in forgetting frequency, with many consistently unforgettable examples and forgetting statistics that remain stable across seeds and architectures.
Takeaways & Limitations
A substantial fraction of training examples can be removed according to forgetting dynamics while maintaining competitive test performance.
Takeaways & Limitations
Forgetting events depend on the trained architecture, optimizer, and number of epochs, whose influence the paper investigates.
Abstract
from arXiv · showhide
Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.
1 INTRODUCTION
The paper asks whether forgetting, usually associated with distribution shifts between tasks, also occurs among examples learned within a single classification task. It studies forgetting dynamics to understand example interactions, dataset compressibility, and whether forgetting identifies informative or noisy examples.
- Forgetting events occur when an example learned correctly is later misclassified during sequential mini-batch optimization.
- The study analyzes forgettable and unforgettable examples, their effects on decision boundaries and generalization error, and the possibility of reducing training data without harming accuracy.
- Existing example-selection methods use loss, hardness, commonality, or predictive-distribution variance, but do not examine persistent forgetting throughout learning.
- The experiments find many examples that are never forgotten after learning, with stability across random seeds and strong correlation across neural architectures.
- Noisy-label examples and images with uncommon, visually complicated features are among the most frequently forgotten.
- Removing a very large fraction of the least forgotten examples still produces extremely competitive test performance.
3 DEFINING AND COMPUTING EXAMPLE FORGETTING
The paper formalizes example forgetting by tracking correctness during stochastic-gradient training and summarizes each example’s transitions, forgetting frequency, and classification margin. Because evaluating every example at every update is expensive, it estimates events when examples reappear in mini-batches.
- The setting is supervised classification with a dataset of observation-label pairs, a neural network, and empirical-risk minimization using cross-entropy loss.
- A forgetting event occurs when an example changes from correctly classified to misclassified between consecutive updates.
- The analysis records forgetting-event distributions and the first learning time, while also examining the classification margin between the correct and strongest competing logits.
- An example is unforgettable if it is learned at some finite time and experiences no later forgetting events; examples forgotten at least once are forgettable.
- For computational efficiency, forgetting statistics are computed only when an example appears in successive mini-batches, yielding a lower-bound estimate.
- Experiments use MNIST, permutedMNIST, and CIFAR-10 with multiple architectures and training schemes designed to achieve competitive test errors.
4 CHARACTERIZING EXAMPLE FORGETTING
The study characterizes example forgetting across datasets, tests its stability and visual correlates, and examines its implications for continual learning and dataset reduction.
- 55,012 MNIST, 45,181 permutedMNIST, and 15,628 CIFAR-10 examples were unforgettable across five seeds, representing 91.7%, 75.3%, and 31.3% of their training sets.Less complex and diverse datasets contained more unforgettable examples.
- Forgetting statistics were stable across random seeds, with average Pearson correlation of 89.2% and correlation of 97.6% between two five-seed groups.The least-forgotten examples also had tight confidence intervals, supporting confident ranking.
- Unforgettable CIFAR-10 examples were visually recognizable, whereas frequently forgotten examples had ambiguous or uncommon class characteristics.Examples such as planes against clear skies were easier to recognize than trucks against brown backgrounds.
- Changing 20% of CIFAR-10 labels made examples more forgettable, and noisy-label examples were among the most forgotten with none unforgettable.The comparison used forgetting-event distributions for noisy and original labels.
- In synthetic continual learning, examples forgotten at least once suffered stronger forgetting, while they could support examples that had never been forgotten.The experiment alternated training over random or forgetting-statistic-based partitions drawn from the same CIFAR-10 distribution.
- Removing examples ordered by forgetting events preserved CIFAR-10 generalization substantially better than removing random examples, while removing the most-forgotten examples harmed performance more.The removal experiment retrained models from scratch on progressively smaller subsets and separately removed contiguous groups of 5,000 examples.
5 REMOVING UNFORGETTABLE EXAMPLES
Removing examples with many forgetting events harms generalization more than removing unforgettable examples, while substantial data removal can preserve performance. This supports viewing frequently forgotten examples as more relevant to the learned decision boundary.
- Removing 30% of CIFAR-10 examples ordered by forgetting events preserves comparable generalization performance, while random removal rapidly degrades it.Removing up to 35% causes only marginal degradation below 0.2%.
- Removing 5,000 examples with increasingly more forgetting events generally worsens generalization error, although the most forgotten examples can sometimes hurt performance.The latter examples may correspond to outliers or mislabeled data.
- Forgettable examples may act as analogs of support vectors because their forgetting frequency correlates with misclassification margin and relevance to the decision boundary.Unforgettable examples have little influence on generalization performance when removed.
- The fraction of unforgettable examples varies across datasets—91.7% for MNIST, 75.3% for permutedMNIST, and 31.3% for CIFAR-10—alongside different removal tolerances.These differences are interpreted as reflecting variation in underlying data-distribution complexity.
6 TRANSFERABLE FORGETTING EVENTS
Forgetting-event rankings stabilize during training and transfer across architectures, enabling a simpler model to estimate example orderings for a larger one. Using the transferred ordering, WideResNet retains near-optimal performance after substantial data removal.
- Forgetting statistics depend on the training architecture, optimizer, and number of epochs, motivating tests of their transferability across these factors.
- The forgetting-event ordering becomes very stable after 75 epochs, with 25 epochs providing a reasonable early estimate.Stability is measured by Spearman rank correlation with the final 200-epoch ordering.
- A simpler convolutional network shows reasonably strong agreement with ResNet18 when identifying its unforgettable examples.The comparison uses precision-recall retrieval of ResNet18’s unforgettable examples.
- WideResNet remains near optimal with 30% of CIFAR-10 removed using the example ordering obtained from ResNet18.Estimating forgetting statistics with ResNet18 can save up to 6 hours compared with training WideResNet on one Titan X GPU.
7 CONCLUSION AND FUTURE WORK
The study finds that example forgetting occurs even within single classification tasks, with stable differences among examples and limited importance of unforgettable examples for final generalization. It concludes that forgetting statistics may reveal intrinsic data properties and support data reduction.
- Catastrophic forgetting-like events occur while neural networks train on a single classification task, without an explicit task distribution shift.
- Some examples are consistently unforgettable while others are more prone to forgetting, and these statistics appear stable across training characteristics.
- Unforgettable examples can be removed from training without hurting generalization, suggesting they contribute little to final classifier performance.
- The authors propose that forgetting statistics uncover intrinsic properties of data and may inform future work in theory, supervised learning, and reinforcement learning.
10 STABILITY OF THE FORGETTING EVENTS
Forgetting-based rankings can identify both unforgettable and highly forgotten examples well before training ends. By 75 epochs, both ends of the spectrum are retrieved with very high precision and recall.
- At 75 epochs, CIFAR-10 examples at both the unforgettable and most-forgotten ends are retrieved with very high precision and recall.The rankings are evaluated against a full 200-epoch ResNet18 run.
- The precision and recall of retrieving unforgettable examples are already high after 50 epochs.
11 Noising THE DATA SETS
Adding noise makes training examples more forgettable: stronger noise reduces unforgettable examples, while pixel noise produces a forgetting distribution resembling label noise. Randomized-gradient controls suggest some forgetting occurs by chance.
- Pixel noise: Increasing Gaussian pixel noise decreases unforgettable examples and increases examples in the second mode of the forgetting distribution.Noise uses σ_noise = λσ_data with λ ∈ {0.5, 1, 2, 10}.
- Pixel versus label noise: 20% pixel noise with σ_noise = 10 produces a forgetting distribution resembling that caused by 20% label noise.The comparison is shown for the same subset of examples before and after pixel corruption.
- Chance forgetting: The randomized-gradient clone estimates forgetting that can occur by chance while preserving gradient statistics from true SGD updates.The base classifier’s gradients are shuffled before being applied to a clone with the same initial random weights.
- Chance forgetting: At most twice, examples are forgotten by chance in the randomized-gradient setting.The forgetting histogram is averaged over 5 seeds.
13 CONFIDENCE ON FORGETTING EVENTS FOR CIFAR-10
Confidence intervals for forgetting-event counts are estimated by aggregating results across many random seeds, showing both average behavior and central variability.
- Confidence estimation: 100 seeds are grouped into 20 averages over 5 seeds to form confidence intervals on forgetting-event counts.The plotted bounds are the bottom 2.5 and top 2.5 percentiles of the 20 curves.
- Confidence estimation: Figure 14 displays the average forgetting curve with its 95% confidence interval across the seed-based averages.The average is shown in green, with lower and upper percentile curves in blue and orange.
14 VISUALIZATION OF FORGETTABLE AND UNFORGETTABLE IMAGES
Visual inspection contrasts the most unforgettable and forgettable examples within each CIFAR-10 class, with forgettable images appearing visually unusual or difficult to classify.
- Example visualization: Examples from every CIFAR-10 class are sorted by forgetting-event count to show the most unforgettable and forgettable images.Ties in the ordering are broken randomly.
- Example visualization: Forgettable examples seem to exhibit peculiar or uncommon visual features.The figure provides additional examples for each class.
15 FORGETTING IN CIFAR-100
CIFAR-100 has a high forgetting burden, especially for images duplicated under conflicting labels, yet removing its unforgettable examples can preserve test performance.
- Forgetting distribution: 3,809 examples, or 7.62% of CIFAR-100’s training set, are unforgettable.CIFAR-100 exhibits the highest percentage of forgetting events among the presented datasets.
- Forgetting distribution: CIFAR-100’s forgetting distribution is closer to noisy CIFAR-10 than to the original datasets.This comparison is reported alongside the CIFAR-100 and noisy CIFAR-10 distributions.
- Ambiguous examples: The 36 most forgotten CIFAR-100 examples all appear multiple times in the training set under different labels.Examples include images labeled as girl and baby, mouse and shrew, and oak, willow, or maple tree.
- Training-set reduction: Removing all unforgettable examples—about 8% of CIFAR-100’s training set—maintains test performance.Selecting examples by the paper’s ordering reduces the training set without affecting generalization, whereas random removal reduces performance faster.