Source-linked AI summary
Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild
Kibok Lee, Kimin Lee, Jinwoo Shin, Honglak Lee
TL;DR
Catastrophic forgetting makes it difficult for class-incremental models to retain earlier-task performance when learning new tasks, especially under limited memory. The paper uses unlabeled data in the wild through global distillation, a three-step training scheme, and confidence-based sampling, achieving its strongest gains when external data are available. On ImageNet, the method reaches 15.8% higher accuracy and 46.5% less forgetting than E2E with external data.
Problem
Class-incremental learning must handle sequential tasks while retaining earlier-task performance, but limited memory prevents storing all previous-task data.
Method
The method combines global distillation, three-step training with current-task and previous-task teachers, and confidence-based sampling of unlabeled external data.
Results
15.8% higher accuracy and 46.5% less forgetting than E2E were obtained on ImageNet with an external dataset.
Takeaways & Limitations
The proposed methods outperform prior methods, with performance gains more significant when unlabeled external data are available.
Takeaways & Limitations
The learning objective can bias performance against the current task because the previous model did not learn that task, requiring an additional current-task teacher.
Abstract
from arXiv · showhide
Lifelong learning with deep neural networks is well-known to suffer from catastrophic forgetting: the performance on previous tasks drastically degrades when learning a new task. To alleviate this effect, we propose to leverage a large stream of unlabeled data easily obtainable in the wild. In particular, we design a novel class-incremental learning scheme with (a) a new distillation loss, termed global distillation, (b) a learning strategy to avoid overfitting to the most recent task, and (c) a confidence-based sampling method to effectively leverage unlabeled external data. Our experimental results on various datasets, including CIFAR and ImageNet, demonstrate the superiority of the proposed methods over prior methods, particularly when a stream of unlabeled data is accessible: our method shows up to 15.8% higher accuracy and 46.5% less forgetting compared to the state-of-the-art method. The code is available at https://github.com/kibok90/iccv2019-inc.
1. Introduction
Class-incremental learning must preserve performance on earlier tasks as new tasks arrive, but limited memory causes catastrophic forgetting. The paper addresses this by using unlabeled data in the wild with global distillation, a three-step learning scheme, and confidence-based sampling.
- Class-incremental learning aims to perform well on both previous and new tasks as tasks arrive sequentially.
- Limited memory prevents storing all previous-task training data, making catastrophic forgetting a central scalability problem.
- The method leverages a continuous stream of unlabeled external data without assuming correlation between that data and the labeled tasks.
- A three-step scheme trains a current-task teacher, distills the previous model, current teacher, and ensemble, then fine-tunes to reduce current-task overfitting.
- Global distillation transfers knowledge across all previous tasks and distinguishes classes across task boundaries, unlike task-wise local distillation.
- 15.8% higher accuracy and 46.5% less forgetting were achieved than E2E on ImageNet when an external dataset was used.
2. Approach
The approach combines global knowledge distillation, three-step training, balanced fine-tuning, and confidence-based sampling of unlabeled data for class-incremental learning.
- External-data sampling: The external dataset is sampled from a large unlabeled stream using a confidence-based procedure, without assuming correlation between unlabeled data and the tasks.The sampled external dataset is used during learning and does not require permanent memory afterward.
- Learning setup: The method trains a class-incremental model using the current task, a previous-task coreset, the previous model, and a sampled unlabeled external dataset.All previously observed classes remain candidates at test time without task boundaries.
- Global distillation: Global distillation transfers knowledge across all previous tasks and preserves discrimination between classes from different tasks, which local distillation misses.The global loss is motivated by the limitation of task-wise local distillation.
- Global distillation: An ensemble of the previous and current-task teachers supplies missing discrimination between previous and current task classes.The previous teacher knows earlier tasks, while the current teacher specializes in the new task; their ensemble can perform all tasks.
- Balanced fine-tuning: Data weighting fine-tunes task-specific parameters by scaling gradients to reduce prediction bias toward the current task.Scaling a gradient is equivalent to feeding the same data multiple times.
- Three-step learning: The three-step strategy trains a current-task teacher, trains the model by distilling previous, current, and ensemble teachers, then fine-tunes task-specific parameters.The final step addresses bias toward the current task caused by imbalanced training data.
3. Related Work
Related work addresses catastrophic forgetting through class-, task-, and data-incremental learning, using model-based constraints or data-based replay and distillation.
- Continual learning settings: Class-incremental learning grows the class-label set, whereas task-incremental learning assumes clear task boundaries and provides the test-time task identity.Class-incremental evaluation uses a single output space without task boundaries.
- Continual learning approaches: Model-based approaches preserve prior knowledge by constraining updates to parameters considered important for previous tasks.These methods reduce or penalize changes around original parameter values.
- Continual learning approaches: Data-based approaches use knowledge distillation and require data whose distribution resembles the data from previous tasks.Prior methods retain a coreset or replay generated data to provide such examples.
- Knowledge distillation: The proposed method differs from prior distillation approaches by using global rather than task-wise local distillation.Related two-teacher methods use model-based preservation or task-wise distillation for task-incremental learning.
- Balanced replay and scalability: Data weighting avoids the diversity loss of undersampling and the scalability cost of oversampling during balanced fine-tuning.Unlike coreset replay, the external dataset is sampled from unlabeled data and discarded after learning, so it needs no permanent memory.
4. Experiments
Experiments on CIFAR-100 and ImageNet compare the proposed methods with established baselines using average incremental accuracy (ACC) and average forgetting (FGT). The proposed three-step scheme and external-data learning improve performance, with the largest gains from combining global distillation and unlabeled data.
- Experimental setup: Experiments evaluate CIFAR-100 and ImageNet using ten CIFAR-100 trials and nine ImageNet trials with different random seeds.Class splits use task sizes of 5, 10, and 20 classes; ImageNet images are downsampled to 32×32.
- Evaluation metrics: ACC measures overall class-incremental accuracy, while lower FGT indicates less forgetting of previous tasks.Evaluation is performed without task boundaries, so predictions range over all classes learned so far.
- Comparison without external data: 4.8% higher ACC and 6.0% lower FGT than E2E are achieved on ImageNet with task size 5 without unlabeled external data.The comparison identifies E2E as the strongest state-of-the-art baseline in this setting.
- Effect of external data: Unlabeled external data consistently improves compared methods, but the gain is more significant for global distillation than for E2E.With ImageNet task size 5, external data improves ACC by 3.2% for E2E and 10.5% for global distillation; FGT decreases by 1.1% and 43.1%, respectively.
- Ablation studies: Ablations show that combining the previous model, current-task teacher, and ensemble reference models yields the best performance, while balanced fine-tuning generally improves FGT.The separate current-task teacher provides less-interfered current-task knowledge, and data-weighted fine-tuning outperforms removing current-task data.
- Ablation studies: The combined external-data sampling strategy performs best, whereas random sampling raises ACC but prediction-based sampling better limits forgetting.Randomly sampled OOD data is not beneficial because it is locally distributed rather than broadly covering the complement of learned data.
5. Conclusion
The paper leverages unlabeled data in the wild for class-incremental learning through global distillation and a confidence-based sampling strategy.
- Global distillation aims to preserve reference-model knowledge without task boundaries.
- The proposed three-step learning scheme uses confidence-based sampling to build an external dataset from unlabeled data.
- The method targets class-incremental learning with a large stream of unlabeled external data.
A. Illustration of Global Distillation
Global distillation uses three reference models while model M learns: the previous model P, the current-task teacher C, and their ensemble Q.
- P is the previous model used as a reference for global distillation.
- C is the teacher specialized for the current task.
- Q is an ensemble of the previous model and current-task teacher.
B. Details on Experimental Setup
The experimental setup uses scalable compared methods and specifies long mini-batch training schedules with staged learning-rate decay and optional fine-tuning.
- Hyperparameters: Training uses mini-batches of 128 samples over 200 epochs, with scheduled learning-rate decay.
- Hyperparameters: With fine-tuning, training lasts 180 epochs before a 20-epoch fine-tuning stage.
- Scalability of methods: All compared methods are scalable and evaluated under a fair condition.
- Scalability of methods: Generative replay methods are excluded because prior work reports challenges for generative continual learning on natural-image datasets.
C.1. More Ablation Studies
Additional studies evaluate accuracy and forgetting across external-data ratios and examine how dataset relatedness affects results.
- ACC and FGT are evaluated against the OOD ratio on CIFAR-100 and ImageNet.Results are averaged over ten trials for CIFAR-100 and nine trials for ImageNet.
- The ImageNet study evaluates ACC and FGT against hierarchical distance between training data and the unlabeled stream.Results are averaged over nine trials.
- The experiment controls correlation between training data and external data using hypernym-hyponym relationships.
C.2. More Results
Additional experiments evaluate ACC and FGT across CIFAR-100 and ImageNet under multiple task sizes, using repeated trials and, for ImageNet, comparisons involving unlabeled data.
- Table C.1 compares methods on CIFAR-100 and ImageNet, reporting means and standard deviations across ten and nine trials, respectively.Higher values are better for metrics marked ↑, while lower values are better for metrics marked ↓.
- Figure C.3 reports ImageNet performance and shows the ACC and FGT gains from learning with unlabeled data.Results are averaged over nine trials.
- Figures C.4 and C.5 report CIFAR-100 ACC and FGT versus the number of trained classes for task sizes 5 and 20.Both figures average results over ten trials.
- Figures C.6, C.7, and C.8 report ImageNet ACC and FGT versus trained classes for task sizes 5, 10, and 20.Each figure averages results over nine trials.