Source-linked AI summary
Incremental Learning In Online Scenario
Jiangpeng He, Runyu Mao, Zeman Shao, Fengqing Zhu
TL;DR
The paper addresses online incremental learning when sequential data include both new classes and new observations of old classes, causing catastrophic forgetting and concept drift. It combines modified cross-distillation, two-step learning, and feature-based exemplar updates, with strong online results and a Food-101 demonstration.
Problem
Online incremental learning must handle sequential streams containing new classes and new observations of old classes while avoiding catastrophic forgetting and concept drift.
Method
The framework combines modified cross-distillation and two-step learning with exemplar updates based on features from new observations of old classes.
Results
The method achieves comparable results to state-of-the-art methods on CIFAR-100 and ImageNet-1000 in online evaluation, with strong Food-101 results.
Takeaways & Limitations
The framework supports online streams containing new and old classes, short updates with limited data, and potential lifelong incremental learning.
Takeaways & Limitations
The learn-from-scratch setting assumes incoming classes have no previous knowledge to retain.
Abstract
from arXiv · showhide
Modern deep learning approaches have achieved great success in many vision applications by training a model using all available task-specific data. However, there are two major obstacles making it challenging to implement for real life applications: (1) Learning new classes makes the trained model quickly forget old classes knowledge, which is referred to as catastrophic forgetting. (2) As new observations of old classes come sequentially over time, the distribution may change in unforeseen way, making the performance degrade dramatically on future data, which is referred to as concept drift. Current state-of-the-art incremental learning methods require a long time to train the model whenever new classes are added and none of them takes into consideration the new observations of old classes. In this paper, we propose an incremental learning framework that can work in the challenging online learning scenario and handle both new classes data and new observations of old classes. We address problem (1) in online mode by introducing a modified cross-distillation loss together with a two-step learning technique. Our method outperforms the results obtained from current state-of-the-art offline incremental learning methods on the CIFAR-100 and ImageNet-1000 (ILSVRC 2012) datasets under the same experiment protocol but in online scenario. We also provide a simple yet effective method to mitigate problem (2) by updating exemplar set using the feature of each new observation of old classes and demonstrate a real life application of online food image classification based on our complete framework using the Food-101 dataset.
1. Introduction
The paper targets online incremental learning systems that must process streams containing new classes and new observations of old classes without catastrophic forgetting or degradation from concept drift. It proposes modified cross-distillation, two-step learning, exemplar updates, and a Food-101 application.
- Online vision systems must process sequential streams containing both new classes and new observations of old classes.
- Catastrophic forgetting degrades old-class performance as new classes are added, while concept drift changes the distribution of already learned classes.
- Existing approaches do not satisfy the combined requirements of mixed data streams, future-data performance, short updates, and lifelong incremental learning.They rely on using all available new data for many epochs, making them impractical for real-life applications.
- The paper introduces modified cross-distillation with two-step learning for online incremental learning under constrained runtime and data.The evaluation follows an established benchmark protocol on CIFAR-100 and ImageNet-1000 in the more constrained online setting.
- The framework updates exemplars using features from new observations of old classes to mitigate concept drift and demonstrates online food classification on Food-101.
2. Related Work
Related work addresses incremental learning through weight constraints, distillation, generative data, and exemplar-based methods. These approaches retain prior knowledge but remain difficult to apply when updates must occur online with limited runtime and data.
- Incremental learning supports continuous knowledge accumulation but remains a longstanding machine-learning challenge.
- Traditional approaches update SVM boundaries or use ensembles and nearest-neighbor classifiers with retained information from earlier data.
- Deep-learning methods preserve old-task knowledge through frozen or constrained weights, distillation, autoencoders, or synthetic data.
- Exemplar-based methods address class imbalance and bias, but their offline training requirements hinder real-life applications.
3. Online Incremental Learning
Online incremental learning updates a model from sequential, tightly bounded data blocks, using each new example once rather than repeated offline epochs. The paper addresses forgetting with modified cross-distillation and addresses concept drift by updating exemplars from new old-class observations.
- Online incremental learning generates successive models from sequential data blocks, with each block containing a strictly limited number of examples.The block size p determines how many new examples update the model.
- Unlike offline learning, each incoming example is used only once rather than across multiple training epochs.
- Catastrophic forgetting occurs when updating a model for m new classes sharply reduces performance on its n old classes.The proposed response is a modified cross-distillation loss combined with two-step learning.
- Concept drift occurs when the joint distribution P(X, Y) changes between training-time data and future data from old classes.
4. Incremental Learning Framework
The framework supports sequential online learning through initial learning, offline retraining, and learning from a trained model. It combines representation-based learning, modified cross-distillation, exemplar maintenance, and two-step updates to address limited data, catastrophic forgetting, and concept drift.
- Framework overview: The framework has three parts: learning from scratch, offline retraining, and learning from a trained model.Lifelong learning is achieved by alternating offline retraining and learning from a trained model after initial learning.
- Learn from scratch: Nearest Class Mean classification supports early online learning when data are scarce by assigning inputs to the closest class representation.A pretrained network extracts representations φ(x_i), and class means are updated online as new data arrive.
- Learn from scratch: The system switches from the NCM classifier to the baseline classifier after the baseline is more accurate for s consecutive blocks, with s = 5.The switch combines early data-efficiency from representation learning with the sequential baseline once sufficient data have arrived.
- Offline retraining: Periodic offline retraining uses all data available up to that time to support lifelong learning as new classes or existing-class data arrive.The retraining step addresses the increasing severity of catastrophic forgetting and concept drift during continued updates.
- Learn from a trained model: Modified cross-distillation combines distillation on old classes with cross-entropy over old and new classes, using an accommodation ratio to mix classifier outputs.β controls the old-class output units: β = 1 uses the new classifier, whereas β = 0 replaces them with the old classifier outputs.
- Learn from a trained model: The exemplar set is updated with new observations of old classes by comparing feature distances to class means and replacing less representative exemplars when appropriate.This makes observations reflecting concept drift more likely to be selected for subsequent model updates.
5. Experimental Results
The experiments evaluate online incremental learning across CIFAR-100, ImageNet-100, and Food-101, including new classes and, for the complete framework, new observations of old classes. The proposed methods achieve strong online and test performance through modified cross-distillation, two-step learning, representation learning, and exemplar updates.
- Experimental Setup: The online benchmark predicts each incoming block before updating the model, using constrained exemplars and comparing against offline incremental-learning results.The CIFAR and ImageNet experiments use the same exemplar count as prior methods, while the proposed protocol also evaluates future data streams containing old-class observations.
- CIFAR-100 and ImageNet-100: Our method shows the best accuracy for all CIFAR-100 incremental learning steps in the online scenario.Classes are divided into random splits of 5, 10, 20, and 50, with block size p = 8 and results averaged over four trials.
- CIFAR-100 and ImageNet-100: On ImageNet-100, EEIL surpasses our method at the second step, but our method attains the best performance as more classes are added.The experiment uses a randomly selected 100-class subset of ImageNet-1000, divided into 10-class increments, with block size p = 16.
- Food-101: On Food-101, combining baseline and representation learning achieves the best online accuracy across incremental steps and higher overall online accuracy.Representation learning performs well when data is scarce, while the baseline improves as more data arrives; automatic switching combines their strengths.
- Food-101: Updating the exemplar set improves overall online and test accuracy and raises online accuracy for old classes under changing data distributions.The exemplar update uses the current class mean from observed old-class data, and the two-step method pairs new data with exemplars.
6. Conclusion
The proposed framework combines online incremental learning techniques to address catastrophic forgetting and concept drift. It performs well on benchmark datasets and improves real-life food image classification over baseline methods.
- The framework combines modified cross-distillation, two-step learning, and exemplar updates for online incremental learning.The first two components address catastrophic forgetting, while exemplar updates use features from new observations of old classes to mitigate concept drift.
- It supports data streams containing both new classes and new observations of old classes in online scenarios.The framework is designed for short update runtimes, limited data, and lifelong handling of an unknown number of classes.
- The method outperforms current state-of-the-art methods on CIFAR-100 and ImageNet-1000 in the challenging online learning scenario.
- Food-101 experiments show significant improvement over baseline methods for a real-life image classification problem.