Source-linked AI summary
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay
Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, Jin Tang
TL;DR
FSCIL asks models to learn new classes from few examples while retaining old knowledge, and the effectiveness of replay for this setting has been questioned. The paper proposes data-free replay with entropy-regularized generation and one-hot-like relabeling. It reports improved benchmark performance over prior methods on CIFAR-100, miniImageNet, and CUB-200.
Problem
FSCIL requires learning new classes from limited examples while avoiding overfitting and forgetting, and the effectiveness of replay-based methods in this setting has been questioned.
Source: §1 · p. 2Method
The method synthesizes old-class data without real-data access, encourages uncertain generated samples with entropy regularization, and relabels them with one-hot-like labels for cross-entropy training.
Source: §1 · p. 3Results
The proposed approach reports final accuracy gains over Cheraghian et al. of 8.5% on CIFAR-100, 6.11% on miniImageNet, and 9.16% on CUB-200, and over CEC of 1.00%, 0.58%, and 0.11%, respectively.
Source: §5.4 · p. 12Takeaways & Limitations
The experiments support adopting replay-based methods for FSCIL, with data-free replay offered as a way to avoid accessing real old data.
Source: §1 · p. 3DeepInversion’s high-confidence class constraint is unsuitable for FSCIL, where predictions for few-shot learned classes are usually uncertain.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Few-shot class-incremental learning (FSCIL) has been proposed aiming to enable a deep learning system to incrementally learn new classes with limited data. Recently, a pioneer claims that the commonly used replay-based method in class-incremental learning (CIL) is ineffective and thus not preferred for FSCIL. This has, if truth, a significant influence on the fields of FSCIL. In this paper, we show through empirical results that adopting the data replay is surprisingly favorable. However, storing and replaying old data can lead to a privacy concern. To address this issue, we alternatively propose using data-free replay that can synthesize data by a generator without accessing real data. In observing the the effectiveness of uncertain data for knowledge distillation, we impose entropy regularization in the generator training to encourage more uncertain examples. Moreover, we propose to relabel the generated data with one-hot-like labels. This modification allows the network to learn by solely minimizing the cross-entropy loss, which mitigates the problem of balancing different objectives in the conventional knowledge distillation approach. Finally, we show extensive experimental results and analysis on CIFAR-100, miniImageNet and CUB-200 to demonstrate the effectiveness of our proposed one.
1 Introduction
FSCIL must learn new classes from few examples while retaining old knowledge, but replay-based methods have been said to be ineffective and may raise privacy concerns. The paper reports that replay is effective and proposes data-free replay with entropy-encouraged samples and one-hot-like relabeling.
1 Introduction
- FSCIL addresses incremental learning with limited examples per new class, where overfitting and forgetting old knowledge are concerns.CIL commonly assumes ample training data for each new class, an assumption the paper says is impractical in real applications.Source: §1 · pp. 1–2
- Replay-based methods significantly outperform TOPIC in the authors’ FSCIL experiments, challenging the claim that replay is ineffective.The comparison includes iCaRL, NCM, and DeepInversion [30].Source: §1 · p. 2, Figure 1 · p. 2
- The paper proposes data-free replay to synthesize old samples without accessing real data, addressing privacy concerns associated with replaying real data.The authors also argue that prior claims against replay could discourage further exploration of replay-based FSCIL methods.Source: §1 · pp. 2–3
- Entropy regularization encourages the generator to produce uncertain samples for knowledge distillation, while one-hot-like relabeling lets the new model train using cross-entropy alone.The relabeling is proposed to avoid balancing cross-entropy and knowledge-distillation losses.Source: §1 · p. 3
- The paper reports state-of-the-art performance for its proposed approach and presents extensive experiments on FSCIL.The supplied introduction passage does not give benchmark values.Source: §1 · p. 3
2 Related Works
The related-work section reviews class-incremental learning, few-shot class-incremental learning, and data-free knowledge distillation. It places the paper in an under-studied replay-based direction for FSCIL.
2.1 Class-Incremental Learning
- Class-incremental learning methods use strategies including storing old real data, combining cross-entropy and distillation losses, and cosine normalization.The section contrasts typical CIL with FSCIL, where new classes have limited training samples.Source: §2.1 · p. 4
2.2 Few-shot Class-incremental Learning
- Prior FSCIL work includes prototype refinement, synthetic feature generation, graph-based classifier updates, and bi-level optimization, while replay-based methods remain under-studied.The cited methods include,,, and.Source: §2.2 · p. 4
2.3 Data-free Knowledge Distillation
- Data-free distillation either trains a generator on original images or synthesizes pseudo-images from a trained network; the paper follows the latter line.The section notes that DeepInversion-based methods cannot be simply transferred to FSCIL.Source: §2.3 · pp. 4–5
3 Preliminaries
The preliminaries define FSCIL as incremental learning of novel classes from few examples and introduce data-free replay as the basis of the approach. The replay procedure trains a generator against an old model and auxiliary model to produce critical samples, with entropy regularization encouraging uncertain samples.
3.1 Problem Setting
- FSCIL uses disjoint class sets across sessions, with many well-sampled classes in the base set and few examples for new classes in later sessions.At each session, training uses the current session’s data and evaluation covers all classes seen so far.Source: §3.1 · p. 5
3.2 Data-free Replay
- Data-free replay trains a generator from a trained model to synthesize critical samples, using an auxiliary model alternately matched to and pushed away from the original model’s outputs.The knowledge-transfer update matches outputs with MSE, while generator evolution maximizes their distance to produce samples hard for the auxiliary model to learn.Source: §3.2 · pp. 5–6
- Entropy regularization encourages the generator to produce uncertain data, which the paper associates with samples near the original model’s decision boundaries.The authors motivate this choice by noting that few-shot models often assign low confidence to input images.Source: §3.2 · p. 6
4 Methodology
The method uses data-free replay to synthesize examples from an old model, then trains on them alongside new-class examples. It encourages uncertain synthetic examples and relabels them to make incremental training rely on cross-entropy alone.
4 Methodology
- Data-free replay synthesizes examples of previously seen classes from the old model for training alongside new-class examples, aiming to alleviate catastrophic forgetting.The method first inverts the old model to obtain a generator.Source: §4 · pp. 6–7
4.1 Entropy-regularized Data-free Replay
- Few-shot learned classes have higher output entropy, so the generator is trained to maximize the old model’s prediction entropy and synthesize uncertain images.The method treats high-entropy predictions as indicating decision-boundary inputs or few-shot learned classes.Source: §4.1 · p. 7, Figure 3 · p. 7
- In the toy experiment, entropy regularization makes generated samples more likely to lie in high-entropy decision-boundary regions.The figure contrasts samples from a normal generator with those from an entropy-constrained generator.Source: §4.1 · p. 8, Figure 4 · p. 8
4.2 Learning Incrementally with Uncertain Data
- Generated images receive hard labels from the old model’s argmax, allowing training with cross-entropy alone rather than balancing cross-entropy and distillation losses.The authors report that vanilla distillation is sensitive to hyperparameter selection, especially in the few-shot setting.Source: §4.2 · pp. 8–9
- With replayed and novel data available, the method can fine-tune old model parameters rather than keeping the backbone fixed.The authors note that updating the backbone can overfit few examples and forget old mappings, while a fixed backbone can generalize poorly to novel classes.Source: §4.2 · p. 9
5 Experiments
Experiments across CIFAR-100, miniImageNet, and CUB-200 show that the proposed method outperforms state-of-the-art methods on all three benchmarks. Analyses also examine replay-based baselines, component ablations, generated-data labels, replay quantity, and DeepInversion.
5.1 Datasets
- Experiments use CIFAR-100, miniImageNet, and CUB-200, with 60, 60, and 100 base classes, respectively; CIFAR-100 increments comprise five classes, while miniImageNet uses five-way five-shot tasks.Each CUB-200 incremental session has 10 images per class.Source: §5.1 · p. 10
5.2 Implementation Details
- The experiments use ResNet20 on CIFAR-100 and ResNet18 on miniImageNet and CUB-200, with DCGAN for replay and generated-data quantity matched to new-class training samples.The generator is trained at the beginning of each incremental session; incremental learning combines generated and novel data.Source: Backbone network · p. 10, Training details · p. 10
5.4 Main Results and Comparison
- The proposed method outperforms state-of-the-art methods across all three benchmarks, exceeding Cheraghian et al. in final accuracy by 8.5%, 6.11%, and 9.16% on CIFAR-100, miniImageNet, and CUB-200.The comparison reports results across CIFAR-100, miniImageNet, and CUB-200.Source: Table 1 · p. 11, §5.4 · p. 12
5.3 Re-implementation of Replay-based Methods.
- Under FSCIL, reimplemented replay methods are competitive; NCM exceeds TOPIC, Zhu et al., and Cheraghian et al. by 17.71%, 3.83%, and 5.44% on CIFAR-100.The comparison uses results marked as the authors’ reimplementations under FSCIL.Source: §5.3 · p. 11
5.5 Analysis
- Removing entropy regularization lowers final accuracy, removing re-labeling causes a significant performance drop, and backbone fine-tuning improves results.The ablation study evaluates entropy regularization, re-labeling, and backbone fine-tuning on CIFAR-100.Source: Table 2 · p. 12, §5.5 · p. 12, Importance of different components · p. 13
- Entropy regularization shifts generated labels toward few-shot learned classes, whereas the generator without it tends to produce more base-class samples.Figure 5(a) compares label distributions for training data and generated data.Source: Figure 5 · p. 13, Label distribution of generated data · p. 13
- Accuracy peaks when replay quantity matches the training-data count—25 images per CIFAR-100 incremental session—and smaller or larger quantities can hurt through imbalance.Too little replay may not prevent forgetting, while too much may hinder adaptation to new classes.Source: Figure 5 · p. 13, Analysis of number of generated data · pp. 13–14
- The proposed replay method exceeds DeepInversion by 1.66% in final accuracy; DeepInversion favors confident predictions and its data-generation process is time-consuming.The authors report that their generator replays data efficiently once trained.Source: Table 3 · p. 14, Analysis of data-free replay · p. 14
6 Conclusion
The conclusion reports that data replay can be used in FSCIL and presents a data-free replay method that synthesizes old samples. The authors report strong performance for avoiding forgetting and quickly adapting to new classes, while noting a potential generator-training limitation on large-scale datasets.
6 Conclusion
- Extensive comparisons with state-of-the-art methods show the approach achieves the best performance for avoiding forgetting and quickly adapting to new classes.Source: §6 · p. 14
- The authors report that data replay can be adopted for FSCIL, and introduce data-free replay to address vanilla replay’s privacy concern.Source: §6 · p. 14
- Entropy regularization encourages uncertain generated data, which the method relabels to learn from replay and address forgetting.Source: §6 · p. 14
- Training the generator on large-scale datasets may be difficult and potentially jeopardize replayed-data quality.Source: §6 · p. 14