Source-linked AI summary
Large Scale Incremental Learning
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Fu
TL;DR
Incremental learning must retain old-class knowledge despite missing past data, while existing exemplar and distillation methods degrade at large class scales. The paper introduces BiC, which corrects new-class bias in the final classifier using a two-parameter linear model and a small validation set. BiC outperforms state-of-the-art methods by 11.1% on ImageNet and 13.2% on MS-Celeb-1M.
Problem
Existing incremental-learning methods struggle with performance degradation when the number of classes becomes large because old-class data are missing and class distributions become imbalanced.
Method
BiC corrects bias toward new classes in the final fully connected layer using a two-parameter linear model learned from a small validation set.
Results
11.1% on ImageNet and 13.2% on MS-Celeb-1M are the reported BiC gains over state-of-the-art algorithms on two large datasets.
Takeaways & Limitations
A small validation set can effectively correct final-layer bias and support strong incremental-learning performance on datasets with 1,000+ classes.
Takeaways & Limitations
BiC trails EEIL during the first three incremental batches on ImageNet-100, largely attributed to EEIL's enhanced data augmentation.
Abstract
from arXiv · showhide
Modern machine learning suffers from catastrophic forgetting when learning new classes incrementally. The performance dramatically degrades due to the missing data of old classes. Incremental learning methods have been proposed to retain the knowledge acquired from the old classes, by using knowledge distilling and keeping a few exemplars from the old classes. However, these methods struggle to scale up to a large number of classes. We believe this is because of the combination of two factors: (a) the data imbalance between the old and new classes, and (b) the increasing number of visually similar classes. Distinguishing between an increasing number of visually similar classes is particularly challenging, when the training data is unbalanced. We propose a simple and effective method to address this data imbalance issue. We found that the last fully connected layer has a strong bias towards the new classes, and this bias can be corrected by a linear model. With two bias parameters, our method performs remarkably well on two large datasets: ImageNet (1000 classes) and MS-Celeb-1M (10000 classes), outperforming the state-of-the-art algorithms by 11.1% and 13.2% respectively.
1. Introduction
Incremental learning degrades as class scale increases because missing old-class data creates imbalance, especially among visually similar classes. BiC corrects new-class bias in the final classifier layer using a two-parameter linear model and performs strongly on large datasets.
- Catastrophic forgetting degrades performance when past data are unavailable, making incremental learning important for applications such as face recognition.The missing old-class data challenge both old-class performance and balance between old and new classes.
- As classes increase from 100 to 1000, iCaRL and EEIL experience greater performance degradation relative to a non-incremental classifier.Figure 1 evaluates both datasets over 10 incremental steps.
- Visually similar classes are especially difficult because data imbalance makes narrow class boundaries sensitive and favors classes with more samples.The paper attributes large-scale difficulty to the coupling of increasing class similarity and imbalance.
- BiC adds a two-parameter linear bias-correction layer after the final fully connected layer to correct its bias toward new classes.Training uses a first-stage training set and a second-stage validation set drawn from old-class exemplars and new-class samples.
- 11.1% on ImageNet and 13.2% on MS-Celeb-1M are the reported gains of BiC over state-of-the-art algorithms on two large datasets.
2. Related Work
Prior incremental-learning methods use weight constraints, distillation, synthetic replay, or old-class exemplars to retain knowledge. BiC also keeps exemplars but addresses imbalance by correcting bias in the classifier with a small validation set.
- Incremental-learning research includes methods using real old data, synthetic data, or no old data.The paper organizes recent deep-learning approaches into three categories based on their old-data requirements.
- Without using old data: Methods without old data preserve prior tasks through frozen layers, weight constraints, or knowledge distillation, but weight conflicts can limit one approach.
- Using synthetic data: Synthetic-replay methods use generative models to reproduce old-task data but depend heavily on generator capability and struggle with complex objects and scenes.
- Using exemplars from old data: Exemplar-based methods retain a small number of images from each old class and use distillation or balanced finetuning to mitigate imbalance.
- Using exemplars from old data: BiC keeps old-class exemplars like prior exemplar methods but corrects classifier bias with a linear model learned from a small validation subset.The validation subset is excluded from training and used only for bias correction.
3. Baseline: Incremental Learning using Knowledge Distillation
The baseline incrementally learns new and old classes using knowledge distillation and classification losses, with their contributions balanced according to the numbers of old and new classes.
- The baseline learns a model for n+m classes using knowledge distillation from an old model covering n classes.New-class samples and selected exemplars from old classes are used during incremental training.
- The distilling loss is computed on both new-class samples and exemplars from old classes.
- The classification loss uses softmax cross entropy over the old and new classes.The output probability is the softmax of the logits for each class.
- Figure 4 compares baseline, BiC, FC-only retraining with all data, and whole-network training with all data on CIFAR-100.
- The overall objective combines distilling and classification losses as L = λLd + (1 −λ)Lc.λ is set to n/(n+m), is zero for the first batch, and approaches one when old classes greatly outnumber new classes.
4. Diagnosis: FC Layer is Biased
Experiments on CIFAR-100 diagnose a strong bias in the last fully connected layer, where class imbalance favors new classes and harms incremental classification.
- The baseline accuracy quickly drops across five incremental CIFAR-100 batches of 20 classes each.
- Retraining only the fully connected layer with all old and new training samples improves accuracy over the baseline.
- 20%: retraining the final fully connected layer with all classes improves accuracy on the final 100-class classifier.The improvement over the baseline validates that the fully connected layer is heavily biased.
5. Bias Correction (BiC) Method
BiC corrects the new-class bias after baseline training by fitting a two-parameter linear layer on a small validation set while keeping the learned network fixed.
- BiC uses two training stages: baseline network training followed by estimation of two bias parameters on a small validation set.
- The validation set is excluded from representation training so it can reflect the distributions of old and new classes for bias correction.Exemplars and new-class samples are split into training and validation subsets.
- The old and new validation subsets are balanced for bias correction.
- BiC keeps old-class logits unchanged and applies a linear model to new-class logits.The correction uses only two shared parameters, α and β, across all new classes.
- The convolutional and fully connected layers remain frozen while the bias parameters are optimized with softmax cross entropy.
- The authors report that the simple linear model effectively corrects bias in the fully connected layer.
6. Experiments
Experiments evaluate BiC across large- and small-scale incremental-learning benchmarks, showing stronger gains as class count and data imbalance increase. Ablations attribute the improvement to correcting new-class bias with a small validation set and show limited sensitivity to exemplar selection.
- Large datasets: On ImageNet-1000, BiC’s average gains over EEIL and iCaRL are 11.1% and 19.7% across ten incremental batches.At the final incremental step, BiC exceeds EEIL by 18.5% and iCaRL by 26.5%.
- Large datasets: On Celeb-10000, BiC’s average gain over iCaRL is 13.2%, increasing to 22.4% at the last incremental step.BiC is less than 3% better than iCaRL during the first couple of incremental steps, but the gap widens from the third step onward.
- Different scales: BiC outperforms state-of-the-art methods on both ImageNet-100 and ImageNet-1000, with a larger gain at the larger scale.Relative to the upper bound, BiC degrades 10.5% on ImageNet-100 and 16.0% on ImageNet-1000.
- Small-scale benchmark: On CIFAR-100, BiC has similar performance to iCaRL and EEIL, outperforming them on 50- and 20-class splits but trailing EEIL slightly on 10- and 5-class splits.The reported margins are small, while EEIL uses additional brightness augmentation and contrast normalization not used by BiC.
- Ablation study: Bias correction raises final-step CIFAR-100 accuracy from 40.34% to 56.69%, leaving a 4.24% gap to the 60.93% upper bound.The ablation also reports that BiC reduces new-class bias and produces a confusion matrix similar to the upper bound.
7. Conclusions
The paper addresses incremental-learning imbalance by identifying and correcting new-class bias in the classifier layer, achieving strong results on large datasets.
- BiC targets the imbalance issue that becomes critical as incremental-learning problems grow to many classes.
- The classifier layer has a strong bias toward new classes because they have substantially more training data.
- A linear model with a small validation set effectively corrects this classifier bias.
- 11.1% on ImageNet ILSVRC 2012 and 13.2% on MS-Celeb-1M are the reported margins over state-of-the-art methods.