PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Incremental Learning

Francisco M. Castro, Manuel J. Marín-Jiménez, Nicolás Guil, Cordelia Schmid, Karteek Alahari

arXiv:1807.09536v2cs.CV

TL;DR

Incremental deep networks must learn new classes without retaining the full historical dataset, because conventional updates suffer catastrophic forgetting and become difficult to sustain as classes grow. The paper combines representative old-class exemplars with cross-entropy and distillation in an end-to-end framework, achieving state-of-the-art performance on CIFAR-100 and ImageNet.

  • Problem

    Deep-learning classifiers suffer catastrophic forgetting and conventionally require all old and new samples when classes are added incrementally.

  • Method

    The method jointly updates features and classifiers using a small representative memory of old-class samples and a cross-distilled loss combining cross-entropy with distillation.

  • Results

    The approach achieves state-of-the-art incremental-learning performance on CIFAR-100 and ImageNet, improving previous average results by more than 5% on ImageNet.

  • Takeaways & Limitations

    Representative memory size and training-set balance substantially influence accuracy, while the end-to-end approach is more robust than methods using an independently learned external classifier.

Abstract

from arXiv · show

Although deep learning approaches have stood out in recent years due to their state-of-the-art results, they continue to suffer from catastrophic forgetting, a dramatic decrease in overall performance when training with new classes added incrementally. This is due to current neural network architectures requiring the entire dataset, consisting of all the samples from the old as well as the new classes, to update the model -a requirement that becomes easily unsustainable as the number of classes grows. We address this issue with our approach to learn deep neural networks incrementally, using new data and only a small exemplar set corresponding to samples from the old classes. This is based on a loss composed of a distillation measure to retain the knowledge acquired from the old classes, and a cross-entropy loss to learn the new classes. Our incremental training is achieved while keeping the entire framework end-to-end, i.e., learning the data representation and the classifier jointly, unlike recent methods with no such guarantees. We evaluate our method extensively on the CIFAR-100 and ImageNet (ILSVRC 2012) image classification datasets, and show state-of-the-art performance.

1 Introduction

Incremental deep learning must absorb classes arriving over time without relearning all old data, while retaining accuracy, manageable memory, and joint representation-classifier learning. The paper addresses this gap with an end-to-end framework using representative old-class samples, distillation, and cross-entropy.

  • Motivation: Incremental recognition requires learning new classes without having all old and new samples available during training.This setting is motivated by applications such as face recognition, where previously learned identities must remain recognizable.
  • Desired properties: A truly incremental system should process classes in arbitrary order and at arbitrary times while preserving performance on old and new classes.It should also maintain reasonable parameter and memory requirements and jointly update features and the classifier.
  • Research gap: Existing methods fail to satisfy all requirements, often decoupling representation and classifier learning or increasing parameters and memory as classes accumulate.Other approaches are restricted to particular datasets, class settings, or non-deep classifiers.
  • Contribution: The proposed end-to-end framework combines representative memory with cross-entropy for new classes and distillation for retaining old-class knowledge.The framework can adapt any deep architecture by replacing its original loss with the incremental loss.

2 Related Work

Prior incremental-learning methods either rely on complete data or fixed representations, suffer catastrophic forgetting, restrict adaptability, or expand model size. The paper positions its approach as jointly learning task-specific features and classifiers while avoiding these limitations.

  • Traditional and lifelong learning: Traditional incremental methods retain classifier information such as support vectors but generally operate with fixed or limited representations.Related lifelong and never-ending learning methods may require the entire dataset or rely on fixed representations.
  • Deep learning methods: Deep networks jointly learn features and classifiers, but incremental updates can cause catastrophic forgetting of the original classes.Early connectionist solutions do not apply directly to contemporary deep computer-vision architectures.
  • Distillation-based methods: Distillation combined with cross-entropy reduces forgetting but was evaluated especially in simplified settings where old and new data come from different datasets.Those settings involve little confusion between old and new classes.
  • Alternative strategies: Other approaches freeze layers, use autoencoders, or regulate learning rates, limiting adaptability or targeting restrictive scenarios.These methods address preservation of old knowledge through mechanisms distinct from the paper’s end-to-end update.
  • Model growth: Growing networks can mitigate forgetting but increase parameters and memory with the number of classes, tasks, and added layers.The proposed model instead makes minimal changes to the original network size.
  • iCaRL: iCaRL decouples classifier and representation learning and uses an auxiliary exemplar set with a nearest-mean classifier.The paper contrasts this external classifier with its end-to-end deep-network approach.

3 Our Model

The model incrementally extends a deep classifier while preserving the feature extractor, stores representative old-class exemplars, and trains with a cross-distilled loss. Its components support joint feature-classifier updates without requiring a specialized architecture.

  • Architecture: The framework uses a standard deep classification architecture with a feature extractor, classification layers, logits, and a softmax layer at test time.The training loss replaces the original classification loss, while the feature extractor supplies inputs to the classification layers.
  • Representative memory: Representative memory stores selected old-class samples and can use either fixed total capacity or a constant exemplar count per class.With fixed capacity K and c stored classes, the exemplar count is n = ⌊K/c⌋; constant per-class storage instead grows with the number of classes.
  • Sample selection: New memory samples are selected by herding, which ranks class samples by distance to the class mean and stores the first n samples.This selects samples considered most representative according to the class mean.
  • Incremental architecture: Incremental updates add classification layers for new classes while leaving the feature-extractor architecture unchanged.The approach can therefore be applied to different deep architectures or pretrained models by adding layers and the incremental loss.
  • Cross-distilled loss: The cross-distilled loss combines cross-entropy over old and new classes with distillation losses on old-class classification layers.Cross-entropy updates class decision boundaries, while distillation retains knowledge from old classes.
  • Distillation: Increasing the distillation temperature gives lower-scoring classes greater influence, encouraging finer separation among class responses.The modified scores are formed by raising the original scores to the exponent 1/T.

4 Incremental Learning

Each incremental step constructs a training set from new-class data and old-class exemplars, trains the model, applies balanced fine-tuning, and updates representative memory. All model weights are updated, allowing the feature extractor and classification layers to adapt together.

  • Incremental Learning: An incremental step has four stages: training-set construction, model training, balanced fine-tuning, and representative-memory updating.The fine-tuning subset contains the same number of samples per class.
  • Incremental Learning: The training set combines new-class samples with old-class exemplars stored in representative memory.Each sample receives a one-hot classification label and logits from old-class layers for distillation.
  • Incremental Learning: During training, all network weights are updated rather than freezing the feature extractor.The changing features require classification layers to adapt across successive incremental steps.
  • Incremental Learning: Balanced fine-tuning addresses imbalance between old and new classes by using equal sample counts per class.This stage is especially relevant when new-class data substantially outnumber stored old-class exemplars.

5 Implementation Details

The experiments use dataset-specific ResNet models, incremental class batches, and representative-memory exemplars, with standard augmentation and training procedures.

  • Model and memory: CIFAR-100 uses a 32-layer ResNet, while ImageNet uses an 18-layer ResNet as the dataset-specific deep model.Representative memory stores K = 2000 distillation samples for CIFAR-100 and K = 20000 for ImageNet.
  • Evaluation protocol: Class-incremental evaluation splits datasets into incremental batches, evaluates after each step on all trained classes, and reports average accuracy across incremental batches.The first batch is excluded from the reported average because it does not represent incremental learning.
  • Data augmentation: Data augmentation applies brightness changes, contrast normalization, and random cropping before training.The contrast transformation is imaltered = (im − mean) × contrast + mean, with contrast sampled from [0.2, 1.8].

6 Evaluation on CIFAR-100

On CIFAR-100, the end-to-end approach generally outperforms competing methods across incremental settings, remains stable for 2–20 classes per step, and benefits from balanced fine-tuning and more exemplars.

  • Fixed memory size: The approach achieves the best results for 2, 5, 10, and 20 classes per incremental step, while matching Hybrid1 and trailing iCaRL by 1% for 50 classes.The paired t-test gives p < 0.01 for 2, 5, 10, and 20 classes, but p = 0.9886 for 50 classes.
  • Fixed memory size: Performance remains stable across incremental step sizes from 2 to 20 classes, unlike the other evaluated methods.Smaller steps improve early accuracy but require more steps, reducing accuracy in later stages.
  • Fixed memory size: The approach significantly outperforms iCaRL with small increments, while iCaRL approaches its performance as more classes are added per step.The approach outperforms LwF.MC in all cases, highlighting the role of representative memory.
  • Fixed number of samples: With a constant number of exemplars per old class, average accuracies are 62.4, 62.7, and 63.3 for 5–20-class incremental steps using 50 exemplars per class.These results indicate stability across incremental step sizes.
  • Ablation studies: Increasing exemplars per class improves accuracy, while balanced fine-tuning improves results in all cases, especially for larger incremental steps.The full model combining balanced fine-tuning and augmentation achieves the best results and establishes a new state-of-the-art on CIFAR-100 incremental learning.

7 Evaluation on ImageNet

ImageNet evaluation uses incremental steps of 10 or 100 classes and reports top-5 accuracy after each step. In both settings, the approach establishes a new state of the art, improving previous average results by more than 5%.

  • ImageNet experiments use 100 or 1,000 classes divided into random incremental splits of 10 or 100 classes, respectively.All methods use the same class sets, and models are evaluated after each step on all classes learned so far.
  • Top-5 accuracy is reported for every incremental step together with average incremental accuracy.The ImageNet experiments are executed once.
  • More than 5%: the approach improves previous average results in both ImageNet incremental settings.The authors describe this as establishing a new state of the art.
  • The approach maintains good accuracy with 100-class incremental steps when new and old class samples are more balanced.The authors suggest this makes the approach suitable for large datasets with many classes.

8 Summary

The paper concludes that its end-to-end CNN approach combines cross-entropy and distillation losses and is more robust than recent alternatives. It also identifies representative memory, memory size, and training-set imbalance as important accuracy factors.

  • The method trains CNNs incrementally by combining cross-entropy and distillation loss functions.
  • The end-to-end approach is reported as more robust than recent methods such as iCaRL, which uses an independently learned external classifier.
  • Representative memory, its size, and unbalanced training sets play important roles in final accuracy.
  • Future work will explore sample-selection strategies using a dynamic number of samples per class.

Fixed number of samples

With a fixed number of old-class samples, the approach performs best across the tested configurations, with larger gains in more challenging settings using fewer classes per incremental step.

  • The experiment evaluates 5, 10, and 20 classes per incremental step with 50, 75, or 100 exemplars for each old class.
  • The approach achieves the best results across the fixed-sample configurations.
  • Its improvement is larger in challenging cases with fewer classes per incremental step.
  • With 5 classes and 100 old-class samples, performance approaches the upper bound using only 20% of the original data.

A priori known task

The known-task evaluation splits CIFAR-100 into five-class groups and selects among only the five classes belonging to the known task. Results are reported for different memory sizes.

  • CIFAR-100 is divided into groups of 5 classes and trained incrementally.
  • At test time, the known task restricts output probabilities to that task’s five classes.This simplifies prediction compared with selecting among all trained classes.
  • Final CIFAR-100 accuracy is reported for different memory sizes.

Unknown task

With the task identity hidden at test time, the method maintains good performance as it incrementally learns classes, although accuracy declines as the class count grows.

  • The CIFAR-100 evaluation uses five classes per incremental step and reports the average accuracy across incremental steps in parentheses.
  • The method performs well without knowing which task produced each test sample.
  • Accuracy decreases as more classes are added because previously learned knowledge must be retained while each old class receives fewer stored samples.
  • GEM’s accuracy decreases dramatically as the number of classes increases, making it noncompetitive when the task is unknown beforehand.

Appendix C: Comparison with similar classes

On datasets containing visually similar classes, the end-to-end approach outperforms competing methods, with especially strong gains on vehicles and a similar pattern on dog breeds.

  • The experiments test whether jointly learning the classifier and feature extractor improves performance when classes share similar visual characteristics.
  • The vehicle experiment uses 10 CIFAR-100 classes across five incremental steps of two classes, while the ImageNet experiment uses 120 dog breeds across twelve steps of ten classes.
  • 73.3% average accuracy for the end-to-end model exceeds iCaRL’s 47.5% on the vehicle experiment.
  • The end-to-end approach achieves the best average accuracy in both the vehicle and dog-breed experiments, with a significant boost over other methods.
  • Figure 1 reports average incremental-step accuracy for the vehicle and dog-breed evaluations.
Loading 1807.09536v2…