PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Drift Compensation for Class-Incremental Learning

Lu Yu, Bartłomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, Joost van de Weijer

arXiv:2004.00440v1cs.CVcs.LG

TL;DR

Class-incremental learning must learn new classes sequentially without access to earlier-task data, yet neural networks can catastrophically forget. The paper studies embedding networks and proposes semantic drift compensation, which estimates earlier-task drift from current-task data and corrects previous prototypes without exemplars. Embedding networks forget less than classification networks, while SDC consistently improves existing forgetting-prevention methods and performs competitively with exemplar-based approaches.

  • Problem

    Class-incremental learning lacks previous-task data during sequential training, causing catastrophic forgetting and class imbalance between old and new tasks.

  • Method

    The paper uses embedding networks with a nearest class mean classifier and estimates current-induced drift to compensate previous-task prototypes without storing exemplars.

  • Results

    Embedding networks suffer significantly less catastrophic forgetting than classification networks, and SDC consistently improves results when combined with existing approaches.

  • Takeaways & Limitations

    Embedding networks are a viable class-incremental learning approach, while semantic drift compensation complements existing methods for preventing forgetting.

  • Takeaways & Limitations

    On the reported multitask settings, methods without exemplars do not significantly outperform the fixed-after-task-one baseline; performance is partly affected by the large number of classes in task one.

Abstract

from arXiv · show

Class-incremental learning of deep networks sequentially increases the number of classes to be classified. During training, the network has only access to data of one task at a time, where each task contains several classes. In this setting, networks suffer from catastrophic forgetting which refers to the drastic drop in performance on previous tasks. The vast majority of methods have studied this scenario for classification networks, where for each new task the classification layer of the network must be augmented with additional weights to make room for the newly added classes. Embedding networks have the advantage that new classes can be naturally included into the network without adding new weights. Therefore, we study incremental learning for embedding networks. In addition, we propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars. We approximate the drift of previous tasks based on the drift that is experienced by current task data. We perform experiments on fine-grained datasets, CIFAR100 and ImageNet-Subset. We demonstrate that embedding networks suffer significantly less from catastrophic forgetting. We outperform existing methods which do not require exemplars and obtain competitive results compared to methods which store exemplars. Furthermore, we show that our proposed SDC when combined with existing methods to prevent forgetting consistently improves results.

1. Introduction

Continual class-incremental learning must add classes sequentially while preserving earlier knowledge, but training without previous-task data causes catastrophic forgetting. This paper studies embedding networks and introduces semantic drift compensation to estimate and correct feature drift without storing exemplars.

  • Motivation: Class-incremental learning trains tasks consecutively with access to one task’s data at a time, causing catastrophic forgetting of previous tasks.Catastrophic forgetting is described as a drastic drop in performance on previous tasks.
  • Motivation: Class-incremental learning is especially challenging without previous-task data, motivating methods that require no storage for privacy and security reasons.The paper considers the setting where no data from previous tasks can be stored.
  • Embedding networks: Embedding networks include new classes without adding weights, unlike classification networks, because they represent data in an embedding space organized by semantic dissimilarity.The paper uses embedding networks for class-incremental learning rather than expanding a classification layer.
  • Semantic drift compensation: Semantic drift compensation estimates drift in previous-task features during new-task training and uses that estimate to compensate previous-task prototypes.The method operates without preventing drift directly and can use current-task data to approximate drift in earlier tasks.
  • Semantic drift compensation: The method is evaluated with a nearest class mean classifier and can be combined with EWC, LwF, or MAS to further improve results.The paper calls class embedding means prototypes and presents SDC as complementary to existing forgetting-prevention methods.

2. Related Work

Related work addresses continual learning through regularization, rehearsal, distillation, and architectural strategies, largely in classification settings. This paper instead uses metric-learning embeddings and compensates feature drift after it occurs.

  • Continual learning: Regularization methods constrain parameter changes or outputs while learning new tasks, including LwF, EWC, and related approaches.These methods preserve earlier knowledge by regularizing probabilities, weights, or synaptic consolidation strength.
  • Continual learning: Rehearsal methods store previous-task examples and combine them with current data, sometimes adding distillation or bias correction.Exemplars address forgetting and class imbalance between old and new classes.
  • Class-incremental learning: Class-incremental methods face scalability limits when architectures require constant expansion as classes increase.The related work discusses class-IL methods that adjust network architectures but do not scale with the number of classes.
  • Paper distinction: This paper differs by using metric-learning loss with an embedding network and estimating feature drift to compensate previous-task prototypes.Unlike methods focused on preventing forgetting, the proposed approach estimates drift caused by new-task learning and corrects prototypes.
  • Deep metric learning: Deep metric learning develops embeddings through objectives such as contrastive, triplet, quadruplet, and multi-similarity losses.These losses organize embeddings by bringing similar examples closer and separating dissimilar examples.
  • Embedding networks: Prior work reports embedding networks as more robust for adversarial examples, out-of-distribution detection, and transfer learning, with preliminary evidence of reduced forgetting.The paper places its embedding-network study within an ongoing comparison with classification networks.

3. Continual Learning for Embeddings

The paper formulates class-incremental learning for embedding networks, which include new classes without architectural expansion, and evaluates nearest class mean classification under continual training. Embedding-based metric learning suffers less forgetting than softmax finetuning, while semantic drift compensation estimates and corrects feature drift using current-task data.

  • 3. Continual Learning for Embeddings: Class-incremental learning uses disjoint class sets across sequential tasks, with no task label available at test time.Training at task t accesses only the current task's image-label data.
  • 3.1. Embedding Networks: Embedding networks map images into a space where distances represent semantic dissimilarity, allowing nearest class mean classification with class prototypes.The prototype for each class is its representative point in the embedding space.
  • 3.2. Softmax Classifier versus Embedding Learning: Softmax classifiers require architectural changes for new classes, whereas embedding networks include new classes without adding network weights.Softmax class-incremental systems can require a new output head for each task and face prediction bias toward new classes.
  • 3.2. Softmax Classifier versus Embedding Learning: Metric-loss finetuning with NCM reduces forgetting relative to softmax-based alternatives, gaining 23.9% on CUB-200-2011, 32.6% on CIFAR100, and 42.9% on ImageNet-Subset.The comparison evaluates average incremental accuracy as tasks are added sequentially.
  • 3.3. Regularizing Embedding Networks: Embedding variants of LwF, EWC, and MAS add losses that constrain embedding or parameter drift during training on the current task.E-EWC uses the previous task's Fisher information matrix, while E-MAS uses parameter importance based on output sensitivity.
  • 3.3. Regularizing Embedding Networks: Semantic drift compensation estimates previous-task prototype drift from current-task drift vectors and uses that estimate to compensate for feature movement.The method does not require exemplars and can be combined with regularization methods for embeddings.

4. Semantic Drift Compensation

Semantic drift compensation updates previously computed class prototypes when embeddings change during sequential learning. It estimates unavailable old-task drift from measurable current-task drift, then corrects prototypes for nearest-class-mean classification.

  • Because previous-task data are unavailable, SDC estimates old-task drift from the drift observed in current-task embeddings during new-task training.
  • SDC compensates for embedding drift by updating previously computed prototypes rather than preventing shared parameters from adapting across tasks.
  • Semantic drift is the difference between a previous class’s true mean after later training and its stored class-mean estimate.
  • The method is combined with E-LwF, E-EWC, and E-MAS because drift compensation may complement methods that restrict parameter adaptation.
  • In a 2-dimensional MNIST visualization, SDC-corrected prototypes move closer to the correct class means, maintaining higher overall accuracy during new-task training.

5. Experiments

Experiments evaluate embedding networks and semantic drift compensation across fine-grained, CIFAR100, and ImageNet-Subset class-incremental settings. SDC reduces prototype drift and improves incremental performance, including against several exemplar-based methods, while a fixed-network baseline remains competitive in difficult settings.

  • Evaluation protocol: Evaluation uses average incremental accuracy throughout, with average forgetting additionally reported for CIFAR100 and ImageNet-Subset.Classes are randomly assigned to tasks, and models are trained and evaluated sequentially.
  • Fine-grained datasets: SDC further improves embedding-network methods, especially E-FT, by 11.9% on birds and 6.2% on flowers.The reported improvements are relative to the corresponding methods without SDC.
  • Prototype drift: SDC reduces the average distance between real class means and prototypes, indicating compensation for part of prototype drift.The analysis compares distances before and after applying SDC across tasks.
  • Longer sequences: In ten-task experiments, E-MAS exceeds LwM by 21.2% on CUB-200 and 29.0% on Caltech-101, while SDC adds 6.4% and 1.4%, respectively.The experiments use randomly divided ten-task sequences.
  • State-of-the-art comparison: On ImageNet-Subset, the method is 35.0% above LwF and has 3.5% less forgetting than Rebalance.It also exceeds iCaRL-CNN by 15.5% and iCaRL-NME by 2.5% in average incremental accuracy.
  • Baseline caveat: Fixing the network after task one yields 46.3% on CIFAR100 and 50.5% on ImageNet-Subset, limiting the advantage of continual-learning methods in these settings.The passage attributes this partly to the large number of classes in the first task.

6. Conclusions

The paper finds that embedding networks exhibit less catastrophic forgetting than classification networks and introduces semantic drift compensation for prototype correction. SDC is complementary to existing incremental-learning methods and consistently improves their results.

  • Conclusion: Embedding networks show a much less pronounced forgetting effect under finetuning than classification networks.This challenges the current dominance of softmax-based methods in continual learning within the reported experiments.
  • Conclusion: Semantic drift compensation estimates prototype drift during new-task training and compensates previous-task prototypes.The method is designed as a complement to existing incremental-learning approaches.
  • Conclusion: Combining SDC with existing incremental-learning approaches consistently improves results.

Appendices

The appendices provide supplementary drift-vector examples and additional experiments controlling possible overlap between CUB bird categories and ImageNet pretraining.

  • Supplementary analyses: Supplementary Figure 8 shows drift vectors estimated by SDC for E-LwF and E-MAS.These examples supplement the main-paper analysis associated with Figure 4.
  • Supplementary experiments: An additional CUB-200-2011 experiment removes 59 bird-related or similar ImageNet classes before pretraining ResNet18.The appendix reports average incremental accuracy results for this constrained pretraining setup.

C. Results with Multi-similarity Loss and Angular Loss

The appendix compares multi-similarity and angular metric losses with the default triplet loss and visualizes SDC’s effect on forgetting through confusion matrices.

  • Metric-loss comparison: Angular loss starts 5.0% below triplet loss on the first task, while multi-similarity starts 4.0% above it.
  • Confusion-matrix analysis: The appendix compares confusion matrices before and after SDC for CUB-200-2011 and Flowers-102 under fine-tuning.The reported comparison indicates that SDC compensates for some forgetting of previous tasks.

E. Experiments on VGG

On VGG16, the proposed embedding-based method outperforms EWC and R-EWC, while SDC provides an additional gain.

  • 30.1% improvement over EWC and 22.1% over R-EWC were achieved by E-EWC on the four-task CUB-200 setting.
  • SDC contributed an additional 1.6% gain to E-EWC.
  • The evaluation followed the protocol of Liu et al. and used a VGG16 backbone to assess generalization.

F. Classification with Embedding Networks on Cars-196 Dataset

Cars-196 was evaluated with an ImageNet-pretrained ResNet-18, and its results matched the conclusions observed on CUB-200 and Flowers-102.

  • Cars-196 contains 16,185 images across 196 car classes.
  • The Cars-196 experiment used an ImageNet-pretrained ResNet-18 backbone.
  • Results after training the final task were consistent with those on CUB-200 and Flowers-102.

G. Experiments on CIFAR100 and ImageNet-Subset

Across CIFAR100 and ImageNet-Subset, embedding-network methods generally outperform embedding fine-tuning, and SDC further improves most methods. Task-wise results show E-EWC outperforming E-Fix on every task except the first.

  • E-LwF, E-EWC, and E-MAS outperform E-FT on both CIFAR100 and ImageNet-Subset.
  • SDC improves all methods except E-LwF, with gains of 7.4% for E-FT on CIFAR100 and 3.5% on ImageNet-Subset.
  • Figure 8 illustrates SDC estimating new positions for saved prototypes of previous-task classes from observed drift vectors.
  • E-EWC and E-MAS indirectly limit embedding drift by constraining important weights, whereas E-LwF directly constrains the embedding.
  • Figure 9 compares confusion matrices before and after SDC for CUB-200-2011 and Flowers-102 under fine-tuning.
  • After eleven tasks, E-EWC outperforms E-Fix on every task except the first.
Loading 2004.00440v1…