PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Feature Replay For Class-Incremental Learning

Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D. Bagdanov, Shangling Jui, Joost van de Weijer

arXiv:2004.09199v1cs.CVcs.LG

TL;DR

Class-incremental learning suffers catastrophic forgetting and bias toward new classes, while image replay is difficult for complex datasets. The paper proposes exemplar-free generative feature replay with feature distillation, achieving strong results on ImageNet and CIFAR-100 with lower storage needs than exemplar-based methods.

  • Problem

    Class-incremental learning must retain prior knowledge without task IDs, but limited old-class data biases classifiers toward new classes and image replay is unsatisfactory on complex datasets.

  • Method

    The method splits the network into a feature extractor and classifier, combining generative feature replay in the classifier with feature distillation in the extractor.

  • Results

    The method outperforms methods without exemplars by a large margin on ImageNet and CIFAR-100, and surpasses exemplar-based methods in several settings.

  • Takeaways & Limitations

    Feature replay uses simpler high-level feature distributions, making the approach computationally efficient, scalable to large datasets, and less storage-intensive than exemplar replay.

  • Takeaways & Limitations

    Generative image replay remains difficult for complex datasets because image distributions require computationally expensive generators trained with substantial data and may lack discriminative features.

Abstract

from arXiv · show

Humans are capable of learning new tasks without forgetting previous ones, while neural networks fail due to catastrophic forgetting between new and previously-learned tasks. We consider a class-incremental setting which means that the task-ID is unknown at inference time. The imbalance between old and new classes typically results in a bias of the network towards the newest ones. This imbalance problem can either be addressed by storing exemplars from previous tasks, or by using image replay methods. However, the latter can only be applied to toy datasets since image generation for complex datasets is a hard problem. We propose a solution to the imbalance problem based on generative feature replay which does not require any exemplars. To do this, we split the network into two parts: a feature extractor and a classifier. To prevent forgetting, we combine generative feature replay in the classifier with feature distillation in the feature extractor. Through feature generation, our method reduces the complexity of generative replay and prevents the imbalance problem. Our approach is computationally efficient and scalable to large datasets. Experiments confirm that our approach achieves state-of-the-art results on CIFAR-100 and ImageNet, while requiring only a fraction of the storage needed for exemplar-based continual learning. Code available at \url{https://github.com/xialeiliu/GFR-IL}.

1. Introduction

Class-incremental continual learning must retain prior knowledge when task identity is unknown, but new-task data biases classifiers toward recent classes. The paper addresses this imbalance with exemplar-free generative feature replay combined with feature distillation.

  • Class-incremental learning is harder than task-incremental learning because the task ID is unknown at test time.
  • Limited data from previous tasks biases class-incremental classifiers toward the most recently learned classes.
  • Generative image replay avoids exemplars but performs unsatisfactorily on larger, complex datasets such as CIFAR-100 and ImageNet.Accurate image generation is feasible for small datasets but difficult for datasets with more classes and larger images.
  • The proposed hybrid model uses generative feature replay in the classifier and feature distillation in the feature extractor.Feature generation is adopted because it is considerably easier than accurately generating images when data are limited or classes are numerous.
  • CCA-based analysis visualizes how and where networks forget across layers.
  • The method outperforms exemplar-free methods by a large margin on ImageNet and CIFAR-100, and exceeds exemplar-based methods in most evaluated settings.The authors also report computational efficiency and scalability to large datasets.

2. Related Work

Related work includes regularization, capacity expansion, and rehearsal-based approaches. Generative replay avoids exemplar storage but inherits challenges from generating useful images, motivating feature-generation alternatives.

  • Continual-learning methods use regularization, capacity expansion with masks, or rehearsal to mitigate catastrophic forgetting.Rehearsal reuses samples from previous tasks while training on current-task data.
  • Rehearsal methods either store real exemplars or generate synthetic samples from previously learned distributions.The number of stored exemplars decreases as the memory budget is allocated across more classes.
  • Exemplar-free generative replay uses mechanisms such as unconditional or class-conditional GANs, generative autoencoders, or feature generation.One prior feature-generation method assumes Gaussian data and relies on pretrained models.
  • GAN-based image generation is limited by training instability, high computational cost, large data requirements, and incomplete visual diversity.High-resolution generation does not guarantee sufficient discriminative variety for downstream tasks.
  • GANs may fail to generate high-quality images for downstream classifier training, especially in few-shot and zero-shot settings.
  • Adversarial feature generation has emerged as an alternative to image generation, particularly for few-shot learning.Prior approaches improve classification-oriented feature generation by combining GAN and VAE strengths.

3. Forgetting in feature extractor and classifier

The analysis examines how continual learning affects representations across CNN layers and how replay or distillation methods mitigate forgetting. Higher-level features show pronounced forgetting, while LwF improves classifier-level similarity but leaves intermediate feature correlations weak.

  • Forgetting analysis: CCA measures forgetting by comparing task activations in the current model with those in the task-specific optimal model.The figure tracks correlation over time vertically and across network layers horizontally.
  • Fine-tuning: Fine-tuning preserves correlation better in earlier layers, but correlation decreases with layer depth, indicating stronger forgetting in higher-level features.This pattern is observed when new tasks are learned without an explicit forgetting-prevention mechanism.
  • Learning without forgetting: LwF increases CCA similarity at the classifier and maintains higher previous-task accuracy than fine-tuning.Its probability distillation is applied using current-task inputs because previous-task data is unavailable.
  • Learning without forgetting: LwF leaves middle- and lower-level feature correlations similar to or below fine-tuning, possibly because probability distillation is too loose for intermediate features.The stated explanation is tentative: the passage says this pattern may be caused by the looseness of the constraint.
  • Generative image replay: Generative image replay uses class-conditional generated images from previous tasks, but image generation is computationally demanding and may produce insufficiently discriminative training data.The method combines replayed images with current data, while complex image distributions require deep generators that are difficult to train.
  • Proposed framework: The proposed framework combines feature generation in the classifier with feature distillation in the feature extractor to address forgetting across the network.The framework duplicates and freezes the previous model while training the current extractor and classifier with feature replay and distillation.

4. Feature distillation and generative feature replay

The method replaces image replay with feature replay to address class imbalance while combining classifier replay with feature distillation to limit forgetting. It uses a feature extractor, classifier, and feature generator, with Gaussian and conditional-GAN replay variants.

  • Feature replay replaces difficult image generation by modeling and sampling class-conditional feature distributions for future tasks.The generator produces features û = G_t(c, z) for previous classes; Gaussian prototypes and conditional GANs provide two variants.
  • Feature distillation constrains the current feature extractor to preserve features produced by the previous extractor.The paper applies an L2 distillation loss to extracted features and can apply it to tensor features as well.
  • The framework duplicates and freezes previous feature extractor and classifier states, trains the current extractor and classifier, then trains the feature generator.Algorithm 1 initializes the first task directly and subsequently trains the classifier and extractor with current data plus replayed features before updating the generator.
  • Conditional GAN replay uses Wasserstein adversarial training and replay alignment so previous classes reproduce corresponding features under the same class and latent input.The replay alignment loss encourages G_t to match G_{t−1} for a previous class c and latent vector z.
  • The classifier is trained with current-task data and generated features from previous classes, avoiding separate task-specific losses and reducing classifier imbalance.Because the feature representation is shared across tasks, the method uses a single expanded classifier and trains only its linear layer during replay.
  • The combined distillation-and-replay model preserves representations across layers and maintains higher accuracy across tasks while addressing forgetting and task aggregation.The reported representation comparison includes the classifier as well as earlier layers.

5. Experimental results

Experiments evaluate class-incremental learning on ImageNet-Subset and CIFAR-100 using average accuracy and average forgetting across different task counts. The method achieves strong performance without storing exemplars and uses substantially less memory than exemplar-based alternatives.

  • Evaluation metrics: Average overall accuracy averages accuracies across tasks seen so far, while average forgetting averages each task’s peak-to-current accuracy decrease.These are the two primary evaluation metrics used in the experiments.
  • Experimental setup: Experiments use ImageNet-Subset and CIFAR-100, with half the classes in the first task and the remainder split into 5, 10, or 25 tasks.ImageNet images are resized to 256×256 with 224×224 training crops; CIFAR-100 uses padded 32×32 crops and horizontal flips.
  • ImageNet-Subset: On ImageNet-Subset, generative feature replay achieves the best average accuracy and forgetting in every tested setting without storing exemplars.Its advantage over Rebalance increases as the number of tasks grows, and generated features are combined dynamically with current data.
  • CIFAR-100: On CIFAR-100, generative feature replay outperforms iCaRL, LwF, MAS, and EWC, matches Rebalance comparably for 5 and 10 tasks, and is slightly worse for 25 tasks.The paper attributes the 25-task difference possibly to lower-quality features from low-resolution images.
  • Storage requirements: Exemplar methods require 6.2MB to 375MB for 100 classes, whereas the proposed generator and discriminator require a constant 4.5MB.For 256×256×3 images, 4.5MB is equivalent to only 24 total exemplars; full ImageNet-1000 would require 3.8GB for 20 samples per class.
  • Ablation: Replaying shallow features produces dramatically lower performance than replaying high-level linear features from Block 4.The paper links this to greater generation complexity for convolutional lower-level features and difficulty keeping head parameters unbiased.

6. Conclusions

The method combines generative feature replay with feature distillation, using simpler high-level feature distributions to provide an efficient approach that scales to large datasets.

  • High-level feature distributions are simpler than pixel-level distributions and can be modeled with simpler generators trained on limited samples.
  • The method is computationally efficient and scalable to large datasets.
  • Experiments on ImageNet and CIFAR-100 report large-margin gains over methods without exemplars and improvements over exemplar-based methods in several settings.The feature generator has small overhead compared with exemplar storage.

A. Comparative analysis on ImageNet-1000

On ImageNet-1000, the proposed method outperforms iCaRL across 5, 10, and 25 tasks, exceeds Rebalance increasingly with more tasks, and achieves lower forgetting with much less storage.

  • The proposed method outperforms iCaRL in average accuracy across 5, 10, and 25 tasks.
  • The method achieves slightly better accuracy than Rebalance in 5 tasks, with the gap widening in 10 and 25 tasks.
  • More than 10% lower average forgetting is reported relative to all compared methods.
  • 4.5 Mb of generator and discriminator storage replaces the 3.8 Gb required by exemplar-based iCaRL and Rebalance training.

B. Ablation study on different regularization

Feature distillation outperforms MAS and EWC in the CIFAR-100 ablation, indicating that constraining features is more effective than constraining parameters for keeping generated features close to real ones.

  • Feature distillation clearly outperforms MAS and EWC in the feature-extractor regularization comparison.
  • Constraining features is superior to constraining parameters in this ablation.
  • Feature constraints ensure that generated features are closer to real features.

C. T-SNE on generated features

The T-SNE visualization shows generated and real feature distributions as very close, supporting joint classifier training with generated features and current data.

  • Generated feature distributions are very close to distributions of real features extracted from images.
  • The similarity allows the classifier to be trained jointly with current data and generated features.
  • Clusters in the visualization represent distributions of different classes.

D. Architecture details

The generator uses two hidden layers of 512 neurons with LeakyReLU, while Figure 6 compares accuracy and forgetting across task settings and exemplar usage; Figure 7 contrasts real and generated features.

  • Network architecture: The generator and discriminator each use two hidden layers of 512 neurons followed by LeakyReLU with parameter 0.2.The generator additionally receives 200-dimensional Gaussian noise concatenated with one-hot vectors.
  • Evaluation figures: Figure 6 compares average accuracy in the top panel and average forgetting in the bottom panel on ImageNet-1000.The first task contains half the classes, while the remaining classes are split into 5, 10, or 25 tasks.
  • Evaluation figures: Figure 6 distinguishes methods without exemplars by symbol-marked lines from methods using 20000 exemplars by lines without symbols.This encoding supports comparison of exemplar-free and exemplar-based methods across task settings.
  • Feature visualization: Figure 7 shows real features in red and generated features in blue for the first task after all tasks are trained.The comparison is shown for 5-, 10-, and 25-task settings on ImageNet-Subset.
Loading 2004.09199v1…