Source-linked AI summary
Generative Models from the perspective of Continual Learning
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, David Filliat
TL;DR
The paper asks which generative models and Continual Learning strategies best support sequential image generation without forgetting. It compares models and strategies across three benchmarks using two quantitative metrics, finding that original GAN with generative replay is especially effective on MNIST and Fashion MNIST, while CIFAR10 remains challenging.
Problem
Generative models in Continual Learning are less studied than classification models, and methods developed for discriminative models may not directly extend to generative settings.
Method
The study compares multiple generative models and four Continual Learning strategies on ten sequential class-generation tasks across MNIST, Fashion MNIST, and CIFAR10 using two quantitative metrics.
Results
Original GAN combined with generative replay performs best overall, with mean Fitting Capacity of 95.81% on MNIST and 81.52% on Fashion MNIST.
Takeaways & Limitations
Generative replay can use the generator as memory to maintain past knowledge, and original GAN is particularly effective on MNIST and Fashion MNIST.
Takeaways & Limitations
Generative models do not perform well enough for continual learning on CIFAR10 because generation errors accumulate.
Abstract
from arXiv · showhide
Which generative model is the most suitable for Continual Learning? This paper aims at evaluating and comparing generative models on disjoint sequential image generation tasks. We investigate how several models learn and forget, considering various strategies: rehearsal, regularization, generative replay and fine-tuning. We used two quantitative metrics to estimate the generation quality and memory ability. We experiment with sequential tasks on three commonly used benchmarks for Continual Learning (MNIST, Fashion MNIST and CIFAR10). We found that among all models, the original GAN performs best and among Continual Learning strategies, generative replay outperforms all other methods. Even if we found satisfactory combinations on MNIST and Fashion MNIST, training generative models sequentially on CIFAR10 is particularly instable, and remains a challenge. Our code is available online \footnote{\url{https://github.com/TLESORT/Generative\_Continual\_Learning}}.
I. INTRODUCTION
The paper examines generative models in Continual Learning, where models must learn changing data distributions without catastrophic forgetting. It compares models and CL strategies on sequential, disjoint image-generation tasks across MNIST, Fashion MNIST, and CIFAR10.
- Motivation: Continual Learning requires adapting to changing data distributions while retaining crucial previously learned information.Neural networks trained with back-propagation commonly suffer catastrophic forgetting when tasks change.
- Motivation: The study focuses on generative models because prior CL research mainly addressed classification and its methods may not transfer directly to generative objectives.Generative models can also serve as memories by producing samples from past tasks.
- Experimental scope: The experiments sequentially train models on ten disjoint tasks, each introducing one class while requiring generation from all previously seen classes.The same setup is applied to MNIST, Fashion MNIST, and CIFAR10.
- Experimental scope: The comparison covers VAEs, GANs, conditional variants, WGANs, and WGAN-GP with fine-tuning, rehearsal, regularization, and generative replay.Generative replay uses generated samples to maintain knowledge from previous tasks.
- Contributions: The paper contributes a broad evaluation of generative models, CL strategies, their success and failure modes, and two generative-model evaluation metrics.The contributions explicitly include comparing two quantitative metrics in a CL setting.
II. RELATED WORK
Related work frames Continual Learning around four strategies: rehearsal, regularization, dynamic architectures, and generative replay. These approaches address forgetting through stored samples, constrained updates, architectural changes, or generated past-task data.
- Continual Learning strategies: Fine-tuning sequentially across tasks leads to catastrophic forgetting of performance on earlier tasks.This background motivates methods that preserve knowledge while learning new tasks.
- Continual Learning strategies: Rehearsal stores samples from previous tasks, but it is unavailable when old data cannot be accessed and requires memory growing linearly with task count.It remains a competitive baseline despite these constraints.
- Continual Learning strategies: Regularization constrains parameter updates to preserve prior knowledge, with Elastic Weight Consolidation estimating parameter importance.Distillation is another established regularization approach.
- Continual Learning strategies: Dynamic architectures preserve past knowledge while learning new information by changing the model structure.Progressive Networks, Learning Without Forgetting, and PathNet exemplify this strategy.
- Continual Learning strategies: Generative replay uses a generative model to produce samples from previous tasks and is also called pseudo-rehearsal.It represents the most recent of the four strategy families described here.
B. Continual learning for generative models
Continual learning for generative models remains less explored than classification because generative and discriminative models differ in architecture and objective. The paper compares existing strategies and models using complementary generative-quality metrics.
- Generative continual learning is less explored than classification because generative and discriminative models differ in architecture and learning objective.
- Prior work includes EWC for CGANs, variational continual learning, student–teacher training, and VASE, but these methods often target restricted model types or task settings.
- Generative replay uses one model as a memory of past tasks and another to learn the combined past-and-current distribution, across adversarial and variational frameworks.
- This study searches for effective model–strategy combinations by comparing several existing continual-learning strategies across a wide range of generative models.
- The evaluation compares FID, based on feature-distribution statistics, with Fitting Capacity, based on a classifier trained from generated samples.
- Qualitative visualization can reveal failures quickly but cannot rigorously compare well-performing models or reliably assess sample variability.
IV. EXPERIMENTAL SETUP
The experimental setup defines the data, tasks, and evaluated approaches used to study continual generation.
- The paper introduces its experimental setup by specifying the data, sequential tasks, and approaches under evaluation.
A. Datasets, tasks, metrics and models
The experiments use sequential class-generation tasks across three benchmark datasets, comparing six generative models with continual-learning strategies and baselines.
- The experiments use 10 sequential tasks on MNIST, Fashion MNIST, and CIFAR10.
- Each task introduces one new class while requiring generation of that class and all previously learned classes; FID and Fitting Capacity are computed after every task.
- Six models are evaluated: original and conditional GANs and VAEs, plus WGAN and WGAN-GP.
- The study compares continual-learning strategies across eight seeds, using 50 epochs per task for MNIST and Fashion MNIST, while CIFAR10 uses the best-performing strategy.
- Fine-tuning is a lower-bound baseline, while joint-data and separate-per-task training provide upper-bound baselines.
- Rehearsal retains a fixed number of samples from each observed task, whereas EWC regularizes model weights and generative replay uses a frozen generative memory.
V. RESULTS
Fitting Capacity is emphasized because it is more stable than FID, and results favor adversarial models—especially GAN with generative replay—although task structure strongly affects regularization outcomes.
- Fitting Capacity is used for interpretation because it correlates with FID but has smaller standard deviations across eight seeds.
- The best combination reaches 95.81% Fitting Capacity on MNIST and 81.52% on Fashion MNIST with Generative Replay + GAN.
- Generative Replay significantly outperforms other approaches for adversarial models, whereas Rehearsal performs best for variational models.
- EWC fails to overcome catastrophic forgetting in this setting and performs like naive Fine-tuning, unlike results reported under a different task definition.
- The Fisher matrix cannot protect class information appropriately when task sequences begin or continue with single-class tasks, causing forgetting in conditional models.
- Adversarial methods perform significantly better than variational methods, while conditional and unconditional models show no clear overall distinction.
2) Corollary results:
On MNIST and Fashion MNIST, Generative Replay with GAN achieved the strongest reported fitting capacity, while model family affected which continual-learning strategy worked best. Conditional models also reached fitting-capacity levels comparable to continual-learning classification results, with an important evaluation caveat.
- VAE performance decreased across several classes with Generative Replay, whereas GAN performance did not; Rehearsal showed the opposite pattern.GAN-based models benefited from sample quality and stability, while VAE-based models were particularly effective and stable with Rehearsal.
- Fine-tuning and EWC produced higher-than-expected Fitting Capacity for unconditional models because fuzzy samples could be misannotated by the expert.This annotation effect can artificially increase label variability and the measured Fitting Capacity of weak models.
- 94.7% on MNIST and 75.44% on Fashion MNIST were the best reported conditional-model fitting-capacity results, achieved with CGAN.The paper describes these results as comparable to continual-learning classification results, while noting that the comparison is not fully fair because the expert annotator was not trained continually.
- Fig. 4 maps Fitting Capacity across task and class indices, with yellow indicating higher accuracy and blue indicating lower accuracy.The top panel shows GAN results and the bottom panel VAE results.
B. CIFAR10 results
On CIFAR10, sequential generative learning remained unstable: Fine-tuning forgot earlier tasks, Rehearsal overfit memory, and Generative Replay accumulated errors. These results indicate that continual learning for real-life image datasets remains challenging.
- The CIFAR10 experiment compared Generative Replay and Rehearsal with naive Fine-tuning and an Upperbound Model using WGAN-GP.Each task contained one category, and the goal was to avoid forgetting previously seen categories.
- All four methods failed to generate images that enabled a classifier to perform well on real CIFAR10 test data.The evaluation displayed samples after ten sequential tasks alongside FID and Fitting Capacity curves.
- Fine-tuning catastrophically forgot previous tasks, while Rehearsal produced unsatisfactory results and overfit the few stored samples, causing mode collapse.Although FID improved at each new task, visualizations showed copied samples and reduced diversity.
- Generative Replay failed on CIFAR10 as minor generation errors accumulated across tasks, producing blurry and indistinguishable categories by task 9.Its FID improved initially but deteriorated with each subsequent task.
- Training generative models sequentially on CIFAR10 does not reduce to training successfully on all data or on each category separately.The paper concludes that designing a continual-learning strategy for such datasets remains a challenge.
VI. DISCUSSION
The discussion contrasts the practical trade-offs and evaluation limits of continual-learning strategies for generative models. It concludes that GAN with Generative Replay works well on MNIST and Fashion MNIST, while CIFAR10 remains unsuitable because errors accumulate.
- Rehearsal violates data-availability assumptions and risks overfitting when few samples represent a task.
- EWC avoids training-time computational overload but requires the Fisher information matrix and storage for previous parameters.
- The evaluated conditional-model metric can expose catastrophic forgetting but may miss mode collapse and overfitting.
- Generative Replay uses generated samples as memory, helping maintain knowledge from previous tasks.
- GAN combined with Generative Replay is particularly effective on MNIST and Fashion MNIST, whereas accumulated generation errors make continual learning on CIFAR10 unusable.
APPENDIX
The appendix provides visual and metric-based evidence for model behavior across sequential tasks. It highlights GAN with Generative Replay on simpler datasets, while showing degradation and instability for some conditional and CIFAR10 settings.
- GAN with Generative Replay is presented as a well-performing solution across successive MNIST and Fashion MNIST tasks.
- Test-set classification accuracy as a function of training samples estimates the samples needed for high accuracy and generator fitting capacity.
- Fitting Capacity and FID are compared for MNIST and Fashion MNIST to assess generation performance.
- Rehearsal performs as well as Generative Replay under the proposed metric despite visually producing more mode collapse.
- With five sequential two-digit tasks, EWC performs well for CGAN and avoids catastrophic forgetting in the illustrated MNIST setting.
- When each task contains one class, the Fisher information matrix cannot capture the importance of the fixed class-index input, weakening EWC protection.
- At the end of ten sequential tasks, the figures compare GAN and CGAN samples across continual-learning strategies on MNIST and Fashion MNIST.
- WGAN-GP with Generative Replay on CIFAR10 shows errors snowballing across tasks until final samples become completely blurry.