Source-linked AI summary
Memory Replay GANs: learning to generate images from new categories without forgetting
Chenshen Wu, Luis Herranz, Xialei Liu, Yaxing Wang, Joost van de Weijer, Bogdan Raducanu
TL;DR
The paper asks how GANs can learn image categories sequentially without forgetting earlier categories. It proposes MeRGANs, which use a replay generator with joint training or replay alignment, and reports competitive generation with substantially reduced forgetting across MNIST, SVHN, and LSUN.
Problem
Sequential fine tuning makes GANs unable to properly generate images from previous categories.
Method
MeRGANs extend conditional GANs with a memory replay generator and use either joint training with replay or replay alignment.
Results
MeRGANs generate competitive images while significantly mitigating forgetting across MNIST, SVHN, and LSUN.
Takeaways & Limitations
Memory replay enables GANs to retain the ability to generate images from previous tasks after learning several new ones.
Abstract
from arXiv · showhide
Previous works on sequential learning address the problem of forgetting in discriminative models. In this paper we consider the case of generative models. In particular, we investigate generative adversarial networks (GANs) in the task of learning new categories in a sequential fashion. We first show that sequential fine tuning renders the network unable to properly generate images from previous categories (i.e. forgetting). Addressing this problem, we propose Memory Replay GANs (MeRGANs), a conditional GAN framework that integrates a memory replay generator. We study two methods to prevent forgetting by leveraging these replays, namely joint training with replay and replay alignment. Qualitative and quantitative experimental results in MNIST, SVHN and LSUN datasets show that our memory replay approach can generate competitive images while significantly mitigating the forgetting of previous categories.
1 Introduction
The paper frames sequential image generation as a setting where models must learn new categories without losing earlier capabilities. It focuses on conditional GANs and introduces memory replay as a way to address this forgetting.
- GAN foundations: GANs learn mappings from low-dimensional latent spaces to complex image distributions through an adversarial generator–discriminator game.The generator produces images, while the discriminator distinguishes generated samples from real training samples.
- Conditional generation: Unconditional GANs provide diverse samples but offer little control over semantic properties such as category.Conditional GANs add semantic inputs, with category labels providing direct control over generated-image categories.
- Sequential learning: Sequential model updates can cause severe degradation when earlier-task data are not revisited, a phenomenon known as catastrophic forgetting.Existing mitigation strategies include weight or activation regularization, exemplar storage, and memory replay.
- Paper focus: The paper studies forgetting in GANs when image categories arrive sequentially as disjoint tasks rather than focusing only on discriminative models.Generated images make forgetting and consolidation visually observable.
- Paper focus: MeRGANs use memory replay to prevent forgetting through joint retraining or replay alignment.Replay alignment synchronizes current-generator replays with those from an auxiliary pre-new-task snapshot.
2 Sequential learning in GANs
The paper first defines a conditional GAN for joint category learning, then adapts it to sequential category-specific tasks. Sequential fine tuning causes catastrophic forgetting, motivating replay and regularization baselines.
- 2.1 Joint learning: The baseline learns from category-specific image sets using a conditional GAN with generator, discriminator, and classifier components.The discriminator and classifier share layers except for task-specific final layers.
- 2.1 Joint learning: The generator receives a latent vector and category condition, while the auxiliary classifier encourages generated images to match their specified labels.Conditional batch normalization dynamically selects normalization parameters according to the category.
- 2.1 Joint learning: Training alternates generator optimization with discriminator-and-classifier optimization using adversarial and classification objectives.The adversarial objective uses the WGAN formulation with gradient penalty.
- 2.2 Sequential fine tuning: Sequential learning presents category-specific training sets as an ordered sequence of tasks, with each task initialized from the preceding task’s parameters.There is no classifier in this setting because each task contains data from only the current category.
- 2.2 Sequential fine tuning: Adjusting parameters toward a new domain causes catastrophic forgetting of previous tasks during sequential fine tuning.The paper notes that this behavior has also been observed in GANs.
- 2.2 Sequential fine tuning: EWC is used as a baseline that regularizes generator updates according to parameter sensitivity estimated by the Fisher information matrix.The regularization strength is controlled by λ_EWC.
3 Memory replay generative adversarial networks
MeRGANs extend conditional GANs with a replay generator that preserves memories of previous categories while learning a current task. The paper leverages these replays through joint retraining and replay alignment, with conditional inputs enabling controlled sampling.
- Memory replay: The framework adds a replay generator that samples memories of previous tasks during training on the current task.This active replay role replaces parameter-only regularization as the central mechanism for mitigating forgetting.
- Joint retraining with replayed samples: Joint training extends the current-task dataset with replayed samples and trains the network on the combined data.This approach is related to deep generative replay but uses category-conditioned generation for finer control and more reliable (x, c) sampling.
- Replay alignment: Replay alignment synchronizes current and replay generators by matching outputs for the same latent vector z and category c.The shared architecture, inputs, outputs, and initially aligned parameters enable pixelwise comparison of generated images.
- Replay alignment: In replay alignment, both generators replay all previous tasks, while the discriminator trains only on images from the current task.The generator objective includes a replay alignment loss, and the method is framed as aligned distillation.
- Replay alignment: Unlike classification distillation, the method addresses image generation, where spatially structured outputs and latent variability require aligned inputs and outputs.The latent vector captures variability such as pose, location, and color, allowing a given (z, c) pair to define a corresponding image comparison.
4 Experimental results
Experiments on MNIST, SVHN, and LSUN compare replay-based MeRGAN variants with sequential fine tuning and other baselines. MeRGANs preserve earlier categories more effectively while generating clearer images, though performance degrades in challenging settings.
- Experimental setup: MeRGAN-JTR and MeRGAN-RA were evaluated against joint training, sequential fine tuning, EWC, and DGR across digit and scene-generation tasks.The experiments used MNIST, SVHN, and LSUN with fixed training settings and dataset-specific complexity.
- Digit generation: SFT completely forgets previous digit tasks, whereas MeRGANs generate clearer and more recognizable images, especially on challenging SVHN.On MNIST, all methods generate digits properly, but MeRGANs produce sharper digits; on SVHN, MeRGAN outputs remain generally clear despite degradation from limited capacity.
- Digit generation: MeRGANs improve classification accuracy over baselines, including about 8% on MNIST and 21% on SVHN over DGR after five tasks.After ten tasks, the approach achieves about a 12% gain and remains more stable as the number of tasks increases.
- Scene generation: MeRGANs outperform EWC and DGR on LSUN using FID, direct classification accuracy, and reverse accuracy after four sequential tasks.FID captures both generated-image quality and diversity in this evaluation.
- Scene generation: Fine tuning forgets previous scene tasks within the first few iterations, while MeRGANs initially forget and then tend to recover during training.MeRGAN-RA appears more stable and slightly more effective than MeRGAN-JTR in the reported evolution.
- Scene generation: Replay alignment preserves specific bedroom instances, whereas joint training with replay preserves the bedroom category without necessarily reproducing the same instance.The distinction reflects instance-level versus category-level not-forgetting constraints.
Conclusions
The paper addresses catastrophic forgetting in sequential GAN-based image generation with MeRGANs, using memory replay enforced through joint training or replay alignment. Results show retention of competitive image generation for previous tasks after learning new ones.
- MeRGANs use memory replay to address catastrophic forgetting in sequential image generation with GANs.The framework applies memory replay through joint training or replay alignment.
- The two replay mechanisms are joint training and replay alignment.Joint training incorporates replays during training, while replay alignment enforces consistency through replay alignment.
- MeRGANs retain the ability to generate competitive images from previous tasks after learning several new ones.
- Image generation also provides a way to visualize task interference and potential forgetting directly through generated images.
A Reverse Classification Accuracy
Reverse classification accuracy evaluates whether generated data supports a classifier that generalizes to real data. Results vary by dataset and architecture: MNIST remains strong, SVHN can degrade substantially, and improved GAN architectures raise reverse accuracy.
- Reverse classification accuracy trains a classifier on generated data and evaluates it on the real dataset.Direct classification accuracy instead trains on real data and tests on generated images.
- 0.992 and 0.985 reverse accuracy are achieved by MeRGAN-JTR and MeRGAN-RA, respectively, on MNIST in the 5-task setting.In the 10-task setting, the corresponding accuracies are 0.968 and 0.939.
- 0.201 is the reverse classification accuracy of JTR on SVHN in the 10-task setting.
- 51.0 and 71.0 are the reverse classification accuracy results for JTR and RA, respectively, with the improved GAN architecture.The architecture uses ResNet-18, a projection-discriminator cGAN, and one-hot conditioning.