PaperlayerView exact revision on arXiv

Source-linked AI summary

GAN Memory with No Forgetting

Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, Lawrence Carin

arXiv:2006.07543v2cs.CVcs.LG

TL;DR

Catastrophic forgetting reflects the difficulty of retaining inaccessible historical data during lifelong learning. The paper proposes GAN memory, which sequentially style-modulates a base GAN to remember datasets and reports no forgetting in its generative memory and improved lifelong classification replay.

  • Problem

    Lifelong learning requires models to acquire new knowledge without forgetting prior experiences, but historical data may be inaccessible and existing generative replay can be blurry or poorly scalable.

  • Method

    GAN memory sequentially modulates the style of a base GAN with task-specific parameters, transferring frozen source knowledge to targeted generative models.

  • Results

    The authors report that GAN memory grows generative power with no forgetting and alleviates catastrophic forgetting in challenging lifelong classification tasks.

  • Takeaways & Limitations

    GAN memory can provide realistic generative replay for high-dimensional data and long, varying task sequences without saving original data.

  • Takeaways & Limitations

    Performance depends on the source model, with a better pretrained model expected to improve results; some inexpensive style parameters may remain task-specific.

Abstract

from arXiv · show

As a fundamental issue in lifelong learning, catastrophic forgetting is directly caused by inaccessible historical data; accordingly, if the data (information) were memorized perfectly, no forgetting should be expected. Motivated by that, we propose a GAN memory for lifelong learning, which is capable of remembering a stream of datasets via generative processes, with \emph{no} forgetting. Our GAN memory is based on recognizing that one can modulate the "style" of a GAN model to form perceptually-distant targeted generation. Accordingly, we propose to do sequential style modulations atop a well-behaved base GAN model, to form sequential targeted generative models, while simultaneously benefiting from the transferred base knowledge. The GAN memory -- that is motivated by lifelong learning -- is therefore itself manifested by a form of lifelong learning, via forward transfer and modulation of information from prior tasks. Experiments demonstrate the superiority of our method over existing approaches and its effectiveness in alleviating catastrophic forgetting for lifelong classification problems. Code is available at https://github.com/MiaoyunZhao/GANmemory_LifelongLearning.

1 Introduction

The paper frames catastrophic forgetting as a memory problem and proposes GAN memory: sequential style modulation of a base GAN to retain and extend generative capabilities.

  • Catastrophic forgetting occurs when learning new information severely interferes with previously learned knowledge in continual learning.
  • The paper seeks a realistic generative memory with growing expressive power to support general lifelong learning.
  • GAN memory transfers and freezes a base GAN, then uses mFiLM and mAdaFM style parameters to adapt its layers to perceptually distant target domains.
  • GAN memory retains existing capabilities while adding generative power through limited task-specific style parameters, which can be further compressed.
  • The method is generalized conditionally and evaluated for realistic replay that alleviates catastrophic forgetting in challenging lifelong classification tasks.

2 Related work

Prior lifelong-learning methods include regularization, dynamic-model, and generative-replay approaches, but generative replay remains limited by image quality and scalability.

  • Existing lifelong-learning methods are grouped into regularization-based, dynamic-model-based, and generative-replay-based approaches.
  • Generative replay is viewed as an effective and general strategy, but many methods produce blurry or distorted images or face scalability issues.
  • A better source model, such as one pretrained on a large-scale dense and diverse dataset, is expected to improve performance.
  • MeRGAN still shows decreased performance on historical tasks, whereas GAN memory is described as realistic, non-forgetting, and scalable to long high-dimensional task sequences.

3 Preliminary

The paper reviews GANs and style-transfer methods as the foundations for modulating latent features or convolutional filters to control generative behavior.

  • GANs use adversarially trained generators and discriminators to synthesize realistic observations from a latent distribution.
  • Style-transfer methods manipulate image texture or attributes by modulating latent-feature statistics, including through FiLM-style affine transformations.
  • FiLM applies element-wise affine transformations using scale γ and shift β, with the transformed feature forwarded to the next layer.
  • AdaFM instead modulates source convolutional filters to alter their style and improve transfer performance.

4 Proposed method

The method builds a GAN memory by freezing a well-behaved source GAN and sequentially learning compact, task-specific style modulations for distant target domains. This preserves source knowledge while enabling realistic generation, interpolation, replay, and compression across tasks.

  • 4.1 A surprising discovery: Style modulation transfers source GAN knowledge to perceptually distant domains by changing function-level generation characteristics rather than retraining the full model.The approach treats kernel statistics as a transferable dimension distinct from kernel shape, enabling adaptation beyond conventional style transfer.
  • 4.1 A surprising discovery: mFiLM adapts fully connected layers and mAdaFM adapts convolutional layers by learning target-specific style parameters while freezing source weights and biases.The modulation removes source statistics and learns target scale, shift, bias, or convolutional transformation parameters.
  • 4.2 GAN memory to sequentially remember a stream of generative processes: Sequentially modulating a frozen source GAN with task-specific parameters forms realistic generators for a stream of target datasets without forgetting prior tasks.The source parameters remain fixed, while each target receives separate style parameters that can be retained for replay.
  • 4.2 GAN memory to sequentially remember a stream of generative processes: The three style-parameter groups control distinct generation perspectives: scales affect texture and structure, shifts affect low-frequency color, and biases affect illumination and localized objects.Different blocks also specialize, with early modulations changing high-level domain structure and later blocks refining generation details.
  • 4.2 GAN memory to sequentially remember a stream of generative processes: GAN memory supports smooth interpolation between generative processes, allowing style parameters from different domains to be dynamically combined for data augmentation.The paper demonstrates interpolation between flower and cat generation and connects this property to downstream robust classification.
  • 4.3 GAN memory with further compression: Keeping 80% matrix energy, approximately 35% of top singular values, preserves nearly the same performance for Γ and B in blocks B0-B4.These parameters are generally low-rank, especially near the noise input, making them compressible for long task sequences.

5 Experiments

Experiments evaluate GAN memory for realistic generation, lifelong classification, and parameter compression across challenging image-task streams. The method maintains performance better than replay and regularization baselines while retaining comparable compressed-generation quality.

  • Experimental scope: The experiments cover realistic generative replay, conditional-GAN pseudo rehearsal for classification, and compression for long sequences of related generation tasks.The evaluation uses high-dimensional image datasets from diverse fields and includes six streaming classification tasks.
  • Lifelong classification: The six classification tasks use fish, bird, snake, dog, butterfly, and insect images from ImageNet, with six categories introduced per task.After task t, the classifier must recognize all first 6t categories.
  • Lifelong classification: Our conditional-GAN memory maintains increasing accuracy across 36 class-incremental categories, performs close to joint training, and outperforms EWC and MeRGAN on the long task sequence.EWC barely works, while MeRGAN degrades as its rehearsal becomes increasingly blurry.
  • Parameter compression: Compression substantially reduces newly introduced style parameters while preserving performance comparable to the naive GAN-memory implementation.Table 1 compares compressed and uncompressed variants, with 52.2M frozen source parameters.

6 Conclusions

The paper concludes that style modulation can support realistic generation for perceptually distant targets and a GAN memory that grows without forgetting. It also identifies compression and generative replay as useful directions while noting a dependence on the quality of the base model.

  • 6 Conclusions: The proposed GAN memory sequentially modulates a well-behaved base GAN to remember generative processes with growing generation power and no forgetting.The source model remains frozen while target-specific style parameters form new generation models.
  • 6 Conclusions: The paper reports compression techniques and empirical advantages over existing methods, while suggesting generative replay for longer base-model updates.The conclusion connects this update strategy to a sleep-inspired long-period update.

Broader impact

The paper describes GAN memory as potentially useful for replay, privacy, and controllable retention of model capabilities. It also warns that GAN-based systems may inherit risks from synthetic media and adversarial training.

  • Broader impact: GAN memory may support realistic generative replay for challenging lifelong applications such as self-driving while avoiding storage of original data.The paper presents privacy protection as a potential societal benefit because original data need not be saved.
  • Broader impact: Flexible control over replayed contents may help with unbalanced training data and selective retention or forgetting of model capabilities.This is presented as a potential practical value of GAN memory.
  • Broader impact: Because GAN memory is built on GANs, it may inherit risks including misuse for synthetic fake images, news, or videos and failures of adversarial training.The passage explicitly cautions that GANs can have negative ethical and societal consequences.

A Experimental settings

The experiments use a pretrained GP-GAN architecture and compare GAN memory with Fine-tuning while analyzing style modulation, interpolation, and normalization or bias ablations.

  • Experimental setup: GAN memory inherits GP-GAN architecture and settings, using a CelebA-pretrained model with style modulation applied to nearly all generator and discriminator layers.The generator excludes its last convolutional layer, and the discriminator excludes its last fully connected layer.
  • Baseline: Fine-tuning updates all GP-GAN parameters on each target dataset and serves as a strong baseline because it uses the whole model capacity.
  • Style modulation: Target-domain style parameters are selectively substituted for source parameters to test modulation through scales, shifts, biases, individual layers, or all components.The settings include “None,” “γ/Γ Only,” “β/B Only,” “All,” and variants omitting biases or targeting FC and residual blocks.
  • Ablation: The ablations remove weight normalization or bias terms to measure their separate contributions to GAN memory.

E More results for Section 5.1

The lifelong generation experiment evaluates GAN memory and MeRGAN on six sequential datasets spanning flowers, buildings, animals, medical images, and anime.

  • Sequential generation: The task stream contains Flowers, Cathedrals, Cats, Brain-tumor images, Chest X-rays, and Anime images, with only the current dataset available at each step.At task t, both methods are expected to generate samples for all learned datasets D1 through Dt.
  • Comparison: GAN memory produces realistic current and historical generations, whereas MeRGAN becomes increasingly blurry along the task sequence.

F Experimental settings for lifelong classification

The classification study uses class-incremental ImageNet tasks and compares replay-based training with EWC under a shared sequential-learning protocol.

  • Benchmark: The benchmark contains six ImageNet tasks, each covering six categories, requiring classification of all 6t categories after task t.Across the stream, the benchmark contains 36 categories, with 1,200 training and 100 test images per category.
  • Replay protocol: GAN memory and MeRGAN train on current real samples plus generated replay for previous tasks, using label-conditioned sampling of reliable (x, y) pairs.
  • Evaluation: The classification curves report per-task accuracy throughout training, while the aggregate evaluation uses all 3,600 test images across the 36 categories.
  • Results: GAN memory stably maintains historical classification accuracy over long sequences, while EWC rapidly forgets early tasks and MeRGAN declines because rehearsal is blurry.When training reaches D6, EWC has totally forgotten knowledge from D1, D2, and D3; MeRGAN degradation is especially clear on the early tasks.
  • Compression analysis: The compression experiments analyze low-rank style parameters by singular values, matrix-energy truncation, and task-specific basis growth.The procedure applies SVD and retains selected singular values while collecting task-specific coefficients and shared knowledge bases.

G.2 Experiments on the compressibility of the GAN memory

The compressibility experiments test shared low-rank representations for GAN memory parameters and examine how compression changes as related tasks are added.

  • Compressibility: Different GAN-memory blocks and layers have different compressibility, with larger scale and bias matrices generally allowing higher compression.The reported configuration keeps 80%, 80%, 90%, and 95% matrix energy for blocks B0 through B3.
  • Parameter growth: Figure 16 measures compression by the ratio of newly added bases to maximum rank and compares compressed with uncompressed parameter growth across six butterfly tasks.
  • Trade-off: Compression can reduce parameter growth, but selecting the matrix-energy threshold requires balancing memory savings against FID performance as task counts increase.The authors identify threshold selection as a future research direction because its negative effect on FID emerges for large task sequences.

H On the robustness of GAN memory to the source model

The method remains robust when the pretrained source GAN changes, achieving comparable Flowers performance and preserving observed modulation behavior. These results support generalization across well-behaved source models.

  • FID=15.0 with LSUN Bedroom as source is comparable to FID=14.8 with CelebA as source for Flowers generation.The comparison is presented as evidence that performance is robust to the pretrained source model.
  • The experiments support the view that GANs capture universal image structure that can manifest as different content or semantics under different styles.The passages further suggest that this structure is carried across various well-behaved source models.
  • The authors therefore believe GAN memory and its reported properties can generalize across different source models.The robustness discussion includes realistic replay on a sequence of six relatively related butterfly-image tasks.
  • Modulations in different GAN blocks exhibit different strength and focus over generation when using an LSUN Bedroom-pretrained source model.The same FC→B6 property is observed as FC→B5 after the architecture change.
Loading 2006.07543v2…