Source-linked AI summary
Lifelong GAN: Continual Learning for Conditional Image Generation
Mengyao Zhai, Lei Chen, Fred Tung, Jiawei He, Megha Nawhal, Greg Mori
TL;DR
Lifelong GAN studies how generative models can learn new conditional generation tasks without forgetting previous ones when only current-task data are available. It uses knowledge distillation from previous networks and validates the framework for image-conditioned and label-conditioned generation, with results showing retention of earlier tasks and a 10% classification-accuracy and 25% reverse-classification-accuracy advantage over memory replay.
Problem
Deep networks can catastrophically forget previous tasks, creating a continual generative-learning problem when training data from earlier tasks are unavailable.
Method
Lifelong GAN distills information from a previously trained network into a new network while training on the current conditional generation task.
Results
Lifelong GAN learns current image-conditioned tasks while retaining previous-task generation, and outperforms memory replay by 10% in classification accuracy and 25% in reverse classification accuracy.
Takeaways & Limitations
Knowledge distillation provides a generic continual-learning framework spanning image-conditioned and label-conditioned generation under current-task supervision.
Takeaways & Limitations
The setting assumes access to previous-task models but not previous-task data, and image-conditioned experiments compare only with joint learning and sequential fine-tuning because replay is inapplicable.
Abstract
from arXiv · showhide
Lifelong learning is challenging for deep neural networks due to their susceptibility to catastrophic forgetting. Catastrophic forgetting occurs when a trained network is not able to maintain its ability to accomplish previously learned tasks when it is trained to perform new tasks. We study the problem of lifelong learning for generative models, extending a trained network to new conditional generation tasks without forgetting previous tasks, while assuming access to the training data for the current task only. In contrast to state-of-the-art memory replay based approaches which are limited to label-conditioned image generation tasks, a more generic framework for continual learning of generative models under different conditional image generation settings is proposed in this paper. Lifelong GAN employs knowledge distillation to transfer learned knowledge from previous networks to the new network. This makes it possible to perform image-conditioned generation tasks in a lifelong learning setting. We validate Lifelong GAN for both image-conditioned and label-conditioned generation tasks, and provide qualitative and quantitative results to show the generality and effectiveness of our method.
1. Introduction
Lifelong learning in deep neural networks is challenged by catastrophic forgetting, especially when models must learn new tasks with access only to current-task data. Lifelong GAN addresses this setting with knowledge distillation and evaluates continual conditional generation across input types and domains.
- Catastrophic forgetting prevents a trained network from maintaining previously learned task abilities after adapting to new tasks.
- Jointly retaining all previous training data is not scalable because storage requirements and training time grow without bound as tasks accumulate.Previous-task data may also be private or privileged.
- Continual learning must extend a model to new tasks without forgetting previous tasks when only current-task training data are available.The setting assumes access to a model trained on previous tasks but not to their data.
- Lifelong GAN uses knowledge distillation to transfer information from previously trained networks to a new network while learning the current task.The networks are encouraged to produce similar output values or visual patterns.
- The framework targets both image-conditioned and label-conditioned image generation and is validated with qualitative and quantitative results across diverse data domains.
2. Related Work
Prior work addressed catastrophic forgetting mainly in discriminative models and, for generative models, relied on parameter regularization or memory replay. Lifelong GAN introduces knowledge distillation for continual generative learning, including image-conditioned generation where replay is not applicable.
- Conditional image generation includes image-conditioned generation and label-conditioned generation, supporting tasks such as image-to-image translation and other visual transformations.
- Knowledge distillation transfers knowledge from a teacher to a student, including knowledge arising from learning resources unavailable to the student.This work focuses on transferring knowledge when the teacher has seen more input data.
- Continual-learning research has widely studied discriminative tasks using distillation losses, attention mechanisms, and parameter regularization.
- Generative continual-learning methods using EWC and memory replay have shown limited capability in remembering previous categories and generating high-quality images.
- Lifelong GAN introduces knowledge distillation into continual generative learning and applies it to both image-conditioned and label-conditioned generation.The paper notes that replay is not applicable to image-conditioned generation.
3. Approach
Lifelong GAN extends BicycleGAN for continual conditional image generation by distilling knowledge from the previous model into the current model while training only on current-task data. It applies distillation across both cVAE-GAN and cLR-GAN cycles, using auxiliary data to address conflicting objectives.
- BicycleGAN: Lifelong GAN builds on BicycleGAN, whose cVAE-GAN cycle encodes a ground-truth image and reconstructs it from the conditional image and latent code.The cVAE-GAN losses include image reconstruction, latent-distribution regularization, and adversarial training.
- BicycleGAN: Its cLR-GAN cycle generates an image from conditional data and a latent code, then reconstructs the latent code to enforce latent-code utilization.The cycle combines latent reconstruction with adversarial training.
- Problem formulation and overview: During task t, model M_t learns the current task while preserving prior tasks by distilling outputs from M_{t−1} into M_t.Both models process current-task data, and their outputs are encouraged to remain similar.
- Knowledge distillation: Knowledge distillation is applied to both BicycleGAN cycles: encoders and generators are matched in cVAE-GAN, while generators and encoders are matched in cLR-GAN.The cVAE-GAN loss compares encoded latent codes and reconstructed images; the cLR-GAN loss compares generated images and reconstructed latent codes.
- Knowledge distillation: L1 distillation losses are used instead of L2 to avoid blurriness in generated images, with β controlling the knowledge-distillation loss weight.The overall Lifelong GAN objective combines the cVAE-GAN and cLR-GAN losses with their distillation terms.
- Conflict removal with auxiliary data: Auxiliary data is introduced to remove conflicts between current-task reconstruction objectives and old-model matching objectives, with new auxiliary data required for each task.Montage and Swap can generate auxiliary images without external data sources, and categorical codes from previous tasks can serve as auxiliary inputs for label-conditioned generation.
4. Experiments
Lifelong GAN is evaluated for image-conditioned and label-conditioned continual image generation, including MNIST segmentation, image-to-image translation, and label-conditioned digit tasks. Across these settings, it learns new tasks while retaining earlier capabilities, outperforming or matching relevant baselines in qualitative and quantitative comparisons.
- Experimental setup: Lifelong GAN is evaluated for both image-conditioned and label-conditioned image generation, with comparisons against joint learning, sequential fine-tuning, and memory replay where applicable.Memory replay is not applicable to image-conditioned generation without ground-truth conditional inputs.
- Experimental setup: The experiments use Acc, r-Acc, and LPIPS to assess generated-image quality and diversity.Acc and r-Acc measure classifier-based quality, while higher LPIPS indicates greater diversity and values closer to real images indicate more realistic generation.
- Image-conditioned image generation: For MNIST digit segmentation, Lifelong GAN learns the current task without forgetting prior tasks, whereas sequential fine-tuning cannot segment digits 0–5 from earlier tasks.The experiment divides digits into three colored groups, producing three sequential image-conditioned tasks.
- Image-conditioned image generation: For edges →shoes followed by segmentations →facades, Lifelong GAN learns the new translation while remembering the old task and maintaining prior-task diversity; sequential fine-tuning forgets the old task.The two models are initialized from the same Task1 model, and the second task uses all available images while the first uses approximately 20,000 pairs.
- Label-conditioned image generation: In label-conditioned MNIST generation, Lifelong GAN and memory replay produce visually similar results to joint learning, while sequential fine-tuning exhibits catastrophic forgetting.The label-conditioned experiment uses four sequential tasks based on groups of MNIST digits.
- Label-conditioned image generation: Lifelong GAN outperforms memory replay by 10% in classification accuracy and 25% in reverse classification accuracy, while memory replay loses diversity during sequential learning.A user study further finds that participants prefer Lifelong GAN over memory replay, although it is not on par with joint learning.
5. Conclusion
Lifelong GAN uses knowledge distillation to extend one generative network to new tasks while retaining previous-task capabilities using only current-task supervision. The framework supports both image-conditioned and label-conditioned generation, including image-to-image translation beyond memory replay methods.
- Lifelong GAN transfers learned knowledge from previous networks to a new network through knowledge distillation during training.
- The framework extends a single network to new generative tasks without forgetting previous tasks under current-task-only supervision.
- Lifelong GAN supports both image-conditioned and label-conditioned generation tasks.
- The framework enables broader generation tasks, including image-to-image translation, that memory replay methods cannot support.