PaperlayerView exact revision on arXiv

Source-linked AI summary

Lifelong Generative Modeling

Jason Ramapuram, Magda Gregorova, Alexandros Kalousis

arXiv:1705.09847v7stat.MLcs.LG

TL;DR

Lifelong learning seeks to retain and reuse knowledge across sequential tasks, but many existing approaches require storing past data or models. This paper uses a student-teacher VAE with replay and a cross-model regularizer; on Celeb-A, its model generates samples from prior distributions while a VAE baseline forgets them.

  • Problem

    Existing lifelong-learning approaches often require preserving past data or models, while deep unsupervised generative learning over sequential tasks remains comparatively underexplored.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    A student-teacher VAE uses teacher-generated replay and functional regularization, including a KL divergence between teacher and student posteriors, to transfer knowledge.

    Source: §4.1 · pp. 7–8, §4.1.1 · p. 9

  • Results

    On Celeb-A, the lifelong model generated instances from previous distributions, whereas the baseline VAE generated only samples from the eyeglasses distribution.

    Source: §6.6 · p. 19

  • Takeaways & Limitations

    Generative replay coupled with functional regularization makes preserving past data and models unnecessary in the described approach.

    Source: §2.2 · p. 4

    The method does not completely solve lifelong generative modeling, and performance degrades slowly over time.

    Source: §8 · p. 21

Abstract

from arXiv · show

Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner, where knowledge gained from previous tasks is retained and used to aid future learning over the lifetime of the learner. It is essential towards the development of intelligent machines that can adapt to their surroundings. In this work we focus on a lifelong learning approach to unsupervised generative modeling, where we continuously incorporate newly observed distributions into a learned model. We do so through a student-teacher Variational Autoencoder architecture which allows us to learn and preserve all the distributions seen so far, without the need to retain the past data nor the past models. Through the introduction of a novel cross-model regularizer, inspired by a Bayesian update rule, the student model leverages the information learned by the teacher, which acts as a probabilistic knowledge store. The regularizer reduces the effect of catastrophic interference that appears when we learn over sequences of distributions. We validate our model's performance on sequential variants of MNIST, FashionMNIST, PermutedMNIST, SVHN and Celeb-A and demonstrate that our model mitigates the effects of catastrophic interference faced by neural networks in sequential learning scenarios.

1 INTRODUCTION

The paper motivates lifelong learning as a way to use prior tasks to support future learning, then applies it to deep unsupervised generative modeling. Its approach targets a setting that avoids retaining prior data and models.

1 INTRODUCTION

  • Unsupervised learning seeks patterns in input data without supervision, while learning assumptions or priors favor some plausible solutions over others.The passage gives image convolutions as an example of a prior favoring local structure.Source: §1 · p. 1
  • Lifelong learning retains representations and models from earlier tasks to aid current and future learning.Unlike single-task models, this setting reflects learning over a sequence of tasks.Source: §1 · pp. 1–2
  • This work addresses deep lifelong unsupervised learning, a more challenging and less explored setting than supervised lifelong learning.The authors also seek to remove the need to store past models and training data.Source: §1 · p. 2

2 RELATED WORK

The related work frames lifelong learning around retaining knowledge across sequential tasks and mitigating catastrophic forgetting. The paper identifies trade-offs in prior approaches and proposes a generative student-teacher model that avoids storing past data or models.

2 RELATED WORK

  • Online learning prioritizes fitting the current problem, whereas lifelong learning aims to retain and reuse past learning across tasks.Lifelong learning also differs from single-task learning by explicitly considering a sequence of tasks.Source: §2 · p. 2
  • Neural networks trained sequentially can forget earlier representations as their parameters become biased toward recent observations.The paper groups mitigation strategies into transfer learning, replay, parameter regularization, and distribution regularization.Source: §2 · p. 2

2.1 LIMITATIONS OF EXISTING APPROACHES.

  • Prior approaches commonly trade storing past data for storing past models or model parameters.Some methods require all or a growing coreset of data, while others retain previous models or parameter subsets.Source: §2.1 · p. 4

2.2 OUR SOLUTION AT A HIGH LEVEL.

  • The proposed method combines replay and functional regularization in teacher-student generative models to preserve past input-output relations.The teacher supplies replayed data to a student trained on both current-task and synthetic data.Source: §2.2 · p. 4
  • The approach avoids assumptions about the parameter-posterior form and is reported to improve sample complexity on future tasks.The authors also state that generative replay with functional regularization removes the need to preserve past models and data.Source: §2.2 · p. 4

3 BACKGROUND

The background introduces latent-variable generative modeling, extends it to sequentially observed tasks, and describes VAEs as the paper’s modeling framework. It outlines variational inference and the ELBO as tools for approximating latent posteriors and training the model.

3.1 LATENT VARIABLE GENERATIVE MODELING

  • A latent-variable generative model samples a latent variable from a prior, then generates an observed variate from a conditional distribution.The paper formulates this for a dataset of continuous or discrete observations.Source: §3.1 · p. 5

3.2 LIFELONG GENERATIVE MODELING

  • Lifelong generative modeling estimates a distribution from sequentially observed tasks without access to earlier task datasets.The goal is for the model’s estimate to approach the unknown distribution as additional tasks arrive.Source: §3.2 · p. 5

3.3 THE VARIATIONAL AUTOENCODER

  • Variational inference approximates an intractable latent posterior with a tractable distribution family and yields the evidence lower bound objective.The paper uses VAEs to infer latent variables and jointly estimate model parameters.Source: §3.1 · p. 5, §3.3 · p. 6
  • VAEs use an encoder for approximate inference and a decoder for the conditional, and optimize their parameters by maximizing the ELBO.Amortized inference shares encoder parameters across data points; a path-wise estimator supports gradients through posterior sampling.Source: §3.3 · pp. 6–7

4 LIFELONG LEARNING MODEL

Lifelong Generative Modeling uses a student–teacher VAE to retain prior distributions through replay and functional regularization. Its base model separates task identity from sample variation so the student can generate and learn from task-specific synthetic data.

4 LIFELONG LEARNING MODEL

  • The teacher acts as a probabilistic knowledge store, while the student learns the new task and accommodates the teacher’s representation of prior tasks.The paper frames this architecture as a lifelong-learning model inspired by replay during biological memory consolidation.Source: §4 · p. 7

4.1 STUDENT-TEACHER ARCHITECTURE

  • Teacher-generated replay augments current-task data, while posterior regularization transfers learned representations between teacher and student.After each task, the student is frozen as the next teacher, so the process does not add a new model per task.Source: §4.1 · pp. 7–8

4.1.1 KNOWLEDGE TRANSFER VIA BAYESIAN UPDATE.

  • The posterior regularizer matches student and teacher posteriors on teacher-generated data, and can be viewed as a VAE ELBO under reparameterized student parameters.The authors report that applying a similar regularizer to the reconstruction term hurts performance.Source: §4.1.1 · pp. 8–9

4.2 BASE-LEVEL GENERATIVE MODEL.

  • A vanilla VAE can mix distributions during replay or undersample them; the model instead factors latent variables into discrete and continuous components.The discrete component represents discriminative information about distributions, while the continuous component captures remaining sample variability; a mutual-information constraint encourages the decoder to use the discrete component.Source: §4.2 · pp. 9–10

4.2.1 CONTROLLED GENERATIONS.

  • Fixing the discrete latent to a task and sampling the continuous prior enables varied generations from that task conditional.This controlled sampling assumes the discrete latent captures each distribution’s discriminative information, but the decoder may ignore it.Source: §4.2.1 · pp. 10–11, Figure 6 · p. 10

4.2.2 INFORMATION RESTRICTING REGULARIZER

  • The information-restricting regularizer minimizes information between generated samples and the continuous latent, a choice reported as essential for empirical performance.The authors favor this over maximizing information with the discrete latent because it provides subtler gradient information.Source: §4.2.2 · p. 11

4.3 LEARNING OBJECTIVE

  • The final student objective combines the sequential VAE ELBO, generative replay, and mutual-information regularization.The λ setting trades sample diversity against use of the discrete latent; the authors chose λ = 0.01 for their experiments.Source: §4.3 · pp. 11–12

4.4 COMPUTATIONAL COMPLEXITY

  • In the most demanding experiment, training epochs increased by an average of 13.53 seconds and test epochs by 6.3 seconds.The added costs include teacher generations, posterior regularization, and an extra encode operation; neural-network matrix-vector products still dominate computation.Source: §4.4 · p. 12

5 REVISITING STATE OF THE ART METHODS.

The section contrasts LGM’s posterior-distribution regularization with parameter-based methods EWC and VCL, then compares LGM with VASE. It emphasizes LGM’s unsupervised inference of discrete latents and its information-restricting regularizer.

EWC and VCL

  • LGM regularizes the output latent distribution Qφ(z|x), whereas EWC and VCL constrain parameters under isotropic-Gaussian assumptions about the parameter posterior.The paper notes active research on relaxing these assumptions [74, 75, 80].Source: §5 · pp. 12–13
  • For an isotropic-Gaussian latent posterior demonstration, LGM can freely change parameters φ while minimizing an analytic KL, unlike EWC’s direct quadratic parameter constraint.The method can operate over distributions with tractable KL divergence.Source: §5 · p. 13

VASE

  • Unlike VASE’s classification-based posterior heuristic, LGM’s objective yields unsupervised inference of the discrete posterior Qφ(zd|x).The section also states that decoding-distribution regularization helps with direct supervised class information but fails in a purely unsupervised generative setting.Source: §5 · p. 13
  • LGM adds an information-restricting regularizer that directly controls the interpretation and flow of information in learned latent variables.The section contrasts this with VASE.Source: §5 · p. 13

6 EXPERIMENTS

The experiments evaluate lifelong generative modeling across sequential datasets using likelihood, ELBO, Fréchet distance, sample complexity, and runtime measures. They report retained generations across tasks, reduced use of real samples over time, and performance across varied distributions.

6.1 PERFORMANCE METRICS

  • Evaluation combines importance-sample log-likelihood, negative test ELBO, and Fréchet distance with qualitative samples because no metric conclusively captures generative quality.Importance-sample log-likelihood is used only for MNIST and FashionMNIST because of its computational cost.Source: §6.1 · pp. 13–14, Table 3 · p. 14

6.2 DATA FLOW

  • Training uses samples from the current task, while each test set accumulates the current and all previous tasks.The experiments include sequential MNIST and FashionMNIST, permuted MNIST, Celeb-A distributions, and an SVHN-to-MNIST transition.Source: Figure 7 · p. 14, §6.2 · p. 15

6.3 SITUATING AGAINST STATE OF THE ART LIFELONG LEARNING MODELS.

  • LGM has smoother performance across tasks despite sharp estimate increases when the model encounters a substantially different distribution.The authors posit that functional regularization, rather than parameter constraints, accounts for the smoother performance.Source: Figure 8 · p. 15, §6.3 · p. 16
  • Real samples used by LGM decrease steadily over time, supporting reduced sample complexity in the lifelong setting.The estimate uses the stopping epoch and the student’s Bernoulli sampling proportion.Source: §6.3 · p. 17, Figure 10 · p. 17

6.4 DIVING DEEPER INTO THE SEQUENCE.

  • LGM drastically outperforms EWC and the naive sequential VAE on both negative test ELBO and log-Fréchet distance in PermutedMNIST.The final reconstructions show different degradation patterns: salt-and-pepper noise for EWC and Gaussian noise for LGM.Source: Figure 11 · p. 17, §6.4 · p. 18, Figure 12 · p. 18

6.5 LEARNING ACROSS COMPLEX DISTRIBUTIONS

  • After training only on MNIST, the final model can still reconstruct previously observed SVHN data.The experiment transitions between colored real-world house-number images and binary handwritten digits.Source: §6.5 · p. 18, Figure 13 · p. 18
  • Interpolating the continuous latent space produces samples that exhibit a common continuous structure across the two distributions.The visualized interpolations use each of the two discrete latent settings.Source: §6.5 · pp. 18–19, Figure 13 · p. 18

6.6 VALIDATING EMPIRICAL SAMPLE COMPLEXITY USING CELEB-A

  • On Celeb-A, the lifelong model generates samples from all four distributions, while the baseline generates only from eye-glasses.The authors also report no significant overhead for the lifelong model on the 44,218-sample male distribution.Source: §6.6 · p. 19, Figure 14 · p. 19, Table 4 · p. 19

7 ABLATION STUDIES

The ablations examine the discrete-continuous posterior factorization and the posterior-consistency and mutual-information regularizers. The combined posterior reaches 87.1% linear test accuracy, while generations without both regularizers are blurry.

7.1 LINEAR SEPARABILITY OF DISCRETE AND CONTINUOUS POSTERIOR

  • The separability test trains linear classifiers on encoded posterior samples from VAEs with discrete-only, Gaussian-only, Bernoulli-only, and combined discrete-plus-Gaussian posteriors.For the combined posterior, the discrete dimension is fixed at 10 while the Gaussian dimension varies with total latent size.Source: §7.1 · p. 20, Figure 16 · p. 20
  • The discrete-plus-Gaussian posterior reaches 87.1% upper-bound linear test-classification accuracy, outperforming the discrete-only, Gaussian-only, and Bernoulli-only posteriors.The authors interpret this as evidence that decoupling the latent representation aids a more meaningful, separable posterior.Source: §7.1 · p. 20

7.2 VALIDATING THE MUTUAL INFORMATION AND POSTERIOR CONSISTENCY REGULARIZERS.

  • The regularizer ablation compares both regularizers, posterior consistency alone, and neither, using negative test ELBO and generated samples.The evaluation uses the MNIST task sequence.Source: §7.2 · p. 20, Figure 17 · p. 21
  • The posterior-consistency and mutual-information regularizers are both necessary for high-quality samples in the MNIST ablation.Without both, generations are blurry; the authors attribute this to unrepresentative discrete sampling and the decoder routing more information through the continuous component, causing catastrophic forgetting and posterior collapse.Source: §7.2 · pp. 20–21, Figure 17 · p. 21

8 LIMITATIONS

The method’s limitations include gradual performance degradation, reliance on known distribution boundaries, blurry VAE generations, and a proposed need for decoupled global memory.

8 LIMITATIONS

  • Model performance degrades slowly over time, which the authors attribute mainly to poor VAE generations compounding upon each other.The authors identify non-degrading Frechet distance and -ELBO as goals for an optimal unsupervised generative lifelong learner.Source: §8 · p. 21
  • The method assumes known, fixed distribution boundaries and focuses on accumulating distributions rather than detecting transitions.The authors exclude anomaly-detection methods for learning distributional boundaries.Source: §8 · p. 21
  • VAEs produce blurrier images than GAN-based methods, a limitation the authors connect to their reconstruction-distribution assumption.The passage also notes that adversarial methods have failed to accurately match posterior-prior distribution ratios in large dimensions [104].Source: §8 · p. 21
  • The authors posit that truly lifelong learning needs global memory decoupled from the learning algorithm, which could support parameter transfer and historical-data compression.The Kanerva Machine and its extensions provide a principled way to do this in the VAE setting.Source: §8 · pp. 21–22

9 CONCLUSION

The paper proposes unsupervised generative learning across sequentially presented distributions without access to old data. Its dual student-teacher architecture and regularizers support knowledge retention and mitigate catastrophic interference.

9 CONCLUSION

  • A dual student-teacher architecture preserves past knowledge to aid future learning without access to old data, while regularizers support knowledge transfer and mitigate catastrophic interference.The method is intended to distill necessary information into a single final model.Source: §9 · p. 22

10 APPENDIX

The appendix analyzes the consistency regularizer, tests a reconstruction regularizer, and details architectures, comparisons, and mechanisms for sequential generative learning. It also contrasts the method with streaming approaches and describes reparameterization and expandable capacity.

10.1 UNDERSTANDING THE CONSISTENCY REGULARIZER

  • The consistency regularizer transforms the standard VAE regularizer; with an isotropic Gaussian posterior, it scales the student posterior’s mean and variance by the teacher variance and adds a volume term.The authors interpret this as preserving the standard VAE learning objective.Source: §10.1 · p. 28

10.2 RECONSTRUCTION REGULARIZER

  • Adding a decoder likelihood cross-model regularizer diminishes performance, and the FashionMNIST comparison reports worse Frechet distance and test ELBO with that term.The comparison uses a simpler dense model; Figures 18 and 19 show the test ELBO and Log-Frechet Distance, respectively.Source: §10.2 · p. 29, Figure 18 · p. 29, Figure 19 · p. 29

10.3 MODEL ARCHITECTURE

  • The experiments use dense and convolutional architectures; lifelong models grow the discrete latent representation, while EWC has twice as many parameters and requires preserving a teacher model.Naive and batch models use fewer parameters, while the vanilla model lacks protection against catastrophic interference and the full model is an upper bound.Source: §10.3 · pp. 29–30

10.4 CONTRAST TO STREAMING / ONLINE METHODS

  • Unlike streaming methods that approximate the posterior using the current dataset and previous posterior, the method reuses generated past data to recompute a posterior retaining earlier distributions.Coupled with the Bayesian update regularizer, generative replay supports information transfer between sequentially learned models.Source: §10.4 · p. 30

10.5 EWC BASELINES: COMPARING CONV & DENSE NETWORKS

  • The appendix compares convolutional and dense EWC baselines on PermutedMNIST and FashionMNIST and uses the best-performing EWC models in experiments.The passage refers to the full range of baselines in Figure 10.5 but does not report specific comparison outcomes.Source: §10.5 · p. 31

10.6 GUMBEL REPARAMETERIZATION

  • The discrete-and-continuous latent model uses Gumbel-Softmax reparameterization, with its temperature tending to zero as the sample converges to a categorical.The reparameterization is defined over encoder logits and an annealed temperature parameter.Source: §10.6 · p. 31

10.7 EXPANDABLE MODEL CAPACITY AND REPRESENTATIONS

  • The model can expand capacity by adding layers and growing the discrete latent representation to accommodate new distributions.Different-sized categorical distributions are compared by zero-padding the teacher distribution; weights around the latent projection are reinitialized when the representation grows.Source: §10.7 · pp. 31–32

Related papers

Loading 1705.09847v7…