PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-based Continual Learning with Adaptive Regularization

Hongjoon Ahn, Sungmin Cha, Donggyu Lee, Taesup Moon

arXiv:1905.11614v3cs.LGstat.ML

TL;DR

Continual-learning methods must adapt to new tasks while retaining prior knowledge, but regularization approaches can require substantial per-weight memory and may lack graceful forgetting. UCL uses Bayesian variational inference with node-wise uncertainty and additional stability-plasticity regularizers, and the paper reports stronger performance across supervised and reinforcement-learning benchmarks with fewer additional parameters.

  • Problem

    Continual learning must balance learning sequentially arriving tasks with retaining past knowledge, while regularization methods require substantial memory for per-weight strengths and may lack graceful forgetting.

  • Method

    UCL applies a variational Bayesian framework with node-wise uncertainty and regularization terms that freeze important past-task parameters while controlling actively learning parameters.

  • Results

    UCL convincingly outperformed state-of-the-art baselines across supervised and reinforcement-learning continual-learning benchmarks with much fewer additional parameters.

  • Takeaways & Limitations

    UCL combines reduced regularization-parameter storage with mechanisms for stability and plasticity across diverse continual-learning settings.

  • Takeaways & Limitations

    Per-parameter regularization methods can require twice the original model memory, while tying node uncertainty can cause network capacity to fill quickly as tasks grow.

Abstract

from arXiv · show

We introduce a new neural network-based continual learning algorithm, dubbed as Uncertainty-regularized Continual Learning (UCL), which builds on traditional Bayesian online learning framework with variational inference. We focus on two significant drawbacks of the recently proposed regularization-based methods: a) considerable additional memory cost for determining the per-weight regularization strengths and b) the absence of gracefully forgetting scheme, which can prevent performance degradation in learning new tasks. In this paper, we show UCL can solve these two problems by introducing a fresh interpretation on the Kullback-Leibler (KL) divergence term of the variational lower bound for Gaussian mean-field approximation. Based on the interpretation, we propose the notion of node-wise uncertainty, which drastically reduces the number of additional parameters for implementing per-weight regularization. Moreover, we devise two additional regularization terms that enforce stability by freezing important parameters for past tasks and allow plasticity by controlling the actively learning parameters for a new task. Through extensive experiments, we show UCL convincingly outperforms most of recent state-of-the-art baselines not only on popular supervised learning benchmarks, but also on challenging lifelong reinforcement learning tasks. The source code of our algorithm is available at https://github.com/csm9493/UCL.

1 Introduction

Continual learning must balance adapting to sequentially arriving tasks with retaining prior knowledge, while regularization-based methods trade network updates against memory overhead. UCL addresses these limitations with node-wise uncertainty and additional mechanisms for stability and plasticity.

  • Problem: Continual learning requires adapting to continuously arriving tasks without forgetting previously learned knowledge.This creates the stability-plasticity dilemma: excessive stability slows new learning, whereas excessive plasticity causes catastrophic forgetting.
  • Related approaches: Regularization-based methods are one of three broad continual-learning categories, alongside dynamic architectures and dual memory systems.The taxonomy is presented as a rough grouping of approaches to catastrophic forgetting.
  • Regularization-based methods: Regularization-based methods constrain parameter updates to reuse limited network capacity as the number of tasks grows.They add penalties against changing network parameters when learning new tasks.
  • Limitations: Per-parameter regularization strengths can double model memory because methods store individual importance parameters alongside the original network parameters.The passage identifies this as a potential deployment limitation for large networks.
  • UCL: UCL interprets the variational KL divergence for Gaussian mean-field inference and defines node-wise uncertainty by tying variances of incoming weights.It also adds regularization terms intended to freeze important weights and control actively learning weights.

2 Related Work

Related work groups neural continual-learning approaches into regularization, dynamic architecture, and dual-memory methods. The paper situates UCL within Bayesian variational inference and regularization-based approaches that preserve prior-task knowledge by constraining updates.

  • Regularization-based methods: Regularization-based methods identify important weights from learned tasks and penalize large updates to those weights during new-task learning.The related-work passage describes this as the main approach within that category.
  • Alternative architectures: PackNet selects task-specific weights through pruning and must save binary masks for those weights.This represents an alternative approach for limited network capacity.
  • Alternative architectures: HAT uses task-identifier embeddings with node-wise attention but requires knowing the number of tasks in advance.The paper characterizes this prior task-count requirement as a critical limitation.
  • Variational inference: Variational inference approximates model posteriors and can learn neural-network variational parameters using back-propagation or re-parameterization.The passage presents these as standard approaches in Bayesian learning.

3 Uncertainty-regularized Continual Learning (UCL)

UCL reinterprets the variational KL divergence to replace weight-level uncertainty storage with node-wise uncertainty, while adding regularization that balances preserving past-task knowledge with learning new tasks.

  • Bayesian online learning: Variational inference approximates the intractable posterior with a Gaussian mean-field distribution learned by reparameterization and back-propagation.The variational objective combines expected negative log-likelihood with KL divergence between approximate and prior distributions.
  • Bayesian online learning: In continual learning, the previous task’s posterior becomes the next task’s prior, making the KL term a sequential regularizer against catastrophic forgetting.VCL applies this procedure task by task, though only partially mitigating forgetting.
  • Motivation: UCL addresses VCL’s memory and sampling drawbacks by interpreting the Gaussian KL term and using node-wise rather than per-weight uncertainty.Per-weight variance doubles stored network parameters, while Monte Carlo likelihood estimation increases time and space complexity with sample size.
  • Stability and plasticity: Additional ℓ1-based and uncertainty regularizers freeze important weights while allowing actively learning weights to change for new tasks.Because preserving node importance can fill network capacity as tasks accumulate, UCL adds a term that supports more graceful forgetting.
  • Node-wise uncertainty: Node uncertainty ties the incoming weights of each node to a shared standard deviation, reducing auxiliary parameters while treating activations as basic information units.Under the Gaussian mean-field approximation, this corresponds to injecting independent zero-mean Gaussian noise with node-specific variances during variational sampling.
  • Adaptive weight regularization: UCL strengthens a weight’s penalty when either connected node has low uncertainty, protecting important nodes from incoming negative transfer and outgoing information loss.The resulting regularization is implemented through the modified KL term and node-derived weight strengths.
  • Optimization: Using one model-weight sample per iteration enables UCL to operate on reinforcement-learning tasks, unlike VCL’s sampling-based procedure.At each new task, UCL initializes the variational parameters from the previous task before subsequent updates.

4 Experimental Results

Across supervised and reinforcement-learning benchmarks, UCL generally outperforms regularization-based baselines while using fewer additional parameters. Its uncertainty-based design supports both retaining past-task knowledge and adapting to new tasks.

  • Permuted / Row Permuted MNIST: 94.5% average test accuracy after 10 Permuted MNIST tasks, exceeding EWC, SI, VCL, and coreset VCL.EWC, SI, and VCL achieved 91.8%, 91.1%, and 91.3%, respectively; UCL withstood the single-headed setting better than HAT after task 7.
  • Uncertainty analysis: UCL’s node standard deviations adapt across tasks, with some uncertain first-layer nodes becoming more certain as new-task learning progresses.Most second-layer nodes show no monotonic trend, supporting the paper’s interpretation of plasticity and graceful forgetting.
  • Ablation study: Removing either freezing or active-learning regularization causes substantial degradation, showing both components contribute to sequential performance.Without term (6), accuracy drops drastically after Task 3 as actively learning weights diminish; the other ablations also identify freezing-related terms as important.
  • Split MNIST and notMNIST: 99.7% average accuracy on Split MNIST matches HAT and exceeds SI and coreset VCL, while UCL does not require knowing the task count beforehand.On Split notMNIST, UCL reaches 95.7%, above HAT at 95.2% and VCL with coreset at 93.7%.
  • CNN benchmarks: 63.4% on Split CIFAR-100, 73.2% on Split CIFAR-10/100, and 83.9% on Omniglot exceed the reported EWC and SI results.The paper reports UCL outperforming baselines across more diverse vision datasets and deeper CNN architectures.
  • Parameter comparison: UCL uses fewer parameters than other regularization-based approaches, with almost half as many as VCL, while HAT requires the task count a priori.The comparison concerns the additional parameter burden used for regularization across benchmarks.
  • Reinforcement learning: In eight lifelong reinforcement-learning tasks, both UCL variants significantly outperform EWC and fine-tuning on cumulative normalized rewards.Fine-tuning mostly suffers catastrophic forgetting, whereas UCL combines retention of prior tasks with learning new tasks.

5 Conclusion

The paper presents UCL as an uncertainty-based regularization method for overcoming catastrophic forgetting and addressing the stability-plasticity dilemma. It reports stronger performance than state-of-the-art baselines with fewer additional parameters across supervised and reinforcement-learning benchmarks.

  • UCL uses node-wise uncertainty motivated by Bayesian online learning to address catastrophic forgetting.
  • UCL adds regularization terms designed to manage the stability-plasticity dilemma.
  • UCL outperformed other state-of-the-art baselines on supervised and reinforcement-learning benchmarks.
  • UCL achieved these results with much fewer additional parameters.

Supplementary Materials for Uncertainty-regularized Continual Learning with

The supplementary materials identify the paper, its authors and affiliations, and its publication context. The paper is associated with NeurIPS 2019 and an arXiv version dated 14 November 2019.

  • The paper is authored by Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon.
  • The authors are affiliated with the Department of Artificial Intelligence and Department of Electrical and Computer Engineering at Sungkyunkwan University.
  • The paper appeared at NeurIPS 2019 and has an arXiv version dated 14 November 2019.

1 Derivation of Eq. (3)

This section derives the Gaussian mean-field KL-divergence expression used in the method. It decomposes the divergence into integrals, evaluates them, and connects the result to the main-paper equation layer by layer.

  • The KL divergence between successive variational distributions is decomposed into two terms.
  • The first decomposed integral is evaluated in closed form.
  • The second decomposed integral is evaluated separately before combining the terms.
  • The resulting expression becomes the main-paper Eq. (3) when decomposed across network layers.

2 Detailed explanation on initializing standard deviation

The supplementary derivation explains variance propagation through forward and backward passes and motivates the standard-deviation initialization used by UCL. The initialization aims to prevent unstable signal scaling and is selected empirically for fully connected layers.

  • Initialization assumptions: The derivation assumes ReLU activations and uses notation motivated by an architecture-adaptive initialization method.
  • Forward propagation: Forward propagation samples weights from a symmetric zero-mean prior and assumes independent identically distributed previous-layer activations.
  • Forward propagation: For convolutional layers, the previous-layer node count is represented as a k^2c-by-1 vector.
  • Forward propagation: The forward derivation combines variance relationships across L layers to characterize signal propagation.
  • Initialization objective: The initialization sets weight variance to keep output variance constant and avoid exponentially increasing or decreasing signals.
  • Empirical choice: Initializing the standard deviation with n_l for fully connected layers achieved the best empirical performance.

3 Detailed explanation on Eq. (6)

The section reformulates the optimization problem in generalized form and characterizes its optimum through convexity and a zero-gradient condition.

  • Equation (10) is converted into a generalized form before analyzing its optimal point.
  • The derivation uses a vector p(l) satisfying the stated element-wise constraint relative to the preceding uncertainty term.
  • Because the generalized objective is convex, its optimum is identified by setting the gradient to zero.
  • For simplicity, the derivation selects p(l) = 1 for all layers after establishing the relevant element-wise condition.

4 Additional experimental results

Additional experiments examine adaptive initialization across fully connected and convolutional networks, including ablations of UCL’s freezing and graceful-forgetting components.

  • Additional experiments evaluate adaptive initialization on various Permuted MNIST, Split MNIST, and Split notMNIST settings.
  • An ablation study evaluates UCL variants on Split CIFAR10/100 with a convolutional neural network.
  • Removing upper freezing fails to prevent early-task catastrophic forgetting but leaves more active learners and high average accuracy.
  • Removing the freezing term lowers Task 1 retention while producing nearly the same average accuracy as original UCL.
  • Removing graceful forgetting causes accuracy to drop sharply after Task 1 despite higher Task 1 accuracy than UCL.

5 Implementation details

The implementation details specify training and hyperparameter settings for supervised benchmarks and describe how UCL represents uncertainty in convolutional networks.

  • Most supervised baselines use mini-batches of 256 for 100 epochs with Adam at learning rate 0.001, while VCL uses longer or full-dataset training in specified experiments.
  • The experiments compare UCL, EWC, SI, HAT, and VCL across benchmark-specific regularization hyperparameter grids.
  • Split CIFAR-10/100 and Split CIFAR-100 use the architecture reported in Table 1, while Omniglot uses the architecture reported in Table 2.
  • For CNNs, UCL assigns uncertainty to convolution channels rather than individual nodes and uses colored channels and filters to denote importance and regularization.
  • Filters connected to channels identified as important for earlier tasks tend to retain their values, while other filters can specialize for later tasks.

5.2 Reinforcement learning

The reinforcement-learning experiments train UCL and baselines on eight environments with PPO, then examine how initialization uncertainty and β affect rewards, forgetting, exploration, and new-task learning.

  • UCL and each baseline are trained on eight tasks using two 16-node fully connected hidden layers, a 44-node input layer, and multiple output layers.
  • PPO is used for reinforcement-learning training, with shared hyperparameters applied equally across the baselines.
  • The experiments report cumulative normalized rewards and normalized rewards for each task throughout learning.
  • Successful continual reinforcement learning requires selecting σ(l)_init and β values appropriately for reward and forgetting objectives.
  • With σ(l)_init = 1x10^-3, UCL with small β effectively overcomes catastrophic forgetting, but Tasks 1 and 3 receive lower rewards than in the manuscript’s experiment.
  • Small initial uncertainty limits exploration capacity and can make a new task difficult to learn even when forgetting is graceful.
Loading 1905.11614v3…