PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent

Mehdi Abbana Bennani, Thang Doan, Masashi Sugiyama

arXiv:2006.11942v4stat.MLcs.LG

TL;DR

Continual Learning requires learning sequential tasks while retaining prior knowledge, but theoretical guarantees for OGD were lacking. The paper develops an NTK-based recursive kernel framework, proves forgetting robustness and generalisation results, and examines practical limits caused by NTK variation.

  • Problem

    Continual Learning must balance adaptation to sequential tasks with retention of previously acquired knowledge, while few theoretical works address this setting.

  • Method

    The paper models Continual Learning algorithms in the Neural Tangent Kernel regime through recursive kernel regression and analyzes SGD and OGD.

  • Results

    The framework proves OGD robust to forgetting under infinite memory and provides the first Continual Learning generalisation bound for SGD and OGD, with cross-task generalisation depending on NTK-based task similarity.

  • Takeaways & Limitations

    The analysis connects Continual Learning with transfer and task similarity while showing that NTK variation matters for OGD's practical forgetting robustness.

  • Takeaways & Limitations

    The theoretical guarantees rely on NTK-regime and infinite-memory assumptions, and NTK variation limits applicability in non-overparameterized practical settings.

Abstract

from arXiv · show

In Continual Learning settings, deep neural networks are prone to Catastrophic Forgetting. Orthogonal Gradient Descent was proposed to tackle the challenge. However, no theoretical guarantees have been proven yet. We present a theoretical framework to study Continual Learning algorithms in the Neural Tangent Kernel regime. This framework comprises closed form expression of the model through tasks and proxies for Transfer Learning, generalisation and tasks similarity. In this framework, we prove that OGD is robust to Catastrophic Forgetting then derive the first generalisation bound for SGD and OGD for Continual Learning. Finally, we study the limits of this framework in practice for OGD and highlight the importance of the Neural Tangent Kernel variation for Continual Learning with OGD.

1 INTRODUCTION

The paper develops an NTK-based theoretical framework for Continual Learning and uses it to analyze OGD's forgetting robustness and generalisation. It also identifies limits arising from NTK variation in practical settings.

  • The paper introduces an NTK-regime framework for Continual Learning that provides closed-form task-wise models and proxies for transfer, generalisation, task similarity, and curriculum learning.The framework treats Continual Learning through recursive kernel regression.
  • OGD is proved robust to Catastrophic Forgetting across an arbitrary number of tasks under infinite memory.
  • The paper derives the first generalisation bound for Continual Learning with SGD and OGD, showing that cross-task generalisation depends on task similarity measured through the NTK.
  • In practical settings, NTK variation negatively affects OGD's Catastrophic Forgetting robustness on non-overparameterized benchmarks.The framework's assumptions therefore constrain its practical applicability.

2 RELATED WORKS

Related work largely studies Continual Learning through heuristics and method families, while theoretical analyses remain comparatively limited. The paper builds on NTK results to study OGD and its generalisation properties.

  • Continual Learning methods are commonly grouped into regularization, memory-based, and dynamic architectural approaches.Memory methods replay buffered past data, whereas dynamic architectures add components for new knowledge.
  • Theoretical work on Continual Learning is limited, including analyses of lifelong-learning regret and the computational difficulty of optimal algorithms.
  • Prior theory shows that task similarity contributes to generalisation for SGD in a two-task setting with overparameterized two-layer ReLU networks.
  • The paper uses NTK and overparameterized-network results to construct a framework for Continual Learning and analyze OGD generalisation.

3 PRELIMINARIES

The preliminaries define Continual Learning as sequential supervised learning with simultaneous adaptation and retention requirements. They introduce OGD's Jacobian-based orthogonal updates and the NTK linearization assumptions.

  • Continual Learning presents sequential supervised tasks to one model, requiring good performance on new tasks while retaining previous-task knowledge.
  • Tasks are modeled as distributions with finite samples, and training uses ridge regularization with task- and iteration-indexed model notation.
  • OGD updates weights using the gradient projection onto the orthogonal complement of Jacobian-induced spaces from previous tasks.Its stated intuition is to preserve past knowledge by maintaining previous prediction-gradient directions.
  • In the infinite-width limit, the NTK remains constant during training and the network evolves as a linear model under certain losses and gradient descent.
  • The analysis assumes an overparameterized network and uses a linear approximation around initialization.

4 CONVERGENCE - CONTINUAL LEARNING AS A RECURSIVE KERNEL REGRESSION

The paper expresses Continual Learning models as recursive kernel regression across tasks, where each task fits a residual relative to prior knowledge. This yields task-similarity and convergence analyses under NTK-based assumptions.

  • 4 CONVERGENCE - CONTINUAL LEARNING AS A RECURSIVE KERNEL REGRESSION: Theorem 1 expresses Continual Learning as recursive kernel ridge regression, with OGD's NTK recursively determined by projected task feature maps.
  • The recursive model requires a learning-rate condition involving the task kernel matrix, ridge regularization, and its spectral norm.
  • Theorem 1 (Continual Learning as a recursive Kernel Regression): Each task fits a residual after subtracting predictions transferred from prior tasks, and this residual serves as a proxy for task similarity.Identical tasks have zero residual, while the kernel increment captures the new knowledge added by the task.
  • For SGD, and for OGD under orthonormal-task assumptions, Corollary 1 relates the limiting model or distance from initialization to task progression and similarity.
  • 4.2 DISTANCE FROM INITIALISATION THROUGH TASKS: The analysis uses ridge regularization because Gram-matrix invertibility is not guaranteed; under additional invertibility assumptions, it can recover a related approximation from prior work.

5 OGD : LEARNING WITHOUT FORGETTING, PROVABLY

The NTK framework proves that OGD preserves previous-task training performance under infinite memory and yields generalisation bounds whose tightness depends on task similarity and ordering.

  • 5.1 MEMORISATION PROPERTY OF OGD: OGD leaves training error on all previous-task samples unchanged under infinite memory, establishing robustness to catastrophic forgetting.The result assumes that previous-sample Jacobians remain in OGD’s memory and follows from orthogonal updates preserving relevant weight dimensions.
  • 5.2 GENERALISATION PROPERTIES OF SGD AND OGD: Theorem 3 gives generalisation bounds for both SGD and OGD, with OGD tighter because SGD incurs an additional residual forgetting term.The bound also contains a regularisation term and a task-similarity term.
  • 5.3 THE IMPACT OF TASK SIMILARITY ON GENERALISATION: Greater task dissimilarity increases the Rademacher-complexity upper bound, enlarging the function class explored by the network with high probability.This links task similarity to the complexity term governing continual-learning generalisation.
  • 5.3 THE IMPACT OF TASK SIMILARITY ON GENERALISATION: The task-similarity contribution is measured through an NTK-based dissimilarity, and the bound does not increase when successive tasks are identical.Knowledge from previous tasks is encoded in the kernel through its feature map.
  • 5.3 THE IMPACT OF TASK SIMILARITY ON GENERALISATION: Because the bound depends on task ordering, alternating between two dissimilar tasks produces a large upper bound, motivating curriculum effects in continual learning.The framework therefore connects task sequence design with generalisation behaviour.

6 THE IMPACT OF THE NTK VARIATION ON OGD

The practical analysis examines the limits of the NTK and infinite-memory assumptions, introducing OGD+ to isolate the effect of NTK variation during continual learning.

  • 6 THE IMPACT OF THE NTK VARIATION ON OGD: The theoretical guarantee applies under the NTK regime and infinite memory, assumptions that do not hold in practical settings.The practical section therefore studies the limits of the framework rather than directly extending the theorem.
  • 6 THE IMPACT OF THE NTK VARIATION ON OGD: OGD+ updates its orthonormal basis using all tasks after each task, making it more robust to NTK variation than OGD.This design decouples NTK-variation effects in the experiments.
  • 6 THE IMPACT OF THE NTK VARIATION ON OGD: Unlike OGD, OGD+ stores feature maps for previous-task samples as well as current-task samples, requiring a dedicated memory for those samples.The additional storage is motivated by NTK variation in practice.

7 EXPERIMENTS

Experiments test OGD’s theoretical robustness across overparameterization and Jacobian variation, then compare OGD+ with continual-learning baselines. Forgetting decreases with overparameterization, while updating the Jacobian is most valuable in less-overparameterized benchmarks.

  • 7.1 Ablation study: OGD and overparameterization: Train error variation decreases with overparameterization for OGD on MNIST and CIFAR100, matching Theorem 2.The experiments vary hidden size as a proxy for overparameterization and measure memorised samples from the first task.
  • 7.2 Ablation study: updating the Jacobian: The analysis assumes overparameterization and a constant Jacobian, but experiments show forgetting when this assumption fails in practice.OGD+ updates stored Jacobians to measure the impact of Jacobian variation.
  • 7.2 Ablation study: updating the Jacobian: For less-overparameterized Rotated MNIST and Permuted MNIST, OGD+ is more robust to catastrophic forgetting than OGD; on CIFAR100 and CUB200, their robustness is equivalent.OGD+ accounts for Jacobian variation, which is smaller in the more-overparameterized benchmarks.
  • 7.3 Benchmarking OGD+: OGD+ improves over OGD and is competitive with other continual-learning methods on Permuted MNIST and Rotated MNIST, but is not competitive on CIFAR100 and CUB200.The comparison uses average accuracy across the listed MNIST, CIFAR100, and CUB200 datasets.

8 CONCLUSION

The paper develops an NTK-regime framework for continual-learning algorithms and uses it to analyze SGD and OGD. It connects the analysis to transfer and curriculum learning while identifying limits in non-overparameterized settings.

  • 8 Conclusion: The paper presents an NTK-regime framework for continual learning and studies convergence and generalisation properties of SGD and OGD.The framework’s applicability is assessed through experiments.
  • 8 Conclusion: The analysis connects continual learning with Transfer Learning and Curriculum Learning through task similarity and task ordering.The paper identifies these connections as implications of its theoretical framework.
  • 8 Conclusion: Experiments expose limits of the framework in non-overparameterized settings, motivating extensions beyond the overparameterization assumption.The conclusion identifies this extension as an important direction for understanding OGD there.

A.1 NTK VARIATION - THE IMPORTANCE OF ORTHOGONALITY FOR THE OGD, OGD+ AND A-GEM ALGORITHMS

The section compares how OGD, OGD+, A-GEM, and A-GEM-NT differ in projection spaces and update rates, emphasizing the role of changing feature maps. In the NTK regime, OGD implies A-GEM-NT without Positive Backward Transfer and yields an upper bound on its generalisation error.

  • Algorithm properties: OGD, OGD+, A-GEM, and A-GEM-NT differ mainly in the span of their orthogonality constraints and how frequently the projection space is updated.OGD+ combines OGD’s larger projection space with less frequent updates, whereas A-GEM updates a smaller projection space at every step.
  • Practical interpretation: In practice, changing feature maps can make OGD’s orthogonality constraint less relevant, while A-GEM’s frequently updated projection may remain more relevant.The text positions OGD+ between these alternatives by using a larger constrained space without updating feature maps at every gradient step.
  • Theoretical connection: OGD provides an upper bound on A-GEM-NT in terms of generalisation error.This comparison is stated as Proposition 1.
  • Theoretical connection: In the NTK regime, OGD implies A-GEM-NT with no Positive Backward Transfer.The implication follows from OGD’s robustness to Catastrophic Forgetting.

B.2 GENERALISATION :

The paper derives continual-learning solutions through recursive kernel regression and analyzes their generalisation using Rademacher complexity and empirical-loss bounds. OGD obtains tighter bounds than SGD because its orthogonality property removes the forgetting terms affecting previous tasks.

  • OGD robustness: Theorem 2 establishes no forgetting for OGD because updates are orthogonal to feature maps stored from previous tasks.This preserves the parameters relevant to memorised samples and leaves their training error unchanged under the theorem’s conditions.
  • Generalisation bounds: OGD’s generalisation bound is tighter than SGD’s because OGD is robust to Catastrophic Forgetting on previous tasks.SGD’s bound contains an additional residual forgetting term, whereas OGD’s bound does not.
  • Interpretation: The analysis connects generalisation across tasks to task similarity measured with respect to the NTK.The framework also provides implications for Transfer Learning and Curriculum Learning.
  • Proof strategy: The generalisation analysis bounds Rademacher complexity and empirical losses for current and previous tasks.The SGD analysis includes residual terms for prior-task forgetting, while the OGD analysis leverages Theorem 2 to obtain tighter bounds.

D.2.7 ALTERNATIVE PROOF FOR THE EMPIRICAL LOSS BOUNDS - NO REGULARISATION CASE

The no-regularisation analysis gives an alternative dynamical proof of OGD’s empirical-loss behavior. Under positive-definite kernel conditions, current-task training error converges to zero while previous-task error remains unchanged.

  • Proof strategy: The proof derives the model’s output dynamics, defines the projection onto the trainable weight space, and analyzes the resulting gradient flow.The projection matrix encodes which weight directions remain trainable under OGD.
  • No-regularisation dynamics: With no regularisation, OGD’s training error on every task converges to zero as training time tends to infinity.The convergence result assumes the relevant kernel matrix is positive definite and uses a suitable learning-rate condition.
  • Past-task preservation: For previous tasks, OGD keeps the training error at zero after the current task is learned.This combines convergence on each task with Theorem 2’s unchanged-error property for past tasks.
  • Experimental setting: The experiments use five independent runs and report means and standard deviations across benchmarks.OGD and OGD+ require smaller architectures on CIFAR100 and CUB200 because of memory limitations.
  • Empirical checks: Increasing hidden size or memory size reduces the observed variation in memorised-sample train accuracy.The hidden-size experiment treats hidden size as a proxy for overparameterization, while the memory experiment varies memory per task.

F.6.1 COMPLEMENTARY RESULTS - CONTINUAL LEARNING METRICS

The complementary benchmark results show that OGD+ helps most on the less overparameterized MNIST settings, while it is not competitive on Split CIFAR100 and CUB200. On Rotated MNIST, its gains over OGD are concentrated in selected metrics.

  • Permuted MNIST: On Permuted MNIST, OGD+ outperforms the baselines on AAC while remaining competitive on AFM and BWT.The comparison is reported in Table 9.
  • Rotated MNIST: On Rotated MNIST, OGD+ is competitive on AAC and AFM but underperforms on the other metrics.It improves over OGD on AAC, BWT, and AFM, possibly because the setting is relatively less overparameterized.
  • CIFAR100 and CUB200: On Split CIFAR100 and CUB200, OGD+ is not competitive on any reported metric and matches OGD across metrics.These settings are described as more overparameterized than the MNIST benchmarks.
  • Evaluation: Test accuracy is tracked through tasks for multiple Continual Learning methods, with means and standard deviations over five runs.The reported test error is measured at the end of each task.
Loading 2006.11942v4…