Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

II. RELATED WORK

• Prior theoretical work largely uses linear or NTK regimes, while related stability analyses remain tied to particular architectures or task structures.The paper positions its trajectory-based analysis as avoiding a prescribed model architecture.Source: §I · p. 1, §II · p. 3

I. INTRODUCTION

• The framework decomposes expected forgetting into intra-task and cross-task generalization gaps plus empirical forgetting, tracking sample perturbations across updates and tasks.It covers constant, intra-task decaying, and continuously decaying schedules; the empirical-forgetting analysis retains cumulative gradient alignment and does not require convexity.Source: §I · pp. 1–2
• For historical tasks, continuous decay yields generalization bounds independent of N with other parameters fixed, while constant and intra-task decay scale as O(N/n) and O(log(eN)/n), respectively, in the stated convex setting.For non-convex losses and historical tasks, continuous-decay IGG and CGG bounds are no larger than intra-task-decay bounds at the same parameter values.Source: §I · pp. 1–2, Table I · p. 2
• The empirical-forgetting bound includes cumulative gradient alignment and a squared-learning-rate term whose scaling differs across the three schedules.The second-order term scales as O(N), O(1), and O(1/N) for constant, intra-task decaying, and continuously decaying rates, respectively.Source: §I · p. 2
• The expected-forgetting bounds combine generalization gaps with changes in empirical risk along the continual-learning trajectory.Their terms account for sample size, local training length, task count, learning-rate schedules, and cumulative gradient alignment.Source: §I · p. 2

A. Continual Learning Setting

• The analysis follows SGD over sequential tasks with independent datasets of n samples each, training each task for N steps.The setting defines task distributions and empirical and population risks; expected forgetting is measured over the underlying data distribution and training and algorithm randomness, generalizing BWT.Source: A. Continual Learning Setting · p. 3

B. Algorithmic Stability

• Current-task stability measures sensitivity to a sample change in the current task, while cross-task stability captures how a historical-task change propagates through later training.These notions account for the sequential training trajectory in continual learning.Source: B. Algorithmic Stability · pp. 3–4

IV. THEORETICAL ANALYSIS

• Forgetting is decomposed into cross-task generalization gap, empirical forgetting, and intra-task generalization gap.The gaps compare population and empirical risks on historical or current tasks; empirical forgetting measures change in historical-task training risk.Source: §IV · p. 4

A. Intra-Task Generalization Gap

• For convex losses, the intra-task gap bound depends on the optimization within the current task, and smaller learning rates or larger training samples can tighten it.The bound is independent of previous tasks and is expressed through the cumulative learning rate within task i.Source: A. Intra-Task Generalization Gap · p. 4
• For non-convex losses, intra-task decay yields an O(N^p/n) bound when 2cL^2 < N, while continuous decay can make the bound independent of N for i > 1.For the stated intermediate regime, continuous decay is no larger than intra-task decay; when 2cL^2 ≥ iN, both schedules give N/n.Source: A. Intra-Task Generalization Gap · p. 5

B. Cross-Task Generalization Gap

• For convex losses, the cross-task gap bound coincides with the intra-task gap bound, reflecting preserved SGD stability across the task sequence.The result relies on the non-expansive property of convex and smooth losses.Source: B. Cross-Task Generalization Gap · p. 5
• Under intra-task decay, the cross-task bound is nondecreasing with subsequent tasks and can require N = o(n^(1/[p(T−i+1)])) for vanishing, a stricter condition than for the intra-task gap.The stronger restriction reflects perturbation propagation through subsequent tasks.Source: B. Cross-Task Generalization Gap · p. 6
• For i > 1, continuous decay gives a cross-task bound no larger than intra-task decay and O(1/n) uniformly in N for fixed c, β, L, T, and i.Continuous decay continues across task boundaries; intra-task decay resets to c, and its perturbation-amplification bound depends on N.Source: B. Cross-Task Generalization Gap · p. 6

C. Empirical Forgetting

• A positive cumulative gradient alignment reduces the empirical-forgetting upper bound, while a negative value increases it; this result holds for convex and non-convex losses.The bound also includes a second-order term involving squared learning rates.Source: C. Empirical Forgetting · pp. 6–7
• The second-order term grows linearly with N under a constant learning rate, approaches βc2L2(T − i) as N increases under intra-task decay, and decreases proportionally to 1/N under continuous decay.Source: C. Empirical Forgetting · p. 7
• The observable CGA measurement accumulates learning-rate-weighted inner products between historical evaluation-loss gradients and actual training gradients, using measurement buffers from previous tasks.The buffers are used exclusively for measurement.Source: C. Empirical Forgetting · p. 7
• On Split CIFAR-100, estimated CGA is largest under constant rates, smaller under intra-task decay, and closer to EF under continuous decay.Under intra-task decay, EF and negative estimated CGA remain visibly separated; under continuous decay, they show similar fluctuations.Source: C. Empirical Forgetting · p. 7, Figure 1 · p. 7

D. Forgetting in Expectation

• Expected forgetting is bounded by accumulating intra-task and cross-task generalization contributions and empirical forgetting over historical tasks.If the bounds on both gaps and the magnitude of empirical forgetting vanish, expected forgetting also converges to zero.Source: D. Forgetting in Expectation · p. 8
• With other parameters fixed, decaying learning rates or more training samples can tighten the generalization bounds, while smaller initial learning rates can reduce them.Dependence on local training length varies across learning-rate schedules.Source: D. Forgetting in Expectation · p. 8
• Positive cumulative alignment between gradients of previous and subsequent tasks contributes to reducing the empirical-forgetting upper bound through −G_T,i.This is one of the implications of the combined bounds.Source: D. Forgetting in Expectation · p. 8

V. EXPERIMENT

• Split CIFAR-100 experiments examine the empirical implications of the theoretical analysis.The section also reports Split MiniImageNet experiments in Appendix F.Source: §V · p. 8
• Split MiniImageNet is also included as an experimental benchmark.Its results are reported in Appendix F.Source: §V · p. 8
• The Split CIFAR-100 experiments are presented in the section, while the Split MiniImageNet results are deferred to Appendix F.Source: §V · p. 8

A. Experimental Setup

• The Split CIFAR-100 setup uses 10 tasks, AlexNet, and SGD; experiments vary epochs, initial learning rate, and samples per task.Observed gaps replace population risk with test loss, while ACC measures average accuracy; figures report means and standard deviations over three seeds.Source: A. Experimental Setup · p. 8

B. Experimental Result

• Decaying schedules generally yield smaller generalization gaps and less forgetting than constant rates, with continuous decay keeping forgetting close to zero across tested settings.Similar overall trends are reported for Split MiniImageNet with ResNet-18.Source: B. Experimental Result · p. 8, Figure 2 · p. 9
• Continuous decay yields the lowest ACC, and fewer epochs or a smaller initial learning rate may reduce forgetting without improving ACC.The authors relate this trade-off to the stability-plasticity dilemma.Source: B. Experimental Result · p. 9
• Both IGG and CGG decrease as training samples per task increase when the number of SGD updates per task is fixed.The observed trend is consistent with the theoretical bounds’ dependence on sample size.Source: Figure 3 · p. 9, B. Experimental Result · p. 9
• GEM has the highest ACC and BWT under constant and intra-task decay, but all three methods show similarly low ACC and near-zero BWT under continuous decay.The authors suggest gradient-alignment benefits depend on the learning-rate schedule.Source: Figure 4 · p. 9, B. Experimental Result · p. 9

VI. CONCLUSION

• The conclusion proposes a stability framework that decomposes forgetting into IGG, CGG, and EF and analyzes both convex and non-convex losses across three schedules.It characterizes effects of local training length, initial learning rate, schedules, and cumulative gradient alignment, and reports empirical validation.Source: §VI · pp. 9–10
• The theoretical analysis assumes L-Lipschitz continuity and β-smoothness, assumptions the authors note may be restrictive in practice.They identify relaxing these assumptions as future work.Source: §VI · p. 10

APPENDIX A PRELIMINARIES ON ALGORITHMIC STABILITY

• Gradient updates are (1 + αβ)-expansive for β-smooth losses, and 1-expansive when the loss is also convex and α ≤ 2/β.The preliminaries adopt Hardt et al.'s notation.Source: Appendix A · p. 13

A. Proof of Theorem 1

• Theorem 1's proof compares datasets that differ in one example from task i to track the effect of that replacement on the algorithm's output.The coupled samples are identical except for the replaced task-i example.Source: A. Proof of Theorem 1 · p. 13

B. Proof of Theorem 2

• Theorem 2's proof couples SGD runs with the same initialization and sampled indices, then tracks a task-i perturbation through later updates.For subsequent tasks, the datasets are identical across the coupled runs.Source: B. Proof of Theorem 2 · pp. 13–14

C. Proof of Theorem 3

• Theorem 3's proof treats both intra-task and continuous decay; its bound also applies under random permutation when the cutoff t0 is below n.The derivation uses a cutoff and evaluates the resulting finite integral.Source: C. Proof of Theorem 3 · pp. 14–15

A. Proof of Theorem 4

• For two independent samples, replacing task i’s k-th example defines the modified sample; separately, the displayed gap averaged over S1:T and A satisfies |E[R_Di(A(S1:T)) − R_Si(A(S1:T))]| = |δ| ≤ εT,i.Source: A. Proof of Theorem 4 · p. 16

B. Proof of Theorem 5

• Theorem 5's proof carries a task-i perturbation into the final model by recursively applying 1-expansivity on tasks with unchanged data.Within task i, it distinguishes updates on matching versus differing sampled examples.Source: B. Proof of Theorem 5 · pp. 16–17

C. Proof of Theorem 6

• Theorem 6's proof analyzes intra-task and continuous decay using a cutoff; for the stated endpoint case, its integral evaluation yields N/n.The result also states that the same inequality holds under random permutation when t0 < n.Source: C. Proof of Theorem 6 · pp. 17–18

• The appendix states empirical-forgetting bounds for constant, intra-task decaying, and continuously decaying learning rates.The theorem statements specify the schedule cases, while the provided passages do not show the bounds' full expressions.Source: Appendix D · p. 19
• The proof bounds the effect of each later-task training step on task i's empirical risk, then sums those changes across subsequent tasks.It applies smoothness to the SGD update before accumulating the expected risk changes.Source: Appendix D · p. 19

A. Proof of Theorem 8

• Theorem 8 combines convex stability bounds with the empirical-forgetting bound to form an upper bound on forgetting in expectation.Its proof evaluates the contributions under the learning-rate schedules.Source: A. Proof of Theorem 8 · p. 19

B. Proof of Theorem 9

• Theorem 9 assembles intra-task generalization, cumulative cross-task generalization, and empirical forgetting into four bound cases.The proof derives separate cutoff cases for the IGG and cumulative CGG contributions before combining them.Source: B. Proof of Theorem 9 · pp. 20–21

APPENDIX F EXPERIMENTS ON SPLIT MINIIMAGENET WITH RESNET-18

• Split MiniImageNet is divided into 20 sequential five-class tasks; the experiments use a ResNet-18 backbone and three random seeds.Epochs per task, initial learning rate, and samples per task are varied; sample-size comparisons use a common budget of N = 1600 updates per task.Source: Appendix F · pp. 21–22
• Figure 5 reports IGG, CGG, EF, FT, and ACC across learning-rate schedules, with initial learning rates in the top row and epochs per task in the bottom row.The initial-rate sweep fixes epochs per task at 50; the epoch sweep fixes c = 0.1.Source: Figure 5 · p. 22, Appendix F · p. 22
• The gradient-alignment measurement compares current-task training gradients with previous-task buffer gradients at the same parameters, accumulating learning-rate-weighted inner products over subsequent updates.The buffers are used only for measurement, and the plotted negative cumulative alignment is shown alongside empirical forgetting without normalization by gradient norms, updates, or tasks.Source: Appendix F · p. 22
• Figure 6 plots EF and cumulative gradient alignment for constant, intra-task decaying, and continuously decaying learning rates, reporting three-seed means and standard deviations.Source: Figure 6 · p. 22
• Figures 7 and 8 examine IGG and CGG across per-task sample counts, and ACC and BWT for original CL, GEM, and A-GEM across learning-rate schedules.Higher ACC indicates better predictive performance, while higher BWT indicates less forgetting.Source: Figure 7 · p. 23, Figure 8 · p. 23