Source-linked AI summary
Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective
Jiuqi Wang, Jayanth Srinivasa, Claire Chen, Shuze Daniel Liu, Ali Payani, Shangtong Zhang
TL;DR
Deep continual learning needs networks to adapt to new tasks, but existing diagnostics have limited theoretical grounding as predictors of that ability. This paper studies trainability, proposes and analyzes Optimization Readiness, and finds it reliably ranks checkpoints across two supervised continual-learning settings, including with limited validation data.
Problem
Existing plasticity diagnostics have been empirically challenged, but theoretical understanding of why they fail remains limited.
Source: §1 · p. 2Method
Results
Optimization Readiness reliably ranks checkpoints by 1-, 10-, and 100-step gains on Slowly-Changing Regression and Permuted MNIST, outperforming several common diagnostics.
Source: §7 · p. 9Takeaways & Limitations
Structural diagnostic values alone do not guarantee future optimization progress, while Optimization Readiness lower-bounds one-step gain under smoothness and learning-rate conditions.
Source: §7 · p. 9The counterexamples use one-hidden-layer ReLU networks, and the empirical evaluation is limited to supervised continual learning.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Deep continual learning requires models to adapt to new tasks without retraining from scratch. However, neural networks can lose their ability to adapt to new tasks after training on previous ones, a phenomenon known as loss of plasticity. There have been several explanations and diagnostics proposed for plasticity loss. Motivated by the philosophy that "all models are wrong, but some are useful", we ask: can existing diagnostics predict a neural network's plasticity? In this work, we take a practical view to interpret plasticity as trainability, i.e., a neural network's future optimization gain on a target task. We first take a theoretical approach, showing, by constructing a few counterexamples, that some widely adopted diagnostics of plasticity, including representation rank and neural tangent kernel rank, can fail to predict the loss of trainability in both regression and classification settings. We instead propose a novel metric, called optimization readiness, which combines gradient strength and gradient reliability. We prove that optimization readiness lower bounds one-step optimization gain under standard smoothness assumptions, providing a theoretical guarantee for its predictive power. Empirically, we show that across commonly used deep continual learning settings, such as Slowly-Changing Regression and Permuted MNIST, optimization readiness more reliably ranks checkpoints by trainability than prior diagnostics, even with substantially fewer samples.
1 Introduction
The introduction asks whether existing diagnostics can predict plasticity, operationalized here as trainability, and notes that theoretical explanations for diagnostic failures remain limited. It presents counterexamples, Optimization Readiness, and evaluations on two continual-learning testbeds as its response.
1 Introduction
- Trainability is the meaningful relative loss reduction a network achieves through additional gradient-based optimization, used here as a concrete aspect of plasticity.Source: §1 · p. 2
- Existing falsification of plasticity diagnostics is largely empirical, leaving limited theoretical understanding of why the metrics can fail.Source: §1 · p. 2
- Counterexamples show that effective rank, 99% energy rank, and eNTK can fail to predict trainability in regression and binary classification.Source: §1 · p. 2
- Optimization Readiness combines gradient strength and reliability, and is proved to lower-bound one-step optimization gain under standard smoothness assumptions.Source: §1 · p. 2
- On Slowly-Changing Regression and Permuted MNIST, Optimization Readiness ranks checkpoint trainability accurately, including with substantially reduced validation data.Source: §1 · p. 2
2 Preliminaries and Problem Setup
This section formalizes continual supervised learning as sequential tasks and defines trainability through expected relative loss improvement under gradient optimization. It also introduces structural and activity-based diagnostics used for comparison.
2 Preliminaries and Problem Setup
- A continual supervised-learning problem consists of sequential tasks with shared pointwise loss, and each task has input and label spaces with a joint distribution.Source: §2 · pp. 2–3
- Trainability is measured by the relative loss reduction from continuing gradient-based optimization on a target task, with the k-step gain serving as its operational proxy.Source: §2 · p. 3
- Normalizing gain by initial loss makes improvement comparable across checkpoints with different starting losses and avoids misleading absolute-loss comparisons.Source: §2 · p. 3
- Representation diagnostics include effective rank and 99% energy rank of penultimate-layer representations, while related diagnostics use eNTK rank and curvature energy rank.Source: §2 · p. 3
- The diagnostic set also includes an active-neuron fraction computed from hidden-layer activations.Source: §2 · p. 3
3 Theoretical Falsifications of Structural Plasticity Metrics
The section constructs regression and binary-classification counterexamples showing that favorable representation and eNTK ranks do not ensure trainability. The failures arise when task gradients are orthogonal to directions usable by the output layer at initialization.
Regression Counterexample
- For regression, a checkpoint with high representation and eNTK rank is completely stuck under full-support gradient descent (full(θpre; T̄reg)=0 for all k≥1 and η>0), whereas random initialization achieves full(θrand; T̄reg)≥1−1/Mgap.Source: §3 · pp. 4–5
Binary Classification Counterexample
- Binary classification likewise has a high-rank checkpoint that is stuck, while random initialization achieves full-support gain at least 1−1/Mgap.The classification construction assumes an even number of support points.Source: §3 · p. 5
Theoretical Falsifications
- The constructions are intentionally stylized to isolate a failure mode, not to model typical practical continual-learning trajectories.Their conclusion is limited to showing that favorable structural values alone do not guarantee target-task optimization progress.Source: §3 · p. 5
- High representation and eNTK ranks do not by themselves imply trainability: both regression and binary-classification counterexamples exhibit this failure.The structural metrics are label-agnostic, while the counterexamples rely on label-dependent null directions.Source: §3 · pp. 3–5
- When the task gradient is orthogonal to usable output-layer directions, full-support gradient descent can leave the checkpoint unchanged and yield zero gain.For the analyzed checkpoint form, the output-layer gradient is zero; if the representation-gradient product is also zero, every full-support step leaves the parameters unchanged.Source: §3 · pp. 4–5
4 Optimization Readiness
Optimization Readiness (OR) combines gradient strength and gradient reliability to predict trainability from expected descent dynamics. Under stated assumptions and a suitable learning rate, it lower-bounds one-step optimization gain.
4 Optimization Readiness
- OR combines gradient strength, which measures whether a gradient signal exists, with gradient reliability, which reflects its signal-to-noise ratio.The metric is derived from expected stochastic-gradient descent dynamics rather than only checkpoint structure or geometry.Source: §4 · pp. 5–6
- Gradient strength normalizes squared gradient magnitude by population loss because raw gradient norm can mislead across checkpoints at different convergence stages.A checkpoint near convergence may have little room for absolute improvement despite a smaller absolute loss reduction.Source: §4 · p. 6
- Under unbiased mini-batch gradients, β-smooth population loss, and the theorem’s learning-rate condition, OR lower-bounds one-step gain.The bound scales with OR and depends on the reliability term and smoothness constant.Source: §4 · p. 6
5 Experiments
Experiments evaluate whether plasticity metrics rank checkpoints by future optimization gain in Slowly-Changing Regression and Permuted MNIST. OR ranks checkpoints most accurately across the tested tasks and horizons, while subsampling reveals a task-dependent loss in accuracy.
Experiments
- The testbeds are SCR regression and P-MNIST classification, with continual training used to generate checkpoints of varying trainability for held-out validation tasks.The MLPs use two hidden layers of five units for SCR and three hidden layers of 100 units for P-MNIST.Source: §5 · pp. 6–7
- The evaluation estimates k-step gain by running SGD on validation mini-batches, while OR is estimated from validation-set and independently sampled mini-batch gradients.The baseline set includes representation-rank metrics, active-neuron fraction, eNTK rank, and Hessian-rank metrics where applicable.Source: §5 · p. 7
Qualitative Analysis
- OR shows an approximately linear relationship with 1-step gain and remains strongly positively related to gain at k = 100.Other metrics do not show a clear positive correlation with trainability in the qualitative analysis.Source: §5 · p. 7
Quantitative Analysis
- OR achieves the highest pairwise ranking accuracy on both tasks for every tested k, with its largest margin in Slowly-Changing Regression.Table 2 reports mean pairwise ranking accuracy and standard error across tasks.Source: Table 2 · p. 8, §5 · p. 8
Qualitative Analysis
- OR’s trajectory aligns most closely with k-step gains, particularly in SCR; gains initially rise rapidly before gradually declining.The authors conjecture the initial rise may relate to forward transfer, temporarily facilitating adaptation to similar future tasks.Source: §5 · p. 8
Quantitative Analysis
- Pairwise ranking accuracy measures the fraction of checkpoint pairs whose metric ordering agrees with their ordering by estimated k-step gain.It has a probabilistic interpretation as the chance that a metric correctly ranks two checkpoints by trainability.Source: §5 · p. 8
Subsampling Ablation
- In SCR, OR remains most accurate with 0.1% of validation data; in P-MNIST, subsampling reduces its accuracy, but it remains strongest at the same data budget.On P-MNIST, its accuracy can fall below representation-rank metrics computed using the full validation set.Source: §5 · p. 8
6 Related Work
Prior work proposes structural, spectral, curvature, and neuron-activity diagnostics for plasticity, but their ability to predict future trainability remains unclear. This work complements empirical critiques with theoretical counterexamples and an optimization-derived metric supported by theory and experiments.
Explanations and Diagnostics of Loss of Plasticity
- Plasticity diagnostics span representation collapse, NTK spectra, Hessian-rank loss, finite-time Lyapunov exponents, and dormant or dead neurons.The cited works associate these structural or geometric properties with degraded trainability or optimization difficulty.Source: §6 · p. 9
Empirical Falsifications and Limitations of Existing Diagnostics
- Because existing diagnostics emphasize checkpoint structure or geometry, their predictive reliability remains unclear; this work directly evaluates future gain and provides theoretical counterexamples.The paper complements primarily empirical prior analysis with explicit theoretical failure modes.Source: §6 · p. 9
- Empirical studies report inconsistent or incomplete indicators, including weight and feature metrics, update norm, neuron dormancy, Hessian rank, sign entropy, and gradient norm.Lyle et al. [2023], Lewandowski et al. [2024], and Baveja et al. [2025] provide these empirical falsifications.Source: §6 · p. 9
7 Discussion and Conclusion
The discussion treats future optimization progress as the practical measure of plasticity, reports theoretical and empirical support for optimization readiness, and identifies limits in the study’s scope.
7 Discussion and Conclusion
- In regression and binary classification constructions, favorable representation-rank and eNTK-rank values coexist with no full-support loss reduction, while random initialization is likely to make substantial one-step progress.The discussion attributes the structural-metric failure to a label-dependent null direction, where the task gradient is orthogonal to directions usable by the output layer at initialization.Source: §7 · p. 9
- Optimization readiness combines gradient strength and reliability, lower-bounds one-step gain under smoothness and a suitable learning rate, and reliably ranks 1-, 10-, and 100-step gains on both evaluated tasks.The subsampling ablation also suggests it remains informative with limited validation data and few Monte Carlo samples.Source: §7 · p. 9
- The counterexamples use one-hidden-layer ReLU networks, and the empirical evaluation is limited to supervised continual learning.The paper identifies deeper networks, other activation functions, continual reinforcement learning, and larger-scale architectures and problems as future directions.Source: §7 · p. 10
B.1 Proof of Lemma 3.1
- When the first-layer gradient is zero and the output-layer condition also holds, gradient descent leaves the checkpoint unchanged and yields zero full-support gain.Source: §B.1 · p. 12
B.2 Proof of Lemma 3.2
- For Lemma 3.2, the proof uses the eigenvalues of a positive-semidefinite matrix B as its singular values, then scales after appending zero columns.Source: §B.2 · p. 12
B.3 Proof of Lemma 3.3
The proof bounds a random-feature variance term and applies matrix Bernstein to obtain a high-probability control under a width condition.
B.3 Proof of Lemma 3.3
- Each random vector h_j has coordinates equal to a with probability 1/2 and zero with probability 1/2, independently across coordinates.Source: §B.3 · p. 12
- The proof invokes matrix Bernstein to obtain a bound holding with probability at least 1 − δ.Source: §B.3 · p. 13
- With M at least Cn^2ρz, the proof states that the matrix-Bernstein bound is at most 1/√ρ.The bound holds after increasing the universal constant C.Source: §B.3 · p. 13
B.4 Proof of Theorem 3.4
The proof of Theorem 3.4 constructs a checkpoint with uniform normalized nonzero representation singular values and zero full-support gain under gradient descent.
B.4 Proof of Theorem 3.4
- At the constructed checkpoint, the normalized nonzero singular values of the representation are all 1/(n − 2).Source: §B.4 · p. 13
- For the zero-output checkpoint, full-support gradient descent leaves the parameters unchanged for every learning rate and has zero gain for every k ≥ 1.Source: §B.4 · p. 14
B.5 Proof of Theorem 3.5
The proof and estimation details cover a classification counterexample, a smoothness-based loss bound, and the procedures used to estimate optimization readiness and comparison metrics.
B.5 Proof of Theorem 3.5
- For the classification construction, full-support gradient descent leaves the checkpoint unchanged and its gain is zero for every k ≥ 1.Source: §B.5 · pp. 15–16
B.6 Proof of Theorem 4.6
- The smoothness proof bounds the updated loss using the inner product of the population gradient with the batch-gradient estimate and a βη² term.Source: §B.6 · p. 17
C.1 Estimation Details
- The validation set contains 10,000 examples, and gain-estimation rollouts use stochastic gradient descent with mini-batches of four samples drawn with replacement.Source: §C.1 · p. 17
- Optimization readiness estimation uses the full validation-set gradient and independently sampled mini-batches to estimate gradient strength and reliability.The procedure uses 128 mini-batches for the reliability estimate.Source: §C.1 · p. 18
- The study computes representation effective rank and 99% energy rank for both tasks, but omits eNTK rank on Permuted MNIST because its vector-output kernel is not directly comparable to the scalar-output case.Source: §C.1 · p. 18
C.2 Compute Resources
C.2 reports compute costs for the two testbeds; C.3 presents figures on learning curves, gain versus metric values, checkpoint trajectories, and subsampling.
C.2 Compute Resources
- SCR experiments ran faster on CPU and took less than 3 hours per seed; P-MNIST took about 10.5 hours per seed on an 8GB NVIDIA RTX 4000 GPU.Jobs ran on a shared Slurm cluster with opportunistic scheduling, so hardware allocation varied across runs.Source: §C.2 · p. 18
C.3 Figures
- Figure 1 shows learning curves for Slowly-Changing Regression and Permuted MNIST.Source: Figure 1 · p. 19
- Figure 2 plots 1-, 10-, and 100-step gain against plasticity metric values for both testbeds, with extreme metric values clipped for outliers.Source: Figure 2 · p. 19
- Figure 3 plots gains and metric values against checkpoints for both testbeds; shaded areas denote standard errors.Source: Figure 3 · p. 20
- Figure 4 presents the subsampling ablation results for both testbeds, with error bars denoting standard errors.Source: Figure 4 · p. 21