PaperlayerView exact revision on arXiv

Source-linked AI summary

Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive

Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen, Shishir Muralidhara, Federico Raue, René Schuster, Stanislav Frolov, Andreas Dengel

arXiv:2608.15901v1cs.LGcs.AI

TL;DR

Continual-learning regularizers use parameter-wise importance, but diagonal Fisher weights do not capture large layer-to-layer curvature differences that matter for forgetting. This paper analyzes a block-diagonal Hessian model, derives sensitivity-weighted regularization, and finds that protecting shallow layers while allowing deeper layers to move improves evaluated EWC and SLCA outcomes.

  • Problem

    EWC’s parameter-wise Fisher penalties lack layer-level sensitivity information, despite evidence that layers differ substantially in their vulnerability to weight changes.

  • Method

    Under a block-diagonal Hessian assumption, the paper derives layer-weighted forgetting bounds and the minimum-regret rule λ_ℓ∝s_ℓ, then applies it to EWC and SLCA.

  • Results

    Experiments support the rule on evaluated settings: Depth-weighted EWC on MediumCNN/CIFAR-100 peaks at α = 2 on avg-acc, while SLCA on ViT-B/ImageNet-21k reaches +0.7% over default.

  • Takeaways & Limitations

    On the evaluated pretrained backbones, protect shallow layers strongly and let deeper layers move when layer sensitivity is shallow-dominated.

  • Takeaways & Limitations

    The main results rely on a block-diagonal Hessian assumption that is violated by skip connections, LayerNorm, and attention.

Abstract

from arXiv · show

Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-parameter looks more flexible than per-layer, but each layer's diagonal Fisher is a weak summary of its actual curvature, missing the top-eigenvalue information that controls forgetting. Adversarial bit-flip attacks and Hessian-spectrum studies show that this missing per-layer sensitivity spans orders of magnitude in neural networks. Under a block-diagonal Hessian assumption, the layer-level analogue of EWC's existing diagonal assumption, we prove three things. Forgetting decomposes as a sum of per-layer terms weighted by each layer's top Hessian eigenvalue. Diagonal-Fisher weights cannot recover this eigenvalue. For instance, two layers with identical Fisher averages can have top eigenvalues differing by a factor as large as the layer width. For the same level of forgetting, uniform regularization loses new-task performance by an amount scaling with the layer condition number. Our theoretical analysis leads to a simple recipe: protect early layers strongly, let deeper layers move. We apply this recipe to EWC and SLCA and show clear improvements in average performance and forgetting metrics.

1 Introduction

Continual learning suffers catastrophic forgetting, while regularization methods penalize parameter drift without capturing layer-wide sensitivity. This paper argues for layer-adaptive regularization, supported by empirical evidence and a block-diagonal Hessian analysis.

  • Motivation: Continual learning trains one network across sequential tasks, causing catastrophic forgetting of earlier-task competence.The network may retain performance on the newest task while losing performance on previous tasks.
  • Regularization: Regularization-based methods prevent forgetting by penalizing parameter drift, with EWC using diagonal-Fisher importance weights.SI and MAS use path-integral importance and synaptic salience, respectively, while later methods retain the same penalty form.
  • Layer heterogeneity: EWC’s parameter-wise Fisher weights miss layer-wide sensitivity: layers with similar Fisher averages can differ tenfold in sensitivity to weight changes.This creates a static, layer-uniform view in which each parameter receives its own scalar importance tag.
  • Empirical motivation: More than 60% ImageNet accuracy can be lost by flipping fewer than 30 bits, while per-layer Hessian eigenvalues span two to three orders of magnitude.The flipped bits cluster in a small number of layers, and Hessian analyses independently find large differences across standard backbones.
  • Theoretical contribution: Under a block-diagonal Hessian assumption, forgetting decomposes into per-layer contributions weighted by each layer’s top sensitivity, which diagonal-Fisher weights cannot recover.The paper identifies this assumption as the layer-level analogue of EWC’s diagonal-Fisher assumption and attributes the mismatch to lost rank information.
  • Methodological contribution: TUNA and SLCA already make layer-dependent choices, and the paper extends EWC with a one-parameter geometric schedule to test layer-adaptive regularization.TUNA uses per-layer low-rank adapters, whereas SLCA uses separate learning-rate schedules for the backbone and head.

2 Related work

Related work spans per-parameter regularization, layer-adaptive updates, layer-wise scaling, and curvature-based sensitivity estimation. This paper positions scalar-per-layer regularization alongside these approaches while using true Hessian information to characterize layer sensitivity.

  • Regularization-based CL: EWC, SI, MAS, RWalk, and online EWC assign per-parameter diagonal importance weights, whereas replay and parameter isolation are orthogonal to this analysis.EWC uses the diagonal Fisher; SI and MAS use path-integral and output-sensitivity importance, and RWalk connects them through a KL-based Riemannian formulation.
  • Layer-adaptive CL: SLCA uses smaller learning rates for pretrained backbones than heads, while prompt methods largely freeze the backbone and update an input-prompt subspace.These methods impose different priors on which layers are more sensitive.
  • Layer-wise scaling: Layer-wise update scaling originates in optimization and fine-tuning, while this work introduces its scalar-per-layer counterpart on the regularization side.Related importance methods include second-order approaches such as Optimal Brain Damage, WoodFisher, EigenDamage, and SparseGPT, plus first-order signals such as magnitude pruning and the Lottery Ticket Hypothesis.
  • Per-layer Hessian sensitivity: Adversarial bit-flip attacks concentrate destructive bits in a small number of layers, providing an empirical lower bound on the spread of per-layer sensitivity.Hessian studies further report a bulk-plus-outlier structure in trained networks.
  • Per-layer Hessian sensitivity: PyHessian and Lanczos provide scalable Hessian estimators, while K-FAC and Shampoo exploit block-diagonal structure and the paper computes the true Hessian instead of using empirical-Fisher approximations.The flat-minima literature operationalizes top-eigenvalue sensitivity during training, and NTK-based work studies continual-learning landscapes.

3 Methodology

The methodology models forgetting through a second-order, block-diagonal Hessian and decomposes it into layer-specific costs governed by each layer’s top eigenvalue. It shows why diagonal-Fisher or uniform penalties can misallocate the forgetting budget, motivating sensitivity-adaptive regularization that protects early layers more strongly.

  • Layer-wise forgetting: Under the block-diagonal Hessian assumption, second-order forgetting decomposes into independent per-layer terms, making allocation of a forgetting budget an explicit optimization problem.The assumption is exact for fully factorized architectures and approximate in real networks with off-diagonal effects from skip connections, LayerNorm, and attention.
  • Layer sensitivity: Each layer’s sensitivity is its Hessian block’s top eigenvalue, which acts as a price on parameter displacement and bounds that layer’s forgetting contribution.Measured sensitivity profiles are heavy-tailed: ResNet50 has κ = 134.6, while ViT-B/16 has κ = 27.4.
  • Diagonal-Fisher limitation: Layers with identical mean diagonal Fisher can have top eigenvalues differing by up to the smaller layer width, so diagonal-Fisher weights can treat substantially different forgetting risks as identical.The discrepancy is especially relevant in the low-rank Hessian regime observed in trained networks, where the bound can be nearly tight.
  • Uniform-regularization regret: Uniform regularization has no regret when κ = 1 or when new-task gradients avoid fragile directions, but otherwise its regret is strictly positive and non-decreasing in κ.The gap isolates the stability-plasticity trade-off by comparing regularizers calibrated to the same actual forgetting budget.
  • Adaptive regularization: Matching layer penalties to top-eigenvalue sensitivities is optimal along worst-case layer directions, yielding geometric penalty schedules when sensitivities vary geometrically with depth.Empirically, early convolutional stages and ViT patch embeddings dominate sensitivity, motivating the rule to protect early layers strongly and let deeper layers move.

4 Experiments

Experiments test depth-adaptive regularization in EWC on small CNNs and SLCA on pretrained ViT-B/16 models. The results show gains over uniform or default schedules, while optimal depth weighting depends on backbone and pretraining.

  • Experiment 1: Depth-weighted EWC on Split-C: Depth-weighted EWC replaces uniform λ with λℓ = c/αℓ−1 and evaluates α and c across four continual-learning metrics on small CNNs.Uniform EWC is α = 1; other α values are layer-adaptive.
  • Experiment 2: SLCA depth-weighted schedule on ViT-B/16: SLCA implements depth weighting through per-layer learning-rate scaling rather than per-layer EWC penalties, with larger α assigning different movement allowances across depth.The schedule uses a different sign convention from Experiment 1.
  • Experiment 1: Depth-weighted EWC on Split-C: 59.8% ± 1.5 average accuracy at α = 2, c = 200 exceeds 57.2%±1.3 at α = 1 with non-overlapping bands.Forgetting and BWT instead prefer α ≤1, indicating different metric optima.
  • Experiment 2: SLCA depth-weighted schedule on ViT-B/16: 92.01% final-task accuracy at c = 1, α = 1.18 improves on default SLCA’s 91.29% by +0.7% absolute.The average-incremental gain is +0.13% (95.03% vs 94.90%) and is direction-consistent across c values.
  • Cross-pretraining comparison: 91.35 →91.80 last-acc and 94.40 →94.51 inc-acc show improvement over default SLCA on ImageNet-21k, while MoCoV3 yields 84.26 vs 84.44.The MoCoV3 result is interpreted as default SLCA already approximating the favorable per-layer schedule.
  • Cross-pretraining comparison: The experiments support protecting shallow layers more strongly on standard pretrained backbones, while the CNN sweep instead favors deeper-layer sensitivity for average accuracy.Changing pretraining on the same backbone shifts the optimal α in the predicted direction.

5 Discussion and limitations

The framework isolates per-layer operator norms by extending EWC’s second-order and diagonal assumptions to a block-diagonal setting. Its limitations are that the quadratic surrogate is local, while block diagonality fails around skip connections, LayerNorm, and attention.

  • Framework scope: The framework is deliberately minimal, inheriting EWC’s second-order approximation and adding a block-diagonal analogue of its diagonal assumption.This isolates per-layer operator norms s_ℓ as the missing piece.
  • Limitations: The quadratic surrogate is valid only inside a trust region around θ⋆.This limitation is documented in Appendix B.
  • Limitations: The block-diagonal assumption is violated at boundaries created by skip connections, LayerNorm, and attention.Appendix A provides a continuous relaxation and a granularity diagnosis.

6 Conclusion

The conclusion identifies a layer-sensitivity gap in EWC: diagonal Fisher penalties treat layers equally despite large differences in Hessian top eigenvalues. It formalizes the resulting forgetting and regret, then gives a closed-form layer-adaptive remedy computable cheaply per layer.

  • 6 Conclusion: Per-layer Hessian top eigenvalues span two orders of magnitude across every standard backbone measured, although EWC’s diagonal Fisher penalty treats layers equally sensitive.This exposes the mismatch between parameter-level Fisher weighting and actual layer-level sensitivity.
  • 6 Conclusion: Forgetting decomposes by s_ℓ, diagonal Fisher penalties cannot recover s_ℓ, and uniform regularization incurs regret (κ −1)/κ.These results formalize the consequences of using layer-insensitive regularization.
  • 6 Conclusion: The closed-form remedy sets λ_ℓ∝s_ℓ and requires only one cheap power iteration per layer.This provides a computationally practical layer-adaptive regularization rule.

A Relaxed block-diagonal Hessian

Allowing bounded cross-layer Hessian coupling yields a decomposition and regret bound with an effective condition number that weakens as coupling and depth increase. Measurements show coupling is substantial and non-banded, while block-diagonal curvature captures only part of realized quadratic terms.

  • Decomposition: Under bounded cross-layer coupling, the second-order term decomposes into block-diagonal curvature plus a controlled coupling contribution.Proposition 11 provides the quantitative replacement for the block-diagonal decomposition.
  • Regret bound: κeff = κ/(1 + ρ(L − 1)) approaches κ as ρ → 0 and κ/L as ρ → 1, attenuating the framework’s gain with depth-coupling.The result is Corollary 12’s effective condition-number replacement for Theorem 6.
  • Granularity diagnostic: On ResNet-50, mean adjacent ρ is 1.34 for 3 super-blocks, 1.27 for 6 residual stages, and 0.92 for 18 blocks, so coarser partitions do not lower coupling.A working threshold of ρ ≤0.3 was not attained at any measured granularity.
  • All-pairs coupling: Mean non-adjacent ρ exceeds adjacent ρ on both ResNet-50 and ViT-B/16, indicating coupling is not banded around the diagonal.The means are 1.80 versus 1.25 on ResNet-50 and 0.98 versus 0.81 on ViT-B/16; normalization by ResNet-50’s fc block can inflate entries up to 2.9.
  • Tightness probes: 0.71 ± 0.16 of the full quadratic is captured by the block-diagonal term on ResNet-18, versus 0.50 ± 0.08 on ResNet-50.Across 9 of 9 runs, the gap between the full and block-diagonal quadratic stayed within Proposition 11’s measured coupling bound; 2 runs showed negative forgetting.

B Full proof of Theorem 6

Under block-diagonal task curvature and regularizer structure, the proof compares the per-layer oracle with uniform regularization and derives a worst-layer lower bound scaling with (κ −1)/κ. If the new-task Hessian couples layers, the same structure survives but the constant can shrink by a factor of L.

  • Assumptions: The proof assumes quadratic new-task loss with µ_QI ⪯ Q ⪯ M_QI, while both Q and H are block-diagonal across layers.These assumptions make the layerwise optimization separable.
  • Per-layer KKT: The oracle’s per-layer update uses (Q^(ℓ,ℓ) + βH^(ℓ,ℓ))^-1, whereas uniform regularization substitutes Q^(ℓ,ℓ) + βµI.This follows from the per-layer KKT subproblem at forgetting budget B and multiplier β.
  • Uniform sensitivity: When κ > 1, the effective uniform sensitivity µ lies strictly between the minimum and maximum layer sensitivities.The proof identifies µ as an average of layer-level quantities, placing it between min_ℓ s_ℓ and max_ℓ s_ℓ.
  • Worst-layer lower bound: (κ −1)/κ: the worst-layer comparison yields Theorem 6’s bound after substituting c_0 = µ_Q/(M_Q + µ_Q).The bound follows by restricting attention to a maximum-h mode in a layer whose sensitivity exceeds µ.
  • Coupled new-task Hessian: For coupled Q, a weaker bound replaces µ_Q with the smallest eigenvalue of an off-block-diagonal-augmented Q matrix; c_0 can shrink by L, but (κ −1)/κ remains.The per-layer KKT step no longer decouples cleanly when Q couples layers.

C Full proof of Theorem 8 · D SLCA LR-to-λ equivalence

The proof shows that matching oracle and scalar penalties along worst-case directions requires layer weights proportional to sensitivity, while scalar penalties cannot reproduce anisotropic Hessians. The SLCA analysis shows that layerwise learning-rate scaling can be matched in displacement norm by an implicitly determined λ-schedule, though not by exact trajectory identity.

  • C Full proof of Theorem 8: For worst-case displacements Δθ(ℓ) = cℓv(ℓ), the two penalties agree only when λℓ/sℓ is layer-independent, so λℓ ∝ sℓ.This matches the oracle penalty and scalar-family penalty on the specified displacement family.
  • C Full proof of Theorem 8: A scalar λℓ has one degree of freedom per layer, whereas a dℓ-dimensional symmetric Hessian has dℓ(dℓ + 1)/2 degrees of freedom.Thus scalar regularization maps only into the one-dimensional identity subspace of Symdℓ.
  • C Full proof of Theorem 8: Exact matrix matching requires H(ℓ,ℓ) itself to lie in the identity subspace, so scalar regularization cannot reproduce a general anisotropic layer Hessian.The condition is λℓI = H(ℓ,ℓ).
  • D SLCA LR-to-λ equivalence: SLCA analyzes gradient descent on the new-task quadratic surrogate ˜Lk(Δθ) = 1/2Δθ⊤QΔθ + g⊤Δθ from Δθ0 = 0.Layerwise learning-rate scaling γℓ produces a separate trajectory in each layer.
  • D SLCA LR-to-λ equivalence: λ-penalized descent replaces each layer curvature block Q(ℓ) with Q(ℓ) + λℓI, yielding fixed point −(Q(ℓ) + λℓI)−1g(ℓ).This fixed point has the same direction as −(Q(ℓ))−1g(ℓ) but smaller magnitude.
  • D SLCA LR-to-λ equivalence: For any γℓ ∈ (0, 1] and total step budget T, some λℓ ≥ 0 can make the learning-rate and penalty trajectories agree in displacement norm.The exact matching λℓ is implicit rather than an explicit algebraic identity.
  • D SLCA LR-to-λ equivalence: SLCA’s γbackbone = 0.1 and γhead = 1 correspond in displacement norm to a two-group λ-schedule whose ratio is implicitly determined by T.The framework predicts only the sign: the more-sensitive group should receive the smaller learning rate.

E Direct sℓmeasurements

Direct Hessian measurements estimate each layer’s top eigenvalue with power iteration and reveal stable, strongly heterogeneous sensitivity profiles across trained backbones. The profiles vary with architecture, pretraining, and initialization, while non-negligible inter-layer coupling limits the strict quantitative block-diagonal guarantee.

  • Measurement procedure: Top per-layer Hessian eigenvalues are estimated by 15-step power iteration using Hessian-vector products, costing O(k) HVPs per layer.Each HVP uses PyTorch double-backward autodiff; ViT-B/16 requires the math SDPA kernel.
  • Measurement stability: Across five calibration batches, layer ordering remains stable while absolute s_ℓ values fluctuate by approximately 5–15%.Table 5 reports κ as the mean over seeds.
  • Backbone scan: ResNet κ increases from 55 for R18 to 195 for R50, with mild saturation at 181 for R101; ViT κ is approximately 25–30.The dominant block is consistently early in ResNets, whereas it varies with depth in vision transformers; all satisfy κ ≫1.
  • Off-diagonal coupling: Adjacent-block coupling ranges from ρ = 0.76 to 1.34, so the qualitative block-diagonal structure holds but the strict ρ < 1 regret bound can fail.The degradation is a genuine caveat at large coupling, especially across adjacent residual stages.
  • Initialization and training state: At random initialization, heterogeneity persists but ordering reverses: deepest blocks become hottest, with ResNet-50 rising from s_stem ≈301 to s_fc ≈127,000.ResNet-18 peaks at layer4 near 32,000, and SmallCNN has κ ≈200; trained-checkpoint profiles govern EWC anchors.

F Depth-weighted EWC on from-scratch backbones … J Full proof of Proposition 5

Across from-scratch, pretrained, and adapter-based experiments, layer-adaptive protection generally improves continual-learning behavior, but its benefit depends on backbone sensitivity and schedule shape. The proof formalizes why diagonal summaries can miss layer-level curvature, with the Fisher-to-top-eigenvalue ratio spanning [1, d].

  • F Depth-weighted EWC on from-scratch backbones: For MediumCNN on Split-CIFAR-100, forgetting rises from ∼0.005 at α = 2−2 to ∼0.16 at α = 23.The result illustrates substantial depth-dependent sensitivity in the discriminating from-scratch setting.
  • F Depth-weighted EWC on from-scratch backbones: On SmallCNN, forgetting at α = 23 is ∼8× higher than at α = 2−2, while average-accuracy differences remain statistically marginal.The maximum average accuracy occurs at c = 200 and α ∈{1, 2}, with overlapping 3-seed standard-deviation bands at α = 0.5.
  • F Depth-weighted EWC on from-scratch backbones: On ResNet-18, the geometric schedule does not consistently beat uniform, because measured per-layer sensitivities are non-monotone in depth and cannot be fit by λℓ∝γℓ−1.The average-accuracy maximum for c = 200 is at α = 1 with mean 0.55, with overlapping bands at α ∈{0.5, 4}.
  • G SLCA depth-weighted sweep on MoCoV3 pretraining: On ImageNet-21k, the geometric SLCA schedule beats uniform on every seed, with a paired Last-acc gain of +0.450 and paired t-test p = 0.0018.Per-seed gains are +0.52/+0.56/+0.51/+0.44/+0.22.
  • H TUNA layer-wise orthogonality sweep: For TUNA on ImageNet-R B0-Inc20, the optimum lies around α ∈{21, 22}, indicating that late-block orthogonality should be relaxed.The reported sweep uses per-layer weights λj = c/αj on a pretrained ViT-B/16 backbone.
  • H TUNA layer-wise orthogonality sweep: On CIFAR-100 B0-Inc10, the accuracy span is only about 0.25%, yet the optimum consistently lies away from α = 1, typically near α ∈{2−1, 21}.The adapter constraint already performs most of the work, leaving limited room for layer-wise tuning.
  • I Schedule-shape comparison: All three monotone-decreasing schedule families beat uniform by ∼2–7% depending on c, while their best points differ by only ∼1–4%.The step family is strongest at higher c, so schedule shape is a second-order choice after protecting shallow layers more strongly.
  • J Full proof of Proposition 5: For positive semidefinite A with identical diagonal entries, λmax(A)/(1/d Σ_i λ_i) continuously spans [1, d], showing diagonal averages cannot determine top curvature.The endpoints correspond to equal eigenvalues and rank(A) ≤1, respectively.

K Direct test of the prescription on EWC · L Forgetting swing vs measured κ · M Trace proxy vs operator-norm sensitivity

Tests of the layer-adaptive prescription show that literal measured-schedules can underperform because extreme ratios and noisy curvature estimates over-protect early layers, while smoothed schedules recover the intended direction more robustly. Forgetting swing increases with measured condition number, and trace estimates preserve layer ordering but understate operator-norm sensitivity.

  • K Direct test of the prescription on EWC: The measured-s schedule underperforms uniform EWC on both backbones by 4–14% absolute.On ResNet-50, the measured s_layer1/s_fc ratio is 134, producing an approximately 134-fold regularization ratio that over-protects layer1 and limits head plasticity.
  • K Direct test of the prescription on EWC: The prescription is narrower than applying λ_ℓ∝s_ℓ literally: the optimum lies in the s_ℓ direction, while geometric schedules robustly sweep that direction without fixing noisy exact ratios.Dynamic range and per-layer measurement noise motivate smoothing rather than abandoning the prescription.
  • K Direct test of the prescription on EWC: Interpolated schedules λ_ℓ∝(s_ℓ/s_med)^β test ratio compression from uniform EWC at β = 0 to the literal measured schedule at β = 1.The sweep used β ∈ {0, 0.25, 0.5, 0.75, 1} and c ∈ {50, 200, 1000, 5000} on Split-CIFAR-100 with three seeds.
  • K Direct test of the prescription on EWC: Per-task re-measurement shows that drift is secondary and magnitude smoothing is primary.The online-smoothed arm uses β = 0.5, averages three calibration batches per boundary, and costs approximately 1.5 s on ResNet-18 and 3.6 s on ResNet-50 per boundary.
  • K Direct test of the prescription on EWC: ResNet-18 peaks at β*= 0.5 with avg-acc 0.590±0.014 versus 0.557±0.021 for uniform, whereas ResNet-50 is best at β = 0 with 0.441±0.015.For ResNet-50, avg-acc decreases monotonically to 0.330 ± 0.120 at β = 1.
  • L Forgetting swing vs measured κ: Forgetting swing is defined as the maximum-across-c gap between α = 2−2 and α = 23, providing an empirical test of the prediction that uniform-regularization regret grows with κ.The comparison interprets larger forgetting under-protection as evidence that more sensitive deep layers incur greater cost when not protected.
  • M Trace proxy vs operator-norm sensitivity: Trace-divided-by-width preserves the qualitative ordering and identifies the same dominant block, but compresses the spread and understates κ’s dynamic range.The dominant blocks are layer1 on ResNet-50 and patch_embed on ViT-B/16; power iteration remains preferable for the operator-norm regret bound, while trace is a cheap first sweep.
  • L Forgetting swing vs measured κ: Across the two pretrained backbones, forgetting swing is monotone in measured κ and consistent in magnitude with the bound’s (κ −1)/κ scaling.SmallCNN and MediumCNN were placed on the left margin because their s_ℓ profiles were not measured.

N LIH on the BERT-base language backbone

On BERT-base, token embeddings dominate layer curvature while transformer blocks and the head remain substantially less sensitive, extending the LIH pattern beyond vision. In text continual learning, uniform EWC improves substantially over sequential fine-tuning, while measured layer weighting provides no significant additional benefit, consistent with BERT’s modest condition number.

  • Implications: The framework’s prescription extends to NLP in principle, but the regularization schedule should depend on the chosen model’s per-layer curvature profile.On BERT, the smaller κ implies a correspondingly smaller rule-of-thumb gain from depth-weighted regularization.
  • Curvature profile: The measured BERT-base layer condition number was κ = 9.15, lower than the 20–200 range observed on vision backbones but still above 1.The ordering indicates a small number of high-curvature layers, especially the token-embedding block, with the remaining architecture spread across roughly an order of magnitude.
  • Text continual-learning benchmark: In a five-dataset text-classification sequence, BERT-base used separate task heads while EWC regularized the shared encoder parameters.The sequence was AG News, Yelp Review Full, Amazon Review Full, Yahoo Answers, and DBpedia, in that fixed order.
  • Text continual-learning benchmark: 10 percentage points: uniform EWC’s average accuracy gain over sequential fine-tuning; forgetting fell from 0.158 to 0.027.The benchmark used BERT-base on the five-dataset sequence, with sequential fine-tuning as the unregularized baseline.
  • Text continual-learning benchmark: −0.6 to +0.3 percentage points: measured-s versus uniform EWC at every c, with no significant difference (p ≥ 0.09).This null result matches the prediction associated with BERT-base’s measured κ = 9.15.

O Per-task sℓvariation and ordering stability … 10. Broader impacts

Layer-importance rankings remain stable across task batches and trained checkpoints, while experience replay is backbone-dependent and can complement EWC. The paper reports efficiency benefits alongside dual-use and privacy considerations, reproducibility details, compute accounting, and responsible-use disclosures.

  • O Per-task sℓvariation and ordering stability: Across all 10 tasks, the per-block sℓ ordering is preserved, with layer1 highest and stem lowest, while magnitudes vary by approximately 2×.The task-wise rank is invariant despite calibration-batch changes.
  • O Per-task sℓvariation and ordering stability: From task 2 onward, sℓ rankings remain stable across task boundaries, supporting re-measuring the regularization schedule at task boundaries rather than fixing it at initialization.Initialization is fc-heavy, but after task 1 the profile becomes stage3-heavy and later rankings stabilize.
  • P Comparison with experience replay: ER dominates both EWC arms on ResNets at every tested memory budget, whereas every EWC arm beats ER on the small from-scratch CNN.The geometric schedule is the best EWC arm overall on the small from-scratch CNN.
  • P Comparison with experience replay: A small replay buffer and geometric EWC compose: ER+EWC combines M = 500 with α = 2 and c = 200, improving over ER alone on MediumCNN.The cited composition uses the best geometric α reported for MediumCNN.
  • Q Broader impacts: Layer-adaptive regularization may reduce compute needed to maintain deployed models, while κ provides a single-number diagnostic for avoiding unnecessary sweeps.The efficiency effect compounds with the total number of model updates over a system’s lifetime.
  • Q Broader impacts: The prescribed sℓ sensitivity overlaps with quantities used to identify layers for bit-flip, weight-perturbation, or trojan-implant attacks, creating a documented dual-use concern.The paper states that it introduces no new attack capability.
  • R Compute resources: Experiments used one NVIDIA GPU, 4–8 CPUs, 32–64 GB RAM, and 3–10 h walltime per job; the full effort consumed roughly 800–1200 GPU-hours.The estimate includes preliminary, debugging, abandoned, and appendix-only runs.
  • NeurIPS Paper Checklist: The paper reports numbered theoretical results with stated assumptions, full or deferred proofs, complete experimental grids, public benchmarks, error bars, and planned release of companion code and launch scripts.The checklist states that no private data, human subjects, new models, or datasets are involved.
Loading 2608.15901v1…