PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Parameter Protection: Fisher-Shaped EWC for Replay-Free Continual Learning

Liying Wang, Simone Magistri, Joost van de Weijer

arXiv:2610.37145v1cs.LGcs.AI

TL;DR

In replay-free continual learning, EWC's local parameter-importance estimate does not generally capture sensitivity after finite parameter displacement. FS-EWC shapes solutions using neighborhood Fisher sensitivity and constructs retained references from optimization trajectories. It generally improves on corresponding EWC baselines across vision and language benchmarks, with controlled results indicating better retention despite greater displacement.

  • Problem

    EWC's local parameter-importance estimate does not generally characterize sensitivity after finite future displacement, leaving a gap in how replay-free learning accounts for retention.

    Source: §1 · p. 3

  • Method

    FS-EWC penalizes worst-case empirical FIM trace within a prescribed neighborhood during task learning and uses trajectory-based model merging to construct the retained reference.

    Source: §1 · p. 3, §3.3.1 · p. 5, §3.3.2 · p. 5

  • Results

    Across diverse vision and language benchmarks FS-EWC generally improves on corresponding EWC baselines, and FS-EWCLoRA's Task 1 test-loss increase was 0.033 versus 0.229 for EWCLoRA.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The results suggest that future forgetting should be considered during learning alongside protection of acquired knowledge.

    Source: §5 · p. 10

    Training accesses only the current task's St, with historical examples neither retained nor replayed, and future drift is considered within a prescribed neighborhood.

    Source: §3.1 · p. 3, §3.2 · p. 4

Abstract

from arXiv · show

In replay-free continual learning, Elastic Weight Consolidation (EWC) uses Fisher information matrix (FIM) to protect acquired knowledge by penalizing changes to historically important parameters. However, a local importance estimate at the retained solution does not in general characterize sensitivity after finite parameter displacement. This motivates controlling Fisher sensitivity over a neighborhood of potential future updates. We propose Fisher-Shaped EWC (FS-EWC), which extends the role of FIM from parameter protection to solution formation for future retention. During task learning, FS-EWC minimizes the worst-case empirical FIM trace within a prescribed neighborhood, explicitly favoring solutions that are less sensitive to bounded future displacement. At task completion, trajectory-based model merging constructs the retained solution from multiple optimization states rather than a single terminal iterate, after which FIM is estimated for subsequent consolidation. Experiments across multiple benchmarks show that FS-EWC improves performance over EWC, while controlled studies indicate a more favorable stability--plasticity balance and greater tolerance to parameter drift. These results suggest that effective knowledge retention requires both forming solutions with future retention in mind and protecting acquired knowledge.

1 INTRODUCTION

EWC protects knowledge using locally estimated parameter importance, but finite future drift can expose sensitivity not captured at the retained solution. FS-EWC shapes solutions for future retention and reports gains over EWC across vision and language benchmarks.

1 INTRODUCTION

  • EWC's local importance estimate can miss sensitivity that emerges after finite parameter drift, motivating protection that accounts for the surrounding parameter neighborhood.Existing extensions mainly refine importance construction or use while treating the solution being protected as given.Source: §1 · pp. 1, 3
  • Reducing the worst-case FIM trace within a prescribed neighborhood tightens a bound on potential loss increase under bounded future displacement.This makes future retention an explicit consideration during current-task learning.Source: §1 · p. 2
  • FS-EWC adds a neighborhood FIM-trace penalty during task learning while preserving historical EWC regularization.At task completion, trajectory-based model merging constructs the retained reference from multiple optimization states, and FIM is estimated there for subsequent protection.Source: §1 · pp. 2–3
  • Across diverse vision and language benchmarks, FS-EWC generally improves on corresponding EWC baselines under both full-parameter and LoRA training.Controlled studies also report improved retained-task and new-task accuracy when solution shaping is combined with historical protection.Source: §1 · p. 2
  • The proposed design principle combines shaping solutions for future retention with protecting acquired knowledge during subsequent learning.The paper reports gains over corresponding EWC baselines and complementarity with alternative FIM computation schemes and matrix representations.Source: §1 · p. 3

2 RELATED WORKS

The related work frames replay-free continual learning as balancing retention and adaptation without historical examples. Prior work strengthens parameter protection or improves FIM computation and representation, while generally treating the protected solution as given.

Continual Learning

  • Replay-free continual learning must balance retaining prior knowledge and adapting to new tasks without revisiting historical examples.Replay-based methods alleviate this tension by revisiting previous examples.Source: §2 · p. 3

Elastic Weight Consolidation

  • EWC extensions refine how historical importance is accumulated and used, but largely treat the solution being protected as given.Examples include online consolidation, trajectory-aware importance estimation, and EWC-style regularization for LoRA adaptation.Source: §2 · p. 3

FIM Estimation and Representation

  • Prior FIM work improves its computation or representation, whereas this paper addresses how the reference solution itself is formed.The cited representation approaches include rotated and Kronecker-factored structures beyond diagonal approximations.Source: §2 · p. 3

3 METHODOLOGY

The methodology formalizes why local Fisher importance may not capture loss changes under finite future drift, then shapes sensitivity across a neighborhood and constructs the retained anchor from trajectory checkpoints.

3.1 PRELIMINARIES

  • Replay-free class-incremental learning trains on each task’s data without retaining or replaying historical examples, while inference distinguishes all observed classes without task identities.The retained parameters and Fisher-based importance from earlier tasks provide the reference for subsequent learning.Source: §3.1 · pp. 3–4

3.2 SOLUTION SENSITIVITY TO FUTURE DRIFT

  • A finite future displacement can change task loss according to Fisher sensitivity along the displacement path, motivating sensitivity control throughout a bounded neighborhood rather than only at the anchor.The future displacement is unknown during current-task learning, so the neighborhood provides a direction-agnostic sensitivity criterion.Source: §3.2 · p. 4

3.3 FISHER-SHAPED EWC

  • FS-EWC combines neighborhood sensitivity shaping during current-task learning with a rule for selecting the retained anchor where current-task importance is estimated.It preserves the inherited EWC penalty while guiding the current solution toward lower sensitivity to bounded future displacement.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.1 NEIGHBORHOOD FISHER SENSITIVITY SHAPING

  • The worst-case neighborhood FIM criterion considers the largest sensitivity within radius r, and its gradient adds a Hessian–vector product correction while retaining EWC’s historical protection.The neighborhood locations used to evaluate sensitivity are distinct from the possible future parameter drift.Source: §3.3.1 · p. 5

3.3.2 TRAJECTORY-BASED ANCHOR CONSTRUCTION

  • The retained task anchor averages selected optimization checkpoints instead of using only the terminal iterate, and current-task FIM is estimated at that anchor for subsequent consolidation.The average can be accumulated online without storing the full optimization trajectory; normalized and damped current-task FIM is consolidated with historical importance.Source: §3.3.2 · p. 5

4 EXPERIMENTS

Across vision and language benchmarks, FS-EWC improves on matched EWC baselines; controlled studies show retention gains despite greater parameter movement and a larger empirically tolerated displacement. Results also depend on retained-state construction, while gains persist across tested Fisher configurations and neighborhood radii.

4.2 MAIN RESULT

  • FS-EWCLoRA raises ImageNet-A FAA from 56.42% to 62.08% and reaches 79.81% FAA on ImageNet-R, surpassing compared LoRA baselines there.Across four standard vision benchmarks, FS-EWC improves FAA over matched EWC under both full-parameter fine-tuning and LoRA.Source: §4.2 · p. 6, Table 1 · p. 6
  • FS-EWCLoRA beats matched EWC on all five heterogeneous vision streams and raises mean FAA from 64.70% to 69.15% (1B) and 70.31% to 73.08% (3B) in continual NLP.For NLP, improvements occur in five of six model–order comparisons.Source: §4.2 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7
  • On ImageNet-A, FS-EWCLoRA has larger final effective-weight displacement than EWCLoRA (1.442 versus 1.254), yet lower Task 1 test-loss increase (0.033 versus 0.229) and higher accuracy (86.64% versus 81.11%).Its path Fisher is also lower (0.015 versus 0.240), so the measured retention gain does not rely on stronger suppression of subsequent updates.Source: §4.2 · p. 7, Figure 3 · p. 8
  • Under matched perturbation radii, FS-EWCLoRA has lower positive training-loss increase and empirical-Fisher trace, while tolerance radius rises from 11 to 12.25 at τ = 0.01.Tolerance is larger for ten of eleven evaluated loss budgets.Source: §4.2 · p. 7, Figure 4 · p. 8
  • On ImageNet-A, shaping Task 1 lowers endpoint trF emp from 31.44 to 21.93 and improves subsequent retention with or without the same historical protection.Retention changes from 82.41% to 86.54% without protection and from 82.95% to 86.32% with C + P; Task 2 shaping improves adaptation from 76.25% to 81.89% while shifting balance away from immediate Task 1 retention.Source: §4.2 · p. 8, Table 4 · p. 8
  • Across six evaluated FIM computation–representation combinations, FS-EWC improves FAA over matched EWC by 6.08 to 9.07 percentage points.These gains exceed those from refining FIM computation or representation alone in the reported settings.Source: §4.2 · pp. 8–9, Table 5 · p. 9

4.3 ABLATION AND HYPERPARAMETER STUDIES

  • Trajectory averaging achieves the highest AAA; adding checkpoint density yields no further improvement, while including initialization lowers FAA and AAA despite improving BWT.Multi-checkpoint construction generally reduces forgetting relative to the endpoint, and the benefit depends on which learned states are combined.Source: §4.3 · p. 9, Table 6 · p. 9
  • On ImageNet-A, Figure 5 shows FS-EWCLoRA remains above the matched EWCLoRA baseline throughout the tested prescribed neighborhood-radius range, while AAA varies only moderately as radius changes.Source: §4.3 · p. 9, Figure 5 · p. 9

5 CONCLUSION

EWC protects knowledge using local parameter importance, but finite future drift depends on sensitivity beyond the retained solution. FS-EWC shapes solutions during learning while retaining EWC protection, and shaped solutions can tolerate greater subsequent displacement with less forgetting.

5 CONCLUSION

  • EWC's local parameter-importance protection does not account for sensitivity after finite future parameter displacement.The conclusion frames this as a constraint on EWC's stability–plasticity trade-off.Source: §5 · p. 9
  • FS-EWC shapes current-task solutions for future retention while preserving subsequent EWC protection.It incorporates Fisher information into solution formation as well as later protection.Source: §5 · p. 10
  • Shaped solutions can undergo larger subsequent displacement while incurring less forgetting, indicating a more favorable stability–plasticity balance.Source: §5 · p. 10

AI USE STATEMENT

The paper discloses how generative AI assisted with the work and how authors reviewed that assistance. It also states the study's ethical scope and identifies materials provided for reproduction.

AI USE STATEMENT

  • Generative AI assisted with mathematical, methodological, experimental, interpretive, translation, and manuscript tasks, but did not generate or modify experimental data.Authors report reviewing and checking AI-assisted work and taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses established public benchmarks and involves no human subjects, personally identifiable information, or private-data collection.The statement notes that model behavior may inherit biases or limitations from the datasets and pretrained models considered.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper specifies its methods and reports that the appendix documents proofs, experimental configurations, and diagnostic settings; code and configurations will be released.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 NOTATION

A.2 EVALUATION QUANTITIES.

The appendix defines evaluation quantities and documents the proof, Bayesian interpretation, Fisher computation and consolidation, practical sensitivity solver, and full learning procedure for FS-EWC.

A.2 EVALUATION QUANTITIES.

  • Final average accuracy, stage-average accuracy, backward transfer, parameter displacement, loss change, and sensitivity tolerance quantify learning outcomes and solution drift.More negative backward transfer indicates greater forgetting; the measured tolerance radius is an empirical grid summary, not a continuous-neighborhood certificate.Source: §A.2 · p. 13

A.3 PROOF OF FINITE-DISPLACEMENT SENSITIVITY

  • The finite-displacement proof bounds loss change using Fisher sensitivity along a displacement path within the prescribed neighborhood.The path lies in the neighborhood when displacement magnitude is at most r.Source: §A.3 · p. 14

A.4 FS-EWC FROM A BAYESIAN PERSPECTIVE

  • In the Bayesian view, FS-EWC selects a distribution-aware center by combining task loss and historical compatibility with controlled loss variation around that center.For fixed distribution shape and radius, the parameter-dependent upper surrogate is the FS-EWC objective.Source: §A.4 · p. 15
  • At task completion, equal-mass late-stage checkpoints define an empirical trajectory distribution whose fitted center recovers the retained anchor; Fisher importance is estimated there.The selected checkpoints are treated as an empirical distribution, not assumed to be exact posterior samples.Source: §A.4 · p. 15

A.5.1 EMPIRICAL FISHER COMPUTATION

  • Practical optimization uses squared batch-averaged gradients as a surrogate for the per-example empirical-Fisher trace, though the quantities are generally not identical.The batch-gradient surrogate omits nonnegative dispersion among per-example gradients and is directly available during stochastic optimization.Source: §A.5.1 · p. 16

A.5.2 FISHER POST-PROCESSING AND CONSOLIDATION

  • Current-task Fisher importance is globally trace-normalized across protected coordinates, then isotropically damped and recursively consolidated.For nondegenerate Fisher statistics, normalization gives trace approximately 1; default damping uses β = 0.2.Source: §A.5.2 · p. 16

A.5.3 PRACTICAL SENSITIVITY SOLVER

  • The practical sensitivity solver uses a finite-difference sensitivity-gradient estimate and a single normalized ascent step to select a point within radius r.The selected point stays within the prescribed neighborhood, and its sensitivity estimate supplies the outer shaping correction.Source: §A.5.3 · p. 17

A.5.4 COMPLETE LEARNING PROCEDURE

  • The complete procedure shapes each task update, averages selected checkpoints into the retained anchor, estimates and normalizes its Fisher importance, then consolidates that importance.Source: §A.5.4 · p. 17

A.5.5 APPROXIMATION ERROR ANALYSIS

The practical sensitivity solver approximates both the sensitivity gradient and the neighborhood maximization. The analysis characterizes the finite-difference error and clarifies that the single-step search need not find the exact maximizer.

A.5.5 APPROXIMATION ERROR ANALYSIS

  • The finite-difference sensitivity estimator has first-order truncation error with respect to probe scale ρ under the stated Hessian Lipschitz condition.Source: §A.5.5 · p. 18
  • The neighborhood point comes from one first-order ascent step rather than an exact solution of the inner maximization.The normalized direction of 2H(θ)g(θ) maximizes the linearized sensitivity objective, while the practical solver uses a finite-difference estimate.Source: §A.5.5 · p. 18
  • The practical solver therefore combines finite-difference gradient estimation and single-step neighborhood search; probe scale ρ and radius r control these respective approximations.Source: §A.5.5 · p. 18

B.1 EXPERIMENTAL CONFIGURATIONS

The experiments cover standard vision, fine-grained and heterogeneous vision, and continual text classification, using benchmark-specific task sequences and training configurations.

B.1 EXPERIMENTAL CONFIGURATIONS

  • Standard vision LoRA experiments cover CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A, with 10 tasks per dataset except DomainNet’s five.They use ViT-B/16 with dataset-specific LoRA ranks; FS-EWC settings include dataset-specific shaping and probe radii and checkpoint intervals.Source: §B.1 · p. 18
  • Full-model fine-tuning is additionally evaluated on CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A with ViT-B/16.FS-EWC runs use dataset-specific shaping radii and retained-checkpoint intervals, damping 0.2, and Fisher estimates from at most 100 current-task batches excluding the classification head.Source: §B.1 · pp. 18–19
  • Fine-grained evaluation covers Aircraft, Cars, CUB-200, Flowers, and the heterogeneous 5Datasets sequence CUB-200 → Aircraft → Cars → Flowers → Pets.The individual fine-grained datasets use 10 tasks except Pets, which has nine; FS-EWC-LoRA uses shaping radius 0.05 and probe scale 0.2 without multi-checkpoint averaging.Source: §B.1 · p. 19, Table 8 · p. 19
  • Continual text classification uses AG News, Amazon Reviews, DBpedia, and Yahoo Answers across three task orders with Llama-3.2-1B and Llama-3.2-3B.LoRA is applied to query and value projections; FS-EWC-LoRA uses λ = 1, shaping radius 0.02, and probe scale 0.2 for both model sizes.Source: §B.1 · p. 19, Table 9 · p. 19

B.2 ADDITIONAL EXPERIMENTAL RESULT

Stage-wise results show that FS-EWCLoRA generally improves sequential-learning performance on fine-grained vision and continual text classification, with some dataset- and order-specific exceptions.

B.2 ADDITIONAL EXPERIMENTAL RESULT

  • FS-EWCLoRA is particularly strong late in learning on Aircraft, Cars, and 5Datasets, and improves over matched EWCLoRA on CUB-200 and Flowers.The gains generally accumulate during sequential learning rather than appearing only at the final stage.Source: §B.2 · p. 20, Figure 6 · p. 20
  • Across most text task orders, FS-EWCLoRA better preserves earlier tasks, especially those exposed to more later updates.Llama-3.2-3B under Order 3 is an exception: weaker final performance on some tasks offsets improved retention on others, yielding slightly lower FAA than EWCLoRA.Source: §B.2 · p. 20, Figure 7 · p. 20

B.3 ADDITIONAL TWO-TASK SOLUTION-SHAPING DIAGNOSTICS

Two-task ImageNet-R diagnostics show that shaping the inherited Task 1 solution improves its retained accuracy under the same current-task fitting and protection objective.

B.3 ADDITIONAL TWO-TASK SOLUTION-SHAPING DIAGNOSTICS

  • Shaping the inherited Task 1 solution raises retained accuracy from 87.95 to 91.82 under LoRA and from 85.53 to 87.89 under full-parameter fine-tuning.These comparisons use the same C + P objective on Task 2.Source: §B.3 · p. 21, Table 10 · p. 21
  • The complete FS-EWC configuration achieves higher final Task 1 accuracy than EWC in both parameterizations, improves Task 2 accuracy under LoRA, and leaves it nearly unchanged under full fine-tuning.All Table 10 variants use endpoint references without trajectory averaging.Source: §B.3 · p. 21, Table 10 · p. 21
  • For LoRA, FS-EWC’s more negative BWT reflects its higher Task 1 starting accuracy rather than lower final retention.The authors interpret this as evidence that retention depends on both the solution passed forward and subsequent protection.Source: §B.3 · p. 21

B.4 COMPLEMENTARITY STUDY: EXPERIMENTAL DETAILS AND SENSITIVITY TO λ

The complementarity study finds FS-EWC gains across Fisher variants and regularization strengths, while retained-state comparisons favor trajectory averaging without implying that denser sampling is better.

B.4 COMPLEMENTARITY STUDY: EXPERIMENTAL DETAILS AND SENSITIVITY TO λ

  • Across all six Fisher-computation and matrix-representation combinations, FS-EWC improves over matched EWC by 6.08–9.07 points in FAA and 3.61–4.39 points in AAA.The multi-seed results cover seeds 0, 1, and 2; Kronecker-factored Fisher generally gives modest gains over diagonal Fisher, particularly for EWC.Source: §B.4 · p. 21, Table 11 · p. 21
  • Under batched empirical Fisher, FS-EWC remains above matched EWC at every numerically valid tested λ and varies less across the evaluated range.The Kronecker-factored EWC run at λ = 2 × 10^6 produced NaN values.Source: §B.4 · p. 22, Table 12 · p. 22

B.5 RETAINED-STATE CONSTRUCTION DETAILS

  • Trajectory averaging achieves the highest AAA for all three seeds, but denser trajectory sampling adds no further improvement.All multi-checkpoint constructions yield less negative BWT than the endpoint reference; endpoint-plus-initial has the least negative BWT but substantially lower FAA and AAA.Source: §B.5 · p. 22, Table 13 · p. 22
  • The retained-state study averages LoRA factors uniformly across selected checkpoints and estimates current-task Fisher after merging.Checkpoint 0 is before the first optimizer update, and checkpoint 50 is the terminal iterate.Source: Table 13 · p. 22
Loading 2610.37145v1…