PaperlayerView exact revision on arXiv

Source-linked AI summary

Spurious Forgetting in Continual Learning of Language Models

Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma

arXiv:2501.13453v1cs.LG

TL;DR

Continual-learning performance drops raise the question of whether new tasks erase knowledge or disrupt task alignment. In four continual-learning scenarios, freezing bottom layers raised sequential fine-tuning task accuracy from 11% to 44%, versus 22% at best for other techniques.

  • Problem

    The paper asks whether performance declines after new tasks reflect lost knowledge or disrupted task alignment, a distinction raised by apparent forgetting despite extensive prior training.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper uses a controlled synthetic biography dataset and theoretical analysis of orthogonal weight updates, then tests freezing the model’s bottom layers to mitigate spurious forgetting.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    In sequential fine-tuning (SEQ), Freeze raised task accuracy from 11% to 44%, versus 22% at best for other techniques, and improved performance across four continual-learning scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that task alignment, rather than underlying knowledge retention, is central to the observed performance declines in its continual-learning experiments.

    Source: §1 · p. 2, §7 · p. 11

    A significant performance gap remains between Freeze and data replay, leaving spurious forgetting a persistent challenge.

    Source: §5.2 · p. 10

Abstract

from arXiv · show

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

1 INTRODUCTION

The introduction reframes apparent catastrophic forgetting as potentially spurious loss of task alignment rather than knowledge, and links early updates on new tasks to reversible performance declines. It presents freezing the model’s bottom layers as an effective way to mitigate those declines.

Introduction

  • The motivating puzzle is that models can decline after new tasks despite extensive earlier training; in safety alignment, as few as ten identity-shift examples can undermine safety.The introduction also notes abrupt declines in continual instruction tuning after models initially excel at specific tasks.Source: §1 · p. 1
  • Older-task performance returned after training on only ten safety instances or irrelevant tasks, neither from the old dataset, challenging the interpretation that the knowledge was genuinely forgotten.The recovery is presented as evidence that apparent forgetting can be reversed without replaying old-task data.Source: §1 · p. 2
  • The authors define “spurious forgetting” as performance loss that may reflect weakened task alignment—the ability to apply existing knowledge—rather than loss of underlying knowledge.They identify this distinction as a contribution of their work.Source: §1 · p. 2
  • In controlled experiments with a synthesized dataset and randomly initialized model, the first 150 new-task optimization steps could undo prior alignment, especially through bottom-layer changes; orthogonal-update theory supports this account.The authors report that significant gradients in the loss landscape can rapidly reduce previous-task performance during these initial steps.Source: §1 · p. 2
  • Retaining a subset of old-task data for replay facilitates re-alignment and restores previous-task performance, suggesting that avoiding the undo-alignment process can preserve it.This finding indicates that old-task performance can be retrieved rather than permanently lost.Source: §1 · p. 2
  • Freeze raises task accuracy in sequential fine-tuning (SEQ) from 11% to 44%, whereas other techniques peak at 22%, and it works in four real-world continual-learning scenarios.The scenarios are safety alignment, continual instruction tuning, continual knowledge editing, and instance incremental learning.Source: §1 · p. 2

2 MOTIVATION: PRELIMINARY EXPERIMENTS ON SPURIOUS FORGETTING

The preliminary experiments investigate whether sudden performance drops during continual learning indicate lost knowledge, given that extensive prior training seems unlikely to be entirely negated by new tasks.

Motivation

  • The observed sudden performance drops raise questions about knowledge retention because extensive prior training seems implausible to be entirely negated by new tasks.The motivation cites examples of 100K safety instances and 5K Science QA instances.Source: §2 · p. 2

Safety Alignment

  • After 10 epochs on 10 identity-shifting instances, AdvBench safety performance fell from 100% to 0%, then rose to approximately 99% after 10 rejection-response examples and 10 epochs.The experiment reproduces AOA alignment and evaluates safety as 100% minus the jailbreak rate on AdvBench (Zou et al., 2023).Source: §2 · p. 2

Continual Instruction Tuning

  • On TRACE’s 8 diverse tasks, replicated LLaMa-3-8B-Instruct accuracy sometimes fell to zero and rebounded with subsequent training, a pattern not confined to specific datasets or hyperparameters.TRACE includes domain-specific QA, code completion, and mathematical reasoning.Source: §2 · pp. 2–3

3 A CLOSER LOOK AT SPURIOUS FORGETTING

Controlled synthetic experiments show that Task 1 sharply reduces Task 0 performance while Task 0 knowledge remains recoverable; loss, weight, and feature analyses trace the effect to changing task alignments and updates rooted in bottom layers.

3.1 CONTROLLED SETTINGS UNDER SYNTHETIC DATASET

  • The 200,000-person Biography dataset separates attribute pretraining from QA finetuning, enabling Task 0 alignment without new knowledge and Task 1 learning of novel knowledge plus alignment.The controlled dataset prevents overlap in knowledge between pretraining and finetuning tasks, helping isolate task alignment from knowledge acquisition.Source: §3.1 · p. 3

3.2 SPURIOUS FORGETTING FROM PERFORMANCE PERSPECTIVE

  • Recovered Task 0 accuracy remains nearly 100% during Task 1’s first 150 steps and reaches 96% by its end, while Task 0 accuracy falls to approximately 10% after 150 steps then rises to 20%.The recovery results support the authors’ conclusion that the performance decline reflects spurious forgetting rather than actual loss of Task 0 knowledge.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 LOSS LANDSCAPE PERSPECTIVE

  • Task 1 initially lowers its loss as Task 0 loss rises, with opposing Task 0 and Task 1 gradients; after 150 steps, Task 1 alignment and new-knowledge learning proceed together.The initial 150 steps undo Task 0 alignment, and Task 1’s learned alignment does not align with Task 0’s.Source: §3.3 · pp. 4–5

3.4 MODEL WEIGHT PERSPECTIVE

  • Pretraining and Task 0 updates occupy similar weight spaces, while Task 1’s first 150 steps resemble Task 0 updates and later updates differ, especially in bottom layers.The authors identify bottom layers, including input embeddings, as crucial for task alignment; near-orthogonal updates there contribute to alignment differences and spurious forgetting.Source: §3.4 · pp. 5–6

3.5 FEATURE PERSPECTIVE

  • Task 0 and Task 1 alignment-learning stages shift feature principal components, but their combined shifts disappear, while shifts appear to originate in bottom layers and propagate upward.The authors interpret the cancellation as evidence of a shared QA-oriented feature pattern despite different task alignments.Source: §3.5 · p. 6

3.6 SUMMARY

  • The summary identifies differing Task 0 and Task 1 alignments as the root cause of spurious forgetting and says data replay or Freeze enables aligned task alignments.Source: §3.6 · p. 6

4 THEORETICAL ANALYSIS

The analysis attributes spurious forgetting to orthogonal weight updates that shift model features without necessarily erasing knowledge. It argues that freezing bottom layers can mitigate the resulting accumulated output shift.

THEORETICAL ANALYSIS

  • The theory studies stacked linear layers with residual connections, a simplification motivated by orthogonal updates’ relation to layer count rather than specific Transformer components.The paper notes that complete Transformer analysis is challenging and cites single-layer Transformers (Li et al., 2023) and diagonal-weight Transformers as other simplified structures.Source: §4 · p. 7
  • Under small weight norms and bounded perturbations in each weight matrix’s left null-space, the layer shift’s inner product with its principal component is at most O(δ + ϵ∆).The assumptions are a weight norm bound ∥Wl∥≤δ and perturbations satisfying ∥∆Wl∥≤ϵ∆ and Wl⊺∆Wl=0.Source: §4 · p. 7
  • Orthogonal weight updates shift outputs away from their prior column space and nearly orthogonally to feature principal components, so apparent forgetting need not mean knowledge loss.The analysis says this interpretation aligns with the paper’s empirical findings.Source: §4 · p. 7
  • ∥˜XL −XL∥≤Lϵ∆(1 + δ)L−1∥X0∥ bounds the final shift, which grows with depth and motivates freezing bottom layers to reduce contributing layers.The proposed mitigation follows because the shift accumulates from bottom to top, and orthogonality is most prominent in bottom layers.Source: §4 · p. 7

5 SOLUTION TO SPURIOUS FORGETTING

Existing continual-learning methods struggle to prevent spurious forgetting, while freezing bottom layers mitigates undoing Task 0 alignment. Across four real-world scenarios, Freeze improves performance over sequential fine-tuning, with the freezing strategy depending on when forgetting occurs and how much plasticity new tasks require.

5.1 REVISITING EXISTING TECHNIQUES FOR FORGETTING

  • Existing methods failed to achieve satisfactory Task 0 accuracy when Task 1 accuracy exceeded 99%, with EWC weakly identifying critical bottom-layer parameters and Task Vector unable to find a viable SEQ-trajectory solution.Task Vector showed a Task 0–Task 1 trade-off despite extensive hyperparameter tuning.Source: §5.1 · pp. 8–9
  • After filtering LAMOL’s 24,000 generated pseudo-old samples, fewer than 20% remained, implying nearly half were hallucinated without real old data.The filtering removed invalid-format and duplicate samples, as well as samples without an exact match to real old data.Source: §5.1 · pp. 8–9
  • Gradient Projection’s best variant achieved only 13.34% because undo-alignment directions varied across trials, with average cosine similarity of 0.4.The method projected component gradients orthogonally to an estimated undo-alignment direction, but variants did not effectively prevent the undo process.Source: §5.1 · p. 9

5.2 MITIGATING SPURIOUS FORGETTING BY FREEZING BOTTOM LAYERS

  • Freezing bottom-layer components, including input embeddings, targets alignment shifts that originate in lower layers and accumulate upward.Replay data can restore Task 0 alignment, but undo alignment remains during initial training even when up to 20% of old data is retained; Freeze encourages reuse of that alignment, though it still trails replay.Source: §5.2 · pp. 9–10
  • On Biography, Freeze raised SEQ performance from 11% to 44% while updating less than half the parameters.Freezing more layers mitigated Task 0 undo alignment but slowed Task 1 learning and could cause late-stage forgetting; early stopping when Task 1 accuracy exceeded 99% improved performance.Source: §5.2 · p. 9

5.3 APPLICATION ON REAL-WORLD SCENARIOS

  • Freeze significantly improves performance over SEQ across four real-world scenarios, with freezing timing and depth tailored to when forgetting begins and how much plasticity new tasks need.For similar task formats and knowledge, forgetting can begin with the first task and freezing more layers may help; for new formats or knowledge, freezing after the first task with fewer layers preserves plasticity.Source: §5.3 · p. 10, Table 2 · p. 10

6 RELATED WORK

Research on LLM behavior under continual learning remains underexplored, and prior findings differ on their susceptibility to catastrophic forgetting. Other studies’ evidence of anti-forgetting capability aligns with the paper’s observation that core knowledge is more resilient than task alignment.

6 RELATED WORK

  • Prior studies disagree on whether LLMs are susceptible to catastrophic forgetting during continual learning.Some studies report susceptibility (Peng et al. (2024); Ren et al. (2024)), while others find strong anti-forgetting capabilities in sequential fine-tuning (Tao et al. (2023); Zheng et al. (2023b)).Source: §6 · p. 11
  • LLM continual-learning behavior remains underexplored.The paper situates its work in a rapidly developing area with limited prior study.Source: §6 · p. 11
  • The reported anti-forgetting findings align with the observation that LLM core knowledge is more resilient than task alignment.The passage attributes evidence of strong anti-forgetting capability in sequential fine-tuning to Tao et al. (2023) and Zheng et al. (2023b).Source: §6 · p. 11
  • The paper points to Appendix A for further discussion of forgetting mechanisms, memorization dynamics, and parameter-freezing strategy.Source: §6 · p. 11

7 CONCLUSION … A.3 PARAMETER FREEZING IN CONTINUAL LEARNING

The paper frames spurious forgetting as a decline in task alignment rather than necessarily a loss of knowledge, and proposes freezing model parameters to mitigate it. It presents Freeze as applicable to full fine-tuning in practical continual-learning settings.

7 CONCLUSION

  • The conclusion identifies spurious forgetting as a key factor in continual-learning performance and argues that task alignment matters more than mere knowledge retention.The paper reports that its Freeze strategy mitigates spurious forgetting and improves performance across learning scenarios.Source: §7 · p. 11

APPENDIX

  • The appendix contents list sections on memorization dynamics and parameter freezing, as well as biography and QA datasets.These are entries in the appendix contents, not findings reported in the supplied passages.Source: APPENDIX · p. 16

A RELATED WORK

  • The related-work section groups continual-learning research in LLMs into forgetting mechanisms and memorization dynamics, and also discusses parameter freezing.Source: A RELATED WORK · p. 17

A.1 FORGETTING MECHANISMS

  • Prior forgetting studies measured degradation on learned tasks, while later probing work examined representation shifts, layer-wise forgetting, and the relation between prior-knowledge retention and new-task learning.The cited examples include French (1999), Kirkpatrick et al. (2017), Davari et al. (2022), Wu et al. (2021), and Chen et al. (2023).Source: §A.1 · p. 17

A.2 MEMORIZATION DYNAMICS

  • Memorization research reports sensitive-information memorization during pretraining, faster memorization and higher forgetting baselines in larger models, unpredictable sample memorization, and incremental knowledge learning in transformers.The cited work spans textual memorization and changes in model weights.Source: §A.2 · p. 17

A.3 PARAMETER FREEZING IN CONTINUAL LEARNING

  • Prior parameter-freezing approaches train only a small parameter subset or freeze the backbone, but some capture less knowledge than full fine-tuning or are limited to classification tasks.Architecture-based approaches include LoRA, prompts, and adapters; cited work also freezes LLM backbones while training classifiers.Source: §A.3 · p. 17
  • Model expansion can prevent forgetting by freezing old layers and adding new ones, but its resource overhead for every new task makes it impractical for real-world use.This approach is contrasted with the proposed Freeze method.Source: §A.3 · p. 17
  • Freeze applies to full fine-tuning in settings such as alignment and continual instruction tuning, distinguishing it from prior approaches and supporting its use across diverse scenarios.The paper characterizes the method as a practical solution for addressing catastrophic forgetting in real-world continual learning.Source: §A.3 · p. 17

B DATASET CONSTRUCTION … B.2 QA DATASET

The section describes a synthetic Biography dataset and a corresponding QA dataset for pre-training and extracting model knowledge. Profiles combine six attributes in varied biographical text entries, while QA pairs target those attributes directly.

B DATASET CONSTRUCTION

B.1 BIOGRAPHY DATASET

  • The Biography dataset contains 200,000 individuals, each with six attributes, and five biography entries made from six sentences apiece.Source: §B.1 · p. 18, §B.1.3 · p. 19

B.1.1 THE CONSTRUCTION OF NAMES AND ATTRIBUTES

  • Each individual has a three-part name and six attributes, with name components and attributes independently and randomly selected uniformly from corresponding real-world pools; rejection sampling ensures all N individuals have unique full names.The pools contain 800 common first names divided between first- and middle-name pools, and 1,000 last names.Source: §B.1.1 · p. 18

B.1.2 THE CONSTRUCTION OF TEMPLATES

  • Each attribute has 50 templates, constructed with GPT-4o from examples and varied to paraphrase the attribute sentence.Templates are filled with an individual’s name and attribute; construction assumes the individual is male by default to simplify pronoun replacement.Source: §B.1.2 · p. 19

B.1.3 THE CONSTRUCTION OF BIOGRAPHICAL TEXT ENTRIES

  • Each individual has five entries, each containing one sentence for each of six attributes, with sentence order randomized.Source: §B.1.3 · p. 19

B.2 QA DATASET

  • The QA dataset extracts knowledge by asking six questions per individual, one for each attribute, and prompting the model to generate the answer.The prompt includes an answer indicator (“Answer:”) after the question.Source: §B.2 · p. 20

C PRE-TRAINING AND FINE-TUNING … D.2 EXPERIMENTAL DETAILS

The paper separates acquiring biography knowledge, aligning it to QA prompts, and learning from QA about new individuals, then evaluates knowledge through QA performance using three accuracy metrics.

C PRE-TRAINING AND FINE-TUNING

  • Knowledge means connections between an individual and six corresponding attributes; the model is trained from scratch, first on Biography data for the first 100k individuals, then QA for the first 50k, and finally QA for individuals 100k–120k.Source: C PRE-TRAINING AND FINE-TUNING · p. 21

C.1 PRE-TRAINING STAGE

  • Biography pre-training uses the standard language-modeling objective to encode broad knowledge in model parameters (Radford & Narasimhan (2018)).The model is trained from scratch on biography data.Source: §C.1 · p. 21

C.2 FINE-TUNING STAGE

  • QA fine-tuning first aligns existing knowledge using a subset of individuals, then trains on QA for new individuals to bestow new knowledge.Performance on the held-out individuals’ QA data tests whether the model aligned existing knowledge or merely learned from the fine-tuning examples.Source: §C.2 · p. 21

D IMPLEMENTATION DETAILS / D.2 EXPERIMENTAL DETAILS

  • The implementation-details sections cover evaluation and the pre-training and fine-tuning implementations; all experiments use PyTorch.The implementation description is presented in Appendix D.2.Source: D IMPLEMENTATION DETAILS · p. 21, §D.2 · p. 22

D.1 EVALUATION DETAILS / D.1.1 SOFT FIRST-TOKEN ACCURACY

  • QA performance indicates knowledge acquisition, measured with soft first-token, hard first-token, and exact match accuracy.Soft and hard first-token accuracy are also monitored during pre-training; soft accuracy tracks the correct token’s generation probability on the six attributes.Source: §D.1 · p. 22, §D.1.1 · p. 22

D.1.2 HARD FIRST-TOKEN ACCURACY

  • Hard first-token accuracy uses greedy decoding and counts knowledge as acquired when the correct token has the highest generation probability.The paper presents it as a closer reflection of real-world performance than soft first-token accuracy.Source: §D.1.2 · p. 22

D.1.3 EXACT MATCH ACCURACY

  • Exact match accuracy uses greedy decoding and counts knowledge as acquired only when every attribute token is generated correctly.It is more computationally demanding than either first-token metric; the paper uses soft first-token accuracy during Task 0 and Task 1 fine-tuning, then exact match after recovery.Source: §D.1.3 · p. 22

D.2.1 EXPERIMENTAL DETAILS OF PRE-TRAINING STAGE · D.2.2 EXPERIMENTAL DETAILS OF FINE-TUNING STAGE · E FORMAL DEFINITION OF SPURIOUS FORGETTING

The formal definition separates performance degradation from knowledge loss: spurious forgetting occurs when task-A loss rises after task-B fine-tuning even though task-A-equivalent performance can be recovered through a transformation.

D.2.1 EXPERIMENTAL DETAILS OF PRE-TRAINING STAGE

  • Pre-training used AdamW with weight decay 0.1, ϵ = 10−6, a 0.001 initial learning rate, 1000-step linear warmup, and cosine decay to 0.0001.Training lasted 80,000 steps with batch size 96.Source: §D.2.1 · p. 22
  • Pre-training shuffled biographical entries each epoch, concatenated them into 512-token sequences separated by <EOS>, and ran on an NVIDIA A800 80GB GPU.Source: §D.2.1 · p. 22

D.2.2 EXPERIMENTAL DETAILS OF FINE-TUNING STAGE

  • Fine-tuning updated all language-model parameters using AdamW with weight decay 0.01, ϵ = 10−6, and an initial learning rate of 5 × 10−6.Source: §D.2.2 · p. 22
  • Fine-tuning used cosine learning-rate decay from 5 × 10−6 to 4.5 × 10−6 for 62,500 steps, with batch size 48 on NVIDIA RTX 3090 GPUs.Source: §D.2.2 · p. 23

E FORMAL DEFINITION OF SPURIOUS FORGETTING

  • Spurious forgetting occurs when MB’s expected loss on task A rises after fine-tuning on B, yet a transformation to MA′ restores task-A loss to MA’s level.The tasks’ datasets have no knowledge overlap, and the transformed model’s expected loss on task A is equivalent to MA’s.Source: E FORMAL DEFINITION OF SPURIOUS… · p. 23
  • In the controlled Biography setup, recovery trains on half of Task 0 and tests on the other half; Task 1 represents B, with no knowledge overlap between recovery and Task 0.The recovery-training half is DA′, while the test half is DA.Source: E FORMAL DEFINITION OF SPURIOUS… · p. 23

F THEORETICAL RESULTS AND PROOF … G.1.4 EXTENDED SETTING 4: DIFFERENT OPTIMIZERS AND LEARNING RATES

The theoretical section analyzes how small residual weights and orthogonal updates affect network outputs and principal components, and extends the spurious-forgetting analysis across varied task and optimization settings. It also shows that freezing bottom layers mitigates output shift and reduces its theoretical bound.

F THEORETICAL RESULTS AND PROOF

  • For residual products, the deviation obeys the recursive bound ∥Pm+1−I∥≤δ(1+ϵm)+ϵm when each weight matrix has norm at most δ.The lemma derives a bound on the product’s deviation from the identity under small weight norms.Source: F THEORETICAL RESULTS AND PROOF · pp. 23–24
  • The perturbed-product difference bound grows as Lϵ_∆(1+δ)^(L−1), while the unperturbed-product bound is linear in L and depends on δ.The discussion characterizes the product as more sensitive to the weight norm δ than to perturbation size ϵ_∆.Source: F THEORETICAL RESULTS AND PROOF · p. 24, §1 · p. 25

1. Bound for (∆Wm+1)P∆

  • With residual connections and small weight norms, principal-component deviation across L layers is O(Lδ), provided the leading-eigenvalue gap is not too small.The bound follows from covariance perturbation growth of O(Lδ) and the Davis–Kahan theorem.Source: §1 · pp. 26–27
  • When ∆W lies in the null space of W⊺, the output shift ∆Y is orthogonal to the column space of Y, including its principal component.This follows from W⊺∆W = 0.Source: §1 · p. 27
  • Under the residual-network assumptions, each layer’s update-induced shift is nearly orthogonal to that layer’s leading principal component.The shift arises from applying ∆Wl to the previous layer’s output, while the orthogonality condition places ∆Wl in Wl’s left null space.Source: §1 · pp. 27–28
  • The final output shift is orthogonal to the principal components and bounded in magnitude by Lϵ_∆(1+δ)^(L−1)∥X0∥.The shift accumulates contributions from perturbations at intermediate layers.Source: §1 · p. 29
  • Under Assumption F.4, freezing L_freeze bottom layers, where L_freeze ≤ L_bottom, mitigates the accumulated shift in the final output.The orthogonality condition is assumed only for updates in the bottom L_bottom layers.Source: §1 · p. 29
  • Freezing bottom layers reduces the bound on output shift by removing their orthogonal-update contribution, leaving shifts from layers above the frozen layers.The proof notes that orthogonality is most prominent in bottom layers in real-world settings, motivating bottom-layer freezing to mitigate cumulative shift.Source: §1 · p. 30

G.1 SPURIOUS FORGETTING UNDER PERFORMANCE PERSPECTIVE

  • The extended experiments test spurious forgetting across task counts, individuals, task types, and optimizer and learning-rate choices.The appendix introduces these experiments after reporting that Task 0 performance decline can be recovered using half of Task 0’s data.Source: §G.1 · p. 31

G.1.1 EXTENDED SETTING 1: MORE TASKS

  • Spurious forgetting persists across subsequent tasks as the number of tasks increases.The experiment fine-tuned sequentially on five tasks; Tasks 4 and 5 used 80K training steps, more than the earlier tasks, because the same number of steps was inadequate as task count increased.Source: §G.1.1 · p. 31, Figure 8 · p. 32

G.1.2 EXTENDED SETTING 2: VARYING NUMBERS OF INDIVIDUALS

  • With 20 individuals per task, spurious forgetting is not pronounced, but it becomes more pronounced as task size increases and forgetting emerges.The experiments compare 20, 200, and 2,000 individuals per task; smaller tasks require fewer new-knowledge updates, helping the model retain prior-task knowledge.Source: §G.1.2 · p. 33, Figure 9 · p. 34, Figure 10 · p. 35

G.1.3 EXTENDED SETTING 3: DIFFERENT TASK TYPES

  • Spurious forgetting still occurs when the new task uses compound question-answer pairs, differing from the prior task type.The model was pretrained on 100,000 individuals, fine-tuned on the original QA task, then fine-tuned on compound QA pairs for 20,000 unfamiliar individuals.Source: Figure 11 · p. 36, §G.1.3 · p. 36

G.1.4 EXTENDED SETTING 4: DIFFERENT OPTIMIZERS AND LEARNING RATES

  • SGD converges much more slowly than AdamW in these experiments; at learning rates of 1 × 10−5 or lower, SGD’s Task 0 training loss does not decrease after 10,000 steps.AdamW still facilitates training at 5 × 10−7.Source: §G.1.4 · p. 36
  • Spurious forgetting occurs at different training steps across optimizer and learning-rate settings, showing it is not caused by initial finetuning instability.With SGD, it appears after approximately 3,000 steps at 1 × 10−4 and 6,000 steps at 5 × 10−5; with AdamW, after 200 steps at 5 × 10−6, 600 at 1 × 10−6, and 1,000 at 5 × 10−7.Source: §G.1.4 · p. 36, Figure 12 · p. 37

G.2 SPURIOUS FORGETTING UNDER LOSS LANDSCAPE PERSPECTIVE … G.4.4 CASE 4: STEP 0 TO STEP 62500 IN TASK 1

The appendices examine spurious forgetting through loss landscapes, weight-update spaces, and feature shifts across training stages. They report early disruption of Task 0 alignment, distinct updates in bottom layers, and feature comparisons across four training intervals.

G.2 SPURIOUS FORGETTING UNDER LOSS LANDSCAPE PERSPECTIVE

  • During replay, the trajectory first reverses Task 0 alignment over 150 steps, then advances along Task 1 knowledge while replay encourages re-alignment, and finally seeks a shared alignment direction.The trajectory is interpreted as balancing Task 1 and Task 0 alignment as training proceeds.Source: §G.2 · p. 38
  • More replay data makes both task loss landscapes flatter, while sequential fine-tuning fails to find effective solutions and has an optimal average loss around 0.8.The flattening is interpreted as identifying common alignment directions or improved Task 1 solutions near Task 0 flat minima.Source: §G.2 · p. 38, Figure 13 · p. 38

G.3 SPURIOUS FORGETTING UNDER MODEL WEIGHT PERSPECTIVE

  • During Task 1’s first 150 steps, updates in all bottom-layer components differ from later updates; this layer-position pattern motivates freezing all bottom-layer components rather than selected components.For Task 0 and Task 1, nearly all components update in the same spaces as pretraining except input embeddings.Source: §G.3 · p. 39, Figure 14 · p. 40

G.4 SPURIOUS FORGETTING UNDER FEATURE PERSPECTIVE

  • The feature analysis centers layer features and projects them into two dimensions, using mean feature difference on the x-axis and an earlier-stage principal component on the y-axis.It compares model checkpoints from the Section 3.2 experiment and connects corresponding features for the same attribute and individual.Source: §G.4 · p. 41

G.4.1 CASE 1: STEP 0 TO STEP 62500 IN TASK 0

  • Case 1 compares features from the same pretraining individuals before and after Task 1 fine-tuning, with circular and cross markers distinguishing the models.The passage identifies the comparison as features during Task 0 fine-tuning and points to Figure 15.Source: §G.4.1 · p. 41, Figure 15 · p. 41

G.4.2 CASE 2: STEP 100 TO STEP 150 IN TASK 1

  • Case 2 compares features from pretraining individuals after 100 versus 150 Task 1 fine-tuning steps, marking the checkpoints with circles and crosses.Source: §G.4.2 · p. 42, Figure 16 · p. 42

G.4.3 CASE 3: STEP 200 TO STEP 62500 IN TASK 1

G.4.4 CASE 4: STEP 0 TO STEP 62500 IN TASK 1

H REVISITING CONTINUAL LEARNING METHODS … I.5.1 EXPERIMENTAL SETTINGS

This section describes experimental settings and evaluations for continual-learning methods across safety alignment and task-wise learning. It covers real-world evaluations, the TRACE benchmark, AOA fine-tuning for safety-alignment removal and recovery, and setups for knowledge editing, instance-incremental learning, and supervised fine-tuning.

H REVISITING CONTINUAL LEARNING METHODS

  • The compared continual-learning methods include sequential fine-tuning as a lower-bound baseline, replay with 20% or 50% old data, and EWC and LAMOL variants with tuned hyperparameters.For replay, each new-task batch contains equal amounts of new and replay-buffer data; the tested LAMOL variants differed in use of a task-specific generation token and had less than 1% difference in final results.Source: H REVISITING CONTINUAL LEARNING METHODS · p. 47
  • Task Vector searches checkpoint endpoints and scaling factors to counteract alignment undoing, while gradient projection tests null-space projections of recorded first-stage updates but does not improve Task 0 performance effectively.The best gradient-projection variant, on Attention Layers, reached 13.34% on Task 0; projecting across All Layers yielded 9.52%.Source: H REVISITING CONTINUAL LEARNING METHODS · p. 47

I.1.1 REMOVING SAFETY ALIGNMENT

  • After 10 epochs on 10 Identity Shifting Data examples, LLaMa-2-7B-Chat’s jailbreak rate rose from 0% to 99.80%, after only 10 training steps.The model was fine-tuned on safety data before AOA alignment; the examples were manually curated to prioritize obedience and contained no explicitly toxic content.Source: §I.1.1 · p. 48

I.1.2 RECOVERING SAFETY ALIGNMENT

  • Fine-tuning on 10 safe instances for 10 epochs reduced the jailbreak rate from 99.80% to 0.77%.The safe instances were generated by the safety-aligned LLaMa-2-7B-Chat in response to 10 harmful instructions, and training used the AOA alignment batch size and learning rate.Source: §I.1.2 · p. 48

I.1.3 APPLYING FREEZE TO SAFETY ALIGNMENT

  • Freezing the bottom 3 or 6 LLaMa-2-7B-Chat layers reduced jailbreak rates to 79.61% and 1.15%, respectively, versus 99.80% without freezing.The authors interpret this as Freeze mitigating spurious forgetting of safety knowledge during fine-tuning on Identity Shifting Data.Source: §I.1.3 · p. 50

I.1.4 CASE STUDY ON SAFETY ALIGNMENT

  • The safety-alignment case study presents five harmful-instruction responses before AOA, after AOA, after recovery on safe instances, and with Freeze during AOA.The examples show refusals before AOA, compliance after AOA, and refusals in the recovery and Freeze conditions.Source: §I.1.4 · pp. 50–51

I.2.1 CONTINUAL INSTRUCTION TUNING ON TRACE

  • TRACE evaluates continual learning across varied tasks, including unseen-target stance detection in C-STANCE and text simplification in 20Minuten.20Minuten uses SARI to assess simplification quality; C-STANCE tests generalization to unseen targets.Source: §I.2.1 · pp. 52–53

I.2.2 IMPLEMENTATION DETAILS

  • TRACE experiments use LLaMa-3-8B-Instruct, a constant learning rate of 1 × 10−5, task-specific epoch counts, batch size 32, and maximum sequence length 2048.Tasks are ordered C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cm, NumGLUE-ds, then 20Minuten.Source: §I.2.2 · p. 53

I.2.3 TASK-WISE RESULTS ON CONTINUAL INSTRUCTION TUNING

  • SEQ shows task-wise spurious forgetting on TRACE: FOMC’s Finance QA score falls to zero after Science QA, then rebounds to 62% after the next task.Similar trends occur on C-STANCE and NumGLUE-cm, consistent with Wang et al. (2023b).Source: §I.2.3 · p. 53, Table 4 · p. 53

I.3.1 CONTINUAL KNOWLEDGE EDITING ON ZSRE

  • For continual knowledge editing, zsRE has 10,000 factual, paraphrased, and unrelated records; it is split into 10 tasks of 1,000, with Efficacy and Paraphrase reported.The evaluation also defines Specificity to measure accuracy on unrelated facts.Source: §I.3.1 · p. 55

I.3.2 IMPLEMENTATION DETAILS

  • Knowledge-editing experiments use LLaMa-3-8B-Instruct, training each task for 20 epochs at learning rate 1 × 10−5, with batch size 32 and maximum sequence length 32.The setup is intended to support learning while mitigating overfitting.Source: §I.3.2 · p. 55

I.3.3 TASK-WISE RESULTS ON CONTINUAL KNOWLEDGE EDITING

  • Freeze outperforms LAMOL and EWC by a large margin on CKE; task-wise CKE results cover SEQ and five Freeze configurations.The comparison is shown in Table 10, and task-wise results are organized in Tables 11–22.Source: §I.3.3 · p. 55, Table 10 · p. 56

I.4.1 INSTANCE INCREMENTAL LEARNING ON CONCEPT-1K

  • Concept-1K contains 16,653 instances across 1,023 concepts and uses instance-level incremental learning, evaluating memorization and generalization accuracy by exact match.Concepts are represented as triplets, with training and test question pairs for concept-specific knowledge.Source: §I.4.1 · p. 60

I.4.2 IMPLEMENTATION DETAILS

  • Instance-incremental experiments use Pythia-410M, with each task trained for 20 epochs at learning rate 1e-5, batch size 64, and maximum sequence length 32.Source: §I.4.2 · p. 60

I.4.3 TASK-WISE RESULTS ON INSTANCE INCREMENTAL LEARNING

I.5.1 EXPERIMENTAL SETTINGS

  • SFT experiments finetune LLaMa-3-8B-Instruct 9, Qwen2.5-7B-Instruct 10, and Mistral-8B-Instruct-2410 11 on math or code data, using one epoch to limit general-ability forgetting.MetaMathQA is evaluated on GSM8K and Math, while Magicoder-Evol-Instruct-110K is evaluated on HumanEval; general ability is assessed using MMLU, BBH, TriviaQA, BoolQ, and HellaSwag.Source: §I.5.1 · p. 64

I.5.2 EXPERIMENTAL RESULTS · J LIMITATIONS, SOCIAL IMPACT, AND REPRODUCIBILITY STATEMENT

The results show that Freeze helps preserve general ability during MetaMathQA finetuning, while Magicoder-Evol-Instruct-110K improves it even without Freeze. Across the reported models, Freeze supports adaptation while limiting forgetting or further improving general ability.

I.5.2 EXPERIMENTAL RESULTS

  • On MetaMathQA, LLaMa-3-8B-Instruct’s average general ability fell from 66.89 to 64.15 at 5 × 10−6; Freeze raised it to 66.11 while math ability remained comparable.Table 35 reports one epoch of finetuning and defines Freeze as freezing the bottom one layer.Source: §I.5.2 · p. 64, Table 35 · p. 65
  • At a learning rate of 2 × 10−5, SFT on MetaMathQA caused significant catastrophic forgetting of general ability.The reported result concerns supervised finetuning on MetaMathQA.Source: §I.5.2 · p. 64, Table 35 · p. 65
  • On MetaMathQA, Freeze also reduced general-ability forgetting while maintaining strong math-dataset adaptation on Qwen2.5-7B-Instruct and Mistral-8B-Instruct-2410.The passage describes similar trends for both models.Source: §I.5.2 · p. 64, Table 35 · p. 65
  • On Magicoder-Evol-Instruct-110K, LLaMa-3-8B-Instruct’s average general ability rose from 68.89 to 71.47 with SFT at 5 × 10−6, then to 71.78 with Freeze.Table 36 reports one epoch of finetuning and defines Freeze as freezing the bottom one layer.Source: §I.5.2 · p. 64, Table 36 · p. 65
  • Magicoder-Evol-Instruct-110K finetuning did not cause catastrophic forgetting of general ability; the code dataset instead enhanced it.This contrasts with the reported MetaMathQA result, where SFT at 2 × 10−5 caused significant catastrophic forgetting.Source: §I.5.2 · p. 64
  • On Magicoder-Evol-Instruct-110K, Freeze consistently improved SFT performance across Qwen2.5-7B-Instruct and Mistral-8B-Instruct-2410 while minimizing forgetting during target-dataset adaptation.The passage reports consistent improvements across both models.Source: §I.5.2 · p. 64, Table 36 · p. 65

J.1 LIMITATIONS · J.2 SOCIAL IMPACT · J.3 REPRODUCIBILITY STATEMENT

Freeze improves SEQ but remains limited by replay’s accuracy advantage and a stability–plasticity trade-off. The paper also notes social risks and commits to publicly releasing code, scripts, and the synthetic Biography dataset.

J.1 LIMITATIONS

  • Freeze significantly improves SEQ performance, but it still falls short of data replay methods.Replay strategies can retain a larger portion of old data.Source: §J.1 · p. 66
  • Although Freeze mitigates spurious forgetting, it does not match the performance enhancements achieved through replay strategies.The passage attributes replay’s advantage to retaining more old data.Source: §J.1 · p. 66
  • Freezing too many layers may hinder adaptation to new tasks, leaving a stability–plasticity trade-off unresolved.Source: §J.1 · p. 66

J.2 SOCIAL IMPACT

  • Continual learning techniques such as Freeze could help AI systems learn and adapt over time across various sectors.Source: §J.2 · p. 66
  • Deployment raises concerns about data use, privacy, and algorithmic bias, requiring transparency and fairness to build trust and mitigate unintended consequences.Source: §J.2 · p. 66

J.3 REPRODUCIBILITY STATEMENT

  • The authors will publicly release all code, scripts, and the synthetic Biography dataset to support validation and further research.They state this is intended to foster collaboration and further advances in the field.Source: §J.3 · p. 66

Related papers

Loading 2501.13453v1…