PaperlayerView exact revision on arXiv

Source-linked AI summary

Spurious Forgetting in Continual Learning of Language Models

Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma

arXiv:2501.13453v1cs.LG

TL;DR

The paper addresses whether performance declines during continual learning reflect erased knowledge or disrupted task alignment. It combines controlled experiments and theoretical analysis, then proposes Freezing bottom layers, which improves performance across four continual learning scenarios. The results support distinguishing task alignment from knowledge retention.

  • Problem

    LLMs can suffer major performance declines on earlier tasks after learning new ones, raising whether continual-learning failures reflect knowledge loss or reduced task alignment.

  • Method

    The paper uses controlled synthetic-data experiments, weight and gradient analysis, theoretical analysis of orthogonal updates, and a strategy that freezes bottom layers.

  • Results

    Freezing bottom layers improves sequential fine-tuning task accuracy from 11% to 44%, while other techniques peak at 22%, and works across four continual learning scenarios.

  • Takeaways & Limitations

    Task alignment can be more critical than knowledge retention for understanding continual-learning performance declines in language models.

  • Takeaways & Limitations

    In safety alignment, the results are not intended to establish Freeze as a defense against jailbreak attacks.

Abstract

from arXiv · show

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

1 INTRODUCTION

The paper argues that apparent catastrophic forgetting in continual learning can reflect disrupted task alignment rather than erased knowledge. Controlled experiments and analysis motivate Freezing bottom layers as an effective mitigation.

  • Problem: Spurious forgetting describes performance loss that reflects declining task alignment rather than loss of underlying knowledge.Task alignment is the model’s ability to apply existing knowledge to specific tasks.
  • Method: Controlled experiments use a synthesized dataset and randomly initialized language model to distinguish new knowledge from old knowledge.This setup is intended to clarify whether observed performance declines represent genuine knowledge loss.
  • Findings: During the first 150 optimization steps on new tasks, significant gradients can rapidly reduce previous-task performance by undoing prior task alignment.The analysis identifies bottom layers as particularly important and links the behavior to orthogonal weight updates.
  • Findings: Data replay can restore previous-task performance by facilitating re-alignment and avoiding the undo alignment process.The result suggests old-task performance remains retrievable after apparent forgetting.
  • Contribution: Freezing bottom layers improves sequential fine-tuning task accuracy from 11% to 44%, while other techniques peak at 22%.The strategy is also reported effective across safety alignment, continual instruction tuning, knowledge editing, and instance incremental learning.
  • Contribution: The paper identifies spurious forgetting, attributes it to lost task alignment, analyzes it theoretically, and proposes Freeze as a mitigation.These contributions frame task alignment as distinct from underlying knowledge retention.

2 MOTIVATION: PRELIMINARY EXPERIMENTS ON SPURIOUS FORGETTING

Preliminary experiments show that continual learning can cause abrupt declines on earlier tasks, including safety and instruction-tuning settings. Performance can later rebound with subsequent training, motivating spurious forgetting as an alternative to simple knowledge loss.

  • Motivation: The experiments examine whether sudden performance drops after new-task training indicate genuine knowledge loss.The motivation is that extensive prior training appears unlikely to be entirely negated by introducing new tasks.
  • Safety Alignment: Safety alignment is evaluated after training LLaMa-2-7B-Chat on 10 identity-shifting instances, using safety performance defined as 100% minus jailbreak rate.The initial safety performance is reported as 100%.
  • Continual Instruction Tuning: TRACE covers eight diverse continual instruction-tuning tasks, including domain-specific question answering, code completion, and mathematical reasoning.The replication uses LLaMa-3-8B-Instruct with the same task order and settings.
  • Continual Instruction Tuning: Task accuracy can drop significantly—occasionally to zero—then rebound with subsequent training, across datasets and training hyperparameters.The reported pattern is not confined to a particular dataset or training configuration.

3 A CLOSER LOOK AT SPURIOUS FORGETTING

Controlled experiments on the synthetic Biography dataset show that Task 1 training rapidly reduces Task 0 performance without eliminating recoverable knowledge. The decline is concentrated in early optimization steps, where contradictory task alignments and weight-update directions disrupt prior alignment.

  • 3.1 CONTROLLED SETTINGS UNDER SYNTHETIC DATASET: The Biography dataset uses synthetic individuals with six attributes and separate pretraining statements and QA finetuning pairs.The controlled setup keeps task knowledge non-overlapping to isolate spurious forgetting.
  • 3.2 SPURIOUS FORGETTING FROM PERFORMANCE PERSPECTIVE: Task 0 accuracy drops from nearly 100% to around 10% within the first 150 Task 1 optimization steps.The authors describe this rapid decline as implausible evidence of complete knowledge disappearance.
  • 3.2 SPURIOUS FORGETTING FROM PERFORMANCE PERSPECTIVE: Recovered Task 0 accuracy remains nearly 100% during the first 150 Task 1 steps and reaches 96% by the end, despite Task 0 accuracy falling to approximately 10%.Recovery uses non-overlapping Task 0 training and test subsets, supporting retained knowledge with degraded task alignment.
  • 3.3 LOSS LANDSCAPE PERSPECTIVE: The initial 150 steps sharply decrease Task 1 loss while increasing Task 0 loss, and the corresponding Task 0 and Task 1 optimization directions are contradictory.After 150 steps, training shifts toward a local minimum for Task 1 and enters a second trajectory stage.
  • 3.4 MODEL WEIGHT PERSPECTIVE: The first 150 Task 1 steps update weights near the Task 0 update space, whereas later steps use a distinct space that especially affects bottom layers and input embeddings.These patterns indicate that early updates undo Task 0 alignment and that bottom layers are crucial for task alignment.
  • 3.5 FEATURE PERSPECTIVE: Task-alignment changes produce principal-component shifts, but combining early and later Task 1 effects can cancel the shift because both tasks align the model toward QA tasks.The authors connect the difference between Task 0 and Task 1 alignments to spurious forgetting and motivate data replay or Freeze.

4 THEORETICAL ANALYSIS

The theoretical analysis models continual-learning updates with residual linear layers and orthogonal weight perturbations. It explains spurious forgetting as feature and output shifts that can accumulate across layers while remaining orthogonal to principal components, motivating bottom-layer freezing.

  • Model and assumptions: The analysis uses stacked linear layers with residual connections because orthogonal updates are related to layer count rather than specific Transformer components.Each layer has the form X_l = (W_l + I)X_{l−1}.
  • Model and assumptions: The assumptions bound each weight norm by δ and require perturbations of size at most ϵ_∆ that lie in the left null-space of the corresponding weight matrix.These assumptions formalize small, orthogonal weight updates.
  • Orthogonal shifts: A weight perturbation in the null-space of W^⊺ shifts the output orthogonally to every vector in the column space of the original output.This proposition provides the local mechanism connecting orthogonal updates to feature shifts.
  • Orthogonal shifts: The shift in each layer’s output has principal-component projection bounded by O(δ + ϵ_∆), so the shift can be nearly orthogonal to the leading feature direction.The result formalizes the near-orthogonality observed empirically.
  • Accumulation and intervention: After L layers, the final-output shift is bounded by Lϵ_∆(1 + δ)^(L−1)||X_0||, making accumulated shifts sensitive to depth.Because orthogonality is most prominent in bottom layers, the analysis suggests freezing them to mitigate the shift.
  • Accumulation and intervention: Figure 7 compares Freeze, which freezes the bottom n layers, with existing continual-learning techniques whose shortcomings are visualized on the Biography dataset.The figure frames bottom-layer freezing as the proposed intervention for the analyzed mechanism.

5 SOLUTION TO SPURIOUS FORGETTING

Existing continual-learning methods struggle to preserve Task 0 alignment while learning Task 1. Freezing bottom layers substantially improves performance, although the number and timing of frozen layers create a stability–plasticity trade-off.

  • None of the existing methods achieve satisfactory Task 0 accuracy when Task 1 accuracy exceeds 99%.
  • Existing techniques: EWC inadequately identifies bottom layers as critical parameters because Fisher-matrix correlations with update angles are weak across model components.
  • Existing techniques: Fewer than 20% of LAMOL’s pseudo-old samples remain after filtering invalid, duplicated, and unmatched examples.
  • Existing techniques: Task Vector exposes a trade-off between Task 0 and Task 1 accuracies, with the best average accuracy reaching (95.76, 30.75).
  • Freezing bottom layers: Freezing bottom layers improves sequential fine-tuning performance from 11% to 44% while updating less than half the parameters.
  • Freezing bottom layers: Freezing more layers mitigates Task 0 undo alignment but slows Task 1 learning, reduces capacity, and can cause later forgetting.
  • Limitations: A performance gap remains between Freeze and data replay, indicating persistent challenges in mitigating spurious forgetting.
  • Real-world scenarios: Across four real-world scenarios, freezing after the first task with fewer layers suits tasks introducing different formats or knowledge, whereas similar tasks may benefit from freezing more layers earlier.

6 RELATED WORK

Related work studies forgetting mechanisms, memorization dynamics, and parameter-freezing strategies in continual learning. Prior freezing approaches often restrict training or task types, while model expansion adds resource costs.

  • Continual-learning research in LLMs examines forgetting mechanisms, memorization dynamics, and parameter-freezing strategies.
  • Prior findings disagree about whether LLMs are susceptible to catastrophic forgetting, with some studies reporting degradation and others reporting robustness.

7 CONCLUSION

The paper identifies spurious forgetting as a major factor in continual-learning performance and argues that task alignment matters more than knowledge retention. It introduces Freeze as a mitigation strategy evaluated across diverse settings.

  • The paper identifies spurious forgetting as a pivotal factor affecting language-model performance during continual learning.
  • Controlled experiments and theoretical analyses support the importance of task alignment relative to mere knowledge retention.
  • Freeze mitigates spurious forgetting and enhances performance across various learning scenarios.
  • The paper evaluates continual learning across safety alignment, continual instruction tuning, continual knowledge editing, and instance incremental learning.
  • Related work: Prior parameter-freezing methods include training LoRA modules, prompts, adapters, or classifiers instead of all model parameters.

B DATASET CONSTRUCTION

The paper constructs synthetic Biography and QA datasets from randomly generated individual profiles, then uses them to pre-train and query a language model under controlled conditions. Templates diversify six-attribute biographies while preserving structured attribute values.

  • The model is pre-trained on a synthetic Biography dataset and then fine-tuned on a corresponding QA dataset.
  • Dataset construction: The Biography dataset contains 200,000 synthetic individuals, each defined by a name and six attributes.
  • Dataset construction: The six attributes are birthday, birth city, university, major, company name, and company city.
  • Dataset construction: Names and attributes are sampled independently from pools, with rejection sampling ensuring unique full names.
  • Dataset assumptions: Because attributes are sampled independently, some combinations may be unrealistic, but the randomly initialized model is trained on these synthetic associations.
  • Biography templates: Each biography entry contains six attribute-specific sentences selected from template pools, with five entries generated per individual and sentence order randomized.
  • QA dataset: The QA dataset extracts model knowledge by asking six questions per individual, each targeting one unique attribute.

C PRE-TRAINING AND FINE-TUNING

The experiments separate pre-training, which provides knowledge, from fine-tuning, which aligns that knowledge for answering questions or adds new knowledge. Performance is evaluated with token-level and exact-match measures.

  • Training stages: Pre-training on Biography data gives the model knowledge, while later stages either align that knowledge for QA or add knowledge about new individuals.The second stage uses overlapping individuals and therefore introduces no new knowledge; the third uses unfamiliar individuals.
  • Training stages: The second fine-tuning stage uses QA pairs for only a subset of pre-trained individuals, leaving held-out individuals to test alignment.Good performance on the remaining individuals’ QA data indicates successful alignment.
  • Evaluation: Knowledge is assessed through soft first-token accuracy, hard first-token accuracy, and exact-match accuracy on QA data.The paper uses soft first-token accuracy during fine-tuning and exact match after recovery.
  • Evaluation: The evaluation implementation uses greedy decoding for hard first-token and exact-match accuracy, with exact match requiring every attribute token to be correct.Hard first-token accuracy requires the correct token to have the highest probability.

D.2.1 EXPERIMENTAL DETAILS OF PRE-TRAINING STAGE

The section specifies the continual-learning setup and formalizes performance degradation, knowledge retention, and spurious forgetting. The definitions distinguish a loss in task performance from recoverability of the underlying knowledge.

  • Formal setup: The setup trains a model from scratch on task A, fine-tunes it on non-overlapping task B, and may further fine-tune it on a third non-overlapping dataset A′.The recovery dataset shares no knowledge with either A or B.
  • Definitions: Performance degradation occurs when fine-tuning on task B significantly increases the expected loss on task A.This definition concerns the observable evaluation performance on task A.
  • Definitions: Knowledge retention occurs when a function applied to the model after task B restores task-A loss to the level achieved by the model trained on task A.The retained knowledge is operationalized through recoverable expected loss on the task-A evaluation.
  • Definitions: Spurious forgetting combines performance degradation with retained knowledge, so a task-performance drop need not imply that the underlying knowledge was lost.The recovery experiment uses half of Task 0 for recovery and the other half for testing, preventing direct relearning from the test half.

F THEORETICAL RESULTS AND PROOF

The theoretical analysis bounds how small weight perturbations accumulate through residual networks and explains why orthogonal updates can shift outputs without directly altering their principal directions. It also links these bounds to the benefit of freezing bottom layers.

  • Product bounds: For layer matrices with norm at most δ, the product’s deviation from the identity is controlled recursively by ϵ_m+1 = δ(1 + ϵ_m) + ϵ_m.The resulting bound tracks how deviations accumulate across layers.
  • Product bounds: With small perturbations bounded by ϵ_∆, the perturbed product differs from the unperturbed product by a bound that grows with depth and perturbation size.The discussion gives growth proportional to Lϵ_∆(1 + δ)^(L−1).
  • Orthogonal shifts: A weight update in the null-space of W^⊺ shifts the output orthogonally to the column space of the original output.The result applies to Y = WX with ∆Y = ∆WX.
  • Residual networks: Across residual layers, principal components remain stable within O(Lδ), while cumulative output shifts are bounded by Lϵ_∆(1+δ)^(L−1)∥X_0∥.The analysis treats the final shift as the cumulative effect of intermediate-layer shifts.
  • Residual networks: Freezing bottom layers reduces the bound on the accumulated shift.This theoretical result motivates restricting updates in lower layers.
  • Empirical implications: Spurious forgetting becomes more pronounced as task size increases and can arise across different task types and optimizer settings.The reported optimizer examples place the phenomenon at different training steps for SGD and AdamW.

G.2 SPURIOUS FORGETTING UNDER LOSS LANDSCAPE PERSPECTIVE

The loss-landscape and feature analyses characterize early Task 1 training as initially undoing Task 0 alignment before moving toward Task 1 knowledge and a shared alignment. Replay flattens both task landscapes, while weight-update comparisons identify bottom layers as the distinctive early-update region.

  • Loss landscape: During the first 150 steps, the trajectory moves opposite to Task 0 alignment, corresponding to the undoing of that alignment.This early phase is identified in the data-replay trajectory analysis.
  • Loss landscape: Later, the trajectory shifts toward Task 1 knowledge while data replay progressively encourages Task 0 realignment.The model searches for a balance between the two task alignments.
  • Loss landscape: Increasing the amount of replay makes both Task 0 and Task 1 loss landscapes flatter.The paper interprets this as finding common alignment directions and Task 1 solutions near Task 0’s flat minima.
  • Loss landscape: Sequential fine-tuning fails to find an effective average-loss solution, with the optimum around 0.8 in the analyzed landscape.The paper connects this landscape behavior to poor Task Vector performance.
  • Weight updates: Weight-update angles are distinctive during the first 150 Task 1 steps in all bottom-layer components, rather than only in particular component types.The comparison motivates freezing all components in bottom layers, including input embeddings.
  • Feature perspective: Feature-shift visualizations center and project layer features into two dimensions, using mean difference and an earlier-stage principal component as axes.The plots compare corresponding features across model checkpoints and label the embedding layer as Layer 0.

H REVISITING CONTINUAL LEARNING METHODS

The paper compares replay, regularization, generative-replay, model-merging, and gradient-based continual-learning methods, finding that freezing bottom layers is especially effective for mitigating spurious forgetting.

  • SEQ provides the baseline for continual-learning performance and establishes the lower bound for comparison.
  • Replay jointly optimizes new data with stored old-task samples, using buffers containing 20% or 50% of old data.Each new-task batch contains half new data and half replay-buffer data.
  • EWC regularizes parameter updates using Fisher-information estimates, but small λ values yield negligible old-task improvements while large values restrict plasticity.
  • Gradient Projection records the first 150 update steps and projects subsequent gradients onto the null space of that direction.
  • On TRACE, SEQ performance on FOMC drops to zero after Science QA but rebounds to 62% after the subsequent task.
  • Freeze outperforms LAMOL and EWC by a large margin, consistent with results on the Biography dataset.

I.5.2 EXPERIMENTAL RESULTS

Experiments on MetaMathQA and Magicoder-Evol-Instruct-110K examine whether freezing the bottom layer preserves general ability during supervised fine-tuning while maintaining target-task adaptation.

  • On LLaMa-3-8B-Instruct, MetaMathQA fine-tuning at 5 × 10−6 reduces average general ability from 66.89 to 64.15, while Freeze raises it to 66.11.The math ability remains comparable to SFT without Freeze.
  • Across Qwen2.5-7B-Instruct and Mistral-8B-Instruct-2410, Freeze reduces general-ability forgetting while maintaining strong adaptation to MetaMathQA.
  • On Magicoder-Evol-Instruct-110K, SFT at 5 × 10−6 improves average general ability from 68.89 to 71.47, and Freeze further raises it to 71.78.
  • Table 35 reports one-epoch MetaMathQA fine-tuning, with Freeze freezing the bottom one layer and bolding the best results.
  • Table 36 reports one-epoch Magicoder-Evol-Instruct-110K fine-tuning, with Freeze freezing the bottom one layer and bolding the best results.

J.1 LIMITATIONS

The paper identifies performance gaps against replay and a stability–plasticity trade-off as limitations of Freeze, while also noting ethical and reproducibility considerations.

  • Freeze improves SEQ but remains less accurate than data replay methods.Replay can retain a larger portion of old data.
  • Freezing too many layers may hinder efficient adaptation to new tasks.
  • The paper identifies data use, privacy, and algorithmic bias as ethical considerations for continual-learning technologies.
  • The authors state that code, scripts, and the synthetic Biography dataset will be publicly available for validation and extension.
Loading 2501.13453v1…