Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

1. Introduction

• Sequential fine-tuning on a target dataset can sharply reduce performance on the original dataset when new-task gradients overwrite established parameter pathways.The paper frames this degradation as a barrier to lifelong, autonomous systems.Source: §1 · p. 2
• At trillion-parameter scale, parameter-protection schemes face computational bottlenecks, while sparse MoE and GQA routing can behave unpredictably under sequential parameter shifts.The paper identifies these scaling and routing challenges as motivation for studying modern foundation models.Source: §1 · p. 2
• A comprehensive comparison of open-weight internal representational drift and closed-source behavioral semantic drift remains lacking.The paper argues that understanding drift mechanisms matters for designing continuous-learning pipelines that adapt without degrading.Source: §1 · p. 2
• The study compares catastrophic forgetting across 20 advanced models, separating 10 closed-source systems from 10 open-weight architectures.It analyzes closed-source models through semantic output drift and open-weight models through internal mechanistic measures, then introduces LRCP to preserve historical task capabilities.Source: §1 · p. 3
• Open-weight analyses use CKA, activation patching, weight-space drift, and MoE routing stability, while closed-source analyses estimate shifts from output probabilities and semantic distances.The closed-source method is needed because commercial model internals are inaccessible.Source: §1 · p. 3
• LRCP restricts updates to the orthogonal complement of historical-task activation subspaces, targeting susceptible mid-network MLP and MoE routing projections.The paper reports that experiments preserve historical task capabilities across sequential training steps.Source: §1 · p. 3

2.1. Continual Learning and Catastrophic Forgetting

• Regularization, replay, and parameter-isolation methods address forgetting, but scaling them to large MoE models raises memory and compute bottlenecks; historical-data buffers also pose privacy and security challenges.Localized importance methods such as RWalk and MAS still face projection misalignment under non-stationary gradients and divergent objectives.Source: §2.1 · p. 3

2.2. Mechanistic Interpretability and Subspace Projections

• Mechanistic interpretability and low-rank subspace work motivate targeting specific circuits, since attention heads and feed-forward networks have specialized or localized activation structure.The paper shifts from global regularization toward localized subspace protection during continual fine-tuning.Source: §2.2 · pp. 3–4
• Unlike GPM and OGP's network-wide constraints and online SVD recalculations, LRCP uses RADL to target susceptible MLP blocks and early attention projections with a static calibration set.This design bypasses online matrix decompositions.Source: §2.2 · p. 3

3.1. Mechanistic Metrics for Open-Weight Models

• For open-weight models, linear CKA compares baseline and fine-tuned activations at each layer, while Routing Gate Drift measures Jensen-Shannon divergence in MoE routing on task samples.These metrics track representation alignment and routing changes, respectively.Source: §3.1 · p. 4

3.2. Behavioral Semantic Drift for Closed-Source Models

• For closed-source models, Semantic Output Drift uses cosine distance between semantic embeddings of baseline and sequentially adapted output distributions.The embedding vectors represent top-k sampled tokens generated under identical prompt parameters.Source: §3.2 · p. 4
• RADL diagnoses layers and components with severe representational shifts under sequential training so mitigation can target those regions.The procedure takes open-weight model states, a calibration dataset, and CKA and routing-drift thresholds to identify highly drifted blocks.Source: §3.2 · p. 4

4.1. Open-Weight Cohort (10 Models)

• The open-weight cohort spans ten architectures, including DeepSeek-V4-Pro, Llama 4 Maverick, Qwen 3.6-27B, and several sparse MoE models.The listed systems range from dense models to MoE architectures with hundreds of billions of parameters.Source: §4.1 · p. 5

4.2. Closed-Source Cohort (10 Models)

• The closed-source cohort includes ten models, among them Claude Fable 5, GPT-5.5 High, Gemini 3.5 Flash, Grok 4.3 Beta, and GPT-4.5.The cohort covers models from Anthropic, OpenAI, Google, Meta, and xAI.Source: §4.2 · p. 5

4.3. Experimental Sequence

• All models undergo sequential coding, math, and medical-QA tasks, using AdamW, a 2 × 10−5 learning rate, cosine decay, and batch size 128.Coding is the source task; math and medical question answering are the two target tasks.Source: §4.3 · p. 5

5.1. Behavioral Evaluation of the Closed-Source Cohort

• GPT-4.5 loses 45.9% absolute performance on Task A; GPT-5.5 High and Claude Fable 5 show relatively lower degradation.Table 1 reports performance degradation and mean Semantic Output Drift across 10 closed-source models, using initial Task A performance as the reference.Source: §5.1 · p. 5, Table 1 · p. 6

5.2. Empirical Analysis of the Open-Weight Cohort

• Mid-layer CKA similarity falls to 0.412–0.612 across evaluated open-weight architectures, while MoE models show considerable Routing Gate Drift.The measurements cover mid-layer MLP and expert blocks.Source: §5.2 · p. 6, Table 2 · p. 6
• Attention-map KL divergence is high in early-to-mid-layer heads, and tracked attention and MLP weight trajectories show rapid orthogonality shifts during adaptation.Figure 1 maps normalized layer depth against attention-map KL divergence and CKA alignment.Source: §5.2 · p. 6, Figure 1 · p. 7, Figure 2 · p. 7
• Activation patching replaces fine-tuned states with saved baseline activations during inference to assess Task A performance recovery.The recovery evaluation uses 500 samples from a SWE-bench subset.Source: §5.2 · p. 6, Figure 3 · p. 7

6.1. Attention Head Entropic Dispersion

• Down-projection matrices (W_down) have the highest parameter drift rates across architectures, corresponding to low middle-layer CKA similarity.Table 3 reports mean Frobenius normalized drift rates across projection matrices.Source: §6.1 · p. 8, Table 3 · p. 8
• Middle layers are most vulnerable to representation changes, while early attention heads show the largest increase in entropy and more diffuse attention distributions.The authors associate this dispersion with reduced focus on syntax and token associations.Source: §6.1 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 9

6.2. Routing Gate Drift in Mixture-of-Experts

• In sparse MoE models, domain shifts alter expert selection, assigning specialized experts to generic tokens and reducing source-task performance.The routing distributions are described as sensitive to domain shifts and susceptible to learning-rate-induced instability.Source: §6.2 · p. 8
• Freezing router parameters severely limits expert-block plasticity; localized L2 penalties on gate-logit deviations during Task A → Task B transition epochs are proposed instead.The authors say this soft regularization prevents routing collapse while allowing new latent concepts to use underutilized expert blocks.Source: §6.2 · p. 8

7. Mitigation Framework: Low-Rank Circuit Projection

• Unlike uniform GPM or OGP projections, LRCP uses targeted RADL to project updates only on identified critical sub-circuits.The projection operator is calculated by SVD of historical task activations.Source: §7 · p. 9
• LRCP projects target-task gradients into the orthogonal complement of historical activation states to minimize interference with learned representations.SVD is performed once on a static calibration set, reducing the online SVD computation bottleneck to O(1) during standard backpropagation.Source: §7 · p. 9
• LRCP attains 74.8% Task A retention and 75.1% Task B accuracy, with a favorable trade-off against alternative regularizers.The comparison concerns final task accuracy after Task A → Task B adaptation on Llama 4 Scout (109B); Figure 7 maps the retention–accuracy Pareto frontier.Source: §7 · p. 10, Table 4 · p. 10, Figure 7 · p. 10

8.1. Why Do Middle Layers Exhibit Peak Drift?

• Middle layers act as a global semantic routing hub, and their substantial adaptation-related parameter changes help explain their susceptibility to forgetting.The discussion contrasts these layers with early layers that process structural and syntactic relationships and deep layers that handle next-token logits.Source: §8.1 · p. 10

8.2. SVD Rank Sensitivity in LRCP

• Rank ratios above 0.30 restrict parameter updates, reducing target-task accuracy and slowing convergence; lower ratios preserve less original capability but allow faster adaptation.The table concerns LRCP rank-ratio sensitivity on Llama 4 Scout (109B) under sequential fine-tuning.Source: §8.2 · p. 10, Table 5 · p. 11

8.3. API Non-Determinism, Version Drift, and Reproducibility Limitations

• Proprietary APIs undergo version drift and nondeterministic sampling even at zero temperature, constraining long-term tracking despite reproducibility safeguards.The safeguards include frozen, timestamped June 2026 snapshots where accessible, greedy decoding, and five independent sampling runs.Source: §8.3 · pp. 10–11
• Long-context sequences exhibit accelerated representation drift compared with short-context samples.The paper tracks forgetting by input sequence position to assess how context length relates to model stability.Source: Figure 8 · p. 11, §8.3 · p. 11

• Open-weight analyses locate peak representational drift in mid-layer feed-forward networks and expert routing blocks, while early attention heads show systematic entropic drift.Source: §9 · p. 11
• LRCP projects updates into the orthogonal complement of historical-activation subspaces and demonstrates a favorable trade-off between task retention and adaptation efficiency.Source: §9 · p. 11

• The research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors.Source: Funding · p. 12

• Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov is credited with conceptualization, methodology, software, formal analysis, data curation, investigation, original-draft writing, and visualization.Source: CRediT authorship contribution statement · p. 12