When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting

• In controlled factual recall, finetuning on new facts alone makes old-fact recall collapse and return without replay; changing where new answers lie can instead produce steady forgetting.Source: §1 · p. 2
• A minimal associative memory reproduces this with shared-structure keys, concentrated new answers, and normalization: a common shift hides old facts while preserving their relative geometry.Source: §1 · p. 2
• In trained models, removing the common shift eliminates collapse in synthetic Transformers, and removing one direction per update restores pretrained-model recall at equal new-fact accuracy.Source: §1 · p. 2
• Forgetting is noncatastrophic when updates move old memories together: shared loss of access can reverse, while fact-specific erosion is lasting.Source: §1 · p. 2

• The comparison uses new biographies C and B′, which overlap with both old sets A and B or only with B, respectively, with no replay.Source: §2 · p. 2
• Finetuning on C steadily reduces recall of A and B, whereas B′ barely affects B as A's recall collapses, partially recovers, and later erodes.Source: §2 · pp. 2–3
• The trajectory raises two questions: why B′ hides A while sparing B, and why A's collapse reverses on its own.Source: §2 · p. 3

3.1 REPRODUCING SPURIOUS FORGETTING IN AN ASSOCIATIVE MEMORY MODEL

• The two-layer associative memory reproduces collapse, recovery, and erosion; removing shared key structure, concentrated new values, or normalization breaks this trajectory.Keys combine a shared component with fact-specific components, and the model uses RMS normalization between layers.Source: §3.1 · p. 3, §3 · p. 3, Figure 2 · p. 4

3.2 KEY SHARING LEADS TO COLLAPSE

• Shared key structure writes a common shift into old hidden states, favoring the region containing new answers and hiding facts in the other region.The collapse weakens as α decreases; when new answers span the vocabulary, the common push cancels and forgetting is gradual.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 NORMALIZATION DRIVES RECOVERY

• Normalization retracts the common shift after new facts are learned, allowing old facts to recover, while accumulating fact-specific drift causes lasting erosion.The shift reverses when the new facts’ individual margin becomes positive; old representations retain their relative geometry during collapse, but recovery becomes less complete as individual displacements accumulate.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

4.1 TRAINING TRANSFORMERS ON SYNTHETIC BIOGRAPHIES MEETS THE CONDITIONS

• Synthetic biographies meet the proposed conditions through shared query structure, answer subsets concentrated by group, and Transformer normalization.Individuals in A and B′ are distinct, while B′ shares its answer subset with B.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 THE TRANSFORMER SHOWS THE SAME GEOMETRIC SPLIT

• In the Transformer, the common shift grows quickly and partly retracts around recovery, while fact-specific displacement grows more slowly and eventually becomes larger.The analysis tracks representations at the readout input and separates their mean displacement from each fact’s remainder.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 SUBTRACTING THE COMMON SHIFT REMOVES THE COLLAPSE

• Removing the common logit shift eliminates the collapse, whereas keeping only that shift produces collapse and recovery; fact-specific drift drives erosion.The shift is carried mostly by attention value and output weights, with little contribution from query and key weights.Source: §4.3 · pp. 7–8

4.4 THE LEARNING RATE CHANGES THE DEPTH OF THE COLLAPSE

• The collapse bottom occurs at nearly the same new-fact accuracy despite a threefold change in steps, while its depth increases with learning rate.The bottom comes later in learning when there are fewer new facts.Source: §4.4 · p. 8

5 THE COLLAPSE IN A PRETRAINED LANGUAGE MODEL

• OLMo 2 1B is finetuned without replay on synthetic individuals or real entities, tracking known CounterFact facts against new answers outside the old answers’ first-token region.Synthetic subjects reveal nothing about their answers; product types can hint at manufacturers.Source: §5.1 · p. 8, §5 · p. 8

5.1 THE COLLAPSE PERSISTS, AND REVERSES WHEN THE NEW FACTS ARE ARBITRARY

• Old-fact recall initially rises slightly in both runs because fact-stating documents favor answers over other continuations, fixing old facts already ranked correctly.This initial increase occurs without replay.Source: §5.1 · p. 8

5.2 REMOVING ONE DIRECTION OF THE UPDATE RESTORES THE OLD FACTS

• Removing each weight update’s top singular component largely removes the collapse; a same-norm random direction does not, and old-fact recall exceeds trained recall at equal new-fact accuracy.Because prompts vary, the edit is applied to weights; in both controlled Transformers, the removed direction aligns with the common shift.Source: Figure 5 · p. 9, §5.2 · p. 9

5.1 THE COLLAPSE PERSISTS, AND REVERSES WHEN THE NEW FACTS ARE ARBITRARY

• Synthetic-individual training recreates collapse, recovery during new-fact learning, then erosion; four times more individuals makes collapse deeper and longer, while real-entity recall never recovers.For real entities, the shared direction continues helping select answers within their output region, so it is never withdrawn.Source: §5.1 · p. 9, Figure 5 · p. 9

6 DISCUSSION

• The authors connect shared-shift recovery to spurious forgetting’s task-alignment account (Zheng et al., 2025), distinguishing recovery without probes, prompts, or replay.The discussion also places the work alongside research separating stored from accessible information and reports of recoverable associations and capabilities.Source: §6 · pp. 9–10
• Forgetting comprises a shared loss of access shaped by common new-data structure and slower fact-specific erosion; answer-predictive subject structure can speed new learning while keeping old facts hidden.The authors suggest that what helps generalize to new facts may also keep old ones from recovering.Source: §6 · p. 10
• The pretrained-model comparison studies one model, and the two new-fact types differ in ways beyond what their subjects reveal about their answers.The minimal model is deliberately extreme, and normalization-driven recovery is likely weaker on real data.Source: §6 · p. 10

A.1 CONTROLLED TRANSFORMER

• Synthetic biographies control old and new facts, with held-out phrasings used to evaluate recall.The controlled Transformer is pretrained on two disjoint groups, then finetuned on new individuals whose attribute values come either from one half of the value pool or both halves.Source: §A.1 · p. 15

D The pretrained language model

• The pretrained-model setup tracks previously recalled CounterFact facts while finetuning on new facts without replay; Table 1 maps analyses across models.The old facts comprise 1,106 facts, with recall evaluated on 422 prompts where the answer is absent.Source: D The pretrained language model · p. 14, Table 1 · p. 14, §A.2 · pp. 15–16

A.2 PRETRAINED LANGUAGE MODEL

• The pretrained-model experiment uses synthetic individuals and little-known real entities as new facts, with held-out phrasings for evaluation.The tracked old facts are those the model answered correctly before finetuning; evaluation measures top-1 accuracy of the first answer token.Source: §A.2 · pp. 15–16

B ANALYSIS OF THE MINIMAL MODEL

• The minimal-model analysis derives old-state displacement, how new facts write into old states, common-shift dynamics, and fact-specific displacement.It also checks the derivations in simulations and tests interventions on the minimal model.Source: B ANALYSIS OF THE MINIMAL MODEL · p. 17

B.1 NOTATION, ASSUMPTIONS AND SIMULATIONS

• The analysis assumes gradient-flow finetuning and high-dimensional, weakly overlapping random keys; its common-coordinate assumption is used only for a closed-form result.The old facts are not trained during finetuning, and the simulations use minibatch gradient descent as an approximation to gradient flow.Source: §B.1 · p. 17

B.2 GRADIENTS OF THE FINETUNING LOSS

• The gradient analysis separates errors at the logits, normalized states, and hidden states, then expresses each new fact’s weight gradient as an error-key outer product.Normalization divides the hidden-state error by the state norm and removes its component along the state.Source: §B.2 · p. 18

B.3 DECOMPOSITION OF THE DISPLACEMENT OF THE OLD FACTS

• Each old hidden state’s displacement splits exactly into a common shift and a fact-specific part; the mean displacement measures the shift, while deviations measure individual motion.When α = 0, the common shift vanishes and all displacement is fact-specific.Source: §B.3 · pp. 18–19

B.4 HOW THE NEW FACTS WRITE INTO THE OLD HIDDEN STATES

• When new answers are concentrated in one vocabulary region, their early errors share a component that writes a common shift into old states; drawing answers across the vocabulary largely removes this push.The shared write dominates when new-fact errors have a common component, and it fades as the new facts are learned.Source: §B.4 · pp. 20–21
• The common write is faster than fact-specific learning: shared error enters every state with weight α, while a single fact’s error enters its own state with weight 1/nB.Normalization adds an individual-margin term to the common-shift dynamics; both margins vanish as the new facts are learned.Source: §B.4 · p. 21, §B.5 · p. 22

B.5 DYNAMICS OF THE COMMON SHIFT

• The common-shift approximation is reliable when projected errors share a sign across new facts, except near the shift’s peak where their signs change at different times.The individual-margin contribution exists because normalization removes the radial component of the error.Source: §B.5 · pp. 21–22

B.6 REDUCTION UNDER THE COMMON-COORDINATE APPROXIMATION

• The common-coordinate approximation yields a one-dimensional shift equation whose direction depends on shared and individual margins; without normalization, the pullback term is absent.Once new facts are learned, the restoring term vanishes, so the shift stops wherever it is rather than returning to its initial value.Source: §B.6 · pp. 22–23

B.7 EMPIRICAL VALIDATION

• The integrated shift dynamics match the measured shift within 1% of its range with normalization and 2% without normalization.The validation integrates the measured equation using an explicit Euler scheme with the learning rate as its time step.Source: §B.7 · p. 23, Figure 7 · p. 23

B.8 THE MECHANISM, STEP BY STEP

• Equation 4’s shared vector reproduces each old state’s update within 16% through the collapse, while new and old states move equally far along that direction.The shared-write speed grows almost in proportion to α.Source: §B.8 · p. 23, Figure 8 · p. 24
• At the collapse’s trough, 93% of old facts still rank their correct answer first within their own half, despite only 17% overall accuracy.The common shift pushes their logits toward the half containing new answers, disrupting recall without scrambling their within-half ordering.Source: §B.8 · p. 23, Figure 9 · p. 24
• The mean pairwise cosine of old states rises from 0.06 to 0.54 as normalization rotates them toward the growing shift; both recover as the shift recedes.At the trough, 82% of predictions for A fall in the half of the new answers, favoring answers common among new facts.Source: §B.8 · p. 24, Figure 10 · p. 25
• Readout weights for old answers change about 5% during the collapse and 10% by finetuning’s end, versus more than 60% for new-answer weights.Old-answer weights receive gradient only briefly, while the model assigns probability to them for new facts.Source: §B.8 · p. 24, Figure 11 · p. 25

B.9 THE OLD FACTS SURVIVE THE COMMON SHIFT

• Removing the common part of the logit change restores old-fact accuracy from 0.17 to 1.00 at the trough, while the remaining decline reflects fact-specific erosion.For new facts B, removing the common part changes almost nothing; wrongly recalled A facts favor the answer of their closest new key twice as often as a random new key.Source: §B.8 · p. 24, §B.9 · p. 24
• Removing the shared update from the store slows learning of new facts, which reach 0.59 at step 200 rather than 0.99.The shared update also carries information common to the new facts.Source: §B.9 · p. 25

B.10 THE ANALYSES OF THE TRANSFORMER AND THE LANGUAGE MODEL, IN THE MINIMAL MODEL

• Subtracting the common logit-change part restores old-fact accuracy from 0.17 to 1.00 at the trough, while keeping only that part produces most of the collapse.Fact-specific drift overtakes the shared shift only after about 10^4 steps in the minimal model.Source: §B.10 · p. 25
• Removing the top singular component of the update restores old facts at the trough, unlike a same-norm random rank-one removal.The top direction aligns with the shared key direction at cosine 0.99; once the shift recedes, its alignment falls to 0.11 by step 200, and the edit no longer helps.Source: §B.10 · pp. 25–26
• The top-direction edit also slows learning new facts, but at equal new-fact accuracy it recalls more old facts than the trained model.The edit therefore does not simply rewind finetuning.Source: §B.10 · p. 26

B.11 FACT-SPECIFIC DISPLACEMENTS

• Fact-specific displacements keep accumulating while new facts produce errors, because their random coefficients include no term that pulls each displacement back.A random direction captures, on average, a fraction 1/d of the squared Frobenius norm of the update.Source: §B.11 · pp. 26–27

B.12 WHY RECOVERY IS INCOMPLETE

• The readout supplies a second persistent bias toward new answers, and freezing it raises recovered old-fact accuracy from 0.56 to 0.61.The common-shift withdrawal force, by contrast, vanishes with new-fact errors.Source: §B.12 · pp. 27–28, Figure 17 · p. 28

B.13 INTERVENTIONS IN THE MINIMAL MODEL

• Projecting out the shared direction prevents collapse but leaves final old-fact accuracy at 0.45 versus 0.51 without projection, because fact-specific erosion remains.Replaying 16 of 128 old facts raises never-replayed facts’ trough accuracy from 0.17 to 0.50, but final accuracy only from 0.54 to 0.62.Source: §B.13 · pp. 28–29, Figure 18 · p. 29

C.1 THE OLD FACTS KEEP THEIR RANKING WITHIN THEIR REGION

• At the collapse’s trough, 86% of old facts still rank their correct answer first within their own answer half, although only 12% are recalled overall.Within-half ranking falls only to 83% after 500 steps as fact-specific drift erodes old facts.Source: §C.1 · p. 30, Figure 19 · p. 30

C.2 TIMING ACROSS LEARNING RATES AND BATCH SIZES

• Across five learning-rate and batch-size conditions, the trough spans steps 34–100 while new-fact accuracy remains 36–44%, so collapse timing follows new-fact learning rather than training steps.The figure marks the trough for each condition.Source: §C.2 · p. 30, Figure 20 · p. 30

C.3 TIMING ACROSS THE NUMBER OF NEW FACTS

• With 125 new individuals, 89% of old facts remain recalled at the trough, versus 38% with 500; fewer new facts delay and shallow the collapse.In the minimal model, the common shift peaks when the new facts’ individual margin turns positive; accuracy at that point depends on the number of new facts, not learning rate or batch size.Source: §C.3 · p. 31, Figure 21 · p. 31, Figure 22 · p. 31

C.4 LOCALIZING THE COMMON SHIFT

• At the trough, attention value and output weights carry 84% of the common shift, and query and key weights carry 11%.Restoring each weight group separately measures its share; after recovery, value and output weights still carry 85%.Source: §C.4 · pp. 31–32, Table 2 · p. 32

C.5 TRANSFORMER WITH MLP BLOCKS

• With B′, old-fact accuracy falls to 0.53 at step 30, recovers to 0.93, then erodes to 0.75; removing the common logit-change part raises trough accuracy to 0.99.The common shift and fact-specific drift reproduce the attention-only model’s mechanism, while 97% of old facts retain their within-half ranking at the trough.Source: §C.5 · p. 32, Figure 24 · p. 33
• MLP and attention value/output weights each carry about half the common shift, at 46% and 47%, respectively.The shift is written into associative memories, not primarily into the query and key weights that determine where attention goes.Source: §C.5 · pp. 33–34
• With MLP blocks, the collapse trough shifts from step 20 to 56 across optimizer settings, while new-fact accuracy there is 23–33%.The timing follows the new facts’ individual margin turning positive, but the relation between that margin and accuracy depends on architecture and number of new facts.Source: Figure 26 · p. 33, §C.5 · p. 34

D.1 WHY RECALL OF THE OLD FACTS FIRST RISES.

• Old-fact recall rises from 0.87 to about 0.93 in the first 25–30 steps without replay, because finetuning raises correct answers above other continuations without changing their answer ranking.Every fact that becomes correct during this rise was already ranked first among possible answers before finetuning.Source: §D.1 · p. 35

D.2 REMOVING ONE DIRECTION OF THE UPDATE

• Removing the top direction of each weight update keeps old facts near their pre-finetuning accuracy through the collapse, while a random same-norm control leaves them at trained accuracy.The edit also keeps ordinary-text loss near its pre-finetuning value, while finetuning alone raises it.Source: §D.2 · p. 35, Table 3 · p. 35
• At matched new-fact accuracy for real entities, the edited model recalls 88%, 88%, and 79% of old facts, versus 81%, 73%, and 72% for the trained model.No point in the training run combines the edited model’s old- and new-fact accuracies.Source: §D.2 · p. 35

D.3 THE SAME EDIT IN THE CONTROLLED TRANSFORMERS

• The edit removes each block-weight update’s top singular component, leaves embeddings and unembedding untouched, and uses same-norm random rank-one removal as control.The removed direction holds 8% of squared update norm in the attention-only Transformer and 14% with MLP blocks.Source: §D.3 · pp. 35–36
• The edit recalls more old facts throughout finetuning, but slows new-fact learning in the attention-only Transformer and barely affects it with MLP blocks.The removed direction carries part of what the new facts share.Source: §D.3 · p. 36
• At collapse, the edit’s readout-state change aligns with the common shift at cosine 0.75 in the attention-only Transformer and 0.97 with MLP blocks, falling to 0.22 and 0.49 after 1,200 steps.In the attention-only Transformer, removing about a quarter of the shift at collapse restores half of the old facts.Source: §D.3 · p. 36, Figure 27 · p. 36
• At step 100, the edited model recalls old facts at 0.60 against 0.21 in the attention-only Transformer and 1.00 against 0.90 with MLP blocks, at matched new-fact accuracy.While new facts are being learned, the edited model recalls more old facts at every new-fact accuracy; the comparison loses meaning after full new-fact accuracy is reached.Source: §D.3 · p. 36
• At collapse, top-direction removal raises old-fact accuracy from 0.12 to 0.57 in the attention-only Transformer and from 0.54 to 1.00 with MLP blocks, unlike control.The random rank-one control leaves old-fact accuracy exactly where training did.Source: Figure 27 · p. 36, §D.3 · p. 36