Source-linked AI summary
How Learning Governs Unlearning across the Memorization-Generalization Spectrum
Hwiyeong Lee, Hyelim Lim, Ingyu Bang, Hoki Kim, Taeuk Kim
TL;DR
The paper investigates how memorization and generalization during learning relate to unlearning’s retain damage, a concern because unlearning can erase unrelated knowledge and skills. It compares models across grokking, a controlled memorization–generalization spectrum, and LLM recall settings, finding that greater reliance on shared generalizing mechanisms generally accompanies greater retain damage.
Problem
Unlearning can remove unrelated knowledge and skills, while how models’ memorization and generalization strategies shape that retain damage remains insufficiently examined.
Source: §1 · p. 1Method
The paper compares unlearning across grokking-based modular addition models, bucketed modular addition with controlled memorization floors, and LLM verbatim and factual recall settings.
Source: §1 · pp. 1–2Results
Across modular addition, bucketed modular addition, and LLM recall, greater reliance on shared generalizing mechanisms generally accompanies greater retain damage.
Source: §7 · p. 10Takeaways & Limitations
Unlearning objectives should distinguish removing training contributions from suppressing targeted outputs and favor replacement answers that preserve shared structure.
Source: §1 · p. 2In modular addition, the proposed mechanism comparison concerns grokked models that rely entirely on the shared circuit, unlike memorizing models.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
While unlearning seeks to negate undesired capabilities acquired through learning, little research has examined how the way models learn shapes their subsequent unlearning. In this paper, we investigate this connection from the perspectives of memorization and generalization, the two most representative yet competing strategies that models employ during training. We first classify memorization- and generalization-heavy models using grokking in modular addition and compare their responses to unlearning, showing that the latter suffer greater retain damage, i.e., a larger performance drop on the retain set. Furthermore, we conduct a finer-grained analysis by introducing bucketed modular addition, in which the respective contributions of the two strategies can be explicitly controlled across the memorization-generalization spectrum. In this setup, we reaffirm that the same trend persists and is nearly monotonic. We further demonstrate that this relationship also holds in LLM unlearning across verbatim and factual recall settings. Finally, we provide two practical insights for developing better unlearning methods, highlighting the importance of accounting for learning dynamics in unlearning.
1 INTRODUCTION
The paper asks how memorization and generalization during training shape unlearning’s retain damage. It tests this relationship from grokking models through a controlled spectrum to LLM recall, then draws practical implications for unlearning objectives.
1 INTRODUCTION
- The paper frames retain damage as the loss of unrelated knowledge and skills while unlearning information owners request removed.It investigates whether models’ differing reliance on memorization and generalization shapes that damage.Source: §1 · p. 1
- Generalizing grokked models suffer greater retain damage than memorizing models, consistent with unlearning disrupting a shared solution mechanism.In the toy model, that mechanism is a Fourier circuit.Source: §1 · p. 1
- Bucketed modular addition controls the balance of memorization and generalization; retain damage rises nearly monotonically with each step toward generalization.The memorization floor measures the sample-specific information required beyond the shared rule.Source: §1 · p. 2
- In verbatim and factual LLM recall, a higher estimated memorization floor again predicts less retain damage.The paper treats linguistic and semantic regularities as generalization and sample-specific remainder as the memorization floor.Source: §1 · p. 2
- Unlearning should distinguish removing a forget set’s training contribution from suppressing its induced answers, and favor replacement answers consistent with shared rules.Retraining without the forget set may still reproduce targeted answers when generalization accounts for them; arbitrary replacement answers incur more retain damage than rule-consistent ones.Source: §1 · p. 2
2 BACKGROUND AND RELATED WORK
The background defines memorization, generalization, and machine unlearning, then situates the paper’s question among prior work on unlearning difficulty.
2.1 MEMORIZATION AND GENERALIZATION
- Memorization captures sample-tied information, while generalization learns a shared solution mechanism that applies across examples.The paper applies these meanings to language models as specific knowledge versus linguistic or semantic regularities.Source: §2.1 · p. 3
2.2 MACHINE UNLEARNING
- Machine unlearning seeks to remove information learned from the forget set while preserving retain-set performance, with retraining without the forget set as an ideal standard.Removing forget-set information can nevertheless degrade retain-set performance.Source: §2.2 · p. 3
- Prior work relates unlearning difficulty to data entanglement and frequency, but the relationship between unlearning, generalization, and memorization remains largely unexplored.The paper addresses that gap by investigating how memorization and generalization shape unlearning.Source: §2.2 · p. 3
3 MEMORIZATION VS. GENERALIZATION: A BINARY VIEW
Using grokking to compare models with different learning strategies, the paper tests whether unlearning damages retain performance more when models rely on shared generalizing mechanisms.
3.1 GROKKING AS A BINARY LENS IN MODULAR ADDITION
- Grokking provides two models that fit training data equally well but differ in strategy: one memorizes examples, while the other relies on a shared circuit.The study uses this contrast to examine how learning strategy relates to retain damage.Source: §3 · p. 3, §3.1 · p. 4
3.2 GENERALIZATION IS A KEY CAUSE OF RETAIN DAMAGE
- The comparison trains a toy transformer through grokking, then applies RL and GD until forget accuracy reaches chance or lower across four checkpoints and five forget-set selection seeds.All checkpoints have 100% training accuracy while spanning poor to perfect test accuracy.Source: §3.2 · p. 4
- At chance-level forget accuracy, mean retain accuracy under RL is 59.5% for the memorizing model and 8.1% for the grokked model; under GD it is 23.0% and 3.3%, respectively.Models sampled during circuit formation and cleanup fall between those endpoints.Source: §3.2 · p. 4, Table 1 · p. 4
- The greater retain damage in the generalized model is consistent with unlearning acting on a shared solution mechanism used by both forget and retain examples.The paper next tests whether unlearning damages the Fourier circuit identified as that mechanism.Source: §3.2 · p. 4, §3.3 · p. 4
3.3 FOURIER CIRCUIT AS A CONCRETE EXAMPLE OF GENERALIZATION
- In the grokked model, unlearning reduces Fourier-circuit FVE from 0.97 to 0.33 and raises restricted loss from near zero to 2.44, while the memorizing model relies little on that circuit.These changes accompany declining forget accuracy and collapsing retain accuracy; restricted loss measures how well the circuit alone solves the task.Source: §3.3 · p. 5, Figure 2 · p. 5
4 FINE-GRAINED ANALYSIS OF UNLEARNING ACROSS THE SPECTRUM
The section extends the memorization–generalization comparison to tasks where a shared rule determines only part of each answer. Bucketed modular addition controls that balance, and retain damage generally falls as the required memorization floor rises.
4 FINE-GRAINED ANALYSIS OF UNLEARNING ACROSS THE SPECTRUM
- The analysis addresses the gap between fully algorithmic tasks and answers partly determined by shared rules by testing intermediate spectrum positions.BMA is designed to control how much of an answer the shared rule determines and how much must be memorized individually.Source: §4 · p. 5
4.1 BUCKETED MODULAR ADDITION
- BMA varies bucket size while fixing 256 answer classes, balancing modular-rule bucket selection against memorization of the correct member.At M = 1 the shared rule determines every answer; at M = 256 labels are entirely random, with intermediate sizes balancing both strategies.Source: §4.1 · pp. 5–6
- The memorization floor is the minimum sample-specific information needed beyond the shared solution, equal to log2 M bits per example in BMA.With 256 possible answers, the floor ranges from none to all 8 bits as M varies from 1 to 256.Source: §4.1 · p. 6
4.2 THE MEMORIZATION FLOOR PREDICTS RETAIN DAMAGE
- The experiments train one-layer transformers across five bucket sizes and compare four unlearning methods using matched forget/retain splits and three selection seeds.Forget fractions are 1%, 5%, or 10%; retain damage is measured at Pf = 0.1.Source: §4.2 · p. 6
- For M = 1, 4, 16, and 64, bucket accuracy rises after training accuracy saturates while within-bucket accuracy stays near chance, separating rule learning from member selection.This pattern indicates that generalization resolves the bucket while individual memorization supplies the correct member.Source: §4.2 · p. 6
- Greater memorization floors generally correspond to lower retain damage at matched forgetting levels across BMA settings.The comparison measures retain damage as 1 − Pr at Pf = 0.1 and averages over three seeds.Source: §4.2 · p. 6
5 THE SPECTRUM IN PRACTICE: LLM UNLEARNING
The paper tests whether the link between memorization floor and retain damage extends to LLM recall. Across verbatim and factual settings, lower-floor forget sets or relations generally incur greater retain damage at comparable forgetting levels.
5 THE SPECTRUM IN PRACTICE: LLM UNLEARNING
- The LLM analysis estimates sample-specific information beyond shared regularities using unintended memorization and paired Hubble models.The standard model serves as a reference for predictability without training on inserted texts; the perturbed model includes those texts.Source: §5 · p. 7
5.2 SEMANTIC REGULARITIES IN FACTUAL RECALL
- At comparable forgetting, lower-floor nationality and email facts generally incur more in-relation damage than higher-floor birthplace and UUID facts.The gap between in-relation and out-relation damage is also generally larger for relations with lower floors.Source: Figure 4 · p. 8, §5.2 · p. 9
5.1 LINGUISTIC REGULARITIES IN VERBATIM RECALL
- At matched forgetting, unlearning the High forget set generally causes less retain damage than unlearning the Low set, and damage is generally greater on ID than OOD passages.The authors interpret the pattern as greater pressure on shared linguistic mechanisms when suppressing lower-floor passages.Source: Figure 4 · p. 8, §5.1 · p. 8
- Verbatim recall uses inserted book passages, ranks them by estimated floor, and compares High and Low forget sets across retained, out-of-distribution, and unseen passages.Four unlearning methods are evaluated at mean forget probability 0.1.Source: §5.1 · pp. 7–8
- Across four models, TF-IDF correlates positively with estimated memorization floor, consistent with rarer vocabulary requiring more individual memorization.The High example contains more rare words and has higher TF-IDF than the Low example.Source: §5.1 · p. 8, Table 2 · p. 8
5.2 SEMANTIC REGULARITIES IN FACTUAL RECALL
- Factual recall tests four YAGO relations using candidate-set accuracy and compares retained facts within the target relation against facts from other relations.The experiment selects correctly recalled facts and measures damage when forget accuracy reaches 10%.Source: §5.2 · pp. 8–9
6 DISCUSSION
The discussion distinguishes removing a forget set’s training contribution from suppressing its targeted outputs. It also argues that where probability is reassigned can affect retain damage.
6 DISCUSSION
- Retraining without the forget set removes its training contribution but may not suppress outputs that remain predictable from shared mechanisms.For modular addition, the zero memorization floor means the rule can be learned from examples outside the forget set.Source: §6 · p. 9
- The training-contribution and behavioral-suppression goals can diverge when the targeted behavior is not exclusively dependent on the forget set.A retrained model may still produce targeted outputs even though retraining is the gold standard for removing the forget set’s contribution.Source: §6 · p. 9
- At 10% forget accuracy, within-bucket RL targets preserve substantially higher retain exact accuracy and forget bucket accuracy than full-answer-space targets.Replacement targets consistent with the modular rule reduce pressure on the shared solution mechanism, so where removed probability is reassigned matters.Source: §6 · p. 9
7 CONCLUSION
The conclusion links learning strategy to retain damage: greater reliance on shared solution mechanisms generally accompanies greater damage. It also highlights distinct unlearning objectives and probability redistribution, alongside disclosures about AI assistance and review.
7 CONCLUSION
- Across the memorization–generalization spectrum, greater reliance on shared solution mechanisms generally accompanies greater retain damage, including in BMA and LLM recall settings.In modular addition, generalizing models suffer more damage than memorizing models as unlearning disrupts the shared circuit.Source: §7 · p. 10
- The paper identifies contribution removal versus output suppression and preserving shared structure during probability redistribution as practical considerations for unlearning.The conclusion presents these as implications of the findings, not as interchangeable objectives or a single prescribed method.Source: §7 · p. 10
- The authors report AI assistance in modeling, methodology, implementation, interpretation, writing, figures, code, and literature review, with outputs reviewed and experimental statistics checked.They report that AI did not propose or refine hypotheses or support qualitative and thematic analysis.Source: AI USE STATEMENT · p. 10
A UNLEARNING METHODS
The section defines the unlearning objectives and experimental measures used to compare methods. It also describes the modular-addition checkpoints and the Fourier-circuit measurements used in the analysis.
A UNLEARNING METHODS
- Gradient ascent minimizes negative forget-set cross-entropy, while random labeling trains toward fixed incorrect answers sampled once for each comparison.For modular addition, alternatives are sampled uniformly from the incorrect answer labels; BMA uses other output classes, and factual recall uses incorrect candidate values.Source: A UNLEARNING METHODS · p. 14
- Gradient difference combines forget-loss ascent with retain-loss descent, while the other methods specify distinct objectives for changing forget-set predictions.All objectives are minimized; only gradient difference includes a retain-loss term in the implementations considered.Source: A UNLEARNING METHODS · p. 14
- DPO prefers a fixed incorrect answer relative to the frozen pre-unlearning model, whereas NPO marks the original answer as dispreferred without specifying an alternative.For DPO, the scored answer log-likelihood is a single-label probability in BMA and a summed answer-token likelihood in factual recall; NPO’s scoring differs by recall setting.Source: A UNLEARNING METHODS · p. 14
- Maximum entropy targets a uniform distribution over the full BMA answer space, while CEU replaces the original token target with the current distribution over remaining vocabulary tokens.CEU recomputes that target at every update; ME’s constant leaves the gradient unchanged and expresses the objective as KL divergence to the uniform answer distribution.Source: A UNLEARNING METHODS · p. 14
B.1 CONFIGURATION
- Four modular-addition checkpoints span memorization through grokking, all with 100% training accuracy, and runs use fixed forget/retain partitions across methods and checkpoints.All 40 checkpoint–method–seed combinations reach the forget-accuracy threshold; retain accuracy is reported with 95% Student-t confidence intervals across five selection seeds.Source: §B.1 · p. 15
B.2 CIRCUIT MEASUREMENTS
- Circuit measurements use fixed key frequencies to project logits, measure explained variance, and compute test cross-entropy from restricted logits.Lower restricted loss indicates that the Fourier circuit can solve the task more effectively on its own.Source: §B.2 · p. 16
B.3 FOURIER GRADIENT FILTERING
The section filters RL gradients along Fourier-circuit directions and tests when this protects the circuit during unlearning. In the grokked model, filtering improves circuit measures and retain accuracy, while the effect depends on training phase.
B.3 FOURIER GRADIENT FILTERING
- Filtering projects the RL logit gradient away from five orthogonal key-frequency directions, removing its direct signal for changing the Fourier circuit’s contribution.This reduces pressure to modify the circuit but does not guarantee protection, because parameter updates need not preserve the logit-gradient constraint.Source: §B.3 · p. 16
- Key-frequency filtering substantially improves retain accuracy during cleanup and grokking, with much smaller effects during memorization and circuit formation.The gains accompany higher FVE and lower restricted loss; non-key filtering remains close to unmodified RL.Source: §B.3 · p. 16, Table 7 · p. 17
B.4 CAUSAL INTERVENTION
- The intervention compares key-frequency filtering with unmodified RL and a same-size non-key control across the same four models, selection seeds, and forget threshold.All conditions preserve the RL objective and output vocabulary, and all 60 runs reach the forget-accuracy threshold.Source: §B.3 · p. 16, §B.4 · p. 17
- In the grokked model, key-frequency filtering raised FVE from 0.186 to 0.509 and retain accuracy from 8.1% to 30.6%, while restricted loss fell from 2.324 to 0.167.Non-key filtering left FVE and restricted loss nearly unchanged, with retain accuracy at 8.2%; the authors conclude that disruption of a shared solution mechanism contributes to retain damage.Source: §B.4 · p. 17, Table 8 · p. 17
C.1 CONFIGURATION
- In BMA, models generalize the modular rule but not the within-bucket index, whose log2 M bits constitute the memorization floor.Test bucket accuracy is 93.57%, 93.82%, and 96.63% for M = 4, 16, 64, while within-bucket accuracy is close to the respective 1/M chance levels.Source: §C.1 · p. 18
- BMA retain damage is measured at Pf = 0.1 by interpolating retain probability between adjacent saved evaluations, with uncertainty reflecting three-seed variation.Initial retain probability exceeds 0.999997, so 1 −Pr closely approximates the decrease from the pre-unlearning model.Source: §C.1 · pp. 18–19
C.2 MEMORIZATION FLOOR AND RETAIN DAMAGE RESULTS
At Pf = 0.1, retain damage falls as the memorization floor grows across all three forget-fraction panels. Larger forget fractions leave more damage at a given bucket size.
C.2 MEMORIZATION FLOOR AND RETAIN DAMAGE RESULTS
- At zero memorization-floor bits, all methods sit just below the 0.9 nonselective baseline; by eight bits, all have moved well clear of it.The baseline corresponds to Pf = Pr = 0.1.Source: §C.2 · p. 19, Figure 5 · p. 19
- At a given bucket size, larger forget fractions leave more retain damage, producing successively higher panel levels.Source: §C.2 · p. 19
- As the memorization floor grows, retain damage falls across all methods and forget fractions.Figure 5 measures retain damage as 1 −Pr at Pf = 0.1.Source: Figure 5 · p. 19, §C.2 · p. 19
C.3 CORRELATION ANALYSIS OF MEMORIZATION FLOOR AND RETAIN DAMAGE
The bucket-size sweep links higher memorization floors with lower retain damage, and the LLM experiments extend this analysis to verbatim and factual recall. Across verbatim recall, lower-floor forget sets leave more retain damage; passage-level lexical rarity also tracks memorization floor.
C.3 CORRELATION ANALYSIS OF MEMORIZATION FLOOR AND RETAIN DAMAGE
- 10 of 12 method–forget-fraction cells have Spearman ρs = −1.000, with retain damage decreasing as bucket size—and thus memorization floor—increases.The exceptions are GD and ME at 1%, each with one reversal; Pearson’s r ranges from −0.924 to −0.997. The sweep supports an association in this task family, not a method-independent bound.Source: §C.3 · p. 19, Table 11 · p. 20
D.1 MODELS AND DATA
- Across the four Hubble model sizes and budgets, each pair shares a base corpus, while only the perturbed model receives the controlled insertions.The standard model supplies reference likelihoods and is not retrained for each forget set.Source: §D.1 · p. 20, Table 12 · p. 20, Table 13 · p. 20
D.2.1 CONFIGURATION
- Verbatim High and Low forget sets are the top and bottom 30% by memorization floor; greedy swaps nearly equalize their token totals.For Hubble-1B-500B, the tier is first restricted to passages with P̄pert ≥ 0.6; the High–Low floor intervals do not overlap and differ by roughly 250 bits per passage.Source: §D.2.1 · p. 21, Table 14 · p. 21, Table 15 · p. 21
D.2.2 LEXICAL RARITY AND MEMORIZATION FLOOR
- Passage-level TF-IDF and memorization floor rise together in all four models, despite every plotted passage having the same 256 insertions.The high-floor example is dense with rare words, while the length-matched low-floor example uses mostly common vocabulary.Source: Figure 6 · p. 22, §D.2.2 · pp. 22–23
D.2.3 MEMORIZATION FLOOR AND RETAIN DAMAGE RESULTS
- Low-floor forget sets leave greater retain damage than High-floor sets across both model sizes and pretraining budgets, for nearly every method and retain group.Damage is largest on ID-seen passages, smaller on OOD-seen passages, and at most a fraction of a percent on unseen passages.Source: Table 17 · p. 22, §D.2.3 · p. 23
D.3.1 CONFIGURATION
- Factual recall uses 50 already-correct facts per model and relation, split into 15 forget facts and 35 retain facts.Runs are reported when forget accuracy first reaches at most 10%; Hubble-1B omits birthplace because too few facts are recalled correctly to form a pool of 50.Source: §D.3.1 · pp. 23–24, Table 18 · p. 23, Table 19 · p. 23
D.3.2 MEMORIZATION FLOOR AND RETAIN DAMAGE RESULTS
In factual recall, relations with higher memorization floors generally leave less in-relation damage, while damage is concentrated within the unlearned relation. The floor–damage association is mostly negative across models and methods, though weaker and less consistent for Hubble-1B.
D.3.2 MEMORIZATION FLOOR AND RETAIN DAMAGE RESULTS
- A relation with a higher memorization floor leaves less in-relation damage in 15 of 16 model–method pairs; damage is generally concentrated within the unlearned relation.The UUID exception in Hubble-8B-500B leaves retained UUID facts untouched; the ordering is clearest for Hubble-8B and weakest for Hubble-1B-500B, where nationality takes the most damage under every method.Source: Table 20 · p. 24, §D.3.2 · p. 24, Table 21 · p. 25
D.3.3 CORRELATION ANALYSIS OF MEMORIZATION FLOOR AND RETAIN DAMAGE
- 15 of 16 Pearson coefficients are negative, linking higher relation-level memorization floors with lower in-relation damage across model sizes and budgets.The association is stronger in Hubble-8B (r = −0.978 to −0.517) than Hubble-1B (r = −0.948 to +0.435); the sole positive value is NPO in Hubble-1B-500B.Source: Table 22 · p. 25, §D.3.3 · p. 25
E.1 BMA REDISTRIBUTION EXPERIMENT
The redistribution experiment finds that restricting incorrect targets to the same bucket preserves more retain accuracy at matched forget accuracy. A separate LLM analysis associates stronger token-weight separation by memorization floor at the stopping point with better retain performance.
E.1 BMA REDISTRIBUTION EXPERIMENT
- The two RL conditions differ only in whether incorrect probability targets are unrestricted or confined to the correct answer’s bucket.The experiments use two task configurations and compare conditions at 10% forget exact accuracy.Source: §E.1 · p. 26
- At 10% forget exact accuracy, within-bucket targets retain 75.7% vs 15.4% at (131, 2), and 71.3% vs 28.0% at (277, 4).Forget bucket accuracy stays near 100% with within-bucket targets, versus 12.5% and 14.7% with unrestricted targets, respectively.Source: Table 23 · p. 26, §E.1 · p. 26
E.2 IMPLICATIONS FOR EXISTING UNLEARNING OBJECTIVES
- The LLM comparison isolates token weights on a shared gradient-ascent loss, separating tokens above versus below a one-bit memorization-floor threshold.The threshold means the perturbed model assigns a token at least twice the probability given by the standard reference model; the experiment omits other components of the original objectives.Source: §E.2 · pp. 26–27, Table 24 · p. 27
- At the stopping point, the share of forget tokens no longer predicted through memorization grows from about a quarter to about half; sharper separation relates to better retain performance (Spearman ρ = 0.76).Starting-point separation shows no such trend (ρ = −0.30); this comparison concerns token weighting, not the published methods as a whole.Source: §E.2 · pp. 26–27, Table 25 · p. 28