PaperlayerView exact revision on arXiv

Source-linked AI summary

The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning

Anna Borisiuk, Andrey Savchenko, Alexander Panchenko, Elena Tutubalina

arXiv:2608.14229v1cs.CL

TL;DR

Popular facts are harder to erase than rare ones, while existing unlearning methods apply uniform pressure. AdaPop adapts forgetting to fact popularity and achieves the lowest adversarial-query scores among stable methods across three models, with the best paraphrase scores on two.

  • Problem

    Existing unlearning methods treat forget examples equally despite popular facts being more deeply encoded and harder to remove, leaving the popularity gap unaddressed.

  • Method

    AdaPop uses external popularity signals to set per-fact gradient exponents and a dual-ascent controller to adapt the retain penalty.

  • Results

    AdaPop attains the lowest adversarial-query scores among stable methods on all three models, the best paraphrase scores on two of three, and stays within 0.05 MMLU of pre-unlearning capabilities.

  • Takeaways & Limitations

    Because model confidence does not reveal encoding depth, closing the popularity gap requires an external signal, with Wikidata effective for entity-centric factual QA.

  • Takeaways & Limitations

    AdaPop depends on external popularity proxies, whose quality may limit benefits outside entity-centric factual QA.

Abstract

from arXiv · show

Popular facts are memorised more deeply during pretraining and resist removal longer than rare ones, yet existing LLM unlearning methods apply uniform gradient pressure regardless of training-data frequency. We propose the AdaPop (Adaptive Popularity) method, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy (e.g., Wikidata sitelinks, LLM-as-Judge), and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch. Across three model families and two benchmarks, AdaPop leaks ~5x less forgotten content than competing methods under paraphrased queries and ~1.6x less under adversarial reformulations. We support our analysis with internal metrics: under our method, forget-set hidden states move further from the pre-unlearning model's states than under other methods, while retain-set representations remain close.

1 Introduction

AdaPop addresses the popularity gap in machine unlearning, where popular facts resist removal while rare facts are over-erased under uniform gradient pressure. It uses external popularity-aware token reweighting and dual-ascent retain control, achieving stronger paraphrase and adversarial-query erasure while preserving general capabilities.

  • Motivation: Machine unlearning removes targeted knowledge without full retraining, using forget-set removal and a held-out retain set to measure preserved capability.It addresses privacy, safety, and compliance risks from memorized pretraining data.
  • Problem: Uniform gradient objectives create a popularity gap: rare facts are over-erased, while popular facts remain recoverable under paraphrase or adversarial prompts.Facts encountered more often during pretraining are more deeply encoded and harder to remove.
  • Method: AdaPop conditions per-fact token-level ascent signals on external popularity through a power-law exponent, directly targeting the popularity gap.Popularity proxies include Wikidata-based scores and LLM-as-a-judge estimates.
  • Method: An epoch-level dual-ascent controller adjusts the retain penalty as a Lagrange multiplier in response to observed retain-loss drift.This removes per-dataset hyperparameter search and prevents retain collapse under increasing forget pressure.
  • Results: Among methods that do not collapse, AdaPop attains the lowest adversarial-query scores on all three models and the best paraphrase scores on two of three.General capabilities remain within 0.05 MMLU of the pre-unlearning checkpoint, while token rankings and hidden states indicate internal answer disruption.

2 Related Work

Prior unlearning methods trade off forgetting strength against linguistic coherence, retention, or convergence, while relying mainly on model-internal signals. Related work also shows that popular facts are harder to remove, motivating AdaPop’s external popularity calibration and adaptive method.

  • Gradient-based unlearning: Ascent-based methods forget aggressively but require retain regularisation, whereas preference-based methods are gentler and converge more slowly on deeply memorised facts.Gradient Ascent can collapse linguistic coherence when unconstrained.
  • Gradient-based unlearning: None of the four methods uses an external signal; their unlearning pressure follows the model’s current output distribution rather than parametric encoding depth.Inference-time output interventions can also leave parametric knowledge intact.
  • Popularity and memorisation: Frequent facts are encoded more linearly and resist gradient-based removal, while long-tail facts are recalled unreliably.This frequency-linked asymmetry has been observed in both vision and LLM unlearning settings.
  • Popularity and memorisation: High-frequency facts are often not unlearned or only superficially forgotten, remaining extractable through adversarial queries.Wikipedia-based fact salience also predicts unlearning difficulty across existing methods in DUET.
  • External calibration: Existing popularity-gap studies provide benchmarks but no training method to close the gap, motivating an adaptive popularity method.Local reweighting methods use model predictive probabilities, while FaLW additionally requires an unavailable unseen same-class reference distribution for fact-level unlearning.

3 Adaptive Popularity Method

AdaPop calibrates forgetting pressure to each fact’s popularity while preserving held-out knowledge under a constrained retain-drift objective. It combines popularity- and confidence-based token weighting with an epoch-level dual-ascent controller that automatically adjusts the retain penalty.

  • Popularity-calibrated forgetting: AdaPop reweights per-token forgetting signals with a popularity-derived power-law exponent, addressing the different erasure and collateral-damage regimes of popular and rare facts.The exponent is computed from an external popularity proxy, while token confidence supplies a local model signal.
  • Problem setup: Each forget-set question-answer pair receives a proxy popularity score, while the disjoint retain set contains held-out knowledge the model must preserve.The proxy can include Wikidata sitelinks.
  • Popularity-calibrated forgetting: More popular facts receive stronger pressure-sustaining weighting because their continuous exponent decreases with popularity and amplifies gradient mass as tokens are erased.The power-law mapping also compresses the proxy’s wide dynamic range into a bounded exponent.
  • Token weighting: The per-token gradient magnitude scales as p_i,t^βi, making β=1 the crossover between vanishing and growing gradients and recovering WGA at β=1.Above β=1, gradients vanish as token probability approaches zero; below β=1, they grow.
  • Adaptive constraint control: The dual-ascent controller updates the retain penalty once per epoch, increasing it for positive retain-loss drift above ε and decaying it when the constraint is satisfied.Epoch-level updates suppress oscillations from noisy per-batch retain losses and govern the trade-off between popularity-calibrated forgetting and retain preservation.

4 Experimental Setup

The experiments evaluate AdaPop across three 7–8B instruction-tuned model families, two knowledge-unlearning benchmarks, and established baselines. They assess both output behavior and internal representation changes to distinguish genuine forgetting from surface suppression.

  • Baselines and protocol: AdaPop is compared with GA, GD, NPO, and WGA, while results for 11 additional methods appear in Appendix H.Main results use lr = 10^-4, and ROUGE-L is reported as mean ± std over three random seeds.
  • Models and training: Experiments use Llama-3.1-8B-Instruct, Gemma-7B-it, and Qwen2.5-7B-Instruct with LoRA fine-tuning at r=32 and αLoRA=64.Full fine-tuning is excluded because it either fails to converge on popular facts or causes catastrophic forgetting.
  • Datasets: DUET spans Wikidata scores of 69–3,763 with median 1,090, whereas RWKU spans 0–704 with median 130, providing differing popularity variance.DUET uses the combined rare-and-popular forget subset and a 500-sample fast retain set; RWKU popularity is computed from Wikidata sitelink counts.
  • Evaluation dimensions: Evaluation covers output-level behavior and internal representation change because correct answers may be suppressed while remaining retrievable under paraphrase or adversarial prompting.All internal metrics compare the unlearned model with the same checkpoint before unlearning.
  • Metrics: Output metrics include ROUGE-L Recall and Cosine Similarity, while internal metrics include ∆LP, ∆Rank, Hid.Cos, and KL.Lower forget-set output metrics and higher retain-set values indicate effective unlearning; low forget-set Hid.Cos and high KL indicate deeper disruption.

5 Results

Across three architectures and two benchmarks, AdaPop delivers the strongest stable forgetting while preserving retention, and remains robust to paraphrased and adversarial queries. Internal metrics indicate deeper forget-side representation changes without additional retain-side disruption, while general capabilities remain preserved.

  • Output-level forgetting and retention: AdaPop has the lowest forget Cosine Similarity in all six model-benchmark cells and the lowest forget ROUGE-L in five of six, while mean retain ROUGE-L is at least 0.855.The exception is Llama/DUET, where WGA’s forget ROUGE-L is marginally lower.
  • Output-level forgetting and retention: AdaPop keeps the lowest forget ROUGE-L in every cell across eleven additional methods, while distributional and preference-based alternatives either under-erase or collapse.FLAT and CE-Collapse, whereas preference-based variants preserve retain quality but under-erase popular facts.
  • Robustness to paraphrase and adversarial queries: AdaPop is lowest on RWKU adversarial attacks across all three models and has a larger margin over WGA there than under paraphrase; on paraphrase, it is lowest on Qwen and Gemma.On Llama, AdaPop and WGA are within seed noise under paraphrase.
  • Internal representation analysis: AdaPop has the lowest Hid.Cos and highest ∆Rank on both forget splits among stable methods, while all three methods are indistinguishable on retain splits.WGA’s model-averaged DUET forget ∆Rank is negative, consistent with surface suppression rather than genuine erasure.
  • General capability preservation: NPO, WGA, and AdaPop all remain within 0.05 MMLU of the pre-unlearning model, while WGA matches or exceeds HellaSwag and AdaPop does so on Llama and Qwen.AdaPop has a small HellaSwag drop on Gemma; GA collapses general capability and GD has the largest HellaSwag drop among non-collapsing methods.
  • Popularity proxy ablation: At matched forgetting, Wikidata-based popularity avoids over-erasing rare facts, whereas LLM and corpus-frequency signals cost up to 0.13 retain in pressure-sustaining runs.Only the Wikidata score keeps rare facts above β = 1 and preserves the other popularity tier at 10^-3.

6 Conclusion

AdaPop combines an external popularity-derived per-fact exponent with dual-ascent control to improve unlearning across models and benchmarks. The conclusion argues that external difficulty signals are necessary and could extend the method beyond Wikidata-covered factual knowledge.

  • Method: AdaPop derives its unlearning exponent from an external popularity signal and uses dual-ascent control to maintain the forget-retain balance without per-dataset tuning.The approach uses an external proxy rather than the model’s own confidence.
  • Results: AdaPop achieves the lowest adversarial-query scores among stable methods on all three models and the best paraphrase scores on two of three.These results span three model families and two benchmarks.
  • Results: ∆Rank and Hid.Cos indicate parametric disruption rather than surface suppression under AdaPop.The conclusion cites these internal metrics as evidence of the nature of the model changes.
  • Interpretation: Model confidence does not reveal how deeply a fact is encoded, so confidence-calibrated methods can suppress surface answers without closing the popularity gap.The conclusion identifies external signals as necessary; Wikidata provides a negligible-cost proxy for entity-centric factual QA, with stable results across proxy sources.
  • Future directions: Future extensions could replace popularity with per-fact difficulty estimates for procedural, creative, or code knowledge, or with criteria such as harm severity.The conclusion notes that β_i is a per-example schedule over the ascent signal and that Wikidata sitelinks do not apply to every domain.

Limitations

AdaPop depends on an external popularity proxy whose usefulness varies by domain and requires calibration to the exponent coefficients’ scale. Evaluation is limited to LoRA fine-tuning, leaving full fine-tuning for future work.

  • External popularity proxy: AdaPop requires per-sample popularity scores, but Wikidata is unavailable for procedural or creative knowledge.Wikidata was prioritized for fast computation and covers entity-centric factual QA.
  • External popularity proxy: 84% of binary labels agree between Wikidata annotation and LLM-based labelling, but approximation quality may limit benefits outside entity-centric factual QA.The popularity-dependent exponent may be less useful in domains far from entity-centric factual QA.
  • Calibration and evaluation scope: The exponent coefficients a and b were derived from the DUET score distribution, so proxies on different scales require rescaling or recalibration.Without calibration, the corpus-frequency arm spans four orders of magnitude, pins β at βmin for most popular facts, and lowers retention.
  • LoRA-only evaluation: All experiments use LoRA fine-tuning, while full fine-tuning is left to future work because it requires different hyperparameters and exhaustive search is computationally expensive.Prior work cited in the passage reports that full fine-tuning is less effective and may fail to converge or induce catastrophic forgetting.

Ethics Statement … B.2 Anchor Points and Parameter Derivation

The paper reports an ethics framework for public, non-sensitive data and author-led research, then specifies AdaPop’s popularity-adaptive exponent, gradient regimes, anchor derivation, and implementation parameters.

  • Ethics Statement: The data come from public sources, exclude sensitive attributes, and involve no human subjects; ChatGPT provided only minor language and grammatical edits.The authors describe machine unlearning as supporting AI safety and data governance, while retaining responsibility for research design, analysis, and interpretation.
  • A AdaPop Hyperparameters: AdaPop uses α0 = 0.5, ε = 0.1, ηλ = 0.1, λmax = 5.0, a=58.7, b=0.796, with βi clipped to [0.05, 2.0].The coefficients a and b are analytically derived from Appendix B anchors; sensitivity analyses perturb a and b by ±20% and separately vary learning rate.
  • B Derivation of AdaPop Popularity Parameters: The mapping βi = a·si^−b is monotonically decreasing, assigning smaller βi to more popular facts and thereby stronger pressure-sustaining weighting.Popularity scores spanning several orders of magnitude are mapped into a bounded gradient-weighting interval using closed-form parameters derived from two anchor points.
  • B.1 The Three Gradient Regimes of β: The effective per-token gradient magnitude depends on p_i,t^(β_i−1), while stop-gradient weighting ensures backward flow occurs only through the token loss.The dataset-level forget loss sums per-fact contributions, normalizes by |ΩF|, and flips sign for descent-based optimization.
  • B.1 The Three Gradient Regimes of β: β > 1 is self-limiting: as ascent lowers p_i,t, the gradient weight attenuates and automatically halts erasure of already-forgotten content.This regime corresponds to shallow-encoded facts in AdaPop’s per-fact assignment.
  • B.1 The Three Gradient Regimes of β: β < 1 is pressure-sustaining: as p_i,t decreases, p_i,t^(β_i−1) grows and amplifies updates on tokens being erased; β = 1 gives uniform weighting.As β approaches 0, the weighting approaches 1/p_i,t, recovering vanilla GA’s inverse-confidence behavior; WGA fixes βi ≡1 globally.
  • B.2 Anchor Points and Parameter Derivation: AdaPop anchors β(sr) = 1.5 for rare facts and β(sp) = 0.1 for popular facts, placing them in self-limiting and pressure-sustaining regimes, respectively.The representative popular-to-rare score ratio spans at least one order of magnitude; the magnitudes are design choices, and alternative proxies can use their own anchors.

B.3 Robustness of the Derivation … E LLM-as-Judge Popularity Proxy

AdaPop’s derivation is robust to approximate popularity calibration, and its analytically derived coefficients remain effective under perturbation. An LLM-as-Judge proxy provides a practical alternative to Wikidata, achieving comparable primary-rate unlearning despite noisier rankings and lower stability at other rates.

  • B.3 Robustness of the Derivation: ±20% anchor perturbations shift b by less than 0.1, while empirical performance remains stable across the resulting (a, b) region.The derivation therefore needs only order-of-magnitude rare/popular anchor estimates, not precise proxy distributions.
  • B.3 Robustness of the Derivation: Continuous proxies spanning at least one order of magnitude are required; binary or coarse thresholds collapse distinct memorisation depths into one β value.Wikidata satisfies these requirements on the reported benchmarks.
  • B.3 Robustness of the Derivation: 84% agreement with Wikidata on the rare/popular split accompanies nearly identical AdaPop performance from the LLM-judged proxy at the primary learning rate.This supports substituting an LLM-as-Judge signal when Wikidata metadata is unavailable.
  • C AdaPop Coefficient Sensitivity: The (−a, +b) configuration achieves strongest forgetting and weakest retention simultaneously by moving farther along the same Pareto frontier, not by becoming unstable.Its signed changes are ∆F = −0.021 and ∆R = −0.041.
  • D LLM-as-Judge Annotation Protocol: Approximately 2 000 facts were annotated with three seeds, requiring ≈9 hours via openai/gpt-oss-20b, while Wikidata sitelinks for the same samples took ≈12 minutes.Facts were scored as triples in batches of 8, with ≈540 tokens per batch.
  • E LLM-as-Judge Popularity Proxy: The LLM-judge proxy reaches 84% split agreement and Spearman correlation 0.596 with Wikidata, but produces noisier runs and a less stable three-tier distribution.At lr = 10−4, the proxies forget comparably; at lr = 10−5, the LLM proxy yields holdout ROUGE-L 0.904 versus 0.923 at effectively equal forgetting 0.370 versus 0.361.
  • E LLM-as-Judge Popularity Proxy: At lr = 10−3, both proxies lose at least one popularity tier to collapse, making Wikidata the more stable choice while leaving the LLM proxy viable at the primary rate.Discordant cases reflect different signals: Wikidata captures deep memorisation for well-linked entities, whereas the judge flags salient attributes of less-cited entities.

F Proxy Validation and Noise Robustness · G Qualitative Examples of Unlearning Generations

AdaPop’s popularity proxies track measured corpus frequency and remain effective under substantial signal corruption, while its controller preserves retain performance. Qualitative generations show less collapse for AdaPop than for several competing methods, though other methods sometimes produce factual or partially coherent outputs.

  • F Proxy Validation and Noise Robustness: Pearson 0.970: Wikidata popularity tracks measured Pile-train corpus frequency almost linearly in log space.Counts cover each fact’s subject and object in Pile-train (383B tokens) using infini-gram.
  • F Proxy Validation and Noise Robustness: Spearman 0.596: the two proxies agree with each other, assigning the same rare/popular label to 84% of facts.The agreement is reported on DUET against measured corpus frequency.
  • F Proxy Validation and Noise Robustness: Retain holds at 0.92 under full label inversion, as dual ascent raises α when retain degrades and pulls training toward the non-adaptive baseline.Clipping β_i to [0.05, 2.0] and near-uniform weighting when β_i ≈1 also limit corrupted-score influence.
  • F Proxy Validation and Noise Robustness: 29.0 minutes: AdaPop training nearly matches WGA at 29.4 minutes and is faster than NPO at 40.0 minutes on matched Llama/DUET hardware.The controller also removes per-dataset search over α, lowering total compute to a deployed checkpoint versus a hand-tuned baseline.
  • G Qualitative Examples of Unlearning Generations: GA and GD repeatedly produce token repetition or linguistic collapse, including “dises dises,” “solidar emphat emphat,” and “is is is.”These outputs are marked with [collapsed loop].
  • G Qualitative Examples of Unlearning Generations: AdaPop generates “Tanzania Dodoma Region” and a coherent Georgia-related response rather than the repeated-token collapses shown for GA and GD.The Georgia response states that Sannake is not among Georgia’s regions and identifies Tbilisi as the capital.
  • G Qualitative Examples of Unlearning Generations: NPO produces geographic outputs ranging from “Lublin Voivodeship” to a longer Gmina Sanniki description, while WGA includes collapsed loops and “Lublin Voievodeship.”The examples include “Lower vhs vhs vhs” and “Maz Pi Pi Pi” for WGA.

H Comparison with Additional Unlearning Methods

The comparison evaluates eleven additional unlearning methods under identical LoRA settings. AdaPop achieves the lowest stable forget ROUGE-L on DUET and RWKU while maintaining retain ROUGE-L ≥0.855 across all model-benchmark combinations.

  • Evaluation setup: Eleven additional methods are evaluated, including UNDIAL, RMU, PDU, NPO-SAM, SimNPO, SatImp, Adaptive RMU, AltPO, FLAT, TPO, and CE-U.All use r=32, αLoRA=64, lr = 10−4, and 5 epochs.
  • Comparison patterns: 0.734–0.884 forget ROUGE-L: UNDIAL and RMU remain barely below the pre-unlearning model on DUET.On RWKU, Llama obtains 0.568 with UNDIAL and 0.801 with RMU, versus 0.078 with AdaPop.
  • Comparison patterns: 0.476–0.677 forget ROUGE-L: PDU reduces forgetting on DUET but incurs a measurable retain penalty.The supplied passage does not provide the corresponding retain value.
  • AdaPop results: 0.043 and 0.016: AdaPop records the lowest forget ROUGE-L among stable methods for Llama/DUET and Qwen/RWKU, respectively.AdaPop maintains retain ROUGE-L ≥0.855 on every model-benchmark combination.

I General Capability Preservation: Extended Discussion … M.1 Dynamic Retain Coefficient as Constrained Optimisation

AdaPop preserves broad capabilities while improving forget-set erasure across models and robustness tests. Its popularity-weighted ascent objective is paired with an epoch-level dual controller that stabilizes retain quality without claiming formal convergence.

  • I General Capability Preservation: Extended Discussion: GA collapses MMLU and HellaSwag, while GD preserves MMLU but substantially degrades HellaSwag on Llama and Gemma.GA scores 0.23–0.25 on MMLU versus 0.47–0.71 before unlearning; GD changes HellaSwag from 0.73 to 0.59 on Llama and 0.64 to 0.37 on Gemma.
  • I General Capability Preservation: Extended Discussion: NPO, WGA, and AdaPop remain within 0.05 MMLU of pre-unlearning values, while WGA and AdaPop generally preserve or improve HellaSwag.AdaPop drops to 0.62 versus 0.64 on Gemma, while matching or exceeding the starting HellaSwag score on Llama and Qwen.
  • J Evaluation Metric Definitions: The evaluation defines four internal metrics—∆LP, ∆Rank, Hid.Cos, and KL—computed per example and averaged over each split.∆LP tracks gold-answer log-probability shifts, ∆Rank tracks gold-token rank shifts, Hid.Cos compares pooled hidden states, and KL captures probability-mass redistribution.
  • K Internal Representation Metrics: Figure 4 reports per-model ∆LP, ∆Rank, Hid.Cos, and KL breakdowns at lr = 10−4, complementing the model-averaged Table 6 results.Forgetting favors lower ∆LP and Hid.Cos with higher ∆Rank and KL, while retain-set directions are reversed.
  • L Model-Specific Observations: AdaPop shows strong model-specific forget-retain separation, including Llama DUET forget ROUGE-L 0.939 → 0.043 with retain 0.959 and Qwen RWKU 0.570 → 0.016.On Gemma, AdaPop reaches forget cosine 0.206 on DUET and 0.068 on RWKU, while retain cosine rises to 0.976 and 0.961.
  • M Design Motivation: Gradient-Based Unlearning and the Retain Controller: AdaPop uses ascent because its retain-side damage can be separated from forget aggressiveness through a dedicated dual-ascent retain controller.The controller adjusts retain stability through α, while popularity-dependent βi controls forget aggressiveness.
  • M Design Motivation: Gradient-Based Unlearning and the Retain Controller: WGA has higher robustness ROUGE-L than AdaPop on Qwen and Gemma paraphrases and across all three models under adversarial attacks.Adversarial values are 0.211–0.396 for WGA versus 0.144–0.262 for AdaPop, with the largest gap under adversarial reformulations.
  • M.1 Dynamic Retain Coefficient as Constrained Optimisation: AdaPop treats retain quality as an inequality constraint, updating λ once per epoch and applying it to the popularity-weighted forget objective.λ is clipped to [0, 5.0], initialized at λ0 = 0, and keeps |δk| ≤ ε = 0.1 across the 3×2 grid and learning-rate sweep; formal convergence is not claimed for the non-convex objective.

N Component Ablation: Dual-Ascent Controller vs. Popularity Exponent · O Learning-Rate Sensitivity and Algorithm Stability · P Per-Epoch Unlearning Dynamics

Across ablations and learning-rate sweeps, AdaPop’s dual-ascent controller stabilizes retain quality while its popularity exponent enables deeper forgetting of popular facts. Baselines commonly collapse, oscillate, under-erase, or require model-specific tuning across epochs and learning rates.

  • N Component Ablation: Dual-Ascent Controller vs. Popularity Exponent: At lr=2 × 10−4, fixed α drops rare-fact retain to 0.769, whereas both dyn-α variants maintain ≥0.927.At lr=5 × 10−3 on popular facts, fixed-α variants fall to ≤0.634, while the controller keeps retain above 0.72.
  • N Component Ablation: Dual-Ascent Controller vs. Popularity Exponent: 0.724 versus 0.683 retain at comparable popular-fact forget ROUGE-L ≈0.02 shows the popularity exponent improves the controller’s Pareto frontier.For popular facts, βi < 1 concentrates updates on tokens that still resist erasure, enabling targeted forgetting with less retain pressure.
  • N Component Ablation: Dual-Ascent Controller vs. Popularity Exponent: Only full AdaPop (α=dyn, β=dyn) achieves low forget and high retain simultaneously across the tested learning-rate range.The controller alone cannot deeply forget popular facts, while the exponent alone lacks a viable operating point.
  • O Learning-Rate Sensitivity and Algorithm Stability: GA drives forget and retain ROUGE-L to zero at every tested rate, while GD loses retain at any rate achieving meaningful forgetting.Both methods are unusable above 10−6 on RWKU.
  • O Learning-Rate Sensitivity and Algorithm Stability: AdaPop is the only method whose forget quality improves monotonically with learning rate while retain remains controlled across both benchmarks and all models.Forget cosine decreases alongside ROUGE-L, confirming representational rather than surface-level erasure.
  • O Learning-Rate Sensitivity and Algorithm Stability: Rare facts reach near-zero forget at lr ≈5×10−5, but popular facts require lr ≈10−4, evidencing an intrinsic popularity gap.WGA’s popular-fact forget cosine remains above AdaPop’s, while NPO’s popular-tier forget curves stay flat.
  • P Per-Epoch Unlearning Dynamics: GA reaches forget/retain 0.000/0.000 after epoch 1, while GD reaches forget 0.012 but retain crashes to 0.345 before later recovery.GD’s forget oscillates nonmonotonically thereafter, making careful early stopping necessary.
  • P Per-Epoch Unlearning Dynamics: WGA is the only baseline with smooth monotonic forgetting, dropping from 0.122 at epoch 1 to 0.020 at epoch 10 while retain stays ≥0.945.GA, GD, and PDU collapse or destabilize; NPO oscillates; RMU and UNDIAL never erase entity-level facts.
Loading 2608.14229v1…