Source-linked AI summary
Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality
Xunlei Chen, Qirui Ye, Yuang Li, Yi Gong, Zhaokun Wang, Wenyi Li, Shiyao Guo, Jinyu Guo
TL;DR
Precise LLM unlearning is difficult because sequence- and token-level penalties can damage structure or suppress benign knowledge while leaving contextual retrieval routes unmodeled. ADU addresses this by decoupling preplan-to-anchor attention pathways, achieving the strongest aggregate forgetting–retention trade-off among evaluated baselines while preserving high utility.
Problem
Sequence- and token-level unlearning methods do not model context-dependent retrieval paths, risking linguistic damage and suppression of benign knowledge.
Method
ADU fixes preplan–anchor paths under the original model and trains attention-projection adapters to suppress them while preserving local attention and retain-set language modeling.
Results
ADU achieves the strongest aggregate forgetting–retention trade-off among evaluated baselines and preserves 87–98% of model utility, averaging 92.9% versus 81.9% for baselines.
Takeaways & Limitations
Contextual pathway suppression provides a more targeted alternative to sequence or token erasure while limiting excessive forgetting and utility degradation.
Abstract
from arXiv · showhide
Large language models (LLMs) require effective unlearning to address privacy regulations and safety concerns. However, achieving precise forgetting without compromising general utility remains challenging. Existing sequence- and token-level methods penalize target outputs without modeling their context-dependent retrieval paths, which can disrupt linguistic structure or suppress benign knowledge. We present ADU, a fine-grained, training-based framework that shifts unlearning from token erasure to contextual attention-pathway decoupling. Exploiting the functional distinction between local and global attention heads, ADU identifies preplan positions that retrieve persistent sensitive anchors and fixes their candidate paths under the original model. It then trains attention-projection adapters to suppress attention mass along these paths while preserving local-attention structure and retain-set language modeling. Post-training activation exchange tests whether the modified attention-output module transmits the learned forgetting effect. ADU achieves the strongest aggregate performance among evaluated baselines on the TOFU and WMDP benchmarks, including a Forget Quality of (0.93) on TOFU. It preserves 87--98% of model utility (92.9% on average versus 81.9% for baselines) while reducing side effects in benign contexts.
Introduction
The paper reframes LLM unlearning as contextual pathway decoupling: suppressing retrieval routes from preplan positions to sensitive anchors rather than erasing sequences or tokens. ADU uses local–global attention specialization to target these routes while preserving language modeling and local-attention structure.
- Motivation: LLM unlearning must reduce access to sensitive or unsafe knowledge while preserving unrelated capabilities.The motivation includes privacy regulation and the need to modify deployed models themselves.
- Limitations of Existing Methods: Sequence-level penalties can damage linguistic structure, while token-level penalties may suppress benign occurrences of context-sensitive entities.Both approaches define forgetting over sequences or tokens rather than the computation that retrieves them.
- Method: ADU fixes candidate preplan–anchor paths under the original model and trains attention-projection adapters to suppress their attention mass.Retain-set language modeling and local-attention preservation constrain collateral behavior, while post-training edge-contribution replacement tests causal mediation.
- Results: ADU achieves the strongest aggregate forgetting–retention trade-off among evaluated baselines across TOFU, WMDP, and MUSE-Harry Potter.The reported evaluation includes conditional theoretical guarantees and tests of the causal role of targeted pathways.
- Core Idea: ADU formulates unlearning as contextual pathway decoupling using a preplan–anchor temporal pattern arising from local and global attention heads.Local heads maintain short-range dependencies, whereas global heads retrieve distant persistent anchors.
Related Work
Prior unlearning methods target whole sequences, selected tokens, or static token salience, but these targets can leave contextual retrieval routes intact. ADU instead suppresses context-specific preplan–anchor pathways while retaining local structure and language modeling.
- Sequence-level Unlearning: Sequence-level methods optimize forgetting over complete question–answer sequences, which can produce unstable updates or coarse interventions.The related work contrasts direct loss maximization and preference-regularized objectives.
- Token-level Unlearning: Token-level methods suppress selected logits or attention toward important tokens, but residual semantics, hallucinations, and static salience can limit contextual discrimination.Static token targeting may weaken an entity in both sensitive and benign contexts.
- Pathway Decoupling: ADU identifies recurring preplan–anchor rhythms and fixes candidate paths before training instead of directly erasing sensitive tokens.Attention-projection adapters suppress mass along the selected paths while retain language modeling and local-attention preservation constrain collateral changes.
Method
ADU unlearns by suppressing fixed, context-specific attention pathways from preplan positions to sensitive anchors, while retaining ordinary behavior through language-modeling and local-attention constraints.
- Core method: ADU learns θ1 from θ0, forget data, and retain data using attention-projection adapters, with pathway decoupling active in every forward pass.Counterfactual activation exchange is reserved for post-training analysis of the internal computation mediating forgetting.
- Head specialization: Local and global heads are partitioned using average backward distance, with the bottom and top ρ fractions forming Hloc and Hglob.The partition is computed once from the original model and kept fixed during training.
- Pathway identification: Large retrospective local-attention shifts identify candidate preplans, while persistent global attention identifies candidate anchors within sensitive continuation positions.Positions lacking a valid future continuation window are excluded from anchor selection.
- Pathway construction: The selected path set contains global-head tuples linking anchors s to later preplans t, enforcing the causal order s < t.The path contribution is manipulated before the effective output projection, with head-space and residual-stream representations distinguished.
- Causal validation: Bidirectional activation exchange replaces selected contributions between the original and trained models, recomputes downstream activations, and tests path-specific mediation of forgetting.The recipient model keeps its own output projection and all unpatched computations.
- Training objective: ADU minimizes selected pathway mass while combining forget loss with retain language modeling and local-attention preservation.The backbone remains frozen, and the forget–retain balance is controlled by α.
Experiment
ADU is evaluated across WMDP, MUSE-Harry Potter, and TOFU to measure forgetting, utility retention, copyright unlearning, and boundary preservation. Across these settings, it improves the forgetting–retention balance and preserves neighboring and general knowledge.
- Evaluation Setup: ADU evaluates sensitive-domain, copyright, and boundary-preservation unlearning using WMDP, MUSE-Harry Potter, and TOFU benchmarks.WMDP measures forgetting and retention; MUSE-Harry Potter measures copyright unlearning; TOFU tests boundaries between forget and neighboring retain knowledge.
- WMDP Forgetting and Retention: On Llama3.1-8B-Instruct, ADU reduces WMDP Bio accuracy from 71.86 to 27.32 and Cyber accuracy from 45.37 to 27.97 while retaining MMLU at 62.84 and GSM8K at 58.82.On Qwen3-14B, ADU achieves the lowest Bio and Cyber accuracy and the best GSM8K retention among unlearning methods.
- Copyright Unlearning: On MUSE-Harry Potter, ADU achieves the lowest ROUGE-L among unlearning methods while retaining the strongest MMLU performance.NPO obtains lower BLEU, but its MMLU falls to 42.70; ADU retains MMLU at 45.64 and fluency at 3.29.
- Boundary Preservation: On TOFU (10%), ADU achieves TUD R-L of 0.11, TR of 0.96, NEK accuracy of 70.8%, and GEK accuracy of 72.3%.These results indicate stronger preservation of neighboring and general knowledge than the compared methods.
Discussions
The analyses examine ADU’s component requirements, pathway-specific causal effects, hyperparameter trade-offs, and robustness to recovery attacks. They support targeted pathway intervention while showing that retention constraints and selected transition points are important for utility and forgetting.
- Component Ablation: Removing the pathway loss sharply raises forgetting metrics, while removing the retain loss lowers MMLU and GSM8K by 6.52 and 6.72 points.The ablation identifies pathway suppression as important for forgetting and retain language modeling as important for utility.
- Component Ablation: Random heads weaken both forgetting and retention, while removing the sensitive anchor filter harms MMLU and GSM8K by penalizing benign high-APS anchors.Removing RAS-based preplan selection also weakens forgetting and retention, supporting intervention at the transition point before sensitive anchors guide later generation.
- Path-Specific Causal Validation: Removing selected contributions from Base lowers WMDP Avg. from 58.62 to 36.37, whereas a matched-random edge removal reaches 57.09.Replacing ADU’s selected contributions with Base counterparts restores WMDP Avg. from 27.65 to 47.23, while matched-random replacement reaches only 28.82.
- Hyperparameter Sensitivity: The default q = 0.4 achieves FP 27.65 and RP 60.83 as a stable forgetting–retention trade-off knee.Small q misses sensitive pathways, whereas large q includes benign anchors and harms retention; the default α = 0.3 gives the best tested balance.
- Robustness Analysis: Under adaptive recovery attacks, ADU has both the smallest increase and the lowest final accuracy among the compared methods.Under prompt scaffolding, ADU still achieves the lowest attacked accuracy, although ASU has a smaller increase.
Conclusion
The paper frames LLM unlearning as contextual pathway decoupling rather than erasing entire sequences, static tokens, or prompt-level refusals. ADU targets sensitive retrieval routes while preserving retain-set language modeling and local-attention structure.
- ADU identifies preplan–anchor rhythms from local and global attention specialization, fixes candidate retrieval paths under the original model, and suppresses them with attention-projection adapters.The framework preserves retain-set language modeling and local-attention structure during training.
- ADU provides a persistent parameter-level mechanism for context-specific sensitive retrieval while limiting excessive forgetting, utility degradation, and recovery under altered prompts.
- Experiments on WMDP, TOFU, and MUSE-Books demonstrate strong forgetting–retention trade-offs.
- Bidirectional edge-contribution interventions establish that selected pathways causally mediate a substantial portion of sensitive retrieval and the learned forgetting effect.