PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song, Jinsong Su

arXiv:2605.18879v3cs.LGcs.AIcs.CL

TL;DR

Large language models retain sensitive information, while existing unlearning methods can be costly or degrade related knowledge and utility. ZeroUnlearn reframes unlearning as targeted knowledge remapping and reports stronger unlearning with preserved general utility.

  • Problem

    LLMs retain private, biased, or outdated information, motivating precise selective knowledge removal for privacy, moderation, and factual updates.

  • Method

    ZeroUnlearn remaps sensitive inputs to a safe target while orthogonalizing their representations through multiplicative parameter updates, with closed-form and gradient-based variants.

  • Results

    Across multiple LLMs and benchmarks, ZeroUnlearn outperformed baselines while maintaining utility; on MCF with Llama-3.1, it reduced unlearning efficacy to 0%.

  • Takeaways & Limitations

    The results support knowledge remapping as an effective approach for targeted few-shot and batch unlearning with a favorable balance between erasure and utility.

  • Takeaways & Limitations

    The theoretically optimal closed-form multi-sample solution is computationally prohibitive because it requires inverting a d^2 × d^2 Kronecker-product matrix.

Abstract

from arXiv · show

Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

1. Introduction

LLMs retain biased, private, and outdated information from extensive web training, making selective knowledge removal important while existing retraining-based approaches remain computationally costly. ZeroUnlearn reframes unlearning as few-shot knowledge remapping through targeted parameter updates that redirect sensitive inputs, remove their representations, and preserve utility.

  • LLMs trained on extensive web corpora inevitably retain biased, private, or outdated information, motivating selective machine unlearning for responsible deployment.
  • Full retraining after excluding knowledge to be forgotten offers an exact solution but incurs prohibitive computational costs for modern LLMs and massive pretraining corpora.
  • Knowledge editing motivates overwriting harmful sensitive information with a neutral target such as “<EOS>” instead of destructively perturbing model weights.
  • ZeroUnlearn reframes unlearning as precise knowledge remapping, redirecting sensitive inputs while minimizing representational similarity to the original knowledge.
  • A closed-form multiplicative update enables efficient one-step few-shot unlearning, while ZeroUnlearn-GD extends the formulation to multi-sample batch unlearning.
  • Experiments across widely used models and benchmarks show that ZeroUnlearn and its variant outperform baselines while maintaining a favorable balance between unlearning efficacy and general utility.

2. Related work

Related work spans knowledge editing, which targets precise and local modifications to factual knowledge, and model unlearning, which removes specific training samples’ influence without costly retraining. Knowledge-editing methods either patch predictions through external modules or directly manipulate parameters, while unlearning commonly uses optimization to suppress undesired outputs.

  • Knowledge Editing: Knowledge editing modifies specific factual knowledge in LLMs with high precision and locality.It is designed for targeted changes rather than broad model retraining.
  • Knowledge Editing: External-memory and auxiliary-module methods intercept and override original predictions for targeted queries without altering core weights.These approaches effectively patch the model for selected queries.
  • Knowledge Editing: Direct parameter-manipulation methods identify layers storing particular knowledge and apply closed-form updates to modify factual associations.This line edits the model through targeted weight changes.
  • Model Unlearning: Model unlearning removes the influence of specific training samples to meet data-protection requirements without costly retraining.The goal is efficient removal of sample-specific effects from the model.
  • Model Unlearning: Optimization-based unlearning often applies gradient ascent on unlearning samples to suppress undesired outputs or behaviors.This formulation treats unlearning as an optimization problem.

3. Background

Machine unlearning is framed as targeted removal of sensitive factual associations using only a forget set, while balancing forget efficacy with utility preservation. ZeroUnlearn’s background further represents knowledge through hidden-state transformations and identifies FFN output-weight editing as the unlearning interface.

  • Unlearning task: Machine unlearning targets specific factual associations or sensitive data in a forget set Df, producing updated parameters θ′ that no longer exhibit knowledge of Df.The setting assumes only Df is available during unlearning, unlike traditional retraining-based paradigms.
  • Unlearning objectives: Effective unlearning requires both neutralizing Df’s influence on outputs and preserving model utility despite having no explicit retain set.Forget efficacy can remap sensitive inputs to non-informative targets such as <EOS> or maximize loss on Df.
  • ZeroUnlearn geometry: ZeroUnlearn geometrically projects the sensitive output mf into a null space, then aligns that projection with a neutral target state mn.The two steps use a projection matrix P followed by optimization toward the neutral representation.
  • Knowledge representation: LLM knowledge is modeled as subject–relation–object triples and, in the feed-forward pathway, as (k, m) pairs where editing Wdown can achieve unlearning.Here k is the activated intermediate representation, m is the MLP output, and W denotes Wdown.

4. Methodology

ZeroUnlearn formulates few-shot LLM unlearning as targeted knowledge re-mapping: sensitive inputs are redirected to a neutral state while general knowledge associations are preserved. Its multiplicative null-space projection yields an efficient closed-form edit, with a gradient-based extension for multi-sample unlearning.

  • Knowledge re-mapping: ZeroUnlearn remaps forget-set keys Kf to a neutral target Mn, such as the “<EOS>” representation, suppressing generation of the original knowledge Mf.The forget objective redirects sensitive inputs to a predefined non-informative or terminal signal rather than merely suppressing outputs.
  • Optimization objective: Its three-term objective enforces orthogonality to the original forget representations, explicit redirection to Mn, and fidelity on general knowledge associations (K0, M0).The utility term is designed to prevent catastrophic forgetting and degradation of fundamental linguistic proficiency.
  • Null-space projection: ZeroUnlearn replaces additive updates with multiplicative editing, setting the updated weight matrix to ˜W = DW and constraining D to the right null space of M⊤f.SVD constructs P = I − VV⊤, an orthogonal projector in that null space, with D reparameterized as D = P˜D.
  • Closed-form solution: The resulting objective includes regularization for stable convergence and admits a closed-form solution balancing targeted erasure, utility preservation, and parameter stability.The solution uses target-key associations and a precision-weighted normalizer based on the key second moment matrix.
  • Implementation: In practice, the edit left-multiplies D∗ with the selected layer’s weight matrix W, while a gradient-based scheme supports multi-sample unlearning without performance degradation.The layer-selection strategy is described in the experimental section, and the algorithm extracts final subject tokens as keys.

5. Few-shot Constraints of ZeroUnlearn and an Alternative Solution

ZeroUnlearn remains effective in few-shot settings because the forget set spans a small subspace, allowing targeted neutralization while preserving most model capacity. For multi-sample unlearning, an additive null-space-constrained formulation yields a closed-form optimum in theory, but ZeroUnlearn-GD avoids its prohibitive computational cost through iterative optimization.

  • Few-shot constraints: With n ≪ d, the forget-set knowledge matrix is inherently low-rank, with rank r ≤ n, so the projection matrix P retains near-maximal rank.P is constructed from the r dominant singular vectors of the forget-set knowledge matrix.
  • Few-shot constraints: Constraining updates to the null space of M_f^⊤ neutralizes forget-set directions while leaving most expressive capacity untouched, preserving general linguistic utility.The retained null space provides d − n degrees of freedom for unlearning.
  • Alternative solution: For multi-sample unlearning, additive editing reparameterizes D_m = ˜D_mP_m so that D_mK_0 = 0, making the utility term vanish identically.P_m projects onto the right null space of K_0^⊤, enforcing the hard utility constraint.
  • Alternative solution: The resulting optimization is a Sylvester equation whose optimal effective update admits a closed-form solution using vectorization and the Kronecker product.The vector solution is reshaped back into the original matrix dimensions.
  • Alternative solution: O(d^6) time and O(d^4) space make direct closed-form computation intractable for modern LLMs, motivating ZeroUnlearn-GD with O(d^2) cost per iteration.Gradient descent avoids explicitly constructing the d^2 × d^2 Kronecker matrix and converges to the global optimum for the convex objective.

6. Experiments

Experiments across few-shot and multiple-unlearning settings show that ZeroUnlearn removes targeted knowledge while better preserving model utility than competing baselines. The method also remains computationally lightweight, with stable runtime and memory demands as forget-set size grows.

  • Experimental Setup: Experiments use Llama-3.2, Llama-3.1, and Qwen-3 with MCF, ZsRE, and MQUAKE, evaluating efficacy, generalization, specificity, and perplexity.Few-shot experiments use ten random seeds with 50 randomly selected samples per seed; multiple unlearning uses 1,000 samples.
  • Unlearning Efficacy: On MCF with Llama-3.1, ZeroUnlearn reduces efficacy to 0%, while ROME retains 24.40% efficacy and AlphaEdit performs worse than ZeroUnlearn.The results indicate complete removal of the targeted sensitive information in this setting.
  • Preserving Model Capabilities: ZeroUnlearn preserves model capabilities better than GA and FT, which exhibit exploded perplexity or collapsed specificity alongside low efficacy.GA has PPL >1000 and FT loses specificity completely on MCF, whereas ZeroUnlearn maintains PPL comparable to the base model.
  • Generalization-Specificity Trade-off: Among methods achieving efficacy near 0, ZeroUnlearn better preserves unrelated knowledge than competing editors, despite a moderate specificity decrease from the base model.ROME and MEMIT retain high specificity because they fail to unlearn effectively, whereas ZeroUnlearn achieves a stronger efficacy-specificity balance.
  • Multiple Unlearning: In multiple unlearning, ZeroUnlearn-GD achieves 0% efficacy on MCF with Llama-3.2 while retaining stable linguistic capabilities and functional neighborhood knowledge.Its specificity decreases as a trade-off for perfect erasure, but it avoids the total locality collapse and model collapse observed with GA and FT.
  • Efficiency: As forget-set size increases from 10 to 1000, average SVD time remains below 0.3 seconds on MCF/ZsRE and below 0.6 seconds on MQUAKE, while memory rises from about 13.8 GB to 14.1 GB.These runtime and memory measurements were obtained on Llama-3.2 for the few-shot closed-form update.

7. Conclusion

ZeroUnlearn reframes machine unlearning as precise knowledge remapping, using multiplicative updates to project sensitive representations into an orthogonal null space. It provides efficient few-shot updates through a closed-form solution and extends the approach to batch processing with ZeroUnlearn-GD.

  • ZeroUnlearn reframes machine unlearning as a precise knowledge remapping process.
  • Its multiplicative parameter updates project sensitive representations into an orthogonal null space, enabling erasure while minimizing collateral damage to general utility.
  • A closed-form solution enables efficient few-shot updates, while ZeroUnlearn-GD extends the method to batch processing.

Impact Statement … C. Dataset Details

The paper frames ZeroUnlearn as a precise knowledge-erasure approach with potential safety and compliance benefits, evaluates it against optimization-based and editing baselines, and uses datasets covering unlearning efficacy and general utility.

  • Impact Statement: ZeroUnlearn aims to remove toxic content, hallucinations, and private data without retraining, supporting safer and more legally compliant AI systems.The authors report no immediate negative societal consequences requiring specific emphasis.
  • A. Notation: The notation section summarizes the symbols used throughout the paper, with vectors and matrices represented in bold.
  • B. Baselines Details: The comparative evaluation includes optimization-based unlearning methods and state-of-the-art knowledge-editing techniques.
  • B. Baselines Details: GA reverses the training objective by maximizing loss on the forget set, but can damage language modeling and unrelated knowledge.GA is described as effective at erasing specific data traces while risking catastrophic forgetting.
  • B. Baselines Details: FT updates parameters through standard gradient descent to map an input k to a new target output v, such as a target token or empty response.ROME instead applies a rank-one FFN update that maps a subject representation to a desired target vector while minimizing interference with other memories.
  • B. Baselines Details: MEMIT extends ROME to multi-edit settings by distributing updates across Transformer layers and solving a constrained least-squares batch-editing problem.AlphaEdit adds a null-space constraint so updates remain orthogonal to preserved knowledge, reducing side effects on neighborhood knowledge.
  • C. Dataset Details: The datasets separately assess knowledge-unlearning efficacy and general utility, including fundamental reasoning and linguistic capabilities.The general-utility benchmarks test whether unlearning compromises the model’s broader capabilities.

C.1. Benchmarks for Knowledge Unlearning … E. Details for Layer Identification

The paper evaluates unlearning across factual benchmarks, tests preservation of general capabilities, derives closed-form optimization solutions, and analyzes layer-wise causal effects for identifying edit locations.

  • C.1. Benchmarks for Knowledge Unlearning: The forget set uses MCF, ZsRE, and a single-hop adaptation of MQUAKE to evaluate factual unlearning while retaining relevant knowledge.MQUAKE’s multi-hop chains are decomposed into atomic question-answer pairs to avoid confounding from multi-hop reasoning.
  • C.2. Benchmarks for General Utility: General utility is assessed on six diverse tasks spanning broad knowledge, sentiment analysis, semantic matching, linguistic acceptability, and natural language inference.The benchmark suite includes MMLU and selected GLUE tasks: SST, MRPC, RTE, COLA, and NLI.
  • D. Proof of Lemmas: The proof section derives the closed-form solutions in Lemma 4.1 and Lemma 5.1 using standard matrix calculus and Frobenius-norm notation.The supplied derivation begins by defining the mathematical notation used for the two lemmas.
  • D.1. Proof of Lemma 4.1 (ZeroUnlearn): For ZeroUnlearn, the optimization minimizes projected-subspace reconstruction error while regularizing weight changes, under an orthogonal projection matrix P.The derivation computes the gradient, imposes the projected-gradient stationarity condition, and constructs a projected optimizer satisfying the Range(P) constraint.
  • D.2. Proof of Lemma 5.1: For multi-sample unlearning, the convex objective yields a generalized Sylvester equation, which is transformed with vectorization and the Kronecker-product identity into a closed-form solution.The derivation introduces auxiliary matrices Q, H, C, and Z before merging the vectorized terms.
  • E. Details for Layer Identification: Layer identification uses average indirect effect visualizations to examine causal influence across MLP outputs, Attention modules, and hidden states.Figures 5–7 provide component-wise or architecture-specific views of these layer-wise effects.
  • E. Details for Layer Identification: Although early MLP layers often show peak causal influence for the last subject token, editing lower layers can significantly compromise general capabilities.This observation motivates practical layer-selection considerations for Llama-3.1 and Llama-3.2.

E.1. Ablation Study … F.2. Multiple unlearning results of Llama-3.2 on MQUAKE

The ablation study shows that adding the neutral target state Mn consistently improves unlearning across model architectures, while the paper reports few-shot and multiple-unlearning results across MCF, ZsRE, and MQUAKE datasets.

  • E.1. Ablation Study: E.1. Ablation Study: ZeroUnlearn compares its full method with a “w/o Mn” variant using only null-space projection on ZsRE across Llama-3.2, Llama-3.1, and Qwen-3.The ablation isolates the contribution of the neutral target state Mn.
  • E.1. Ablation Study: E.1. Ablation Study: Including Mn consistently reduces Efficacy (Eff.) and Generalization (Gen.) scores, for which lower values indicate better unlearning.The passage attributes this improvement to guiding outputs toward a neutral value rather than relying solely on null-space projection.
  • E.1. Ablation Study: E.1. Ablation Study: On Llama-3.1, the Efficacy score improves from 36.02 to 32.67 when Mn is included.This is the reported example of the neutral target state's effect.
  • F. Complete Results: F. Complete Results: The paper includes few-shot unlearning results for ZeroUnlearn on the MCF dataset.These results are presented in Table 5.
  • F.1. Few-shot unlearning results of Qwen-3: F.1. Few-shot unlearning results of Qwen-3: The paper includes few-shot unlearning results for ZeroUnlearn on the ZsRE dataset.These results are presented in Table 6.
  • F.1. Few-shot unlearning results of Qwen-3: F.1. Few-shot unlearning results of Qwen-3: The paper includes few-shot unlearning results for ZeroUnlearn on the MQUAKE dataset.These results are presented in Table 7.
  • F.2. Multiple unlearning results of Llama-3.2 on MQUAKE: F.2. Multiple unlearning results of Llama-3.2 on MQUAKE: The paper reports multiple-unlearning results for ZeroUnlearn on the MQUAKE dataset.These results are presented in Table 8.

F.3. Multiple unlearning results of Llama-3.1 · F.4. Multiple unlearning results of Qwen-3 · G. Complete PCA Visualization

The paper reports multiple unlearning results for Llama-3.1 and Qwen-3, followed by complete PCA visualizations of MLP representation shifts across the evaluated models.

  • F.3. Multiple unlearning results of Llama-3.1: Table 9 presents multiple unlearning results for Llama-3.1.
  • F.3. Multiple unlearning results of Llama-3.1: The Llama-3.1 results are reported as a dedicated evaluation table.
  • F.4. Multiple unlearning results of Qwen-3: Table 10 presents multiple unlearning results for Qwen-3.
  • F.4. Multiple unlearning results of Qwen-3: The Qwen-3 results are reported as a dedicated evaluation table.
  • G. Complete PCA Visualization: The paper provides full PCA visualizations of MLP representation shifts at the critical editing layers for all evaluated models.Figures 8, 9, and 10 cover Llama-3.1, Llama-3.2, and Qwen-3, respectively.
  • G. Complete PCA Visualization: Figure 8 visualizes MLP representation shifts at Layer 19 of Llama-3.1.
  • G. Complete PCA Visualization: Figure 9 visualizes MLP representation shifts at Layer 16 of Llama-3.2.
  • G. Complete PCA Visualization: Figure 10 visualizes MLP representation shifts at Layer 9 of Qwen3-4B.

H. Additional Experiments … H.4. Further Exploration of Neutral Target Mn

The appendix reports additional Llama-3.2 experiments covering RWKU, NPO comparisons, sample-size sensitivity, and alternative neutral target states. These evaluations examine unlearning performance, utility, and implementation choices across several datasets and conditions.

  • H. Additional Experiments: The supplementary experiments were conducted on Llama-3.2 and include RWKU, NPO variants, sample-size sensitivity, and neutral-target analyses.ZeroUnlearn and its baselines update only the down-projection matrices of three FFN layers.
  • H.2. Performance of NPO on Knowledge Editing Datasets: For fair NPO comparisons, the experiments implement both a three-layer NPO variant and an all-layer NPO-full variant.This design matches the proposed method for NPO while separately evaluating broader fine-tuning.
  • H.1. Evaluation on RWKU: RWKU experiments compare ZeroUnlearn with GA, FT, ROME, MEMIT, AlphaEdit, NPO, and NPO-full across three difficulty levels.The results are reported separately for Levels 1, 2, and 3.
  • H.2. Performance of NPO on Knowledge Editing Datasets: Knowledge-editing experiments compare NPO variants with the proposed method on the MCF, ZsRE, and MQUAKE datasets.NPO fine-tunes the same three layers as ZeroUnlearn, whereas NPO-full fine-tunes all layers.
  • H.3. Sample Size Sensitivity of ZeroUnlearn: Sample-size sensitivity experiments vary the number of forgotten samples and report corresponding unlearning and utility metrics.The evaluated conditions include MCF and ZsRE, with results presented under different sample sizes.
  • H.4. Further Exploration of Neutral Target Mn: Neutral-target experiments evaluate ZeroUnlearn with several Mn choices, including <EOS>, “I don’t know.”, and “Hello.”.Results are reported on MCF, ZsRE, and MQUAKE under different Mn settings.
Loading 2605.18879v3…