PaperlayerView exact revision on arXiv

Source-linked AI summary

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

Ravi Ranjan, Olivera Kotevska, Agoritsa Polyzou

arXiv:2609.10439v1cs.LGcs.AI

TL;DR

LLM unlearning seeks to remove sensitive or undesirable memorized content without sacrificing retained capabilities, but broad updates can degrade utility and fail after quantization. FOM-UL selects layers with high forget-set influence and low retain-set sensitivity, concentrating updates to improve targeted forgetting and deployment robustness. Across diverse evaluations, it suppresses residual memorization while preserving utility, including under low-bit quantization, without formally guaranteeing erasure.

  • Problem

    Existing unlearning methods can use broad or fixed updates that degrade utility and allow forgotten knowledge to partially re-emerge after post-training quantization.

  • Method

    FOM-UL scores transformer layers by forget-set influence relative to retain-set sensitivity and restricts updates to a small selected subset.

  • Results

    Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization while preserving retain-set utility, including a 1.22% residual memorization rate under 4-bit quantization on Llama-3.

  • Takeaways & Limitations

    Concentrated layer updates provide an efficient empirical path toward quantization- and adversarial-recovery-resistant unlearning while preserving much of the model unchanged.

  • Takeaways & Limitations

    Effectiveness depends on layer-attribution reliability, and entangled or redundant knowledge may require more layers or iterations, increasing compute and utility trade-offs.

Abstract

from arXiv · show

Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge. We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding. Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains stronger memorization suppression and utility preservation than competing methods, and adversarial prompt evaluations show lower recovery of forgotten content. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.

1 Introduction

LLM unlearning must remove targeted knowledge while preserving general capabilities, but broad updates can cause utility loss and quantization-driven recovery. FOM-UL selectively updates influential layers to improve forgetting, efficiency, and robustness.

  • LLM memorization of sensitive, copyrighted, or undesirable content creates legal, ethical, security, and regulatory risks.
  • Full retraining is costly, while coupled transformer representations make targeted removal difficult and can cause utility degradation or catastrophic forgetting.
  • FOM-UL selects transformer layers with high forget-set influence and comparatively low retain-set sensitivity.
  • Iterative layer-budget expansion adds layers only when forgetting criteria remain unmet, improving the erasure–utility trade-off.
  • Restricting updates to selected layers reduces trainable parameters, memory use, and runtime while remaining compatible with GA, NPO, and KLD-style losses.
  • FOM-UL is evaluated under 8-bit and 4-bit post-training quantization to test whether concentrated updates reduce residual memorization relative to global or fixed-selection baselines.

2 Preliminary and Related Work

LLM unlearning methods trade off targeted removal, retained utility, and computational cost, while low-bit quantization can restore suppressed behavior. Existing global, partial, and parameter-efficient approaches each have limitations in robust deployment.

  • Machine unlearning selectively removes undesirable knowledge while aiming to preserve overall model performance.
  • Global unlearning updates the full parameter space, potentially causing broad utility degradation and higher computational cost.
  • Partial unlearning limits updates to selected components but can incompletely forget knowledge distributed across multiple network parts.
  • Post-training quantization maps full-precision parameters to low-bit representations, improving inference efficiency but potentially weakening unlearning updates.
  • When edited coordinates remain in the same quantization bin, the quantized model can become closer to the quantized original and allow suppressed knowledge to re-emerge.
  • Existing quantization-based, editing, and parameter-efficient methods retain significant limitations for robust unlearning scenarios.

3 Proposed Method: FOM-UL

FOM-UL selectively identifies and updates transformer layers that strongly affect forgetting while limiting retain-set disruption. Its pipeline combines layer attribution, significance-based selection, targeted losses, and iterative expansion to approximate retraining while preserving utility and improving quantization resilience.

  • Scope and motivation: The framework aims to approximate retraining on the forget set while preserving retain-set utility and making parameter shifts more resilient to quantization-induced reversal.The paper explicitly does not claim exact erasure.
  • Method overview: FOM-UL targets layers most responsible for encoding sensitive content, freezing low-attribution layers and updating only high-attribution layers.The pipeline uses layer attribution to create a binary saliency mask before targeted unlearning.
  • Layer selection: The method scores layers by forget-set versus retain-set gradient behavior, prioritizing high forgetting leverage with comparatively low retain interference.Its significance score identifies layers responsive to the forgetting objective but relatively insensitive to the retain objective.
  • Selective update: FOM-UL applies forgetting, mismatch, and retain losses only to selected layers, while layers outside the selected set remain frozen.This concentrates optimization on the candidate layer set rather than updating the full model.
  • Iterative expansion: If forgetting criteria remain unmet, FOM-UL expands the selected layer set incrementally instead of updating a fixed region or the full model from the outset.Expansion continues until the forgetting metric converges or the maximum epoch limit is reached, reducing the risk of early utility loss.

4 Experiments

FOM-UL is evaluated across multiple benchmarks, model architectures, metrics, baselines, and deployment settings. The experiments assess forgetting, retained utility, adversarial recovery, quantization robustness, efficiency, and component contributions.

  • Experimental setup: FOM-UL is evaluated on TOFU, KnowUnDo, and MUSE benchmarks using memorization, privacy leakage, and retain-set utility metrics.The evaluation spans factual forgetting, privacy- and copyright-oriented unlearning, and BOOKS and NEWS settings.
  • Unlearning results: Across GPT-2, Llama-3.2-1B, and Gemma-3-1B, FOM-UL achieves the best overall forgetting-utility balance while keeping retain-set utility close to the vanilla model.The gains remain stable across GA, NPO, and KLD-style base objectives.
  • Unlearning results: Across NEWS, KnowUnDo, and BOOKS, FOM-UL matches or ties the best memorization scores, reduces privacy leakage, and preserves retain-set utility close to the vanilla model.The reported utility pattern indicates that the gains are not due to destructive over-unlearning.
  • Robustness analysis: 11.6% ALR is reported for FOM-UL, compared with 16.5% for SURE+NPO and 19.8% for LUNAR under jailbreak-style recovery prompts.Lower ALR indicates fewer successful recoveries of forgotten knowledge.
  • Efficiency: FOM-UL requires 6–8 GB of GPU memory and 10–30 minutes of runtime while updating only a small subset of parameters.The method is presented as competitive in unlearning performance while avoiding full-model updates.
  • Ablation study: The full FOM-UL configuration balances forgetting, privacy, and utility better than forget-only, retain-only, or mismatch-only components.Forget-only produces high privacy leakage, while retain-only weakens forgetting.

5 Discussion

Discussion results indicate that selective layer updates improve forgetting while limiting utility loss and remain effective under adversarial prompts and low-bit quantization. The method’s efficiency and robustness are linked to concentrated edits in attribution-identified layers.

  • Layer selection: High-Sig(ℓ) layer selection outperforms early-only or over-expanded updates and yields the lowest attack leakage rate under jailbreak-style recovery prompts.The reported validation connects layer choice with both forgetting-utility trade-offs and adversarial recovery resistance.
  • Quantization robustness: FOM-UL consistently reduces residual targeted knowledge with limited utility loss and remains robust under aggressive post-training quantization.Conventional methods are described as exhibiting severe reversals from discretization artifacts.
  • Cross-architecture generality: Across LLaMA-2, LLaMA-3, GPT-2, and Gemma-3, FOM-UL reduces residual memorization and privacy leakage while maintaining strong retained utility.Under 4-bit quantization on Llama-3, the reported residual memorization rate is 1.22%.
  • Efficiency: Selective updates, fewer trainable parameters, and lower memory requirements make FOM-UL practical at scale.The method is described as enabling larger batches and reduced checkpointing.

6 Conclusion

FOM-UL localizes unlearning to layers with high forget-set influence and low retain-set sensitivity, improving forgetting-utility trade-offs, efficiency, and deployment robustness without claiming complete erasure.

  • Method and scope: FOM-UL restricts updates to a small subset of layers identified by high forget-set influence and low retain-set sensitivity.This avoids modifying the full model and reduces unnecessary parameter changes.
  • Main findings: Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization while preserving retain-set utility.The conclusion reports the result relative to strong unlearning baselines.
  • Deployment robustness: Concentrated updates improve robustness under adversarial recovery prompts and low-bit post-training quantization.The conclusion contrasts these updates with diffuse changes that can be weakened or erased.
  • Limitations: The paper leaves formal guarantees of complete erasure to future work.The framework is presented as approximate unlearning rather than formally guaranteed erasure.

Limitations

FOM-UL improves forgetting and quantization robustness by concentrating updates, but its effectiveness depends on reliable layer-attribution signals and does not formally guarantee erasure.

  • Layer-attribution reliability can vary across prompts, domains, and evaluation setups, limiting FOM-UL’s effectiveness.
  • Highly entangled or redundantly encoded knowledge may require more updated layers or additional iterations.These expansions can increase compute and introduce utility trade-offs under adversarial or distribution-shifted conditions.
  • FOM-UL is not a formal guarantee of erasure.
  • The evaluation uses diverse transformer checkpoints, including Llama-2, Llama-3.2, GPT-2, and Gemma-3 1B, with FP32, 8-bit, and 4-bit formats.

B.2 Datasets and Splits

The evaluation follows disjoint forget and retain splits across TOFU, KNOWUNDO, and MUSE-style BOOKS and NEWS settings, with fixed implementation conditions for reproducibility.

  • Disjoint Forget and Retain splits are used across TOFU, KNOWUNDO, and MUSE BOOKS/NEWS evaluations.
  • The BOOKS forget set contains copyrighted Harry Potter text, while retain data includes FanWiki and general-domain text.
  • The NEWS evaluation includes a holdout split reserved for privacy and leakage evaluation and excluded from updates.
  • Experiments use PyTorch, HuggingFace transformers, bitsandbytes quantization, NVIDIA A100 GPUs, released environment details, and fixed splits.

C Evaluation Metrics

FOM-UL evaluates forgetting, privacy, utility, layer selection, and quantization robustness using complementary metrics and a selective-update procedure with iterative expansion.

  • Evaluation Metrics: M1 measures verbatim memorization on the forget set, with lower values indicating more effective removal.
  • Evaluation Metrics: M2 measures generalized forgotten knowledge through QA, with lower values indicating better removal.
  • Evaluation Metrics: M3 measures privacy leakage through membership inference, with values closer to 0 indicating parity with retraining.
  • Evaluation Metrics: M4 measures retained utility on Dretain, where higher values indicate better preservation of benign knowledge and task utility.
  • Evaluation Metrics: FOM-UL targets low M1/M2, M3 ≈0, and high M4, including under post-training quantization stress tests.
  • Layer Selection: Layers are ranked by forget-to-retain significance, and only selected layers receive updates while other layers remain frozen.
  • Quantization Robustness: FOM-UL concentrates updates in high-significance layers, making the forgetting signal less likely to be erased by quantization.

D.5 Empirical Quantization-Bin Crossing Analysis

The bin-crossing analysis measures whether unlearning edits survive quantization and connects layer selection, iterative expansion, and quantization persistence to FOM-UL’s robustness mechanism.

  • Metrics: The Bin-Change Fraction measures the fraction of edited parameters whose quantized values differ after unlearning.Higher BCF indicates that more unlearning edits survive post-training quantization.
  • Metrics: Higher BCF reflects a greater fraction of unlearning-induced parameter changes remaining distinguishable after quantization.
  • Empirical Findings: FOM-UL produces more quantization-surviving edits in selected layers than diffuse global updates.
  • Layer Selection: Selecting layers with high forget effect per retain sensitivity prioritizes forgetting responsiveness while constraining retain disruption.
  • Theoretical Scope: The layer-ranking lemma is justified only under small-step, first-order behavior and a retain-gradient utility constraint, not as global optimality for deep non-convex objectives.
  • Iterative Expansion: Expanding the selected layer set cannot worsen the best attainable forgetting progress under the same retain-stability constraint.
  • Quantization Persistence: Quantization can remove unlearning changes when edited coordinates remain in their original quantization bins.
  • Quantization Persistence: FOM-UL’s localized shifts are more likely to exceed effective quantization steps, improving robustness to quantization-induced recovery.

F Adversarial Robustness Analysis

Adversarial prompts test whether forgotten knowledge can be recovered without changing model parameters. FOM-UL shows the lowest adversarial memorization and leakage while preserving utility, and remains comparatively stable across prompt variants.

  • Adversarial evaluation: Adversarial wrappers test recovery of forgotten knowledge without modifying model parameters.The wrappers are applied only during evaluation.
  • Adversarial evaluation: FOM-UL achieves the lowest adversarial VerMem/KnowMem and leakage rate while retaining the highest clean utility.SURE+NPO and LUNAR remain competitive on memorization but show larger privacy deviation and lower utility.
  • Prompt variation: Recovery increases when prompts depart from the clean template, showing that clean-prompt metrics can understate residual knowledge.The audit considers paraphrased questions and alternate extraction templates in addition to jailbreak prompts.
  • Prompt variation: FOM-UL remains comparatively stable across paraphrase, alternate-template, and jailbreak settings.This pattern suggests reduced prompt-specific hiding rather than suppression limited to the canonical test format.

G Sensitivity Analysis

Sensitivity analyses identify a moderate mid-to-late-layer selection regime that improves forgetting while largely preserving utility. Additional evaluations show stable performance across backbones and explain how concentrated updates withstand quantization.

  • Layer budget and location: Increasing the selected-layer budget from Top-1/Top-2 to Top-4/Top-8 substantially improves forgetting while largely preserving utility.Lower M1/M2 and M3 closer to zero indicate stronger forgetting and lower privacy leakage, while M4 measures retain-set utility.
  • Layer budget and location: Early-layer selection under-performs on forgetting and can degrade utility, while expanding into early layers yields diminishing returns and greater collateral risk.The sensitivity trend supports concentrating updates in mid-to-late layers and expanding only as needed.
  • Cross-model stability: FOM-UL-Full maintains consistently low memorization and privacy-leakage scores with high utility across GPT-2, Llama-3, and Gemma-3.Small standard deviations indicate stable behavior across runs.
  • Cross-model stability: Error bars summarize mean ± standard deviation across repeated trials for VerMem, KnowMem, PrivLeak, and Utility.The evaluation repeats unlearning with different random seeds and plots each metric with vertical ±σ error bars.
  • Quantization stress: Concentrating updates in responsible layers stabilizes forgetting after low-bit quantization while leaving remaining layers intact.The qualitative comparison reports stable refusal, neutral, or corrected responses after int4 quantization with preserved utility relative to broad updates.
Loading 2609.10439v1…