PaperlayerView exact revision on arXiv

Source-linked AI summary

MAAT: Multi-phase Adapter-Aware Targeted Unlearning

Suryash Yagnik, Shubham Gaur, Saksham Thakur, Vinija Jain, Aman Chadha, Amitava Das

arXiv:2605.30514v1cs.LGcs.CL

TL;DR

Machine-unlearning benchmarks scarcely represent causal Why-type knowledge, leaving important failures difficult to measure. The paper introduces balanced 5WBENCH and the three-phase MAAT framework, which reaches a new forget–retain operating point on Why-type causal knowledge. The authors report this result across balanced 5W evaluation while noting limitations in model and evaluation scope.

  • Problem

    Existing benchmarks contain near-zero Why-type causal questions, so balanced evidence about unlearning causal knowledge is missing.

  • Method

    The paper combines balanced 5WBENCH with MAAT, a three-phase framework that performs structured operations directly on LoRA adapter weights.

  • Results

    MAAT is the first method reported to achieve high forgetting and high retention on Why-type causal knowledge, exceeding 60% FSR and 60% RSR across all five 5W categories on Llama 3.2-3B.

  • Takeaways & Limitations

    Balanced causal evaluation makes failures quantifiable and identifies a new forget–retain operating point that no baseline reaches on Why-type knowledge.

  • Takeaways & Limitations

    Evaluation covers two model families at 3–4B scale and uses LLM-as-a-Judge, with extension to larger models, architectures, and complementary evaluation left for future work.

Abstract

from arXiv · show

Machine unlearning evaluation is structurally skewed: Why-type questions, which probe causal and relational knowledge, comprise less than 0.06% of CounterFact, 0.6% of ZSRE, and less than 1.3% of TOFU, MUSE, and WMDP-Cyber. This near-zero representation means that methods that fail on causal knowledge can score highly in aggregate, and this failure is undetectable without balanced evaluation. We present 5WBENCH, a balanced 5,000-sample benchmark with 1,000 examples per 5W category (Who, What, When, Where, Why), making causal unlearning failures quantifiable for the first time. Using 5WBENCH, we show that no existing baseline simultaneously achieves high forgetting and high retention on Why-type questions: aggressive forgetting degrades retained knowledge, while conservative methods fail to forget causal facts. Why-type difficulty stems from multi-hop reasoning chains (44% of Why entries vs. less than or equal to 2% for others) and gradient dilution over 40.1-token answer spans. We present MAAT (Multi-phase Adapter-Aware Targeted Unlearning), a three-phase framework operating on LoRA adapter weights, combining gradient-projected ascent, SVD rank-dimension pruning, task vector negation, and hybrid KL-hidden-state retain repair. MAAT is the first method to simultaneously achieve high forgetting and high retention on Why-type causal knowledge, reaching a new operating point on the forget-retain Pareto frontier. We make our code publicly available.

1 Introduction

Existing benchmarks scarcely test causal knowledge, creating a measurement gap for unlearning methods that fail on relational and causal facts. The paper introduces balanced 5WBENCH and MAAT to expose and address this gap.

  • Why-type questions comprise less than 0.06% of CounterFact, 0.6% of ZSRE, 1.2% of TOFU, 0.5% of MUSE, and 1.2% of WMDP-Cyber.
  • This systematic underrepresentation makes failures on causal knowledge statistically undetectable without balanced evaluation.
  • 44% of Why-type answers involve multi-hop reasoning chains, versus ≤2% for other categories, while answers average 40.1 tokens versus 4.2–10.5 tokens.The paper attributes the difficulty to relational complexity and gradient dilution rather than a unique weight-space encoding.
  • MAAT is a three-phase LoRA adapter framework combining gradient projection, SVD-based rank-dimension pruning, task vector negation, and retain repair.
  • MAAT is the first method reported to achieve high forgetting and high retention on Why-type causal knowledge, reaching a new forget–retain Pareto operating point.
  • 5WBENCH provides 5,000 balanced samples, with 1,000 examples for each Who, What, When, Where, and Why category.

2 Related Work

Prior unlearning work spans gradient, localization, representation, adapter-aware, and geometry-aware methods, but existing benchmarks lack balanced causal Why-type coverage. 5WBENCH targets this evaluation gap directly.

  • Gradient-Based and Preference-Based Unlearning: Gradient-based methods trade off forgetting against utility: aggressive updates degrade utility, whereas conservative steps under-forget, particularly on difficult causal knowledge.
  • Localization, Weight Saliency, and Structured Editing: ROME and MEMIT localize factual associations in MLP weights, while AlphaEdit projects updates away from retained-knowledge covariance.
  • Localization, Weight Saliency, and Structured Editing: SalUn selects weights with highest forget-set gradient saliency, establishing a principled weight-selection mechanism for unlearning.
  • Representation-Based and Adapter-Aware Methods: Representation methods steer or reroute forget representations, while LUNE, KGA, LoKU, and FILA use adapter-based or Fisher-guided strategies.
  • Second-Order and Geometry-Aware Methods: Second-order methods use parameter-space geometry, Fisher information, or Hessian updates to modify unlearning optimization.
  • Benchmarks and Evaluation: ZSRE, CounterFact, TOFU, WMDP, MUSE, and RWKU each address aspects of editing or unlearning, but none provide balanced causal Why-type coverage.
  • Benchmarks and Evaluation: 5WBENCH supplies 1,000 balanced Why-type samples, making systematic causal-unlearning failures quantifiable for the first time.

3 The 5WBENCH Benchmark

5WBENCH is constructed from Factify-5WQA with stratified sampling across five question categories and includes rephrased variants for robustness testing. Its Why-type items specifically expose long, multi-hop causal reasoning challenges.

  • Dataset Construction: 5WBENCH extracts a primary subject entity using Stanford CoreNLP dependency parsing before sampling examples by 5W label.
  • Dataset Construction: The benchmark samples exactly 1,000 examples per 5W category, drawing uniformly within Who, What, When, Where, and Why.
  • Sample Format: Each record contains a question, ground-truth answer, 5W label, and up to three rephrased question variants.The variants test robustness to surface-form variation.
  • Why-Type Complexity: Why-type answers average 40.1 tokens, and 44% involve multi-hop reasoning chains, compared with 4.2–10.5 tokens and ≤2% multi-hop chains for other types.The benchmark therefore supports direct measurement of gradient-dilution failures in causal unlearning.

4 The MAAT Framework

MAAT performs adapter-only unlearning through conditional gradient projection, targeted rank-dimension surgery, task-vector negation, and hybrid retain repair. Its design concentrates forgetting on forget-relevant LoRA directions while repairing retained performance.

  • MAAT operates exclusively on LoRA adapter matrices while keeping the base model weights frozen.
  • Phase 1: Gradient-Projected Unlearning: Conditional gradient projection removes forget updates aligned with retain gradients only when the gradients conflict, preserving the full ascent direction otherwise.Conflict is detected when gf · gr > 0.
  • Phase 2a: SVD Rank-Dimension Pruning: SVD pruning scores MLP rank dimensions by forget-set gradient-column norms and zeros the highest-scoring fraction in both adapter matrices.Attention modules are excluded because pruning them at non-trivial ratios destroys the instruction-following pathway.
  • Phase 2b: Task Vector Negation: Task-vector negation subtracts a forget task vector formed from the top-kF forget-scored dimensions, targeting suppression without erasing retain-associated directions.The described setting uses kF = 50%.
  • Phase 3: Hybrid Repair: Hybrid retain repair combines multiple objective terms after ascent and pruning, including KL anchoring, hidden-state representation distance, entropy maximization on forget predictions, and task-vector separation.The negative entropy term discourages recovery of forgotten content during repair.

5 Experimental Setup

The experiments evaluate MAAT and adapter-based baselines on balanced and established unlearning datasets using two instruction-tuned model families. Forgetting and retention are measured primarily with an LLM judge, supplemented by ROUGE metrics.

  • Models: Experiments use LLaMA 3.2-3B-Instruct and Gemma 3-4B-Instruct, both fitting on a single consumer GPU with ≤24 GB VRAM.Inference uses greedy decoding, temperature = 0, and max_new_tokens = 100.
  • Datasets: 5WBENCH experiments use 100 examples per 5W label in each split, with 500 forget and 500 retain examples stratified for equal representation.The full benchmark contains 1,000 examples per label and 5,000 total.
  • Datasets: TOFU uses the forget05/retain95 split but contains no meaningful Why-type questions because of severe category imbalance.
  • Baselines: Baselines include Gradient Ascent, KL-regularised Gradient Ascent, and Retain-Only Fine-Tuning, all operating on the fine-tuned LoRA adapter.
  • Evaluation Metrics: The primary LLM-as-Judge evaluation reports Forget Success Rate and Retain Success Rate based on whether outputs contain the ground-truth answer.ROUGE-1, ROUGE-2, and ROUGE-L are additionally reported for forget and retain splits.

6 Results

5WBENCH reveals substantial differences in forget–retain behavior that category-skewed TOFU results cannot resolve. MAAT achieves the strongest aggregate balance, maintains performance across categories and architectures, and avoids the under-forgetting and catastrophic-retention failures seen in baselines.

  • Aggregate performance: 77.4% FSR / 71.6% RSR on Llama 3.2-3B and 64.0% / 61.8% on Gemma 3-4B give MAAT the best aggregate forget–retain balance.Against RO-FT on Llama 3.2-3B, MAAT matches 77.4% average FSR while improving RSR from 35.2% to 71.6%.
  • Per-category consistency: MAAT is the only method exceeding 60% FSR and 60% RSR on all five 5W categories on Llama 3.2-3B.Its category results include 63% FSR / 65% RSR on Why-type questions, while no baseline reaches the threshold on any single category.
  • Baseline trade-offs: AN reaches 99.8% FSR but collapses to 0.4% RSR on Llama 3.2-3B, whereas GA+KL reaches 33.8% FSR and sacrifices forgetting for retention.These baselines occupy opposite failure regions: indiscriminate forgetting versus under-forgetting.
  • Why-type results: MAAT achieves 63% FSR / 65% RSR on Why-type questions on Llama 3.2-3B and 55% / 56% on Gemma 3-4B.Compared with GA on Llama 3.2-3B, MAAT forgets 19 points more Why-type knowledge while losing 13 RSR points.
  • Architecture sensitivity: MAAT’s FSR ranges from 63–83% across categories on Llama 3.2-3B, while RO-FT shows larger architecture-dependent swings when moving to Gemma 3-4B.MAAT degrades more uniformly across architectures, maintaining a balanced operating point.
  • TOFU comparison: On TOFU, MAAT achieves 67% FSR / 46.6% RSR on Llama 3.2-3B among methods exceeding 60% FSR, but category imbalance limits per-category resolution.TOFU contains 84.7% What-type questions and 1.2% Why-type questions, whereas 5WBENCH uses balanced 500-sample category splits.

7 Conclusion

The paper introduces 5WBENCH to quantify causal unlearning failures and MAAT to improve the forgetting–retention tradeoff, especially for Why-type knowledge. MAAT reaches a new operating point while performance remains architecture-dependent.

  • Contributions: MAAT is a three-phase structured LoRA adapter framework that concentrates forgetting on rank dimensions activated by forget-set inputs.The framework operates through targeted adapter manipulation rather than uniform forgetting pressure.
  • Results: No existing baseline simultaneously achieves high forgetting and high retention on Why-type causal knowledge, whereas MAAT exceeds 60% FSR and 60% RSR across all five 5W categories on Llama 3.2-3B.The reported result includes Why-type causal questions, where baselines face a fundamental forget–retain tradeoff.
  • Limitations and future work: Performance differs between Llama 3.2-3B (77.4/71.6) and Gemma 3-4B (64.0/61.8), indicating architecture-dependent knowledge separability in adapter rank dimensions.The paper identifies architecture-aware adapter rank selection as an open direction.
  • Broader scope: 5WBENCH’s balanced coverage reveals performance differences and localized category failures that TOFU’s skewed evaluation cannot resolve.The format also supports model-editing evaluation, including causal-knowledge insertion and modification, although this paper evaluates only unlearning.

Limitations

The evaluation is limited by reliance on a single LLM judge, dataset construction choices, and a narrow model and task scope. These boundaries motivate broader validation and architecture-specific investigation.

  • Evaluation: FSR and RSR are evaluated with a single Qwen 2.5-7B LLM judge, whose calibration may differ from proprietary judges or human evaluation.Judge availability and versioning may also affect reproducibility over time.
  • Evaluation: Future work should complement LLM-judge evaluation with adversarial paraphrase probing to distinguish genuine unlearning from verbatim-recall suppression.
  • Dataset scope: 5WBENCH derives from Factify-5WQA, which was designed for fact verification rather than knowledge editing, and subject extraction may introduce noise for implicit or pronominal subjects.
  • Model and benchmark scope: Evaluation covers two model families at 3–4B scale on 5WBENCH and TOFU, leaving 7B+ checkpoints, additional architectures, and other editing operations for future work.Phase 2 pruning and masking hyperparameters may require architecture-specific tuning.

Ethical considerations

The paper frames unlearning as supporting privacy and safety needs while acknowledging that the same techniques could be used to remove safety-relevant knowledge from aligned models.

  • Benefits: The methods address legitimate needs including GDPR compliance and correction of harmful associations.
  • Risks: The authors acknowledge a dual-use concern: adversaries could apply the techniques to remove safety-relevant knowledge from aligned models.

A Ablation Study

The ablation isolates how MAAT components reshape the forget–retain tradeoff: hybrid repair improves retention but can recover forgotten content, attention pruning boosts forgetting at high collateral cost, and task vector negation restores balance. Across full profiles, MAAT occupies the Pareto-dominant region, while Why-type questions remain especially difficult because of long, multi-hop answers.

  • Ablation setup: The ablation progressively introduces MAAT components on a 200-sample 5WBENCH subset, evaluating FSR, RSR, and post-unlearning ROUGE.The setup uses 20 forget and 20 retain samples per label; lower forget ROUGE indicates stronger erasure, while higher retain ROUGE indicates better preservation.
  • Condition A: 48% FSR and 83% RSR in Condition A show that gradient projection, MLP SVD pruning, and KL-only repair provide strong retention but limited erasure.The retention benefit is attributed to gradient projection, while the absence of further structural erasure limits unlearning efficacy.
  • Condition B: 43% FSR and 90% RSR in Condition B show that hybrid repair maximizes retention but over-corrects by recovering forgotten content without a structural erasure anchor.Condition B adds the full hybrid repair objective to Condition A.
  • Condition C: 70% FSR and 54% RSR in Condition C show that attention pruning sharply increases forgetting while causing severe collateral damage to retained distributions.Attention modules are pruned at ρattn = 0.01 in addition to MLP pruning.
  • Condition D: 71% FSR and 76% RSR in Full MAAT show that task vector negation balances deep erasure with substantially improved retention over attention pruning.Full MAAT replaces attention pruning with task vector negation on forget-scored rank dimensions and retains the full hybrid repair phase.
  • Complete profiles: MAAT consistently occupies the Pareto-dominant region across aggregation metrics, whereas baselines reveal retention collapse, under-forgetting, or lower overall tradeoff quality.Adapter Negation’s Geometric and Harmonic scores approach zero because retention collapses, while GA+KL trades forgetting for retention.
  • Category difficulty: Why-type questions are particularly difficult because their answers are longer and more often require multi-hop reasoning chains than other 5WBENCH categories.The benchmark contains 1,000 samples per category, and Why entries are identified as having severe gradient dilution during standard unlearning.

F Encoding Analysis

Encoding analysis finds that 5W facts are broadly and uniformly distributed across layers, so Why-type difficulty does not reflect a uniquely concentrated representation. Qualitative traces show MAAT removing causal forget facts while preserving retained knowledge, unlike competing behaviors illustrated in the tables.

  • Encoding distribution: Low Gini coefficients and small top-3 layer mass confirm distributed encoding across all 5W categories on both models.Gemma’s Gini values range from 0.152–0.177 with 17.8–19.3% top-3 mass; Llama’s range from 0.258–0.277 with 28.1–29.9%.
  • Encoding distribution: Why-type questions do not deviate from the distributed-encoding baseline, with Gini values of 0.172 for Gemma and 0.276 for Llama.The analysis reports no category-specific representation concentration for Why questions on either model.
  • Evaluation limitations: TOFU contains only 2 Why samples in its forget split, providing insufficient statistical signal for label-wise unlearning computation.Its inferred label distributions are highly skewed across the Forget10 and Retain splits.
  • Evaluation protocol: The Qwen 2.5-7B judge evaluates semantic ground-truth containment, requiring the core causal reason for Why questions and using greedy decoding at temperature 0.The same prompt template is used for forget and retain evaluation, with verdict direction determining FSR versus RSR.
  • Qualitative traces: For a Why-type forget example, MAAT removes the target causal link and produces a fluent alternative, while GA and GA+KL reproduce the causal detail.AN produces a severely degraded output in the same qualitative comparison.
  • Qualitative traces: On a Why-type retain example, MAAT preserves the core fact with minor paraphrasing and fluent output, whereas AN remains incoherent.The retained example concerns the significance of PM Modi’s visit to Vladivostok.
Loading 2605.30514v1…