Source-linked AI summary
Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation
Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale, Tuhina Tripathi, Abhishek Lalwani, Somya Chatterjee, Sadid Hasan
TL;DR
The paper asks whether clean-query unlearning evaluations establish that targeted information is truly inaccessible. It compares unlearning methods under shared conditions and adversarially tests strong performers using ASR. Fine-tuning-based methods show high clean-query forgetting yet substantial adversarial recoverability, while multilingual leakage is much lower and ASR validation is imperfect.
Problem
Standard unlearning benchmarks use clean, non-adversarial queries and leave the recoverability of supposedly forgotten information under strategic prompting insufficiently assessed.
Method
The study compares prompt-based and fine-tuning-based methods on TOFU with four clean-query metrics, then applies hacker-generated adversarial prompts and ASR to fine-tuning-based models.
Results
72.8%–84.3% ASR remained for fine-tuning-based methods despite strong forgetting, versus 87.5% for the unprotected base model; clean multilingual probing produced 2.95% measured leakage.
Takeaways & Limitations
Strong standard-metric performance alone is insufficient to establish robustness after unlearning, supporting adversarial stress-testing as a complementary evaluation component.
Takeaways & Limitations
The study uses one victim model family, and its LLM-judge audit covers only ten purposively selected examples with one annotator.
Abstract
from arXiv · showhide
Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing benchmarks primarily assess unlearning under clean, non-adversarial queries, leaving open whether information that appears forgotten can still be recovered through strategic prompting. We address this gap through a unified evaluation of prompt-based and fine-tuning-based unlearning methods on TOFU using Llama-3.2-3B-Instruct, followed by an adversarial robustness evaluation of methods that perform strongly under standard metrics. We introduce Attack Success Rate (ASR), an LLM-as-judge metric that measures the fraction of adversarial responses whose leakage score exceeds $0.2$, and evaluate recovery across eight attack suites. Our results reveal a substantial gap between clean-query forgetting and adversarial robustness. Although several fine-tuning-based methods achieve Forget Quality above $0.91$, targeted information remains recoverable with ASRs between $72.8\%$ and $84.3\%$, close to the $87.5\%$ ASR of the unprotected base model. In contrast, clean multilingual reformulations yield only $2.95\%$ measured leakage. A manual audit further finds agreement between binary ASR decisions and human factual assessments in seven of ten cases, indicating that ASR provides a useful, though imperfect, signal of behavioral recoverability. These findings show that strong standard-metric performance alone is insufficient to establish robustness after unlearning and motivate adversarial stress-testing as a complementary component of unlearning evaluation.
1 Introduction
Machine unlearning seeks to remove targeted information while preserving retained capabilities, but standard clean-query metrics may miss whether supposedly forgotten knowledge remains recoverable. This paper addresses that gap through unified cross-paradigm evaluation and adversarial testing.
- Sensitive information incorporated into model parameters can be difficult to remove, motivating unlearning for privacy, compliance, and responsible deployment.
- Prior evaluations often use method-specific models, datasets, objectives, and metrics, limiting direct comparison between prompt-based and fine-tuning-based methods.
- The study compares both method families under a shared framework, then evaluates fine-tuning-based methods with adversarial and multilingual queries.
- Attack Success Rate measures whether adversarial prompting recovers a factually correct, human-readable answer from the forget set.
- Strong clean-query performance can create a false sense of security because standard metrics do not establish resistance to strategic knowledge recovery.
2 Related Work
Prior LLM unlearning work spans prompt-based and parameter-modifying approaches, but results are often difficult to compare across differing experimental settings. This paper places both paradigms in a shared framework and extends evaluation with adversarial and multilingual probing.
- Prompt-based methods operate at inference time without modifying model parameters, whereas fine-tuning-based methods update weights or internal representations.
- ICUL uses prepended forget examples and anchor demonstrations, but is constrained by context-window capacity and requires targeted information in the prompt.
- Early gradient-ascent unlearning can degrade retained capabilities, while NPO instead penalizes forget-set responses relative to a reference model.
- Adversarial recovery research uses generated jailbreak prompts, and prior TOFU and WMDP experiments show vulnerability despite strong standard unlearning metrics.
- The paper compares methods using a shared checkpoint, data splits, and evaluation framework before adversarial and multilingual probing.
3 Method
The method uses a common TOFU-fine-tuned checkpoint to compare six unlearning methods with clean-query metrics, then stress-tests the fine-tuning-based methods using adversarial and multilingual prompts. ASR quantifies recovery when judge-assigned leakage exceeds 0.2.
- Unified Benchmarking: The unified pipeline comprises preparation, unlearning, and evaluation stages using a common experimental setup.
- Preparation: All methods begin from Llama-3.2-3B-Instruct fine-tuned on complete TOFU, exposing the checkpoint to all 200 fictitious authors.
- Unlearning: Six methods span prompt-based approaches that leave parameters unchanged and fine-tuning-based approaches that modify parameters or internal representations.
- Unified Benchmarking: The clean-query evaluation uses Forget Quality, Model Utility, Forget Truth Ratio, and PrivLeak to measure forgetting, retention, truth-ratio behavior, and privacy leakage.
- Adversarial Robustness Evaluation: Adversarial evaluation targets the four fine-tuning-based methods because prompt-based methods degrade to near-zero Forget Quality as forget-set size increases.
- Adversarial Robustness Evaluation: A hacker model generates and refines prompts across role-playing, injection, rephrasing, routing, and multilingual query conditions.
4 Experimental Setup
Experiments use the controlled TOFU benchmark and identical forget/retain splits across six methods, with Llama-3.2-3B-Instruct as the common checkpoint. Evaluation combines four OpenUnlearning metrics with adversarial and multilingual robustness procedures.
- Dataset: TOFU contains 200 fictitious authors with synthetic biographies and 4,000 author-specific question–answer examples, enabling clearly defined forgetting targets.
- Evaluation Metrics: OpenUnlearning metrics capture forgetting quality, retained utility, forget-set truth-ratio behavior, and membership-inference privacy leakage.
- Evaluation Metrics: FQ and FTR favor stronger forgetting at higher values, MU favors better retained capability, and PrivLeak favors values closer to zero.
- Implementation: All six methods share the same TOFU-fine-tuned checkpoint, while adversarial testing uses a hacker model and a TOFU-fine-tuned judge model.
5 Results
Under unified clean-query evaluation, fine-tuning-based methods generally outperform prompt-based methods on Forget Quality, but adversarial tests reveal that apparently forgotten information remains highly recoverable. Multiple metrics and attack formulations expose substantial variation beyond standard forgetting scores.
- Unified evaluation: NPO, SimNPO, and AltPO achieve substantially stronger Forget Quality than prompt-based methods across the 1%, 5%, and 10% forget splits.RMU shows more variable forgetting performance.
- Unified evaluation: Forget Quality for NPO, SimNPO, and AltPO ranges from 0.919 to 0.999, while SPUL and ICUL decline sharply as the forget split increases.RMU scores 0.765, 0.466, and 0.581 across the 1%, 5%, and 10% splits.
- Unified evaluation: NPO, SimNPO, and AltPO combine consistently high Forget Quality with relatively stable Model Utility, whereas RMU preserves similar utility but less consistent forgetting.This represents the strongest clean-query forgetting–utility trade-off in the evaluation.
- Unified evaluation: FTR generally ranges from approximately 0.68 to 0.74, while PrivLeak varies across methods and splits, supporting evaluation with multiple complementary metrics.No method remains uniformly close to zero on PrivLeak across all three splits.
- Adversarial evaluation: SimNPO reaches Forget Quality 0.999 on the 1% split, yet an authority-framing prompt recovers the target birthplace exactly with judge leakage and ROUGE-L both at 1.000.The example demonstrates that clean-query performance does not necessarily imply resistance to adversarial recovery.
- Adversarial evaluation: Average ASRs remain 72.8%–84.3% for the four fine-tuning-based methods, close to the unprotected base model’s 87.5% ASR despite strong clean-query Forget Quality.Prompt-injection attacks average approximately 86.6%, while attack effectiveness varies substantially by formulation and method.
- Adversarial evaluation: Leakage patterns across methods are weakly correlated, with pairwise Pearson correlations ranging from r = −0.022 to r = 0.290.This suggests that methods can exhibit different leakage patterns across matched attack-suite–question pairs.
- ASR validation: Leakage scores correlate strongly with ROUGE-L at r = 0.924, but 27 high-judge, low-ROUGE cases require human inspection to assess factual knowledge recovery.The correlation and divergence analysis alone cannot establish that every divergent case represents factually correct recovery.
6 Conclusion
The unified evaluation finds that clean-query unlearning performance does not reliably predict resistance to adversarial recovery. Adversarial prompting exposes substantial recoverability gaps, while multilingual reformulations produce much lower measured leakage.
- 72.8%–84.3% average ASR remained for fine-tuning methods on the 1% forget split, compared with 87.5% for the unprotected base model.These results show that strong clean-query forgetting does not establish resistance to strategic knowledge recovery.
- NPO, SimNPO, and AltPO maintain high Forget Quality across TOFU splits, whereas prompt-based SPUL and ICUL decline sharply as forget-set size increases.The shared evaluation exposes differences that method-specific metrics make difficult to compare.
- 2.95% of responses exceeded the ROUGE-L leakage threshold under clean multilingual probing across seven non-English languages.Cross-lingual recovery remained possible particularly in German and Spanish, but the largest robustness gap appeared under adversarial English prompts.
- Adversarial stress-testing provides a complementary perspective because strong standard-metric performance alone does not demonstrate robustness to strategic knowledge recovery.The conclusion frames adversarial evaluation as extending, rather than replacing, clean-query benchmarking.
7 Limitations and Future Work
The study’s conclusions are bounded by its model, synthetic benchmark, adversarial coverage, and evaluation design. Future work should test broader models and data, larger forget splits, additional attacks, and better multilingual leakage validation.
- All experiments use Llama-3.2-3B-Instruct, leaving generalization to larger, more strongly tuned, or architecturally different models open.The paper specifically identifies Mistral and Gemma as families for future evaluation.
- TOFU uses synthetic biographies of fictitious authors, so real-world removal requests may involve more complex, redundant, and context-dependent information.The reported adversarial recovery patterns require evaluation on real-world personal or sensitive information before broader generalization.
- Adversarial stress-testing is limited to the 1% TOFU forget split, excluding direct evidence for the 5% and 10% splits.This limitation is especially relevant for RMU, whose clean-query Forget Quality varies substantially across evaluated splits.
- The LLM judge combines strong correlation with ROUGE-L and a small audit, but the audit agreed with binary ASR decisions in only seven of ten purposively selected examples.Larger random samples and multiple annotators are needed to estimate accuracy and calibrate the 0.2 threshold more precisely.
- Multilingual leakage is defined using ROUGE-L greater than 0.3 without independent calibration against human judgments.Alternative criteria and human annotation could test whether the observed cross-lingual patterns remain stable.
- The eight attack suites do not exhaust extraction strategies, including adaptive multi-turn, gradient-based, automated-search, and indirect multi-hop attacks.These attack classes remain important directions for future evaluation.
A Manual Audit of LLM-Judge Leakage Decisions
The manual audit compares binary ASR decisions with blinded human judgments of factual disclosure. Agreement was substantial but imperfect, with errors concentrated in factually incorrect or threshold-near cases.
- Manual labels distinguish no leakage, partial leakage, and clear leakage according to whether correct reference information is absent, incomplete, or centrally disclosed.Partial and clear leakage count as successful disclosure for comparison with binary ASR.
- 7/10 audited examples matched after partial and clear leakage were mapped to successful disclosure and no leakage to unsuccessful disclosure.The annotator judged responses while blinded to the method, attack suite, automated scores, and ASR decision.
- Two disagreements involved factually incorrect responses scored above the 0.2 ASR threshold, while S06 disclosed the target information but scored 0.190.These cases illustrate both false positives and false negatives from judge assessment and thresholding.
- The audit indicates that clear disclosures and clear non-disclosures were generally consistent, while incorrect or decision-boundary cases were more prone to disagreement.The evidence supports ASR as a meaningful but imperfect signal rather than a definitive test of complete removal.
- The audit used one annotator and ten purposively selected examples, so its agreement rate is qualitative rather than a population-level reliability estimate.More randomly sampled examples and multiple annotators would support more precise validation.
B AI Disclosure
Claude was used as a writing assistant for presentation and technical-support tasks, while the team retained responsibility for the project’s substantive decisions and core work.
- Claude assisted with phrasing, organization, document structure, references, flow diagrams, and experimental-script debugging.The listed uses concern report preparation and technical assistance.
- Team members determined the project idea, designed the methodology, selected related work, interpreted papers, wrote core implementation logic, and reviewed generated text and code.