PaperlayerView exact revision on arXiv

Source-linked AI summary

Position: LLM Unlearning Benchmarks are Weak Measures of Progress

Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, Virginia Smith

arXiv:2410.02879v2cs.CL

TL;DR

Empirical benchmarks are increasingly used to assess LLM unlearning, but their fixed queries may not reliably represent practical unlearning goals. This paper modifies popular benchmarks and finds that dependencies between forget and retain information and ambiguity in targets can expose failures or encourage query-fitting. It urges caution in interpreting benchmark results and calls for definitions and metrics aligned with practical use cases.

  • Problem

    Empirical benchmarks assess LLM unlearning, but their fixed queries may not reliably indicate whether unlearning succeeds beyond those evaluations.

    Source: §I · p. 1

  • Method

    The paper analyzes popular benchmarks and tests how simple modifications, dependencies between forget and retain information, and target ambiguity affect evaluation.

    Source: §I · pp. 1–2

  • Results

    Small benchmark modifications can contradict reported success, dependencies between forget and retain sets can cause catastrophic failures, and ambiguous targets can encourage query-fitting.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The authors recommend caution with benchmark-only claims and advocate practical-use definitions and metrics, alongside provable algorithms.

    Source: §I · p. 2

    The paper offers examples of general benchmark properties rather than targeting particular methods or benchmarks, and fully comprehensive benchmarks are extremely challenging to develop.

    Source: §I · p. 2

Abstract

from arXiv · show

Unlearning methods have the potential to improve the privacy and safety of large language models (LLMs) by removing sensitive or harmful information post hoc. The LLM unlearning research community has increasingly turned toward empirical benchmarks to assess the effectiveness of such methods. In this paper, we find that existing benchmarks provide an overly optimistic and potentially misleading view on the effectiveness of candidate unlearning methods. By introducing simple, benign modifications to a number of popular benchmarks, we expose instances where supposedly unlearned information remains accessible, or where the unlearning process has degraded the model's performance on retained information to a much greater extent than indicated by the original benchmark. We identify that existing benchmarks are particularly vulnerable to modifications that introduce even loose dependencies between the forget and retain information. Further, we show that ambiguity in unlearning targets in existing benchmarks can easily lead to the design of methods that overfit to the given test queries. Based on our findings, we urge the community to be cautious when interpreting benchmark results as reliable measures of progress, and we provide several recommendations to guide future LLM unlearning research.

I. INTRODUCTION

LLM unlearning benchmarks use curated forget and retain queries because directly evaluating unlearning at LLM scale is difficult. The paper finds that small benchmark changes can expose failures and that benchmark ambiguities encourage query-fitting, making benchmark results alone an unreliable measure of practical progress.

I. INTRODUCTION

  • Curated forget and retain queries help assess approximate unlearning when model and data scale make evaluation difficult.Forget queries probe whether information was unlearned; retain queries test preservation of information unrelated to the unlearning data.Source: §I · p. 1
  • Small, nonadversarial benchmark modifications can reveal behavior contradicting reported unlearning success, especially when forget and retain information have dependencies.Popular benchmarks often use largely unrelated forget and retain queries, while even rough dependencies between them can cause catastrophic failures in methods optimized on those benchmarks.Source: §I · pp. 1–2
  • Existing benchmarks can encourage fitting to evaluation queries when targets or the intended effects on test metrics are ambiguous.The information to unlearn may be implicitly defined by test queries, and retain accuracy may appropriately fall when retain questions correlate with the unlearned data.Source: §I · p. 2
  • The authors urge caution about benchmark-only progress claims and recommend definitions and metrics aligned with practical use cases, alongside provable algorithms.They frame their attacks on particular algorithms as examples of general benchmark properties, not as criticism of those methods or any single benchmark.Source: §I · p. 2

THE RISE OF LLM UNLEARNING BENCHMARKS

Forget/retain benchmarks have become a prominent way to evaluate LLM unlearning, despite limits in what their queries and metrics can establish. The paper surveys their prevalence and describes common benchmarks, their caveats, and how benchmark components can leave evaluations vulnerable to overfitting.

A. Focus of this work: forget set/retain set benchmarks

  • 82% of the 72 surveyed 2024 repository papers used the forget/retain evaluation structure.The authors note that the repository may reflect curation or field-trend biases, while treating the survey as an indicator of broader patterns.Source: A. Focus of this work: forget… · p. 2
  • Nearly half of surveyed papers used one of the five most popular benchmarks, and those benchmarks accounted for 80% of survey-set citations.The five are TOFU, WMDP, TDEC, PKU-SafeRLHF, and Who’s Harry Potter; Figure 1 also reports that TOFU and WMDP alone appear in 31% of papers.Source: A. Focus of this work: forget… · p. 2, Figure 1 · p. 3

B. Description of common unlearning benchmarks

  • The five benchmarks test different targets, from Harry Potter knowledge and synthetic author profiles to hazardous knowledge, celebrity information, and extractable training examples.Their approaches include automated question scoring, multiple-choice accuracy, adversarial probes, and exact-sequence extraction evaluation.Source: B. Description of common unlearning… · p. 3
  • Benchmark caveats warn against overgeneralizing results, but their ambiguity has not prevented uncritical use as measures of success.The authors note that it can be unclear what conclusions an evaluation supports, despite papers citing these benchmarks as evidence of success.Source: B. Description of common unlearning… · p. 3
  • A benchmark combines forget and retain test queries with optional base-model, unlearning-data, and metric specifications, and incomplete specification can make query overfitting easier.The figure identifies the test queries as the typical evaluation component and notes that base models, task-specific data, unlearning data, and metrics may also be specified.Source: Figure 2 · p. 4

C. Works proposing improvements to benchmark datasets and metrics

Prior benchmark extensions add data or metrics, but the paper identifies dependencies between forget and retain information and test-query overfitting as distinct problems. It also argues that empirical brittleness supports considering provable metrics and efficient exact unlearning.

C. Works proposing improvements to benchmark datasets and metrics

  • Prior extensions add benchmark data or metrics, but generally preserve a separation between forget and retain performance; this paper instead highlights dependencies and test-query overfitting as fundamental axes.Despite calls for more complex metrics, most work continues to rely on the benchmark’s original measures, typically accuracy.Source: C. Works proposing improvements to… · p. 4
  • The paper argues that empirical brittleness warrants considering formal, provable unlearning metrics, though such guarantees often remain confined to simpler or finetuned models.The authors suggest testing weaker benchmark perturbations before extending evaluation to stronger threat models.Source: C. Works proposing improvements to… · p. 4

D. Provable methods, definitions, and metrics

The paper describes limits in provable unlearning work and analyzes how benchmark design, threat assumptions, and familiar LLM-evaluation brittleness shape empirical assessment. It argues that benign, in-distribution queries can expose unlearned information or damage retained utility.

D. Provable methods, definitions, and metrics

  • Provable work spans data-deletion definitions, approximate adaptive unlearning, convex-loss methods, and SISA, but size and runtime constraints limit LLM applicability.The paper notes that its empirical focus does not exhaustively cover later work on provable algorithms.Source: D. Provable methods, definitions, and… · p. 4

A. Anatomy of an LLM unlearning benchmark

  • Benchmarks may specify base models, data sources, forget/retain evaluation sets, and metrics, whose choices affect which methods appear most effective.Accessing evaluation sets to perform unlearning can lead to overfitting, while many benchmarks ignore dependencies between forget and retain sets.Source: A. Anatomy of an LLM unlearning… · pp. 4–5

B. Threat model

  • The paper assumes guarantees should hold against benign, essentially in-distribution queries, excluding optimizer-interactive, white-box, incoherent, out-of-domain, or non-English queries.Under this threat model, the authors report crafting queries that reveal unlearned information or degrade retained utility.Source: B. Threat model · p. 5

C. Aren’t all LLM evaluations brittle?

  • Unlike capability-benchmark brittleness that mainly warrants caution about leaderboard variance, evaluation brittleness in privacy-critical unlearning can mean the difference between privacy and a privacy violation.The paper says query modifications can systematically reveal unlearned information or degrade retained performance, and unlearning can make models more vulnerable to such changes.Source: C. Aren’t all LLM evaluations brittle? · p. 5

IV. FORGET-RETAIN EVALUATIONS ARE DECEPTIVE

The section shows that mild query changes linking forget and retain information can undermine unlearning results on separate benchmark splits. In TOFU and WMDP, paired or modified retain queries expose failures not indicated by the original evaluation setup.

IV. FORGET-RETAIN EVALUATIONS ARE DECEPTIVE

  • Separate forget and retain evaluations permit classifier-like behavior, although real queries may link the two sets and make such classification difficult or impossible.The authors conclude that simple combinations of queries can cause failures in methods optimized for existing benchmarks.Source: §IV · p. 5, A. TOFU: Asking about both forget and… · p. 7

A. TOFU: Asking about both forget and retain entities

  • DPO and ECO retain performance falls significantly on paired forget-retain queries, while gradient ascent is more stable but has lower overall F1 than their retain-only scores.The figure reports ROUGE-L precision, recall, and F1 on the retain response; DPO and ECO tend to treat the whole paired query as forget-related and refuse.Source: Figure 3 · p. 6, A. TOFU: Asking about both forget and… · p. 6
  • TOFU models that perform well on separate forget and retain queries fail on combined questions, often treating both authors as either forgotten or retained.The experiment combines a forget-set author question with a retain-set author question; a retrained model hallucinates for the former but answers the latter correctly.Source: A. TOFU: Asking about both forget and… · pp. 5–6

B. WMDP: Modifying multiple-choice responses

  • A benign incorrect-choice swap with a forget-associated phrase drives unlearned WMDP retain performance near-random; RMU drops nearly 30% in college biology and 28% overall, while the base model generally improves.With LAT, the RMU result drops below random in every category.Source: Figure 4 · p. 7, A. TOFU: Asking about both forget and… · p. 7

V. BENCHMARKS MAY ENCOURAGE TAILORING TO THE TEST SET

The paper argues that benchmark design can encourage tailoring methods to evaluation queries. It says this concern persists even when forget-retain dependencies are addressed and applies to newer method classes.

V. BENCHMARKS MAY ENCOURAGE TAILORING TO THE TEST SET

  • Current benchmark design can explicitly require or implicitly encourage tailoring algorithms to test-set design or content, including direct test-set training.The preceding section's mild perturbation results hint at this issue but do not exhaust it.Source: §V · p. 7
  • The concern can persist even when dependencies between forget and retain sets are carefully considered.The paper distinguishes this test-tailoring concern from the dependency problem discussed in Section IV.Source: §V · p. 7
  • Entirely new classes of unlearning methods are being developed that do not appropriately factor into this concern.Source: §V · p. 7

A. Conflating the forget data and test query set

The section shows how benchmark structure can reward query-specific filters or classifiers rather than general unlearning, and how test-set access can encourage overfitting. Its examples include a TOFU keyword filter, an ECO classifier exploiting a spurious feature, and training on the same test queries used for evaluation.

A. Conflating the forget data and test query set

  • Pre- or postprocessing can be effective in practice, but designing such methods may require inspecting test queries, creating a training–test conflation.Without a clear separation, query-specific preprocessing or output filtering may overfit to the test queries.Source: A. Conflating the forget data and test… · pp. 7, 9
  • On TOFU, an exact-author-name filter replaces matching queries or responses with a refusal, performing effectively on this dataset but failing one forget-10% pair containing only a last name.The filter also performs perfectly on queries outside TOFU, but may be difficult to apply to real-world individuals; it makes the truth-ratio metric undefined.Source: A. Conflating the forget data and test… · p. 8, Table I · p. 8
  • The linear classifier performs comparably to RMU in several categories and comparably to or better than RMU on retain; Table II reports SCRUB as 5% better than filtering on WMDP-Bio.The table caption also says the reported results are strictly better than LLMU and SCRUB on all other columns.Source: Table II · p. 8, A. Conflating the forget data and test… · p. 9
  • ECO [52] can classify WMDP forget versus retain queries using the prepended topic phrase; without it, the classifier reaches 37% accuracy, below random.The phrase’s “college” keyword distinguishes many MMLU retain questions from WMDP forget questions, letting the classifier exploit a spurious feature.Source: A. Conflating the forget data and test… · p. 8
  • Some work claims success on PKU-SafeRLHF [38] using the same test set for training and evaluation, despite a held-out test set being available.The authors recommend held-out queries to measure generalization when benchmarks provide them.Source: A. Conflating the forget data and test… · p. 9

B. Changing the query type

Changing a query’s format can reveal information that a benchmark suggests was unlearned. Because benchmarks often test only one query type, covering the possible formats is difficult.

B. Changing the query type

  • Algorithms that succeed on one query format may reveal information after the format changes, while sufficiently covering potential input formats in a benchmark is challenging.Prior examples include Harry Potter information recovered through multiple-choice queries or slight input modifications after training on sentence completions.Source: B. Changing the query type · p. 9

C. Benchmarks with underspecified forget set

Benchmarks with underspecified forget targets may use metrics that do not reliably correspond to unlearning and may implicitly encourage fitting to test queries. The recommendations also caution that separate forget and retain measures miss dependencies between the two.

C. Benchmarks with underspecified forget set

  • RWKU [6] specifies a famous-entity target but no forget data, while requiring preservation of neighboring information; baseline methods and follow-up work fail to perform well on both sets simultaneously.The evaluation implicitly assumes forget and neighbor sets are disjoint, although the target’s influence is underspecified.Source: C. Benchmarks with underspecified… · p. 9
  • When forget data is underspecified, forget- and neighbor-set accuracy may be ambiguous or inconsistent with actual unlearning, potentially setting algorithm developers up for failure.Forget and retain metrics are not direct measures of unlearning, so their performance is not guaranteed to correlate with it.Source: C. Benchmarks with underspecified… · p. 9

VI. RECOMMENDATIONS

  • Separating forget and retain query performance overlooks realistic queries dependent on both data types; simple accuracy metrics are insufficient for evaluating such dependencies.Even a benchmark that tests these dependencies may leave methods vulnerable to other query modifications.Source: §VI · p. 9

A. Design benchmarks that do not encourage fitting to the test set

The recommendations target test-query overfitting, unclear distinctions between unlearning and censorship, and unspecified threat models. They call for more robust query evaluation, defined forget data, and threat models that reflect realistic access and attacks.

A. Design benchmarks that do not encourage fitting to the test set

  • Use multiple query formats, held-out queries with unseen formats, and reviewer scrutiny to assess whether methods generalize beyond supplied test queries.Pre- and postprocessing filters may be effective but require inspecting test queries to design.Source: A. Design benchmarks that do not… · p. 10
  • Specify the data to unlearn and consider finetuning-data settings, such as TOFU, where retraining can provide an accurate performance baseline.Source: A. Design benchmarks that do not… · p. 10
  • Use membership-inference attacks to measure data retention; the authors argue they can be practical when unlearning finetuning data or using quantile-regression approaches.Source: A. Design benchmarks that do not… · p. 10

B. Separate unlearning from general censorship tasks

  • Define unlearning narrowly as removing the influence of specific training-data points, and call censoring harmful or general-topic information alignment or censorship.This distinction is intended to clarify the problem statements and goals of algorithms and benchmarks, not discourage broader knowledge-editing research.Source: B. Separate unlearning from general… · p. 10

C. Define clear threat models

  • Specify a threat model so readers can understand expected operating conditions and compare methods defending against the same threat.The authors suggest articulating relevant threat dimensions and question whether restrictions, such as limiting weight access rather than query types, match reality.Source: C. Define clear threat models · p. 10

D. Focus on efficient exact unlearning methods

The paper argues for moving beyond brittle empirical measures toward formal definitions and guarantees, while recognizing that exact unlearning methods remain costly or limited in scope. It also highlights privacy-by-construction as a possible tool when data may be sensitive.

D. Focus on efficient exact unlearning methods

  • Formal guarantees for unlearning remain concentrated in simpler or fine-tuned models, where exact unlearning is less computationally expensive.The authors argue that brittle empirical measurement warrants developing formal and provable metrics, even when they are more challenging to achieve.Source: D. Focus on efficient exact unlearning… · p. 10

E. Focus on privacy by construction

  • Because identifying what counts as public data is difficult and deletion can affect other users’ privacy, differential privacy may help when learning from potentially sensitive data.The paper notes that strict privacy definitions can be difficult to satisfy alongside utility.Source: E. Focus on privacy by construction · p. 11

F. Work towards formal definitions and provable metrics

  • Most LLM unlearning papers lack a formal objective beyond the function implicitly defined by their test sets; the authors suggest mutual information as a potentially more salient metric than test-query accuracy.They argue that formalizing and operationalizing such definitions for large models is important for trustworthy, useful guarantees.Source: F. Work towards formal definitions and… · p. 11

VII. CONCLUSION

  • The authors urge caution in interpreting benchmark results and conclude that reliable solutions in privacy-critical domains require formal guarantees beyond even exhaustive benchmarking.They suggest adding queries that depend on both forget and retain information, or expanding the query types in each benchmark.Source: §VII · p. 11

APPENDIX

The appendix describes ROUGE as a word-sequence similarity metric and ROUGE-L as using longest-common-subsequence overlap. It also specifies that β weights precision versus recall in F1 and is set to 1 in the experiments.

A. ROUGE score

  • ROUGE measures similarity between two sequences of words, while ROUGE-L uses their longest common subsequence.The appendix computes precision, recall, and F1 scores for ROUGE-L.Source: A. ROUGE score · p. 14
  • β weights precision versus recall in F1, and the experiments set β to 1.The appendix states that precision, recall, and F1 are computed for ROUGE-L.Source: A. ROUGE score · p. 14

Related papers

Loading 2410.02879v2…