PaperlayerView exact revision on arXiv

Source-linked AI summary

DUSK: Do Not Unlearn Shared Knowledge

Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, Soeun Kim, Seungju Han, Youngjae Yu, Albert No

arXiv:2505.15209v3cs.CL

TL;DR

Most unlearning evaluations assume forget and retain data are disjoint, despite real-world overlap. DUSK benchmarks unlearning with shared and unique content, and its evaluation finds that methods often struggle to remove forget-specific knowledge without disrupting shared information.

  • Problem

    Existing unlearning evaluations often assume forget and retain sets are disjoint, although documents can contain overlapping information.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    DUSK benchmarks unlearning under data overlap by distinguishing document-specific knowledge from shared knowledge that should be retained.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Evaluated methods often struggle to separate forget-specific knowledge from shared information, leading to unintended degradation of retain data.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    DUSK offers a structured framework for evaluating unlearning in realistic multi-source settings.

    Source: §5 · p. 9

    DUSK uses synthetic data that may not fully reflect the linguistic richness of real-world corpora.

    Source: D Limitations · p. 20

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed in real-world applications, raising concerns about the unauthorized use of copyrighted or sensitive data. Machine unlearning aims to remove such 'forget' data while preserving utility and information from the 'retain' set. However, existing evaluations typically assume that forget and retain sets are fully disjoint, overlooking realistic scenarios where they share overlapping content. For instance, a news article may need to be unlearned, even though the same event, such as an earthquake in Japan, is also described factually on Wikipedia. Effective unlearning should remove the specific phrasing of the news article while preserving publicly supported facts. In this paper, we introduce DUSK, a benchmark designed to evaluate unlearning methods under realistic data overlap. DUSK constructs document sets that describe the same factual content in different styles, with some shared information appearing across all sets and other content remaining unique to each. When one set is designated for unlearning, an ideal method should remove its unique content while preserving shared facts. We define seven evaluation metrics to assess whether unlearning methods can achieve this selective removal. Our evaluation of nine recent unlearning methods reveals a key limitation: while most can remove surface-level text, they often fail to erase deeper, context-specific knowledge without damaging shared content. We release DUSK as a public benchmark to support the development of more precise and reliable unlearning techniques for real-world applications.

1 Introduction

DUSK addresses the gap between disjoint-set evaluations and real-world data overlap by testing whether unlearning can remove forget-specific content while preserving shared facts. Its evaluation of nine methods finds that separating forget-specific information from retained knowledge remains difficult.

1 Introduction

  • DUSK targets realistic overlap, where unlearning a document’s unique information should not disrupt widely supported facts also present in the retain set.For example, differently phrased documents can convey the same earthquake’s location, magnitude, and timing.Source: §1 · pp. 1–2
  • DUSK constructs five documents from 120 synthetic professor profiles, each with 12 document-unique profiles and 60 shared profiles, to test selective removal.The benchmark defines seven metrics: Verbatim Memorization, Unique Forget Knowledge, Shared Knowledge, Unique Retain Knowledge, Downstream Capability, Privacy Leakage, and Retain Deviation.Source: §1 · p. 2
  • Nine evaluated unlearning methods often fail to separate forget-specific content from retained knowledge, degrading shared knowledge and potentially compromising overall utility.The findings identify precise disentanglement of forget-specific information from retained knowledge as a fundamental challenge in current approaches.Source: §1 · p. 2
  • DUSK is released as a public benchmark for evaluating unlearning in real-world scenarios.It is designed to support further research on precise removal of sensitive information while preserving shared knowledge.Source: §1 · p. 2

2 Related Work

Related work covers LLM unlearning methods and benchmarks, but existing benchmarks largely overlook documents that mix forget-specific and retained knowledge. DUSK addresses this gap with a multi-source unlearning benchmark.

Machine Unlearning in LLMs: Methods and Applications

  • LLM unlearning seeks to remove forget-data influence while preserving retain performance, commonly using fine-tuning with gradient ascent or preference optimization.Applications include removing copyrighted content, sensitive or harmful knowledge, mitigating bias, and model editing; guardrail-based and in-context approaches aim to scale unlearning, while studies report fragility and practical reliability challenges.Source: §2 · pp. 2–3

Machine Unlearning in LLMs: Benchmarks

  • Existing benchmarks largely assume forget documents contain only removable information, whereas DUSK evaluates settings where forget-specific and retained knowledge coexist.Earlier benchmarks range from entity-specific forgetting (WHP) and hazardous-knowledge removal (WMDP) to synthetic author profiles (TOFU), multidimensional evaluation (MUSE), verbatim-only forgetting (CoTAEval), and inaccessible-corpus evaluation of real-world knowledge (RWKU).Source: §2 · p. 3
  • LLM unlearning benchmarks expanded from entity-specific and hazardous-knowledge tasks to synthetic, multidimensional, verbatim-only, and real-world evaluation settings.WHP [12] targets entity-specific forgetting; WMDP [39] addresses biosecurity and cybersecurity; TOFU [45] uses synthetic author profiles; MUSE [52] assesses six dimensions; CoTAEval [56] targets verbatim memorization; and RWKU [33] evaluates real-world knowledge without corpus access.Source: §2 · p. 3

DUSK Benchmark

DUSK frames unlearning as selective removal in overlapping corpora: remove knowledge unique to the requested document while preserving shared facts and information unique to retained sources. It instantiates this setting with multi-style profiles and evaluates forgetting, retention, and distributional behavior.

3.1 Problem Setting

  • Each document contains shared knowledge to preserve and document-specific knowledge to forget, while unlearning must also preserve information unique to other retained documents.This reflects the challenge of multi-source settings, where forget and retain content is not cleanly separable.Source: §3.1 · p. 3

3.2 Problem Formulation and Notations

  • Unlike the common assumption that forget and retain knowledge do not overlap, DUSK uses 120 fictitious profiles across five documents, with 60 shared profiles and 12 unique profiles per document.A separate holdout set contains 120 professors excluded from training the Retrain and Target models.Source: §3.2 · p. 4

3.3.1 Data Generation Pipeline

  • DUSK generates five stylistically distinct documents from GPT-4-synthesized profiles, designating D1 as the forget set and the other four as retain data.The styles are Chronological, Feature Story, Interview, Inverted Pyramid, and Listicle; four rounds of prompt refinement address skewed attribute distributions, including an overrepresentation of 2010 graduation years.Source: §3.3.1 · pp. 4–5

3.4.1 Forget Assessment

  • Forget Assessment tests whether models reproduce forget-document wording and retain knowledge unique to that document, using text-similarity metrics and targeted questions.Verbatim Memorization uses ROUGE-1, ROUGE-L, their Recall variants, Levenshtein Distance, LCS, and Cosine Similarity; Unique Forget Knowledge uses ROUGE-L, with lower overlap indicating more effective unlearning.Source: §3.4.1 · p. 5

3.4.2 Retain Assessment

  • Retain Assessment measures preservation of shared knowledge, retain-exclusive knowledge, and downstream capabilities after unlearning.Shared Knowledge and Unique Retain Knowledge use ROUGE-L against ground-truth answers; capabilities are assessed on MMLU, ARC-c, GSM8K, TriviaQA, TruthfulQA (MC1), and BBQ.Source: §3.4.2 · pp. 5–6

3.4.3 Distributional Assessment

  • Distributional Assessment uses Min-K%++ membership inference to measure Privacy Leakage on forget versus holdout data and Retain Deviation on retain data.Privacy Leakage near zero indicates forget data is treated similarly to holdout data, while positive values signal over-unlearning and negative values under-unlearning; increasing Retain Deviation signals divergence from the model’s original behavior on retained data.Source: §3.4.3 · pp. 6–7

4 Experiments

Experiments evaluate nine unlearning configurations on LLaMA-3-8B and find that suppressing forget-set text does not reliably erase its deeper facts or preserve overlapping shared knowledge. General downstream capabilities remain relatively stable, but realistic overlap creates tradeoffs between forget-set removal and retain-set behavior.

4.1 Unlearning Methods

  • The experiments combine five removal methods—GA, NPO, RMU, TV, and TAU—with GD or KL utility-preserving objectives for nine configurations.GA maximizes forget-set loss; NPO treats forget samples as negative preferences; RMU alters forget-set representations; TV subtracts fine-tuning parameter changes; TAU combines selective ascent and task-vector subtraction. GD preserves retain-set prediction loss, while KL aligns predictions with a Target model.Source: §4.1 · p. 7

4.2 Experimental Setup

  • Experiments start from LLaMA-3-8B, use a full-corpus fine-tuned Target and retain-only Retrain model, and stop unlearning when URK first falls below 70.All methods use AdamW with learning rate 1 × 10−5, batch size 32, and a first-epoch warm-up; the stopping rule standardizes utility levels for comparison.Source: §4.2 · p. 8

4.3.1 Forget Assessment Results.

  • Most methods suppress verbatim overlap, with TAU achieving the largest reductions, but they largely fail to erase unique forget facts that models continue to answer correctly.Verbatim Memorization is assessed with ROUGE, Levenshtein distance, LCS, and Cosine similarity; UFK questions rely exclusively on information from the forget set.Source: §4.3.1 · p. 8

4.3.2 Retain Assessment Results.

  • Nearly all methods substantially degrade shared knowledge, and SK accuracy falls more than URK across most methods, despite relatively stable general downstream-task performance.The six downstream tasks are MMLU, ARC-c, GSM8K, TriviaQA, TruthfulQA, and BBQ; knowledge close to the forget set remains more vulnerable to collateral forgetting than broad capabilities.Source: §4.3.2 · pp. 8–9, Table 1 · p. 8

4.3.3 Distributional Assessment Results.

  • No method fully removes forget-set influence while preserving retain-set behavior: most over-unlearn with rising retain deviation, while NPO under-unlearns and also raises deviation.The results motivate jointly monitoring Privacy Leakage and Retain Deviation when forget and retain data overlap.Source: §4.3.3 · p. 9

5 Conclusion · A Details of

DUSK evaluates unlearning when forget and retain data overlap by separating unique from shared knowledge. Experiments find that methods often remove verbatim content but struggle to preserve shared information while removing forget-specific knowledge.

5 Conclusion

  • Many methods remove verbatim content but struggle to disentangle forget-specific knowledge from shared information, degrading retain data.This unintended degradation occurs when methods fail to separate knowledge specific to the forget set from information shared with the retain set.Source: §5 · p. 9
  • DUSK benchmarks machine unlearning in realistic multi-source scenarios where forget data overlaps with retain data.Source: §5 · p. 9
  • DUSK explicitly separates unique and shared knowledge to provide a fine-grained testbed for assessing unlearning performance.Source: §5 · p. 9
  • The benchmark aims to advance more precise and reliable unlearning methods.The paper presents DUSK as a foundation for bridging theoretical formulations and real-world applications.Source: §5 · p. 9

A.1 Dataset Construction Details

DUSK’s dataset uses fictional professor biographies to create style-varied profiles with shared and unique knowledge. Its construction controls attribute diversity and balances training instances across the two knowledge sets.

A.1 Dataset Construction Details

  • GPT-4 generated 20 question–answer pairs for each of 120 fictional professors, yielding 2,400 pairs for diverse biographical, academic, and professional attributes.The researchers checked the generated professor names for duplicates. The QA prompt required each question to name the professor and answers to be one word or a compound noun.Source: §A.1 · pp. 13, 15
  • Prompt refinement assigned 60 nationalities and eight belief systems, sampled temporal attributes uniformly, and produced more balanced distributions than the initial outputs.The initial outputs showed mode collapse in attributes such as nationality and employment year; the refined distributions were more balanced across nationality, religion, and employment year.Source: §A.1 · p. 13, Figure 6 · p. 14, Figure 7 · p. 14
  • Each professor’s QA-derived information produced five profiles—in Chronological, Feature Story, Interview, Inverted Pyramid, and Listicle styles—for 600 profiles total.The profile-generation prompt required biographies of at least 300 words constructed exclusively from the provided QA pairs.Source: §A.1 · pp. 13, 15
  • The shared set contains 60 professors across five style-specific documents, while the unique set groups the other 60 professors’ five-style profiles into five documents.Each unique-set document contains 12 professors with five profiles each; this arrangement gives every professor the same total number of training instances across styles.Source: §A.1 · pp. 13–14

A.2 Example Data Instances

Table 3 illustrates how the same facts can appear in distinct document styles, creating a multi-source unlearning challenge: remove forget-set-exclusive information while preserving semantically aligned content in retain sources.

A.2 Example Data Instances

  • The benchmark presents five formats—Chronological, Feature Story, Interview, Inverted Pyramid, and Listicle—that express the same core information through different narrative styles.Table 3 illustrates this with the fact that Ikebana is Professor Miyashimizu’s hobby.Source: §A.2 · pp. 15–16, Table 3 · p. 16
  • Knowledge explicitly forgotten in one source may persist implicitly in stylistically distinct sources that convey semantically aligned content.The examples show that variation in tone, structure, and surface form does not necessarily change the underlying information.Source: §A.2 · p. 16, Table 3 · p. 16
  • Effective unlearning must remove information exclusive to the forget set while preserving semantically aligned content that also appears in the retain set.Source: §A.2 · p. 16

B Experiment Details … B.2 Evaluation Metric Definitions

The experiment details describe unlearning approaches that target forget-set content while preserving utility, then define metrics for testing surface-level memorization, unique and shared knowledge, and downstream capabilities.

B.1 Unlearning Baseline Methods

  • The baseline objective is to remove knowledge from the forget set Df while preserving general utility.This frames the methods in the next subsection.Source: §B.1 · p. 16

Unlearning Methods.

  • GA increases prediction error on Df, NPO treats forget samples as negative examples, and GD/KL regularize on retain data to preserve general capabilities.GD uses retain-set prediction loss, while KL encourages output distributions to remain similar to the Target model on the retain set.Source: Unlearning Methods · pp. 16–17
  • RMU pushes forget-set activations toward a random direction and aligns retain activations with a frozen Target model, updating only layers 5–7 in this implementation.The paper sets l = 7, updates layers l −2, l −1, and l, and freezes all other layers.Source: Unlearning Methods · pp. 16–17
  • Task Vector subtracts forget-associated weight updates, while TAU combines selective gradient ascent with task-vector subtraction to reduce memorization.The implementation runs SGA for five epochs before subtracting a task vector obtained by retraining on Df.Source: Unlearning Methods · p. 17

B.2 Evaluation Metric Definitions

  • Verbatim Memorization (VM) tests whether a model regenerates exact forget-document spans by comparing continuations with ground-truth suffixes using surface and semantic similarity metrics.Metrics include ROUGE-1, ROUGE-L F1 and Recall, Levenshtein distance, LCS, and sentence-embedding cosine similarity.Source: §B.2 · pp. 17–18
  • QA metrics test forget-set-unique knowledge with ROUGE-L on Kf ∖Kr, shared knowledge on Kf ∩Kr, and retain-set-unique knowledge with accuracy on Kr ∖Kf.Shared-knowledge answers are supported by both sources.Source: §B.2 · p. 18
  • Downstream Capability (DC) averages performance across MMLU, ARC-c, GSM8K, TriviaQA, TruthfulQA (MC1), and BBQ to assess broader utility.Evaluation uses lm-evaluation-harness3 with default settings; the tasks reflect reasoning, factuality, and robustness.Source: §B.2 · p. 18

B.3 Experimental Setup … C.2 Downstream Capability

TAU leads on verbatim unlearning but sacrifices downstream capability, while GA, GAGD, and TV preserve utility near the Retrain baseline. The experiments used common optimization settings and required approximately 2,500 GPU hours.

B.3 Experimental Setup

  • Task arithmetic and preference optimization methods use loss-function hyperparameters α and β, respectively; forget and regularization loss coefficients are both 1.0.All methods use a 1 × 10−5 learning rate with AdamW for fair comparison; selected epochs and α/β settings are summarized in Table 4.Source: §B.3 · p. 18, Table 4 · p. 18

B.4 Hardware Specification

  • Experiments and method development used 512 CPU cores, eight Nvidia RTX L40S GPUs with 48GB each, and 1024 GB of RAM.The work required approximately 2,500 GPU hours across experiments, evaluations, analyses, and development.Source: §B.4 · p. 18

C.1 Verbatim Memorization

  • TAU achieves the strongest verbatim unlearning across all reported metrics, with GA and its variants also performing strongly.Metrics include ROUGE-1 and ROUGE-L, LCS, cosine similarity, and Levenshtein distance.Source: Table 6 · p. 19, §C.1 · p. 19
  • RMU and NPO retain relatively high verbatim memorization, with cosine similarity scores higher than Retrain.This indicates insufficient removal of verbatim traces.Source: Table 6 · p. 19, §C.1 · p. 19

C.2 Downstream Capability

  • GA, GAGD, and TV average above 0.40 downstream, close to Retrain at 0.4055, while most methods show only slight average degradation.TAU has a notable utility drop, especially on GSM8K and TruthfulQA, illustrating a trade-off between unlearning and capability preservation.Source: §C.2 · p. 19

C.3 Distributional Assessment

The distributional assessment finds that methods struggle to unlearn precisely when forget and retain sets overlap: most either over- or under-unlearn, and only RMU preserves retain behavior within the acceptable range.

C.3 Distributional Assessment

  • Only RMU keeps Retain Deviation within the acceptable range, but its Privacy Leakage indicates under-unlearning; all other methods fail to preserve retain behavior.The ideal bound for Retain Deviation is 5%.Source: Table 8 · p. 20, §C.3 · p. 20
  • 86.1 to 128.0 positive Privacy Leakage scores for GA, GAGD, and GAKL indicate over-unlearning.The passage reports this range as an example for the methods, not as method-specific values.Source: Table 8 · p. 20, §C.3 · p. 20
  • NPO’s −45.0 and RMU’s −98.6 Privacy Leakage scores indicate under-unlearning.Source: Table 8 · p. 20, §C.3 · p. 20
  • TV’s 49.0 and TAU’s 47.2 Retain Deviation scores show extreme deviations.Source: Table 8 · p. 20, §C.3 · p. 20

D Limitations · E Broader Impact

DUSK offers a structured framework for evaluating unlearning under multi-source overlap, but its synthetic data, clean forget/retain distinctions, and document-level focus limit its scope. Selective unlearning may support privacy and user control, while also creating risks that call for transparent, reproducible, robust methods.

D Limitations

  • DUSK uses synthetic data that may not capture the linguistic richness of real-world corpora.The benchmark is carefully designed to represent diverse attributes and overlapping content.Source: D Limitations · p. 20
  • DUSK assumes clearly defined forget and retain sets, which may diverge from settings where those distinctions are less precise.Its current evaluation focuses on document-level unlearning; entity- or attribute-level removal remains an open challenge.Source: D Limitations · p. 20
  • Despite these limitations, DUSK provides a structured framework for assessing trade-offs between knowledge retention and removal in multi-source settings.It is presented as a step toward evaluating unlearning in more realistic overlap scenarios.Source: D Limitations · p. 20

E Broader Impact

  • By distinguishing unique from shared knowledge, DUSK could support precise removal of sensitive information while preserving general knowledge, aligning with privacy regulations such as GDPR.The benchmark is described as having potential to improve data privacy and user control in machine learning.Source: E Broader Impact · p. 21
  • Selective removal of documents or entities could be exploited to suppress perspectives or manipulate historical records.This is identified as a potential risk of the approach.Source: E Broader Impact · p. 21
  • Unlearning can cause unintended knowledge loss, potentially affecting AI-system reliability and fairness.Source: E Broader Impact · p. 21
  • Risk mitigation calls for unlearning methods that are effective, transparent, reproducible, and robust against adversarial manipulation.Future work should also consider the environmental impact of training large models and potential biased outcomes in multi-source data settings.Source: E Broader Impact · p. 21
Loading 2505.15209v3…