Source-linked AI summary
TOFU: A Task of Fictitious Unlearning for LLMs
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, J. Zico Kolter
TL;DR
TOFU addresses the lack of well-defined, robust evaluation for unlearning private information from generative language models. It constructs a fictitious-author benchmark with complementary forgetting and utility metrics, finding that elementary baseline methods are largely unsuccessful. The benchmark focuses on entity-level forgetting and excludes behavior-level and in-context unlearning.
Problem
Unlearning evaluation for generative models is poorly defined, while LLMs trained on massive web corpora make it difficult to identify and remove specific information.
Method
TOFU creates a controlled synthetic-author task with forget and retain sets, multiple severities, and evaluation across forget quality and model utility.
Results
Existing baseline methods are largely unsuccessful because learning and unlearning are entangled, making it difficult to forget target data while preserving retain-set performance.
Takeaways & Limitations
TOFU provides a well-defined benchmark and aggregated metrics for studying whether unlearned models behave like models trained without the forget data.
Takeaways & Limitations
The framework covers entity-level forgetting but leaves instance-level unlearning for future work and excludes behavior-level and in-context unlearning.
Abstract
from arXiv · showhide
Large language models trained on massive corpora of data from the web can memorize and reproduce sensitive or private data raising both legal and ethical concerns. Unlearning, or tuning models to forget information present in their training data, provides us with a way to protect private data after training. Although several methods exist for such unlearning, it is unclear to what extent they result in models equivalent to those where the data to be forgotten was never learned in the first place. To address this challenge, we present TOFU, a Task of Fictitious Unlearning, as a benchmark aimed at helping deepen our understanding of unlearning. We offer a dataset of 200 diverse synthetic author profiles, each consisting of 20 question-answer pairs, and a subset of these profiles called the forget set that serves as the target for unlearning. We compile a suite of metrics that work together to provide a holistic picture of unlearning efficacy. Finally, we provide a set of baseline results from existing unlearning algorithms. Importantly, none of the baselines we consider show effective unlearning motivating continued efforts to develop approaches for unlearning that effectively tune models so that they truly behave as if they were never trained on the forget data at all.
1 Introduction
TOFU frames unlearning as a privacy-oriented task for generative language models, addressing poorly defined evaluation and the difficulty of separating forgetting from retained knowledge. It introduces a controlled benchmark, evaluates forget quality and model utility, and finds existing baseline methods largely unsuccessful.
- Web-trained LLMs can reproduce private information, motivating benchmarks that measure privacy and guide mitigation strategies.
- TOFU uses 200 fictitious author profiles and designates a subset for forgetting, isolating the information source that unlearning methods must remove.
- The benchmark defines three severities—forgetting 2, 10, or 20 authors—and requires compute that scales linearly with the number of forget samples.
- TOFU evaluates unlearning across forget quality and model utility, including a true-versus-false answer probability metric and comparison with retain-only reference models.
- Existing baseline methods are weak attempts because learning and unlearning remain entangled, making isolated forgetting difficult while preserving retain-set performance.
2 New Task: Fictitious Author Question Answering
TOFU frames unlearning as forgetting information about selected fictitious authors while preserving knowledge outside the forget set. It uses controlled synthetic biographies, severity-based splits, and multi-axis evaluation to make forgetting measurable.
- TOFU targets unlearning information about specific individuals rather than label-specific data in conventional NLP tasks.
- The benchmark uses GPT-4-generated biographies for fictitious authors, with 20 question-answer pairs per author and no pretraining remnants.Authors are generated from seeded attributes including birthplace, gender, birth year, genre, awards, and parents’ professions.
- Seeding author attributes reduces recurring artifacts in generated biographies, including repeated words such as ‘tides’ and ‘shadows’.The initial 50-author dataset showed concentrated birth years, recurring book-title vocabulary, and similar upbringings and writing styles.
- The evaluation combines utility metrics across datasets with Truth Ratio distributions and a KS-Test to assess forgetting quality.The datasets span the forget set, retain set, real authors, and world facts, providing different relevance distances from the forgotten data.
3 Baseline Unlearning Methods
The baseline section evaluates several post-finetuning methods that reduce performance on the forget set while attempting to preserve the retain set. The methods differ in objectives, retained-task information, and computational requirements.
- 3.1 Model Finetuning: Finetuning exposes pretrained LLMs to fictitious-author information by computing answer-token loss on the question-answer dataset.The loss is optimized with AdamW for five epochs, using warm-up during the first epoch and an effective batch size of 32.
- Baseline Unlearning Methods: Gradient Ascent maximizes training loss on the forget set so the model deviates from its original predictions.For each forget-set instance, the method reduces the likelihood of correct predictions.
- Baseline Unlearning Methods: Gradient Difference combines increased forget-set loss with an objective to maintain performance on the retain set.Retain examples are randomly sampled alongside forget examples to satisfy the compute constraint.
- Baseline Unlearning Methods: KL Minimization preserves similarity to the original model on the retain set while maximizing conventional loss on the forget set.The original and current models provide the prediction distributions used in the KL objective, with retain instances sampled under the compute constraint.
- Baseline Unlearning Methods: Preference Optimization aligns the model to refrain from revealing information about specific authors, while the original DPO objective is reported as unstable and difficult to optimize.Across the evaluated baselines, the authors describe existing methods as weak attempts because forgetting and retaining performance remain entangled.
4 Baseline Results
Across Phi and Llama-2-7B baselines, improving forget quality generally reduces model utility, while small forget sets remain difficult to unlearn effectively. The methods therefore fail to produce models both indistinguishable from retain-only models and useful on retained knowledge.
- Model comparison: On Llama-2-7B, model utility is overall higher than on Phi, but the same baseline trends remain and forget quality never exceeds 0.01 for 1% forget sets.The baseline methods fail to find useful models under this setting.
- Small forget sets: For 1% forget sets, trajectories are nearly vertical, but forget quality remains low and models remain distinguishable from retain-only models.None of the tested baselines crosses the p-value significance threshold of 0.05, even after up to 10 epochs.
- Larger forget sets: For forget sets larger than 1%, methods can achieve higher forget quality only alongside severe utility losses, making the resulting models unusable.Several 5% checkpoints show high forget quality, but some remain low on the utility scale.
- Trade-offs: All four methods reduce model utility as forgetting improves, producing an upward-left trade-off in forget quality versus utility.After two epochs, models can generate gibberish across all four evaluation datasets, even with oracle models or retain data.
- Knowledge entanglement: Fine-grained ROUGE results show that forgetting fictitious authors also affects pretrained knowledge, with the strongest declines closest to the forget data.Retain-set performance drops sharply, Real Authors performance also declines but remains higher, and World Facts stays relatively unchanged.
- Non-monotone behavior: Preference Optimization and Gradient Difference follow zig-zag trajectories, first sacrificing utility for forgetting before utility recovers as forget quality decays.The passage attributes this pattern to balancing forget and retain losses in an unstable way.
5 Discussion
The discussion identifies important scope boundaries and methodological limitations of TOFU while positioning the benchmark as a foundation for improved unlearning research. It also emphasizes that elementary unlearning attempts remain largely unsuccessful and require aggregated evaluation metrics.
- Scope boundaries: TOFU targets entity-level forgetting and leaves instance-level unlearning, which forgets a specific answer about a person, for future work.The authors state that entity-level unlearning is not yet clear and that instance-level unlearning is outside the current framework.
- Scope boundaries: The framework excludes behavior-level unlearning related to aligning models with human values and focuses instead on the Right to be Forgotten.The authors separate alignment-related unlearning from their privacy-oriented setting.
- Practical limitations: TOFU evaluates the execution of unlearning algorithms but assumes provided forget and retain sets, leaving real-world data selection as a separate challenge.The authors identify finding forget and retain sets as difficult practical problems while supplying both in the benchmark.
- Scope boundaries: The benchmark omits in-context unlearning, which is relevant when users can query but cannot modify model weights.The paper distinguishes this setting from weight modification and notes its relevance to API-based products and services.
- Benchmark design: The benchmark studies information learned during finetuning rather than pretraining, enabling controlled and inexpensive experiments but limiting realism.This design permits a retain-only model to serve as an upper bound for utility and as a reference for forget-quality evaluation.
- Conclusions and future work: Elementary unlearning attempts are largely unsuccessful, and their flaws become visible only through aggregating multiple metrics.The authors present TOFU as a well-defined task and metric suite intended to motivate stronger algorithms and future benchmark extensions.
A Kolmogorov-Smirnov Test Details
The KS-Test compares the empirical Truth Ratio distributions of unlearned and retain models, using their maximum CDF difference and a significance threshold to assess similarity.
- Dn,m = supx |FU(x) − FR(x)| measures the largest difference between unlearned and retain empirical CDFs.FU and FR are the empirical CDFs for n unlearned-model samples and m retain-model samples.
- The null hypothesis that both sample sets come from the same distribution is rejected when the KS statistic exceeds the chosen significance threshold.The threshold depends on significance level α and critical value c(α).
- The p-value is the smallest α at which the null hypothesis can be rejected, quantifying confidence that forget-set Truth Ratio distributions match.Forget quality is defined from this distributional similarity between unlearned and retain models.
B Hyperparameters
The experiments use AdamW, fixed learning-rate and epoch settings, and a first-epoch linear warmup, with 10^-5 selected for baseline methods.
- AdamW training uses learning rates of 10^-5, 10^-6, and 5 · 10^-7 in experiments, with 10^-5 chosen for baseline methods.The passage reports that 10^-6 produces no appreciable model-weight updates.
- Both finetuning and unlearning run for 5 epochs with linear warmup during the first epoch.Weight decay settings are reported as 0.01 and 0.
- Training uses AdamW with weight decay settings of 0.01 and 0.
C Preference Strings
This section contains many alternative model-response strings expressing uncertainty, lack of knowledge, or inability to answer; it also references KS-Test comparisons of Truth Ratio distributions.
- The strings repeatedly express uncertainty or lack of knowledge, such as “I’m not certain about that” and “I have no knowledge on that subject.”
- Several strings state that the model lacks training or database coverage for the requested information.
- Other strings directly state an inability or unwillingness to provide an answer.
- The referenced Table 3 compares KS-Test p-values for Llama-2-7B retain models trained on 90%, 95%, and 99% of the data.It evaluates Truth Ratio distributions on retain and forget data using set intersections.
D Continued Unlearning
Figure 9 shows zoomed-in unlearning trajectories over 10 epochs for experiments using the 1% forget set.
- Figure 9 plots extended unlearning trajectories across 10 epochs on the 1% forget set.
E Sanity Checks
The paper checks whether Model Utility and Forget Quality behave according to predefined expectations using KS-Test p-values and Truth Ratio distributions. The reported comparisons support the intended interpretation of these metrics across retain, forget, pretrained, finetuned, and random models.
- Model Utility: Model Utility is expected to be high for pretrained models and low for models with random weights.
- Forget Quality: Forget Quality is evaluated statistically using KS-Tests on Truth Ratio distributions.
- KS-Test expectations: High p-values are expected when comparing two retain models on Truth Ratio distributions from the intersected forget sets.
- KS-Test comparisons: The tests compare Truth Ratio distributions across retain and forget data for TOFU, pretrained, random, and finetuned models.
- Sanity-check results: The reported results show that retain models are indistinguishable from finetuned models on retain-set Truth Ratios and from pretrained models on forget-set Truth Ratios.
- Sanity-check results: Other KS-Test comparisons are reported to distinguish the expected model pairs appropriately.
F Knowledge Entanglement
The paper examines knowledge entanglement, where unlearning one target can also degrade unrelated knowledge. Figures 10–33 vary model, unlearning algorithm, and forget-set size to illustrate this phenomenon, while generation quality can deteriorate across evaluation sets.
- Knowledge entanglement: Knowledge entanglement causes models to forget other information while unlearning a specified target.The paper relates this phenomenon to catastrophic forgetting in continual learning.
- Experimental coverage: Figures 10–33 examine Llama-2-7B and Phi under gradient ascent, preference optimization, gradient difference, and KL Minimization.
- Observed degradation: Even with access to an oracle model or retain set, generation across all four evaluation sets shows decreasing ROUGE.
- Observed degradation: When evaluation curves overlap at extremely low ROUGE, the model begins producing gibberish.
- Evaluation implication: The authors use these observations to motivate multiple evaluation datasets for holistic unlearning assessment.