PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Unlearning

Yuanshun Yao, Xiaojun Xu, Yang Liu

arXiv:2310.10683v2cs.CLcs.AIcs.LG

TL;DR

LLMs can produce outputs that do not align with human values and policy, motivating ways to remove undesirable behavior. This paper studies unlearning from negative examples and reports better alignment performance than RLHF using only 2% of its computational time.

  • Problem

    How to perform unlearning on LLMs remains an early research question, while aligning their outputs with human values and policy is a major practitioner task.

    Source: §1 · pp. 1, 3

  • Method

    The paper studies unlearning by seeking a model whose outputs on undesirable prompts deviate from unwanted outputs in a dataset designated to forget.

    Source: §2 · p. 4

  • Results

    With only negative samples, unlearning achieved better alignment performance than RLHF using 2% of RLHF’s computational time.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Unlearning is a promising alignment approach when resources are limited and the priority is stopping undesirable outputs rather than producing desirable ones.

    Source: §1 · p. 3

    Hallucination unlearning remains harder, and the authors frame its goal as reducing in-distribution rather than eliminating general hallucinations.

    Source: §8 · p. 14

Abstract

from arXiv · show

We study how to perform unlearning, i.e. forgetting undesirable misbehaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) reducing hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in RLHF (RL from human feedback). (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. We show that if practitioners only have limited resources, and therefore the priority is to stop generating undesirable outputs rather than to try to generate desirable outputs, unlearning is particularly appealing. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

1 Introduction

The paper frames LLM unlearning as removing undesirable learned behaviors using only negative examples, and presents it as a resource-efficient alternative for aligning LLMs.

1 Introduction

  • Unlearning targets the impact of unwanted training samples using negative examples; these are cheaper to collect, and unlearning is efficient and especially useful when causal samples are known.The paper says negative examples can be gathered through user reporting or red teaming, with discovery highly automatable using an unaligned pretrained LLM.Source: §1 · p. 2
  • In low-resource alignment, the paper prioritizes stopping undesirable outputs over generating desirable ones, arguing that one harmful answer can destroy user trust.It argues that non-harmful outputs—including nonsensical or prompt-unrelated responses—can be the higher priority when human-written targets and compute are scarce.Source: §1 · p. 2
  • The paper demonstrates unlearning harmful responses, copyright-protected content without retraining from scratch, and wrong facts learned during training.For harmful-response removal, the stated goal is non-harmful—not necessarily helpful—outputs; for hallucinations, the goal is to forget wrong facts.Source: §1 · pp. 2–3
  • LLM unlearning is harder than classification-model unlearning because natural-language behaviors are ill-defined, models demand efficiency, and training data and ground-truth models are unavailable or costly.The paper notes that retraining to obtain ground-truth models is too expensive, making evaluation challenging.Source: §1 · p. 3
  • Despite using only negative samples, unlearning achieves better alignment performance than RLHF with only 2% of RLHF’s computational time.The authors present this as evidence that unlearning is promising for alignment when resources are limited.Source: §1 · p. 3

1.1 Related Work

2 Setting and Goal

The section defines LLM unlearning as moving outputs away from undesirable prompt-output examples while preserving performance on normal tasks. It sets four goals—effectiveness, generalization, utility, and low cost—under an assumption that desirable outputs are unavailable.

Setting

  • Unlearning seeks to make the model’s output on an undesirable prompt differ from its undesirable target, rather than reproduce the behavior characterized by the forget dataset.The forget dataset contains undesirable prompt-output pairs, such as harmful prompts and responses or attempts to extract copyrighted information.Source: §2 · p. 4
  • Unlike traditional discriminative-model unlearning, the goal is not to match a model retrained without the forget data, but to preserve the original model’s utility on other tasks.A normal dataset can support this preservation; its prompts may come from a different domain, and its responses may be AI- or human-generated.Source: §2 · pp. 4–5

Unlearned Data

  • Negative examples can be collected through user reports or internal red teaming, a procedure described as automatable and more efficient and less expensive than collecting positive outputs for RLHF.Positive outputs often require humans to write helpful, high-quality responses.Source: §2 · p. 4

Setting

  • Forget examples need not be exact training samples: they can represent broader concepts and should generalize to similar examples, improving robustness to paraphrasing attacks.Normal examples likewise need not be an exact subset of the original training data.Source: §2 · p. 5

Goal

  • The four goals are to forget undesirable outputs, generalize forgetting to similar prompts, preserve performance on normal prompts, and avoid retraining-level computational cost.Effectiveness is context-dependent: for a harmful prompt, the desired output is non-harmful. Utility means keeping outputs on normal prompts close to the original model.Source: §2 · p. 5

Remark

  • Without positive examples or an external model to generate them, the aim is to stop undesirable outputs, not necessarily produce helpful answers.For harmful prompts, acceptable outputs may be unrelated or nonsensical; for copyrighted content, they may be unrelated strings rather than polite responses.Source: §2 · p. 5

3 Preliminary

The preliminary section motivates gradient-ascent unlearning and identifies three differences from traditional classification unlearning: loss is a weak effectiveness signal, preserving normal-task utility is difficult, and normal-data format affects utility.

3.1 Why Gradient Ascent?

  • Gradient ascent fits negative-only unlearning by directly reducing harmful-token probabilities, costs about as much as finetuning, and is more tolerable for high-capacity LLMs.The authors also argue Hessian-based unlearning is too costly for LLMs and that GA’s coarse updates can be disruptive in smaller-capacity classification models.Source: §3.1 · p. 6

3.2 How Does LLM Unlearning Differ from Traditional Unlearning?

  • With gradient ascent, harmful-sample loss can rise as high as 60+ after ∼200 batches while harmful responses persist; the model only stops outputting them after ∼1000 batches, and authors propose randomly mismatched forget-input/response loss.Source: §3.2 · p. 6
  • After ∼1000 batches of gradient ascent, the model stops harmful responses but also produces nonsensical outputs on normal prompts, destroying utility.The authors find that preserving performance on normal prompts can be harder than forgetting harmful responses.Source: §3.2 · pp. 6–7, Table 2 · p. 7
  • Minimizing divergence between the unlearned model’s outputs on normal inputs and the original model works better for preserving normal performance than optimizing normal-data cross-entropy alone.The authors say this approach is like existing RLHF work [Ouyang et al., 2022, Touvron et al., 2023, Zheng et al., 2023, Holtzman et al., 2019].Source: §3.2 · p. 7
  • When normal and forget datasets use substantially different formats, the model can use prompt format as a shortcut instead of truly unlearning; matching their formats better preserves utility.For example, if the forget data uses Q&A format, the normal data should also use Q&A format.Source: §3.2 · p. 8

4 Method

The method combines losses for forgetting undesirable outputs and maintaining normal performance. It computes losses on generated response tokens conditioned on prompts, uses random responses to reinforce forgetting, and uses forward KL to preserve the original model’s utility.

4 Method

  • The method weights three losses with hyperparameters, and computing losses only on response tokens is more effective than computing them on prompts and responses.The three losses are for forgetting, random-response prediction, and preserving normal utility.Source: §4 · pp. 8–9
  • Gradient ascent on harmful response tokens forgets unlearned samples, while predicting random responses on unrelated prompts further reinforces forgetting.The loss excludes prompt tokens; random responses can be collected from a normal dataset.Source: §4 · pp. 8–9
  • Forward KL between the original and unlearned models’ predicted distributions preserves normal utility by encouraging coverage of the original model’s distribution.The method uses forward rather than reverse KL because it forces the unlearned model to cover all areas of the original model’s distribution.Source: §4 · pp. 8–9
  • The random-response loss can also help preserve normal utility and stabilize unlearning when gradients on prompts and responses are small.The paper reports the utility benefit empirically and points to Appendix A for detailed analysis.Source: §4 · p. 9
  • Continue unlearning for 3x-5x more batches after forgetting loss becomes abnormally high, and match normal-data format to the forget-data format.For harmful Q&A data from PKU-SafeRLHF, the paper uses TruthfulQA as normal data.Source: §4 · p. 9

5 Evaluation Design

The evaluation measures unlearning on targeted and similar unseen misbehavior prompts, alongside utility preservation on normal prompts.

Unlearning Performance

  • Unlearning is evaluated on both targeted and similar unseen misbehavior-causing prompts to test generalization beyond the prompts directly targeted.Unlearned prompts may or may not exactly appear in training data, depending on the unlearning task.Source: §5 · p. 9
  • Unlearning efficacy tracks reduced target misbehavior or copyrighted-information leakage, while diversity measures the percentage of unique output tokens.Source: §5 · p. 9
  • Following prior work [Lu et al., 2022], fluency measures generated-text perplexity on a reference LLM and is meaningful only when diversity is not extremely low; above 80% single-character repetition is labeled NM (Not Meaningful).Source: §5 · p. 10
  • The study excludes membership-inference metrics because LLM training corpora are often inaccessible and performing such attacks on LLMs remains difficult and under study.Membership inference attacks are a popular traditional unlearning metric [Shokri et al., 2017].Source: §5 · p. 10

Utility Preservation

  • For normal prompts from a different distribution, utility is assessed against the original LLM using reward models and BLEURT output similarity [Sellam et al., 2020].This treats the original model as ground truth because retraining is prohibitive, assuming that forgetting targeted samples does not affect normal outputs.Source: §5 · p. 10

6 Application: Unlearning Harmfulness

GA and GA+Mismatch nearly eliminate harmful responses and generalize to unseen harmful prompts while preserving normal-prompt utility; random mismatch improves that utility.

Setting

  • The unlearning setting uses only negative samples, and unlike traditional unlearning, those samples need not come from the LLM’s training set.The evaluation samples 200 unlearned-harmful, unseen-harmful, and normal prompts, and measures output similarity against the original model using maximum pairwise BLEURT.Source: §6 · pp. 10–11

Dataset and Model

  • The evaluation uses PKU-SafeRLHF harmful Q&A and TruthfulQA normal data across OPT-1.3B, OPT-2.7B, and Llama2-7B.Harmful samples are split into unlearned and unseen evaluation prompts; the baseline finetunes on remaining BookCorpus data, while the proposed methods test plain GA and GA with random mismatch.Source: §6 · p. 10

Results

  • GA and GA+Mismatch achieve near-zero harmful rates, generalize to unseen harmful prompts, and retain normal-prompt utility near the original model’s level.Because the method receives no helpful responses, mostly whitespace or nonsensical non-harmful outputs are expected; adding random mismatch improves utility reward and helps preserve coherent, linguistically meaningful outputs.Source: Table 3 · p. 11, §6 · p. 11

7 Application: Unlearning Copyrighted Contents

This application studies removing a requested copyright-protected corpus from an LLM after training, without retraining from scratch. GA and GA+Mismatch nearly eliminate leakage from unlearned extraction attempts while retaining normal-task utility.

Dataset and Model; Setting

  • The experiment fine-tunes pretrained LLMs on Harry Potter text, then splits that corpus into unlearned and test sets to study removal without retraining from scratch.The task measures leakage by comparing completions against the ground-truth text; nonsensical repeated-character outputs are expected because the methods use no positive examples of good completions.Source: §7 · pp. 11–13

Results

  • Both GA and GA+Mismatch reduce leakage on unlearned copyright-extraction attempts to nearly zero.The comparison uses Table 4’s leak-rate results; the paper notes that completions are mostly single-character repetitions, an expected outcome without positive examples of good completions.Source: Table 4 · p. 12, §7 · pp. 12–13
  • Both methods generalize to unseen extraction attempts, distinguishing copyright-related prompts from other prompts.The authors interpret this as the unlearned LLM learning to distinguish copyright-related prompts from other prompts.Source: §7 · p. 13
  • Both methods achieve normal-completion utility similar to the original LLM.BookCorpus serves as the normal completion dataset.Source: §7 · pp. 12–13
  • Adding mismatching loss preserves similar normal utility to plain GA while yielding relatively higher similarity to the original LLM’s outputs.The passage attributes a higher normal utility for GA+Mismatch than the original LLM to sampling randomness.Source: §7 · pp. 11, 13

8 Application: Reducing Hallucination

This application uses unlearning to reduce hallucinated answers by teaching models which question types to answer rather than teaching correct answers. Both GA variants reduce hallucinations on unlearned and in-distribution questions while preserving original-like answers out of distribution, but hallucination remains harder to unlearn than the earlier behaviors.

Approach

  • Because the goal is to avoid wrong answers, not guarantee correct ones, unlearning can generalize by distinguishing normal questions from questions resembling the unlearned ones.The section argues that learning this question-distribution distinction is easier than producing correct answers.Source: §8 · p. 13

Dataset and Model

  • The evaluation uses HaluEval hallucinated Q&A as negative samples and TruthfulQA as normal data, whose misleading and straightforward questions create an in-distribution versus out-of-distribution distinction.HaluEval negatives are split into 70% training, 10% validation, and 20% testing; the section does not assume the hallucinated Q&A samples were in the model’s training set.Source: §8 · p. 13

Setting

  • The hallucination rate counts test outputs whose BERTscore similarity to a hallucinated answer is 10% higher than their similarity to the correct answer.Source: §8 · p. 13

Results

  • Both GA and GA+Mismatch significantly reduce hallucination on unlearned questions, generalize to in-distribution questions, and preserve similar-to-original answers on out-of-distribution questions.The comparison is reported in Table 5; no cell values are provided in the supplied passages.Source: Table 5 · p. 14, §8 · p. 14
  • Hallucination remains harder to unlearn: its rate does not reach the roughly 10% level seen in the previous applications, so the goal should be reducing in-distribution hallucination rather than eliminating it generally.Source: §8 · p. 14

9 Ablation Studies

The ablations compare unlearning with RLHF and test templated outputs as an alternative to random outputs.

9.1 Comparing to RLHF

  • With negative samples alone, unlearning uses about 2% of full RLHF’s runtime, achieves a lower harmful rate than full RLHF, and a far lower rate than SFT.The comparison assumes RLHF also has expensively collected positive samples; unlearning’s runtime is comparable to mere fine-tuning.Source: §9.1 · pp. 14–15, Table 6 · p. 15
  • When the goal is to stop undesirable responses rather than generate desirable ones, negative-sample-only unlearning may be more appealing than RLHF.The reported non-harmful rate is promising for that goal.Source: §9.1 · p. 15

9.2 Templated Outputs

  • To avoid nonsensical outputs on harmful prompts, templated answers can replace random outputs during unlearning or serve as post-processing replacements for detected nonsensical strings.The authors also describe making the model output a template after it learns to respond differently to harmful prompts.Source: §9.2 · pp. 15–16
  • GA+Template achieves similar unlearning performance to GA and GA+Mismatch, with no significant difference in metrics from using templated rather than random answers.The experiment uses “I can’t assist it.” as the template.Source: §9.2 · pp. 15–16, Table 7 · p. 16

10 Conclusion and Future Work

The paper presents LLM unlearning as a promising approach for stopping undesirable outputs, especially when practitioners have limited resources for other alignment techniques. It explores three applications and identifies evaluation frameworks and influence-function methods as future directions.

10 Conclusion and Future Work

  • Unlearning removed harmful responses, copyrighted content, and hallucinations, and the experiments support its promise for aligning LLMs when resources for techniques such as RLHF are limited.The ablation study reports better alignment performance than RLHF using only a fraction of its computational time, despite relying only on negative samples.Source: §10 · p. 16
  • The paper takes a first step toward formalizing LLM unlearning through its setups, goals, and evaluations.The authors note that LLM unlearning differs substantially from traditional classification-model unlearning.Source: §10 · p. 16
  • Future work includes a unified, comprehensive framework for evaluating LLM unlearning and influence-function approaches with computational efficiency and theoretical guarantees.Source: §10 · p. 16

A Analysis on Random Mismatch Loss

Random mismatch loss uses grammatically intact random targets to help forget undesirable content without sacrificing coherent text generation, and its random term can help when GA’s forgetting-sample gradients are small.

A Analysis on Random Mismatch Loss

  • Random mismatch loss encourages forgetting the undesirable answer while preserving coherent text generation, and empirically helps retain normal utility.It forces the model to predict a random but grammatically intact answer, favoring removal of the undesirable content over reducing general text-generation ability.Source: A Analysis on Random Mismatch Loss · p. 20
  • GA’s first-order loss increase is positive but limited when the forgetting-sample gradient is small; adding a random term changes the loss increase calculation.The analysis models the change as ∇θℓ(x, y; θ) · ∆θ; with GA alone, ∆θ = λ · ∇θℓ(θ, x, y), while the random-term expression includes an additional gradient interaction.Source: A Analysis on Random Mismatch Loss · p. 20
  • The random-target intuition is that an irrelevant answer may, with some probability, lie in the opposite direction from the undesirable answer.The analysis suggests this can support unlearning even when the gradient on the forgetting sample is small.Source: A Analysis on Random Mismatch Loss · p. 20

B Experimental Settings … C.4 Templated Responses

The appendix provides hyperparameter settings for unlearning harmfulness, copyrighted content, and hallucinations, followed by generated-output examples across these tasks and a templated-response comparison.

B.1 Unlearning Harmfulness

  • Hyperparameter settings for unlearning harmfulness are summarized in Table 8.The passage identifies the table but does not provide its parameter values.Source: §B.1 · p. 20, Table 8 · p. 21

B.2 Unlearning Copyrighted Content

  • Hyperparameter settings for unlearning copyrighted content are summarized in Table 9.The passage identifies the table but does not provide its parameter values.Source: §B.2 · p. 20, Table 9 · p. 21

B.3 Reducing Hallucination

  • Hyperparameter settings for reducing hallucinations are summarized in Table 10.The passage identifies the table but does not provide its parameter values.Source: §B.3 · p. 20, Table 10 · p. 22

C.1 Unlearning Harmfulness

  • Tables 11–19 provide generated-text examples for unlearning harmfulness.For Llama 2, the appendix notes that the base text-completion model may continue with unnecessary questions after answering; these can be ignored.Source: §C.1 · p. 21, Table 19 · p. 31

C.2 Unlearning Copyrighted Content

  • Tables 20–28 provide generated-text examples for unlearning copyrighted content, limited to three examples per experiment to avoid overexposing copyrighted text.Source: §C.2 · p. 21

C.3 Reducing Hallucinations

  • Tables 29–37 show examples of generated text in unlearning harmfulness, with an untruthful-content warning; the passage does not identify them as reducing hallucinations.Source: §C.3 · p. 21

C.4 Templated Responses

  • Table 38 compares templated output with OPT-1.3B after unlearning harmfulness.The supplied passage identifies the comparison but does not give its outcome.Source: §C.4 · p. 21, Table 38 · p. 50

Related papers

Loading 2310.10683v2…