PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Unlearning

Yuanshun Yao, Xiaojun Xu, Yang Liu

arXiv:2310.10683v2cs.CLcs.AIcs.LG

TL;DR

The paper asks how LLMs can forget undesirable behaviors when alignment resources and training-data access are limited. It formulates LLM unlearning around negative examples, and reports successful applications with better RLHF alignment performance at 2% of RLHF’s computational time. The approach is presented as promising for stopping undesirable outputs, though hallucination unlearning remains harder and standard membership-inference evaluation is omitted.

  • Problem

    The paper addresses how to remove unwanted LLM behaviors, including harmful outputs, copyright leakage, hallucinations, and policy-inconsistent behavior, using limited information and resources.

  • Method

    The paper formulates unlearning as training an LLM with negative examples representing undesirable behaviors, while using normal data to preserve utility on unrelated tasks.

  • Results

    Across three applications, unlearning successfully removes harmful responses, erases copyrighted content, and reduces hallucinations; an ablation achieved better alignment performance than RLHF using 2% of its computational time.

  • Takeaways & Limitations

    Unlearning is a promising alternative to RLHF when practitioners primarily need to stop undesirable outputs and lack positive examples or substantial computational resources.

  • Takeaways & Limitations

    Hallucination unlearning remains harder than the other applications, and membership-inference metrics are excluded because LLM training corpora are often inaccessible and LLM MIA evaluation is non-trivial.

Abstract

from arXiv · show

We study how to perform unlearning, i.e. forgetting undesirable misbehaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) reducing hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in RLHF (RL from human feedback). (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. We show that if practitioners only have limited resources, and therefore the priority is to stop generating undesirable outputs rather than to try to generate desirable outputs, unlearning is particularly appealing. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

1 Introduction

LLM unlearning is proposed as an alignment approach for removing harmful responses, erasing requested copyrighted content, and reducing hallucinations. It is designed for settings with limited resources and only negative examples, where stopping undesirable outputs is prioritized.

  • Negative examples are cheaper and easier to collect through user reporting or red teaming than positive examples required by RLHF.Negative-example discovery can also be highly automated using pretrained, unaligned LLMs.
  • Unlearning is computationally efficient and especially effective when practitioners know which training samples cause unwanted behavior.Its cost is described as similar to LLM finetuning.
  • In low-resource settings, the paper prioritizes stopping undesirable outputs over generating desirable outputs when positive examples and computation are limited.The proposed fallback can be non-harmful but nonsensical or unrelated responses.
  • LLM unlearning targets harmful responses, copyright-protected content, and hallucinations learned during training.
  • LLM unlearning is more challenging than classification unlearning because language outputs are broader, efficiency requirements are higher, and training data and ground-truth models are often inaccessible.These constraints also make evaluation difficult because retraining is too expensive.
  • 2% of RLHF’s computational time was sufficient for unlearning to achieve better alignment performance despite using only negative samples.

2 Setting and Goal

The paper defines LLM unlearning as removing the influence of undesirable prompt-output pairs while preserving utility on unrelated tasks. Its setting uses negative forget data and optional normal data, without access to desirable outputs.

  • The forget dataset contains undesirable prompt-output pairs, and unlearning seeks outputs that deviate substantially from the unwanted outputs.The unlearned model should no longer produce behaviors characterized by the forget dataset.
  • Unlike traditional discriminative unlearning, the goal is not to reproduce a retrained model exactly but to preserve the original model’s utility on unrelated tasks.
  • Negative forget samples can be collected through user reporting or internal red teaming, which is more efficient and less expensive than hiring humans to write positive outputs.The collection procedure is described as highly automatable.
  • Forget prompts and outputs may represent broad concepts such as harmfulness or hallucination rather than exact training examples, requiring generalization to similar samples and paraphrasing attacks.
  • Normal data consists of benign prompts and responses from different domains, used to maintain performance on tasks outside the unlearning target.Normal data need not be an exact subset of the original training data.
  • Without positive samples or external models, the setting provides no information about what a desirable output should look like.

3 Preliminary

The section identifies three empirical differences that make LLM unlearning challenging: loss on forgotten samples poorly predicts effectiveness, utility degrades easily, and data format strongly affects preservation.

  • Key Difference 1: Gradient-ascent loss on unlearned samples can rise above 60 while harmful responses persist, so loss is a weak effectiveness indicator.After roughly 200 batches, harmful outputs remain despite the high loss.
  • Solution 1: Effective forgetting requires continuing unlearning for roughly 1,000 batches, well beyond the point where harmful-sample loss first appears promising.The method also adds a random-mismatch loss to facilitate forgetting.
  • Key Difference 2: Gradient ascent can eliminate harmful responses while also producing nonsensical outputs on normal prompts, damaging model utility.This failure appears after roughly 1,000 batches in the reported example.
  • Solution 2: Minimizing divergence from the original LLM preserves normal performance better than merely optimizing cross-entropy on normal data.The comparison is motivated by the difficulty of maintaining normal performance during unlearning.
  • Key Difference 3: When forgetting and normal datasets use substantially different formats, the LLM can rely on prompt format shortcuts instead of truly unlearning the concept.Matching the normal-data format to the forgetting-data format better preserves normal utility.

4 Method

The method combines gradient ascent on forgotten outputs, random mismatching to reinforce forgetting, and forward-KL preservation of the original model’s normal behavior.

  • Overall objective: The update combines weighted forgetting, random-mismatch, and normal-performance losses during unlearning.The weights ϵ_i control the relative contributions of the three losses.
  • Loss definitions: The per-example loss is cross-entropy on output tokens y conditioned on prompt x and previously generated tokens.The predicted token probability is h_θ(x, y_<i), and the loss sums over output positions.
  • Forgetting losses: The forgetting loss applies gradient ascent to the target outputs y_fgt, while random mismatch pairs forgotten prompts with unrelated responses.Random responses are drawn from a set with no connection to the forgotten prompts.
  • Utility preservation: The normal-data loss compares the current model’s predicted distribution with the original LLM using forward KL divergence.This term is intended to preserve utility on normal prompts.
  • Design choices: The design computes ascent or descent only on output tokens, continues unlearning after high forgetting loss, and matches normal and forgotten data formats.These choices are presented as the main lessons from the empirical observations.

5 Evaluation Design

The evaluation separates forgetting performance from remaining-model utility, testing harmfulness-related behavior on seen and unseen prompts while comparing normal outputs with the original LLM.

  • Evaluation scope: Evaluation measures two categories: performance on unlearned samples and utility on remaining samples.The utility comparison uses normal prompts from a different distribution than unlearned prompts.
  • Generalization: The evaluation includes both unlearned prompts and unseen similar prompts to test whether forgetting generalizes beyond the exact examples.This distinguishes concept-level generalization from forgetting only individual samples.
  • Unlearning metrics: Unlearning efficacy measures reduced unwanted behavior, while diversity and fluency assess whether generated outputs remain informative and reasonable.Fluency is based on perplexity from a reference LLM and is meaningful only when diversity is not extremely low.
  • Evaluation limitation: Membership-inference metrics are omitted because the full training corpus is often inaccessible and LLM membership-inference evaluation remains difficult.The authors describe LLM membership inference as an ongoing research area.
  • Utility metrics: Normal-prompt utility is measured with reward-model scores and BLEURT output similarity against the original LLM.The original model serves as ground truth because retraining is prohibitively expensive.

6 Application: Unlearning Harmfulness

In harmfulness experiments, gradient ascent methods achieve near-zero harmful rates and generalize to unseen harmful prompts, while random mismatch improves preservation of normal utility.

  • Setup: The experiments use PKU-SafeRLHF harmful Q&A as forgotten data, TruthfulQA as normal data, and OPT-1.3B, OPT-2.7B, and Llama2-7B models.Forgotten data are split into unlearned and unseen harmful samples for evaluation.
  • Setup: The harmful rate is the primary unlearning-efficacy measure, while a reward model evaluates utility on TruthfulQA answers.The PKU moderation model flags harmful outputs.
  • Results: GA and GA+Mismatch achieve near-zero harmful rates and generalize to unseen harmful prompts.The results indicate forgetting extends beyond the exact harmful prompts used for unlearning.
  • Results: Both methods retain normal-prompt utility at a level similar to the original model, although outputs may be nonsensical when only negative examples guide unlearning.The paper treats nonsensical non-harmful outputs as the best expected outcome without helpful responses.
  • Ablation: Adding random mismatch improves utility reward compared with plain gradient ascent, likely by preserving coherent linguistic output.The reported explanation is that grammatically correct but semantically disconnected responses maintain coherent language ability.

7 Application: Unlearning Copyrighted Contents

The paper evaluates unlearning for removing copyrighted text without retraining from scratch, finding that GA and GA+Mismatch nearly eliminate leakage while preserving normal completion utility.

  • Motivation: Copyright unlearning removes requested content without retraining the LLM from scratch.The setting uses a copyrighted corpus and studies removal after training.
  • Dataset and Model: The evaluation uses Harry Potter text for unlearning, with BookCorpus as normal completion data.The model is first fine-tuned on the copyright corpus, then evaluated on held-out copyright and normal book-text data.
  • Setting: Leakage is measured by the percentage of extraction prompts whose completions exceed a BLEU-based leakage threshold.The threshold is set from randomly sampled corpus sentences, and the resulting metric is the leak rate.
  • Results: Both GA and GA+Mismatch reduce leak rates on unlearned extraction attempts to nearly zero.The table caption also reports that both methods distinguish copyright-related prompts from other prompts.
  • Results: Both methods generalize to unseen extraction attempts while maintaining utility similar to the original LLM on normal completion.Adding mismatching loss yields similar normal utility to plain GA but relatively higher similarity to original outputs.

8 Application: Reducing Hallucination

The paper applies unlearning to hallucinated answers, aiming to reduce wrong responses and generalize beyond exact prompts. GA and GA+Mismatch reduce hallucinations on related questions while preserving behavior on out-of-distribution questions, although hallucination remains harder to eliminate.

  • Motivation: Hallucination unlearning targets factually wrong answers without assuming the unlearned Q&A samples appeared in training data.The goal is to make the model unlearn wrong answers associated with fact-related questions.
  • Approach: The method generalizes by learning which questions to answer normally and which resemble questions associated with unlearned wrong answers.The paper frames refusing incorrect answers as easier than generating factually correct answers.
  • Results: The methods remove hallucinations on related in-distributional questions while maintaining answers similar to the original LLM on out-of-distributional questions.The distinction relies on a distribution shift between misleading HaluEval questions and benign TruthfulQA questions.
  • Setting: The evaluation defines hallucination rate using BERTScore comparisons between generated outputs and hallucinated or correct answers.An answer is hallucinated when similarity to the hallucinated answer is 10% higher than similarity to the correct answer.
  • Results: Both GA and GA+Mismatch significantly reduce hallucination rates and generalize de-hallucination to in-distribution questions.The table caption reports that both methods distinguish in-distributional from out-of-distributional questions.
  • Limitation: Hallucination remains harder than the previous applications, with rates not reaching the similar low level of ∼10%.The paper therefore frames the goal as reducing in-distributional hallucination rather than eliminating general hallucination.

9 Ablation Studies

The ablations compare unlearning with RLHF and test templated refusal outputs. Unlearning achieves stronger harmfulness alignment at a fraction of RLHF’s runtime, while templated outputs preserve similar unlearning performance.

  • Comparing to RLHF: The RLHF comparison gives RLHF positive and negative samples, while unlearning uses only negative samples.The comparison is therefore made despite RLHF having access to expensively collected positive examples.
  • Comparing to RLHF: 2% of RLHF’s runtime is required by unlearning on a single NVIDIA A100 SXM4 80 GB GPU.The cost is comparable to mere fine-tuning.
  • Comparing to RLHF: Unlearning achieves a lower harmful rate than full RLHF and a far lower harmful rate than SFT.This result is reported for harmfulness unlearning on OPT-1.3B with PKU-SafeRLHF data.
  • Comparing to RLHF: When the priority is stopping undesirable outputs rather than generating desirable ones, unlearning may be more appealing than RLHF.This setting matches unlearning’s use of negative samples without positive responses.
  • Templated Outputs: Templated answers can replace random outputs on harmful prompts without changing the unlearning optimization.The paper uses “I can’t assist it.” as a template and reports similar unlearning performance to GA and GA+Mismatch.
  • Templated Outputs: A post-processing heuristic can replace nonsensical outputs with templated strings after unlearning.The heuristic checks whether generated text is nonsensical before substitution.

10 Conclusion and Future Work

The paper presents LLM unlearning as an alignment approach for removing harmful responses, copyrighted content, and hallucinations, particularly when resources are limited. It identifies unified evaluation and theoretically grounded influence-function approaches as future directions.

  • Conclusion: Unlearning successfully targets harmful responses, copyrighted content, and hallucinations across three alignment scenarios.The paper frames unlearning as a way to stop undesirable outputs when practitioners lack resources for other alignment techniques.
  • Conclusion: Unlearning can outperform RLHF in alignment performance despite using only negative samples and a fraction of RLHF's computational time.
  • Future Work: A unified and comprehensive evaluation framework is needed because LLM unlearning differs substantially from traditional classification-model unlearning.
  • Future Work: Influence-function approaches combining computational efficiency with theoretical guarantees are identified as another future-work direction.

A Analysis on Random Mismatch Loss

The analysis explains why random mismatch loss supplements gradient ascent: it can preserve coherent text while enabling forgetting even when forgetting-sample gradients are small. The resulting update combines the forgetting gradient with a random-answer gradient.

  • Random Mismatch Loss: Random mismatch loss encourages grammatically intact random answers, reducing the risk that unlearning harms the model's general ability to generate coherent text.The authors report empirically that this loss helps preserve normal utility.
  • Gradient Analysis: Gradient ascent alone can have limited unlearning effect when the forgetting-sample gradient norm approaches zero.The analysis uses the loss increase as a proxy for unlearning effectiveness.
  • Update Rule: The update combines the forgetting gradient and a random mismatch gradient: ∆θ = λ · (∇θℓ(x, y; θ) − ∇θℓ(x, yrdn; θ)).
  • Gradient Analysis: Even with a small forgetting gradient, the random term can support a positive unlearning direction when its loss gradient points oppositely to the undesirable answer.The intended random answer is irrelevant but may oppose the undesirable answer with some probability.

B Experimental Settings

The experimental appendix records hyperparameter settings and generated-text examples for harmfulness, copyrighted-content, and hallucination unlearning. It evaluates OPT-1.3B, OPT-2.7B, and Llama 2 (7B) using unlearned prompts and separate harmful and normal test prompts.

  • Hyperparameter Settings: Hyperparameter settings are provided separately for harmfulness, copyrighted-content, and hallucination unlearning.
  • Generated Examples: Generated-text examples cover harmfulness, copyrighted content, and hallucination-related experiments.The appendix also includes a comparison between unlearning harmfulness and templated output.
  • Hallucination Reduction: Additional hallucination tables document normal-test results for OPT-2.7B and Llama 2 (7B).
Loading 2310.10683v2…