PaperlayerView exact revision on arXiv

Source-linked AI summary

Offset Unlearning for Large Language Models

James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, Muhao Chen

arXiv:2404.11045v2cs.CLcs.AIcs.LG

TL;DR

LLMs can memorize sensitive training-corpus information, raising ethical and legal concerns, while unlearning aims to remove target data without harming performance outside its scope. δ-Unlearning estimates a logit offset from two smaller models and applies it to a black-box LLM; experiments report performance similar to or stronger than direct fine-tuning at the same forget performance.

  • Problem

    Sensitive information memorized from LLM training corpora raises ethical and legal concerns, and unlearning aims to remove target data without harming out-of-forget-scope performance.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    δ-Unlearning learns a logit offset by contrasting two smaller models’ logits and applies that offset to a black-box LLM without modifying its internal weights.

    Source: §1 · p. 2

  • Results

    At the same forget performance, δ-Unlearning maintains similar or stronger out-of-forget-scope performance than directly fine-tuned larger models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    δ-Unlearning is compatible with different unlearning algorithms, offering a versatile approach for adapting them to black-box LLMs.

    Source: §6 · p. 10

    The formulation treats a retain set as optional, so preserving performance on a designated retain set is not required in every setup.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Despite the strong capabilities of Large Language Models (LLMs) to acquire knowledge from their training corpora, the memorization of sensitive information in the corpora such as copyrighted, biased, and private content has led to ethical and legal concerns. In response to these challenges, unlearning has emerged as a potential remedy for LLMs affected by problematic training data. However, previous unlearning techniques are either not applicable to black-box LLMs due to required access to model internal weights, or violate data protection principles by retaining sensitive data for inference-time correction. We propose {\delta}-Unlearning, an offset unlearning framework for black-box LLMs. Instead of tuning the black-box LLM itself, {\delta}-Unlearning learns the logit offset needed for unlearning by contrasting the logits from a pair of smaller models. Experiments demonstrate that {\delta}- Unlearning can effectively unlearn target data while maintaining similar or even stronger performance on general out-of-forget-scope tasks. {\delta}-Unlearning also effectively incorporates different unlearning algorithms, making our approach a versatile solution to adapting various existing unlearning algorithms to black-box LLMs.

1 Introduction

LLMs can memorize sensitive training content, raising ethical and legal concerns, while existing unlearning approaches face barriers for black-box models. δ-Unlearning learns a logit offset using smaller models; experiments report comparable or stronger out-of-scope performance than direct fine-tuning at the same forget performance.

1 Introduction

  • Memorized copyrighted, biased, and private content can be reproduced or disclosed by LLMs, motivating efforts to erase problematic training-data influence.The passage cites prior studies of copyrighted-material reproduction, harmful and biased content, and private-information disclosure.Source: §1 · pp. 1–2
  • Existing unlearning methods often require access to model weights, while in-context correction retains sensitive examples for inference and can struggle with growing forget sets.Exact unlearning would retrain from scratch, but the paper describes approximate post-hoc methods as more practical given retraining costs.Source: §1 · p. 2
  • δ-Unlearning adapts black-box LLMs by learning a logit offset from two smaller white-box models instead of tuning the black-box model.The framework also avoids retaining sensitive data for inference and supports plug-and-play customization with a pool of smaller models.Source: §1 · p. 2
  • At matched forget performance, δ-Unlearning maintains similar or stronger out-of-forget-scope performance than directly fine-tuned larger models, without updating their parameters.The evaluation uses TOFU, a benchmark about fictitious authors.Source: §1 · p. 2
  • δ-Unlearning integrates with different unlearning algorithms, and the authors report matching or outperforming direct-fine-tuning baselines on general tasks outside the unlearning scope.The paper presents algorithm compatibility as a contribution alongside the framework itself.Source: §1 · p. 3

2 Related Work

Related work covers post-hoc unlearning and logit-based adaptation of language models. The paper distinguishes unlearning from model editing and notes that prior approaches either require weight access or retain sensitive information at inference.

2 Related Work

  • The related-work review organizes its discussion around machine unlearning for LLMs and logit ensemble.The paper presents these as two research lines related to its approach.Source: §2 · p. 3

Machine Unlearning for LLM

  • Most prior unlearning methods use post-hoc adaptation because retraining is costly, but they require internal model weights; in-context unlearning retains sensitive information for inference.Unlearning seeks to remove the effect of specified training data without necessarily creating new answer mappings, unlike model editing as characterized here.Source: §2 · p. 3

Logit Ensemble

  • Prior work combines logits from different models or layers for generation control, domain adaptation, instruction following, and other downstream tasks.The paper positions its use of smaller models' logit offsets for unlearning as another application of logit combination.Source: §2 · p. 3

3 Method

The method formulates unlearning as approximating a model trained without a target forget set while preserving out-of-scope performance. δ-Unlearning adds a learned offset from smaller models to a black-box model's logits, and presents black-box applicability, privacy protection, and customization as merits.

3.1 Problem Definition

  • Unlearning seeks a model resembling one trained without the forget set while preserving performance on out-of-scope data; a retain set may optionally be included.Retraining on the remaining data is the ideal formulation, while its cost motivates approximation.Source: §3.1 · pp. 3–4

3.2 Offset Unlearning

  • δ-Unlearning adds the difference between two smaller models' logits to the black-box model's logits, updating one offset model while keeping the base model and its reference frozen.The two offset models start from the same checkpoint, so their initial logit difference is zero; the learned difference steers predictions away from sensitive information.Source: §3.2 · p. 4
  • The logit-space addition is interpreted as a product-of-experts, with α controlling the strength of the offset applied to the black-box model.For non-sensitive queries, the smaller models' probability ratio should remain close to one; for sensitive queries, it shifts to steer the ensemble away from the base model's distribution.Source: §3.2 · p. 4
  • The framework applies unlearning objectives to the ensemble prediction rather than directly optimizing the original black-box model.The paper gives Gradient Ascent as an example objective for reducing the likelihood of forget-set data.Source: §3.2 · p. 5

3.3 Merits of δ-Unlearning

  • δ-Unlearning applies to black-box LLMs by using their logits and learning the needed correction with smaller white-box models.It does not require access to the larger model's internal weights.Source: §3.3 · p. 5
  • After unlearning, δ-Unlearning does not require storing explicit sensitive information for inference, unlike the in-context approach described here.The cited passage characterizes the in-context approach as relying on an ever-growing list of sensitive information.Source: §3.3 · p. 5
  • A pool of smaller models can be combined with the same base LLM in a plug-and-play manner for version control and user customization.Specialized unlearning modules are presented as a way to curate knowledge available to different applications.Source: §3.3 · p. 5

4 Experiment

The experiment evaluates δ-Unlearning on TOFU and general benchmarks, comparing it with direct fine-tuning across several unlearning algorithms. At matched forget performance, it achieves similar or stronger retain performance and is effective across algorithms.

4.1 Evaluation Setting

  • TOFU evaluates forgetting on a fictitious-author Forget Set and retention on Retain Set, Real Author, and World Fact, with closer-to-forget data posing greater knowledge-entanglement challenges.Forget metrics include ROUGE-L recall, probability, truth ratio, and forget quality; retain utility aggregates ROUGE-L, probability, and truth ratio.Source: §4.1 · pp. 5–6
  • The evaluation matches models to the retraining baseline's Forget Set ROUGE score by adjusting learning rate, and also tests zero-shot accuracy on ARC, HellaSwag, WinoGrande, and OpenBookQA.The retraining baseline's forget quality is 1.0, so it cannot serve as the forget-performance stopping criterion.Source: §4.1 · p. 6, §4.3 · p. 6

4.2 Model Configuration

  • The comparison applies gradient ascent, gradient difference, KL minimization, and data relabeling to δ-Unlearning and corresponding direct-fine-tuning baselines, with retraining as a reference.Experiments use Llama2-13b-chat-hf as the larger model and Llama2-7b-chat-hf as the smaller offset model; both are trained for five epochs.Source: §4.2 · p. 6

4.3 Main Results

  • Both direct fine-tuning and δ-Unlearning reach forget-set ROUGE performance similar to retraining, while δ-Unlearning produces higher truth ratio and forget quality in all three cases where direct fine-tuning lowers the correct-answer probability.A higher truth ratio indicates that other answers are more likely, making the correct answer less distinguishable.Source: §4.3 · pp. 7–8
  • On TOFU retain data, δ-Unlearning preserves more utility than direct fine-tuning in three of four methods, with gains above 15 points for gradient ascent and KL minimization.Direct fine-tuning performs better on the Retain Set, while δ-Unlearning outperforms it on Real Author and World Fact.Source: Table 3 · p. 7, §4.3 · p. 8
  • On general benchmarks, δ-Unlearning is competitive on most metrics, consistently beats direct fine-tuning with KL minimization, and improves three of four tasks with gradient difference and data relabeling.It also closes the performance gap between the model before and after unlearning.Source: Table 2 · p. 7, §4.3 · p. 8
  • Across the evaluated algorithms, δ-Unlearning is a strong alternative to direct fine-tuning, with matching or superior forget and retain performance.The experiments also show effectiveness across different unlearning algorithms.Source: §4.3 · p. 8

5 Analysis

The analysis tracks how unlearning changes forget and retain performance over training and as offset strength varies. It also examines how offset-model size affects unlearning and retain performance.

5.1 Unlearning Trajectory

  • At higher unlearning levels, δ-Unlearning balances forget and non-forget performance better as direct fine-tuning loses more non-forget performance.Retain Set performance declines early, followed by Real Author; World Fact declines only slightly near the end.Source: §5.1 · p. 9

5.2 Effect of Offset Strength

  • Increasing offset strength makes forgetting more prominent, but beyond the training strength performance degrades across all four datasets.At α=5, Forget Set ROUGE falls below 10, lower than the retraining baseline of 38.9, while the model becomes unusable on non-forget sets.Source: §5.2 · p. 9, Figure 3 · p. 9
  • At α=1, the model responds fluently with the sensitive ground-truth information removed, while stronger offsets produce gibberish and eventually unusable outputs.The authors conclude that using the training offset strength gives the best overall results.Source: §5.2 · pp. 9–10, Table 4 · p. 10

5.3 Choice of Offset Model

  • All tested offset models reach retraining-like forget performance by ROUGE, while stronger offset models generally improve retain performance.Truth ratio and forget quality are more sensitive to offset-model choice, especially for weaker models.Source: §5.3 · p. 10, Table 5 · p. 10

6 Conclusion

δ-Unlearning adapts a black-box LLM without access to its internal weights, using a learned logit offset rather than parameter updates. Experiments report performance on par with or sometimes stronger than direct fine-tuning, and compatibility with various unlearning algorithms.

6 Conclusion

  • δ-Unlearning learns a logit offset to steer a black-box LLM without modifying its parameters, and is compatible with various unlearning algorithms.Experiments show it is on par with and sometimes stronger than direct fine-tuning in forget quality and model utility.Source: §6 · p. 10
Loading 2404.11045v2…