PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Prompting for Unlearning in Large Language Models

Karuna Bhaila, Minh-Hao Van, Xintao Wu

arXiv:2406.12038v2cs.CLcs.AIcs.LG

TL;DR

LLM unlearning addresses concerns about unwanted training data, but existing approaches often rely on fine-tuning. This paper uses learned soft prompts with a frozen model, and reports improved forgetting–utility trade-offs across classification and question-answering evaluations.

  • Problem

    Unlearning unwanted training examples in LLMs matters for data protection, while model scale and inaccessible pre-training data make retraining impractical.

    Source: §1 · p. 1

  • Method

    SPUL optimizes soft prompt tokens with a multi-objective loss for forgetting and utility preservation while keeping pre-trained LLM parameters frozen.

    Source: §3.1 · p. 3, §3.3 · p. 4, §5 · p. 9

  • Results

    SPUL induces forgetting while preserving pre-trained utility, with significant improvements over baselines on sentiment classification and multiple-choice question answering.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SPUL offers a less resource-intensive alternative to fine-tuning for unlearning subsets of data in LLMs.

    Source: §1 · p. 1

    Evaluation focuses on open-source models and sentiment classification/QA; prompt learning still requires access to frozen-model gradients.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The widespread popularity of Large Language Models (LLMs), partly due to their unique ability to perform in-context learning, has also brought to light the importance of ethical and safety considerations when deploying these pre-trained models. In this work, we focus on investigating machine unlearning for LLMs motivated by data protection regulations. In contrast to the growing literature on fine-tuning methods to achieve unlearning, we focus on a comparatively lightweight alternative called soft prompting to realize the unlearning of a subset of training data. With losses designed to enforce forgetting as well as utility preservation, our framework \textbf{S}oft \textbf{P}rompting for \textbf{U}n\textbf{l}earning (SPUL) learns prompt tokens that can be appended to an arbitrary query to induce unlearning of specific examples at inference time without updating LLM parameters. We conduct a rigorous evaluation of the proposed method and our results indicate that SPUL can significantly improve the trade-off between utility and forgetting in the context of text classification and question answering with LLMs. We further validate our method using multiple LLMs to highlight the scalability of our framework and provide detailed insights into the choice of hyperparameters and the influence of the size of unlearning data. Our implementation is available at \url{https://github.com/karuna-bhaila/llm_unlearning}.

1 Introduction

The introduction frames LLM unlearning as a response to privacy and other concerns about training data, then proposes soft prompting as a lighter alternative to fine-tuning. It presents SPUL for unlearning subsets of training data and reports improved forgetting–utility trade-offs on classification and question-answering tasks.

1 Introduction

  • Privacy regulations motivate removing unwanted training examples’ influence, while LLM scale and inaccessible training data make retraining impractical.The introduction connects machine unlearning to opt-out and right-to-be-forgotten requests.Source: §1 · p. 1
  • SPUL learns soft prompts to guide a frozen LLM toward unlearning goals without updating its large-scale parameters.The prompts are prepended to input tokens at inference and trained with a multi-objective loss.Source: §1 · pp. 1–2
  • SPUL induces forgetting while preserving pre-trained utility, with significant improvements over baselines on sentiment classification and multiple-choice question answering.The evaluation also examines loss components, prompt size, multiple LLMs, and unlearning-set size.Source: §1 · p. 2

2 Related Work

The related work reviews soft prompting as a lightweight alternative to fine-tuning and surveys LLM unlearning approaches that modify model parameters, units, or inputs.

2.1 Soft prompting

  • Soft prompting uses trainable prompt parameters; prior work reports performance comparable to fine-tuning across classification, generation, and other tasks.Methods include continuous embeddings, prefixes, prompts combined with discrete prompts, and deep prompts inserted across layers.Source: §2.1 · p. 2

2.2 Unlearning in LLMs

  • LLM unlearning work includes gradient-based fine-tuning, relabeling, optimizer-focused updates, and localization objectives that target model units.The surveyed work addresses unwanted responses or information while also seeking to maintain utility.Source: §2.2 · pp. 2–3
  • Input-modification approaches include in-context examples and guardrails; this paper instead automates prompt optimization using unlearning objectives.The passage notes that in-context unlearning is limited in the size of data it can unlearn.Source: §2.2 · p. 3

3 Soft Prompting for Unlearning

SPUL adapts a frozen pre-trained LLM for unlearning by optimizing soft prompt tokens, while the problem formulation seeks to forget selected training outputs and preserve utility on retained samples. Its method combines a forgetting objective with losses that preserve retain-set predictions and keep prompted outputs close to the base model.

3.1 Soft Prompting

  • Soft prompting appends learnable prompt embeddings to input embeddings and updates only the prompts while freezing the pre-trained LLM parameters.The prompts are appended to the input sequence before it is passed to the encoder or decoder.Source: §3.1 · p. 3

3.2 Problem Formulation

  • The problem formulation divides training data into a forget set and a retain set, with the goal of forgetting selected token sequences while maintaining inference utility on retained samples.For text classification and question answering, unlearning is framed as forgetting predictive output sequences for forget samples while preserving their predictive performance on retain samples.Source: §3.2 · p. 3

3.3 Method

  • SPUL trains prompts to associate forget inputs with generic outputs rather than their true labels, aiming to decorrelate the inputs from their predictive labels.The generic label set is disjoint from task labels and includes tokens such as neutral, unknown, or None.Source: §3.3 · pp. 3–4
  • The forget loss uses a uniformly sampled generic label and cross-entropy, with the aim of changing predictions for arbitrary examples containing distinctions learned from forget samples.The method also seeks to preserve predictive performance on samples outside the forget set.Source: §3.3 · p. 4
  • SPUL uses a true-label loss on retain samples to preserve inference utility after adding the prompt tokens.The method also constrains prompted predictions to resemble the base model’s predictions on retain samples.Source: §3.3 · p. 4
  • A KL-divergence loss on retain samples discourages large deviations in the base model’s output caused by the forgetting loss.It compares the prompted model’s predictive distribution with the base model’s distribution on the same input.Source: §3.3 · p. 4
  • SPUL optimizes the prompt tokens using a combined loss of forgetting, retain utility, and KL-divergence components, weighted by hyperparameters α and β.The pre-trained model parameters remain frozen during prompt optimization.Source: §3.3 · p. 4

4 Experiments

The experiments evaluate SPUL on sentiment-classification and multiple-choice question-answering unlearning tasks, using constructed forget and retain sets and comparisons with fine-tuning baselines. Across the reported results, SPUL achieves better forgetting–utility trade-offs while preserving retain-set performance.

4.1 Experimental Setup

  • The evaluation covers SST-2 and Yelp sentiment classification plus WMDP+SciQ multiple-choice question answering, with forget sets designed around related reviews or hazardous knowledge.SST-2 forget examples are selected by named entities, while Yelp examples are grouped by semantic similarity; WMDP forget data covers hazardous knowledge and SciQ supplies general-science retain examples.Source: §4.1 · p. 4

4.2 Experimental Results

  • SPUL reduces forget-set accuracy and F1 while retaining performance close to the base model, including on test examples unseen during prompt training.The reported test-retain metrics reflect the training-retain trends; similar performance trends are reported for Yelp and WMDP+SciQ.Source: §4.2 · p. 5
  • SPUL achieves significantly better overall trade-offs between forgetting and utility than the compared methods across both tasks.On WMDP+SciQ, both GA+KL and GA+GD underperform on forgetting; GA+KL performs poorly across all three datasets, while GA+GD does not improve forgetting quality on Yelp.Source: Table 2 · p. 6, §4.2 · p. 6
  • Embedding visualizations show forget and retain samples separating after SPUL, with positive and negative forget examples mixed while retain examples remain grouped by label.The plots use 500 randomly sampled SST-2 training examples per label; colors distinguish forget from retain examples and shapes distinguish positive from negative examples.Source: Figure 1 · p. 6, §4.2 · p. 7
  • At fixed α, changing β has little effect on forgetting, while β = 0.5 generally gives the best retain performance; higher α favors retention and lower α favors forgetting.The ablation varies α and β on SST-2 with prompt length p fixed at 30.Source: §4.2 · pp. 7–8, Table 3 · p. 7
  • On SST-2, SPUL preserves utility across differing forget-set sizes with minimal loss as more forget samples are added, while its forget-metric performance improves when more samples are present.On Yelp, results highlight SPUL’s robustness against large forget sets when more than 5,000 samples are assigned to Dtr_f.Source: §4.2 · p. 8
  • On SST-2, retain-set utility is largely unaffected by prompt length, while p = 30 gives the most competitive forgetting performance.Larger prompt lengths generally improve forgetting metrics but may require longer training for optimal performance.Source: Figure 2 · p. 7, §4.2 · p. 8
  • With p = 30, SPUL trains 604K, 1.19M, or 1.49M parameters across the evaluated models, versus 1.42B, 6.74B, or 13B updated by fine-tuning.SPUL, GA + KL, and GA+GD each take about 1020 GPU seconds per SST-2 training epoch; GA and RL take about 40 seconds, while SPUL avoids updating LLM parameters.Source: §4.2 · pp. 8–9

5 Conclusion

SPUL unlearns subsets of training data in text classification and question answering while keeping pre-trained LLM parameters frozen. The reported evaluation finds it more efficient than fine-tuning-based baselines and robust to a high number of unlearning samples.

5 Conclusion

  • SPUL optimizes a small number of prompt tokens with a multi-objective loss on forget and retain subsets while keeping pre-trained LLM parameters frozen.The framework targets text classification and multiple-choice question answering.Source: §5 · p. 9
  • SPUL demonstrates superior efficiency over fine-tuning-based unlearning baselines on sentiment classification and question-answering datasets.Source: §5 · p. 9
  • SPUL is reported to adapt to multiple LLMs and remain robust to a high number of unlearning samples.Source: §5 · p. 9

Limitations

The evaluation primarily covers open-source LLMs, sentiment classification, and question answering. It does not comprehensively evaluate the framework, including its robustness to model-stealing attacks, membership-inference attacks, or jailbreaking.

Limitations

  • The framework requires access to frozen pre-trained parameters to compute prompt gradients, so experiments primarily focus on open-source LLMs.Source: Limitations · p. 9
  • The unlearning formulation and evaluation cover sentiment classification and question answering, leaving text generation and summarization for future study.Source: Limitations · p. 9
  • The evaluation emphasizes performance metrics and is not comprehensive, while the paper notes a lack of an extensive LLM-unlearning evaluation pipeline.Source: Limitations · p. 9
  • The framework's robustness to model-stealing attacks, membership-inference attacks, and jailbreaking remains unevaluated.Source: Limitations · p. 9

Broader Impacts

The study aims to support resource-efficient LLM unlearning in response to user requests while maintaining model efficiency. It reports using publicly available datasets and pre-trained LLMs in accordance with their licenses.

Broader Impacts

  • The study aims to enable forgetting of unwanted knowledge in response to user requests while maintaining model efficiency to avoid exploitation of sensitive information.Source: Broader Impacts · p. 9
  • The evaluation datasets are publicly available and used within their intended use.Source: Broader Impacts · pp. 9–10
  • The study uses publicly available pre-trained LLMs in accordance with their associated licenses.Source: Broader Impacts · p. 10

Related papers

Loading 2406.12038v2…