PaperlayerView exact revision on arXiv

Source-linked AI summary

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

Ruiqi Zhang, Licong Lin, Yu Bai, Song Mei

arXiv:2404.05868v2cs.LGcs.AIcs.CLstat.ML

TL;DR

LLM unlearning seeks to remove undesirable training-data influence while preserving other capabilities, but practical methods can fail to unlearn effectively or cause catastrophic collapse. This paper proposes Negative Preference Optimization and finds that it provides effective unlearning with improved stability, including for high percentages of training data.

  • Problem

    LLM unlearning must remove specific training-data influence while preserving other knowledge and capabilities, but existing gradient-ascent methods can fail or cause catastrophic collapse.

  • Method

    The paper proposes Negative Preference Optimization, an alignment-inspired negative-sample-only objective, and theoretically analyzes its divergence speed relative to gradient ascent.

  • Results

    NPO-based methods achieve state-of-the-art unlearning performance and are the only effective methods reported for forgetting 50%-90% of the data, while existing methods struggle at 10%.

  • Takeaways & Limitations

    NPO-based methods offer a better trade-off between forget quality and model utility and avoid the rapid utility deterioration associated with gradient ascent.

  • Takeaways & Limitations

    Retraining from scratch on the retain data is the principled ideal but is often intractable in practice.

Abstract

from arXiv · show

Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail to effectively unlearn the target data or suffer from catastrophic collapse -- a drastic degradation of the model's utilities. In this paper, we propose Negative Preference Optimization (NPO), a simple alignment-inspired method that could efficiently and effectively unlearn a target dataset. We theoretically show that the progression toward catastrophic collapse by minimizing the NPO loss is exponentially slower than GA. Through experiments on synthetic data and the benchmark TOFU dataset, we demonstrate that NPO-based methods achieve a better balance between unlearning the undesirable data and maintaining the model's utilities. We also observe that NPO-based methods generate more sensible outputs than GA-based methods, whose outputs are often gibberish. Remarkably, on TOFU, NPO-based methods are the first to achieve reasonable unlearning results in forgetting 50% (or more) of the training data, whereas existing methods already struggle with forgetting 10% of training data.

1 Introduction

LLM unlearning seeks to remove specific training data while preserving other capabilities, but gradient-ascent methods can fail or trigger catastrophic collapse. The paper introduces NPO, showing theoretically and empirically that it improves the trade-off between forgetting and utility.

  • Machine unlearning removes specific training samples’ influence while preserving other model knowledge and capabilities.
  • Gradient-ascent methods optimize forget-set prediction loss but often provide unsatisfactory unlearning performance on TOFU.TOFU evaluates forgetting 1%, 5%, and 10% of biographies from a model trained on all 200 authors.
  • NPO is an alignment-inspired loss using only negative samples that addresses catastrophic collapse associated with gradient ascent.The paper also describes NPO as a preference-optimization variant without positive samples.
  • NPO’s progression toward catastrophic collapse is exponentially slower than gradient ascent, while synthetic experiments show a superior Forget Distance–Retain Distance Pareto frontier.NPO-based methods also exhibit greater learning stability than GA-based methods.
  • NPO-based methods consistently improve the balance between Forget Quality and Model Utility across TOFU tasks and produce more readable outputs than GA-based methods.The paper reports superior balance, improved stability, and improved output readability relative to baselines.
  • 50%-90%: NPO-based methods are reported as the only effective methods for forgetting this data range, while existing methods struggle with 10%.

2 Preliminaries on Machine Unlearning

Machine unlearning aims to make a model behave like one retrained without a designated forget set, while retaining performance on the remaining data. Gradient ascent can briefly improve forgetting but may rapidly destroy utility, motivating slower-diverging alternatives such as NPO.

  • Machine unlearning fine-tunes an initially trained model to behave like a model retrained only on the retain set.The forget set is removed from the original training data, and the retain set contains the remaining samples.
  • Retraining from scratch on the retain set is theoretically best but often computationally intractable in practice.
  • Gradient ascent: Gradient ascent maximizes forget-set prediction loss as an approximation to reversing the original training updates.
  • Forget quality and model utility are competing evaluation goals: a model can forget effectively yet lose performance on retained knowledge.Simply answering “I don’t know” can yield good forget quality but poor model utility.
  • Catastrophic collapse: Gradient ascent can cause catastrophic collapse: utility quickly reaches zero, forget quality briefly improves, and later outputs become gibberish as KL divergence rises.
  • Catastrophic collapse: GA’s gradient can retain constant scale on already-unlearned samples, producing divergence that grows linearly with the number of steps.
  • Motivation for NPO: NPO is motivated by GA’s undesirably fast divergence and instead diverges at a slower speed.

3 Negative Preference Optimization

Negative Preference Optimization (NPO) adapts preference optimization to unlearning using only negative examples. Unlike gradient ascent (GA), NPO is lower-bounded at finite temperature, converges to GA as β →0, and diverges exponentially more slowly.

  • NPO objective: NPO is a drop-in replacement for the GA loss that remains lower-bounded and stable at any finite temperature.Its loss reduces to GA in the high-temperature limit.
  • NPO objective: NPO treats each forget-set example as a negative response and omits the positive-response term from DPO.Minimizing the resulting loss makes forget-set prediction probabilities as small as possible.
  • Connection with GA: β →0 recovers the GA loss and its gradient from NPO, making NPO a strict generalization of GA.Figure 3 illustrates the convergence of the NPO trajectory to the GA trajectory as β approaches zero.
  • Stability: NPO’s adaptive smoothing weight becomes small after an example is unlearned, reducing its gradient relative to GA.When πθ(y|x) ≪ πref(y|x), the weight satisfies Wθ(x,y) ≪1 and ∥∇θLNPO,β∥2 ≪∥∇θLGA∥2.
  • Theory: NPO diverges exponentially slower than GA in the analyzed logistic-regression setting.The theorem characterizes GA as diverging linearly and NPO as diverging logarithmically under stated assumptions.

4 Synthetic Experiments

Synthetic experiments compare unlearning methods using forget and retain distances in random-feature binary classification. NPO-based methods provide more stable optimization and a better trade-off between forgetting and retention than GA-based baselines.

  • 4.1 Setup: The experiment uses a random-feature model with fixed Gaussian weights and trains initial and retrained models on the full and retain datasets, respectively.The trainable parameter has dimension 128.
  • 4.1 Setup: The evaluated methods include NPO, NPO+RT, GA, GA+RT, IDK+RT, and DPO+RT.Retain-loss variants add a retain loss term during unlearning.
  • 4.1 Setup: Forget distance and retain distance are KL divergences from the retrained model on the forget and retain sets, and lower values are better.A perfectly unlearned model would have both distances equal to zero.
  • 4.2 Results: All methods except IDK+RT reach forget distance below 0.005 within 1200 steps, while GA and GA+RT retain distances diverge.NPO+RT and DPO+RT retain distances increase slowly and stabilize.
  • 4.2 Results: NPO+RT achieves a better Pareto frontier than the baselines, and NPO outperforms GA among methods that do not use the retain set.When α = 0, NPO+RT attains the smallest forget and retain distances.

5 Experiments on the TOFU Data

Experiments on TOFU show that NPO-based methods provide a stronger balance between forgetting target data and preserving model utility than baseline methods. They remain more stable, produce more readable outputs, and extend meaningful unlearning to larger forget sets.

  • Experimental setup: TOFU contains 200 fictitious author profiles and evaluates forgetting 1%, 5%, and 10% of the data using Forget Quality and Model Utility.Forget Quality above 0.05 is treated as evidence of meaningful forgetting.
  • Trade-off results: NPO-based methods consistently outperform GA-based methods across Forget01, Forget05, and Forget10 in the trade-off between Forget Quality and Model Utility.In Forget05, NPO methods are the only ones above 0.05 forget quality; in Forget10, NPO+RT uniquely preserves meaningful forgetting while greatly preserving utility.
  • Unlearning stability: GA and GA+RT show temporary forget-quality gains followed by drastic decline, whereas NPO methods reach and maintain a stable plateau after several epochs.GA-based methods often require early stopping at instance-dependent times to avoid catastrophic collapse.
  • Output quality: NPO+RT generates fluent incorrect answers on forget questions while preserving fluent and correct answers elsewhere, unlike GA gibberish and IDK or DPO excessive ignorance.The sampled responses illustrate improved fluency and diversity under NPO+RT.
  • Retain-loss analysis: Increasing the retain-loss weight in NPO+RT monotonically improves Model Utility, while Forget Quality first improves and then deteriorates.Forget Quality increases as the retain-loss weight rises from 0 to 2, but drops when the weight increases further, such as from 2 to 5.
  • Forgetting beyond 10%: NPO+RT remains the sole method with meaningful Forget Quality on Forget20 and Forget30, and tuned NPO+RT achieves Forget Quality above 0.05 with Model Utility above 0.55 on Forget50 and Forget90.The extended tasks target substantially larger fractions of TOFU than the initial 1%, 5%, and 10% settings.

6 Conclusion

The paper proposes Negative Preference Optimization (NPO) for LLM unlearning and reports state-of-the-art performance, including effective unlearning at high forget rates.

  • NPO is introduced as a simple objective for LLM unlearning.
  • NPO-based methods address catastrophic collapse in gradient-ascent unlearning.
  • NPO-based methods achieve state-of-the-art performance on LLM unlearning.
  • NPO-based methods achieve the first effective unlearning result when forgetting a high percentage of training data.

A.2 Proof of Theorem 2

The proof analyzes gradient-ascent and NPO dynamics under bounded feature correlations, establishing linear divergence for GA and exponentially slow divergence for NPO.

  • Both gradient updates remain in the span of the forget-set feature vectors.
  • GA iterations converge to infinity linearly under the theorem’s assumptions.The result uses constants depending on model and feature bounds.
  • NPO iterations converge to infinity exponentially slowly under the theorem’s assumptions.The constants may additionally depend on the temperature parameter β.
  • The analysis assumes pairwise feature correlations satisfy max_i≠j |γ_i,j| ≤ C0/nf.

B.1 Synthetic Experiment

The synthetic experiments examine forget KL across optimization steps for two α settings and show that unlearning performance is best at a moderate forget KL.

  • Moderate forget KL is associated with better unlearning performance in the synthetic experiments.Excessively large or small forget KL can deteriorate performance.
  • Excessively large forget KL, as in GA and GA+RT after 1200 steps, may deteriorate unlearning performance.
  • The figure reports forget KL versus optimization steps for all methods with α = 1 and α = 0.Error bars denote ±1 standard deviation over 5 runs.

C Experimental details of the synthetic experiments

The synthetic experiments construct initial and retrained models by gradient descent, then unlearn from the initial model for 2000 steps using grid-searched learning rates selected for stability.

  • The initial model is trained on the full dataset, while the retrained model is trained on the retain dataset.Both are obtained with gradient descent using cross-entropy loss.
  • Unlearning starts from the initial model and runs each loss function for 2000 steps.
  • Learning rates are selected by grid search to keep training stable within 2000 steps.Different learning rates may change the steps needed to reach minimal forget or retain distance.

D Experiments on the TOFU dataset

This section describes the TOFU experiments and presents the associated experimental details and full results across task levels. It also includes a table specifying synthetic-experiment learning rates and β values.

  • The TOFU experiments cover metric definitions, baseline methods, hyperparameters, and full results across different task levels.
  • The section organizes the TOFU evaluation around methodological details followed by results for different levels of the tasks.
  • Table 1 reports learning-rate and β values for synthetic experiments under α = 1 and α = 0.

D.1.1 Dataset

The TOFU evaluation uses fictitious author profiles and complementary evaluation sets to assess forgetting, retained knowledge, and general capabilities. Forget Quality compares the unlearned model with a retain-only retrained model using Truth Ratio distributions.

  • TOFU contains 200 fictitious author profiles, each with 20 GPT-4-generated question-answer pairs, enabling controlled LLM-unlearning evaluation.
  • The retain set complements the forget set, while an exclusive 400-question retain subset is never included in any forget task.
  • Real Authors and Real World sets measure general capacities involving neighboring author concepts and commonsense knowledge, respectively.
  • For Forget20, Forget30, Forget50, and Forget90, the original TOFU forget and retain sets are reorganized for larger targeted forget sets.
  • Model Utility aggregates nine metrics across the retain, Real Authors, and Real World sets using their harmonic average.
  • Forget Quality uses a KS-test p-value to assess whether the unlearned model's Truth Ratio distribution is indistinguishable from the retrained model's.

D.1.3 Baseline Methods

The baseline suite includes gradient-ascent, IDK, DPO, and KTO-based objectives, alongside task-specific statistics for Forget01 through Forget50. These methods differ in how they encourage forgetting, retention, or divergence control.

  • GA-based Methods: GA-based methods optimize a linear combination of gradient-ascent loss and additional forgetting, retention, and KL-divergence terms.
  • GA-based Methods: The experiments include GA, GA+RT, and GA+KL as representative gradient-ascent baselines.
  • IDK-based Methods: IDK+RT replaces forgotten answers with “I don't know” responses while combining the resulting loss with the retain loss.
  • DPO-based Methods: DPO-based methods treat “I don't know” responses as positive and forget-set answers as negative, with retain-loss or retain-KL variants.
  • KTO-based Methods: KTO and KTO+RT use non-paired preference data, estimating the KL term through averaged log-probability ratios.
  • Figures 12–17 report statistics for NPO-based methods and baselines on Forget01, Forget05, Forget10, Forget20, Forget30, and Forget50.
Loading 2404.05868v2…