PaperlayerView exact revision on arXiv

Source-linked AI summary

Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, Sijia Liu

arXiv:2410.07163v4cs.CLcs.AIcs.LG

TL;DR

LLM unlearning seeks to remove unwanted data influence while preserving model utility, but NPO can suffer from reference model bias. The paper proposes reference-free SimNPO and reports consistent improvements over NPO across standard benchmarks, with additional robustness benefits. The authors note that balancing unlearning effectiveness and utility remains unresolved.

  • Problem

    NPO’s reliance on the pre-unlearning reference model can unevenly allocate optimization power and weaken early gradient smoothing, motivating improved LLM unlearning.

  • Method

    SimNPO removes the reference model from NPO through a reference-free simple preference optimization framework.

  • Results

    SimNPO consistently outperforms NPO across TOFU, MUSE, and WMDP, with additional advantages in unlearning robustness and reasoning-model unlearning.

  • Takeaways & Limitations

    Removing reference-model reliance can improve unlearning effectiveness across benchmarks while supporting robustness against relearning-based attacks.

  • Takeaways & Limitations

    Both SimNPO and NPO still promote divergence, causing some utility loss; SimNPO does not fully resolve the effectiveness–utility trade-off, and theoretical guarantees remain open.

Abstract

from arXiv · show

This work studies the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences (e.g., copyrighted or harmful content) while preserving model utility. Despite the increasing demand for unlearning, a technically-grounded optimization framework is lacking. Gradient ascent (GA)-type methods, though widely used, are suboptimal as they reverse the learning process without controlling optimization divergence (i.e., deviation from the pre-trained state), leading to risks of over-forgetting and potential model collapse. Negative preference optimization (NPO) has been proposed to address this issue and is considered one of the state-of-the-art LLM unlearning approaches. In this work, we revisit NPO and identify another critical issue: reference model bias. This bias arises from using the reference model (i.e., the model prior to unlearning) to evaluate the unlearning success, which can compromise NPO's effectiveness. Specifically, it leads to (a) uneven allocation of optimization power across forget data with varying difficulty levels and (b) ineffective gradient weight smoothing during the early stages of unlearning optimization. To overcome these challenges, we propose a simple yet effective unlearning optimization framework, called SimNPO, showing that `simplicity' in removing the reliance on a reference model (through the lens of simple preference optimization) benefits unlearning. We provide deeper insights into SimNPO's advantages through an analysis based on mixtures of Markov chains. Extensive experiments further validate SimNPO's efficacy on benchmarks like TOFU and MUSE, as well as its robustness against relearning attacks. Codes are available at https://github.com/OPTML-Group/Unlearn-Simple.

1 Introduction

LLM unlearning must remove unwanted data influence while preserving utility, but uncontrolled divergence can cause under- or over-forgetting. The paper identifies reference model bias in NPO and proposes SimNPO to allocate unlearning power more effectively.

  • Motivation: Unlearning reverses learning effects, but uncontrolled divergence from the pre-trained state can cause under-forgetting, over-forgetting, or utility loss.NPO is presented as an approach with better divergence control than gradient ascent.
  • Reference model bias: Reference model bias arises because NPO regulates unlearning by enlarging the distance from the pre-unlearning reference model.Reference-model confidence varies across samples, so the same objective can treat easy and hard examples inequitably.
  • Research question: The paper asks how to identify and address NPO’s limitations in order to improve LLM unlearning effectiveness.
  • Contribution: SimNPO removes reliance on a reference model through simple preference optimization and aims to allocate optimization power according to forget-data difficulty.The authors report experiments across TOFU, MUSE, WMDP, and relearning-based attacks.
  • Reference model bias: NPO can over-allocate optimization power to weakly memorized, easier-to-unlearn samples, hindering strongly memorized samples.The paper links this imbalance to NPO’s worse performance on strongly memorized forget data relative to SimNPO.

2 Related work

Related work frames machine unlearning as removing data influence while preserving unrelated utility without costly retraining. The paper connects this problem to preference optimization as a route toward efficient reference-free unlearning.

  • Machine unlearning: Retrain is the gold standard for machine unlearning but requires access to the full training set and incurs high computational cost.Retraining excludes forget data from the original training set and starts training again.
  • LLM unlearning: LLM unlearning targets undesired data influences or behaviors while preserving unrelated knowledge generation and avoiding retraining.Applications include protection of copyrighted and personally identifiable information and prevention of harmful content generation.
  • Preference optimization: The paper advances LLM unlearning through preference optimization, motivated by more efficient offline alternatives to complex online optimization.
  • Preference optimization: SimPO is a reference-free, length-normalized DPO variant that the paper integrates into LLM unlearning to improve NPO.

3 A Primer on LLM Unlearning

LLM unlearning balances a forget objective against a retain objective that preserves utility. The primer contrasts unbounded gradient ascent with NPO’s bounded loss and reference-based adaptive gradient weighting.

  • Problem formulation: LLM unlearning is formulated as regularized optimization over a forget set and a retain set, balancing removal of unwanted influence against utility preservation.The retain term penalizes divergence caused by unlearning.
  • Gradient ascent: Gradient ascent uses the negative token prediction loss on forget data, discouraging the model from learning those examples but leaving the forget loss unbounded below.
  • Negative preference optimization: NPO treats forget data as negative preferences, replacing the unbounded gradient-ascent loss with a lower-bounded loss and adaptive gradient weighting.These design choices are intended to prevent catastrophic collapse and control divergence speed.
  • Negative preference optimization: NPO’s adaptive weight is typically below 1 during forgetting, producing more gradual divergence from the reference model than gradient ascent.Gradient ascent corresponds to an adaptive weight of 1.
  • Tasks and evaluation: The evaluation tasks cover synthetic fictitious unlearning, memorization removal, and hazardous-content removal, with effectiveness and utility assessed using task-specific metrics.

4 Uncovering Reference Model Bias in NPO

The paper attributes NPO’s weaknesses to reference model bias: reference-dependent weighting can misallocate optimization power and provide ineffective early gradient smoothing. Experiments show sensitivity to reference choice and response length.

  • Reference model bias: Reference model bias makes NPO’s objective favor enlarging distance from the reference model rather than directly reflecting sample-specific unlearning difficulty.
  • Analysis design: The paper analyzes these limitations using experiments on reference perturbation, memorization, response length, and optimization trajectories.
  • Reference sensitivity: Perturbing NPO’s reference model reduced forget quality from 0.79 to 0.27 while model utility remained nearly unchanged at 0.52 versus 0.57.The study uses TOFU Forget05 and compares the original reference with a perturbed reference.
  • Response length: For short-response forget data, NPO achieved FQ 0.58, compared with FQ 0.81 for the longer-response half of the forget set.The short-response result was farther from Retrain, while the longer-response result was closer.
  • Gradient weight smoothing: NPO’s early gradient weights remain approximately 1 across forget samples despite differing unlearning difficulties, indicating ineffective early weight smoothing.Figure 3 tracks weights against response length and across unlearning epochs, alongside forget quality and utility.

5 SimNPO: Method and Rationale

SimNPO removes NPO’s reference-model dependence while retaining bounded optimization and length-normalized gradient smoothing. Synthetic analyses show more balanced allocation of unlearning power across response lengths and memorization difficulty.

  • Method: SimNPO replaces NPO’s reference-model reward with the reference-free, length-normalized reward (β/|y|) log πθ(y|x).The resulting loss uses the SimPO formulation and includes a reward-margin parameter γ, set to 0 unless otherwise specified.
  • Method: SimNPO’s bounded-below loss preserves controlled optimization while its gradient smoothing depends on response length rather than the reference model.The loss has minimum value 0, and its gradient separates into weight smoothing and a gradient-ascent component.
  • Rationale: Length normalization allocates less optimization power to longer responses and reduces uneven forgetting across data with different response lengths.As β approaches 0, SimNPO becomes a length-weighted gradient-ascent objective rather than ordinary gradient ascent.
  • Gradient smoothing: During the first two unlearning epochs, SimNPO prioritizes shorter, initially harder-to-forget samples; later, its gradient weights become more uniform.NPO’s weights instead concentrate near 1 during the early unlearning stage, making early smoothing ineffective.
  • Synthetic analysis: In synthetic mixture-of-Markov-chain experiments, SimNPO achieves more balanced tradeoffs across forget datasets with different lengths and memorization levels.Compared with NPO, it incurs less retain distance for the same Forget2 quality and avoids the larger Forget1–Forget2 gap associated with uneven unlearning.

6 Experiments

Experiments on TOFU, MUSE, and WMDP compare SimNPO with retraining, NPO, and other unlearning methods using effectiveness, utility, privacy, and robustness measures. SimNPO is reported to improve the utility–unlearning balance and resist relearning attacks, especially for short responses.

  • Experiment setup: Experiments cover TOFU, MUSE, and WMDP, including TOFU Forget05/Forget10, MUSE News/Books, and hazardous-biosecurity knowledge unlearning.The evaluation compares Retrain, SimNPO, NPO, GA, GradDiff, and task-specific baselines.
  • TOFU: On TOFU Forget05, SimNPO outperforms NPO in both forget quality and model utility and is the closest approximate unlearning method to Retrain.FQ and MU are the primary TOFU indicators, with results averaged over five random trials.
  • TOFU: NPO-unlearned models sometimes generate repeated text unlike Retrain, although the repetition does not reveal the information targeted for unlearning.This difference is reported in qualitative response examples accompanying the TOFU evaluation.
  • MUSE and WMDP: On MUSE, SimNPO preserves higher utility while achieving stronger unlearning than NPO and attains PrivLeak values closer to 0.The reported Dr, Df, and PrivLeak values differ across News and Books, while sequential News unlearning also favors SimNPO as requests increase.
  • Relearning robustness: Under shortest-relearn and random-relearn attacks, SimNPO maintains higher forget quality and a slower decline than NPO, remaining resilient to both attack types.NPO is reported as less robust against shortest-relearn than random-relearn, whereas SimNPO is resilient to both.

7 Conclusion

The paper identifies reference model bias in NPO and proposes SimNPO, which consistently outperforms NPO across standard benchmarks while improving robustness and reasoning-model unlearning.

  • SimNPO addresses NPO’s reference model bias with a simpler reference-free preference-optimization framework.
  • SimNPO consistently outperforms NPO across TOFU, MUSE, and WMDP, with additional advantages in relearning robustness and reasoning model unlearning.

Broader Impact

Preference optimization supports more efficient machine unlearning, but selective erasure of essential knowledge creates ethical and legal risks requiring strict safeguards.

  • Preference optimization can improve unlearning efficiency, data privacy protection, and compliance with regulatory requirements.
  • Selective erasure of essential concepts or knowledge could be misused, raising ethical and legal concerns.

Limitations

SimNPO reduces NPO’s reference model bias but still relies on divergence promotion, leaving utility loss and the unlearning–utility trade-off unresolved, especially for knowledge removal.

  • Both NPO and SimNPO rely on promoting divergence, which inevitably causes some degree of utility loss.
  • The limitation is especially evident in knowledge unlearning or capability removal, such as on the WMDP benchmark.
  • SimNPO has not fully resolved the balance between unlearning effectiveness and model utility, and theoretical guarantees remain future work.

NeurIPS Paper Checklist

The checklist states that the abstract and introduction accurately reflect the paper’s contributions and scope, with details provided in later sections.

  • The checklist marks the alignment of the abstract and introduction with the paper’s main claims as Yes.
  • The justification points to Sections 3, 4, 5, and 6 for comprehensive details supporting those claims.

2. Limitations

The paper reports that its limitations are discussed in Appendix 7. Its supporting statements also clarify that the work uses no human subjects and uses LLMs only for writing refinement.

  • The paper discusses limitations of the authors’ work in Appendix 7.
  • LLMs were used solely for writing refinement rather than as an important component of the core methodology.

F Further Results on Response Length Normalization in SimNPO

SimNPO’s length normalization removes the response-length bias in NPO’s unlearning dynamics. Analyses and experiments show length-agnostic unlearning steps while preserving a more controllable divergence rate than gradient ascent.

  • Length normalization: When β → 0, SimNPO becomes length-normalized weighted-GradDiff, whereas NPO becomes GradDiff.
  • Length normalization: Length normalization gives weighted-GradDiff substantially better forget quality than GradDiff, explaining SimNPO’s advantage over NPO.
  • Unlearning speed: NPO unlearns longer responses faster than shorter responses, preventing equal unlearning of short and long responses simultaneously.
  • Unlearning speed: SimNPO makes the number of steps needed to unlearn a sample agnostic to response length.
  • Divergence control: SimNPO and NPO have asymptotically identical unlearning speed, with similar logarithmic divergence rates that are slower and more controllable than GA.

I.2 Experiment Setups

Experiments evaluate SimNPO and baseline unlearning methods across TOFU, MUSE, and WMDP, including utility, unlearning effectiveness, and relearning robustness. Results report advantages for SimNPO on TOFU Forget10, sequential MUSE requests, WMDP utility preservation, and generated-response quality, while retaining some unlearning failures.

  • Methods and evaluation: Evaluation compares Retrain, GA, GradDiff, NPO, and task-specific baselines, with NPO using retain regularization as the primary baseline.Retrain is treated as the gold standard where available, while relearning attacks test whether forgotten information can be recovered through fine-tuning.
  • TOFU Forget10: SimNPO achieves the highest Forget Quality and Model Utility among methods on TOFU Forget10.The comparison is reported in Table A3 for the LLaMA2-7B-chat model.
  • MUSE: On sequential MUSE News requests, SimNPO has smaller KnowMem on Df and larger KnowMem on Dr than NPO, indicating stronger unlearning and utility preservation.The comparison is shown in Figure A5 for the same unlearning requests.
  • WMDP: On WMDP, SimNPO preserves utility better than NPO, while SimNPO and NPO outperform RMU in unlearning efficacy but preserve less utility than RMU.Retrain is unavailable for WMDP, whose forget set represents hazardous biosecurity knowledge.
  • Generated responses: Generated answers from SimNPO are closer to Retrain than NPO outputs, although both methods sometimes fail to unlearn targeted information or produce excessive repetition.NPO additionally produces repetitive or erroneous words in several examples, whereas SimNPO avoids those listed behaviors.
Loading 2410.07163v4…