PaperlayerView exact revision on arXiv

Source-linked AI summary

RADNPO: Reference-free Adaptive Negative Preference Optimization for LLM Unlearning

Shenghan Tan, Ziyi Zhou, Wenpeng Hu, Mengyuan Zhang

arXiv:2609.34251v1cs.CR

TL;DR

LLM unlearning must address memorized sensitive, private, and copyrighted information, but suppressing a target alone leaves the resulting next-token distribution underconstrained. RADNPO contrasts forget targets with likely alternatives and adapts token-level forgetting strength; experiments report a better forgetting–utility trade-off than baselines.

  • Problem

    Preference-based unlearning can suppress forget targets without controlling how their probability mass is redistributed, leaving the post-unlearning distribution potentially concentrated.

    Source: §1 · p. 2

  • Method

    RADNPO contrasts each forget target with alternatives favored by the current next-token distribution and adapts token-level forgetting using target confidence and next-token concentration.

    Source: §1 · p. 2

  • Results

    RADNPO improves the trade-off between forgetting quality and model utility over baselines while preserving retained knowledge.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    On TOFU Forget10, RADNPO produces less repetitive and more lexically diverse responses than NPO.

    Source: §6 · p. 11

    RADNPO is approximate and recoverable unlearning: fine-tuning on the original forget set can substantially restore forgotten behavior.

    Source: §5.4 · p. 11, Ethics Statement · p. 12

Abstract

from arXiv · show

Large language models (LLMs) can memorize sensitive, private, or copyrighted content during pre-training, making machine unlearning necessary for removing targeted knowledge. Recent preference optimization (PO)-based unlearning methods improve stability over gradient ascent (GA)-based methods by introducing alignment-style objectives, which effectively suppress the probability of forget targets. However, target suppression alone does not sufficiently constrain the next-token distribution after unlearning. Existing methods provide limited control over how suppressed probability mass is redistributed and insufficiently adapt forgetting strength to target confidence and distributional concentration. Even after target suppression, probability mass may remain concentrated on a few non-target tokens, potentially producing repetitive or uninformative outputs. To address these limitations, we propose Reference-free ADaptive Negative Preference Optimization (RADNPO), which explicitly guides next-token probability redistribution. Specifically, RADNPO contrasts each forget target with alternative tokens favored by the current next-token distribution and adaptively modulates token-level forgetting strength using target confidence and next-token concentration. Experiments on TOFU and MUSE demonstrate that RADNPO achieves a better trade-off between forgetting quality and model utility than current baselines.

1 Introduction

Pre-training can leave LLMs with sensitive, private, or copyrighted information, motivating machine unlearning. RADNPO addresses limits of target suppression by guiding probability redistribution and adapting forgetting strength, and experiments on TOFU and MUSE report a better forgetting–utility trade-off than baselines.

1 Introduction

  • Pre-training can encode sensitive, private, or copyrighted content, while retraining from scratch is prohibitively expensive, making machine unlearning a practical removal approach.The paper links memorized content to privacy leakage, toxic generation, and unauthorized reproduction.Source: §1 · p. 1
  • Target suppression alone leaves probability redistribution uncontrolled, potentially concentrating mass on implausible, repetitive, or uninformative alternatives.When target likelihood falls but the local distribution remains concentrated, the paper calls this low-entropy forgetting.Source: §1 · p. 2
  • RADNPO contrasts each forget target with high-probability alternatives from the current next-token distribution and scales token-level forgetting by target confidence and distributional concentration.The adaptive signal uses next-token partial entropy to characterize concentration.Source: §1 · p. 2
  • RADNPO improves the trade-off between forgetting quality and model utility over baselines while preserving retained knowledge.Experiments on TOFU and MUSE demonstrate the result.Source: §1 · pp. 2–3

2 Related Work

Machine unlearning spans exact retraining and more efficient approximate approaches. In LLM unlearning, preference optimization offers greater stability than gradient ascent, but target-focused methods leave probability redistribution underconstrained.

Machine unlearning

  • Exact unlearning retrains from scratch on the retain set, whereas approximate unlearning uses efficient fine-tuning or localized editing to avoid full-retraining overhead.The paper identifies exact retraining as the gold standard.Source: §2 · p. 3

LLM unlearning

  • LLM unlearning includes fine-tuning, in-context learning, model editing, and preference optimization, but benchmarks show difficulty balancing forgetting quality and model utility.Recent preference-optimization methods improve stability but largely focus on suppressing target responses.Source: §2 · p. 3

Preference optimization

  • DPO directly optimizes preference data but requires a frozen reference model, while KTO, SimPO, and ORPO offer alternative approaches to streamline preference optimization.SimPO discards the reference model; ORPO combines supervised fine-tuning and preference alignment in a single stage.Source: §2 · p. 3

3 Preliminary

The preliminary section frames unlearning as removing forget-set influence while preserving retain-set capabilities, then describes NPO as a more stable alternative to gradient-ascent forgetting.

3.1 Problem formulation of LLM unlearning

  • LLM unlearning aims to remove the influence of forget-set targets while preserving general capabilities on the retain set.The formulation distinguishes forget and retain losses; gradient ascent reduces forget-target likelihood, while retain loss preserves utility.Source: §3.1 · pp. 3–4

3.2 LLM unlearning with preference optimization

  • NPO repurposes DPO using only negative forget-set samples, with a reference-model-based adaptive weight that modulates forgetting according to relative target likelihood.This weighting is presented as stabilizing gradient-ascent forgetting and preventing catastrophic utility degradation.Source: §3.2 · p. 4

4 RADNPO: Method and Analysis

RADNPO combines a reference-free contrastive objective that guides local probability redistribution with token-level adaptive scaling based on target confidence and next-token partial entropy. Its regime analysis explains how the scaling responds to target probability and local distribution concentration.

4 RADNPO: Method and Analysis

  • RADNPO contrasts forget targets with high-probability non-target candidates and adaptively scales forgetting using target confidence and next-token distribution concentration.Soft clamping limits extreme contrastive margins to support stable optimization.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Reference-Free Contrastive Log-Odds Objective

  • The contrastive log-odds objective compares each forget target with selected alternatives as a group, establishing local competition that shifts preference toward plausible alternatives.Soft clamping smoothly bounds extreme margins so they do not dominate the forgetting objective.Source: §4.1 · pp. 5–6

4.2 Adaptive Scaling based on Target Confidence and Partial Entropy

  • The adaptive coefficient combines target confidence and partial entropy, strengthening forgetting for higher-confidence targets and, at fixed confidence, lower-entropy distributions.Partial entropy uses selected-token probabilities without renormalizing and captures both their probability mass and allocation.Source: §4.2 · pp. 6–7

4.3 Regime Analysis of RADNPO

  • At the same target confidence, lower partial entropy yields a larger adaptive coefficient, assigning greater relative scaling to collapsed than dispersed forgetting states.The coefficient is an adaptive scale for the local contrastive objective, not the effective gradient magnitude itself.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

5 Experiment

RADNPO is evaluated on TOFU and MUSE, with ablations and additional analyses examining its components, robustness, generation quality, and recoverability.

5.2 Experiment results

  • RADNPO achieves the best Forget Quality among compared methods on TOFU while reducing residual memorization and preserving competitive retain performance.Its Forget Quality is close to retraining, and it substantially reduces Exact Memorization and Extraction Strength relative to existing PO-based baselines.Source: Table 1 · p. 9, §5.2 · p. 10

5.1 Experiment setups

  • Evaluations use TOFU and MUSE with the Open-Unlearning framework, primarily on LLaMA-2 7B, with additional Qwen3 experiments.Gradient ascent and gradient difference are included for completeness but excluded from the main comparison because they cause catastrophic forgetting and severe utility degradation.Source: §5.1 · p. 9, §5.2 · p. 9

5.2 Experiment results

  • On MUSE News, RADNPO is among the strongest utility-preserving methods for forgetting; on Books, it has the lowest ES and lower VerbMem than utility-preserving baselines.It also preserves retained knowledge better than UNDIAL on News and forgets more effectively than NPO and SimNPO while retaining competitive knowledge on Books.Source: Table 2 · p. 10, §5.2 · p. 10

5.3 Ablation study

  • Removing the contrastive objective causes the largest decline in forgetting quality and utility, while removing soft clamping causes a smaller, consistent decline.The authors associate soft clamping with improved optimization stability by bounding extreme log-ratios.Source: §5.3 · p. 10, Figure 4 · p. 11
  • Without confidence scaling, forgetting becomes more aggressive and utility deteriorates; without entropy scaling, forgetting quality falls with a relatively modest utility effect.The ablations attribute the contrastive objective's main redistribution signal to it, and associate soft clamping and adaptive scaling with stability and token-level selectivity.Source: §5.3 · p. 10

5.4 Additional Evaluation and Analysis

  • Across evaluated settings, utility and forget–retain behavior are relatively stable over λent and βbase ranges, while γfocal mainly controls the forgetting operating point.Across seeds, MU, TR, EM, and ES are stable, whereas FQ and PrivLeak vary more.Source: §5.4 · p. 11
  • RADNPO stays closer to the Original model than NPO on repetition and diversity statistics, and GPT-4o evaluation finds lower forget-set leakage and better retain-side semantic performance.Layer-wise probing shows a shift toward Retrain-like representations earlier than NPO in deeper layers, particularly layers 30–31; fine-tuning on the original forget set substantially restores forgotten behavior.Source: §5.4 · p. 11

6 Conclusion

The conclusion frames RADNPO as a response to concentrated post-suppression distributions and degraded generation. It reports favorable forgetting–utility trade-offs on TOFU and MUSE, while emphasizing local redistribution and approximate unlearning.

6 Conclusion

  • Target suppression can coexist with concentrated local distributions and degraded generation, motivating explicit guidance of probability redistribution.The conclusion describes this as a failure mode of LLM unlearning.Source: §6 · p. 11
  • RADNPO compares each forget target with alternatives selected from the current next-token distribution and scales token-level forgetting by target confidence and partial entropy.The method is reference-free and uses a local contrastive log-odds objective with adaptive scaling.Source: §6 · p. 11
  • Experiments on TOFU and MUSE show favorable forgetting–utility trade-offs, and TOFU Forget10 responses are less repetitive and more lexically diverse than NPO.The authors conclude that redistribution should be guided by the current token state rather than target suppression alone.Source: §6 · p. 11

AI Use Statement

The authors describe generative AI assistance in manuscript revision and supplementary experiments.

AI Use Statement

  • Generative AI tools assisted with language, clarity, organization, mathematical and methodological feedback, figure layouts, supplementary experiments, and semantic evaluation.The authors state that they take full responsibility for the manuscript, procedures, results, and conclusions.Source: AI Use Statement · p. 12

Ethics Statement

The paper presents its work as approximate unlearning on TOFU and MUSE, not a guarantee of complete or irreversible deletion. The accompanying algorithm initializes from an original model and combines forget and retain optimization.

Ethics Statement

  • The intended application is reducing reproduction of designated information while preserving useful behavior, but the results do not guarantee complete or irreversible deletion.Fine-tuning on the original forget set can restore forgotten behavior, so practical use should assess residual leakage and unintended behavior changes.Source: Ethics Statement · p. 12

A Algorithm of RADNPO

  • The algorithm takes forget and retain datasets, an initial model, a learning rate, and parameters for focal, entropy, cutoff, and clamping controls.It initializes the policy from the original model and outputs an unlearned model.Source: A Algorithm of RADNPO · p. 16
  • For each forget token, the algorithm applies soft-clamped contrastive odds, dual-state-aware dynamic beta, and a token forget loss.These are the named stages in its forget optimization loop.Source: A Algorithm of RADNPO · p. 16
  • The algorithm then computes a normal retain NLL loss and updates the model using the combined forget and retain losses.Source: A Algorithm of RADNPO · p. 16

B Theoretical analysis of RADNPO gradient rescaling

The appendices outline RADNPO coefficient handling, derive the NPO gradient, and organize evaluation metrics into three groups.

B Theoretical analysis of RADNPO gradient rescaling

C Gradient Derivation of the NPO Objective

D Evaluation Metrics

D.1 Memorization Metrics

The evaluation definitions cover forget-set reproduction, distributional and membership-inference diagnostics, and retained model utility. The analyses use LLaMA-2-7B with TOFU Forget10 unless otherwise stated.

D.1 Memorization Metrics

  • Probability and ROUGE assess target-output confidence and reference overlap, while Truth Ratio compares correct-answer preference with a perturbed alternative.Source: §D.1 · p. 18
  • Exact Memorization measures the share of response tokens matching the ground truth, while Extraction Strength measures the shortest prefix needed to reconstruct the target suffix.Source: §D.1 · p. 18
  • Verbatim Memorization evaluates generated continuations against reference text, with generation determined by the prompt and decoding configuration.Source: §D.1 · p. 18

D.2 Privacy Metrics

  • Forget Quality compares truth-ratio distributions with a retrained reference using a KS-test-derived score; it is diagnostic and does not establish model equivalence.Source: §D.2 · p. 19
  • Privacy Leakage compares attack AUC scores for the unlearned and retrained models under the same membership-inference protocol.Source: §D.2 · p. 19

D.3 Utility Metrics

  • Model Utility summarizes retained performance using a harmonic mean across retain, Real Authors, and World Facts evaluations.Source: §D.3 · p. 19

E Additional Evaluation Analyses

E.1 Generation-Level and Semantic Evaluation

On TOFU Forget10, RADNPO is reported to reduce generation degeneration relative to NPO and to score better in LLM-judged forgetting and retention. Further analyses probe internal representations, recovery after re-exposure, baseline methods, and sensitivity to hyperparameters.

E.1 Generation-Level and Semantic Evaluation

  • RADNPO stays close to the Original model on generation statistics, while outperforming NPO in overall judge score, forget-set leakage, and retain-side semantic performance.The reported comparisons indicate less degeneration and a better forgetting–retention trade-off than NPO.Source: Table 3 · p. 20, §E.1 · p. 20, Table 4 · p. 20

E.2 Representation-Probing Protocol

  • At layers 30–31, RADNPO shifts representations away from Original-like patterns earlier than NPO, though both methods have near-zero probe scores at the final layer.The probe compares representations from the Original and Retrain models; its scores indicate distributional similarity, not irreversible erasure.Source: Table 5 · p. 21, §E.2 · p. 21

E.3 Recovery Evaluation

  • Fine-tuning the RADNPO-unlearned model on the exact forget set substantially restores forgotten behavior, so the result supports approximate rather than irreversible unlearning.The experiment does not establish recovery from arbitrary downstream fine-tuning or incidental exposure to related data.Source: Table 6 · p. 21, §E.3 · p. 21

F Baseline Methods

  • The baseline descriptions span likelihood-based gradient ascent, preference methods, representation perturbation, and logit adjustment toward a uniform distribution.Examples include GradAscent, DPO, RMU, and UNDIAL.Source: F Baseline Methods · pp. 22–23

G.1 Computing Resources

  • Experiments were run on four NVIDIA 4090 GPUs in a single node.Source: §G.1 · p. 23

G.2 Experiment Setups

  • RADNPO uses a 10-epoch LLaMA-2 7B setup on TOFU, with different base-temperature and focal-exponent settings for MUSE; both use λent = 0.1.The setup also uses K = 10 for alternative-token selection and partial-entropy computation.Source: §G.2 · pp. 23–24

G.3 Sensitivity to RADNPO Hyperparameters

  • RADNPO is relatively stable across the evaluated λent and βbase ranges, while γfocal has a stronger effect on the forgetting operating point.Increasing γfocal from 1 to 2 lowers FQ and moves PrivLeak closer to its desired value, while slightly increasing EM, ES, and Forget TR.Source: Table 7 · p. 24, §G.3 · p. 24

G.4 Robustness Across Random Seeds

Three-seed results support stable forgetting–utility behavior, with greater variation in FQ and PrivLeak. A separate loss-weight sweep shows trade-offs between retained utility and forgetting; additional tables extend evaluation to LLaMA3-8B and Qwen3-4B.

G.4 Robustness Across Random Seeds

  • Across three seeds, MU, Forget TR, Retain TR, EM, and ES are stable, while FQ and PrivLeak vary more; the authors make no statistical-significance claim from three runs.Source: Table 8 · p. 25, §G.4 · p. 25

G.5 Sensitivity to the Retain-Forget Loss Balance

  • Intermediate retain weights of 0.4 and 0.6 yield lower FQ with comparable utility, while retain-heavy and forget-heavy settings trade utility against forgetting.The retain-heavy setting (0.8) has the highest MU and Retain TR but worse FQ; the forget-heavy setting (0.2) lowers Forget TR at a utility cost.Source: Table 9 · p. 25, §G.5 · p. 25

G.6 Extended experiments on LLaMA3-8B

  • The extended evaluation table covers TOFU Forget10 with the LLaMA3-8B-chat model and includes retrained and original references.Source: Table 10 · p. 26

G.7 Extended experiments on Qwen3-4B

  • The extended evaluation table covers TOFU Forget10 with the Qwen3-4B-Instruct model and includes retrained and original references.Source: Table 11 · p. 26
Loading 2609.34251v1…