Source-linked AI summary
Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models
Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, Elita Lobo
TL;DR
LLM unlearning can suppress forgotten knowledge while producing incoherent or inconsistent responses and reducing utility. AltPO combines negative feedback with in-domain positive feedback and adds evaluations for forget-set response quality. The paper reports strong unlearning, improved response behavior, and preserved overall performance, while noting scope limits for QA-formatted factual knowledge and benchmark diversity.
Problem
Existing LLM unlearning methods can produce nonsensical or inconsistent forget-set responses, while standard evaluations overlook these behaviors and utility changes on forgotten entities.
Method
AltPO uses prompt-specific in-domain alternate answers as positive samples, contrasts them with forget responses using DPO, and preserves retain-set behavior with NLL.
Results
AltPO achieves the highest unlearning scores on existing metrics while obtaining better and near-perfect scores on existing and new evaluation metrics.
Takeaways & Limitations
Combining negative feedback with in-domain positive feedback supports stable factual-knowledge unlearning while avoiding undesirable model behavior and maintaining overall performance.
Takeaways & Limitations
AltPO is designed for factual knowledge represented as QA datasets, and broader evaluation would require adaptation and more diverse benchmarks.
Abstract
from arXiv · showhide
Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality of responses related to the forget set. Extensive experiments show that our approach not only enables effective unlearning but also avoids undesirable model behaviors while maintaining overall model performance. Our implementation can be found at https://github.com/molereddy/Alternate-Preference-Optimization.
1 Introduction
Machine unlearning seeks to remove specific training data, but LLM unlearning can produce incoherent responses and reduce utility. AltPO addresses this by combining negative feedback with plausible alternative positive responses and introduces evaluations for overlooked failure modes.
- Approximate unlearning modifies post-training model weights to reduce the influence of data targeted for forgetting.
- Existing LLM unlearning methods can produce incoherent or inconsistent responses related to forgotten knowledge.
- AltPO combines negative feedback with plausible alternative responses to achieve more stable and effective unlearning.
- The paper identifies failure modes not captured by existing metrics and introduces new evaluation metrics to address them.
- On TOFU, AltPO-unlearned models achieve the highest existing unlearning scores while obtaining better and near-perfect scores on existing and new metrics.
2 Preliminaries
The framework defines forgetting and retention on QA-based LLM data, evaluates both unlearning and retained utility, and contrasts negative-feedback losses with positive-feedback mechanisms. Negative feedback suppresses forgotten responses but can harm coherence and utility when overgeneralized.
- TOFU evaluates forgetting subsets containing 1%, 5%, or 10% of fictitious authors while measuring performance on the remaining retain set.
- Model Utility aggregates average probability, ROUGE, and Truth Ratio on non-forget data to measure retained general performance.
- Negative feedback lowers the likelihood of forget-set responses, while positive feedback preserves language generation and performance beyond the forget set.
- Unlearning methods pair negative feedback with positive feedback because overgeneralized suppression can produce nonsensical responses and harm model utility.
- DPO-based losses contrast positive and negative samples, increasing the former while reducing the latter.
3 AltPO: Alternate Preference Optimization
AltPO uses prompt-specific, in-distribution alternate answers as positive feedback while contrasting them with forgotten responses and preserving retain-set behavior. Multiple alternatives are used to avoid confident replication of any single replacement answer.
- 3 AltPO: Alternate Preference Optimization: Prior NPO and IdkPO methods can produce nonsensical or inconsistent outputs because they lack suitable in-distribution positive feedback or use prompt-independent alternatives.
- 3 AltPO: Alternate Preference Optimization: AltPO generates plausible, prompt-specific alternate answers as in-distribution positive feedback, addressing the instability and incoherence of prior methods.
- 3 AltPO: Alternate Preference Optimization: The objective aligns the model with alternate labels while contrasting them against forget responses through a DPO variant.
- 3 AltPO: Alternate Preference Optimization: Retain-set NLL is added to prevent the model from generalizing unlearning to unrelated contexts.
- 3 AltPO: Alternate Preference Optimization: AltPO samples multiple alternate responses so the model does not become certain of a single replacement answer.
4 Improving Unlearning Evaluations
The section identifies incoherent forget-set generations and privacy leakage as failure modes that existing TOFU metrics do not adequately capture. It introduces Forget Utility and Cleanness Indistinguishability to evaluate response quality and privacy-related indistinguishability.
- 4.1 Failure Modes of Prior Approaches: Existing unlearning methods can produce nonsensical or prompt-inconsistent forget-set responses, including contradictions and altered names.These failures can degrade model performance and may expose information about the training data or unlearning process.
- 4.1 Failure Modes of Prior Approaches: TOFU’s forget-quality metric evaluates predefined-sentence probabilities rather than generated forget-set responses, while its utility metrics omit forget entities.Consequently, incoherence and reduced utility on forget prompts are not fully reflected.
- 4.2 New Evaluation Metrics: The authors introduce Forget Utility (FU) and Cleanness Indistinguishability (CI) to capture forget-set response quality and privacy leakage.FU penalizes nonsensical and prompt-inconsistent outputs, whereas CI measures distinguishability between unlearned and retain models using nonsensical-response behavior.
- 4.2 New Evaluation Metrics: CI uses Text Cleanness scores and a KS test to compare nonsensical-response distributions between unlearned and retain models.Text Cleanness is based on the probability that a generated response is non-gibberish, estimated with a DistilBERT-based classifier.
- 4.2 New Evaluation Metrics: Mean Text Cleanness provides a cheaper alternative to FU that detects nonsensical responses but does not penalize inconsistent answers.The authors therefore report FU in the main results and provide Text Cleanness scores in the appendix.
5 Related Work
Related work stabilizes unlearning with alternate positive labels, whereas AltPO generates multiple prompt-consistent answers without selecting anchor words and aligns them using a DPO-style objective. The paper also evaluates prior approaches against newly identified failure modes and reports the limitations of reverse perplexity for nonsensical text.
- 5 Related Work: Eldan and Russinovich (2023) and Dong et al. (2024) substitute privacy-sensitive anchor words with alternate positive token-level labels to stabilize unlearning.Eldan and Russinovich use GPT-4 to identify anchors, while Dong et al. treat all nouns as anchors.
- 5 Related Work: AltPO avoids selecting anchor words by generating multiple alternate answers consistent with the original question.It directly instructs an LLM to generate alternatives rather than deriving completions from next-token probabilities or GPT-4-proposed substitutions.
- 5 Related Work: Unlike prior cross-entropy approaches, AltPO uses a DPO-style loss that aligns the model with alternate answers while explicitly incorporating negative feedback.The paper reports ablations examining how these design elements affect performance.
- 5 Related Work: Reverse perplexity was found to be non-robust under greedy decoding because repetitive nonsensical text can receive high probabilities.This limits its reliability as an evaluation of nonsensical generations.
6 Experiments
Across TOFU splits on Llama2, AltPO combines effective forgetting with retained utility, coherent forget-set responses, and more stable training behavior. Ablations support using both negative and in-domain positive feedback, DPO-style loss, and multiple alternate answers.
- Extent of forgetting: AltPO achieves superior forgetting across the 1%, 5%, and 10% TOFU splits while remaining nearly indistinguishable from the gold retain model on Truth Ratio and Text Cleanness distributions.The reported KS-test p-values significantly exceed 0.05 for these distributions.
- Model performance post-unlearning: AltPO retains the initial model utility of 0.62 and produces more coherent, question-consistent responses on forget prompts than methods with comparable MU.The advantage appears in the FU metric, which evaluates forget-prompt response quality.
- Ablation experiments: Using both negative and positive forget feedback is necessary: negative-only feedback can damage coherent responses, whereas positive-only feedback does not sufficiently remove learned knowledge.The ablations compare NPO with AltPO and AltNLL-pos or AltPO-pos with their counterparts.
- Ablation experiments: AltPO outperforms its NLL-based variant across forgetting and utility metrics, supporting the DPO-style loss for delivering negative feedback.The result is consistent with the NPO-versus-GradDiff comparison, where NPO yields better FU and MU with fewer gibberish or inconsistent responses.
- Ablation experiments: Increasing alternate answers generally improves forgetting and lowers self-confidence, with M = 5 used by default, while M = 10 reduces FQ to 0.25 from 0.74 at M = 5.Even at M = 10, the reported FQ remains above the 0.05 statistical significance threshold; low self-confidence can instead reflect nonsensical outputs for NPO.
7 Conclusion
The paper proposes AltPO for factual knowledge unlearning in LLMs, combining negative feedback with in-domain positive feedback and introducing broader evaluation metrics. It reports effective unlearning while avoiding undesirable behaviors and maintaining overall model performance.
- AltPO combines negative feedback with in-domain positive feedback on forget-set prompts for more stable and effective factual knowledge unlearning.The method addresses nonsensical responses associated with negative-only feedback or incorrectly applied positive feedback.
8 Limitations
The study focuses on forgetting specific factual knowledge and does not determine the ideal behavior of an unlearned model. AltPO is designed for QA-represented factual data, and broader validation remains needed.
- The study does not address whether an unlearned model should refuse forgotten questions or behave like a model retrained without the forget set.Practitioners are expected to adapt post-unlearning behavior after the initial forgetting step.
- AltPO is specifically designed for factual knowledge represented as QA datasets, so applying it to other training-data formats requires further adaptation.
- The work would benefit from more extensive experiments across diverse benchmarks and datasets.Reliable retain models for forget-quality evaluation are difficult to construct because open-source models may already contain relevant QA data.
A Literature Review
The literature review distinguishes early classification-focused unlearning, parameter- or prompt-based LLM methods, and direct weight modification for factual knowledge. It situates AltPO among approaches addressing incoherent outputs and evaluation limitations.
- Early approaches: Early machine-unlearning methods targeted computer-vision classification, using input or representation perturbations, randomized targets, or outputs from randomly initialized models.
- LLM unlearning methods: LLM unlearning has also used classifiers, prompt-space changes, auxiliary models, and parameter-efficient fine-tuning without directly modifying forget-set-influenced parameters.These approaches identify forget prompts, list unlearning data in prompts, or train auxiliary components for the forget set.
- Weight modification: For factual knowledge unlearning, direct weight-modification methods include gradient ascent on forget-set responses, but prior work reports brittle nonsensical outputs from simple negative forgetting.
- Related benchmarks: RWKU shares key elements with AltPO through DPO-style unlearning of real-world entities, but it differs by lacking a defined forget dataset introducing the entities.
- Evaluation: TOFU evaluates probabilities assigned to correct answers and normalizes them, addressing limitations of ROUGE-based and lower-is-better forgetting evaluations.The review notes that even a model retrained without forget-set knowledge can assign probability to plausible texts.
B Additional Implementation Details
The additional details describe training, tuning, alternate-answer construction, evaluation prompts, and CI/TC trajectory visualizations for TOFU experiments.
- Training: Experiments use one-epoch warmup, paged AdamW, bf16 precision, and metrics from the last checkpoint on an NVIDIA A100 GPU.
- Alternate-answer construction: Alternate-answer training expands the dataset by M, pairing each example with every alternate label and shuffling the resulting examples.Each alternate label is seen exactly once per epoch, though labels for one example may span update steps.
- Training objectives: Retain-set positive feedback is included in all methods except GradAsc because prior work identifies it as essential.
- Hyperparameter tuning: Grid search selects method-specific hyperparameters over learning rates, DPO β values, and retain-loss weights for fair baseline comparison.
- Additional evaluations: Figures 5 and 6 show CI and TC trajectories throughout unlearning for the 10% TOFU split on Llama2.
- Evaluation prompts: The alternate-answer prompt uses a zero-shot design, while Table 5 documents the GPT4o-mini system prompt for LLM-as-judge text-quality evaluation.The zero-shot choice was made because few-shot examples biased generated alternate answers.
D Additional Results
Additional results report comprehensive baseline comparisons across TOFU forget-split sizes and models, alongside ablations and metric trajectories. The appendix also documents evaluation prompts, alternate labels, and post-unlearning responses.
- Additional Results: Tables 8–10 compare all baselines and AltPO across TOFU 10%, 5%, and 1% forget splits on Llama2-7b.The reported metrics are Forget Quality, Cleanness Indistinguishability, Model Utility, Text Cleanness, and Forget Utility.
- Additional Results: Results on Llama3.2-3B-Instruct for the 10% forget split are provided, with trends reported as consistent with Llama2.
- Additional Results: Ablations on the Llama2 10% split compare different loss functions and report forgetting and utility performance.The ablation results are summarized in Table 12, with M set to 5 unless otherwise specified.
- Additional Results: Figures 5–6 track CI and TC over training, while Figures 7–21 show metric trajectories for methods across all forget splits on Llama2.The trajectory plots include more fine-grained TOFU metrics.
- Additional Results: Supplementary materials include the judge prompt for Forget Utility, alternate labels before unlearning, and generated forget-set responses after unlearning.The response table contrasts the true answer with each method’s generated response.