Large Language Model Unlearning via Embedding-Corrupted Prompts

1 Introduction

• Prior LLM unlearning methods can impair general capabilities through knowledge entanglement and may become impractical as gradient-based optimization costs scale with model size.The entanglement arises from the fuzzy boundary between retention and forgetting, such as forgetting one person without affecting related people.Source: §1 · p. 2
• ECO decomposes unlearning into identifying target prompts and ensuring responses to them no longer reflect prior knowledge about the target.The framework is designed to address both knowledge entanglement and unlearning inefficiency.Source: §1 · p. 2
• A prompt classifier flags target prompts, then learned zeroth-order corruptions are applied in embedding space during inference.The corruption is applied to prompts rather than to the model’s weights.Source: §1 · p. 2
• ECO Prompts imposes an unlearned state on an intact LLM, with reported performance across three tasks, virtually zero side effects, and no added cost when scaling to larger models.The authors report universal effectiveness across 100 LLMs, from 0.5B to 236B parameters.Source: §1 · p. 3

2.1 Threat Model

• Users interact through a chat interface or structured API and may see completions or per-token logits; adversaries are assumed to issue in-distribution prompts and attempt jailbreaks.The paper also identifies timeliness as a concern and aims for unlearning that can operate in real time.Source: §2.1 · p. 3
• The threat model covers entity leaks, hazardous knowledge, and extraction of memorized copyrighted passages.For hazardous knowledge, the objective is to prevent access to dangerous knowledge while retaining capabilities in benign but related domains.Source: §2.1 · p. 3

2.2 Problem Setup

• The weak-unlearning setup compares output behavior with a retained model across forget, retain, and unknown general-domain distributions.Evaluation asks whether metric values are empirically similar across those distributions; users are not assumed to access model weights.Source: §2.2 · pp. 3–4

3.1 Method Overview

• The prompt classifier decides whether to replace an incoming prompt with a corrupted version or pass the original prompt to the LLM.A prompt is selected for forgetting when its predicted forget probability exceeds its retain probability.Source: §3.1 · p. 4
• ECO corrupts token embeddings rather than token identities, using a parameterized corruption function to produce the altered prompt embeddings.The corruption parameter controls corruption strength.Source: §3.1 · p. 4

3.2 Decision Threshold Calibration and Conformal Prediction

• Simple thresholding uses a calibrated cutoff τ to decide whether to apply corruption, with calibration aiming to minimize false positives and false negatives.The cutoff is selected on a separate calibration set.Source: §3.2 · p. 5
• Conformal prediction offers an alternative calibrated decision rule based on a target error rate and non-conformity scores from a separate calibration set.The method forms a prediction set for a new prompt and applies corruption when the forget label is included.Source: §3.2 · p. 5

3.3 Embedding-Corrupted Prompts

• The optimization objective minimizes a metric gap between the corrupted model and retained-model behavior, using a surrogate retained metric when available.If that metric is unavailable, the corruption strength may instead be tuned by inspecting forget-set outputs against the unlearning criteria.Source: §3.3 · p. 5
• Zeroth-order optimization treats the distance as a black box and estimates the update to scalar corruption strength using finite differences.The update uses an initial strength, step size, and smoothing parameter.Source: §3.3 · p. 6
• The method corrupts only the first dimension of each token embedding, and the authors report similar outcomes across tested corruption functions.Tested alternatives include Gaussian noise and zeroing top entries.Source: §3.3 · p. 6

4.1 Prompt Classifier

• Prompt classifiers distinguish retain and forget prompts and generalize to unseen prompts with low false-positive rates.Classifier training uses the full forget set for entity and copyrighted-content tasks, but a synthetic surrogate for WMDP and MMLU.Source: §4.1 · p. 6

4.2 Entity Unlearning

• TOFU measures forgetting by output-distribution similarity and utility by performance on retained authors and real-world knowledge.The experiments use synthetic author biographies and assess forgetting 1%, 5%, or 10% of fictitious authors.Source: §4.2 · p. 6
• ECO-RN and ECO-ZO achieve near-perfect forget quality across models and forget-set sizes while preserving utility on TOFU.The setup uses 1%, 5%, and 10% fictitious-author forget sets; ECO-ZO's outputs are nearly identical in distribution to the retained model.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Hazardous Knowledge Unlearning

• ECO reaches near-random accuracy on WMDP subsets across models while maintaining original MMLU performance.The WMDP experiments use ECO-RN with a corruption strength optimized on synthetic questions for Zephyr-7B and reused across models.Source: §4.3 · pp. 7–8, Table 1 · p. 8
• On MMLU subset unlearning, ECO forgets the three selected subjects without the related-subject accuracy losses reported for RMU.The selected subjects are economics, law, and physics, with econometrics, jurisprudence, and math as closely related retain subjects.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Copyrighted Content Unlearning

• Across Harry Potter and BBC News unlearning, ECO maintains similarity to the retained model, meaningful and diverse outputs, and utility.The evaluation compares average similarity gap, perplexity, and unique-token ratio; results are also reported across 19 models.Source: §4.4 · p. 9, Table 3

• Traditional LLM unlearning methods use gradient updates, weight editing, or weight arithmetic, with compute that scales with model size.Optimization approaches may combine forgetting updates with retained-data training to preserve utility.Source: §5 · pp. 9–10
• Prompt-based in-context unlearning and prompt-injection guardrails depend on instruction-following models and can degrade regular-task performance.The cited guardrail baseline reports significant performance degradation on regular tasks.Source: §5 · p. 10
• Embedding perturbations can elicit desired responses, but jailbreak-style approaches are described as prohibitively expensive and require specifying desirable tokens.Prior work also showed that perturbations to token embeddings avoid optimizing over discrete tokens.Source: §5 · p. 10

6 Conclusion

• ECO uses a thresholded prompt classifier and leaves the LLM intact, with reported effectiveness across models from 0.5B to 236B parameters.The conclusion reports minimal side effects and no additional computational overhead.Source: §6 · p. 10

A Broader Impact

• Selective removal of critical information could enable misinformation or biased outputs, while classifier errors could leave target knowledge or harm non-sensitive tasks.The discussion calls for auditing, transparency, and ongoing classifier monitoring and refinement.Source: A Broader Impact · p. 21

B Limitations

• ECO requires API access, can be bypassed through open-weight access, and does not address motivated attacks on the classifier or LLM.Its classifier typically examines only the first or last K tokens; neutral text at both ends can evade it, while a sliding-window technique is suggested as mitigation.Source: B Limitations · p. 21

C Detailed Experimental Setup

• The experimental setup covers evaluation metrics, prompt-classifier training and evaluation, model preparation, and baseline formulations.The setup is described in the appendices, including metrics in C.1, model preparation in C.2, classifier procedures in C.3, and baselines in C.4.Source: C Detailed Experimental Setup · p. 21

C.1.1 TOFU

• TOFU utility aggregates answer probability, truth ratio, and ROUGE-L across retain, real-author, and world-fact subsets; forget quality compares truth-ratio distributions.For real-author and world-fact subsets, answer probability contrasts the correct answer with four perturbed answers; forget quality uses a KS-test p-value comparing retained and unlearned models on the forget set.Source: §C.1.1 · pp. 21–22

C.1.2 WMDP and MMLU

• WMDP and MMLU use four-choice accuracy, where 0.25 indicates random guessing; a linear probe also tests whether logits reveal correct answers.The probe is trained on half the data and evaluated on the other half; successful unlearning should yield chance accuracy.Source: §C.1.2 · p. 22

C.1.3 Harry Potter Book and BBC News Articles

• Copyrighted-text evaluations compare generated text with reference text using four similarity metrics and assess fluency, diversity, and similarity to retained-model outputs.The metrics are ROUGE-L, SacreBLEU, BERTScore, and METEOR; average similarity gap aggregates their absolute differences, while perplexity and unique-token ratio assess fluency and diversity.Source: §C.1.3 · p. 23

C.1.4 Why Not Membership Inference Attacks (MIAs)?

• The study omits membership-inference attacks because their data and model-access requirements are impractical or outside its threat model, and it excludes privacy risks.The authors also cite weak attack performance, potential overestimation, and distribution-related confounds; forget-quality and probing evaluations instead assess behavioral indistinguishability.Source: §C.1.4 · pp. 23–24

C.2 Preparing LLMs for Unlearning

• TOFU models are fine-tuned on the full dataset and on 99%, 95%, or 90% retained splits; WMDP evaluation uses pretrained models without task-specific fine-tuning.For copyrighted content, models are fine-tuned on the selected corpora; BBC articles are from February 2024, and Harry Potter text is split into chunks of up to 256 tokens.Source: §C.2 · p. 24

C.3 Prompt Classifiers

• The classifier appendix reports dataset split statistics and classifier false-positive and false-negative rates, with error rates calculated using calibrated thresholds.It reports results for the original, simple-thresholding, and conformal-prediction classifiers.Source: §C.3 · pp. 24–25, Table 4 · p. 25

C.3.1 Prompt Classifiers’ Training Datasets

• TOFU classifiers follow the dataset’s forget and retain splits; real-author and world-fact data assess false positives, without a test forget set for entities.Source: §C.3.1 · p. 26
• WMDP classifier training uses limited real questions plus MMLU negatives; synthetic data was expanded after topic-line overfitting, and the topic line was removed.The initial synthetic set had 300 questions; later training included 3,000 synthetic WMDP problems and synthetic retain problems based on MMLU development data.Source: §C.3.1 · pp. 26–27
• With only two chemistry questions in the out-of-distribution setup, conformal prediction reduced false negatives by about 14%; uncertain two-label predictions count as positive.The conformal-prediction performance is slightly worse than simple thresholding because uncertain predictions are counted as positive.Source: Table 7 · p. 26, §C.3.1 · p. 27
• Separate MMLU classifiers target economics, physics, and law; each uses only 10–15 forget-target development examples and synthetic training data.The topic line was removed to avoid overfitting, and false-positive and false-negative rates are reported on the forget and related-retain subjects.Source: §C.3.1 · p. 27
• The Harry Potter classifier treats sufficiently long book sentences as positive and BookMIA snippets as negative; the BBC classifier uses article titles as positive samples.The BBC setup uses CC-News articles as negatives and title-only classification; sentence-level training is suggested as a possible approach for extraction attacks.Source: §C.3.1 · pp. 27–28
• General utility is evaluated on 41,297 samples from eleven out-of-distribution LLM benchmarks, using test sets when labels are public and validation sets otherwise.TruthfulQA uses its MC1 subset.Source: §C.3.1 · p. 28

C.3 Prompt Classifiers

• The best prompt classifiers achieve low false-negative rates on forget data and low false-positive rates on retain data; most have zero false positives on general benchmarks.Classifiers use RoBERTa-base or Llama-3.1-1B-Instruct, with class-wise losses reweighted by inverse frequency.Source: §C.3.2 · p. 28

C.3.3 Robustness Against Out-of-Distribution Prompts

• The original TOFU classifier remained robust to common human-written prompt perturbations without explicit training on those perturbations.The test prompts included rephrasings, adversarial prompts, irrelevant context, jailbreak prefixes or suffixes, and keyword-only prompts.Source: §C.3.3 · pp. 29–30, Table 10 · p. 29
• Training on synthetic out-of-distribution prompts improved both false positive and false negative rates when evaluated on human-generated prompts.The synthetic training prompts were generated without overlap with the earlier perturbed test set; the procedure was repeated for all classifiers.Source: Table 11 · p. 29, §C.3.3 · p. 30

C.4 Baseline Methods

• The baseline-method appendix formulates the prompting and optimization-based unlearning methods used in the paper.The prompting baselines follow the style of [88], with author names or unlearning subjects supplied in the prompts.Source: §C.4 · p. 30, §C.4.1 · p. 30

C.4.1 Prompting Baseline

• Prompting baselines could potentially improve through more sophisticated prompts, but prompt development was left to future work.The TOFU prompt names authors to forget; WMDP and MMLU prompts substitute the subject to unlearn.Source: §C.4.1 · p. 30

C.4.2 Optimization-Based Unlearning Baselines

• Optimization-based baselines use 5 epochs and tune learning rates over {5e-5, 2e-5, 1e-5}; SSD uses log-perplexity criteria on forget and retain sets, while RMU uses a hidden representation at layer ℓ and a fixed random unit vector.Source: §C.4.2 · pp. 30, 32

C.6 Usage of Compute Resources

• Experiments used a node with eight NVIDIA A100 or A6000 GPUs, with at most three required per experiment; runs on models above 100B parameters typically took 2–5 days.Source: §C.6 · p. 32

D.1 Prompt Classifier Thresholding

• Thresholding experiments compared no thresholding, simple thresholding, and conformal prediction; on out-of-distribution WMDP, conformal prediction reduced false negatives by more than 50%.Raising a simple threshold brought false positives near-perfect levels but increased false negatives; the authors recommend choosing based on a relatively large held-out set.Source: §D.1 · p. 33

C.5 A Toy Example of Conformal Prediction

• In the conformal-prediction toy example, scores 0.18 and 0.82 both fall below the 0.93 quantile, yielding the prediction set {0, 1}.The example uses α = 0.05.Source: §C.5 · p. 32, §D.1 · p. 33

D.2 Corruption Function Variants

• The tested variants included Gaussian noise, top-k zeroing, sign flipping, reversing or shuffling embeddings, and noise or zero-out applied to selected entries.For sign flipping, random noise, and zero-out, selection could use the first N or top-k entries; random noise also tested random N entries.Source: §D.2 · p. 33
• Sign flipping and zero-out had consistently high forget quality on TOFU, while randomized corruption could require extra strength tuning, especially for larger forget sets.The ablations used the same strength for similar corruption functions rather than tuning each function separately.Source: §D.2 · p. 33, Table 12 · p. 33
• Most tested corruption functions reached ASG below 5 while maintaining low perplexity and high unique token ratio on BBC News and HP Book.The reported ablations cover BBC News on OLMo-7B and HP Book on OLMo-7B.Source: §D.2 · p. 33, Table 13 · p. 34, Table 14 · p. 34
• Corruption choice and strength mattered less for BBC News and HP Book than for TOFU under text-similarity metrics; forget quality instead measures distributional similarity to a retrained model.The authors recommend searching for the best function and strength using the available criteria.Source: §D.2 · p. 34

D.3 Task-Agnostic Selection of the Surrogate Metric Value

• Task-agnostic selection of the surrogate metric value maintained unlearning effectiveness, suggesting task-dependent selection was not necessary in the reported experiments.The approach minimized text similarity between model outputs and synthetic “I do not know” or refusal responses, generated independently of the tasks.Source: §D.3 · p. 34, Table 15 · p. 35

E Additional Experiments

• The appendix presents these results as additional experiments supporting claims and findings in the main paper.Source: E Additional Experiments · p. 35

E.1 Time Delays of Prompt Content Detection

• Prompt content detection usually added no more than 21 ms per example, with the largest reported delay at 85 ms for TOFU (Forget10).The 85 ms case included an additional BERT token classifier to identify names; measurements used Llama-2-7B(-Chat), batch size 4, and one NVIDIA A100.Source: §E.1 · p. 35, Table 16 · p. 35

E.2.1 Full Results

• The full-result tables report conditional answer probability, truth ratio, ROUGE-L, model utility, and forget quality across retain and forget sets, real authors, world facts, and forget-set sizes.Figure 2 plots model utility and forget quality from the final two columns of those tables.Source: §E.2.1 · p. 35, Table 17 · p. 36, Table 18 · p. 37
• The sign-flip variant had low forget quality across all splits in both Llama-2-7B-Chat and Phi-1.5 results.The authors hypothesize that sign flipping drastically alters embeddings, producing a higher truth ratio than the retained model and substantially different truth-ratio distributions.Source: §E.2.1 · p. 35, Table 17 · p. 36, Table 18 · p. 37

E.2.2 TOFU Example Generations Under Embedding Corruptions

• The appendix includes examples of Llama-2-7B-Chat responses to original prompts and embedding-corrupted prompts.Source: Table 19 · p. 38

E.3.1 Full Results

• Prompting-baseline unlearning becomes less effective as a model's original WMDP performance increases.For ECO, post-unlearning accuracy does not depend on the model's original performance.Source: §E.3.1 · p. 39
• ECO reduces WMDP performance to random-guess level in models specialized in biology, chemistry, or coding.The table caption says this holds regardless of the models' prior domain knowledge.Source: Table 20 · p. 39
• For 78 general LLMs (0.5B–236B parameters) on WMDP, ECO reduces accuracy to close to random-guess level (about 0.25) without exceptions, regardless of each model’s original task performance.Source: §E.3.1 · p. 39, Table 21 · p. 40

E.3.2 Probing Evaluation

• Before ECO, linear probes recover a substantial proportion of correct WMDP labels from logits of Zephyr-7B, Yi-34B-Chat, and Mixtral-8x7B-Instruct.Source: §E.3.2 · p. 41
• After ECO, linear-probe accuracy drops to random-chance level across all three models, regardless of model size or performance.The evaluation probes output logits for recoverable answers.Source: §E.3.2 · p. 41, Figure 3 · p. 41

E.4.1 Full Results

• ECO consistently reduces MMLU subset accuracy to random-guessing across model sizes while causing minimal to no performance loss on retain subjects.The reported economics forget accuracy of 35.8 reflects a bias toward answer D, which accounts for about 35% of correct answers, so the authors still consider it random-guessing.Source: §E.4.1 · p. 42, Table 22 · p. 43

E.5 Copyrighted Content

• Across models, ECO maintains stable performance on BBC News and HP book unlearning, with generated text exhibiting low perplexity and high diversity, rivaling LLMU [143].The evaluation reports BERTScore F1, METEOR, ROUGE-L, SacreBLEU, utility on eleven benchmarks, perplexity, and unique token ratio.Source: §E.5 · p. 44

E.5.1 Copyrighted Content Example Generations Under Embedding Corruptions

• The appendix tables provide examples of embedding-corrupted prompt completions for Harry Potter and BBC News content.The supplied captions identify examples from Harry Potter and the Sorcerer’s Stone and BBC News articles from February 2024, without reporting their contents here.Source: Table 61 · p. 54, Table 62 · p. 55

E.6 What Are Embedding-Corrupted Prompts to LLMs?

• For Llama-3-70B-Instruct, across corruption methods, increasing corruption strength prevents repeating the prompt while answers remain possible; corrupted prompts seem to act as ghost tokens that models can answer but not repeat.The same pattern was observed across models ranging from a few billion to hundreds of billions of parameters; further investigation is needed to understand behavior on different prompt types.Source: §E.6 · p. 56

E.6.1 Model’s Interpretation and Responses Under Noise-Corrupted Prompts

• Under random-noise corruption, stronger corruption makes the model unable to interpret the prompt and its responses deviate from the original question’s context.The described examples use Llama-3-70B-Instruct and corrupt all prompt tokens.Source: §E.6.1 · p. 56, Table 63 · p. 57

E.6.2 Model’s Interpretation and Responses Under Sparsely-Corrupted Prompts

• Under zero-out corruption, the model may claim there is no text to repeat and state it cannot understand the query, unlike random corruption, which steers answers away from the question.The difference between corruption schemes suggests that more nuanced input corruption could enable more precise output steering.Source: §E.6.2 · p. 59

E.7 Incorporating Template Answers as A Baseline

• The authors argue that corrupted-prompt responses better preserve response indistinguishability than template answers, which can diverge from retained-model performance and reveal classifier targeting.The comparison of template responses and other unlearning methods is reported in Table 71.Source: §E.7 · p. 62, Table 71 · p. 63

2. Limitations

• The authors report providing task-specific experimental setups in Section 4 and Appendix C, including models, datasets, hyperparameters, and other relevant details.They also state that training and test details are covered in those sections.Source: §2 · pp. 65–66
• The authors released their code at the ECO GitHub repository.The passage supplies the repository URL: https://github.com/chrisliu298/llm-unlearn-eco.Source: §2 · p. 66
• Each experiment was repeated at least three times and average results are reported, but statistical significance was not computed for every experiment due to computational constraints.The authors answer No to reporting error bars or other suitable statistical-significance information for every experiment.Source: §2 · p. 66
• The authors state that compute-resource details are briefly provided in Appendix C.6.Source: §2 · p. 67
• The authors report discussing the method’s broader impacts in Appendix A and releasing no data or models that pose high risks.Source: §2 · pp. 67–68
• The authors report that the work introduces no new assets and involves neither crowdsourcing experiments nor research with human subjects.Source: §2 · p. 69