SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning

1 Introduction

• Unlearning must remove undesirable data influences and associated generation capabilities without compromising utility outside the unlearning scope.Retraining from scratch is described as impractical for LLMs because of its high cost; fine-tuning can instead over-forget or under-forget.Source: §1 · p. 1
• SOUL applies second-order optimization to LLM unlearning, linking it to influence unlearning and extending that approach into an iterative framework.The authors describe SOUL as loss-agnostic and suitable for augmenting existing unlearning approaches.Source: §1 · p. 2
• Prior fine-tuning methods commonly use first-order optimization, while the impact of optimizer choice on LLM unlearning had not been specifically studied.Existing methods include regularized objectives combining unlearning efficacy loss with model utility loss.Source: §1 · p. 2
• The paper reports that SOUL improves unlearning efficacy while preserving model utility across various objectives and formulations.Figure 1 illustrates fictitious unlearning on TOFU with example outputs and a comparison of forget quality and model utility.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

• Machine unlearning research spans image classification, image generation, and federated learning, in addition to its data-protection origins.The passage describes the field's applications as having expanded across these areas.Source: §2 · p. 2
• Exact unlearning retrains after excluding forgotten data, while costly retraining and full-data access needs have motivated scalable approximate methods.The passage also notes methods that provide provable and certified data removal, often using differential privacy.Source: §2 · pp. 2–3
• LLM unlearning approaches include gradient ascent, preference optimization, weight localization, hidden-representation changes, and input-based strategies.TOFU and WMDP are cited as benchmarks for fictitious and hazardous-knowledge unlearning, respectively.Source: §2 · p. 3
• Despite the proliferation of LLM unlearning research, the influence of optimizer selection remains unexplored.The passage reviews work on approaches and benchmarks before identifying this gap.Source: §2 · p. 3

3 Primer on LLM Unlearning

• The prevailing optimization setup fine-tunes a pretrained model over forget and retain sets, balancing their losses with a regularization parameter.The forget set contains information intended for removal, while the retain set contains data irrelevant to that target.Source: §3 · p. 3
• The choice of forget loss and the balance between unlearning and utility remain challenging design questions.The paper presents GradDiff, PO, and NPO as representative approaches to these formulation choices.Source: §3 · pp. 3–4
• GradDiff maximizes training loss on the forget set while minimizing it on the retain set to uphold performance on unrelated tasks.At λ = 0, the formulation reduces to maximizing forget-set training loss, termed gradient ascent.Source: §3 · pp. 3–4
• Preference Optimization replaces the unbounded gradient-ascent loss with an alignment loss based on designated unlearning responses.Those responses may be reject-based or irrelevant to the unlearning target.Source: §3 · p. 4
• Negative Preference Optimization treats forget-set data as negative examples and uses them alone in preference optimization.The passage contrasts this with PO, which specifies an unlearning response, and describes NPO as replacing GA loss with a negative-examples-based preference loss.Source: §3 · p. 4

4 Second-Order Optimization to Enhance LLM Unlearning: Why & How

• Influence unlearning estimates the weight change from a pretrained model to one retrained on the retain set by adjusting data weights for the forget set.The retrained model is implicitly defined by a weighted training problem, and influence functions approximate its change using a first-order Taylor expansion.Source: §4 · pp. 4–5
• Influence unlearning is limited by costly Hessian inversion and reduced accuracy from Taylor-expansion and second-order-information approximations.Prior use was limited to vision tasks and small models, and it was not considered state of the art even for vision unlearning.Source: §4 · pp. 4–5
• SOUL extends influence unlearning’s static, one-shot model update into an iterative second-order optimization process for LLM unlearning.The authors propose this transition to mitigate accuracy loss from influence unlearning’s approximations while avoiding exact Hessian inversion through a practical optimizer.Source: §4 · p. 5
• Sophia uses an exponential moving average of gradients and diagonal Gauss–Newton Hessian estimates, with element-wise clipping to limit update magnitudes.Without clipping, with γ = 1 and ϵ approaching 0, its update simplifies to a Newton update using a diagonal Hessian estimate.Source: §4 · p. 5
• Influence unlearning’s inverse-Hessian gradient term has the form of a second-order ascent step, linking it to Newton-style optimization.The paper uses this structural connection to motivate integrating second-order optimization into an iterative unlearning approach.Source: §4 · p. 5
• For GradDiff, SOUL uses second-order descent on the retain set and ascent on the forget set; for preference-optimization problems, it uses descent for both losses.The preference response is already defined in preference-optimization problems, so the forget loss is minimized rather than maximized.Source: §4 · p. 6

5.1 Experiment setups

• Experiments cover TOFU fictitious unlearning, Harry Potter copyright removal, and detoxification, using OPT-1.3B and LLaMA2-7b base models.Methods include GradDiff, PO, NPO, GA, and prompt-enabled unlearning; evaluation covers unlearning efficacy and preserved utility.Source: §5.1 · p. 6, Table 1 · p. 6

5.3 Results on copyright removal

• Across TOFU, copyright removal, and detoxification, SOUL improves unlearning efficacy while maintaining or improving utility relative to first-order methods.The detoxification findings extend to both LLaMA2-7B and smaller OPT-350M models.Source: §5.2 · p. 7, §5.3 · p. 8

5.2 Results on fictitious unlearning in TOFU

• SOUL-based methods improve TOFU forget quality, MIA, accuracy, and Rouge-L versus first-order counterparts while preserving competitive utility; SO-PO has high MIA.SO-PO does not explicitly reduce Min-k% probability for the correct answer, so forgotten data can still be recognized as training data.Source: §5.2 · p. 7
• On TOFU examples, SO-PO answers utility questions accurately and declines forget-set questions, while GradDiff can produce nonsensical forget-set responses.World-facts performance also favors PO over GradDiff for accurate responses.Source: §5.2 · p. 7

5.3 Results on copyright removal

• 0.21 BLEU decrease for SO-GradDiff versus FO-GradDiff at prompt length 300 on LLaMA2-7B; SOUL also improves perplexity and zero-shot accuracy.GA achieves low unlearning scores on LLaMA2-7B but has poor utility, with perplexity 15.66.Source: §5.3 · p. 8
• Detoxification results show conclusions similar to TOFU and copyright removal for both LLaMA2-7B and smaller OPT-350M models.Source: §5.3 · p. 8

5.4 Iterative unlearning benefits from SOUL

• On TOFU, SOUL converges to better forget performance faster than GA and GradDiff, while adapting retain performance; GA sharply loses retention at the final epoch.Figure 2 plots forget accuracy and retain accuracy against optimization epochs.Source: §5.4 · p. 8, Figure 2 · p. 8
• On TOFU, SOUL improves static influence unlearning through iterative optimization, shows better jailbreak robustness than first-order methods, and incurs no substantial time cost.Sophia’s efficient Hessian-diagonal estimate avoids the extensive computation typically associated with second-order optimization.Source: §5.4 · p. 8

• The paper investigates optimizer choice in LLM unlearning and links second-order optimization to influence unlearning.Source: §6 · p. 8
• SOUL is a loss-agnostic second-order LLM unlearning framework proposed to augment existing approaches.Source: §6 · p. 9
• Experiments across various unlearning tasks, models, and metrics consistently show the superiority of second-order unlearning.Source: §6 · p. 9

7 Limitations

• Experiments primarily use OPT-1.3B and LLaMA2-7b, leaving applicability to larger models for future investigation.Source: §7 · p. 9
• Robustness remains incompletely tested across jailbreaking attacks and changing unlearning targets.Source: §7 · p. 9

A Algorithm

• SOUL iteratively computes the loss gradient, updates its EMA, estimates and smooths the Hessian diagonal, then updates parameters using both estimates.Source: A Algorithm · p. 13
• For PO-type problems, SOUL uses descent mode because the preference is already defined and the corresponding forget loss is minimized.This unifies optimization of forget and retain losses through descent mode.Source: A Algorithm · p. 13

B.1 Datasets, tasks and models

• The study evaluates TOFU fictitious unlearning, Harry Potter copyright removal, and detoxification, with task-specific forget sets and C4 retain data for the latter two tasks.TOFU uses a 10% forget ratio (400 examples about 20 authors); copyright removal uses 200 book chunks, and detoxification uses 200 negative PKU-SafeRLHF samples.Source: §B.1 · p. 13
• Experiments use OPT-1.3B and LLaMA2-7b; TOFU uses fine-tuned LLaMA2-7b-chat, while detoxification uses the original models.For copyright removal, both models are fine-tuned on the Harry Potter series before unlearning.Source: §B.1 · p. 13

B.2 Unlearning configurations.

• GradDiff, PO, and NPO are evaluated with first-order optimization or SOUL, alongside gradient ascent and an input-prompt unlearning method.Gradient ascent is GradDiff with λ = 0; PO uses a reject-based answer as its target, and the input-based approach uses system prompts as prefixes.Source: §B.2 · pp. 13–14
• AdamW is the first-order optimizer, while SOUL uses Sophia with selected hyperparameters, including β1 = 0.9, β2 = 0.95, γ = 0.04, and ϵ = 1 × 10−5.Learning rate and λ were selected through grid search; the reject-based answers and input-method system prompt are summarized in Tables A1 and A2.Source: §B.2 · p. 14, Table A1 · p. 14, Table A2 · p. 14

B.3 Evaluation metrics

• TOFU unlearning is assessed with forget quality, MIA, and generated-answer accuracy and Rouge-L recall, with the same accuracy and Rouge-L measures applied to retained information, real authors, and world facts.Forget quality is 1−p-value from a KS test, while MIA uses Min-k% Probability AUC to detect whether forgotten data was in training.Source: §B.3 · pp. 15–16
• Copyright removal is evaluated by BLEU and Rouge-L recall on 300 truncated Harry Potter excerpts at prompt lengths of 100 and 300 tokens.Source: §B.3 · p. 16
• Detoxification uses Toxic-BERT toxicity scores on real toxic prompts and PKU-SafeRLHF, while utility is assessed with Wikitext perplexity and mean zero-shot accuracy across eight tasks.Instruction-following ability is not evaluated because the primary models are pretrained LLMs not adapted using RLHF.Source: §B.3 · p. 16

B.4 Additional visualization

• In copyright-removal examples, first-order PO can still generate Harry Potter-related content, whereas second-order outputs reject; a similar pattern is reported for GradDiff.Source: §B.4 · p. 16
• In detoxification examples, SO-PO generates non-toxic content, while FO-PO occasionally retains toxic elements; PO is reported to perform best in these examples.Source: §B.4 · p. 16

B.5 Results on LLM detoxification

• Second-order unlearning reduces toxicity scores on both Real Toxicity Prompts and PKU-SafeRLHF while maintaining or improving utility.For LLaMA2-7B, SO-PO reduces the PKU-SafeRLHF toxicity score and improves zero-shot accuracy compared with FO-PO.Source: §B.5 · p. 17

B.6 Performance comparison between IU and SOUL

• Traditional influence unlearning does not yield satisfactory LLM unlearning effectiveness in the reported comparison, while SOUL replaces its static one-shot update with iterative, optimization-driven influence-aware unlearning.Source: §B.6 · p. 17

B.7 Adversarial evaluation for SOUL

• Jailbreaking can degrade unlearning efficacy, as reflected by increased forget accuracy, but the tested non-optimization-based prompts may leave forget accuracy unchanged after unlearning.Source: §B.7 · p. 17
• After jailbreak prompts, SOUL consistently has lower forget accuracy than first-order methods, indicating a robustness benefit in this evaluation.The comparison concerns forget accuracy before and after jailbreaking on TOFU.Source: §B.7 · p. 17, Table A9 · p. 17

• Sophia approximates diagonal Hessian values with element-wise squared gradients, making the additional computational overhead minimal compared with first-order updates.Source: §B.8 · p. 17
• The experiments update the Hessian at every optimization step.Source: §B.8 · p. 17
• The TOFU time comparison shows that second-order optimization does not incur significantly greater overhead than first-order methods.Source: Table A10 · p. 17, §B.8 · p. 17