PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond

Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, Kilian Q. Weinberger

arXiv:2502.19301v1cs.LG

TL;DR

LLM unlearning must remove targeted undesirable knowledge without damaging non-targeted behavior, yet existing objectives lack a unified analysis framework. The paper introduces the gradient-effect toolkit to evaluate objective impacts across data, update steps, and layers, finding retention risks and motivating improved objectives such as WGA and TNPO. The G-effect also exposes limitations in current objectives and supports further methodological development.

  • Problem

    Existing LLM unlearning objectives lack a unified framework for understanding how they remove targeted knowledge while preserving non-targeted model integrity.

  • Method

    The paper introduces the G-effect, a gradient-based toolkit that analyzes unlearning objectives across data points, update steps, and model layers.

  • Results

    The analysis finds that unlearning affects shallow layers more, compromises retention, and can make excessive unlearning harm common responses; WGA and TNPO are introduced as new state-of-the-art objectives.

  • Takeaways & Limitations

    GA-based objectives remain promising, but excessive unlearning and harm to model integrity must be mitigated through improved objectives and analysis.

  • Takeaways & Limitations

    The G-effect assumes low variance in the singular values of matrix A, requires approximate Hessian estimation, and uses NLL as a potentially suboptimal risk metric.

Abstract

from arXiv · show

Large language models (LLMs) should undergo rigorous audits to identify potential risks, such as copyright and privacy infringements. Once these risks emerge, timely updates are crucial to remove undesirable responses, ensuring legal and safe model usage. It has spurred recent research into LLM unlearning, focusing on erasing targeted undesirable knowledge without compromising the integrity of other, non-targeted responses. Existing studies have introduced various unlearning objectives to pursue LLM unlearning without necessitating complete retraining. However, each of these objectives has unique properties, and no unified framework is currently available to comprehend them thoroughly. To fill the gap, we propose a toolkit of the gradient effect (G-effect), quantifying the impacts of unlearning objectives on model performance from a gradient perspective. A notable advantage is its broad ability to detail the unlearning impacts from various aspects across instances, updating steps, and LLM layers. Accordingly, the G-effect offers new insights into identifying drawbacks of existing unlearning objectives, further motivating us to explore a series of new solutions for their mitigation and improvements. Finally, we outline promising directions that merit further studies, aiming at contributing to the community to advance this important field.

1 INTRODUCTION

LLM unlearning seeks to remove targeted knowledge while preserving non-targeted model behavior, but existing objectives lack a unified analytical framework. The paper proposes the G-effect toolkit and reports findings about layer effects, retention, excessive unlearning, risk weighting, and regularization.

  • LLM unlearning aims to remove targeted knowledge while preserving the integrity of non-targeted responses.
  • The G-effect provides a gradient-based toolkit for analyzing unlearning objectives across data points, update steps, and model layers.
  • Shallow layers are more affected than deeper layers, indicating substantial alterations to general knowledge encoded in shallow layers.
  • Current unlearning objectives fail to retain overall performance during unlearning.
  • Excessive unlearning can deteriorate common model responses more than it improves unlearning.
  • Risk weighting is effective for unlearning, while regularization—especially KL—helps maintain overall model integrity.
  • Experiments on the TOFU fictitious unlearning datasets identify several new state-of-the-art methods and research directions.

2 LLM UNLEARNING

LLM unlearning addresses risks from sensitive training data by removing targeted knowledge while retaining performance on non-targeted data. The paper distinguishes full removal from influence removal and discusses gradient ascent as a practical but potentially excessive objective.

  • LLM unlearning responds to legal and ethical risks from sensitive data in web-sourced training corpora.
  • Its goals are to remove targeted data or knowledge and preserve model responses on non-targeted data.
  • Removal requires targeted-data performance to deteriorate, whereas retention requires non-targeted-data performance to remain stable or improve.
  • The paper focuses on full removal, which erases targeted knowledge as much as possible, rather than exclusively matching behavior from a model never trained on that data.
  • Gradient ascent increases targeted-data NLL to implement unlearning but is prone to excessive unlearning.

3 G-EFFECT

The G-effect evaluates unlearning objectives by comparing their update gradients with risk gradients, revealing whether updates support removal or retention. Its gradient formulation also enables analysis across data, steps, and layers.

  • Performance differences before and after unlearning evaluate removal and retention but provide limited insight into underlying mechanisms.
  • The G-effect quantifies alignment between an unlearning objective’s gradient and the risk metric’s gradient using their dot product.
  • A positive dot product indicates potential improvement in the measured risk, whereas a negative dot product indicates potential harm.
  • The removal G-effect compares the unlearning gradient with targeted-data risk, while the retention G-effect compares it with non-targeted-data risk.
  • Effective unlearning seeks negative removal effects and non-negative retention effects.
  • Figure 1 identifies the intersection of the removal and retention regions as satisfying both unlearning goals.

4 ANALYSIS FOR UNLEARNING OBJECTIVES

The G-effect analysis compares unlearning and retention impacts across update steps, layers, instances, and objectives. It finds that several methods trade unlearning strength against model integrity, while weighting and regularization offer mitigation paths with important limitations.

  • Gradient Ascent (GA): GA’s unlearning G-effect declines rapidly from about 0 to −3.5 × 10^5, while retaining effects show similarly scaled deterioration on non-targeted data.Later near-zero values do not indicate relearning because G-effects accumulate across updates; severe damage can already occur earlier.
  • Gradient Ascent (GA): GA affects shallow layers more than middle and deep layers, while the input embedding layer is relatively negligible and the output unembedding layer remains notably large.The layer pattern suggests stronger distortion of general knowledge in shallow layers, with distinct behavior at the output layer.
  • Gradient Ascent (GA): WGA reduces GA’s negative impact on common data relative to targeted-data improvements by controlling the extent of unlearning.The analysis characterizes this mechanism as early stopping, and notes that sufficiently controlled GA can sometimes outweigh integrity deterioration with unlearning improvement.
  • Negative Preference Optimization (NPO): NPO converges faster than GA with smaller G-effect magnitudes, indicating weaker unlearning strength but better preservation of model integrity.When β = 1 or 2, retaining-effect magnitudes outweigh unlearning magnitudes, indicating a retention-related trade-off.
  • Negative Preference Optimization (NPO): NPO weighting can prioritize points with small negative retaining effects, but may sacrifice unlearning strength and misassign weights in important failure cases.The desired mechanism would prioritize points with near-zero retaining G-effect and large negative unlearning G-effect, which current weighting does not reliably achieve.
  • Negative Preference Optimization (NPO): TNPO produces sufficiently large negative unlearning effects while retaining effects remain overall close to zero, supporting token-level prioritization during unlearning.The reported token-wise weighting is presented as achieving unlearning efficacy, with WTNPO discussed as a further improvement.
  • More Objectives: PO has limited validity for erasing targeted knowledge beyond early updating phases and may subsequently facilitate knowledge relearning.Its suitability for LLM unlearning is therefore constrained by its changing behavior across update stages.
  • More Objectives: RMU is highly sensitive to the representation layer and scaling hyper-parameter: middle layers perform better than shallow or deep layers, while wrong configurations may oppose unlearning.The analysis evaluates outputs from the 11-th, 22-th, and 33-th layers before unembedding.

5 EVALUATIONS

The evaluation benchmarks unlearning objectives on TOFU using two LLMs and targeted-data ratios of 1%, 5%, and 10%. Results show trade-offs between removal and retention, with WGA and WTNPO generally performing best while metric disagreement remains.

  • Experimental setup: The experiments evaluate Phi-1.5 and Llama-2-7B on TOFU with 1%, 5%, and 10% targeted-data ratios.TOFU separates targeted from non-targeted author-profile questions for evaluating removal and retention.
  • Evaluation metrics: Forget quality measures unlearning, while model utility assesses output changes relative to a gold-standard model trained without targeted data.The evaluation uses log-scaled p-values for model utility and seeks high values for both metrics.
  • Existing objectives: GA removes targeted data most effectively but compromises model integrity, whereas NPO and RMU provide better removal-retention balance, with NPO usually outperforming RMU.The exception is 10% unlearning with Llama-2-7B, and the authors attribute NPO’s advantage to its more stable G-effect.
  • New objectives: WGA mitigates GA’s excessive unlearning while retaining strong removal, and TNPO and WTNPO improve NPO’s unlearning and retention performance.Across setups and models, WGA and WTNPO stand out; WGA is recommended as the default because it requires tuning only one hyper-parameter.
  • Metric analysis: The proposed methods generally outperform prior counterparts on model utility and forget quality, but one 5% Llama-2-7B case shows disagreement between evaluation metrics.In that case, NPO’s forget-quality score surpasses the proposed methods despite their stronger retention.

6 CONCLUSIONS

The paper uses the G-effect to analyze LLM unlearning objectives, finding that GA-based methods remain promising but require controls for excessive unlearning and harm to model integrity. It introduces WGA and TNPO as advanced objectives and identifies regularization, weighting, and representation methods as directions for improvement.

  • The G-effect reveals that unlearning affects shallow layers more, altering general knowledge encoded there.
  • Current unlearning objectives compromise retention, failing to preserve overall model performance during unlearning.
  • Excessive unlearning can make deterioration in common responses outweigh improvements in targeted-data removal.
  • The G-effect has limitations because its assumptions about matrix singular values, Hessian estimation, and NLL-based risk may reduce accuracy or insight.The paper specifically notes that the smoothness matrix may be inadequately modeled, Hessian computation requires approximation, and NLL may not optimally characterize knowledge parametrization.

ETHIC STATEMENT AND REPRODUCIBILITY

The paper frames the G-effect as a gradient-based toolkit for analyzing how unlearning objectives affect removal and retention across update steps and model components. Its formal analysis connects these effects to gradient alignment while noting assumptions and trade-offs underlying the approximation.

  • The G-effect quantifies how an unlearning objective changes model performance through the dot product of risk and objective gradients.
  • The matrix A captures model smoothness, learning-rate effects, and mini-batch ordering in the accumulated gradient updates.
  • A good objective should jointly support removal of targeted data and retention of performance on common data.
  • The approximation depends on the unlearned and original parameters remaining sufficiently close; otherwise, first-order Taylor errors may become large.
  • The framework distinguishes objectives by whether their gradients favor retention, unlearning, both, or neither.
  • Experiments support the framework by showing stronger targeted-data removal for GA than NPO and smaller retaining effects for WGA than GA.

B.1 TOFU BENCHMARKS

The appendix evaluates G-effect dynamics on TOFU fictitious unlearning benchmarks and examines how update steps, layers, and weighting relate to removal and retention. The results show that GA can become excessively aggressive, especially as shallow-layer effects and inverse confidence grow.

  • B.1 TOFU BENCHMARKS: The evaluation uses TOFU fictitious author profiles with Phi-1.5 and Llama-2-7B under 1%, 5%, and 10% unlearning settings.
  • B.1 TOFU BENCHMARKS: The benchmark compares unlearning and retaining behavior using extraction strength, where lower targeted-data values indicate stronger removal.
  • C.1 GA: GA’s G-effect dynamics align with unlearning risk, including a drop from about the 20-th to 40-th steps.
  • C.1 GA: More than 10^17 inverse confidence around the 30-th step contributes to excessive unlearning.
  • C.1 GA: GA can damage model integrity more than it improves unlearning, with shallow-layer G-effects larger than those in middle and deep layers.

C.2 NPO

The paper analyzes NPO, RMU, WGA, and TNPO through G-effect dynamics, showing how weighting, layer choice, and confidence control shape the removal–retention trade-off. These analyses motivate new objectives that moderate excessive unlearning while preserving removal strength.

  • C.2 NPO: NPO’s G-effect magnitudes increase as β decreases, while the gap between retaining and unlearning effects narrows.
  • C.2 NPO: As β approaches 0, NPO increasingly resembles GA, and its weighting mechanism may contribute to faster convergence.
  • C.3 RMU: RMU favors perturbing middle or shallow layers over deep layers, whose perturbation makes unlearning notably unstable.
  • C.3 RMU: The paper recommends middle-layer representations for RMU because deep-layer perturbations are unstable and scaling parameters produce instability.
  • D MORE DISCUSSIONS FOR NEW UNLEARNING OBJECTIVES: The paper introduces WGA, TNPO, and WTNPO to address excessive unlearning and better exploit weighting mechanisms.
  • D.1 WGA: WGA controls inverse-confidence weighting to mitigate GA’s excessive unlearning, while larger α can cause premature convergence.

D.2 TNPO AND WTNPO

TNPO modifies NPO by applying its weighting mechanism token-wise, while WTNPO adds confidence weighting to stabilize the procedure. The analyses indicate trade-offs between unlearning strength and preservation of model integrity.

  • TNPO applies NPO’s weighting mechanism to individual tokens rather than entire data points, increasing control over the unlearning procedure.
  • At β = 1, TNPO improves unlearning with negligible deterioration in model integrity, making it a preferred objective in the reported analysis.
  • For β = 0.1, TNPO improves unlearning between the 30-th and 40-th steps, but later reductions in unlearning G-effect also reduce retaining G-effect.
  • WTNPO combines TNPO with WGA’s weighting mechanism, and confidence weighting stabilizes unlearning while weakening its strength.
  • WTNPO therefore involves a trade-off across different α values between stability and unlearning strength.

E REGULARIZATION

The section reviews regularization terms intended to preserve non-targeted behavior or embedding features during unlearning. It also motivates further analysis of loss weighting as a way to improve unlearning while preserving integrity.

  • GD, KL, and RR are regularization terms integrated into unlearning objectives to improve stability or preserve model properties.
  • GD decreases the negative log-likelihood for non-targeted data, while KL maintains responses close to those before unlearning.
  • RR is designed to maintain embedding features during unlearning.
  • The experiments integrate these regularization terms directly into unlearning objectives without additional trade-off hyper-parameters.
  • Loss weighting is highlighted as a promising direction for enhancing unlearning while preserving model integrity.

F.1 NPO WEIGHTING MECHANISMS

The analysis examines how NPO’s weighting mechanism relates to point-wise G-effects across inverse temperatures and unlearning steps. It finds useful prioritization but also unstable and semantically counter-intuitive weighting patterns.

  • NPO weighting is analyzed through relationships between w_npo,su and PG-effect across inverse-temperature settings and checkpoints.
  • Larger β enhances distinctions between PG-effect distributions, whereas β approaching 0 makes w_npo,su converge to 1 and NPO resemble conventional GA.
  • At β = 1 and β = 2, NPO can assign moderate-to-high weights to data points with large negative retaining G-effect, indicating failures in protecting integrity.
  • Across checkpoints, weighting makes more errors initially than later, and changing distribution layouts indicate instability that loss weighting might improve.
  • Token-wise analyses visualize TNPO and WTNPO weights to investigate which information receives attention.
  • The token-weighting mechanism does not consistently assign higher weights to informative knowledge, such as key prize-related tokens, and this issue is more pronounced for WTNPO.

G MORE RESULTS

The experiments benchmark the studied unlearning methods with UWC and ES metrics on TOFU fictitious unlearning datasets. These evaluations support comparisons across methods and hyper-parameter settings.

  • The study benchmarks its methods using UWC and ES metrics on TOFU fictitious unlearning datasets.
  • The UWC evaluation framework quantifies knowledge parametrization and helps address hyper-parameter challenges arising from the trade-off between unlearning and retention.
  • The reported tuning tables cover WGA, NPO, TNPO, WTNPO, and RMU configurations across shallow, middle, and deep settings.
Loading 2502.19301v1…