PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Saliency for Machine Unlearning

Cedar Site Bai, Amber Yijia Zheng, Raymond A. Yeh, Brian Bullins

arXiv:2608.15548v1cs.LGcs.AI

TL;DR

Machine unlearning seeks to remove specific training data’s influence without full retraining while preserving utility. SSU thresholds weak singular components of unlearning gradients, with experiments reporting consistent improvements across classifiers, diffusion models, and LLMs.

  • Problem

    Machine unlearning seeks efficient removal of specific training data’s influence without full retraining while preserving model utility for data retained in training.

  • Method

    SSU applies singular-value thresholding to suppress weak spectral components of unlearning gradients, supported by a forgetting-retention trade-off analysis.

  • Results

    SSU consistently improves existing unlearning objectives, reducing the average CIFAR-10 gap by 30.6% and improving retained-class generation quality by 23.6%.

  • Takeaways & Limitations

    The spectral thresholding perspective offers a broadly applicable way to mitigate utility degradation in gradient-based unlearning across multiple model types.

  • Takeaways & Limitations

    The theory characterizes the forgetting-retention trade-off but does not provide a complete quantitative prediction of end-to-end unlearning performance.

Abstract

from arXiv · show

Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, using gradient-based updates to reduce the influence of a forget-set by counteracting the previously learned behavior. Recently, Muon, a gradient descent variant, has been introduced. Muon applies spectral magnitude normalization to encourage exploration of rare directions and demonstrates promising performance. Inspired by Muon, we adopt the spectral view for unlearning and propose Spectral Saliency Unlearning (SSU). SSU thresholds weak singular components and updates only those directions supported by a confident unlearning signal. We further provide theoretical justification for this thresholding approach from the perspective of the forgetting-retention trade-off. Experiments across image classifiers, diffusion models, and LLMs demonstrate SSU's effectiveness.

1 Introduction

The paper frames machine unlearning as an efficient, gradient-based reversal of forget-set influence and proposes SSU, which thresholds weak singular components to reduce forgetting–retention interference. Experiments across classifiers, diffusion models, and language models show consistent improvements with existing unlearning objectives.

  • Motivation: Machine unlearning removes specific training-data influence without full retraining, motivated partly by data-governance requirements such as GDPR.Recent approximate methods use a few gradient-based updates to reduce forget-set influence.
  • Conceptual framing: The paper conceptualizes approximate unlearning as partially reversing SGD’s accumulation of designated forget-set effects in model parameters.This perspective treats unlearning as undoing gradient-descent effects rather than retraining from scratch.
  • Motivation: Inspired by SignSGD, Muon, and SalUn, the paper hypothesizes that suppressing weak spectral components can improve unlearning by targeting rare, weakly expressed directions.Muon equalizes gradient-spectrum magnitudes to promote rare directions, while SalUn thresholds low-magnitude unlearning-gradient coordinates.
  • Method and theory: SSU applies singular-value thresholding to suppress weak spectral components of unlearning gradients and can be integrated into general gradient-based unlearning methods.The paper justifies this design through the forgetting-retention trade-off and extends the perspective to coordinate-wise masking.
  • Empirical evaluation: 30.6% average gap reduction on CIFAR-10 classification, perfect unlearning efficacy with DDPM, and 23.6% generation-quality improvement on retained classes demonstrate SSU’s empirical effectiveness.Evaluations cover image classifiers, diffusion models, and large language models, with consistent improvements when SSU is combined with existing MU objectives.

2 Preliminaries

The preliminaries formulate machine unlearning as gradient-based optimization that reduces forget-set influence while preserving retain-set utility, and characterize Muon through spectral matrix-sign updates.

  • Unlearning formulation: Machine unlearning updates pretrained parameters to reduce the forget set D_f's influence while preserving utility on the complementary retain set D_r := D \ D_f.The framework uses a forgetting loss L_f(W; D_f) and a retaining loss L_r(W; D_r).
  • Unlearning formulation: The forgetting and retaining losses are jointly optimized through gradient updates G_u = G_f + G_r, with G_f = ∇L_f(W; D_f) and G_r = ∇L_r(W; D_r).These updates minimize the unlearning objective.
  • Muon and matrix sign: Muon orthogonalizes layer gradients and is equivalent to applying the matrix sign in the gradient's spectral domain.For G = U diag(σ) V, msign(G) := U diag(sign(σ)) V.
  • Muon and matrix sign: Muon is therefore viewed as a matrix counterpart of SignSGD.This interpretation follows from its spectral matrix-sign operation.

3 Method

SSU replaces the base unlearning gradient with a singular-value-thresholded update that suppresses weak spectral components and retains dominant singular directions. It defines spectral saliency through singular magnitudes and applies thresholding before standard gradient-based updates.

  • Spectral Saliency Unlearning: SSU replaces the base unlearning gradient Gu with a singular-value-thresholded gradient that retains only dominant singular directions for updating.The full procedure is summarized in Algorithm 1.
  • Spectral Saliency: Spectral saliency is defined as singular magnitude σ_i, with larger values indicating stronger and more reliable unlearning signals.Small singular magnitudes represent weakly supported directions that may experience forgetting–retention interference.
  • Singular Value Thresholding: Singular value thresholding sets components below threshold τ to zero while preserving components above τ, equivalently discarding the sorted singular-value tail.The threshold satisfies τ ≥ 0.
  • Singular Value Thresholding: SSU retains the top γ fraction of singular components by setting k = ⌊γm⌋, then reconstructs the thresholded gradient for standard updates.The keep ratio satisfies γ ∈ (0, 1].
  • Parameter Handling: For vector- and scalar-structured parameters, SSU uses coordinate-wise thresholding as in SalUn rather than spectral updates.This mirrors Muon’s vector-based optimizer handling for such parameters.

4 Theoretical Justification

The theory characterizes forgetting–retention conflict directionally and justifies suppressing weak spectral components as a principled SSU mechanism, while clarifying that it does not predict end-to-end performance quantitatively.

  • Directional analysis: In the effective subspace of the unlearning gradient, each singular direction is a canonical one-dimensional update mode, and forget-retain alignment measures projected-gradient directional agreement.The effective subspace is induced by non-zero singular components, with each mode having magnitude σ_i.
  • Forgetting–retention conflict: Strongly negative forget-retain alignment creates an unavoidable trade-off: improving forgetting necessarily worsens retention, or vice versa, along that direction.Under smooth losses, sufficiently negative alignment and a sufficiently small step size imply ΔL_fΔL_r < 0.
  • Spectral thresholding: Small singular values identify directions with negatively aligned forgetting and retention gradients when those directions carry nontrivial signal; sufficiently separated alignment scores also imply σ_i < σ_j.Directions with no forgetting or retention signal lie in the null space of the unlearning gradient and are unaffected by thresholding.
  • Coordinate-wise thresholding: The same trade-off justifies coordinate-wise masking: a coordinate with nontrivial forgetting or retention signal but small combined gradient magnitude necessarily improves one objective while worsening the other.This provides theoretical support for SalUn-style thresholding beyond its primarily empirical motivation.
  • Scope and limitations: The results support SSU as a principled proxy for mitigating forget-retain conflict, but characterize the trade-off rather than provide complete quantitative end-to-end guarantees.Alignment-based thresholding could be more fine-grained but requires separately computing and projecting forgetting and retention gradients, whereas singular values come directly from G_u.

5 Experiments

Experiments across image classification, image generation, and language modeling show that SSU improves the forgetting–utility or forgetting–quality balance across model types and tasks. SSU also maintains minimal computational overhead and achieves strong qualitative removal of unlearned concepts while preserving retained outputs.

  • 5.1 Image Classification: On CIFAR-10 ResNet-18 with 10% data forgotten, SSU achieves the lowest average gap to the retrained oracle, improving on SalUn.The average gap is 0.77 for SSU versus 1.11 for SalUn, with comparable performance across all metrics.
  • 5.2 Image Generation: SSU’s spectral filtering adds minimal overhead because reshaped U-Net gradients typically form moderate-sized matrices suitable for efficient GPU SVD.On one NVIDIA L40S, SSU takes approximately 40 seconds per 100 training steps, nearly identical to SalUn.
  • 5.2 Image Generation: For Stable Diffusion on Imagenette, SSU achieves the best generation quality while maintaining strong forgetting across unlearned classes.SSU reaches an average FID of 1.24, while SalUn reaches 1.55 and ESD 1.71; SSU’s average FA is 0.20%, matching SalUn and improving on ESD’s 0.30%.
  • 5.2 Image Generation: SSU removes recognizable features from unlearned Stable Diffusion classes while retaining high-fidelity, semantically accurate generations for retained classes.For example, generated tench images become dog-like after unlearning.
  • 5.3 Language Modeling: SSU improves Llama2-7B unlearning when applied to the strongest baseline for each task, enhancing utility and forget efficacy across forgetting scenarios.For untargeted unlearning, model utility increases from 0.7245 to 0.7477 for forget01 and from 0.7312 to 0.7480 for forget10; for targeted unlearning, average scores are 0.7856, 0.7562, and 0.7475.

6 Related Work

Prior machine-unlearning work addresses forgetting–retention interference through retain-subspace projection, gradient alignment, saliency-based masking, and optimization-based updates. SSU instead uses singular values of the unlearning gradient as spectral saliency to suppress weak components and can enhance methods jointly optimizing forgetting and retention losses.

  • Forgetting-retention trade-off and spectral structure: Forgetting–retention interference motivates SVD-based projections onto subspaces orthogonal to estimated retain directions.This trade-off is also connected to catastrophic forgetting in continual learning.
  • Forgetting-retention trade-off and spectral structure: SSU operates directly on the unlearning gradient’s singular basis, suppressing weak spectral components through an optimizer-agnostic mechanism.Unlike approaches based solely on estimated retain subspaces, it targets forget–retain interaction directly.
  • Forgetting-retention trade-off and spectral structure: Related methods study forget–retain conflict through Euclidean gradient alignment, whereas SSU uses the unlearning-gradient spectrum as a structured interference proxy.SVD-based low-rank parameterization targets efficient updates, while SSU uses singular values for thresholding.
  • Saliency-based MU: SalUn thresholds low-magnitude coordinates from a forget-set-gradient saliency map, whereas SSU selects dominant matrix-valued directions using singular values.SSU’s spectral directions reflect signals from both forgetting and retention, unlike coordinate masking.
  • General machine unlearning: SSU complements optimization-based unlearning, including gradient ascent, influence-function approximations, Fisher-information updates, and fine-tuning, as a plug-in for joint forgetting–retention objectives.Many existing methods fit a common template that jointly optimizes forgetting and retention or utility-preserving losses.

7 Conclusion · Appendix · A Proof of the Theoretical Justification

The paper concludes by presenting SSU as a spectral, threshold-based enhancement for gradient-based unlearning and theoretically justifying it through the forgetting-retention trade-off. The appendix contains formal proofs, additional experiment details, and extra image-generation results.

  • 7 Conclusion: SSU applies singular-value thresholding to suppress weak spectral components of the unlearning gradient.The method is designed as a drop-in enhancement for gradient-based unlearning pipelines.
  • 7 Conclusion: SSU is motivated by a spectral view of gradient-based optimization.The conclusion frames the method through the spectral structure of the unlearning gradient.
  • 7 Conclusion: The theoretical justification for SSU is developed from the forgetting-retention trade-off.The conclusion characterizes weak directions in relation to forget-retain interference.
  • Appendix: The appendix begins by outlining its organization across three sections.Sections A, B, and C cover proofs, experiment details, and additional image-generation results, respectively.
  • A Proof of the Theoretical Justification: Section A provides the formal assumptions, theorem statements, and complete proofs for the main-paper theorems.This section corresponds to the proof material for the theoretical justification.
  • Appendix: Section B provides additional experiment details.The paper states that the code will be open-sourced upon acceptance.
  • Appendix: Section C provides additional image-generation results after unlearning with SSU.These results supplement the image-generation experiments described in the paper.

A.1 Formal Statement and Proof of Proposition 4.2

Proposition 4.2 establishes that, in singular directions where forgetting and retention gradients significantly conflict, improving either objective necessarily worsens the other. The proof uses the SVD-based update and smoothness-based sign relationships for the two loss changes.

  • Gradient decomposition: The unlearning gradient combines forgetting and retaining gradients as Gu = Gf + Gr and is decomposed into singular directions for analysis.The effective rank distinguishes significant nonzero singular values from approximately zero components.
  • Smoothness and sign conditions: Under smoothness, progress in forgetting requires f_i > 0, while progress in retention requires r_i > 0; opposite signs imply worsening of the corresponding loss.Specifically, f_i < 0 implies the forgetting loss increases, and r_i < 0 implies the retaining loss increases.
  • Singular-direction update: The update along singular direction vi is W′ = W − ησ_i u_i v_i^⊤.This direction-specific update underlies the loss-change analysis in Proposition 4.2.
  • Proposition 4.2: In significantly conflicting directions, ΔL_f < 0 implies ΔL_r > 0, so progress in forgetting necessarily damages retention.The conflict condition is defined through sufficiently negative forget-retain alignment.
  • Proposition 4.2: Conversely, in significantly conflicting directions, ΔL_r < 0 implies ΔL_f > 0, so progress in retention necessarily impedes forgetting.Thus, the proposition formalizes an unavoidable forgetting-retention trade-off along these singular directions.

A.2 Formal Statement and Proof of Proposition 4.3

The appendix defines retain and forget signals on an effective gradient subspace and proves that spectral thresholding favors directions with reliable forgetting signals under bounded spectral disparity.

  • Formal setup: The effective subspace E is spanned by the right singular vectors Ve, with projection P = VeV_e^⊤, and projected Gram matrices Mr and Mf represent retain and forget signals.These constructions restrict analysis to directions supported by the joint update signal.
  • Assumption A.3: Bounded spectral disparity assumes the projected retain and forget Gram matrices have spectra that are not excessively uneven across admissible directions.The assumption formalizes bulk-distributed signals within E and avoids orders-of-magnitude variation across directions.
  • Proposition 4.3(a): If a direction has nontrivial forget and retain signal and σ_i ≤ ρξ, its alignment satisfies a(v_i) ≤ 2ρ^2 − 1 < 0.The proposition uses signal magnitude at least ξ and a sufficiently small singular value to establish negative alignment.
  • Proposition 4.3(b): Under Assumption A.3, sufficiently separated alignment scores force a corresponding separation between the singular values of conflicting directions.The condition compares a(v_i) ≤ −δ_i and a(v_j) ≥ δ_j through the stated spectral-gap inequality.

A.3 Theoretical Justification for Salun

Under smooth forgetting and retaining losses, Proposition 4.4 shows that a coordinate-wise update with nontrivial forgetting/retention signal and sufficiently small combined unlearning gradient decreases one loss while increasing the other.

  • The analysis assumes that the forgetting loss Lf and retaining loss Lr are βf- and βr-smooth.
  • For each coordinate, the forgetting/retention signal is at least ξ, while the combined unlearning gradient satisfies |gu,i| ≤ ρξ with ρ ∈ (0, 1].
  • The resulting coordinate-wise update guarantees ΔLfΔLr < 0, establishing opposite changes in forgetting and retaining losses.
  • The proof derives sign(ΔLf) = −sign(gf,igu,i) and sign(ΔLr) = −sign(gr,igu,i), yielding the opposite-loss-change guarantee.

B Detailed Experiment Setup … B.3 Details of unlearning on large language models.

The experiments specify reproducible image-classification, image-generation, and large-language-model unlearning setups, combining fine-tuning objectives with layer-specific spectral filtering or saliency masks. Across tasks, the protocols define datasets, model configurations, optimization schedules, and filtering ratios.

  • B.1 Details of unlearning on image classification.: Image classification forgets 10% of training data, corresponding to 5,000 samples, selected with random seed 42.Evaluation uses retain-set, forget-set, and test-set accuracy with 45,000, 5,000, and 10,000 samples, respectively.
  • B.1 Details of unlearning on image classification.: Image-classification unlearning fine-tunes on the 45,000-sample retain set for 10 epochs at learning rate 0.013, using γ = 0.30 spectral filtering.SVD is applied on-the-fly to convolutional layers, while fully-connected layers use pre-computed saliency coordinate masks; γ controls the retained singular-value proportion and was grid-searched.
  • B.2 Details of unlearning on image generation.: CIFAR-10 DDPM unlearning uses a U-Net with 128 base channels, 1000 diffusion timesteps, and class-conditional generation targets.The architecture has channel multipliers [1] [2], two residual blocks per resolution, and self-attention at resolution 16.
  • B.2 Details of unlearning on image generation.: CIFAR-10 DDPM unlearning runs 1000 iterations with batch size 128 and gradient ascent-based loss, applying γ = 0.50 eigenvalue filtering.On-the-fly SVD reconstruction is used for convolutional layers, while non-convolutional layers use pre-computed saliency coordinate masks.
  • B.2 Details of unlearning on image generation.: Stable Diffusion on Imagenette uses a latent U-Net with 320 base channels, CLIP text conditioning, and a 1000-timestep linear noise schedule.The model has channel multipliers [1] [2] [4], two residual blocks per resolution, eight attention heads, and transformer depth 1.
  • B.2 Details of unlearning on image generation.: Imagenette unlearning fine-tunes for 5 epochs with batch size 8, random forget-class labels, Adam at learning rate 1 × 10−5, and γ = 0.50 filtering.SVD reconstruction is applied on-the-fly to cross-attention layers, while other layers use pre-computed saliency coordinate masks; the keep ratio is grid-searched.
  • B.3 Details of unlearning on large language models.: LLM unlearning fine-tunes for 5 epochs with batch size 8 and four gradient-accumulation steps, yielding effective batch size 32, under IDK+AP or ME+GD objectives.SVD reconstruction targets Q, K, and V attention projections with αqkv = 0.20, while feed-forward and other layers use coordinate masks at ratio 0.10.
  • B.3 Details of unlearning on large language models.: LLM experiments use method-specific learning rates and coefficients, top eigenvalue ratio 0.2, AdamW with weight decay 0.01, and distributed training across 2 GPUs.IDK+AP uses learning rate 3 × 10−5 and forget coefficient 1.0; ME+GD uses 2 × 10−5 and 0.1, while both use regularization coefficient 1.0.

C Additional generation results

Additional generation results evaluate SSU-based class-wise unlearning on Imagenette across different random seeds. The results organize generated images by unlearned and prompt classes, with diagonal outputs representing the forgetting target.

  • Additional generation results: Class-wise Imagenette unlearning results use SalUn and evaluate SSU under different random seeds.Figures 2–4 present these generated-image results.
  • Additional generation results: Each result figure is a matrix whose rows denote the unlearned class and columns denote the prompt class.This layout separates the intended unlearning target from generated outputs.
  • Additional generation results: Diagonal images represent the forgetting class, while non-diagonal images represent the remaining class in SSU-generated examples.The diagonal outputs highlight SSU’s removal of specific concepts.
Loading 2608.15548v1…