Spectral Saliency for Machine Unlearning

• Approximate machine unlearning reduces a forget set’s influence through a few gradient-based updates, framed here as undoing the effect of gradient descent on that set [12, 24, 45].The introduction situates this need in data-governance requirements such as the GDPR.Source: §1 · p. 1
• Muon promotes rare spectral directions through magnitude normalization, motivating SSU’s suppression of weak components that may encode entangled forgetting and retention.The introduction relates weaker singular directions to specific structure learned later [27] [31] [38].Source: §1 · pp. 1–2
• SSU thresholds weak singular components of unlearning gradients, can plug into general gradient-based methods, and is justified as mitigating utility degradation from forgetting–retention interference.The paper characterizes how weak directions relate to interference between forgetting and retention.Source: §1 · p. 2
• The paper extends its forgetting–retention analysis to coordinate-wise masking, offering a theoretical explanation for SalUn-style unlearning, previously used largely as a heuristic [12].The introduction describes SalUn as thresholding low-magnitude coordinates of the unlearning gradient [12].Source: §1 · p. 2
• SSU yields a 30.6% reduction in the average gap on CIFAR-10, perfect DDPM unlearning efficacy with 23.6% better retained-class generation quality, and a 0.0125 average gain on TOFU.The introduction reports these results when SSU is used with existing machine-unlearning objectives.Source: §1 · p. 2

2 Preliminaries

• Approximate unlearning updates pretrained parameters to reduce the influence of forget set Df while preserving utility on retain set Dr := D \ Df [16] [17] [29] [36].The objective jointly optimizes forgetting loss Lf(W; Df) and retaining loss Lr(W; Dr).Source: §2 · p. 2
• At each step, the unlearning gradient is Gu = Gf + Gr, with Gf and Gr denoting the gradients of the forgetting and retaining losses, respectively.The two losses are optimized jointly via gradient-based updates to minimize the unlearning objective.Source: §2 · p. 2
• Muon orthogonalizes layer gradients, an operation equivalent to applying the matrix sign in the gradient’s spectral domain [9].The paper therefore views Muon as a matrix counterpart of SignSGD.Source: §2 · p. 2

3.1 Spectral Saliency

• A direction’s spectral saliency is its singular magnitude: larger magnitudes indicate stronger, more reliable unlearning signals, while small ones are weakly supported and may invite forgetting–retention interference.The paper states that this intuition is formally justified in Sec. 4.Source: §3.1 · p. 3

3.2 Singular Value Thresholding

• SSU keeps the top γ fraction of singular components (k=⌊γm⌋), zeros the tail, then applies standard gradient updates; vector/scalar parameters use coordinate-wise thresholding.Unlike SalUn [12], which selects coordinates using dataset-specific computations, SVD intrinsically orders matrix directions; vector/scalar parameters follow SalUn-style handling.Source: §3.2 · pp. 3–4

4.1 Why Spectral Thresholding Helps

• Strong negative alignment between forgetting and retention gradients makes their objectives conflict, so an update along that direction improves one objective while worsening the other.The stated result assumes smooth losses, significant negative alignment, and a sufficiently small step size.Source: §4.1 · pp. 4–5
• When a direction has nontrivial forget-retain signal but a small singular value, its alignment is negative; under bounded spectral disparity, sufficiently positively aligned directions have larger singular values than sufficiently negative ones.The first relation requires the singular value to be small relative to the signal; the second holds when the separation between alignment scores exceeds a gap threshold.Source: §4.1 · p. 5
• Suppressing small singular components preferentially removes directions likely to have strong forget-retain conflict, mitigating unnecessary retention degradation while preserving the dominant unlearning signal.Directions with neither forgetting nor retention signal lie in the null space of the unlearning gradient and are unaffected by thresholding.Source: §4.1 · p. 5

4.2 Justification for SalUn-Style Masking

• For coordinate-wise masking, a small combined gradient despite nontrivial forgetting or retention signal suggests cancellation, making updates likely to couple forgetting progress with retention degradation.Under the proposition’s stated signal and gradient-magnitude conditions, the forgetting and retention progress have opposite signs.Source: §4.2 · p. 5

4.3 Limitations and Discussion

• The theory characterizes the forgetting-retention trade-off rather than providing a complete quantitative prediction of end-to-end unlearning performance; weaker assumptions and tighter performance-level guarantees remain future work.The analysis presents SSU as a principled mechanism for mitigating conflict by suppressing weak spectral components.Source: §4.3 · pp. 5–6
• Alignment-based thresholding would require separately computing and projecting the forgetting and retention gradients, whereas singular values come directly from the unlearning gradient and serve as a principled conflict proxy.The paper focuses on singular-value thresholding as a simple, broadly applicable choice, while noting that a finer alignment-aware criterion may improve thresholding.Source: §4.3 · p. 6

5.1 Random Subset Unlearning in Image Classification

• 0.77 average gap to the retrained oracle improves on SalUn’s best-baseline 1.11 in CIFAR-10 random-subset unlearning.Baselines expose a forgetting–utility trade-off: some forget insufficiently, while SSD’s lower forgetting-set accuracy comes with reduced remaining- and test-set accuracy.Source: Table 1 · p. 6, §5.1 · p. 6

5.2 Class-wise Unlearning in Image Generation

• The method requires SVD computation during training but has minimal overhead; on one NVIDIA L40S, it takes approximately 40 seconds per 100 training steps, nearly identical to SalUn, while providing better unlearning through spectral filtering.For U-Net architectures, gradients are reshaped into moderate-sized 2D matrices, typically no larger than 512 × 512, for SVD.Source: §5.2 · p. 7
• 1.24 average FID is SSU’s best overall result on Stable Diffusion, versus 1.55 for SalUn and 1.71 for ESD; average FA is 0.20%, matching SalUn.SSU improves across most classes and its generated samples show forgetting of targeted concepts while retaining high-fidelity, semantically accurate generations for retained classes.Source: Table 3 · p. 7, §5.2 · p. 7, Figure 1 · p. 8

5.3 Random Subset Unlearning in LLM

• 0.7477 versus 0.7245 Model Utility on forget01 and 0.7480 versus 0.7312 on forget10: SSU on ME+GD also improves Forget Efficacy across all scenarios.ME+GD is the strongest untargeted baseline among the evaluated gradient-based, preference-based, and knowledge-manipulation methods.Source: §5.3 · p. 8, Table 4 · p. 9
• 0.7856, 0.7562, and 0.7475 are SSU’s average scores across the three targeted-unlearning scenarios, improving on the IDK+AP baseline.The gains are particularly noticeable for forget01, where SSU improves both model utility and forget efficacy.Source: §5.3 · p. 8, Table 4 · p. 9

5.4 Ablation Studies

• Keep ratio 0.3 achieves the best balance: increasing the ratio generally strengthens forgetting, but excessive values can reduce remaining- and test-set accuracy.The ablation varies the ratio parameter for random-subset unlearning on CIFAR-10.Source: §5.4 · p. 9, Table 5 · p. 9
• Applying SVD only to cross-attention layers achieves the best trade-off between forgetting and generation quality; applying it to all layers slightly degrades generation quality.The proposed explanation is that all-layer SVD perturbs low-level visual features and global denoising dynamics less directly related to semantic concepts.Source: §5.4 · p. 9, Table 6 · p. 9

Forgetting-retention trade-off and spectral structure

• Many unlearning methods use SVD-based projections to place updates orthogonal to an estimated retain subspace, addressing interference also studied as catastrophic forgetting in continual learning [32].Such projections follow parameter-isolation strategies [3] [14] [40]; related methods include [5] [8].Source: §6 · p. 9
• SSU suppresses weak singular components of the unlearning gradient, using its spectrum as a structured proxy for forget–retain interference rather than projecting updates onto an estimated retain subspace or relying on Euclidean gradient alignment.Unlike SVD-based low-rank parameterization, SSU uses singular values as a saliency signal; its mechanism is optimizer-agnostic.Source: §6 · pp. 9–10

Saliency-based MU

• SalUn [12] thresholds low-magnitude coordinates using forget-set gradients, while [10] argues for retained-data saliency; SSU instead thresholds singular values to select matrix-valued directions reflecting forgetting and retention signals.SSU's analysis offers a complementary theoretical explanation for how saliency-based masking can improve the forgetting–retention trade-off; [23] models saliency with an explicit weight-saliency matrix.Source: §6 · p. 10

General MU

• General MU includes gradient ascent [36], influence-function approximations [24] [28], Fisher-based selective updates [2] [18], and fine-tuning [18] [30] across image, diffusion, and language models.Many methods jointly optimize forgetting and retention or utility losses [12], a structure for which SSU can serve as a plug-in enhancement.Source: §6 · p. 10

7 Conclusion

• SSU suppresses weak spectral components of the unlearning gradient through singular-value thresholding and can enhance gradient-based unlearning pipelines as a drop-in method.The conclusion presents this as the paper’s central method contribution.Source: §7 · p. 10
• The theoretical justification links weak directions to forget-retain interference and argues that suppressing them can mitigate utility degradation.The paper also extends this perspective to coordinate-wise thresholding to explain the empirical effectiveness of SalUn-style masking.Source: §7 · p. 10
• Across image classification, diffusion-model unlearning, and large language model unlearning, SSU improves the forgetting-utility trade-off.The conclusion characterizes SSU as consistently effective across these settings.Source: §7 · p. 10

A Proof of the Theoretical Justification

• Section A provides the formal assumptions, theorem statements, and complete proofs for the theorems stated in the main paper.Source: Appendix · p. 14

Appendix

• Section B provides additional experiment details, and the paper says its code will be open-sourced upon acceptance.Source: Appendix · p. 14
• Section C provides additional image-generation results after unlearning with SSU.Source: Appendix · p. 14

A.1 Formal Statement and Proof of Proposition 4.2

• For the unlearning objective, the combined gradient is Gu = Gf + Gr and is represented by an SVD whose effective rank retains significant singular directions.The compact SVD uses only the re effective-rank components, treating approximately zero singular values as insignificant.Source: §A.1 · p. 14
• Under the stated smoothness assumptions, forgetting progress requires fi > 0, while retention progress requires ri > 0.Conversely, fi < 0 implies forgetting loss increases, and ri < 0 implies retaining loss increases.Source: §A.1 · pp. 14–15
• Along any singular direction with significantly conflicting retain and forget gradients, an update that improves either objective necessarily worsens the other.The proof establishes both directions: forgetting progress worsens retention, while retention progress worsens forgetting.Source: §A.1 · pp. 15–16

A.2 Formal Statement and Proof of Proposition 4.3

• The proof projects retain and forget Gram matrices onto E, the effective subspace spanned by the right singular vectors V_e.It defines P = V_eV_e^⊤ as the corresponding projection.Source: §A.2 · p. 16
• Assumption A.3 requires bounded spectral disparity, formalizing that projected retain and forget signals are bulk-distributed in E rather than highly heavy-tailed.The passage notes projected spectra within E are typically less heavy-tailed than spectra in the full parameter space.Source: §A.2 · p. 16
• If ∥G_f v_i∥ + ∥G_r v_i∥ ≥ ξ and σ_i ≤ ρξ, then a(v_i) ≤ 2ρ^2 − 1 < 0.Here ξ > 0 and ρ is the relative singular-value threshold.Source: §A.2 · p. 16
• Part (b) considers directions with opposite-sign alignment scores under Assumption A.3 and requires their conflict magnitudes to exceed a spectrum-dependent separation threshold.The stated condition is (δ_i + δ_j) > ϵ/(1−ϵ) · s^2+1/s.Source: §A.2 · p. 17

A.3 Theoretical Justification for Salun

• The proposition assumes that both the forgetting loss Lf and retaining loss Lr are smooth, with constants βf and βr, respectively.Source: §A.3 · p. 18
• Under Proposition 4.4’s per-coordinate signal and gradient conditions, the update satisfies ΔLfΔLr < 0.The conditions require |gf,i| + |gr,i| ≥ ξ and |gu,i| ≤ ρξ, with ξ > 0 and ρ ∈ (0, 1].Source: §A.3 · p. 18
• The proof establishes that the forgetting and retention signals have opposite signs, yielding opposite-signed changes in their losses under the update.It derives sign(ΔLf) = −sign(gf,i gu,i) and sign(ΔLr) = −sign(gr,i gu,i).Source: §A.3 · pp. 18–19

B.1 Details of unlearning on image classification.

• Image-classification unlearning forgets a seeded 10% sample and evaluates retain-, forget-, and test-set accuracy.The 5,000 forget samples are selected with random seed 42; the retain set has 45,000 samples and the test set has 10,000.Source: §B.1 · p. 19
• Image-classification updates use on-the-fly SVD for convolutional layers and precomputed saliency masks for fully connected layers.The model is fine-tuned on the retain set for 10 epochs at learning rate 0.013, using keep ratio γ = 0.30 selected by validation grid search.Source: §B.1 · p. 19

B.2 Details of unlearning on image generation.

• CIFAR-10 class-conditional unlearning uses a U-Net DDPM with 128 base channels and a 1,000-step linear noise schedule.The model has channel multipliers [1] [2], two residual blocks per resolution, and self-attention at resolution 16; βstart = 0.0001 and βend = 0.02.Source: §B.2 · p. 19
• The diffusion model is fine-tuned for 1,000 iterations with batch size 128 using gradient-ascent-based unlearning; convolutional-layer eigenvalue filtering retains γ = 0.50 of eigenvalues during training.On-the-fly SVD reconstruction is applied only to convolutional layers, while pre-computed saliency coordinate masks are used for non-convolutional layers. Adam uses learning rate 1 × 10^-5, αremain = 1 × 10^-3, and αforget = 1.0; evaluation generates 5,000 images for each class to compute FID and FA.Source: §B.2 · p. 19
• Imagenette unlearning uses Stable Diffusion with a latent-space U-Net conditioned on CLIP text.The U-Net has 320 base channels, channel multipliers [1] [2] [4], two residual blocks per resolution, eight attention heads, transformer depth 1, and context dimension 768; the 1,000-step linear schedule uses βstart = 0.00085 and βend = 0.012.Source: §B.2 · p. 19
• During unlearning, the model is fine-tuned for 5 epochs with batch size 8 using gradient-ascent-based unlearning loss and randomly assigned labels for the forget class.On-the-fly SVD reconstruction is applied only to cross-attention layers, while precomputed saliency coordinate masks are used for other layers. The keep ratio γ = 0.50 is chosen by grid search for best validation performance; Adam uses learning rate 1 × 10^-5 and classifier-free guidance scale 7.5, and evaluation generates 300 images per class to compute FID and FA.Source: §B.2 · pp. 19–20

B.3 Details of unlearning on large language models.

• LLM unlearning uses IDK+AP and ME+GD objectives with five epochs of training and an effective batch size of 32.The batch size is 8 with four gradient-accumulation steps; SVD reconstruction filters attention projections (Q, K, V) using αqkv = 0.20, while other layers use coordinate masks with ratio 0.10.Source: §B.3 · p. 20
• LLM experiments use AdamW with weight decay 0.01 and distributed training across two GPUs.IDK+AP uses learning rate 3 × 10^-5, forget coefficient 1.0, and regularization coefficient 1.0; ME+GD uses 2 × 10^-5, 0.1, and 1.0, respectively. Both use top eigenvalue ratio 0.2, chosen by validation grid search.Source: §B.3 · p. 20

• Figures 2–4 present SSU’s class-wise unlearning results on Imagenette under SalUn across random seeds, with rows marking unlearned classes and columns marking prompt classes.Source: C Additional generation results · p. 20, Figure 2 · p. 21, Figure 3 · p. 22
• Diagonal images correspond to the class being unlearned, illustrating SSU’s removal of the targeted concept.Source: C Additional generation results · p. 20, Figure 2 · p. 21, Figure 3 · p. 22
• Off-diagonal images show generations for other classes, demonstrating the model’s ability to generalize and distinguish among remaining categories.Source: C Additional generation results · p. 20, Figure 2 · p. 21, Figure 3 · p. 22