SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation

• Machine unlearning mitigates the influence of particular data points on a pretrained model.Source: §1 · p. 1
• Exact or certified unlearning offers guarantees, whereas approximate unlearning prioritizes practical speed and accuracy without requiring certified-unlearning assumptions.Source: §1 · pp. 1–2
• Approximate unlearning can exhibit substantial performance variance, including sensitivity to hyperparameter selection.Source: §1 · p. 2
• Existing MU methods developed for image classification are insufficient for image-generation unlearning.Source: §1 · p. 2
• SalUn introduces weight saliency to reduce the gap with exact unlearning and support MU in image classification and generation.Source: §1 · p. 2

Unlearning in image classification

• Exact unlearning retrains models after removing data, but its computational demands motivate probabilistic and approximate alternatives for image classification.Source: §2 · p. 2

Unlearning in image generation

• Diffusion-model studies address concept erasure, but achieving precision comparable to exact unlearning remains challenging.Source: §2 · p. 3

Data and model saliency analyses

• Prior saliency research emphasizes input sensitivity and data attribution, while model saliency and weight-focused analysis remain less explored.Source: §2 · p. 3

MU objective and setup

• The forgetting dataset contains points designated for removal, while the remaining dataset is its complement.Source: §3 · p. 3
• Retrain is the gold standard, but MU seeks an accurate and computationally efficient substitute obtained from the original model.Source: §3 · p. 3

MU for image classification

• Classification MU includes class-wise forgetting and random data forgetting.Source: §3 · p. 3
• Full-stack evaluation considers unlearning accuracy, membership inference, remaining accuracy, testing accuracy, and runtime efficiency.Source: §3 · pp. 3–4

MU for image generation

• Conditional diffusion models estimate noise from noisy data or latent features during reverse diffusion, and training commonly uses mean-squared-error loss.Source: §3 · p. 4
• Image-generation MU must prevent undesired content while maintaining normal-image generation quality.Source: §3 · p. 4

Challenges

• Current MU methods face limitations in unlearning stability and generality.Source: §4 · p. 4

The instability limitation

• Performance gaps relative to Retrain can change when the forgetting-data quantity increases from 10% to 50%.Source: §4 · pp. 4–5, Figure 2 · p. 4
• Influence unlearning shows high variance across hyperparameter choices, whereas SalUn integration reduces this instability.Source: Figure 2 · p. 4, §4 · p. 5

The generality limitation

• Existing classification MU methods do not remain effective when adapted to image-generation unlearning.Source: §4 · p. 5

Weight saliency

• Weight saliency uses gradients of a forgetting loss to identify model weights for targeted unlearning.Hard thresholding converts the gradient magnitudes into a saliency map.Source: §5 · pp. 5–6

Weight update

• The unlearned model updates salient weights while leaving intact weights at their original values.The model is expressed as a masked combination of updated and original weights.Source: §5 · p. 6

Image classification

• SalUn combines the saliency-based weight update with random labeling for image classification.The random-labeling loss updates salient weights while a regularization term on retained data preserves generalization.Source: §5 · p. 6

Method integration

• SalUn is designed as a plug-and-play method that can be applied on top of existing unlearning methods.The paper specifically reports integrating weight saliency with random labeling as a promising solution.Source: §5 · p. 6

Image generation

• For image generation, SalUn misaligns forgetting concepts with alternative images and adds diffusion-training loss on retained data.The regularization balances concept unlearning against preserving image-generation quality.Source: §5 · p. 6

Implementation and evaluation

• The default implementation uses hard thresholding, although the paper also considers a soft-thresholding variant.The supplied table caption describes evaluation across two random-forgetting scenarios on CIFAR-10 with ResNet-18.Source: §5 · p. 6, Table 1 · p. 7

Experimental setups

• Experiments cover random-data and class-wise forgetting in image classification, plus class-wise and concept-wise forgetting in diffusion-based image generation.Classification uses CIFAR-10, CIFAR-100, SVHN, Tiny ImageNet, ResNet-18, VGG-16, and Swin-T; generation uses DDPM and stable diffusion.Source: §6.1 · p. 7

Image classification

• SalUn achieves the smallest average performance gap with Retrain in both random-data forgetting scenarios.SalUn-soft has the second or third-smallest gap, while hard thresholding is hypothesized to improve unlearning efficacy through strict sparsity.Source: §6.2 · p. 8
• SalUn offers the best trade-off between unlearning efficacy, preserved model fidelity, and computational efficiency across the evaluated classification metrics.The comparison uses UA and MIA for unlearning, RA and TA for retained-model fidelity, and RTE for efficiency.Source: §6.2 · p. 8
• 50% forgetting increases the Avg. Gap for BS from 2.67 to 8.13 and for ℓ1-sparse from 2.26 to 5.72, while SalUn stays consistently effective.The results indicate that increasing the forgotten-data proportion makes unlearning more challenging for all methods.Source: §6.2 · p. 8

Image generation

• Proper weight saliency outperforms random masking for DDPM generation by avoiding noisy over-forgetting and degradation on non-forgetting classes.Random masking produces noisy airplane-condition outputs and lower-quality outputs for non-forgetting examples such as C2, C3, C6, and C7.Source: §6.2 · p. 8
• On Imagenette, SalUn outperforms ESD and FMN in UA across forgetting classes while maintaining good FID, whereas FMN has lower FID but weaker forgetting.For NSFW concept forgetting, SalUn generates the fewest harmful images across all evaluated nude body-part classes and outperforms ESD especially for male and female breast categories.Source: §6.2 · p. 9

• SalUn applies weight saliency to focus unlearning on selected model weights rather than the entire model.The framework is presented for both image classification and generation tasks.Source: §7 · p. 9
• The classification and generation algorithms compute a saliency mask from gradients and update parameters using masked gradients.The pseudocode uses relabeled forgetting data, a thresholded saliency mask, and iterative SGD updates.Source: A PSEUDO CODE OF RL-BASED SALUN · p. 15
• An ℓ1 penalty and proximal-gradient steps constrain changes between the unlearned model θu and the original model θo.The proximal update implements soft thresholding around the original weights.Source: B SOFT-THRESHOLDING SALUN · pp. 15–16
• Higher weight-saliency sparsity corresponds to fewer changed model weights and can be enforced through the regularization parameter β.Increasing β drives θ′ − θo toward zero.Source: B SOFT-THRESHOLDING SALUN · pp. 15–16
• The soft-thresholding solution uses the positive-part operator to project parameter differences and then maps back to the original variable θ.The operation is applied elementwise to the displacement from θo.Source: B SOFT-THRESHOLDING SALUN · p. 16

Additional training and unlearning settings

• Classification baselines use distinct training schedules, including 182 epochs for Retrain, 10 epochs for FT and RL, and 5 epochs for GA.The settings also search IU’s Hessian approximation parameter α over [1, 20].Source: §C.1 · p. 16
• SalUn uses 1,000 DDPM iterations and 5 Stable Diffusion epochs, with 50% weight-saliency sparsity in both settings.The DDPM and Stable Diffusion configurations use different optimizers, batch sizes, sampling procedures, and β values.Source: §C.1 · p. 17
• NSFW concept forgetting constructs forgetting and retaining image sets from nude and clothed prompts, then corrects the forgotten concept using the clothed concept.The nudity prompt also supplies the saliency mask during unlearning.Source: §C.1 · p. 17
• In Fig. A1, higher saliency sparsity can produce under-forgetting, while lower sparsity can produce over-forgetting or increased FID.Points above Retrain indicate over-forgetting and points below Retrain indicate under-forgetting.Source: §C.1 · pp. 17–18, Figure A1 · p. 17
• Forgetting the airplane class at 10% sparsity reduces image-generation quality, whereas 90% sparsity fails to forget the class.Fig. A3 relates the optimal sparsity level to the amount of forgetting data.Source: Figure A2 · p. 17, §C.1 · p. 18

Additional datasets and architectures

• Fig. A3 examines how the optimal saliency sparsity changes with the amount of forgotten data under random forgetting on CIFAR-10.Source: Figure A3 · p. 18

Saliency-enhanced baselines

• Adding the weight-saliency mask improves FT, RL, GA, and IU baselines toward the Retrain benchmark, with saliency-augmented RL performing best among them.The comparison is reported using Avg. Gap across forgetting scenarios.Source: §C.2 · p. 18, Table A1 · p. 18

Class-wise and iterative forgetting

• SalUn and SalUn-soft maintain a robust balance across UA, MIA, RA, and TA for class-wise forgetting on CIFAR-10.The methods do not always achieve the highest UA, but their overall metric balance is emphasized.Source: §C.2 · pp. 18–19, Table A2 · p. 18
• SalUn achieves the smallest Avg. Gap to Retrain during five iterative 10%-data forgetting steps totaling 50% of the dataset.The comparison includes FT and Retrain on ResNet-18 trained on CIFAR-10.Source: §C.2 · p. 19, Table A3 · p. 19

Additional datasets and architectures

• Across SVHN, CIFAR-100, VGG-16, and Swin-T settings, SalUn and SalUn-soft deliver promising or balanced performance across diverse forgetting amounts.The reported evaluations emphasize adaptability rather than universal dominance on every metric.Source: §C.2 · p. 19
• On Tiny ImageNet, SalUn has smaller UA and MIA gaps than ℓ1-sparse while maintaining comparable RA and TA gaps.The comparison uses ResNet-18 with 10% random data forgetting.Source: §C.2 · p. 20, Table A4 · p. 20

CIFAR-10 generation

• On CIFAR-10 DDPM class-wise forgetting, SalUn has a 0.84% UA drop relative to ESD but significantly outperforms ESD in FID.The authors attribute ESD’s 100% UA partly to poor generation quality and report SalUn as closest to Retrain.Source: §C.3 · p. 20, Table A5 · p. 20
• SalUn’s CIFAR-10 generation results are illustrated with class-wise examples, marking the forgotten class with a red border.Source: §C.3 · p. 20

ImageNette generation

• ImageNette figures contrast the unlearned class with the prompt class in a matrix, with diagonal images corresponding to the target unlearning class.Off-diagonal images represent other classes and broader differentiation behavior.Source: §C.3 · p. 20

NSFW prompt evaluation

• Table A11 supplies the harmful I2P text prompts used for Stable Diffusion generation experiments.Source: §C.3 · p. 20

Broader impacts

• SalUn is described as enhancing unlearning effectiveness while maintaining model utility under strict unlearning requirements.Source: D BROADER IMPACTS AND LIMITATIONS · p. 20

Limitations

• The method’s demonstrated scope is vision, while scalability and adaptability to language and graph domains remain unresolved.Source: D BROADER IMPACTS AND LIMITATIONS · p. 20
• The paper states that unlearning’s effects on fairness, privacy, and security require further consideration alongside transparent, accountable, and inclusive development.Source: D BROADER IMPACTS AND LIMITATIONS · p. 20

Additional evaluations

• Supplementary evaluations report machine-unlearning performance across varying forgetting-data amounts on CIFAR-10 ResNet-18, SVHN, CIFAR-100, VGG-16, and Swin-T.Source: Table A6 · p. 21, Table A7 · p. 22, Table A8 · p. 23
• Supplementary figures show class-wise CIFAR-10 DDPM results and SalUn-generated images, distinguishing forgetting classes from remaining classes.Source: Figure A4 · p. 26, Figure A5 · p. 27, Figure A6 · p. 28