Source-linked AI summary
Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening
Jack Foster, Stefan Schoepf, Alexandra Brintrup
TL;DR
Machine unlearning must remove specified information while preserving retained-data performance, but existing methods face retraining, computational, and storage burdens. SSD uses Fisher information to selectively dampen forget-specialized parameters without retraining, and experiments find it competitive with retrain-based methods while substantially faster than Fisher Forgetting. The paper concludes that retrain-free post hoc unlearning is viable, although SSD lacks certification and repeated forgetting can eventually degrade retained-data performance.
Problem
Machine unlearning must forget specified data while preserving retained-data performance, but existing approaches can require costly retraining and persistent training-data storage.
Method
SSD selects parameters disproportionately important to the forget set using Fisher information, then dampens them according to their relative forget-versus-retain importance.
Results
SSD is competitive with retrain-based methods and orders of magnitude faster than comparable retrain-free Fisher Forgetting approaches across multiple unlearning settings.
Takeaways & Limitations
SSD demonstrates the viability of fast, selective, retrain-free post hoc unlearning across single-class, subclass, and random-sample settings.
Takeaways & Limitations
SSD has no mathematical unlearning certificate, and without repair only a finite number of forget requests can be processed before retained-data performance degrades.
Abstract
from arXiv · showhide
Machine unlearning, the ability for a machine learning model to forget, is becoming increasingly important to comply with data privacy regulations, as well as to remove harmful, manipulated, or outdated information. The key challenge lies in forgetting specific information while protecting model performance on the remaining data. While current state-of-the-art methods perform well, they typically require some level of retraining over the retained data, in order to protect or restore model performance. This adds computational overhead and mandates that the training data remain available and accessible, which may not be feasible. In contrast, other methods employ a retrain-free paradigm, however, these approaches are prohibitively computationally expensive and do not perform on par with their retrain-based counterparts. We present Selective Synaptic Dampening (SSD), a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data. First, SSD uses the Fisher information matrix of the training and forgetting data to select parameters that are disproportionately important to the forget set. Second, SSD induces forgetting by dampening these parameters proportional to their relative importance to the forget set with respect to the wider training data. We evaluate our method against several existing unlearning methods in a range of experiments using ResNet18 and Vision Transformer. Results show that the performance of SSD is competitive with retrain-based post hoc methods, demonstrating the viability of retrain-free post hoc unlearning approaches.
Introduction
Machine unlearning must forget specified data while preserving performance on retained data, but existing approaches often require costly retraining and persistent training-data storage. SSD addresses this with a retraining-free, post hoc method that selectively dampens forget-specialized parameters.
- Machine unlearning seeks to forget specified information without degrading performance on remaining data.
- Retraining-based methods preserve performance but add overhead and require permanent storage of training data.
- SSD uses Fisher information to identify parameters specialized toward the forget set and dampens them according to their specialization.
- SSD is designed as a retraining-free, post hoc approach for lightweight and timely unlearning.
- Experiments cover single-class, subclass, and random-observation forgetting, with SSD reported as faster than previous retrain-free methods and comparable to retrain-based methods.
- SSD modifies only selected parameters and requires access to training data once to compute the Fisher information matrix before discarding it.
Related Work
Prior work divides post hoc deep-network unlearning into retraining-based and retraining-free methods, with retraining-based approaches generally leading in performance. SSD is positioned as a fast, selective alternative that avoids additional models.
- Differential privacy and machine unlearning both address whether information about individuals remains exposed through model outputs.
- Membership inference attacks compare model outputs on training and unseen data, making MIA a key unlearning-performance measure.
- Post hoc deep-network methods are categorized as retraining-based or retraining-free according to whether traditional training steps are used during unlearning.
- Retraining-free methods commonly use the Fisher information matrix to approximate parameter sensitivity and importance.
- Some alternative methods rely on additional models, whereas SSD does not.
- Retraining-based methods are described as the current state of the art in performance, including student-teacher, relabeling, and noise-based approaches.
- SSD combines post hoc operation, speed, retraining freedom, parameter selectivity, and independence from additional models.
Preliminaries
The paper defines unlearning over a forget subset and a retain set of a labeled training dataset, with SSD evaluated across several forgetting scenarios.
- The training dataset D contains labeled samples, and unlearning targets a forget subset D_f while preserving the retain set D_r = D \ D_f.
- The forget set may represent full-class, subclass, or randomly sampled observations.
- The model is a parameterized function mapping inputs to class-probability outputs.
Proposed Method
SSD identifies parameters specialized to the forget set and modifies them selectively to balance forgetting with retained-model performance. It replaces binary pruning with proportional dampening and uses Fisher information to guide these updates.
- Parameter selection: SSD uses Fisher information to identify parameters that are disproportionately important to forget-set samples but less important to retained data.This targets specialized information while protecting generalized information useful across data.
- Parameter selection: A naïve FIM-based approach sets every parameter with non-zero forget-set importance to zero, which can severely degrade retained-set performance.The degradation follows from overlap between parameters important to the forget and retain sets.
- Parameter selection: SSD applies a stricter criterion, selecting parameters only when forget-set importance exceeds α times retain-set importance.The hyper-parameter α controls how specialized a parameter must be before selection.
- Selective dampening: Binary pruning creates a coarse forgetting-performance trade-off because parameters just above and far above the threshold receive the same update.Large α protects retained performance but can reduce forgetting by making selection too restrictive.
- Selective dampening: SSD replaces pruning with dampening proportional to a parameter’s relative forget-set importance, enabling granular updates that protect generalized parameters.With λ = 1, the dampening factor approaches zero as specialization toward the forget set increases, while remaining bounded above by 1.
- Approximation and evaluation: The method uses D rather than Dr for experiments because |Df| << |D| makes their importance values near-identical, though repeated requests may eventually cause divergence.If divergence occurs, dampening can falsely protect purged parameters because it only reduces parameter contributions.
Experimental Setup
The experiments evaluate SSD across image-classification datasets, model architectures, and three unlearning scenarios, comparing it with retraining-based and other unlearning baselines. Runtime is measured in a realistic pipeline that includes loading and verification.
- Datasets and models: Experiments use CIFAR10, CIFAR20, CIFAR100, and PinsFaceRecognition for image-classification unlearning.PinsFaceRecognition substitutes for the inaccessible VGG-Face dataset and contains 17,534 faces of 105 celebrities.
- Datasets and models: The study evaluates ResNet18 and Vision Transformer models trained with early stopping, a multi-step scheduler beginning at lr = 0.1, and Adam.Experiments ran on NVIDIA RTX4090 systems with Intel Xeon processors.
- Unlearning tasks: Benchmarks cover single-class, subclass, and random-observation forgetting scenarios across CIFAR20, CIFAR100, and PinsFaceRecognition.The subclass setting includes forgetting rocket within the vehicles superclass.
- Baselines: Comparisons include Fisher Forgetting, Baseline, Finetune, Retraining, and other state-of-the-art methods, while some compute-intensive methods are excluded from all experiments.Fisher Forgetting was reported as 50–250 times slower than SSD in initial results.
- Runtime measurement: Figure 2 measures end-to-end time from model loading through forgetting verification, and precomputing []D saves 84.32%±0.21% of pure SSD computation time excluding baseline time.The zoom-in separates []D calculation from the rest of SSD.
- Configuration: SSD hyper-parameters were selected with 50 Optuna TPE runs over α ∈ [0.1, 100] and λ ∈ [0.1, 5], then fixed by task and model.For example, ResNet18 CIFAR tasks use λ = 1 and α = 10, with task-specific changes for Vision Transformer and face recognition.
Results and Discussion
SSD is competitive with retraining-based unlearning while requiring substantially less computation and modifying only a small fraction of parameters. Across class, subclass, and random-sample tasks, its results generally track retraining, though evaluation standards and repeated forgetting impose important qualifications.
- Selectivity: SSD changes only 1.7% of parameters when forgetting the CIFAR100 rocket class.
- Efficiency: 0.4–3.8% of Fisher’s runtime is required by SSD, while including FIM computation makes SSD the second-fastest method overall.
- Class unlearning: For CIFAR100 rocket forgetting, SSD reduces baseline MIA from 93% for ResNet and 94% for ViT to approximately 2%, with retained-data accuracy dropping 2% for ResNet and negligibly improving for ViT.
- Subclass unlearning: Subclass experiments show SSD produces MIA and forget-set accuracy closer to retraining than Amnesiac and Bad Teacher, which can reduce both metrics below retrained-model behavior.
- Random-sample unlearning: Random-sample experiments show similar SSD and retraining performance on ResNet and ViT, whereas some competing methods reduce MIA far below the retrained model.
- Overall assessment: SSD outperformed Fisher, was competitive with state-of-the-art methods and full retraining, and was on average strongest in similarity to the fully retrained model.
- Overall assessment: Unlearning comparisons remain difficult because evaluations are not standardized and the desirable MIA score is not yet agreed upon.
- Limitations: SSD has no mathematical unlearning guarantee, and unsuitable α and λ choices can cause degradation during repeated forgetting.
Conclusion
SSD is a fast, retraining-free unlearning method evaluated across multiple tasks, datasets, and architectures. Its viability is supported by strong speed and performance comparisons, while larger forgetting fractions remain an open direction.
- Conclusion: SSD is a two-step retraining-free method evaluated on single-class, subclass, and random-sample unlearning across multiple datasets and architectures.It selects parameters disproportionately important to the forget set, then dampens them according to their relative importance.
- Conclusion: SSD is orders of magnitude faster than Fisher Forgetting and rivals state-of-the-art retrain-based approaches in speed and performance.The paper describes these results as demonstrating the viability of retrain-free post hoc unlearning.
- Conclusion: Evaluating how to forget larger subsets without retraining remains an important future direction because experiments typically forget no more than 5–10% of the data.The paper also identifies random-subset evaluation and overlapping parameter importance as future research areas.
Appendix
SSD reuses gradients and computes forget-set information for each request, giving it a computational structure dominated by the retained or full dataset. The appendix tables define the reported datasets, hyperparameters, metrics, and model settings.
- Appendix: A backward pass over the full dataset costs O(bfull·p), while forget-set and retain-set passes cost O(bforget·p) and O(bretain·p), respectively.Here, bretain ≈ bfull >> bforget, with b denoting batches and p the number of model parameters.
- Appendix: SSD computes gradients for D once, reuses them, and then processes each forget request with one Df computation followed by O(bforget·p) dampening.This supports repeated requests without recomputing the full-dataset gradients.
- Appendix: Table 7 reports Cifar20 subclass unlearning with ResNet18 and Vision Transformer, using Dr, Df, MIA, and time t as evaluation measures.The caption also specifies α and λ values for each architecture.
- Appendix: Table 8 reports Cifar100 class unlearning and includes a result for changing α from 10 to 20: Dr = 72.98, Df = 0, and MIA = 0.20.The table uses ResNet18 and Vision Transformer with architecture-specific α and λ settings.
- Appendix: Table 9 reports Cifar20 class unlearning with architecture-specific α and λ settings, using Dr, Df, MIA, and time t as metrics.The caption identifies several dataset categories, including electrical devices, natural scenes, vehicle2, and vegetables.