Corrective Machine Unlearning

• Corrective Machine Unlearning targets removing manipulation effects from a trained model using a small identified subset, despite unknown manipulation type and extent.The authors say this requires methods to work across different manipulation types.Source: §1 · p. 2
• Privacy-oriented unlearning targets deleting sensitive user data, whereas corrective unlearning targets correcting effects of corrupted training data with only a small subset identified.The paper argues that the two settings have different requirements.Source: §1 · p. 2
• The benchmark covers feature-and-label BadNet poisoning and label-only Interclass Confusion, which entangles model representations.Source: §1 · pp. 2–3
• No benchmarked method succeeded across both attacks; RewoD needed over 80% of manipulated samples identified, while SSD mitigated BadNet with 10% but failed on Interclass Confusion.SSD also caused a significant drop in overall test accuracy when unlearning poisoned data.Source: §1 · p. 3

2.1 Motivation

• The threat model includes adversaries that alter features, labels, or both to inject harmful properties, including backdoors and systematic mislabeling.Source: §2.1 · p. 3
• Developers may identify compromised sources after training, but the manipulation type is unknown; the setting assumes a representative identified subset and excludes user feedback or red-team signals.Source: §2.1 · p. 3

2.2 Objective of Corrective Unlearning

• Corrective unlearning takes the trained model, training data, and identified subset as inputs, and seeks to improve affected-domain accuracy while maintaining accuracy outside that domain.Source: §2.2 · p. 4
• As the identified fraction α grows, corrected accuracy should rise; with the whole manipulated set known, retraining without it is the reference outcome.The section compares methods by how much of the manipulated set must be identified to approach that outcome.Source: §2.2 · p. 4

2.3 Differences from Privacy-Oriented Unlearning

• Privacy-oriented unlearning removes influence of untampered private data, while corrective unlearning targets manipulated samples and expects affected-domain accuracy to improve.The authors note that corrective unlearning may also improve retained accuracy through better representations.Source: §2.3 · p. 4
• At α < 1, unflagged manipulated samples remain in the retained set, so retraining on it can perpetuate adverse effects rather than serve as corrective unlearning’s gold standard.Privacy-oriented unlearning treats retraining on the retained set as its gold standard, despite its computational cost.Source: §2.3 · p. 5
• Whether identified examples can reveal other manipulated data depends on subset size, data dimensionality, and detection complexity; small subsets and hard-to-detect high-dimensional manipulations may make this impossible.The authors cite a poisoning setting with 500 identified samples in 3,072-dimensional data.Source: §2.3 · p. 5
• Corrective unlearning does not require privatizing identified or manipulated samples, and privacy-focused evaluations are orthogonal to its goals.The authors therefore call for new evaluation strategies.Source: §2.3 · p. 5

3.1 Manipulation Types and Evaluation Benchmarks

• Poisoning modifies features and labels with a trigger, whereas Interclass Confusion flips labels between two classes, confusing predictions across both affected classes.Source: §3.1 · pp. 5–6

3.2 Experimental details

• Experiments use CIFAR10 and CIFAR100 plus PCam, vary identified fraction α from 0.1 to 1.0, and report unseen-test means and standard deviations over three seeds.Source: §3.2 · p. 6
• The benchmark includes RewoD, CF, SSD, BadT, and Scrub, spanning retraining, fine-tuning, parameter dampening, and distillation-based approaches.Source: §3.2 · p. 6
• Hyperparameters are selected by equally weighting validation accuracy and the fraction of identified samples whose classifications change, since corrective accuracy may be unavailable.Source: §3.2 · pp. 6–7

3.3 Experimental Results

• At α=0.1, SSD recovers poisoning Acccorr but fails on IC and reduces utility; no method handles both manipulations well when ≤80% is identified.RewoD shows no poisoning improvement below 80% identification, despite being the traditional gold standard when all manipulated samples are known.Source: Figure 2 · p. 7, §3.3 · pp. 7–8

3.4 Why does SSD fail for IC test but work for poisoning?

• Poisoning's importance-ratio outliers (e.g., 800) contrast with IC's maximum below 200; SSD can target weights linking poison features to wrong labels.Source: Figure 3 · p. 8, §3.4 · p. 8

3.5 Application on a Medical Dataset: PCam - Binary Classification

• On binary PCam, SSD works at small α, while Scrub improves because moving away from manipulated labels infers the only other class.Source: §3.5 · p. 8

• Robust training can reduce sensitivity to manipulated data, but that may harm accuracy and fairness; unlearning discovered samples is presented as a complementary post-training mitigation.Source: §4 · p. 9
• Prior detection work addresses mislabeled, biased, and poisoned data, but the paper assumes developers monitor data sources and cannot simply discard costly trained models.Source: §4 · p. 9
• Known manipulations may permit specialized defenses or re-annotation, whereas this work restricts its scope to settings where the precise manipulation is unknown.Source: §4 · p. 9
• Privacy-oriented unlearning targets retrain indistinguishability, but exact retraining-based methods may fail when the full manipulation set is unknown and become less efficient as deletion grows.Source: §4 · p. 9
• The paper unifies poisoning and mislabeling under corrective unlearning, adds the constraint of partial identification, and argues retraining is not a sufficient gold standard for that setting.Source: §4 · p. 9

Limitations and Future work

• The study excludes more complex manipulations after state-of-the-art methods, including RewoD, failed on its two simple manipulations.Future work should test adaptive attacks, synthetic samples, self-supervised contexts, and false positives; corrective unlearning may not be tractable in all these settings.Source: §5 · pp. 9–10

Broader Impact

• Corrective unlearning is a post-training strategy to mitigate manipulated or incorrect training data, with societal benefits likened to defenses against backdoors and systematic biases.The authors also note that defenses could have negative societal effects in scenarios such as collective action, but call these possibilities speculative.Source: §5 · p. 10

Conclusion

• The paper characterizes corrective machine unlearning as mitigating negative effects of manipulated data discovered after training, even when developers know only a small representative subset.The subset could be collected through careful inspection of a uniform training-data subsample.Source: §5 · p. 10
• Popular unlearning methods, including RewoD, generally need nearly all manipulated samples identified; SSD mitigates BadNet effects with a partial subset but does not generalize to Interclass Confusion.The authors present this as evidence of feasibility in one setting, not across settings.Source: §5 · p. 10

A.1 Training Details

• Models trained for 4,000 steps on CIFAR10 and PCAM and 6,000 steps on CIFAR100, using batch size 512 throughout.Each step uses one batch.Source: §A.1 · p. 16
• Training used SGD with momentum 0.9, weight decay 5e-4, a linear scheduler with tmult = 1.25, and warmup steps set to 1/100 of total training steps.The same hyperparameters apply during unlearning unless otherwise specified.Source: §A.1 · p. 16
• Experiments ran on a PC with an Intel Xeon E5-2640 2.40 GHz CPU, 128GB RAM, and one GeForce RTX 2080 GPU.Source: §A.1 · p. 16

(1) Retrain without Deletion set (RewoD)

• RewoD retrains the entire model from scratch on the training data excluding the identified forget set, using the original training algorithm.The paper treats this as an inefficient gold standard in prior work.Source: §A.2 · p. 16

(2) Catastrophic Forgetting (CF)

• Catastrophic forgetting is induced by finetuning with the original training procedure on the remaining data for 1,000 steps.The method is intended to unlearn the forget set by losing knowledge when updating without those samples.Source: §A.2 · p. 16

(3) Modifying learnt parameters with high influence from Sf

• Influence-based methods identify parameters relevant to the forget set using statistics such as the Fisher Information Matrix, then damage them with noise or reduced magnitudes.The benchmark uses SSD and tunes its selection threshold α and dampening constant γ.Source: §A.2 · p. 16

(4) Pushing Sf outputs towards random

• Bad Teacher distills from a randomly initialized network on the forget set and the original model on the remaining data, with 1,000 finetuning steps.This represents procedures that push forget-set outputs toward random predictions.Source: §A.2 · p. 16

(5) Alternating between Forgetting and Preservation Steps

• SCRUB alternates gradient-ascent forget steps with preservation steps using distillation and task loss on remaining data.The benchmark runs 1,000 unlearning steps, using forget steps only in the first 200; it lowers the learning rate to 0.0025 and tunes α across eight values.Source: §A.2 · pp. 16–17

A.2 Detailed Description of Unlearning Methods

• Table 2 identifies the datasets, models, and manipulation sizes used for the Poisoning and Interclass Confusion evaluations.Source: Table 2 · p. 17

B Further Results

• Results span different manipulation-set sizes and assess both unseen affected-domain samples and training manipulation samples for poisoning and Interclass Confusion.The former measures generalization of manipulation effects; the latter measures memorization.Source: B Further Results · p. 17
• The appendix reports computational efficiency by measuring each method's unlearning time.Source: B Further Results · p. 17
• The appendix includes a hyperparameter sensitivity analysis for all methods.Source: B Further Results · p. 17

Unlearning Poisoning

• SSD classifies manipulated poisoning-training samples correctly after unlearning, with trends similar to unseen affected-domain samples but higher absolute accuracy.The appendix compares manipulation sizes of 0.2% and 2% with the main paper's 1% setting.Source: §B.1 · p. 17

Unlearning Interclass Confusion

• For interclass confusion, smaller manipulation settings show similar but weaker fidelity, while the smallest settings may leave unseen samples unaffected even as training samples remain misclassified.This motivates checking unseen and manipulated training samples separately, particularly when manipulations are too small to affect unseen-sample behavior.Source: §B.1 · p. 17

Computational Efficiency

• RewoD and CF are retrained from scratch for the strongest unlearning, which the authors still find insufficient; Table 3 reports average unlearning time by method.The supplied passage gives no time values.Source: §B.2 · p. 17, Table 3 · p. 20

Poisoning

• For poisoning, methods other than SSD, including RewoD, perform poorly when at most 80% of poisoned data is identified, even with 1% poisoning.The figure panels cover clean-label accuracy on test samples and manipulated training samples with a poison trigger.Source: §B.2 · p. 18

Interclass Confusion

• For interclass confusion, SSD does not improve on the original model for unseen samples, and other methods require identifying a large fraction of manipulated data.At lower manipulation sizes, unseen-sample outputs are not much affected, so the results also examine manipulated training samples.Source: §B.2 · p. 19

B.3 Hyperparameter Sensitivity

• The sensitivity analysis plots Acccorr across α values from 0.1 to 1 and compares Acccorr with Accretain at α = 0.5.CF, RewoD, and BadT have no crucial hyperparameters beyond those used in original training.Source: §B.3 · p. 20

B.3.1 Unlearning across deletion fraction

• Scrub's unlearning improves gradually as α increases across hyperparameter values on IC, but good poisoning unlearning appears only at high α for specific settings.The poisoning result indicates high hyperparameter sensitivity.Source: §B.3.1 · p. 20, Figure 8 · p. 20
• SSD never outperforms the original model's Acccorr on IC, though specific hyperparameters produce good poisoning unlearning across α values.The poisoning performance is highly sensitive to hyperparameter choice.Source: §B.3.1 · p. 20, Figure 9 · p. 21, Figure 10 · p. 21

B.3.2 Unlearning-Utility Tradeoff

• At α = 0.5, Scrub's tuned hyperparameters can achieve unlearning without sacrificing utility, while SSD is sensitive and only a few settings combine highest utility with high unlearning.The tradeoff plots compare Acccorr with Accretain.Source: §B.3.2 · p. 21, Figure 11 · p. 21, Figure 12 · p. 21