PaperlayerView exact revision on arXiv

Source-linked AI summary

Forget or Fine-tune? A Comparative Study of Machine Unlearning Strategies for Noisy Label Correction

João L. P. Santana, Filipe R. Cordeiro

arXiv:2608.30046v1cs.LGcs.AI

TL;DR

Noisy-label correction after training is expensive because the standard remedy retrains on a cleaned dataset, while the relative value of machine-unlearning strategies across noise types is unclear. This paper compares five strategies across synthetic and real-world datasets and finds that the best choice depends on noise structure, with substantial runtime savings in supported settings.

  • Problem

    The relative effectiveness of different machine-unlearning strategies for post-training noisy-label correction across noise structures remains unclear, despite retraining being computationally expensive.

  • Method

    The study compares NegGrad, FT, RL, SalUn, and MUNBa across four synthetic noise types on CIFAR-10 and CIFAR-100 and real-world noise on Food-101N.

  • Results

    The preferred method depends on noise structure: FT is strong across most closed-set settings, RL and SalUn are most consistent, and MUNBa helps mainly under extreme symmetric noise; open-set retraining can reduce accuracy.

  • Takeaways & Limitations

    Machine unlearning provides practical post-training correction alternatives within the evaluated scope, but strategy selection and the decision to unlearn should account for noise structure.

  • Takeaways & Limitations

    The evaluation does not measure forgetting-specific influence removal and covers only limited repetitions and architectures, so conclusions concern utility recovery within the evaluated scope.

Abstract

from arXiv · show

Noisy labels remain a critical challenge for training deep neural networks, since memorizing incorrect labels degrades generalization. Once noisy samples are identified after training, the standard solution is to retrain the model from scratch on the cleaned dataset, which is increasingly expensive as datasets and models grow. Machine Unlearning (MU) has recently emerged as a computationally efficient alternative, but the relative effectiveness of different MU strategies for noisy-label correction remains poorly understood. In this work, we conduct a comparative empirical study of five MU methods (NegGrad, Fine-Tuning (FT), Random Labeling (RL), SalUn, and MUNBa) across symmetric, asymmetric, instance-dependent, and open-set noise on CIFAR-10, CIFAR-100, and the real-world noisy dataset Food-101N. Our central finding is that the appropriate unlearning strategy is conditioned on the noise structure. Simple FT is a strong baseline across most closed-set scenarios; RL and SalUn are the most consistently robust methods and, under instance-dependent noise, approach retraining accuracy at a fraction of the computational cost; MUNBa shows advantages mainly under extreme symmetric noise. Under open-set noise, in contrast, we show that retraining on the cleaned subset degrades accuracy relative to the noisy baseline, so approximating the retrained model is not an adequate objective in this regime. On Food-101N, all MU methods remain competitive and achieve accuracies close to retraining despite reducing runtime by an order of magnitude. These findings provide practical guidelines for selecting MU strategies for post-training noisy-label correction.

I. INTRODUCTION

Post-training noisy-label correction traditionally relies on costly retraining, motivating machine unlearning as a cheaper alternative. This study compares five unlearning strategies across noise structures and finds that method choice should depend on the noise type.

  • Motivation: Post-training identification of noisy samples typically requires retraining on the cleaned dataset, which is computationally expensive for large datasets and models.Machine unlearning instead removes the influence of identified samples from an already trained model using far fewer training epochs.
  • Research gap: Prior work had not systematically compared different machine unlearning strategies for noisy-label correction across qualitatively different noise structures.The study evaluates NegGrad, Fine-Tuning, Random Labeling, SalUn, and MUNBa on CIFAR-10, CIFAR-100, and Food-101N.
  • Findings: The preferred unlearning strategy is conditioned on the noise structure: RL and SalUn suit closed-set and instance-dependent noise, FT suffices for asymmetric noise, and MUNBa helps mainly under extreme symmetric noise.These findings form the study’s central practical comparison.
  • Findings: Under open-set noise, retraining on the cleaned subset degrades test accuracy relative to the noisy-data model, making retraining approximation inadequate.The decision to unlearn should therefore also depend on the noise structure.
  • Findings: RL and SalUn match full retraining within a few percentage points under instance-dependent noise at roughly twenty times lower cost.This benefit persists even when only some noisy samples are identified.

II. RELATED WORK

Earlier work established machine unlearning and several noisy-label correction methods, but evaluations generally considered one method or limited noise scenarios. This paper addresses the resulting gap by comparing distinct strategies across multiple noise structures.

  • Prior methods: Machine unlearning removes the influence of selected training samples without complete retraining, while related methods include gradient-based unlearning and activation projection.These approaches motivated applying unlearning to noisy-label correction.
  • Prior methods: Random Labeling relabels forgotten samples randomly, SalUn updates parameters using saliency, and MUNBa balances forgetting and retention through bargaining.These methods represent distinct unlearning paradigms evaluated in the study.
  • Research gap: Previous noisy-label unlearning studies generally evaluated a single method under a limited set of noise scenarios.The present work instead compares five strategies across qualitatively different noise structures.
  • Research gap: The paper’s contribution is characterizing when fine-tuning is competitive as a function of noise structure, rather than merely observing that it can be competitive.This distinction defines the study’s novelty relative to prior work.

A. Label Noise

The paper models noisy labels as potentially differing from hidden true labels through class- and image-dependent transitions. It defines machine unlearning as removing identified noisy samples’ influence while approximating retraining at lower cost.

  • Label noise: A noisy observed label yi may differ from the hidden true label ˆyi through a transition probability ηjc(xi).The transition can depend on the source and target classes and, for instance-dependent noise, on the image.
  • Noise types: Symmetric noise flips labels randomly at a fixed rate, asymmetric noise flips between similar classes, and instance-dependent noise depends on both classes and the image.These definitions distinguish the synthetic noise structures evaluated later.
  • Noise types: Closed-set noise keeps noisy labels within the valid label set, whereas open-set noise includes samples whose hidden class lies outside that set.Open-set samples are drawn from an unknown distribution and assigned arbitrary valid labels.
  • Machine unlearning: Machine unlearning removes the influence of an identified subset Df from a model trained on D, while retraining applies the training algorithm to the remaining subset Dr.An unlearning algorithm produces θu from the original model, forget set, and retain set, ideally approximating θr at lower cost.

IV. METHODOLOGY

The study evaluates five unlearning strategies on CIFAR-10, CIFAR-100, and Food-101N under synthetic and real-world noise. The methods span direct forgetting, retain-set fine-tuning, relabeling, saliency-based updates, and bargaining-based optimization.

  • Datasets and noise: Experiments use CIFAR-10, CIFAR-100, and Food-101N, with synthetic symmetric, asymmetric, and instance-dependent noise added to the CIFAR datasets.Symmetric noise rates are η ∈ {0.2, 0.5, 0.8}, while asymmetric noise uses ηjc = 0.4.
  • Compared methods: NegGrad performs gradient ascent on the forget set, while Fine-Tuning updates the original model using only the clean retain set.FT relies on catastrophic forgetting to erode the removed samples’ influence.
  • Compared methods: Random Labeling assigns forgotten samples uniformly random labels before fine-tuning on the retain set and relabeled forget set.The method aims to destroy memorized noisy associations.
  • Compared methods: SalUn masks parameter updates using a saliency map, applying ascent to salient weights on forgotten data and descent on retained data elsewhere.Its update rule combines targeted forgetting with retention updates.
  • Compared methods: MUNBa computes a Pareto-optimal update that balances forgetting and retention gradients to avoid unstable unconstrained ascent.It frames unlearning as a cooperative bargaining game.

C. Implementation

The study evaluates corrective unlearning by comparing predictive utility and computational cost, using full retraining on cleaned data as the reference procedure. Experiments use defined CIFAR and Food-101N training protocols, with five MU methods run for 10 unlearning epochs.

  • Datasets and training: CIFAR-10 and CIFAR-100 experiments use ResNet-18 trained for 200 epochs, with results averaged across three independent random seeds.The protocol uses SGD with momentum, weight decay, batch size 256, and standard crop-and-flip augmentation.
  • Retraining baseline: Full retraining produces the reference model by training from scratch on the cleaned subset Dr = D \ Df.For synthetic CIFAR noise, known noisy samples define Df; Food-101N uses the dataset’s noisy identification.
  • Unlearning protocol: All five MU methods start from the fully trained noisy-data model and undergo 10 unlearning epochs with the original augmentations.FT, RL, SalUn, and MUNBa use learning rate 0.013, whereas NegGrad uses 10^-4.
  • Evaluation scope: Evaluation targets test accuracy and runtime efficiency rather than privacy-oriented forgetting guarantees.The stated goal is recovering generalization lost to memorized noisy labels.

V. RESULTS

Across closed-set and instance-dependent noise, several MU methods improve substantially over noisy-label training, with method effectiveness depending on the noise structure and severity. RL and SalUn are especially robust under instance-dependent noise, while MUNBa becomes advantageous mainly at extreme symmetric noise.

  • Experimental reporting: Table I covers CIFAR-10 with symmetric and asymmetric noise and CIFAR-100 with symmetric noise, while reporting accuracy and runtime efficiency for evaluated methods.Accuracies are mean±std over three runs, and MU methods within one standard deviation of the best mean are also highlighted.
  • Closed-set noise: NegGrad is consistently weakest in closed-set settings and becomes increasingly unstable as noise increases, whereas FT, RL, SalUn, and MUNBa improve over the noisy-label baseline.At moderate noise, RL, SalUn, and FT can overlap within one to two standard deviations.
  • Symmetric noise: MUNBa achieves 79.73% accuracy under 80% symmetric noise on CIFAR-10, outperforming the remaining MU methods in that setting.On CIFAR-100, the difference between MUNBa and FT falls within one standard deviation.
  • Instance-dependent noise: RL and SalUn consistently emerge as the most robust MU methods under instance-dependent noise, often offering the best accuracy–runtime trade-off.Their advantage is strongest in the challenging content-dependent corruption setting.
  • Instance-dependent noise: RL remains within approximately three percentage points of retraining on CIFAR-100 with 50% instance-dependent noise while reducing runtime from 15.10 to 1.15 minutes.This is presented as near-retraining performance at a fraction of the computational cost.

A. Open-Set Noise

Open-set noise differs from closed-set corruption because discarded out-of-distribution images can still contribute useful representations. In this regime, the noisy baseline can outperform retraining, making the standard retraining-approximation objective inadequate.

  • Benchmark construction: With 0% closed-set noise, open-set configurations insert out-of-distribution CIFAR-100 images with arbitrary CIFAR-10 labels while genuine CIFAR-10 labels remain correct.This construction explains why the open-set baseline is close to a model trained on clean CIFAR-10.
  • Open-set limitation: 95.11% versus 93.77% at 0/30 open-set noise, the noisy baseline outperforms the retrained model.The corresponding comparison is 95.11% versus 90.98% at 0/60.
  • Open-set limitation: Discarding identified open-set samples removes real images that can contribute to learned representations, while the reduced training set penalizes retraining.Thus, test accuracy can favor retaining rather than unlearning these samples.
  • Method dependence: In pure open-set columns, NegGrad is the best MU method because it perturbs the model least.When closed-set noise is mixed in, SalUn and RL become the strongest active approaches and surpass retraining at 30/30.

B. Real-World Noise

On Food-101N, machine-unlearning methods remain close to retraining while substantially reducing runtime, but imperfect noise identification remains a practical concern.

  • Food-101N: RL reaches 73.10% accuracy, only 1.09 percentage points below retraining, while reducing runtime from 422.33 to 41.26 minutes on Food-101N.Food-101N results use a single run per method because of computational cost.
  • Food-101N: FT, SalUn, and MUNBa remain within approximately 1.5 percentage points of RL on Food-101N.The relatively small performance differences suggest computational constraints can guide method selection in this setting.
  • Imperfect identification: Test accuracy increases monotonically with the forget rate for FT, SalUn, and MUNBa under symmetric noise on CIFAR-10 and CIFAR-100.The analysis varies forgetting from 25% to 100% of identified noisy samples across 20%, 50%, and 80% noise rates.
  • Imperfect identification: Even partial unlearning yields substantial gains over the baseline, while falsely flagged clean samples remain an untested failure mode.The experiments assess missed noisy samples but defer the complementary false-positive case.

C. Discussion and Limitations

The study finds that simple fine-tuning is broadly effective for closed-set noise, whereas open-set noise can make retraining-oriented unlearning counterproductive. Its conclusions are bounded by the evaluated backbone, repetition counts, metrics, and identification-error coverage.

  • Discussion: Simple Fine-Tuning is a strong baseline across most closed-set noise types, but open-set noise can make unlearning toward the retrained model counterproductive.The effectiveness of machine unlearning is strongly influenced by noise structure.
  • Limitations: The experiments use only a ResNet-18 backbone, so the cost argument for larger models is extrapolated rather than measured.Food-101N provides evidence at larger data scale, but larger architectures were not evaluated.
  • Limitations: Three synthetic runs and one Food-101N run limit statistical power for per-cell comparisons.The paper reports ties explicitly instead of strict rankings because of this limitation.
  • Limitations: The evaluation measures accuracy-oriented correction rather than forgetting-specific metrics such as membership inference.Therefore, the conclusions concern utility recovery rather than certified forgetting.
  • Limitations: The imperfect-identification analysis covers missed noisy samples but does not test unlearning falsely flagged clean samples.False negatives and false positives are both recognized, but only the former is experimentally varied.

VI. CONCLUSION

The paper systematically compares five machine-unlearning strategies for post-training noisy-label correction across synthetic and real-world noise. It finds that strategy choice depends on noise structure, with efficient methods often recovering most retraining benefits but open-set noise potentially favoring no unlearning.

  • VI. CONCLUSION: The study compares NegGrad, FT, RL, SalUn, and MUNBa on CIFAR-10, CIFAR-100, and Food-101N across multiple noise structures.The evaluated settings include symmetric, asymmetric, instance-dependent, open-set, and real-world noise.
  • VI. CONCLUSION: No single MU method dominates all scenarios; the appropriate strategy is conditioned on the noise structure.FT is strong for most closed-set settings, RL and SalUn consistently approach retraining, and MUNBa helps mainly under extreme symmetric noise.
  • VI. CONCLUSION: Under open-set noise, retraining on the cleaned subset degrades accuracy relative to the noisy baseline, so deciding whether to unlearn should also depend on noise structure.Approximating the retrained model is therefore not always an adequate objective.
  • VI. CONCLUSION: In many practical scenarios, simple and efficient approaches recover most of the benefits of full retraining.The conclusion frames machine unlearning as a practical alternative within the evaluated scope.
Loading 2608.30046v1…