PaperlayerView exact revision on arXiv

Source-linked AI summary

SELF: Learning to Filter Noisy Labels with Self-Ensembling

Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, Thomas Brox

arXiv:1910.01842v1cs.CVcs.LGstat.ML

TL;DR

Noisy labels can cause DNNs to memorize data and lose generalization. SELF progressively filters potentially noisy supervision using self-ensemble predictions while retaining filtered samples in an unsupervised loss. It outperforms prior approaches across image datasets and noise types while remaining robust to architecture changes.

  • Problem

    DNNs can eventually memorize noisy labels, and traditional training on noisy data severely deteriorates generalization.

  • Method

    SELF uses running averages of a network’s predictions across training epochs to filter potentially noisy labels, then combines clean supervision with unsupervised learning on the full dataset.

  • Results

    SELF outperforms previous works across CIFAR-10, CIFAR-100, and ImageNet under symmetric and asymmetric noise and remains robust to increasing noise ratios and architecture changes.

  • Takeaways & Limitations

    SELF transforms noisy-label learning into a semi-supervised task by separating clean labels from noisy labels treated as unlabeled data.

  • Takeaways & Limitations

    SELF does not repair all noisy labels, and filtering can be challenging when many classes make the correct hidden label difficult to identify.

Abstract

from arXiv · show

Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.

1 INTRODUCTION

SELF addresses the degradation caused by noisy labels by progressively filtering potentially erroneous supervision using self-ensemble predictions. It combines cleaner supervised subsets with ensemble-based stabilization and unsupervised use of filtered samples, achieving robust performance across noise settings and datasets.

  • Motivation: Noisy labels can cause DNNs to memorize training data and substantially degrade generalization, making noise mitigation practically important.Crowdsourced and web annotations provide scalable but imperfect alternatives to high-quality human annotation.
  • SELF framework: SELF identifies potentially noisy labels during training and prevents filtered samples from contributing supervision.The framework progressively focuses learning on samples considered consistently labeled.
  • SELF framework: Self-ensemble predictions average outputs across training iterations to detect samples whose predictions agree with their provided labels.Clean samples tend to produce consistent predictions, whereas wrongly labeled samples produce inconsistent or oscillating predictions.
  • SELF framework: The framework stabilizes learning with a running-average model and uses filtered samples through an unsupervised loss rather than discarding the entire dataset.The running-average model supplies a more stable supervisory signal than noisy model snapshots.
  • Results: SELF consistently outperforms existing approaches across CIFAR-10, CIFAR-100, and ImageNet under symmetric and asymmetric noise, while remaining robust to network architecture choices.The method can transform noisy-label learning into semi-supervised learning by separating clean labeled data from noisy unlabeled data.

2 SELF-ENSEMBLE LABEL FILTERING

SELF progressively filters potentially noisy labels using moving-average ensembles of model snapshots and predictions, while retaining filtered samples through unsupervised learning. The framework repeatedly trains on cleaner label sets and stops when validation performance no longer improves.

  • 2.1 OVERVIEW: Each iteration trains on a detected set of potentially correct labels, maintains a running-average model, and evaluates it across the dataset for an additional learning signal.The process uses Mean Teacher model snapshots and validation-based selection of the best model.
  • 2.2 PROGRESSIVE LABEL FILTERING: SELF does not repair every noisy label, but filtering from the original label set allows clean labels removed earlier to be reconsidered later.The authors note that recovering hidden correct labels can be extremely challenging when many classes are present.
  • 2.2 PROGRESSIVE LABEL FILTERING: The algorithm initializes training on the noisy dataset, filters labels using the best model’s ensemble predictions, retrains on the filtered set, and repeats until validation accuracy stops improving.Filtered samples are removed from the supervised dataset, while the best model is retained at termination.
  • 2.1 OVERVIEW: SELF progressively identifies and removes potentially noisy labels from supervised training using ensembles of models and predictions.Filtering decisions rely on agreement between the provided label and the ensemble’s maximal-likelihood prediction.
  • 2.3 SELF-ENSEMBLE LEARNING: Moving-average predictions accumulated across training epochs provide a more stable filtering basis than predictions from a single epoch.For sample k at epoch j, the ensemble prediction is updated as z_j = αz_{j−1} + (1 − α)ẑ_j, with negligible memory and computation overhead.
  • 2.2 PROGRESSIVE LABEL FILTERING: The framework can reduce computation by continuing training from the previous best model rather than retraining from scratch, with a configurable maximum number of filtering iterations.The new filtered dataset is expected to require only gradual adaptation to its changed noise ratio.

3 RELATED WORKS

Prior work addresses noisy labels through natural robustness, semi-supervised learning, label weighting, collaborating networks, or loss modification. SELF differs by progressively filtering labels with restricted ensemble-based predictions without modifying the primary loss.

  • Natural robustness: Earlier studies found that DNNs can be naturally robust to label noise under particular random-noise conditions without modifying the network or training procedure.SELF relies on this robustness to initiate its self-ensemble filtering process.
  • Semi-supervised learning: Semi-supervised approaches counteract noise but typically learn from a static initial noisy label set and lack mechanisms to repair labels.Their supervised losses can remain high until the model strongly overfits the label noise.
  • Label weighting: Label-weighting methods reduce the influence of suspected errors, but lower weights can still permit learning from wrong labels under nonlinear classification losses.SELF instead assigns extreme weights of zero or one to filtered samples.
  • Collaborating networks: Collaborating-network methods use two simultaneously learned networks, whereas SELF restricts its second network to running averages of the first network.The authors present ensemble learning as a way to counteract model fluctuations while avoiding a freely predictive second network.
  • Loss modification: Loss-modification methods alter the classification objective, while SELF preserves the primary loss and uses progressive filtering to support transfer beyond classification.The cited approaches include noise-transition estimation, human-in-the-loop learning, and alternative cross-entropy losses.

4 EVALUATION

SELF is evaluated on CIFAR-10, CIFAR-100, and ImageNet under symmetric and asymmetric label noise, across noise ratios and network architectures. It consistently outperforms prior methods, while ablations show that progressive filtering, model ensembling, and moving-average predictions jointly improve robustness.

  • Evaluation setup: SELF is evaluated on CIFAR-10, CIFAR-100, and ImageNet with symmetric and asymmetric label noise, alongside architecture and component ablations.The experiments include uniform noise, semantically related class flips, pair-wise flips, and multiple network architectures.
  • Symmetric label noise: SELF remains robust up to 60% uniform noise on CIFAR-10 and CIFAR-100 and outperforms previous works, although performance drops at 80% noise.A clean validation set improves performance but is not necessary; the 80% result is particularly affected by reliance on an extremely noisy validation set.
  • ImageNet: More than 5% absolute improvement over the best previously reported results is achieved by SELF with ResNext50 on ImageNet under 40% label noise.Even ResNext18 outperforms MentorNet, which uses the more powerful ResNet101 architecture.
  • Asymmetric label noise: SELF retains high performance under asymmetric noise, with only a small performance drop at 40% noise.The asymmetric scenarios include semantically similar class flips on CIFAR-10 and cyclic class flips on CIFAR-100.
  • Architecture robustness: SELF outperforms other methods in all tested scenarios except CIFAR-10 with 80% noise and remains consistent across underlying architectures.Table 3 reports compatibility with all tested architectures.
  • Ablation study: Ablations show that progressive filtering and model ensembles improve robustness, while moving-average predictions provide the final boost for reliable sample detection.The combined framework remains robust until 80% noise, where partial variants still fail or lose performance.

5 CONCLUSION

SELF trains robust deep models under noisy labels by filtering hard-to-learn samples, using clean supervision on retained samples, and applying unsupervised loss to the full dataset. It achieves strong generalization across datasets, noise types, noise ratios, and architectures.

  • 5 CONCLUSION: SELF filters potentially noisy samples using ensembles of predictions across training epochs.The retained samples provide clean supervision, while the entire dataset contributes through an additional unsupervised loss.
  • 5 CONCLUSION: SELF outperforms previous works on CIFAR-10, CIFAR-100, and ImageNet under symmetric and asymmetric noise.The framework remains robust as noise ratios increase and network architectures change.

A.1 MEAN TEACHER MODEL FOR ITERATIVE FILTERING

The iterative Mean Teacher procedure trains a student on potentially clean labels while maintaining a detached teacher model whose weights are updated by exponential moving average. It returns the best teacher model after evaluating performance during training.

  • A.1 MEAN TEACHER MODEL FOR ITERATIVE FILTERING: Each filtering iteration initializes a student network and a detached Mean Teacher copy using the potentially clean dataset Dfilter.At iteration i = 0, Dfilter is the entire labeled dataset.
  • A.1 MEAN TEACHER MODEL FOR ITERATIVE FILTERING: The training loss combines the student’s classification loss with a consistency loss between student and teacher outputs.
  • A.1 MEAN TEACHER MODEL FOR ITERATIVE FILTERING: During each training iteration, the optimizer updates the student, and the teacher weights follow an exponential moving average of the student weights.
  • A.1 MEAN TEACHER MODEL FOR ITERATIVE FILTERING: The procedure returns the best teacher model selected from performance evaluations during training.

A.2.1 CIFAR-10 AND CIFAR-100

The CIFAR-10 and CIFAR-100 evaluation uses contaminated validation data, standard training configurations, and several augmentation and optimization settings.

  • CIFAR-10 and CIFAR-100 validation sets are contaminated with the same noise ratio as training unless stated otherwise.
  • SELF training uses SGD with Nesterov momentum, cosine learning-rate annealing, weight decay of 2e-4, and 600 total epochs.Each filtering step permits up to 300 iterations and uses 50-epoch patience.
  • The evaluation includes uniform-noise classification tasks on CIFAR-10 and CIFAR-100, with training and validation noise uncorrelated.Accuracy on the noisy validation set provides a noisy estimate of generalization to unseen test data.
  • Mean-teacher training uses mean-squared-error consistency loss, consistency weight 100.0, logit-distance cost 0.01, and batches split into labeled and unlabeled samples.The total batch size is 512, with 124 labeled and 388 unlabeled samples.
  • Input preprocessing normalizes data to zero mean and unit standard deviation, followed by random translation, reflection, and horizontal flipping.

A.2.2 IMAGENET-ILSVRC-2015

The ImageNet-ILSVRC-2015 setup uses ResNet and ResNeXt models, large-image semi-supervised training, and top-5 label filtering to accommodate multiple objects per image.

  • ImageNet evaluation uses ResNet and ResNeXt networks, with ResNeXt variants configured at cardinality 32 and base width 4.
  • ImageNet training uses a total batch size of 40, split evenly between labeled and unlabeled samples, with mean-squared-error consistency loss weighted by 1000.Consistency ramp-up lasts 5 epochs, and patience is four epochs for stopping the current filtering iteration.
  • ImageNet filtering keeps a sample when its provided label appears among the model’s top 5 predictions.Top-5 filtering is used because images may contain multiple objects, making maximum-likelihood filtering too strict.
  • ImageNet augmentation includes random rotations up to 10 degrees, resizing to 224x224, horizontal flips, and color jittering.Validation images are resized to 256x256.
  • The entropy loss encourages extreme predictions for individual samples while maximizing mean-prediction entropy balances predictions across samples.

A.2.3 SEMI-SUPERVISED LOSSES

The semi-supervised-loss analysis compares push-away, entropy, and mean-teacher strategies within iterative filtering, showing that retaining filtered samples as unlabeled data is important.

  • Push-away-loss: Negative sample reweighting can perform gradient ascent away from potentially wrong labels, but negative-log-likelihood gradients vanish when predictions approach noisy labels.
  • Push-away-loss: The push-away-loss is proposed as a baseline with improved gradients for pushing the model away from potentially wrong labels.
  • Entropy-loss: Entropy minimization encourages extreme per-sample predictions while balancing predictions across classes over many samples.It can be applied to all samples or combined with filtering that removes potentially wrong labels from supervised learning.
  • Limitation: At high noise ratios, wrong-label predictions fluctuate across training iterations, so relying on a single noisy snapshot can produce many incorrect filtering decisions.
  • Sample retention: Deleting filtered samples creates a performance gap close to 9% at 80% label noise compared with SELF, which uses them as unlabeled data.Continuing to use filtered samples in the unsupervised loss provides meaningful learning signals.
  • Mean-teacher comparison: With filtering, mean teacher performs best, while entropy-loss variants outperform the standard push-away-loss.The reported advantage is attributed to temporal ensembling of models and sample predictions for filtering.
  • Training dynamics: At 60% and 80% CIFAR-100 noise, mean-teacher models outperform student models, while progressive filtering avoids the rapid overfitting seen in regular training.With filtering, performance increases slowly while training accuracy approaches 100%.
Loading 1910.01842v1…