PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Label Noise Modeling and Loss Correction

Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, Kevin McGuinness

arXiv:1904.11238v2cs.CV

TL;DR

Label noise can be fitted by CNNs, motivating an unsupervised way to estimate which samples are mislabeled. The paper fits a beta mixture to per-sample losses, uses its posteriors for dynamic bootstrapping, and combines this with mixup; experiments report strong robustness, while extreme-noise convergence remains a limitation.

  • Problem

    CNNs can fit random labels, so training needs to distinguish clean from noisy samples without relying on a clean dataset.

  • Method

    A two-component beta mixture models per-sample losses, and its posterior probabilities dynamically weight a bootstrapping loss combined with mixup augmentation.

  • Results

    The combined approach achieves remarkable accuracy improvements under high noise, including outperforming mixup at 80% label noise.

  • Takeaways & Limitations

    The method provides an unsupervised loss-correction procedure that retains noisy samples for representation learning while limiting overfitting to their labels.

  • Takeaways & Limitations

    The proposed M-DYR-H approach fails to converge on CIFAR-10 with 90% label noise, requiring modifications for extreme noise.

Abstract

from arXiv · show

Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE

1. Introduction

Label noise is common in visual datasets and can harm CNN generalization because networks readily fit incorrect labels. The paper proposes unsupervised loss modeling, dynamic correction, and mixup-based extensions to learn useful representations without discarding noisy samples.

  • Noisy labels arise from error-prone automatic or manual annotation and complicate computer-vision training.
  • CNNs can fit noisy labels, harming generalization, even though noisy labels may still provide useful representation-learning information.
  • The paper models clean and noisy samples with a two-component beta mixture fitted to per-sample losses, without requiring clean data.
  • Posterior probabilities from the mixture dynamically weight a bootstrapping loss that corrects sample losses while retaining noisy samples for learning.
  • The approach combines dynamic loss correction with mixup and guides mixup to converge under extreme label noise.

2. Related work

Related work addresses closed-set label noise through removal, relabeling, loss correction, and curriculum strategies, but several methods require clean data or discard useful noisy samples. This paper instead uses only training losses and combines unsupervised modeling with bootstrapping and mixup.

  • Closed-set noise keeps true labels within the known label set, whereas open-set noise may include out-of-distribution samples.
  • Uniform noise flips labels with equal probability, while class-conditional noise uses class-specific flipping probabilities.
  • Removing corrupted data can confuse difficult and noisy samples and prevents using noisy examples for representation learning.
  • Graphical-model, Conditional Random Field, and CNN relabeling methods commonly assume access to a small clean set, limiting applicability.
  • Loss-correction methods modify losses or class probabilities, while curriculum methods reduce the influence of samples considered noisy.
  • This paper uses only per-sample training losses to model clean and noisy samples, then combines bootstrapping and mixup for closed-set noise.

3. Learning with label noise

The paper models label noise from per-sample losses, using a beta mixture to distinguish clean and noisy examples and dynamically correct their training losses. It combines this correction with bootstrapping and mixup, while adding regularization to address instability under high noise.

  • 3.1. Label noise modeling: Random labels typically have higher early-training loss because networks learn clean labels before fitting noisy ones.This makes clean and noisy samples distinguishable from the loss distribution alone.
  • 3.1. Label noise modeling: The method fits a two-component mixture model to sample losses, representing clean and noisy examples without requiring labeled clean data.The posterior probability of the noisy component estimates each sample’s likelihood of being mislabeled.
  • 3.1. Label noise modeling: The beta mixture model better captures skewed loss distributions than a Gaussian mixture, improving clean-noisy classification ROC-AUC by around 5 points at 80% CIFAR-10 label noise.The beta distribution models symmetric and skewed distributions over normalized losses in [0, 1].
  • 3.2. Noise model for label correction: Dynamic bootstrapping sets each sample’s prediction weight to its estimated noisy probability, reducing reliance on labels for suspected noisy samples.The BMM is estimated after each epoch from standard cross-entropy losses, while the corrected training loss may differ from that estimation loss.
  • 3.3. Joint label correction and mixup data augmentation: Mixup is fused with dynamic bootstrapping to retain mixed-example regularization while correcting labels using independently inferred weights for each constituent sample.At high noise levels, mixing two incorrectly labeled samples can reduce mixup’s effectiveness, motivating the combination.
  • 3.3. Joint label correction and mixup data augmentation: At high label-noise levels, dynamic bootstrapping may fail to converge because unreliable predictions can encourage the network toward one class.A regularization term based on class priors and mean softmax probabilities is added to prevent assignment of all samples to a single class.

4. Experiments

Experiments on CIFAR-10 and CIFAR-100 evaluate dynamic loss correction, its combination with mixup, and extensions for extreme label noise. The proposed methods improve robustness under high corruption, including comparisons on alternative noise settings and TinyImageNet, while some configurations have convergence limits.

  • 4.1. Datasets and implementation details: Experiments use CIFAR-10 and CIFAR-100 with 50K training images, 10K validation images, 32×32 resolution, and PreAct ResNet-18.Training uses SGD with batch size 128.
  • 4.2. Static and dynamic loss correction: Dynamic bootstrapping improves over static bootstrapping, with CIFAR-10 80% noise last accuracy increasing from 25.7% for ST-H to 64.2 for DY-H.Hard bootstrapping outperforms the soft alternative in these experiments.
  • 4.3. Joint mixup and dynamic loss correction: At 80% noise, M-DYR-H improves CIFAR-10 best (last) accuracy from mixup’s 71.6 (46.7) to 86.8 (86.6), and CIFAR-100 from 30.8 (17.6) to 48.2 (47.2).Similar best and last performance across noise levels indicates robustness to varying corruption levels.
  • 4.3. Joint mixup and dynamic loss correction: Dynamic mixup uses per-sample noise probabilities to reduce noisy-label contributions while retaining mixup behavior for clean-clean and noisy-noisy pairs.The per-sample mixing weights are δp = p(k = 0 | ℓp) and δq = p(k = 0 | ℓq).
  • 4.4. On the limits of the proposed approach: MD-DYR-H shows more stable convergence at 90% label noise in both datasets, whereas M-DYR-H fails to converge on CIFAR-10 at that noise level.A decreasing-softmax variant, MD-DYR-SH, reports 69.1 accuracy on CIFAR-10 and 24.3 on CIFAR-100 at 90% noise.
  • 4.4. On the limits of the proposed approach: At 90% CIFAR-10 noise, the proposed approach exceeds reported best and last state-of-the-art accuracies of 58.3% and 58.0%.The comparison uses results reported with a PreAct ResNet-32.
  • 4.5. Comparison with related approaches: Under the alternative criterion excluding the true label from random candidates, MD-DYR-SH outperforms related work on both CIFAR-10 and CIFAR-100.M-DYR-H is slightly below Jiang et al. for low noise on CIFAR-100.
  • 4.6. Generalization of the proposed approach: On TinyImageNet, the proposed approach clearly outperforms mixup across different noise levels using the same network, hyperparameters, and learning-rate policy as CIFAR.On Clothing1M, it exceeds 71% test accuracy but remains below the 72.23% state of the art.

5. Conclusions

The paper presents an unsupervised BMM-based approach for training CNNs under label noise without clean data, combining dynamic bootstrapping with mixup. It reports strong CIFAR results, generalization to TinyImageNet, and limitations under non-uniform Clothing1M noise.

  • The method fits a beta mixture model to per-sample cross-entropy losses to model label noise without clean data.The BMM guides dynamic bootstrapping between network predictions and potentially noisy labels.
  • Dynamic bootstrapping is combined with mixup to produce a robust loss-correction approach.The BMM also guides mixup combinations for convergence under extreme noise.
  • Experiments on CIFAR-10 and CIFAR-100 demonstrate outstanding performance while revealing both strengths and weaknesses of the approach.
  • The approach generalizes well to TinyImageNet but has limitations under non-uniform noise in Clothing1M.

A. Beta Mixture Model (BMM)

The BMM estimates clean/noisy membership from sample losses and supports loss correction, with strong discrimination at multiple noise levels. Its usefulness is reflected in improved classification and a substantial gain over fixed bootstrapping weights, while Gaussian mixtures perform worse.

  • BMM performance under low levels of label noise: With near 0% noise, the loss distribution may not form a two-component mixture, causing estimation errors and reliance on network predictions instead of clean labels.For 20% noise, the method still outperforms the compared state-of-the-art at the end of training.
  • BMM parameter estimation frequency: The BMM parameters are re-estimated after every epoch following an initial warm-up, using losses computed with the original labels.
  • BMM classification accuracy and robustness: The BMM exceeds 0.98 AUC on CIFAR-10 with 20%, 50%, and 80% label noise.AUC increases during training and improves faster at lower noise levels.
  • Effect of BMM classification accuracy on image classification accuracy: BMM prediction accuracy and image classification accuracy tend to increase together, especially at higher noise levels.
  • GMM comparison: At 80% label noise, the GMM reduces clean/noisy AUC from 0.98 to 0.94 and image classification accuracy from 86.6 to 83.5.
  • Performance attributable to the BMM: At 80% label noise, replacing the BMM with fixed weights reduces M-DYR-H accuracy from 86.6 to 74.6 in the last epoch.The comparison keeps mixup for robustness while assigning fixed weights of 0.8 to ground truth and 0.2 to network prediction.

B. Hyperparameters

Experiments use shared configurations across datasets rather than carefully tuned hyperparameters. Warm-up, learning-rate schedules, mixup, and BMM estimation settings define the main training choices, with robustness to the number of EM iterations.

  • General configuration: All datasets use the same hyperparameter configuration, producing consistent improvements without carefully tuned hyperparameters.The authors suggest results may be suboptimal without a label-noise-free validation set, whose availability is not assumed.
  • Warm-up: A high-learning-rate warm-up helps learn structured data and separate clean and noisy losses for BMM fitting.
  • Experiment setup: Experiments use normalized, randomly flipped images with 32×32 crops after four-pixel zero padding, a PreAct ResNet-18, and SGD with momentum.The optimizer uses momentum 0.9, weight decay 10^-4, and batch size 128.
  • Training schedule without mixup: Without mixup, training lasts 120 epochs with learning-rate reductions after epochs 30, 80, and 110, following a 30-epoch warm-up.
  • Training schedule with mixup: With mixup, training lasts 300 epochs, reduces the learning rate after epochs 100 and 250, and uses a 105-epoch warm-up with α = 32.
  • BMM parameter estimation: Changing BMM estimation from 10 EM iterations to 5 or 20 yields similar CIFAR-10 accuracy at 80% label noise.The reported best and last epoch accuracies are 87.4 (87.2) and 86.9 (86.3), respectively.
Loading 1904.11238v2…