PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey

Görkem Algan, Ilkay Ulusoy

arXiv:1912.05170v3cs.LGcs.CVstat.ML

TL;DR

Deep-learning image classifiers need extensive labeled data, but practical collection and annotation processes produce noisy labels, which networks can memorize through overfitting. This paper surveys deep-learning methods for noisy labels, grouping them into noise model based and noise model free approaches. It also discusses experiments, benchmarks, and practical method-selection guidance while noting important evaluation and scope limitations.

  • Problem

    The literature lacked a comprehensive survey focused explicitly on deep-learning classification methods for noisy labels, despite costly annotation and DNN overfitting challenges.

  • Method

    The paper surveys noisy-label algorithms and categorizes them as noise model based methods or noise model free methods.

  • Results

    The survey finds that methods have different advantages and disadvantages, with method choice depending on the use case and available noise-structure information.

  • Takeaways & Limitations

    Noise model based methods are suggested when noise is domain-specific and prior structural information is available, while dataset cleansing or pruning suits preprocessing-oriented purification.

  • Takeaways & Limitations

    Synthetic-noise results on toy datasets are not appropriate for fair algorithm comparison, while web datasets often have unknown noise rates and class-distribution bias.

Abstract

from arXiv · show

Image classification systems recently made a giant leap with the advancement of deep neural networks. However, these systems require an excessive amount of labeled data to be adequately trained. Gathering a correctly annotated dataset is not always feasible due to several factors, such as the expensiveness of the labeling process or difficulty of correctly classifying data, even for the experts. Because of these practical challenges, label noise is a common problem in real-world datasets, and numerous methods to train deep neural networks with label noise are proposed in the literature. Although deep neural networks are known to be relatively robust to label noise, their tendency to overfit data makes them vulnerable to memorizing even random noise. Therefore, it is crucial to consider the existence of label noise and develop counter algorithms to fade away its adverse effects to train deep neural networks efficiently. Even though an extensive survey of machine learning techniques under label noise exists, the literature lacks a comprehensive survey of methodologies centered explicitly around deep learning in the presence of noisy labels. This paper aims to present these algorithms while categorizing them into one of the two subgroups: noise model based and noise model free methods. Algorithms in the first group aim to estimate the noise structure and use this information to avoid the adverse effects of noisy labels. Differently, methods in the second group try to come up with inherently noise robust algorithms by using approaches like robust losses, regularizers or other learning paradigms.

1. Introduction

Deep learning systems need large labeled datasets, but label noise commonly arises from costly, automated, crowdsourced, and difficult annotation processes. This survey addresses the lack of a comprehensive deep-learning-focused review by organizing noisy-label methods into two groups.

  • Motivation: Deep neural networks can overfit even completely random noise, making noisy-data training an important challenge.The paper notes that many methods have therefore been proposed to train DNNs effectively with noise.
  • Sources of label noise: Large-scale supervised learning is costly because collecting and labeling the required datasets demands substantial resources.Web data, crowdsourcing, and automated labeling reduce costs but introduce label noise.
  • Sources of label noise: Label noise can arise from inexperienced annotators, difficult examples, automated systems, crowdsourcing, or intentional data poisoning.These sources make label noise a natural outcome of dataset collection and labeling.
  • Contribution: Existing surveys cover older machine-learning techniques under label noise, but the literature lacked a comprehensive survey centered on deep learning.The paper fills this absence by briefly explaining and grouping relevant methods.
  • Organization: The survey categorizes proposed solutions, presents experimental setups and a benchmark leaderboard, then concludes with practical guidance.Methods are discussed across the paper’s methodology sections, experiments, and conclusion.

2. Preliminaries

The preliminaries define label noise, formalize learning under a noisy distribution, and describe its sources and the survey’s taxonomy. The paper distinguishes noise model based methods from noise model free strategies focused on avoiding overfitting.

  • Problem statement: With noisy labels, training data are drawn from a noisy distribution Dn, so minimizing risk over Dn can produce parameters different from the desired classifier.Classical risk minimization is therefore insufficient in this setting.
  • Noise taxonomy: Label noise may depend on neither features nor class, only class, or both features and class.The paper describes these as random, Y-dependent, and xy-dependent noise, respectively.
  • Sources of label noise: Real-world label noise occurs in medical imaging, segmentation, crowdsourcing, social tagging, financial analysis, web data, multi-expert annotation, and poisoning.Expert disagreement and task complexity can produce noise even without automated labeling.
  • Methodologies: The taxonomy is practical rather than exclusive because algorithms often lack sharp boundaries and may fit multiple categories.For consistency, methods are assigned to the subclass they most resemble.
  • Methodologies: Noise model based methods estimate the noise structure and use it to neglect, de-emphasize, or correct noisy information.Their performance depends heavily on accurately estimating the underlying noise, while decoupling estimation from classification can ease integration.
  • Methodologies: Noise model free methods avoid explicit noise modeling and instead regularize training to reduce overfitting-related degradation.These methods assume the classifier is not inherently too sensitive to noise.

3. Noise Model Based Methods

Noise model based methods seek to recover the hidden noise structure while training the classifier, using a known or estimated mapping from clean to noisy distributions.

  • Noise model based methods: When the mapping M from clean distribution D to noisy distribution Dn is known, it can help reverse the effect of noisy samples.The methods simultaneously estimate the underlying noise structure and train the base classifier.

1. Noisy Channel

The noisy-channel subsection lists approaches that estimate noise types or relevance through explicit, iterative, or complex channel calculations.

  • Noisy Channel: Noisy-channel methods include explicit calculations from noisy, clean, or easy data, as well as iterative estimates using EM, network layers, anchor points, or Dirichlet distributions.The taxonomy also includes complex noisy channels for noise-type and relevance estimation.
  • Noisy Channel: Noise model based methods are presented as a distinct methodological family for modeling label corruption.The section heading places noisy-channel approaches within the broader noise-model-based category.

1. Robust Losses

Robust-loss methods and noisy-channel methods address label noise through different mechanisms: the former changes the loss, while the latter models noise and corrects predictions during training. Noisy-channel performance depends strongly on accurately estimating the underlying noise structure, which becomes difficult as the number of classes grows.

  • Robust Losses: Robust-loss methods include non-convex, MAE, generalized cross-entropy, symmetric, information-theoretic, and other loss formulations designed for noisy-label learning.The listed methods include 0-1 loss surrogates, unbiased estimators, modified cross-entropy, linear-odd losses, classification-calibrated losses, and robust-loss SGD.
  • Noisy Channel: The mapping Q can be represented by a transition matrix T, where T_ij = p(ỹ = j|y = i), and prediction correction is called loss-correction.Each row of T contains transition probabilities that sum to one.
  • Noisy Channel: Transition-matrix estimation becomes intractable as class counts increase, so some methods restrict connections to probable or predefined label nodes.These restrictions are determined by human experts and can insert additional noise information into training.
  • Noisy Channel: A noisy channel maps network predictions to observed noisy labels so the base classifier can receive noise-free gradients during training.The channel is removed during evaluation to obtain noise-free predictions from the base classifier.

3.2. Label Noise Cleaning

Label-noise cleaning estimates which observed labels are incorrect and replaces them with cleaner posterior information. Methods may use clean subsets, jointly trained cleaning networks, or incremental posterior estimation when only noisy labels are available.

  • Label Noise Cleaning: Label cleaning replaces suspicious noisy labels using G(ỹ_i, x_i) = p(y_i|ỹ_i, x_i), estimated from features and noisy labels.The feature extractor may be pretrained or may be the base classifier, creating an iterative interaction between classifier quality and label-cleaning accuracy.
  • Using Clean Data: When clean data exists, models can blend noisy and predicted labels, use ensemble agreement, or learn noise structure from conflicting clean and noisy annotations.Jointly trained label-cleaning networks can correct labels on the clean subset and provide corrected labels for noisy data.
  • Using Data With Just Noisy Labels: Without noise-free data, methods incrementally estimate cleaner posterior label distributions while regularizing against collapse to a single predicted class.Compatibility losses and confidence policies help preserve contributions from the observed noisy-label distribution.
  • Using Data With Just Noisy Labels: Some methods infer whether samples are noisy from loss values, including fitting instance losses with a beta mixture model in an unsupervised manner.Other approaches use staged inference procedures to identify or correct noisy samples.

3.3. Dataset Pruning

Dataset-pruning methods reduce the influence of label noise by removing suspicious samples or labels rather than correcting every noisy annotation. Their central trade-off is between limiting noise impact and avoiding unnecessary loss of training information.

  • Dataset Pruning: Pruning noisy samples can improve performance by preventing harmful noise, but removing too many examples causes unnecessary data loss.Methods therefore aim to remove as few samples as possible.
  • Dataset Pruning: Pruning either removes noisy samples entirely or removes only their labels, producing labeled and unlabeled subsets for semi-supervised learning.The second strategy retains the inputs while discarding unreliable annotations.
  • Filtering Strategies: Filtering methods identify noisy examples through base-network errors, ensemble noisiness scores, probabilistic subsets, or prediction confidence.One approach estimates the noise rate from confidently clean and noisy subsets, then removes the least-confident samples accordingly.
  • Iterative Pruning: Iterative label removal updates the training dataset across epochs and can combine noisy-label assessment with semi-supervised learning.Consistency between provided labels and moving-average model predictions is used to evaluate label noisiness.

3.4. Sample Choosing

Sample-choosing methods dynamically control which examples enter training, aiming to approximate a noise-free data distribution without modifying the underlying learning system. They favor easier or lower-loss examples first, but may delay hard informative samples and discard too much data if poorly balanced.

  • Sample Choosing: Sample choosing uses a binary operator V(x_i, y_i) to decide whether each training example enters the next iteration.When V equals one for all examples, the procedure reduces to classical risk minimization; a static V corresponds to dataset pruning.
  • Sample Choosing: Dynamic selectors continuously monitor the base classifier and choose samples for subsequent training iterations, tracing a path intended to mimic the noise-free distribution.Because they operate on the input stream, these methods are easier to attach to existing algorithms.
  • Sample Choosing: Sample-choosing systems must preserve data balance because prioritizing low-loss samples can slow learning by postponing hard, informative examples.The paper identifies unnecessary data omission and delayed learning of difficult examples as practical concerns.
  • Curriculum Learning: Curriculum and self-paced learning begin with easy or low-loss examples and progressively introduce harder instances as training improves.In noisy-label settings, clean labels are treated as easier while noisy labels are treated as harder.
  • Multiple Classifiers: Multiple classifiers can help select later training batches by correcting one another’s mistakes through differences in learned representations.The initialization of these classifiers is essential for this approach to work.

3.5. Sample Importance Weighting

Sample importance weighting assigns dynamic, instance-specific weights to emphasize cleaner samples during training. Unlike binary sample selection, these weights vary across instances, but rapid changes and accumulated errors can destabilize or bias learning.

  • 3.5. Sample Importance Weighting: Instance-dependent weights emphasize cleaner samples to improve model-weight updates under label noise.The approach minimizes empirical risk while assigning weights according to estimated noisiness.
  • 3.5. Sample Importance Weighting: Unlike sample choosing, weighting uses non-binary β values that differ across instances.If β were binary, the formulation would match sample choosing.
  • 3.5. Sample Importance Weighting: Dynamic weights keep changing during training, making it challenging to prevent rapid or sharp shifts from disrupting stabilization.
  • 3.5. Sample Importance Weighting: Accumulated errors can bias weighting methods toward a particular subset of the data.The literature proposes methods for estimating β to reduce noise effects.
  • 3.5. Sample Importance Weighting: When clean data are available, a simple strategy is to assign clean examples greater weight, although this uses information poorly and requires such data.Other methods estimate weighting factors through meta-learning, multilayer perceptrons, or iterative noisy-sample detection.

3.6. Labeler Quality Assessment

Labeler quality assessment addresses disagreement among annotators by modeling their characteristics and estimating ground-truth labels. The surveyed methods distinguish label quality from other noise-robust strategies such as cleaning, pruning, and sample weighting.

  • 3.6. Labeler Quality Assessment: Crowdsourced and expert datasets can contain contradictory labels because annotators differ in expertise.
  • 3.6. Labeler Quality Assessment: Figure 3 contrasts label correction, dataset pruning, and sample importance weighting as distinct responses to noisy labels.It depicts weighting by changing sample size, with clean samples up-weighted and noisy samples down-weighted.
  • 3.6. Labeler Quality Assessment: Modeling annotator characteristics can significantly improve performance on datasets with multiple labelers.
  • 3.6. Labeler Quality Assessment: Expectation-maximization can jointly estimate noisy labeler characteristics and ground-truth labels.When noise depends on the true class, annotator characteristics can be represented with a noise transition matrix.
  • 3.6. Labeler Quality Assessment: Noise model-based methods depend on accurately estimating noise structure, creating a coupled estimation dilemma.Regularization and assumptions about the noise structure are commonly used to stabilize estimates, but assumptions reduce applicability across setups.
  • 3.6. Labeler Quality Assessment: A benefit of noise model-based methods is that prior or domain-specific noise information can be incorporated, while noise estimation and classification remain decoupled.This separation can make the methods easier to implement with an existing classifier.

4. Noise Model Free Methods

Noise model-free methods seek robustness without explicitly modeling label noise, using robust losses, regularization, meta-learning, and alternative learning paradigms. Their generality is limited when noise is structured rather than anomalous or random.

  • 4. Noise Model Free Methods: Noise model-free algorithms design robustness directly into the learning procedure instead of explicitly estimating noise.They commonly treat noisy data as anomalies and regularize memorization of noisy examples.
  • 4. Noise Model Free Methods: Robust losses aim to preserve classification accuracy despite noisy labels, but noise can still substantially degrade their performance.These losses also treat clean and noisy data identically, limiting use of prior information.
  • 4. Noise Model Free Methods: For uniform noise, a loss satisfying Σ_k l(fθ(x), k) = C for every x is noise tolerant.Here C is a constant, and the condition is presented as sufficient for uniform-noise tolerance.
  • 4. Noise Model Free Methods: MAE is reported as more noise tolerant than categorical cross entropy, while IMAE scales gradients to adjust MAE’s weighting variance.CCE produces larger gradients for abnormal samples, whereas MAE weights data points equally and can underfit.
  • 4. Noise Model Free Methods: Meta-learning methods select noise-tolerant procedures or parameters by optimizing objectives involving noisy data, synthetic corruption, clean data, or teacher outputs.Examples include damage recovery through scaled gradient ascent and teacher-student consistency or distillation.
  • 4. Noise Model Free Methods: Regularizers such as weight decay, dropout, adversarial training, mixup, and label smoothing frame noisy-label degradation as overfitting to noise.The assumption is generally valid for random noise but may fail for more complex noise.
  • 4. Noise Model Free Methods: Bagging is more robust to label noise than boosting, while BrownBoost and LogitBoost are reported as more robust than AdaBoost.AdaBoost can overfit noise because it assigns excessive weight to noisy samples.
  • 4. Noise Model Free Methods: Other paradigms include complementary labels, autoencoder-based noise discrimination, clean prototypes, and multiple-instance learning.These approaches respectively exploit non-membership labels, reconstruction error, representative class features, or grouped predictions.

5. Experiments

Experiments in noisy-label research commonly use synthetic-noise toy datasets or imperfect real-world benchmarks, limiting fair comparison. The survey presents Clothing1M results under a standardized data and model setup while noting remaining dataset and method trade-offs.

  • 5. Experiments: Toy datasets such as MNIST, MNIST-Fashion, and CIFAR are often tested with synthetic label noise for quick implementation.Because noise types, architectures, and hyperparameters differ across studies, these results mainly serve as proof of concept rather than fair comparisons.
  • 5. Experiments: Food101N has about 20% noise, which the survey considers too low for evaluating noise-robust algorithms.
  • 5. Experiments: WebVision contains 2.4 million images but requires substantial computation and still fails to provide a benchmarking dataset for noise-robust evaluation.Some studies restrict experiments to its first 50 classes to reduce computational demands.
  • 5. Experiments: The survey introduces a web-collected dataset with one million images across 14 classes and an estimated real-world noise ratio of around 40%.It also provides 50K, 14K, and 10K verified clean images for training, validation, and testing, respectively.
  • 5. Experiments: Methods evaluated on the one-million-image noisy training set generally use pretrained ResNet50 and stochastic gradient descent without the additional 50K clean training images.
  • 5. Experiments: Table 2 ranks Clothing1M algorithms by test accuracy using results from studies that exclude additional 50K clean training data.The survey cautions that methods also differ in computational cost and memory requirements.

6. Conclusion

The paper concludes that learning with noisy labels is important for practical machine learning, while deep learning in this setting remains understudied. It organizes available methods by whether they model noise explicitly and highlights unresolved evaluation and data-scarcity challenges.

  • Label-noise robustness is important for using web-scale, self-sustained learning systems and for extending noisy-label learning beyond image classification.
  • Noise model-based methods are most appropriate when the noise structure is domain-specific and prior information about it is available.
  • Noise model-free methods avoid prior assumptions about noise structure and are easier to implement when noise is assumed random and degradation is linked to overfitting.
  • Future research: Understanding which network layers are affected by different noise models could clarify how label noise corrupts learned representations.
  • Evaluation challenges: Benchmarking remains limited because synthetic noise differs across studies, web-collected datasets have unknown rates and class bias, and adjustable verified-noisy datasets are lacking.
  • Future research: Learning effectively from small noisy datasets without ground truth remains underexplored, despite potential applications such as medical imaging.
Loading 1912.05170v3…