PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Noisy Labels with Deep Neural Networks: A Survey

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, Jae-Gil Lee

arXiv:2007.08199v7cs.LGcs.CVstat.ML

TL;DR

Noisy labels severely challenge DNN generalization, motivating robust training for real-world learning with imperfect annotations. This survey reviews 62 methods, organizes them into five methodological groups, compares six properties, and examines noise-rate estimation and evaluation practices. It concludes that no ideal method supports all required properties and identifies research directions for future work.

  • Problem

    Noisy labels are common when high-quality annotations are costly or difficult to obtain, and they can severely degrade DNN generalization.

  • Method

    The survey reviews 62 robust-training methods, groups them into five methodological categories, compares six properties, and analyzes noise-rate estimation and evaluation practices.

  • Results

    No existing method supports all the evaluated properties, which vary across methodological categories.

  • Takeaways & Limitations

    Robust-training methods require different trade-offs, while noise-rate estimation and evaluation guidelines remain important for practical use.

Abstract

from arXiv · show

Deep learning has achieved remarkable success in numerous domains with help from large amounts of big data. However, the quality of data labels is a concern because of the lack of high-quality labels in many real-world scenarios. As noisy labels severely degrade the generalization performance of deep neural networks, learning from noisy labels (robust training) is becoming an important task in modern deep learning applications. In this survey, we first describe the problem of learning with label noise from a supervised learning perspective. Next, we provide a comprehensive review of 62 state-of-the-art robust training methods, all of which are categorized into five groups according to their methodological difference, followed by a systematic comparison of six properties used to evaluate their superiority. Subsequently, we perform an in-depth analysis of noise rate estimation and summarize the typically used evaluation methodology, including public noisy datasets and evaluation metrics. Finally, we present several promising research directions that can serve as a guideline for future studies. All the contents will be available at https://github.com/songhwanjun/Awesome-Noisy-Labels.

I. INTRODUCTION

Noisy labels can cause deep neural networks to memorize corrupted training data, leaving a substantial generalization gap despite standard regularization. The survey formalizes this problem and reviews robust-training research and its methodological organization.

  • Deep neural networks can fit corrupted labels, causing poor generalization on unseen data.
  • Standard regularization does not completely overcome noisy-label overfitting, and clean- versus noisy-data test accuracy remains substantially different.
  • The survey tracks recent studies and groups robust deep-learning methods into five categories according to methodological differences.
  • Robust training addresses noisy labels within supervised learning, whereas adversarial input perturbations and missing inputs are outside this survey’s scope.
  • The supervised-learning formulation trains a DNN mapping inputs to labels by minimizing empirical risk, then applies the procedure to noisy training data.

B. Taxonomy of Label Noise

Label noise is classified by how corruption depends on true labels and input features. The survey reviews conventional strategies while emphasizing that deep networks require approaches designed for their high expressive power.

  • Most existing algorithms address instance-independent noise, while instance-dependent noise makes corruption probability depend on both features and class labels.
  • Under symmetric noise, each true label is retained with probability 1 − τ and flipped uniformly to other classes with probability τ/(c−1).
  • Conventional approaches include data cleaning, surrogate losses, probabilistic methods, and model-based methods.
  • Deep learning is more susceptible to label noise than traditional machine learning because of its high expressive power.

D. Regression with Noisy Labels

The survey focuses primarily on classification, while noting that robust classification approaches can often extend to regression with simple modifications. It also organizes robust deep-learning research into methodological categories.

  • Regression models relationships between features and continuous target variables, unlike classification, which uses discrete label spaces.
  • Many robust classification approaches can be extended to regression with simple modifications, so the survey focuses on classification where most methods are defined.
  • The survey identifies five research directions for making supervised learning more robust to label noise.
  • These directions include noise-adaptation architectures, reduced overfitting, robust losses, loss adjustment, and sample selection.
  • The survey discusses combining sample selection with orthogonal methods or semi-supervised learning toward state-of-the-art performance.

A. Robust Architecture

Robust architectures model label-transition behavior within or alongside a DNN to improve learning under noisy labels. Their main limitation is inaccurate transition estimation, especially with only noisy data or high noise.

  • 1) Noise Adaptation Layer:: Noise-adaptation architectures modify DNN outputs using an estimated label-transition probability.They add a noise-adaptation layer above the softmax output and remove it during test prediction.
  • 1) Noise Adaptation Layer:: Examples differ in how they estimate or regularize the transition matrix, including confusion-matrix initialization, identity initialization, dropout, EM, and human constraints.The c-model addresses instance-dependent noise, while NLNN iterates between transition estimation and DNN training.
  • 1) Noise Adaptation Layer:: A common drawback is that these architectures treat all examples equally and cannot identify false-labeled examples.Transition-matrix estimation error generally becomes large when only noisy data are available or the noise rate is high.
  • 2) Dedicated Architecture:: These methods include noise-adaptation layers and dedicated architectures designed for complex or realistic noise.Dedicated designs aim to improve estimation of label-transition probabilities beyond label-dependent noise.
  • Regularization:: Explicit regularization can improve robustness but often introduces sensitive model-dependent hyperparameters or requires deeper architectures.Performance gains are possible when the regularization is optimally tuned.

2) Implicit Regularization:

Implicit regularization improves robustness by altering training stochasticity or targets without reducing representational capacity. Robust losses provide theoretical support but can be limited by task complexity and slower convergence.

  • 2) Implicit Regularization:: Implicit regularization uses stochastic training effects, such as data augmentation and mini-batch stochastic gradient descent.Unlike explicit regularization, it operates through the training procedure or data.
  • 2) Implicit Regularization:: Adversarial training handles perturbed inputs, while label smoothing mixes one-hot labels with a uniform distribution to reduce overfitting.For label smoothing, α controls the smoothing degree and the target becomes (1 − α)·[ỹ = i] + α/c.
  • 2) Implicit Regularization:: Mixup creates virtual examples by linearly interpolating two noisy inputs and their labels.The balance parameter λ ∈ [0, 1] controls the interpolation between examples.
  • 2) Implicit Regularization:: Implicit regularization preserves representational capacity and avoids sensitive model-dependent hyperparameters, but extended feature or label spaces slow convergence.These methods improve generalization without reducing capacity.
  • C. Robust Loss Function: Robust-loss research designs modified losses whose empirical risk on noisy data can approach the Bayes-optimal classifier under stated conditions.The theoretical framework uses Lipschitz and classification-calibration assumptions together with DNN universal approximation.
  • C. Robust Loss Function: MAE, GCE, bi-tempered loss, SCE, curriculum loss, and APL provide different approaches to noise-tolerant classification.MAE can generalize better than CCE under noisy labels, while GCE combines advantages associated with MAE and CCE.
  • C. Robust Loss Function: Robust-loss methods perform well mainly in simple settings, such as easy learning or few classes, and modified losses increase convergence time.These limitations constrain their effectiveness on more complicated classification problems.

D. Loss Adjustment

Loss adjustment reduces noisy-label effects by modifying every example’s loss through correction, reweighting, or label refurbishment. Its effectiveness depends on accurate transition estimation or suitable weighting decisions.

  • D. Loss Adjustment: Loss adjustment changes each example’s loss before updating the DNN through correction, reweighting, or label refurbishment.The three groups differ in whether they estimate transitions, assign example weights, or replace labels.
  • D. Loss Adjustment: Loss adjustment accumulates false-correction error, particularly with many classes or mislabeled examples.Correction methods also depend heavily on precise transition estimation and often require anchor points or clean validation data.
  • 1) Loss Correction:: Loss correction uses an estimated transition matrix to modify either the loss or the DNN’s softmax outputs.Backward correction uses an inverse transition matrix, whereas forward correction multiplies softmax outputs by the estimated transition probability.
  • 1) Loss Correction:: Clean validation data or anchor points can provide additional information for more accurate transition-matrix estimation.Other methods seek to infer or factorize the matrix without directly relying on anchor points.
  • 2) Loss Reweighting:: Loss reweighting assigns smaller weights to suspected false-label examples and larger weights to examples believed to have true labels.Examples with smaller weights contribute less to DNN learning.
  • 2) Loss Reweighting:: Reweighting methods may estimate distribution ratios or use prediction uncertainty and variance to determine example weights.Active bias emphasizes examples with uncertain and inconsistent label predictions.
  • 2) Loss Reweighting:: Reweighting approaches require manually specified weighting functions and additional hyperparameters that vary with noise type and training data.This variation makes practical application difficult.

3) Label Refurbishment:

Label refurbishment replaces noisy targets with combinations of observed labels and model predictions, while later methods refine confidence estimation or automate updates through meta learning. These methods work best when corrected labels are reliable and clean validation data is available.

  • 3) Label Refurbishment:: Label refurbishment forms a new target by combining the noisy label with the DNN’s current prediction.The loss is backpropagated using the refurbished label rather than the original noisy label.
  • 3) Label Refurbishment:: The refurbishment equation uses α ∈ [0, 1] as the confidence assigned to the noisy label.A higher confidence gives greater weight to the observed noisy label in the convex combination.
  • 3) Label Refurbishment:: Bootstrapping estimates label confidence through cross-validation, while dynamic bootstrapping adjusts confidence for individual examples.D2L uses local intrinsic dimensionality to evaluate confidence and limit later representation expansion.
  • 3) Label Refurbishment:: SELFIE corrects only refurbishable examples whose label predictions remain consistent over time.This selective correction is intended to identify examples whose predictions correspond to their true labels with high probability.
  • 3) Label Refurbishment:: Refurbishment works well when confusing classes are uncommon but can overfit to wrongly refurbished labels when confusion is substantial.Unlike correction and reweighting, these methods explicitly replace noisy labels or their combinations.
  • Meta Learning:: Meta learning automates reweighting and refurbishment rules to adapt across data and label-noise types.Meta-DNNs or meta models can use small clean validation datasets to guide target-model updates.
  • Meta Learning:: Unbiased clean validation data is essential for meta learning’s auxiliary objective but may be unavailable in real-world datasets.This requirement limits the practical availability of the approach.

E. Sample Selection

Sample selection identifies likely clean examples and updates the network using them, often exploiting DNN memorization and small-loss behavior. Multi-network, multi-round, and hybrid variants improve selection reliability or use more data, but add computational or error-accumulation costs.

  • Sample selection updates the DNN only with identified clean examples, excluding mini-batch instances likely to have false labels.
  • DNNs tend to learn simple patterns before noisy patterns, so small-loss examples are commonly treated as likely clean.
  • Hybrid Approach: Sample selection may accumulate errors from incorrect choices and discard useful examples, motivating combinations with loss correction or semi-supervised learning.
  • Multi-network Learning: Multi-network methods guide selection through disagreement, mentor supervision, or peer exchange to reduce confirmation bias.
  • Multi-round Learning: Multi-round learning repeatedly refines the clean set, while iterative methods alternate selection and retraining or expand an initial safe set.
  • Multi-round Learning: Selected clean sets can become expanded and purified through iterative refinement, but training cost increases linearly with the number of rounds.

3) Hybrid Approach:

Hybrid robust-training methods combine sample selection with semi-supervised learning, loss correction, or robust losses to exploit more noisy examples. The survey compares methods across six properties and finds that no existing method supports them all.

  • 3) Hybrid Approach:: Hybrid methods treat selected examples as labeled clean data and remaining examples as unlabeled before applying semi-supervised learning.
  • 3) Hybrid Approach:: SELF progressively filters false-labeled examples using a semi-supervised approach with a mean-teacher backbone.
  • 3) Hybrid Approach:: SELFIE combines sample selection with loss correction so more training examples contribute to DNN updates.
  • 3) Hybrid Approach:: Combining robust-training techniques significantly improves noise robustness, but introduces additional hyperparameters and inevitably increases computational cost.
  • IV. METHODOLOGICAL COMPARISON: The survey compares 62 robust deep-learning methods using six properties, including flexibility, no pre-training, full exploration, no supervision, heavy noise, and complex noise.
  • IV. METHODOLOGICAL COMPARISON: No existing method supports all six properties, and support patterns differ substantially across methodological categories.

V. NOISE RATE ESTIMATION

Noise-rate estimation supports practical robust training, but its accuracy has not been analyzed adequately despite its importance for loss adjustment and sample selection. The survey reviews transition-matrix, GMM, and cross-validation approaches.

  • Noise-rate estimates reweight examples for robust classifiers or determine how many examples are selected as clean.
  • A. Noise Transition Matrix: Noise transition matrices represent class-posterior probabilities for noisy and clean data and can be estimated using anchor points.Anchor points have perfectly clean labels, but are typically unknown in real-world data and must be identified theoretically or heuristically.
  • B. Gaussian Mixture Model (GMM): The GMM approach fits two Gaussian components to bimodal training-loss distributions and uses posterior probabilities to identify false-labeled examples.
  • B. Gaussian Mixture Model (GMM): The area under the loss curve preserves separability between true- and false-labeled examples after training-loss signals become less separable during later epochs.
  • C. Cross Validation: Cross-validation estimates noise rates from test accuracy, but commonly requires clean validation data that is difficult to acquire in real-world applications.An alternative uses two randomly divided noisy training datasets under the assumption that they share the same noise transition matrix.

VI. EXPERIMENTAL DESIGN

Experimental comparisons of robust training methods commonly use image-classification benchmarks and evaluate performance on clean data or the quality of selected or refurbished labels.

  • Robust training methods are commonly evaluated on image-classification benchmark datasets.
  • Datasets: Seven widely used clean datasets include MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, SVHN, ImageNet, and Tiny-ImageNet.
  • Datasets: Six widely used real-world noisy datasets contain mislabeled examples annotated by non-experts, including ANIMAL-10N, CIFAR-10N, CIFAR-100N, Food-101N, and Clothing1M.
  • Evaluation Metrics: Test accuracy on unbiased clean examples is the typical robustness metric because overfitting false labels degrades generalization.
  • Evaluation Metrics: Validation accuracy can replace test accuracy when test data are unavailable.
  • Evaluation Metrics: Sample-selection methods can be assessed with label precision and recall, while label-refurbishment methods can use correction error.Precision and recall measure the quality of examples selected as true-labeled; correction error measures incorrectly refurbished labels.

VII. FUTURE RESEARCH DIRECTIONS

The survey identifies future challenges involving realistic noise assumptions, multi-label data, and broader applicability of robust learning methods. Instance-dependent noise and noisy multi-label learning remain especially underexplored.

  • Most existing methods assume instance-independent noise, while instance-dependent noise has received comparatively little research attention.Some recent studies focus only on binary classification or restricted small-scale models such as logistic regression.
  • Sample-selection methods may work under instance-dependent noise but can worsen considerably because true- and false-labeled loss distributions overlap.
  • Most robust methods address single-label multi-class classification rather than multi-label data with multiple true class labels.
  • Multi-label noise is further complicated by class imbalance, which can prevent simple extensions from learning proper correlations among labels.

C. Class Imbalance Data with Label Noise

Future robust-training research must address class imbalance, fairness under noisy protected groups, connections with input perturbation, and computational efficiency. Existing methods and evaluations often rely on restrictive assumptions or omit efficiency analysis.

  • C. Class Imbalance Data with Label Noise: Class-imbalance assumptions are strong because real-world training data often contain a few classes accounting for most examples.
  • C. Class Imbalance Data with Label Noise: Learning-dynamics methods may discard minority-class examples because models reach small losses on majority classes earlier.
  • D. Robust and Fair Training: Fairness criteria cannot be directly applied when protected-group labels are noisy, and joint robust-fair training has mostly been studied in binary classification.
  • E. Connection with Input Perturbation: Adversarial training has been reported to make DNNs robust to label noise, motivating connections between input perturbation and noisy-label learning.
  • F. Efficient Learning Pipeline: Multiple DNNs or repeated training rounds can significantly reduce robust-learning efficiency, while most surveyed work did not report efficiency or time complexity.
  • DNNs overfit false labels despite conventional regularization, which significantly decreases generalization performance.
  • The survey finds no ideal method supporting all six required properties, with supported properties varying by methodological category.
Loading 2007.08199v7…