Learning from Noisy Labels with Deep Neural Networks: A Survey

• Deep neural networks can fit corrupted labels, causing poor generalization on unseen data.Source: §I · p. 1
• Standard regularization does not completely overcome noisy-label overfitting, and clean- versus noisy-data test accuracy remains substantially different.Source: §I · p. 1, Figure 1 · p. 1
• The survey tracks recent studies and groups robust deep-learning methods into five categories according to methodological differences.Source: §I · p. 1, Figure 2 · p. 2, A. Related Surveys · p. 2
• Robust training addresses noisy labels within supervised learning, whereas adversarial input perturbations and missing inputs are outside this survey’s scope.Source: B. Survey Scope · p. 2
• The supervised-learning formulation trains a DNN mapping inputs to labels by minimizing empirical risk, then applies the procedure to noisy training data.Source: A. Supervised Learning with Noisy Labels · p. 2

• Most existing algorithms address instance-independent noise, while instance-dependent noise makes corruption probability depend on both features and class labels.Source: B. Taxonomy of Label Noise · p. 3
• Under symmetric noise, each true label is retained with probability 1 − τ and flipped uniformly to other classes with probability τ/(c−1).Source: 1) Instance-independent Label Noise · p. 3
• Conventional approaches include data cleaning, surrogate losses, probabilistic methods, and model-based methods.Source: C. Non-deep Learning Approaches · p. 3
• Deep learning is more susceptible to label noise than traditional machine learning because of its high expressive power.Source: C. Non-deep Learning Approaches · p. 4

• Regression models relationships between features and continuous target variables, unlike classification, which uses discrete label spaces.Source: D. Regression with Noisy Labels · p. 4
• Many robust classification approaches can be extended to regression with simple modifications, so the survey focuses on classification where most methods are defined.Source: D. Regression with Noisy Labels · p. 4
• The survey identifies five research directions for making supervised learning more robust to label noise.Source: §III · p. 4
• These directions include noise-adaptation architectures, reduced overfitting, robust losses, loss adjustment, and sample selection.Source: • (§III-A) Robust architecture · p. 4, • (§III-B) Robust regularization · p. 4, • (§III-D) Loss adjustment · p. 4
• The survey discusses combining sample selection with orthogonal methods or semi-supervised learning toward state-of-the-art performance.Source: • (§III-E) Sample selection · p. 4

1) Noise Adaptation Layer:

• Noise-adaptation architectures modify DNN outputs using an estimated label-transition probability.They add a noise-adaptation layer above the softmax output and remove it during test prediction.Source: A. Robust Architecture · p. 4, 1) Noise Adaptation Layer · p. 4
• Examples differ in how they estimate or regularize the transition matrix, including confusion-matrix initialization, identity initialization, dropout, EM, and human constraints.The c-model addresses instance-dependent noise, while NLNN iterates between transition estimation and DNN training.Source: 1) Noise Adaptation Layer · p. 5, 2) Dedicated Architecture · p. 5
• A common drawback is that these architectures treat all examples equally and cannot identify false-labeled examples.Transition-matrix estimation error generally becomes large when only noisy data are available or the noise rate is high.Source: 1) Noise Adaptation Layer · p. 5

2) Dedicated Architecture:

• These methods include noise-adaptation layers and dedicated architectures designed for complex or realistic noise.Dedicated designs aim to improve estimation of label-transition probabilities beyond label-dependent noise.Source: 2) Dedicated Architecture · p. 5

Regularization:

• Explicit regularization can improve robustness but often introduces sensitive model-dependent hyperparameters or requires deeper architectures.Performance gains are possible when the regularization is optimally tuned.Source: B. Robust Regularization · p. 5, 1) Explicit Regularization · p. 5

2) Implicit Regularization:

• Implicit regularization uses stochastic training effects, such as data augmentation and mini-batch stochastic gradient descent.Unlike explicit regularization, it operates through the training procedure or data.Source: 2) Implicit Regularization · p. 5
• Adversarial training handles perturbed inputs, while label smoothing mixes one-hot labels with a uniform distribution to reduce overfitting.For label smoothing, α controls the smoothing degree and the target becomes (1 − α)·[ỹ = i] + α/c.Source: 2) Implicit Regularization · pp. 5–6
• Mixup creates virtual examples by linearly interpolating two noisy inputs and their labels.The balance parameter λ ∈ [0, 1] controls the interpolation between examples.Source: 2) Implicit Regularization · p. 6
• Implicit regularization preserves representational capacity and avoids sensitive model-dependent hyperparameters, but extended feature or label spaces slow convergence.These methods improve generalization without reducing capacity.Source: 2) Implicit Regularization · p. 6

C. Robust Loss Function

• Robust-loss research designs modified losses whose empirical risk on noisy data can approach the Bayes-optimal classifier under stated conditions.The theoretical framework uses Lipschitz and classification-calibration assumptions together with DNN universal approximation.Source: C. Robust Loss Function · p. 6
• MAE, GCE, bi-tempered loss, SCE, curriculum loss, and APL provide different approaches to noise-tolerant classification.MAE can generalize better than CCE under noisy labels, while GCE combines advantages associated with MAE and CCE.Source: C. Robust Loss Function · p. 6
• Robust-loss methods perform well mainly in simple settings, such as easy learning or few classes, and modified losses increase convergence time.These limitations constrain their effectiveness on more complicated classification problems.Source: C. Robust Loss Function · p. 6

D. Loss Adjustment

• Loss adjustment changes each example’s loss before updating the DNN through correction, reweighting, or label refurbishment.The three groups differ in whether they estimate transitions, assign example weights, or replace labels.Source: D. Loss Adjustment · p. 6
• Loss adjustment accumulates false-correction error, particularly with many classes or mislabeled examples.Correction methods also depend heavily on precise transition estimation and often require anchor points or clean validation data.Source: D. Loss Adjustment · p. 7, 1) Loss Correction · p. 7

1) Loss Correction:

• Loss correction uses an estimated transition matrix to modify either the loss or the DNN’s softmax outputs.Backward correction uses an inverse transition matrix, whereas forward correction multiplies softmax outputs by the estimated transition probability.Source: 1) Loss Correction · p. 7
• Clean validation data or anchor points can provide additional information for more accurate transition-matrix estimation.Other methods seek to infer or factorize the matrix without directly relying on anchor points.Source: 1) Loss Correction · p. 7

2) Loss Reweighting:

• Loss reweighting assigns smaller weights to suspected false-label examples and larger weights to examples believed to have true labels.Examples with smaller weights contribute less to DNN learning.Source: 2) Loss Reweighting · p. 7
• Reweighting methods may estimate distribution ratios or use prediction uncertainty and variance to determine example weights.Active bias emphasizes examples with uncertain and inconsistent label predictions.Source: 2) Loss Reweighting · p. 7
• Reweighting approaches require manually specified weighting functions and additional hyperparameters that vary with noise type and training data.This variation makes practical application difficult.Source: 2) Loss Reweighting · p. 7

3) Label Refurbishment:

• Label refurbishment forms a new target by combining the noisy label with the DNN’s current prediction.The loss is backpropagated using the refurbished label rather than the original noisy label.Source: 3) Label Refurbishment · p. 7
• The refurbishment equation uses α ∈ [0, 1] as the confidence assigned to the noisy label.A higher confidence gives greater weight to the observed noisy label in the convex combination.Source: 3) Label Refurbishment · p. 7
• Bootstrapping estimates label confidence through cross-validation, while dynamic bootstrapping adjusts confidence for individual examples.D2L uses local intrinsic dimensionality to evaluate confidence and limit later representation expansion.Source: 3) Label Refurbishment · p. 7
• SELFIE corrects only refurbishable examples whose label predictions remain consistent over time.This selective correction is intended to identify examples whose predictions correspond to their true labels with high probability.Source: 3) Label Refurbishment · p. 8
• Refurbishment works well when confusing classes are uncommon but can overfit to wrongly refurbished labels when confusion is substantial.Unlike correction and reweighting, these methods explicitly replace noisy labels or their combinations.Source: 3) Label Refurbishment · p. 8

Meta Learning:

• Meta learning automates reweighting and refurbishment rules to adapt across data and label-noise types.Meta-DNNs or meta models can use small clean validation datasets to guide target-model updates.Source: 4) Meta Learning · p. 8
• Unbiased clean validation data is essential for meta learning’s auxiliary objective but may be unavailable in real-world datasets.This requirement limits the practical availability of the approach.Source: 4) Meta Learning · p. 8

• Sample selection updates the DNN only with identified clean examples, excluding mini-batch instances likely to have false labels.Source: E. Sample Selection · p. 8
• DNNs tend to learn simple patterns before noisy patterns, so small-loss examples are commonly treated as likely clean.Source: E. Sample Selection · p. 8

Hybrid Approach

• Sample selection may accumulate errors from incorrect choices and discard useful examples, motivating combinations with loss correction or semi-supervised learning.Source: E. Sample Selection · p. 8

Multi-network Learning

• Multi-network methods guide selection through disagreement, mentor supervision, or peer exchange to reduce confirmation bias.Source: 1) Multi-network Learning · pp. 8–9

Multi-round Learning

• Multi-round learning repeatedly refines the clean set, while iterative methods alternate selection and retraining or expand an initial safe set.Source: 2) Multi-round Learning · p. 9
• Selected clean sets can become expanded and purified through iterative refinement, but training cost increases linearly with the number of rounds.Source: 2) Multi-round Learning · p. 9

3) Hybrid Approach:

• Hybrid methods treat selected examples as labeled clean data and remaining examples as unlabeled before applying semi-supervised learning.Source: 3) Hybrid Approach · p. 9
• SELF progressively filters false-labeled examples using a semi-supervised approach with a mean-teacher backbone.Source: 3) Hybrid Approach · p. 9
• SELFIE combines sample selection with loss correction so more training examples contribute to DNN updates.Source: 3) Hybrid Approach · p. 11
• Combining robust-training techniques significantly improves noise robustness, but introduces additional hyperparameters and inevitably increases computational cost.Source: 3) Hybrid Approach · p. 11

IV. METHODOLOGICAL COMPARISON

• The survey compares 62 robust deep-learning methods using six properties, including flexibility, no pre-training, full exploration, no supervision, heavy noise, and complex noise.Source: §IV · p. 11
• No existing method supports all six properties, and support patterns differ substantially across methodological categories.Source: §IV · p. 12

• Noise-rate estimates reweight examples for robust classifiers or determine how many examples are selected as clean.Source: §V · p. 12

A. Noise Transition Matrix

• Noise transition matrices represent class-posterior probabilities for noisy and clean data and can be estimated using anchor points.Anchor points have perfectly clean labels, but are typically unknown in real-world data and must be identified theoretically or heuristically.Source: A. Noise Transition Matrix · p. 12

B. Gaussian Mixture Model (GMM)

• The GMM approach fits two Gaussian components to bimodal training-loss distributions and uses posterior probabilities to identify false-labeled examples.Source: B. Gaussian Mixture Model (GMM) · p. 12, Figure 7 · p. 13
• The area under the loss curve preserves separability between true- and false-labeled examples after training-loss signals become less separable during later epochs.Source: B. Gaussian Mixture Model (GMM) · p. 13

C. Cross Validation

• Cross-validation estimates noise rates from test accuracy, but commonly requires clean validation data that is difficult to acquire in real-world applications.An alternative uses two randomly divided noisy training datasets under the assumption that they share the same noise transition matrix.Source: C. Cross Validation · p. 13

• Robust training methods are commonly evaluated on image-classification benchmark datasets.Source: §VI · p. 13, A. Publicly Available Datasets · p. 13

Datasets

• Seven widely used clean datasets include MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, SVHN, ImageNet, and Tiny-ImageNet.Source: 1) Clean Datasets · p. 13
• Six widely used real-world noisy datasets contain mislabeled examples annotated by non-experts, including ANIMAL-10N, CIFAR-10N, CIFAR-100N, Food-101N, and Clothing1M.Source: 2) Real-world Noisy Datasets · pp. 13–14

Evaluation Metrics

• Test accuracy on unbiased clean examples is the typical robustness metric because overfitting false labels degrades generalization.Source: B. Evaluation Metrics · p. 14
• Validation accuracy can replace test accuracy when test data are unavailable.Source: B. Evaluation Metrics · p. 14
• Sample-selection methods can be assessed with label precision and recall, while label-refurbishment methods can use correction error.Precision and recall measure the quality of examples selected as true-labeled; correction error measures incorrectly refurbished labels.Source: B. Evaluation Metrics · p. 14

• Most existing methods assume instance-independent noise, while instance-dependent noise has received comparatively little research attention.Some recent studies focus only on binary classification or restricted small-scale models such as logistic regression.Source: A. Instance-dependent Label Noise · p. 14
• Sample-selection methods may work under instance-dependent noise but can worsen considerably because true- and false-labeled loss distributions overlap.Source: A. Instance-dependent Label Noise · p. 14
• Most robust methods address single-label multi-class classification rather than multi-label data with multiple true class labels.Source: B. Multi-label Data with Label Noise · p. 14
• Multi-label noise is further complicated by class imbalance, which can prevent simple extensions from learning proper correlations among labels.Source: B. Multi-label Data with Label Noise · pp. 14–15

C. Class Imbalance Data with Label Noise

• Class-imbalance assumptions are strong because real-world training data often contain a few classes accounting for most examples.Source: C. Class Imbalance Data with Label Noise · p. 15
• Learning-dynamics methods may discard minority-class examples because models reach small losses on majority classes earlier.Source: C. Class Imbalance Data with Label Noise · p. 15

D. Robust and Fair Training

• Fairness criteria cannot be directly applied when protected-group labels are noisy, and joint robust-fair training has mostly been studied in binary classification.Source: D. Robust and Fair Training · p. 15

E. Connection with Input Perturbation

• Adversarial training has been reported to make DNNs robust to label noise, motivating connections between input perturbation and noisy-label learning.Source: E. Connection with Input Perturbation · p. 15

F. Efficient Learning Pipeline

• Multiple DNNs or repeated training rounds can significantly reduce robust-learning efficiency, while most surveyed work did not report efficiency or time complexity.Source: F. Efficient Learning Pipeline · p. 15

• DNNs overfit false labels despite conventional regularization, which significantly decreases generalization performance.Source: §VIII · p. 15
• The survey finds no ideal method supporting all six required properties, with supported properties varying by methodological category.Source: §VIII · p. 16