A Survey of Label-noise Representation Learning: Past, Present and Future

Bo Han; Quanming Yao; Tongliang Liu; Gang Niu; Ivor W. Tsang; James T. Kwok; Masashi Sugiyama

Source-linked AI summary

A Survey of Label-noise Representation Learning: Past, Present and Future

Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W. Tsang, James T. Kwok, Masashi Sugiyama

arXiv:2011.04406v2Nov 9, 2020cs.LG

TL;DR

Noisy labels are common in real-world data, yet classical statistical-learning methods may not robustly train deep models under label corruption. This survey formally defines LNRL, studies its behavior theoretically and empirically, organizes methods by data, objective, and model, and derives supported future directions. It concludes that robust LNRL can approach the clean-distribution Bayes optimum under stated theoretical conditions, while practical research remains constrained by realistic noise modeling and datasets.

Problem
Real-world training data often contain corrupted labels, while statistical-learning methods may not robustly train deep models with such noise.
Method
The survey defines LNRL, analyzes noisy-label effects through theory and experiments, and organizes methods by data, objective, and model.
Results
Under the stated objective-function analysis, noise-tolerant learning on corrupted data can converge toward the Bayes-optimal classifier for the clean distribution as hypothesis richness and sample size increase.
Takeaways & Limitations
The survey identifies essential components of robust LNRL and proposes directions including realistic datasets, instance-dependent noise, and learning with other types of noisy data.
Takeaways & Limitations
The commonly used class-conditional noise model only approximates real-world noise, while instance-dependent noise is more realistic but harder to formulate.

Abstract

from arXiv · show

Classical machine learning implicitly assumes that labels of the training data are sampled from a clean distribution, which can be too restrictive for real-world scenarios. However, statistical-learning-based methods may not train deep learning models robustly with these noisy labels. Therefore, it is urgent to design Label-Noise Representation Learning (LNRL) methods for robustly training deep models with noisy labels. To fully understand LNRL, we conduct a survey study. We first clarify a formal definition for LNRL from the perspective of machine learning. Then, via the lens of learning theory and empirical study, we figure out why noisy labels affect deep models' performance. Based on the theoretical guidance, we categorize different LNRL methods into three directions. Under this unified taxonomy, we provide a thorough discussion of the pros and cons of different categories. More importantly, we summarize the essential components of robust LNRL, which can spark new directions. Lastly, we propose possible research directions within LNRL, such as new datasets, instance-dependent LNRL, and adversarial LNRL. We also envision potential directions beyond LNRL, such as learning with feature-noise, preference-noise, domain-noise, similarity-noise, graph-noise and demonstration-noise.

1 INTRODUCTION

Noisy labels are unavoidable in many large-scale and specialized datasets, motivating robust deep learning methods. The survey defines LNRL, explains why noise harms deep models, organizes existing methods, and identifies future directions.

Motivation: Crowdsourcing and expert disagreement introduce noisy labels in large-scale, medical, and speech datasets.Large datasets may rely on imperfect distributed annotation, while medical labels can vary across observers and affect health-related decisions.
Problem Definition: LNRL treats features as intact while labels are sampled from a corrupted distribution.The survey distinguishes this representation-learning setting from earlier statistical-learning approaches focused on noise-tolerant losses and unbiased risk estimation.
Contributions: The survey formally defines LNRL and clarifies its goal and solution space from a machine-learning perspective.The definition is intended to be broad enough to include existing methods while remaining specific about the learning objective.
Contributions: Learning theory and empirical study are used to examine how noisy labels affect deep-model performance and generalization.The survey connects theoretical understanding with reported empirical behavior under noisy labels.
Contributions: Existing LNRL methods are categorized by data, objective, and optimization, with their advantages, disadvantages, and category-level insights discussed.The taxonomy provides a unified framework for reviewing representation-learning methods.
Future Directions: The survey proposes future directions within LNRL and beyond it, including noisy features, preferences, domains, similarities, graphs, and demonstrations.It also positions the survey relative to prior work that focused on statistical learning, image applications, or different methodological categorizations.

2 RELATED LITERATURE

Label-noise learning developed from statistical methods toward deep representation learning as deep models became mainstream. Related work includes risk correction, noise modeling, data reweighting, small-loss selection, regularization, and newer mixture-model and optimization approaches.

Statistical Learning: Early statistical-learning research established robustness guarantees and methods for random, class-conditional, symmetric, and asymmetric label noise.Representative approaches included unbiased risk estimation, kernel modeling, anchor-point noise estimation, importance reweighting, and specialized losses.
Transition to Representation Learning: Since 2015, label-noise research shifted toward representation learning because deep models achieved stronger empirical performance.The survey identifies robust training of deep models with noisy labels as an urgent research goal.
Early LNRL Methods: Seminal deep-learning methods modeled noisy-label distributions, added consistency through bootstrapping, and corrected losses using noise-transition structures.These methods included constrained or nonlinear noise layers, soft and hard bootstrapping, and forward or backward correction.
Core Method Families: Other representation-learning approaches used data reweighting, small-loss selection, regularization, and designed loss functions to reduce the influence of noisy examples.The survey describes small-loss tricks as a way to exploit deep networks’ training behavior.
Recent Directions: Since 2019, newer work has modeled loss distributions, used pre-training, introduced specialized losses, filtered labels progressively, and separated clean from noisy data dynamically.Examples include beta-mixture modeling, barrier hinge loss, partially Huberised loss, self-ensemble filtering, and mixture-based partitioning.

3 OVERVIEW

Section 3 formalizes LNRL as learning with intact features and corrupted labels, then frames its difficulty through data, objectives, and optimization. It uses theory and empirical evidence to motivate robust losses, sufficient hypothesis/data richness, and early-stopped optimization.

3.1 Notation: The formal setup separates corrupted training data from clean test inputs and seeks a hypothesis approximating the Bayes-optimal classifier.The hypothesis is parameterized by θ and selected through an optimization policy over the hypothesis space.
3.2 Problem Definition: LNRL is a machine-learning problem whose experience contains noisy supervised information while deep neural networks directly model the target task.The training features remain intact, but labels are sampled from a corrupted distribution.
3.2 Problem Definition: Noisy labels arise in large-scale crowdsourced image annotation, expert-dependent medical labeling, and subjective speech ratings, making fully correct supervision difficult.These scenarios connect label noise to practical learning tasks across image, healthcare, and speech domains.
3.4 Core Issues: The survey analyzes label-noise learning through three ingredients: data, objective function, and optimization policy.This perspective supports a unified taxonomy of existing LNRL methods.
3.5.2 Perspective of Objective Function: Theoretical guarantees indicate that noise-tolerant corrected losses can approach the Bayes-optimal classifier as hypothesis richness and sample size increase.The approximation error can vanish when a suitable deep network provides a universal hypothesis class and training data increase.
3.5.3 Perspective of Optimization Policy: Gradient descent with early stopping can be robust under stated cluster and step conditions, while keeping final weights close to their initialization.The cited result links robustness to a bounded training duration and limited movement from the initial model.

4 DATA

Data-oriented LNRL methods model or exploit the relationship between clean and noisy labels, chiefly through the noise transition matrix T. These approaches include adaptation layers and loss corrections, whose effectiveness depends strongly on estimating T accurately.

Noise Transition Matrix: The noise transition matrix T models the corruption process from latent clean labels y to observed noisy labels ȳ.Its entries are T_ij = p(ȳ = e_j|y = e_i).
Noise Transition Matrix: Sym-flipping distributes a clean class uniformly among other classes, whereas Pair-flipping redirects it to an adjacent fine-grained class.These structures are commonly used to generate simulated label noise.
Noise Transition Matrix: The relationship p(Ȳ|x) = T · p(Y|x) makes T a bridge between clean and noisy class posteriors.Estimating T remains a key step because noisy posteriors can be estimated from noisy training data.
Adaptation Layer: An adaptation layer inserts a constrained linear or nonlinear noise model between the base network and cross-entropy loss to mimic T.The linear version parameterizes the layer by a probability matrix T, while nonlinear variants model the noise channel separately.
Adaptation Layer: Jointly minimizing cross-entropy over the base model and T may not recover the optimal transition matrix, motivating trace-norm or ridge regularization.The nonlinear EM-based approach also has potential drawbacks including local optimality and scalability.
Loss Correction: Forward correction applies T to network predictions, while backward correction uses T^-1 to correct the loss; both have theoretical guarantees under their assumptions.Backward correction is unbiased, whereas forward correction preserves the clean-loss minimizer but has a weaker guarantee.
Loss Correction: The quality of estimating T directly determines learning performance under loss correction.Because T is generally unknown, robust two-stage training first estimates it and then retrains with the corrected loss.
Loss Correction: Gold Loss Correction estimates T from a small trusted set when severe noise prevents accurate estimation using noisy data alone.The trusted set is produced by experts, while the larger untrusted set may come from crowdsourced workers.

5 OBJECTIVE

Objective-oriented LNRL methods modify the noisy-label training objective through regularization, dynamic reweighting, or related changes to objective subfunctions.

Objective Modification: Objective modification addresses LNRL by changing the empirical objective through explicit or implicit regularization and dynamic weighting of sub-objectives.Implicit regularization operates at the algorithm level while remaining equivalent to modifying the objective.

5.1 Regularization

Regularization-based LNRL modifies the objective to improve generalization and reduce overfitting to noisy labels. The survey covers explicit penalties, smoothness constraints, bootstrapping, Mixup, and gradient-based implicit regularization.

Explicit and Implicit Regularization: Regularization adds a term r to the original objective, aiming to improve generalization and avoid or alleviate overfitting noisy labels.The survey distinguishes explicit regularization from implicit regularization produced by the optimization algorithm.
Explicit Regularization: Auxiliary group-norm regularization encourages sparse use of features from a small number of clean examples while filtering mislabeled and irrelevant data.The operation is described as promoting clean-data contributions and disregarding noisy additional activations.
Explicit Regularization: Virtual adversarial training enforces local smoothness of the conditional label distribution around labeled or unlabeled inputs.The adversarial perturbation maximizes distribution divergence within an l2-norm radius before the smoothness penalty is applied.
Bootstrapping: Bootstrapping combines training labels with current model predictions to generate targets, allowing the learner to disagree with perceptually inconsistent labels.The parameter β balances predictions and original targets; hard bootstrapping uses a MAP estimate of the model prediction.
Mixup: Mixup linearly interpolates pairs of features and labels to construct virtual examples, diluting noise effects and smoothing the data manifold.Its formulation uses x̃ = λx_i + (1 − λ)x_j and ỹ = λy_i + (1 − λ)y_j.
Implicit Regularization: SIGUA uses gradient descent on good data and learning-rate-reduced gradient ascent on bad data to counter progressive fitting of noisy labels.The survey contrasts this data-dependent procedure with parameter shrinkage, which acts on all data.

5.2 Reweighting

Reweighting-based LNRL assigns learned weights to sample-specific objective terms so that clean or more reliable examples influence training more strongly than corrupted ones.

Reweighting: Reweighting modifies the objective by assigning different learned weights to sub-objectives associated with individual training samples.Larger weights bias optimization toward sub-objectives that better overcome label noise.
Importance Reweighting: Importance reweighting treats noisy training data as a source domain and clean test data as a target domain, rewriting clean risk using noisy samples.Its weights can be learned from a transition matrix or a small clean dataset under an instance-independent noise assumption.
Probabilistic Reweighting: Reweighted probabilistic models infer latent sample weights that down-weight corrupted labels and up-weight clean labels.Prior choices on the weights trade off extremely low-likelihood terms, and include Beta, scaled Dirichlet, and Gamma priors.
Mixture-Based Reweighting: A two-component beta mixture model dynamically weights samples using posterior probabilities of clean and noisy components without discarding noisy samples.The model uses EM to estimate mixture parameters and sets each sample’s weight from its component posterior.
Meta-Reweighting: Meta-Weight-Net learns an explicit loss-to-weight mapping with a one-hidden-layer MLP using a small clean and balanced meta-dataset.Classifier and weighting-network parameters are updated iteratively with SGD and meta-learning.

5.3 Redesigning

Redesigning methods address label noise by replacing or augmenting the training loss with mechanisms such as robustness penalties, curriculum selection, abstention, consistency, label filtering, or representation-dynamics monitoring. These approaches derive from distinct principles and target different ways of reducing the influence of corrupted examples.

Loss redesign: Redesigning replaces the conventional loss with a specialized format motivated by different observations and principles.The survey presents redesigning as a broad family of diverse loss-based approaches for different scenarios.
Gradient-based redesign: Partially Huberized loss limits over-confident descent steps by applying gradient clipping in noisy-label settings.Its design is explicitly motivated by controlling gradients under label noise.
Robust losses: The generalized cross-entropy loss ℓq combines MAE’s noise robustness with CCE’s implicit weighting and works under closed-set and open-set noise.ℓq reduces to CCE as q approaches 0 and becomes MAE when q = 1; its truncated variant discards additional large-loss samples.
Abstention: Abstention-based DAC adds an abstention output, with its penalty controlling whether the classifier abstains and enabling data cleaning under structured and unstructured noise.Large α drives abstention probability toward zero, whereas small α may cause the classifier to abstain broadly; an auto-tuning procedure selects α.
Curriculum losses: Curriculum loss provides a tighter upper bound of the 0-1 loss and adaptively selects samples for stagewise training.Noise-pruned curriculum loss adapts this construction to deep learning models.
Label ensemble: Self-ensembling and SELF use predictions across training realizations or epochs to form consensus labels and progressively filter likely noisy labels.SELF combines a Mean Teacher model ensemble with a prediction ensemble over multiple epochs.
Representation dynamics: LID-based learning monitors representation-subspace dimensionality because clean-label training yields decreasing LID and increasing test accuracy, whereas noisy-label training reverses both trends after several epochs.The LID score uses nearest-neighbor distances and oversees deep-network training dynamics.

6 OPTIMIZATION POLICY

Optimization-policy methods exploit deep networks’ tendency to learn clean patterns before memorizing noisy ones. The section develops early stopping, small-loss selection, self-training, meta-reweighting, and co-training strategies, including methods that preserve network diversity or combine filtering with semi-supervised learning.

Optimization policy: Optimization-policy methods change training procedures such as early stopping to exploit memorization effects and reduce overfitting to noisy labels.Small-loss tricks provide another way to use this phenomenon by updating parameters only with selected examples.
6.1 Memorization Effects: Deep networks tend to fit easy clean patterns before gradually overfitting hard noisy patterns.This memorization behavior motivates optimization policies based on training dynamics.
6.1 Memorization Effects: Small-loss methods treat low-loss examples as clean and backpropagate only a selected 1 − τ fraction, where τ is the noise rate.They construct a restricted loss by sorting losses from small to large and retaining the smallest-loss examples.
6.2 Self-training: MentorNet learns a data-driven curriculum for StudentNet, selecting low-loss examples through a robust weighting rule rather than relying only on a predefined curriculum.When λ2 = 0, samples with loss below λ1 are selected; otherwise, samples with loss above λ1 + λ2 are excluded.
6.2 Self-training: Meta-learning reweights training examples according to gradient directions so that their weighted training update improves performance on an unbiased clean validation set.The procedure uses noisy training gradients, validation loss, and second-order gradients to update example weights dynamically.
6.3 Co-training: Co-teaching trains two networks that exchange small-loss examples, reducing peer-filtered error flow, although convergence can eventually make it functionally resemble MentorNet.Co-teaching+ adds disagreement-based updates to keep the networks diverged before applying cross-updates.
6.3.2 Beyond Co-teaching: DivideMix partitions data with a Gaussian Mixture Model, then applies co-refinement to labeled data and co-guessing to unlabeled data within a MixMatch-style procedure.INCV instead uses iterative cross-validation to select a lower-noise subset and removes large-loss samples before Co-teaching.

7 FUTURE WORKS

The survey proposes future work both within LNRL and beyond label noise, emphasizing realistic datasets, instance-dependent and adversarial noise, and other corrupted data modalities.

7.1 Build up New Datasets: Real-noise benchmark datasets are needed beyond simulated MNIST and CIFAR-10, spanning images, language, speech, and sensor data.Clothing1M and web-label noise provide realistic examples, but existing benchmarks remain concentrated on image classification.
7.2 Instance-dependent LNRL: Instance-dependent noise models label corruption probabilities as functions of both class and features, better reflecting ambiguous or poor-quality instances.However, the model is difficult to formulate and is unidentifiable without additional assumptions or information.
7.3 Adversarial LNRL: Adversarial LNRL considers adversarially perturbed features together with noisy labels, extending the usual setting of adversarial training.MART separates misclassified from correctly classified examples, while FAT uses early-stopped PGD and min-min optimization.
7.4 Beyond Labels: Noisy Data: Beyond LNRL, the survey identifies feature, preference, domain, similarity, graph, and demonstration noise as broader noisy-data learning directions.These settings include wild domain adaptation, robust graph learning, and imitation learning from diverse-quality demonstrations.
7.4 Beyond Labels: Noisy Data: Graph-noise research must address corrupted nodes, edges, labels, or features because LNRL methods do not directly transfer to discrete and combinatorial graph optimization.Cross-Graph and GNN pre-training are identified as preliminary approaches.
7.4 Beyond Labels: Noisy Data: Learning from diverse-quality demonstrations becomes difficult without expert-provided quality information, motivating probabilistic quality modeling such as VILD.VILD estimates demonstration quality with a reward function and uses variational methods and importance sampling for efficiency.

8 CONCLUSIONS

The survey reviews LNRL history, defines the problem formally, analyzes deep-network behavior under label noise, and organizes methods through a unified taxonomy. It also identifies future directions within LNRL and broader data-noise representation learning.

8 CONCLUSIONS: The survey formally defines LNRL and studies deep-network mechanisms under label noise using representation-learning theory and empirical experiments.It organizes methods by data, objective, and model perspectives.
8 CONCLUSIONS: The survey discusses advantages and disadvantages across LNRL categories and summarizes essential components of robust LNRL.These components are intended to inform new research directions.
8 CONCLUSIONS: Four future directions include new datasets, instance-dependent LNRL, adversarial LNRL, and learning with preference-, domain-, similarity-, graph-, and demonstration-noise.The final direction extends beyond label noise toward a general data-noise representation-learning framework.

9 APPENDIX 1: RELATED LITERATURE

Label-noise learning evolved from theoretically robust statistical methods toward representation learning for large-scale deep models. The literature includes transition modeling, objective modification, regularization, loss design, and memorization-based sample selection.

9.1 Statistical Learning Stage: Statistical label-noise learning established robust learning under random and classification noise through online algorithms, unbiased risk estimation, and related theoretical methods.Its historical development begins with work showing robustness when random noise affects less than half of examples.
9.1 Statistical Learning Stage: Class-conditional noise research developed asymmetric-noise handling, symmetric-noise losses, anchor-point estimation, and importance-reweighted surrogate losses.These methods target explicit noise structures and corrections.
9.1 Statistical Learning Stage: The field shifted toward representation learning because statistical methods focused on small-scale data and did not empirically scale well to deep, non-convex models.Deep models became mainstream across vision, language, speech, and video, increasing demand for robust LNRL methods.
9.2 Emerging Stage: Early LNRL methods modeled noisy-label distributions with adaptation layers or corrected objectives using forward and backward loss correction.Other foundational work introduced consistency-based soft and hard bootstrapping.
9.2 Emerging Stage: From 2018, major LNRL directions included transition-matrix estimation, regularization, loss design, and small-loss methods based on deep-network memorization.Co-teaching uses peer networks to reduce the accumulated error associated with single-network sample-selection bias.
9.2 Emerging Stage: Later work explored trusted-example correction, mixup, robust losses, abstention, gradient clipping, label filtering, mixture models, and curriculum losses.These methods modify supervision or training dynamics to address noisy labels.

SPECTIVES

LNRL methods can be organized by how they use noise information: transition-matrix modeling, objective-function modification, and optimization policies exploiting deep-network training dynamics.

Data Perspective: Transition-matrix methods either add an adaptation layer, correct the loss with an estimated matrix, or use prior knowledge to ease matrix estimation.More accurate transition-matrix estimation is associated with better classification accuracy.
Data Perspective: Data-perspective research also models structured annotation noise, web noise, and distillation with side information.These approaches address noise structure or auxiliary information beyond standard label transformations.
Objective Perspective: Objective-based methods augment losses with explicit or implicit regularization and can reweight per-sample sub-objectives.Examples include minimum entropy regularization, virtual adversarial training, and learned weighting.
Objective Perspective: Other objective-oriented work includes online crowdsourcing, graphical models linking noisy and clean labels, and active-bias training.These methods incorporate annotation processes or auxiliary information into robust learning.
Optimization Perspective: Optimization-policy methods exploit memorization through self-training, co-training, semi-supervised learning, pre-training, and deep nearest-neighbor approaches.These methods alter training dynamics or use overparameterized-model behavior.
Optimization Perspective: Additional optimization strategies change training dynamics through multitask learning, random grouping with attention, and disagreement-based decoupling.Decoupling updates networks using examples on which the networks disagree.

11 APPENDIX 3: EXPERIMENTAL DETAILS

The appendix reports MNIST experimental settings for Figures 1 and 5, including the network architecture, noise conditions, and observed training and validation behavior.

Experimental setting: Experiments for Figures 1 and 5 use MNIST.Figure-specific experimental details are provided in the appendix.
Network structure: The network follows a memorization-effects setup with convolutional, batch-normalization, pooling, fully connected, and softmax layers.Its listed structure includes two Conv-BN-ReLU-MaxPool blocks followed by fully connected layers and a 10-class softmax.
Figure 1: Figure 1 uses MNIST with 35% symmetric noise and compares clean training with forward correction of the original loss.Forward correction produces a corrected loss for comparison with the original loss setting.
Figure 5: Figure 5 evaluates memorization under 0%–80% symmetric noise using the original loss.The training curve is expected to rise gradually, while validation improves initially and then declines toward convergence.