Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates

Yang Liu; Hongyi Guo

Source-linked AI summary

Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates

Yang Liu, Hongyi Guo

arXiv:1910.03231v7Oct 8, 2019cs.LGstat.ML

TL;DR

Noisy-label learning often depends on known or estimated noise rates, which can require extra data and introduce estimation error. The paper introduces peer loss functions within ERM, using peer prediction to avoid explicit noise-rate specification. Under stated conditions, peer-loss ERM can recover an optimal or near-optimal clean-data classifier, supported by extensive experiments.

Problem
Existing noisy-label methods often require known noise rates or additional estimation procedures, which can require extra data and introduce estimation error.
Method
Peer loss functions adapt peer prediction to standard ERM by evaluating classifier predictions with noisy labels and peer samples without specifying or estimating noise rates.
Results
Formal results show that peer-loss ERM can recover an optimal or near-optimal classifier as if trained on clean data, with extensive experiments validating usefulness.
Takeaways & Limitations
Peer loss offers a candidate loss function for training classifiers when reliable noise-rate estimates are unavailable.
Takeaways & Limitations
Peer loss assumes label noise is homogeneous across training instances and does not yet cover instance-based or margin-based noise.

Abstract

from arXiv · show

Learning with noisy labels is a common challenge in supervised learning. Existing approaches often require practitioners to specify noise rates, i.e., a set of parameters controlling the severity of label noises in the problem, and the specifications are either assumed to be given or estimated using additional steps. In this work, we introduce a new family of loss functions that we name as peer loss functions, which enables learning from noisy labels and does not require a priori specification of the noise rates. Peer loss functions work within the standard empirical risk minimization (ERM) framework. We show that, under mild conditions, performing ERM with peer loss functions on the noisy dataset leads to the optimal or a near-optimal classifier as if performing ERM over the clean training data, which we do not have access to. We pair our results with an extensive set of experiments. Peer loss provides a way to simplify model development when facing potentially noisy training labels, and can be promoted as a robust candidate loss function in such situations.

1. Introduction

The paper addresses noisy-label learning when noise rates are unknown, introducing peer loss functions that fit ERM without specifying or estimating those rates. Theory and experiments support peer loss as a practical approach for asymmetric noise.

Motivation: Existing noisy-label methods with theoretical guarantees typically require prior noise rates or additional estimation procedures.Estimating rates may require clean or additional data and can introduce estimation error.
Method: Peer-loss risk on noisy data becomes an affine transformation of clean-data risk, making the loss invariant to label noise during optimization.This property removes the need to estimate noise rates.
Contribution: Peer loss functions adapt to standard ERM and handle asymmetric label noise without prior knowledge or estimation of noise rates.The method builds on peer prediction by evaluating predictions against noisy labels and peer samples.
Theory: Formal results show that ERM with peer loss can recover an optimal or near-optimal classifier as if trained on clean data.The paper also provides risk guarantees for peer loss functions.
Experiments: Experiments on 10 UCI benchmarks evaluate peer loss under increasing label-noise rates against several noisy-label methods and a neural-network baseline.The study also reports preliminary generalization to multiclass classification.
Context: The paper positions peer loss as a simpler candidate for training in unknown noisy environments, while related work includes symmetric-loss, correction, information-theoretic, and peer-prediction approaches.Implementation is publicly available, and the full proof and experiment details are referenced in an extended version.

2. Preliminaries

The preliminaries formalize binary classification with homogeneous, feature-independent label noise and review standard risk minimization and peer prediction. Existing unbiased loss correction removes noise in expectation but generally requires known error rates.

Problem Setting: The setting uses binary labels, i.i.d. training samples, and noisy labels generated from true labels under a uniform error model.The class prior is denoted p, and the noisy-data distribution is denoted ˜D.
Problem Setting: The noise model assumes feature-independent errors with e+1 := P(˜Y = −1|Y = +1) and e−1 := P(˜Y = +1|Y = −1).The condition e+1+e−1 < 1 means noisy labels remain positively correlated with true labels.
Risk Minimization: The learner minimizes a surrogate loss over a hypothesis space, while the Bayes classifier minimizes clean 0-1 risk.The paper distinguishes clean classification risk from surrogate risk.
Noise Correction: Unbiased correction losses are constructed so that evaluating predictions against noisy labels recovers the clean loss in expectation.These corrections require the noise error rates e−1 and e+1.
Peer Prediction: Peer prediction evaluates an agent’s report using reports from other agents without observing ground truth, with scoring functions designed to favor truthful reporting.Correlated Agreement uses a Δ matrix to capture stochastic correlations between prediction sources.
Peer Prediction: Correlated Agreement operates in a multitask setting using independent peer tasks, a sign matrix M, and a scoring function S.Its established mechanism motivates the peer-loss construction developed later.

3. Learning with Noisy Labels: a Peer Prediction Approach

The paper reframes noisy-label learning as peer prediction: classifier predictions act as one agent’s reports and noisy labels as reference reports. Proper scoring functions then make the Bayes-optimal classifier the loss minimizer under the stated reporting assumptions.

Approach: Peer prediction scoring functions can be specified so that their maximizer, or corresponding loss minimizer, is the Bayes-optimal classifier.This establishes the conceptual basis for constructing peer loss functions.
Elicitation Formulation: The noisy-label problem is recast by treating classifier predictions and noisy labels as reports from two agents observing the same task features.The noisy-label agent’s observations are modeled through Z(X), while the optimal-classifier agent is represented by Z*(X).
Elicitation Formulation: The peer prediction function S evaluates classifier reports using noisy-label reports as references, as summarized by Figure 1.The associated peer loss changes the reward-maximization view into loss minimization.
Optimality Result: If S induces truthful reporting and the optimal classifier has the correct class prior, minimizing the negative score over the reporting space yields the Bayes-optimal classifier.The theorem relies on the stated reporting-space and prior assumptions.
Scope: The construction provides an anchor for connecting peer-prediction scores to noisy-label learning before specifying a particular peer-loss form or noise-rate requirement.The paper emphasizes that the conceptual connection precedes the concrete loss construction.

4. Peer Loss Function

Peer loss adapts correlated-agreement ideas into ERM losses for noisy-label learning. Under stated conditions, it can recover clean-data minimizers without explicitly specifying noise rates, with extensions for class imbalance and multiple classes.

4. Peer Loss Function: Peer loss is a family of ERM-compatible losses inspired by the correlated agreement mechanism.The construction is intended to induce the minimizer over a hypothesis space under broad conditions.
4.1. Preparing CA for Noisy Learning Problem: Peer samples pair a training example with two independently drawn examples, then use one peer input with the other peer label.This pairing supplies the independent peer term used in the scoring function.
4.2. Peer Loss: The generic peer loss subtracts a peer loss evaluated on independently paired samples from the ordinary loss on the current noisy example.The peer term penalizes excessive agreement with noisy labels.
4.1. Preparing CA for Noisy Learning Problem: When the total binary noise rate is below one, the sign matrix is the identity, so the scoring matrix awards matching labels.This condition is sufficient to specify the simple peer-loss form without individual noise-rate values.
4.3. Property of Peer Loss: In expectation, peer loss under noisy labels is scaled by (1−e−1−e+1) relative to clean-data peer loss, preserving the minimizer.This noise invariance connects noisy-distribution ERM to clean-distribution optimization.
4.4. α-weighted Peer Loss: With balanced classes, peer-loss ERM induces the same minimizer as clean-data 0-1-risk minimization; unequal priors require an α-weighted variant.For unequal priors, the unweighted objective minimizes an unweighted sum of class-conditional errors rather than the prior-weighted true risk.
4.4. α-weighted Peer Loss: The framework generalizes to multi-class problems when the noise matrix yields an identity scoring matrix.The paper reports experiments on multi-class tasks as an extension of the theoretical development.
4.4. α-weighted Peer Loss: The α-weighted formulation converges to the optimal classifier under the stated guarantees, while classification calibration holds under specified conditions.When class priors differ, tuning α on validation data may improve performance, but α* depends on noise-rate knowledge.

5. Experiments

Experiments evaluate peer loss on UCI benchmarks, synthetic decision-boundary tasks, and CIFAR-10, comparing it with established noisy-label methods. Results indicate competitive performance without requiring noise-rate specifications and sharper boundaries under noise.

UCI benchmarks: The study evaluates peer loss with a two-layer ReLU MLP on 10 UCI benchmarks under increasing label-noise rates.Comparisons include surrogate loss with true noise rates, symmetric loss, DMI, C-SVM, PAM, and a neural-network baseline.
Training dynamics: Figure 2 tracks test-set accuracy during training on Splice with e−1 = 0.4 and e+1 = 0.4.The accompanying discussion states that peer loss can prevent over-fitting under noisy labels.
UCI benchmarks: Peer loss is competitive across UCI datasets and outperforms surrogate loss with true noise rates on several datasets.It also outperforms symmetric loss and DMI in a number of datasets.
Decision boundaries: In a 2D synthetic experiment, cross entropy produces a sharp clean-data boundary but a less sharp boundary with e+1 = e−1 = 0.2 noise.Peer loss returns sharp boundaries even under a high noise rate.
Multi-class classification: Preliminary CIFAR-10 results use ResNet under universal class noise, flipping each label to any other class with probability ϵ/9.Peer loss is reported as competitive against cross entropy and DMI.

6. Conclusion and Discussion

The paper concludes that peer loss enables noisy-label training without explicit noise-rate knowledge. Its principal scope boundary is the assumption that label noise is homogeneous across training instances.

Conclusion: Peer loss enables classifier training over noisy labels without using explicit knowledge of label noise rates.This is presented as the paper’s concluding contribution.
Discussion: The method assumes label noise is homogeneous across training data instances.Future work is proposed for instance-based and margin-based label noise.

Proof for Lemma 2

The proof of Lemma 2 decomposes peer-loss expectations into separate terms and uses conditional and sample independence assumptions to derive the result.

Proof setup: The proof introduces random variables for the peer samples x_n1 and ỹ_n2.These variables represent the feature and noisy-label components used in peer evaluation.
Expectation decomposition: The first expectation term is analyzed using the law of total probability and conditional independence of Ỹ and X given Y.The proof then adds and subtracts noise-rate-weighted loss terms.
Expectation decomposition: The second expectation term is considered separately before subtracting the two right-hand-side terms.Independence between the peer sample indices n1 and n2 is used in the derivation.

Related work with more details

The paper situates peer loss among methods for learning with noisy labels, emphasizing asymmetric noise and simple ERM-compatible optimization without prior noise-rate specification.

Learning from Noisy Labels: Prior work studies random classification noise and class-conditional noise, including rigorous analyses for asymmetric binary classification.
Symmetric loss: Symmetric loss methods can be robust without specifying noise rates, whereas this work exclusively focuses on asymmetric noise.
More recent works: Related approaches include importance re-weighting, noisy-neural-network training, massive noisy-data learning, robust cross-entropy, and loss correction.
More recent works: Unlike an information-theoretical loss requiring joint-distribution estimation, peer loss targets a simple loss that adapts to existing ERM solutions.
Peer Prediction: Peer loss also builds on peer-prediction research, including strictly proper scoring rules for eliciting truthful reports from self-interested agents.

Illustration of our implementation of peer loss

The paper refers readers to Figure 5 for an illustration of the peer loss implementation.

Figure 5 is cited as an illustration of the peer loss implementation.
The implementation illustration is presented separately from the surrounding theoretical discussion.
The cited passage directs attention to the figure rather than describing implementation details in prose.

Other peer prediction functions

The paper considers quadratic and logarithmic scoring functions alongside a lemma connecting scoring rules to strict truthfulness under stochastic relevance.

The paper introduces quadratic and logarithmic scoring functions as additional peer-prediction examples.
A lemma states that the scoring rules in Examples 1 and 2 induce strict truthfulness when yA and yB are stochastically relevant.
Under stochastic relevance, correlated agreement, quadratic, and logarithmic scoring rules are strictly truthful.

Proof for Theorem 1

The proof establishes optimality by decomposing noisy-label expectations, exploiting conditional independence, and relating truthful peer scoring to the Bayes classifier under stated conditions.

The proof decomposes expected noisy-data scores by classifier predictions and conditional outcomes.
Conditional independence of noisy labels and features given true labels permits dropping disagreement conditions in the expectations.
The proof compares reporting strategies corresponding to constant −1 and +1 predictions while rearranging terms involving the noise parameters.
The derivation uses truthfulness of the scoring rule and excludes classifiers predicting all +1 or all −1 labels.
The argument represents disagreements with the optimal classifier using sets X+_dis and X-_dis and parameters ϵ+ and ϵ−.
The proof concludes that f* is optimal, and minimizing the correlated-agreement loss finds the Bayes optimal classifier when the noisy label and f* are categorical.

Details for Example 1

The example derives how peer loss relates noisy-label expectations to clean risk, showing that under stated conditions its minimizer matches the clean 0-1-risk minimizer. It also extends the result to multi-class settings and establishes calibration and finite-sample guarantees for α-peer losses.

Clean-risk equivalence: Under balanced labels and suitable noise conditions, minimizing peer loss on the clean distribution returns the same minimizer as minimizing true clean 0-1 risk.For the multi-class extension, the corresponding matrix is diagonal when the class-noise conditions hold, including Σ_j e_j < 1.
Asymmetric noise: For binary asymmetric noise, Proposition 1 relates pairwise α-peer-loss differences to the factor 1 − e−1 − e+1.The result is stated for any classifiers f and f′ and identifies the noise-dependent scaling governing comparisons.
Binary derivation: Peer loss uses independent peer samples and conditional label-noise assumptions to express noisy-label expectations through clean-distribution quantities.The derivation uses independence between peer samples and assumes noisy labels are conditionally independent of X given Y.
Statistical guarantees: The α-peer loss is classification calibrated under the stated positivity and calibration conditions, while empirical-risk deviations are bounded with high probability.The finite-sample result gives a uniform deviation bound involving (1 + α)L, the Rademacher complexity of F, and probability 1 − δ.