Source-linked AI summary
Error-Bounded Correction of Noisy Labels
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, Chao Chen
TL;DR
Label noise undermines learning, and the use of noisy classifiers to identify trustworthy labels lacks theoretical justification. The paper proves that classifier confidence can indicate label corruption, then derives a likelihood-ratio correction algorithm integrated into neural-network training. The resulting method is reported to outperform many recent methods across datasets and noise patterns, while its guarantees rely on stated noise assumptions and focus on identifying incorrect labels.
Problem
Data-recalibrating methods use classifiers trained on corrupted labels to identify clean data, but lack a theoretical justification for why this works.
Method
The paper proves confidence-based bounds for detecting corrupted labels and uses them in a likelihood-ratio correction algorithm integrated with iterative neural-network training.
Results
The method produces models robust to different noise patterns and outperforms many recently proposed methods on various datasets.
Takeaways & Limitations
The theory explains existing data-recalibrating methods and provides a principled label-correction procedure for noisy-label training.
Takeaways & Limitations
The analysis assumes transition probabilities are independent of the true data distribution and features, and focuses on identifying incorrect rather than correct labels.
Abstract
from arXiv · showhide
To collect large scale annotated data, it is inevitable to introduce label noise, i.e., incorrect class labels. To be robust against label noise, many successful methods rely on the noisy classifiers (i.e., models trained on the noisy training data) to determine whether a label is trustworthy. However, it remains unknown why this heuristic works well in practice. In this paper, we provide the first theoretical explanation for these methods. We prove that the prediction of a noisy classifier can indeed be a good indicator of whether the label of a training data is clean. Based on the theoretical result, we propose a novel algorithm that corrects the labels based on the noisy classifier prediction. The corrected labels are consistent with the true Bayesian optimal classifier with high probability. We incorporate our label correction algorithm into the training of deep neural networks and train models that achieve superior testing performance on multiple public datasets.
1. Introduction
Label noise harms model performance, while existing data-recalibrating methods lack a theoretical explanation and reliable control of individual label decisions. This paper provides a theorem linking noisy-classifier confidence to label corruption and uses it to design a guaranteed label-correction method.
- Motivation: Label noise is common in real-world data and particularly impairs deep networks because they can memorize incorrect labels.Noise may arise from annotator mistakes, automatic annotation, or malicious attacks.
- Related approaches: Existing robust-learning methods adjust noise models, architectures, priors, or losses, but may require strong global assumptions.Data-recalibrating methods instead assess individual examples using their own context.
- Open problem: Data-recalibrating methods lack theoretical justification and face a trade-off between accumulating errors through aggressive selection and slowing progress through conservative selection.A theoretical guarantee would support decisions that are reasonably close to the truth for individual data.
- Main theorem: The main theorem shows that a noisy classifier can identify corrupted labels, with confidence thresholds separating labels likely to be corrupted from those likely to be clean.The paper also empirically shows that the theorem’s bound is tight.
- Method and outcome: The proposed likelihood-ratio label-correction algorithm corrects or preserves labels using noisy-classifier predictions and a threshold, with guaranteed success and high-probability recovery of true labels.Integrated into neural-network training, the method is evaluated across datasets with varied noise patterns and levels and reported to outperform state-of-the-art methods.
2. The Main Theorem: Probing Label Purity Using the Noisy Classifier
The theorem establishes when a noisy classifier’s confidence indicates whether an observed label is corrupted, under the Tsybakov condition. It motivates likelihood-ratio label correction and is supported by synthetic and CIFAR10 experiments showing tight bounds.
- The paper asks how to identify corrupted labels without ground truth when only a classifier trained on corrupted labels is available.
- A noisy classifier with low confidence in an observed label indicates that the label is likely corrupted, providing the theoretical basis for data-recalibrating methods.
- The theorem assumes the true conditional probability satisfies the Tsybakov condition and that the noisy classifier is ϵ-close to the noisy conditional probability.
- The resulting upper bound is controlled by ϵ, becoming tighter as the classifier better approximates the noisy conditional probability.
- The proposed likelihood-ratio correction algorithm corrects or preserves labels using the noisy classifier’s prediction and confidence threshold, with a guaranteed success rate.
- On CIFAR10, the estimated Tsybakov constants were C = 0.23 and λ = 1.04, while the bound was small and nearly linear in ϵ.
3. The Algorithm: Likelihood Ratio Test for Label Correction
The paper uses a likelihood-ratio test on a noisy classifier’s confidence to decide whether to replace or preserve each label, with theoretical guarantees for correction quality. This procedure is integrated into iterative neural-network training as AdaCorr.
- Likelihood-ratio correction: The label-correction algorithm compares the noisy classifier’s confidence in the observed label with its confidence in its predicted label.It computes LR(f, x, ey) = fey(x)/fmx(x).
- Likelihood-ratio correction: If the likelihood ratio falls below threshold δ, the algorithm replaces the observed label with the classifier’s predicted label; otherwise, it preserves the label.The threshold defines a hypothesis test for whether the observed label equals the Bayes-optimal label.
- Theoretical guarantees: The correction theorem guarantees high-probability recovery of the Bayes-optimal label for both flipped and preserved-label cases.The bounds depend on classifier approximation error, threshold approximation error, and the probability that the true label is neither candidate label.
- AdaCorr training: AdaCorr iteratively alternates label correction using the current network with neural-network training on the updated labels.The method uses a burn-in stage, then applies correction during later epochs while training with cross-entropy and retroactive losses.
- AdaCorr training: Experiments use four public datasets, with burn-in lengths of 30 epochs for CIFAR10 and CIFAR100 and 25 epochs for MNIST and ModelNet40.The confidence threshold δ is set slightly below 1.
4. Experiments
The experiments evaluate AdaCorr across standard image and 3D datasets under injected noise and on Clothing 1M with real-world noisy labels. The reported results show stronger performance than several baselines, including on Clothing 1M.
- Datasets: AdaCorr is evaluated on MNIST, CIFAR10, CIFAR100, ModelNet40, and the real-world Clothing 1M dataset.The first four datasets use specified noise transition matrices, while Clothing 1M contains 1M images with real-world noisy labels.
- Baselines: The comparisons include Standard, Forward Correction, Decoupling, Coteaching, MentorNet, and Forgetting.These baselines represent standard training, transition-matrix correction, paired-network methods, curriculum filtering, and related approaches.
- Experimental setup: All methods use the same backbone families and are trained for 180 epochs to ensure convergence across the benchmark datasets.The setup uses preactive ResNet-34 for MNIST, CIFAR10, and CIFAR100, and PointNet for ModelNet40.
- Results: 71.47% accuracy on Clothing 1M exceeds Standard at 68.94%, Forward Correction at 69.84%, and Backward Correction at 69.13%.Other listed baselines did not report results on this dataset.
- Results: On CIFAR10 with 40% uniform noise, AdaCorr continues correcting labels while clean-label test accuracy does not drop, unlike Standard training.The Standard model overfits noisy labels and its test performance degrades catastrophically.
5. Conclusion
The paper concludes that its theoretical analysis supports data-recalibrating methods and motivates a label-correction algorithm for noisy labels. Experiments report robustness across noise patterns and stronger performance than many recent methods.
- Contributions: The paper proves theoretical guarantees for data-recalibrating methods under noisy labels.The conclusion presents these guarantees as the theoretical basis for the proposed approach.
- Contributions: The proposed label-correction algorithm is designed to combat label noise and produce models robust to different noise patterns.The conclusion connects the algorithm to robustness across varied noise settings.
- Empirical findings: Experiments on various datasets show that the method outperforms many recently proposed methods.The conclusion states this as the paper’s empirical outcome without specifying a single benchmark value.
— Supplementary Material —
The supplementary-material passage provides author information rather than scientific content.
- Author information: The supplementary-material passage lists the paper’s authors.It contains names and affiliations are not specified in the passage.
1. Additional (Synthetic) Experiment for Validation of the Bound
The synthetic mixture-of-Gaussians experiment evaluates the Tsybakov-condition bound and the LRT-Correction algorithm under symmetric and asymmetric label noise. The empirical correction results are close to clean labels, with success limited by noise asymmetry.
- Experimental setup: The experiment uses a known mixture-of-Gaussians distribution, noise process, η(x), and noisy classifier eη(x) to evaluate the theoretical bound.The synthetic setting permits direct evaluation of the Tsybakov constants and the bound.
- Tsybakov-condition estimation: The estimated Tsybakov constants are C = 0.58 and λ = 1.27, with R2 = 0.904 and p-value less than 10^-4.The constants are estimated by ordinary least-squares regression of the empirical quantities.
- Bound validation: Figure 1 compares the proportion of incorrect labels with the proposed upper bound under symmetric and asymmetric noise.The figure also visualizes clean, symmetrically corrupted, and asymmetrically corrupted data using t-SNE.
- Bound validation: When f(x) < ∆ with a perfect noisy classifier, the probability that the observed label is correct is zero in the synthetic experiment.This behavior appears for both symmetric and asymmetric corruption in the reported experiment.
- Label correction: The LRT-Correction algorithm produces corrected labels very close to clean labels, with success rate limited by the asymmetry level of the noise pattern.Figure 2 validates the correction-error bound across symmetric and asymmetric noise settings.
2. Proof of Theorem 2
Theorem 2 bounds the probability that a noisy label is correct when a noisy classifier has low confidence in that label. Its proof rewrites this event using the noisy-label posterior and applies the multi-class Tsybakov condition.
- Theorem statement: Theorem 2 assumes that η(x) satisfies the multi-class Tsybakov condition and establishes a bound under that assumption.The theorem is stated for constants C, λ > 0 and t0 ∈ (0, 1].
- Proof strategy: The proof identifies label correctness with the event η_ey(x) ≥ η_sx(x) together with low noisy-classifier confidence f_ey(x) < ∆.This event formulation connects the observed label, the Bayes-optimal class, and the classifier confidence threshold.
3. Proof of Theorem 3
Theorem 3 extends the label-correction guarantee to an estimated threshold, while the proof decomposes error by rejected and accepted labels. The resulting bounds include approximation, threshold-estimation, and classifier-selection error terms.
- Rejected labels: For rejected labels, the error bound is C[O(ϵ)]^λ plus the probability that the selected classes differ from both the maximizer and the observed label.This is the bound stated for the baseline threshold setting.
- Proof decomposition: The proof analyzes separately the cases in which the observed label is rejected and accepted by LRT-Correction.Each case is decomposed into terms corresponding to different sources of correction error.
- Proof limitation: The proof states that one residual term cannot be handled properly by the algorithm and is left as a future research problem.This limitation is identified for the second term in the accepted-label analysis.
- Binary specialization: In the binary case, the analysis chooses δ = (1 − |τ10 − τ01|)/(1 + |τ10 − τ01|) under an additional Tsybakov-related error condition.The theorem also records a specialized parameter choice for binary classification.
- Estimated threshold: The theorem introduces an estimated threshold with error ξ, yielding bounds of C[O(max(ϵ, ξ))]^λ plus the classifier-selection error term.The threshold-estimation error affects both rejected- and accepted-label guarantees.