Source-linked AI summary
Learning with Feature-Dependent Label Noise: A Progressive Approach
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, Chao Chen
TL;DR
Real-world label noise is heterogeneous and feature-dependent, limiting methods built on i.i.d. assumptions or lacking guarantees. The paper introduces PMD noise and a progressive confidence-based correction algorithm, proving convergence to a well-behaved classifier and reporting superior performance across synthetic and real-world evaluations.
Problem
Real-world label noise is heterogeneous and feature-dependent, while existing approaches either rely on unrealistic i.i.d. assumptions or lack theoretical guarantees.
Method
The paper defines PMD noise and alternates high-confidence label correction with model refinement, progressively lowering the correction threshold.
Results
The method outperforms various baselines under feature-dependent synthetic noise and shows superior performance on real-world noisy datasets.
Takeaways & Limitations
PMD noise provides a broader theoretical setting for heterogeneous, feature-dependent label noise, while the proposed data-recalibrating method offers guaranteed convergence to a well-behaved classifier.
Takeaways & Limitations
The analysis is asymptotic and leaves sample complexity for future work.
Abstract
from arXiv · showhide
Label noise is frequently observed in real-world large-scale datasets. The noise is introduced due to a variety of reasons; it is heterogeneous and feature-dependent. Most existing approaches to handling noisy labels fall into two categories: they either assume an ideal feature-independent noise, or remain heuristic without theoretical guarantees. In this paper, we propose to target a new family of feature-dependent label noise, which is much more general than commonly used i.i.d. label noise and encompasses a broad spectrum of noise patterns. Focusing on this general noise family, we propose a progressive label correction algorithm that iteratively corrects labels and refines the model. We provide theoretical guarantees showing that for a wide variety of (unknown) noise patterns, a classifier trained with this strategy converges to be consistent with the Bayes classifier. In experiments, our method outperforms SOTA baselines and is robust to various noise types and levels.
1 INTRODUCTION
The paper addresses heterogeneous, feature-dependent label noise, which limits feature-independent methods and leaves data-recalibrating approaches without theoretical convergence guarantees. It introduces PMD noise and a progressive confidence-based correction method with theoretical and empirical support.
- Incorrect labels are inevitable in large-scale datasets because of ambiguous examples and annotation mistakes, making noise-resilient supervised learning important.
- Feature-independent noise assumptions are unrealistic because real label corruption can depend on example difficulty, visual quality, and occlusion.
- Existing data-recalibrating methods progressively select or correct labels, but lack theoretical explanations for convergence and therefore require careful tuning.
- The paper defines PMD label noise, allowing arbitrary noise levels near the decision boundary while restricting noise farther from it.
- The proposed algorithm alternates high-confidence label correction with model retraining, progressively refining both labels and the classifier.
- The method outperforms other approaches on CIFAR-10/100 synthetic noise and is evaluated against SOTAs on three real-world datasets with unknown noise patterns.
- PMD noise encompasses a broad range of heterogeneous, feature-dependent patterns and provides a theoretical setting for studying label noise.
2 METHOD
The method models noise through PMD conditions and progressively corrects labels using a neural network’s confidence. It begins with warm-up training, alternates retraining and correction, and extends the correction rule to multi-class classification.
- Preliminaries: The paper focuses on binary classification, defining clean and corrupted posterior probabilities, directional noise functions, and the Bayes-optimal classifier.
- PMD Noise: PMD noise imposes a polynomial, monotonically decreasing upper bound on noise where the Bayes classifier is confident, while allowing arbitrary noise near the decision boundary.
- PMD Noise: Compared with boundary consistent noise, PMD noise requires only a monotonic upper bound and permits arbitrary noise strength in a wider boundary buffer.
- Progressive Label Correction: The algorithm first warms up a neural network on noisy data, then corrects only labels for which the classifier has very high confidence.
- Progressive Label Correction: Progressive correction alternates classifier training and label updates while increasing the confidence threshold across rounds after warm-up.
- Multi-class Extension: For multi-class data, the method compares the predicted class confidence with the corrupted-label confidence and corrects when their gap exceeds a threshold.
3 ANALYSIS
The analysis formalizes assumptions under which progressive label correction can expand reliable regions and produce a classifier agreeing with the Bayes classifier on most data. Its guarantees apply asymptotically to PMD noise and depend on model approximation, distribution regularity, initialization, and algorithm parameters.
- 3.1 ASSUMPTIONS: The analysis assumes a hypothesis class that approximates the Bayes classifier and a distribution with continuous, bounded level-set density.These conditions are summarized as (α, ϵ)-consistency of H and (c∗, c∗)-boundedness of D.
- 3.2 MAIN RESULT AND PROOF SKETCH: Under Assumption 1 and PMD noise, the theorem guarantees that the final classifier matches the Bayes-optimal classifier on most of the distribution.The theorem defines e0 from the PMD margin and approximation parameters and specifies initializations including T0, m, and N.
- 3.2 MAIN RESULT AND PROOF SKETCH: A pure level set is one where the model’s predicted labels agree with the Bayes-optimal labels throughout the set.The set is defined by points whose clean-posterior margin is at least e.
- 3.2 MAIN RESULT AND PROOF SKETCH: Each correction round expands the region on which the model is reliable by replacing labels only where confidence exceeds the current threshold.The update uses yf(x) for sufficiently confident points and retains the existing noisy label otherwise, after which the model is retrained.
- 3.2 MAIN RESULT AND PROOF SKETCH: Warm-up rounds enlarge an initially tiny reliable region before the larger-level-set flipping procedure begins.The required number of rounds depends on ℓ, α, ϵ, T0, and e0 through the stated logarithmic bound.
- 3.2 MAIN RESULT AND PROOF SKETCH: Given a reasonably pure initial super-level set and suitable parameter conditions, Algorithm 1 progressively corrects a large fraction of corrupted labels.The limiting accuracy is controlled by the neural network’s approximation ability, characterized by ϵ.
4 EXPERIMENTS
Experiments evaluate the method under synthetic feature-dependent and hybrid noise, hyperparameter variations, and real-world noisy datasets. Across these settings, the method generally outperforms baselines and remains robust to substantial noise and parameter changes.
- Synthetic datasets: Synthetic experiments use CIFAR-10 and CIFAR-100 with feature-dependent, hybrid, uniform, and asymmetric noise.Feature-dependent noise is tested at 35% and 70%, while hybrid corruption reaches real noise levels of 50%–70%.
- Feature-dependent noise: The method achieves the best performance across three feature-dependent noise types and both tested noise levels.Table 1 reports average accuracy and standard deviation over three trials.
- Hybrid noise: Under hybrid noise reaching 50%–70% corruption, the method outperforms baselines, while some baselines deteriorate under additional 30% and 60% uniform noise.The deteriorating baselines become worse than the standard approach at high noise levels.
- Ablation studies: Performance is robust across a wide range of the initial confidence threshold θ0 and step size β.The ablations vary θ0 and β while evaluating CIFAR-10 with 35% feature-dependent noise.
- Real-world datasets: On Clothing1M, the method achieves the best performance among the listed baselines.Clothing1M contains one million noisy clothing images with an unknown noise structure.
- Real-world datasets: The method improves upon previous approaches on Food-101N and outperforms existing baselines on ANIMAL-10N.Food-101N has an estimated label purity of 80%, whereas ANIMAL-10N has an estimated noise rate of 8%.
5 CONCLUSION
The paper introduces a broad feature-dependent noise family and a theoretically guaranteed data-recalibrating method. Experiments report stronger performance than baselines across synthetic and real-world noisy datasets.
- The proposed feature-dependent label-noise family is much more general than traditional i.i.d. noise.
- The data-recalibrating method is theoretically guaranteed to converge to a well-behaved classifier under the proposed noise assumption.
- On synthetic datasets, the method outperforms various baselines under different feature-dependent noise patterns covered by the assumption.
- On real-world noisy datasets, the method achieves superior performance over existing approaches.
A APPENDIX
The appendix develops purity-improvement lemmas for progressive label flipping and combines them into a convergence theorem. The analysis also identifies a stopping boundary where further purity improvement is not guaranteed.
- Purity improvement: Each label-flipping round improves level-set purity by a factor of (1 + εℓ^α), yielding a logarithmic bound on warm-up rounds.The proof states that repeating the flipping step for sufficiently many rounds obtains the target purity T0.
- Iterative analysis: Starting from a pure level set, Algorithm 1 preserves and expands purity through warm-up and subsequent iterations under its initialization conditions.The analysis increases T by a factor β while balancing label-purification progress against corrupted labels.
- Correctness of flipping: When |η(x) − 1/2| ≥ 1/2 − 3ε, the updated classifier predicts the same label as η*(x), supporting correct label flipping.The condition follows from the bound |f(x) − η(x)| ≤ 3ε.
- Stopping boundary: Purity cannot be guaranteed to improve once T reaches 1/2 − 3ε, because classifier and posterior labels need not remain consistent there.
- Main theorem: Theorem 1 sets the initial pure-region threshold to e0 = max(t0, (α + ε)/(1 + 2α)) for PMD noise with margin t0.The theorem then applies the algorithm under stated iteration and initialization requirements.