Source-linked AI summary
Probabilistic End-to-end Noise Correction for Learning with Noisy Labels
Kun Yi, Jianxin Wu
TL;DR
Noisy labels are easy to collect at scale but can cause deep networks to overfit, motivating methods that do not depend on clean auxiliary data or prior noise information. PENCIL jointly learns network parameters and probabilistic label distributions, and it reports strong results on synthetic, real-world, and nearly clean datasets.
Problem
Large-scale clean annotations are expensive, while noisy labels can cause serious overfitting and reduced accuracy.
Method
PENCIL represents labels as distributions and iteratively updates both those distributions and network parameters through end-to-end back-propagation.
Results
PENCIL outperformed current state-of-the-art methods by large margins across synthetic-noise experiments and achieved 1.33% higher accuracy than the previous state of the art on Clothing1M.
Takeaways & Limitations
PENCIL is independent of the backbone network structure and does not require an auxiliary clean dataset or prior information about noise.
Takeaways & Limitations
At very high noise rates such as 80% symmetric noise, correct labels become too weak to bootstrap correction and PENCIL tends to fail.
Abstract
from arXiv · showhide
Deep learning has achieved excellent performance in various computer vision tasks, but requires a lot of training examples with clean labels. It is easy to collect a dataset with noisy labels, but such noise makes networks overfit seriously and accuracies drop dramatically. To address this problem, we propose an end-to-end framework called PENCIL, which can update both network parameters and label estimations as label distributions. PENCIL is independent of the backbone network structure and does not need an auxiliary clean dataset or prior information about noise, thus it is more general and robust than existing methods and is easy to apply. PENCIL outperforms previous state-of-the-art methods by large margins on both synthetic and real-world datasets with different noise types and noise rates. Experiments show that PENCIL is robust on clean datasets, too.
1. Introduction
PENCIL addresses the practical difficulty of learning from noisy labels by jointly updating network parameters and probabilistic label estimates without requiring clean auxiliary data or prior noise information. It reports strong performance across noisy and nearly clean datasets.
- Clean annotations are expensive and time-consuming, whereas large-scale web-collected datasets can contain substantial label noise.JFT300M is described as containing about 20% noisy labels.
- Existing noise-handling methods may require auxiliary clean labels, prior noise information, or complex procedures, while deep networks can memorize random labels.
- PENCIL models each image label as a distribution and updates both network parameters and label distributions through back-propagation.Noisy labels initialize the distributions, which are then iteratively corrected while the network loss uses the distributions as supervision.
- PENCIL is independent of the backbone network structure and does not need an auxiliary clean dataset or prior information about noise.The framework uses back-propagation to probabilistically update and correct image labels.
- PENCIL achieves state-of-the-art accuracy on datasets with synthetic and real-world noisy labels, including CIFAR-10, CIFAR-100, and Clothing1M.
- PENCIL is reported as robust both to noisy labels and to datasets with zero or small amounts of potential label noise, such as CUB-200.
2. Related Works
Prior noisy-label methods generally construct noise-aware models or robust losses, but often rely on impractical prerequisites or discard difficult samples. PENCIL instead updates estimated labels directly, making its approach more principled than running-average label replacement.
- DLDL uses constant label distributions for uncertainty, whereas PENCIL updates distributions to correct label noise.
- Deep networks with sufficient capacity can memorize randomly generated labels, causing noisy-label overfitting and reduced accuracy.
- Deleting unreliable samples can remove difficult examples that are important for accuracy, motivating more substantive noisy-label handling methods.
- Noise-handling methods mainly construct special noise-aware models or design robust loss functions.
- PENCIL replaces ad-hoc running averages of network predictions with probabilistic correction of label distributions.
- Existing methods may require an additional clean dataset, a groundtruth noise transition matrix, or complex model designs that hinder practical deployment.
3. The Proposed PENCIL Method
PENCIL jointly learns network parameters and per-image label distributions, using back-propagation and three losses to correct noisy labels while preserving valid probabilistic estimates.
- 3.1. Probabilistic modeling of noisy labels: PENCIL represents each image’s unknown clean label as a soft distribution yd rather than a fixed categorical label.yd is initialized from the noisy label and serves as pseudo-groundtruth during learning.
- 3.2. End-to-end noise correction in labels: PENCIL updates both network parameters θ and label distributions yd through back-propagation in every iteration.The auxiliary variable ˜y is unconstrained, while yd remains normalized as a valid probability distribution.
- 3.2. End-to-end noise correction in labels: ˜y is initialized from noisy labels using ˜y = Kˆy, with K = 10, making the initial yd approximately equal to ˆy.The noisy labels initialize the distributions but do not directly affect parameter learning afterward.
- 3.3. Compatibility loss: The compatibility loss keeps estimated distributions from becoming completely different from noisy labels, which may still contain many correct annotations.It is defined as a cross-entropy loss between the noisy labels and label distributions.
- 3.4. Classification loss: PENCIL replaces the classic KL-based classification loss with an asymmetric variant because the classic form performed poorly for noise handling.The proposed loss produces stronger updates when predictions support a class absent from the noisy label and preserves distributions where predictions disagree less decisively.
- 3.5. Entropy loss: The entropy loss encourages peaked predictions and prevents training from stalling when network predictions approach the label distributions.A one-hot distribution has the smallest entropy, favoring concentration on one category.
- 3.6. Overall loss: The overall objective combines classification loss, compatibility loss weighted by α, and entropy loss weighted by β.The resulting framework can equip any deep neural network backbone with noisy-label handling.
- 3.7. Implementation: After training, PENCIL-specific components are unnecessary because the backbone network alone performs prediction on future test examples.The framework uses automatic gradient computation for its back-propagation updates.
4. Experiments
Experiments evaluate PENCIL across synthetic and real-world noisy-label datasets, including high-noise, asymmetric, and nearly clean settings. PENCIL generally outperforms competing methods, estimates correct labels, and remains robust without validation data or noise priors, while failing at 80% symmetric noise.
- CIFAR-100: PENCIL significantly outperformed previous methods in nearly all CIFAR-100 symmetric and asymmetric noise cases, except 80% symmetric noise.It also outperformed Forward T in most cases despite Forward T using the groundtruth noise transition matrix.
- CIFAR-100: At 80% symmetric noise, correct labels were too weak to bootstrap correction, revealing a failure mode at very high noise rates.The authors contrast this with JFT300M, which has about 20% noisy labels.
- CIFAR-10: PENCIL remained robust across shared hyperparameters and showed no obvious best-to-last accuracy drop under CIFAR-10 symmetric noise.Unlike the cross-entropy baseline, PENCIL did not show the accuracy decline associated with memorizing noisy labels.
- CIFAR-10: PENCIL was the overall accuracy winner on CIFAR-10 asymmetric noise without requiring prior information about noise labels.Forward and CNN-CRF require the ground-truth noise transition matrix, which is rarely available in applications.
- CIFAR-10: With 70% symmetric noise and 30% asymmetric noise on CIFAR-10, PENCIL effectively and stably estimated correct labels for most examples.Estimated labels were determined by the maximum value in each learned label distribution.
- CUB-200: On CUB-200, PENCIL produced competitive results across a wide range of hyperparameters, and its final label distributions had correct maxima.Rows #4 to #7 exceeded the baseline, with row #4 achieving a 0.71% higher accuracy.
5. Conclusion
PENCIL is an end-to-end framework that supervises network learning with label probability distributions and updates those distributions through back-propagation. It outperformed state-of-the-art methods across synthetic and real-world noisy-label settings and remained robust across datasets and hyperparameters.
- PENCIL uses label probability distributions both to supervise network learning and to update labels through back-propagation each epoch.
- PENCIL outperformed current state-of-the-art methods by large margins on CIFAR-100 and CIFAR-10 across different noise types and rates.
- PENCIL achieved 1.33% higher accuracy than previous state-of-the-art on the real-world Clothing1M dataset.
- PENCIL was robust across different datasets and hyperparameters, including the noise-free CUB-200 dataset.