Source-linked AI summary
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, Kevin Gimpel
TL;DR
Label noise from automatic, non-expert, and adversarial labeling can severely degrade deep classifiers, while prior methods often assume no labels are trusted. The paper proposes Gold Loss Correction, which uses a small trusted set to estimate label corruption and correct training; across vision and language tasks, it outperforms existing methods and remains data-efficient with few trusted labels.
Problem
Label noise increasingly affects large supervised-learning datasets, but many existing robustness methods assume that all training labels may be corrupted and do not exploit trusted examples.
Method
Gold Loss Correction uses trusted examples to estimate a K × K label-corruption matrix and trains a classifier with the resulting loss correction.
Results
Across vision and natural language experiments with varied corruption types and strengths, GLC surpasses previous robustness methods, including under severe noise and with few trusted labels.
Takeaways & Limitations
A small set of gold-standard labels can provide substantial, data-efficient robustness to moderate, severe, and adversarially relevant label corruption.
Takeaways & Limitations
GLC’s corruption-matrix estimate depends on classifier quality, sufficient trusted examples from each class, and the conditional-independence assumption.
Abstract
from arXiv · showhide
The growing importance of massive datasets used for deep learning makes robustness to label noise a critical property for classifiers to have. Sources of label noise include automatic labeling, non-expert labeling, and label corruption by data poisoning adversaries. Numerous previous works assume that no source of labels can be trusted. We relax this assumption and assume that a small subset of the training data is trusted. This enables substantial label corruption robustness performance gains. In addition, particularly severe label noise can be combated by using a set of trusted data with clean labels. We utilize trusted data by proposing a loss correction technique that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers. Across vision and natural language processing tasks, we experiment with various label noises at several strengths, and show that our method significantly outperforms existing methods.
1 Introduction
Label noise increasingly threatens supervised learning as datasets grow and labels come from automatic, non-expert, or adversarial sources. This work uses a small trusted subset to improve robustness, including under severe corruption.
- Label noise can arise from automatic labeling, expensive annotation settings, or data-poisoning attacks, sharply degrading classification performance.
- Unlike prior work assuming every training example may be corrupted, the paper assumes that a small subset of training labels is trusted.
- A new trusted-label loss correction is evaluated across vision and natural language datasets with multiple corruption types and strengths.
- The method recovers from extremely high noise, including cases where most untrusted labels are corrupted.
- The method is significantly more accurate than corrections without trusted data under moderate to severe noise and is more data-efficient than another trusted-data method.
2 Related Work
Earlier approaches modify architectures or correct losses while often treating labels as potentially corrupted and making assumptions about the noise structure. This paper instead incorporates a small set of clean labels into label-noise modeling and correction.
- Label-noise robustness matters because labeled-data systems degrade noticeably under natural noise and can degrade even more under adversarial noise.
- Prior label-noise methods modify model architectures or implement loss corrections for binary, symmetric-noise, or asymmetric multi-class settings.
- Sukhbaatar et al. introduce a stochastic corruption matrix and forward loss correction, while Patrini et al. estimate the matrix without clean labels under strong assumptions.
- This work assumes access to a small set of clean labels during training, departing from approaches that do not use trusted data.
3 Gold Loss Correction
Gold Loss Correction uses trusted examples to estimate a class-conditional corruption matrix, then trains a classifier that corrects predictions for noisy labels. Its estimate depends on classifier quality, per-class trusted examples, and a conditional-independence assumption.
- Gold Loss Correction: The method assumes an untrusted dataset with potentially corrupted labels and a much smaller trusted dataset drawn from the true distribution.
- Estimating The Corruption Matrix: Figure 1 compares the corruption matrix with three estimates for corrupted CIFAR-10 data, where Cij denotes corruption from class i to class j.
- Gold Loss Correction: GLC estimates the K × K corruption-probability matrix Cij = p(ey = j | y = i) from trusted examples.
- Estimating The Corruption Matrix: The estimate relies on a good untrusted-data classifier, enough trusted examples per class, and approximate conditional independence of noisy and true labels given x.
- Training a Corrected Classifier: The corrected classifier transforms its softmax output with bC^T and trains on noisy labels, while trusted examples receive no correction during training.
- Estimating The Corruption Matrix: It first trains on untrusted data, averages predicted noisy-label probabilities over trusted examples in each true class, and uses those averages to estimate the matrix.
4 Experiments
The experiments compare label-correction methods across vision and NLP datasets under multiple corruption processes and trusted-data fractions. GLC generally performs best, including under severe weak-classifier noise, while its advantage depends on the dataset and trusted fraction.
- Label Noise Corrections: The Forward method estimates the corruption matrix without trusted training data, whereas Forward Gold replaces its trusted-example estimate with the identity matrix.Forward Gold isolates the effect of training on trusted labels within the Forward correction framework.
- Label Noise Corrections: Distillation trains a network on a large trusted dataset to provide soft targets for untrusted data, making its utility dependent on reliable trusted-data supervision.The method is compared with GLC despite its need for a large trusted dataset.
- Results: Across all experiments, GLC achieves better area under the error curve than the baselines, Forward, and Distillation methods.The rankings among the remaining methods and baselines are mixed.
- Results: On MNIST, trusted-data-only training outperforms all methods except GLC and Confusion Matrix, but it performs significantly worse on CIFAR-100 even with large trusted fractions.This demonstrates that the value of using trusted data alone varies substantially across datasets.
- Results: GLC consistently outperforms Ren et al.'s trusted-set meta-learning correction and averages best in weak-classifier label experiments.With only 1% trusted data on CIFAR-10, GLC reaches 26.94% error versus 60% originally, although uncorrected training reaches 28.32% in that experiment.
5 Discussion
The GLC is designed to remain data-efficient when trusted labels are scarce, while improving label-noise correction through estimates of the corruption matrix. On Clothing1M, it substantially outperforms Distillation with few trusted examples, and achieves strong accuracy when using the full trusted set.
- Data Efficiency: Clothing1M contains 1 million noisily labeled images and 50,000 human-annotated examples from which trusted subsets are sampled.The dataset supports direct comparison of GLC and Distillation under very small trusted fractions.
- Data Efficiency: GLC outperforms Distillation by a large margin on Clothing1M, especially when fewer trusted examples are available.The advantage decreases as the number of trusted examples increases because Distillation benefits from a larger trusted training set.
- Full Trusted Set: 80.67% accuracy is achieved on Clothing1M with the full trusted set, compared with 79.03% for Forward correction and 78.24% reported previously.Performance saturates as more trusted labels are added.
- Corruption-Matrix Estimation: 0.96 percentage points of area-under-error-curve improvement is obtained when the true corruption matrix replaces the estimated matrix, while calibration and base-rate adjustments do not change original GLC performance.This suggests the estimator is robust to uncalibrated networks, but further gains may require improving the network used to estimate the corruption probabilities.
6 Conclusion
The paper concludes that a small trusted set can substantially improve neural-network robustness to label noise. Its Gold Loss Correction method uses trusted labels to estimate noise and performs strongly across vision and NLP settings, including severe corruption.
- Conclusion: Gold Loss Correction uses a small set of correct labels to estimate the label-noise distribution and improve neural-network robustness.The method is explicitly designed to be data-efficient.
- Conclusion: GLC surpasses previous label-noise robustness methods across vision and natural language processing domains, corruption types, and corruption strengths.The experiments include severe noise strengths.
- Conclusion: The results characterize GLC as a powerful, data-efficient method for improving robustness to label noise.
B Additional Results and Error Plots
Additional results report evaluation conventions for vision datasets and Ren et al.’s method, alongside error curves covering multiple datasets, corruption types, and corruption strengths in vision and NLP.
- Table 4: Table 4 reports Ren et al.’s method, with other methods copied from Table 5 and trusted-data percentages shown as trusted fraction multiplied by 100.Values are generally percentages for area under the error curve computed at 11 test points, with the best mean result bolded.
- Table 5: Table 5 reports vision-dataset results and adds SVHN relative to the paper’s original results.The table uses the same trusted-fraction and area-under-error-curve conventions described for Table 4.
- Error Plots: Figure 5 provides vision error curves across numerous label-correction methods, corruption types, and corruption strengths.
- Error Plots: Figure 7 provides NLP error curves across numerous label-correction methods, corruption types, and corruption strengths.