Source-linked AI summary
Deep Learning is Robust to Massive Label Noise
David Rolnick, Andreas Veit, Serge Belongie, Nir Shavit
TL;DR
Expensive, carefully annotated datasets motivate studying whether deep networks can learn from abundant but unreliable labels. The paper evaluates extreme non-adversarial label noise while preserving the original clean examples and finds strong generalization, with larger data and batch sizes partly offsetting noise-related effects.
Problem
Carefully annotated datasets are difficult and expensive to obtain, motivating evidence about learning when labels are massively unreliable.
Method
The paper adds α noisy examples per original training example, studies uniform and structured noise, and assumes unreliable labels arise from an unknown stochastic rather than adversarial process.
Results
Deep networks generalize across datasets and noise patterns, including 90%+ MNIST performance with 100 noisy labels per clean label.
Takeaways & Limitations
Robust learning under massive non-adversarial noise requires sufficiently many clean labels; noise also reduces effective batch size, which larger batches and learning-rate adjustment can partly mitigate.
Takeaways & Limitations
The studies focus on non-adversarial noise, still require expert-vetted test sets, and find that clean labels outperform noisy labels when training-data quantity is held constant.
Abstract
from arXiv · showhide
Deep neural networks trained on large supervised datasets have led to impressive results in image classification and other tasks. However, well-annotated datasets can be time-consuming and expensive to collect, lending increased interest to larger but noisy datasets that are more easily obtained. In this paper, we show that deep neural networks are capable of generalizing from training data for which true labels are massively outnumbered by incorrect labels. We demonstrate remarkably high test performance after training on corrupted data from MNIST, CIFAR, and ImageNet. For example, on MNIST we obtain test accuracy above 90 percent even after each clean training example has been diluted with 100 randomly-labeled examples. Such behavior holds across multiple patterns of label noise, even when erroneous labels are biased towards confusing classes. We show that training in this regime requires a significant but manageable increase in dataset size that is related to the factor by which correct labels have been diluted. Finally, we provide an analysis of our results that shows how increasing noise decreases the effective batch size.
1. Introduction
The paper investigates whether deep networks can learn from extremely unreliable labels and finds that they can generalize across substantial noise when enough data is available. It also identifies dataset size and effective batch size as important factors governing robustness.
- Deep neural networks generalize from massively noisy data rather than merely memorizing noise, even when label accuracy is only 1 percent above chance.This behavior appears across datasets and noise patterns, including labels biased toward confused classes.
- The minimum dataset size required for effective training increases with noise level.The required clean-data amount rises slightly faster than linearly with the ratio of noisy to clean examples.
- Increasing dataset size beyond the needed threshold does not appreciably increase accuracy.Performance rapidly levels off after the critical amount of clean data is reached.
- High label noise decreases effective batch size because noisy labels roughly cancel, leaving a small learning signal.The paper states that larger batch sizes and learning-rate scaling can partly compensate for this effect.
2. Related Work
Prior work addresses noisy labels through noise-robust learning or label cleansing, while related robustness studies examine other forms of neural-network resilience. This paper differs by holding the number of original examples fixed while adding noise, separating noisy-label quantity from clean-data scarcity.
- Existing approaches to noisy data mainly learn directly from noisy labels or remove and correct mislabeled examples.
- Related robustness research has examined parameter redundancy, layer deletion, and adversarial examples in neural networks.
- The paper’s MNIST comparison includes a perceptron, MLPs with 1, 2, and 4 hidden layers, and a 4-layer ConvNet.With 100 noisy labels per clean label, the 4-layer ConvNet attains 91% performance.
- Unlike earlier studies that reduce both correct-label proportion and absolute count, this work shows that reducing clean examples is more destructive than adding noisy labels.
3. Learning with massive label noise
The experiments test whether deep networks can learn when noisy labels vastly outnumber correct ones, using uniform, structured, and out-of-vocabulary noise across MNIST, CIFAR-10, and ImageNet. Performance remains strong across these settings, while robustness depends on architecture, noise structure, noise source, and sufficient clean-data volume.
- Experimental setup: The experiments model noise by adding α noisy examples per original example while retaining the original n examples and assume labels arise from an unknown stochastic, non-adversarial process.The resulting training set contains αn noisy labels, separating dilution of correct labels from reducing their absolute number.
- Uniform label noise: 100 noisy labels per clean example still yielded over 90% MNIST accuracy, while CIFAR-10 exceeded 85% with 10 and ImageNet exceeded 70% top-5 accuracy with 5.These results used randomly labeled examples and show substantial generalization despite extreme dilution of correct labels.
- Uniform label noise: Larger networks were more robust to label noise, with ConvNets and ResNets degrading more slowly than smaller architectures.On CIFAR-10, the residual network’s accuracy dropped more slowly than that of smaller ConvNets.
- Structured label noise: Structured noise preserved robustness when the correct label remained most likely, although bias toward random classes hurt slightly more than bias toward confused classes.The study varied structure using labels biased toward easily confused, hardly confused, or random classes.
- Source of noisy labels: Noise from CIFAR-100 caused about half the performance drop of same-dataset CIFAR-10 noise, while matched white-noise inputs caused no performance drop.The comparison used a six-layer ConvNet across varying noise levels.
4. The importance of larger datasets
Large datasets enable deep networks to learn from noisy labels, but increasing noise raises the critical amount of clean data needed for high accuracy more than linearly.
- 4. The importance of larger datasets: Dataset size is central to learning from noisy data, making larger training sets particularly important in high-noise settings.The paper frames data size as underlying the ability of deep networks to learn from noisy data.
- 4. The importance of larger datasets: With sufficient data, networks reach similar results across noise levels, although each higher noise level requires more clean training examples.The required clean-data threshold increases as noise rises.
- 4. The importance of larger datasets: The required amount of clean data increases slightly more than linearly with the ratio of noisy to clean data.Figure 9 relates the clean-data requirement to noise for different target accuracies.
5. Analysis
The analysis explains noisy-label robustness through gradient averaging: noise reduces the effective batch size, so larger batches and lower learning rates can mitigate its effects.
- 5. Analysis: For ImageNet, increasing noise raises the critical number of clean training examples needed for high accuracy, after which accuracy plateaus.The figure reports this relationship for an 18-layer ResNet.
- 5. Analysis: Increasing batch size improves robustness to noisy labels, as observed for a 2-layer ConvNet on MNIST across batch sizes from 32 to 256.The experiments hold other settings fixed while varying batch size.
- 5. Analysis: The analysis models noisy labels as uniformly random labels added to n training examples, with αn noisy labels and clean-label probability 1/(1+α).The expected batch loss is then represented by a noisy loss function Hα.
- 5. Analysis: The noisy loss Hα simulates αn noisy labels with infinite batch size, where increasing α decreases the true-gradient magnitude without changing its direction.This explains why MNIST accuracy does not decrease as α increases under the infinite-batch approximation.
- 5. Analysis: Noisy labels reduce the effective batch size because erroneous signals roughly cancel, leaving only a small learning signal.This provides the paper’s central mechanism for understanding optimization under label noise.
- 5. Analysis: For CIFAR-10, lower learning rates are generally optimal as noise increases, consistent with scaling learning rate to the reduced effective batch size.The analysis links the learning-rate trend to noisier, effectively smaller batches.
6. Conclusion
The paper concludes that deep networks can learn with essentially arbitrary non-adversarial label noise when enough clean labels are available, while noise increases data requirements and reduces effective batch size.
- 6. Conclusion: Deep networks remain robust to essentially arbitrary label noise when the number of clean labels is sufficiently large.The required clean-label count rises slightly above linearly with the noisy-to-clean label ratio.
- 6. Conclusion: Noisy labels reduce effective batch size, an effect mitigated by larger batches and downscaling the learning rate.This is the paper’s final optimization implication.
- 6. Conclusion: Clean labels still outperform noisy labels when the quantity of training data is held constant.The conclusion also emphasizes the continuing need for expert-vetted test sets and the focus on non-adversarial noise.
- 6. Conclusion: The work leaves label-cleaning, semi-supervised methods, and annotation-versus-acquisition cost trade-offs for future investigation.The paper specifically asks whether these approaches can lower the training-set-size threshold.