Source-linked AI summary
Training Convolutional Networks with Noisy Labels
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, Rob Fergus
TL;DR
The paper asks how discriminatively trained ConvNets can learn from abundant labels that may be inaccurate, when manual annotation is impractical. It adds a trainable noise layer that maps softmax outputs to the noisy-label distribution, and reports improved robustness across synthesized and real noise settings, including ImageNet. The approach is lightweight to implement, but learning the noise distribution is not uniquely identifiable without additional constraints.
Problem
Manual labeling is impractical in many settings, while freely available labels may be misleading and their effects on ConvNet training require study.
Method
The paper adds a constrained linear noise layer above the softmax and trains its probability-matrix weights jointly with the ConvNet by back-propagation.
Results
Across small and large datasets with synthesized and real noise, the proposed models improve performance over a standard model, with smaller gains on real data; on ImageNet, learned Q gains 8.5% over the unaltered model.
Takeaways & Limitations
The noise models handle both label-flip and outlier noise and can be added with minimal effort and little overhead to existing deep-learning training procedures.
Takeaways & Limitations
Minimizing the training loss alone does not identify the true noise matrix, because infinitely many Q and C pairs can produce the same combined solution.
Abstract
from arXiv · showhide
The availability of large labeled datasets has allowed Convolutional Network models to achieve impressive recognition results. However, in many settings manual annotation of the data is impractical; instead our data has noisy labels, i.e. there is some freely available label for each image which may or may not be accurate. In this paper, we explore the performance of discriminatively-trained Convnets when trained on such noisy data. We introduce an extra noise layer into the network which adapts the network outputs to match the noisy label distribution. The parameters of this noise layer can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks. We demonstrate the approaches on several datasets, including large scale experiments on the ImageNet classification benchmark.
1 INTRODUCTION
The paper examines how ConvNets behave when trained with freely available but unreliable labels, covering label flips and outliers. It proposes a noise layer that adapts model outputs to the noisy-label distribution and can be trained end-to-end.
- Manual image labeling is impractical for many problems, motivating the use of abundant but potentially misleading user tags or search keywords.
- Label flips confuse classes within the dataset, whereas outliers are unrelated images incorrectly assigned one of the task’s labels.
- Standard ConvNets are surprisingly robust to both noise types, but significant performance degradation occurs at high noise levels.
- A constrained linear noise layer atop the softmax adapts outputs to the noisy-label distribution and supports effective training under high label noise.
- The noise layer and base model train jointly with conventional back-propagation, automatically learning the noise distribution without supervision.
2 RELATED WORK
Prior work addressed noisy labels through preprocessing, robust classical classifiers, unbiased estimators, or limited neural-network noise models. The paper distinguishes its setting from semi-supervised learning because all data have noisy labels and no clean labels are assumed.
- Removing or correcting suspected labels is difficult because informative hard samples can be mistaken for harmful mislabeled examples.
- Robust methods for SVMs, kNN, and logistic regression exist, while generic unbiased estimation had been proposed for binary classification with noisy labels.
- Earlier neural noise models considered binary classification or assumed symmetric label noise, limiting their treatment of multiclass asymmetric noise.
- The paper assumes every example has a noisy label and an unknown trustworthy fraction, without access to any clean human-provided labels.
- Semi-supervised learning requires annotating a subset of noisy data, whereas this approach is presented as a natural precursor to SSL.
- Many spectral SSL methods have O(n^3) complexity, which is problematic for datasets with n = 10^6 examples.
3 LABEL NOISE MODELING
The paper models label noise with a constrained probability-matrix layer between the Convnet’s softmax and cost layers, allowing noisy-label training while targeting clean predictions. It learns the noise distribution jointly with the base model, using regularization and specialized handling for outliers.
- Label flip noise: Label flips are modeled by a noise distribution Q that maps true labels to observed noisy labels and can represent asymmetric confusions.The model assumes label flips are independent of x, while allowing class-specific error patterns such as cat-to-dog being more likely than cat-to-tree.
- Label flip noise: A constrained linear noise layer inserted after softmax transforms base-model predictions into a distribution matching noisy labels.Its weights represent the probability matrix Q, and the combined model is trained with cross-entropy on noisy labels.
- Label flip noise: When Q equals the true nonsingular noise distribution Q*, training the combined model forces the base model’s confusion matrix toward identity, yielding true-label predictions.The noise layer implements the transformation from clean-label predictions to noisy-label predictions while preserving the clean prediction objective for the base model.
- Learning the noise distribution: Jointly minimizing noisy-label loss is insufficient because infinitely many Q,C pairs can produce the same combined confusion matrix QC.A sufficiently capable base model may absorb the noise itself, making C approach Q* and Q approach the identity instead of recovering clean predictions.
- Learning the noise distribution: Regularization on Q, implemented in practice as weight decay, encourages the noise layer to absorb label diffusion while the base model learns clean labels.The trace-based argument is exact only under strong assumptions, although the paper reports empirical effectiveness beyond those assumptions.
- Outlier noise: Outlier noise is handled by adding an outlier class, but the corresponding Q* is fixed and requires manually setting the outlier fraction α.Extra outlier images can make the noise matrix nonsingular; the reported experiments find limited sensitivity to the exact α value.
4 EXPERIMENTS
Experiments on synthetic and real-world label noise show that the added noise model improves Convnet robustness across datasets, noise types, and training scales, including ImageNet.
- Experimental setup: The experiments evaluate label-flip and outlier noise on SVHN, CIFAR-10, ImageNet, and a real-world web-image dataset.Controlled experiments deliberately corrupt clean labels, while other experiments use datasets with inherent label noise.
- SVHN: 70% noise is tolerated on SVHN with the noise layer, whereas the normal model degrades badly beyond 30%.Beyond 70% noise, false labels overwhelm correct ones and the method breaks down.
- SVHN: The learned SVHN noise model performs as well as a model using the true noise distribution Q∗, indicating effective unsupervised estimation of Q.The learned and ground-truth noise distributions also differ negligibly in the reported example.
- SVHN and CIFAR-10: The noise model is more robust to label flips on both SVHN and CIFAR-10, with especially large gains at high noise levels and large training-set sizes.Figure 4 reports test errors across varying training amounts and noise levels; brighter colors indicate better accuracy.
- ImageNet: On ImageNet random label noise, the learned-Q model gains 8.5% over the unaltered model but remains 3.8% behind the ground-truth-Q model.The learned-Q model also outperforms training the unaltered model only on the clean-label subset; under adversarial noise it remains superior to the unaltered model.
- Outlier noise: For CIFAR-10 outlier noise, the noise model reduces the effect of outliers particularly for small training sets and consistently achieves higher average precision for inlier detection.The model is not sensitive to the precise value of α, although the experiment sets α using the true outlier count.
5 CONCLUSION
The paper evaluates two simple models for training convolutional networks with noisy labels across varied settings. They provide significant gains on synthesized noise, smaller gains on real data, and minimal implementation overhead.
- The authors proposed two simple models to improve convolutional-network robustness to noisy labels.
- Across small and large-scale datasets with synthesized and real label noise, both approaches were evaluated in varied settings.
- Significant performance gains over a standard model occurred with synthesized noise, whereas gains on real data were smaller.
- Both approaches require minimal effort to implement in existing deep-learning systems and add little overhead to training.
- The training set was carefully filtered to remove images also present in the CIFAR-10 test set.