Source-linked AI summary
Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach
Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, Lizhen Qu
TL;DR
Large-scale training labels are often noisy, while existing corrections either lack broad architectural applicability or require noise rates known in advance. The paper introduces forward and backward loss corrections with multiclass noise estimation, reports robustness across datasets and architectures, and proves noise-independent curvature for ReLU networks.
Problem
Large-scale labeling introduces class-dependent noise, while prior practical approaches may lack theoretical grounding and theoretically grounded corrections require noise rates known a priori.
Method
The paper uses backward and forward loss corrections based on a class-noise transition matrix T and extends noise estimation to the multiclass setting.
Results
Experiments across image-recognition and sentiment-analysis datasets and diverse architectures confirm robustness, with forward correction often performing better and new state of the art established on Clothing1M.
Takeaways & Limitations
Loss correction provides an architecture- and application-domain-independent framework for training deep neural networks under class-dependent label noise.
Abstract
from arXiv · showhide
We present a theoretically grounded approach to train deep neural networks, including recurrent networks, subject to class-dependent label noise. We propose two procedures for loss correction that are agnostic to both application domain and network architecture. They simply amount to at most a matrix inversion and multiplication, provided that we know the probability of each class being corrupted into another. We further show how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and thus providing an end-to-end framework. Extensive experiments on MNIST, IMDB, CIFAR-10, CIFAR-100 and a large scale dataset of clothing images employing a diversity of architectures --- stacking dense, convolutional, pooling, dropout, batch normalization, word embedding, LSTM and residual layers --- demonstrate the noise robustness of our proposals. Incidentally, we also prove that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise.
1. Introduction
The paper develops theoretically grounded loss corrections for deep networks trained with class-dependent label noise, combining correction procedures with multiclass noise estimation. Experiments across domains and architectures support robustness, while ReLU networks additionally have noise-independent loss curvature.
- Large-scale labeling through crowdsourcing and keyword-based image searches introduces label noise that can adversely affect model training.
- Existing architecture-specific approaches often lack a solid theoretical framework and require substantial clean labels for pre-training or hyperparameter validation.
- The paper introduces backward correction using T^-1 and forward correction using T, where T contains class-flip probabilities.Backward correction multiplies the loss by T^-1, whereas forward correction multiplies network predictions by T.
- Both procedures have formal robustness guarantees and are independent of network architecture, application domain, and chosen loss function.
- The authors extend noise-rate estimation to multiclass problems, forming an end-to-end framework that combines estimation with loss correction.
- The paper proves that the loss Hessian is independent of class-dependent label noise when ReLU is the only non-linearity.
2. Related work
Related work spans theoretical noise robustness, corrected surrogate losses, noise-rate estimation, and deep-learning methods for noisy labels. The paper positions itself at their intersection by combining estimation with loss correction while avoiding architectural noise-modeling heuristics.
- Noise robustness: Theory distinguishes low-capacity models, where even mild symmetric noise can be damaging, from settings where Bayes-optimal classification remains unchanged under broader noise assumptions.
- Surrogate losses: Noise-corrected losses can recover clean-data optimization when noise rates are known, while some losses are intrinsically robust under specific symmetric-noise conditions.
- Noise rate estimation: Noise-rate estimators infer label-flip probabilities from noisy samples using perfect-instance assumptions, kernel embeddings, or post-processing of class-probability scores.
- Deep learning with noisy labels: Deep-learning approaches often model noise through specialized architectures and may require a small clean-label set for pre-training or hyperparameter selection.
- Deep learning with noisy labels: The method in [39] adds a linear layer representing T but requires trace regularization and a fixed update schedule; this paper instead estimates T first and then applies correction.
- The paper identifies its novelty as combining noise-corrected losses with noise-rate estimators for modern deep neural networks.
3. Preliminaries .=
The preliminaries define multiclass data, neural-network outputs, and loss functions used throughout the analysis. They formalize networks as layered transformations whose final predictions are compared with labels through softmax-based losses.
- Multiclass classification uses feature space X ⊆ R^d and labels represented by one-hot canonical vectors e_i in R^c.
- Examples are drawn from an unknown joint distribution p(x,y)=p(y|x)p(x), with each label vector containing exactly one nonzero coordinate.
- An n-layer neural network is a composition of intermediate transformations mapping X to R^c, with trainable weights and biases.
- The network’s final scores are converted by softmax into class-probability estimates, and the predicted label is the class with the largest score.
- A loss compares the observed label with the network output; cross-entropy is given by -log p̂(y=e_i|x), with an equivalent vector-form loss defined over all possible labels.
- The formal results apply under mild conditions to a generic loss function, with cross-entropy used as a principal example.
4. Label noise and loss robustness
The paper studies class-conditional label noise, where labels are stochastically flipped according to a transition matrix while features remain unchanged. It proposes backward and forward loss corrections, extends noise-rate estimation to multiclass settings, and establishes robustness properties under stated assumptions.
- Noise model: Class-conditional noise flips labels according to a row-stochastic transition matrix T while leaving feature vectors untouched.The matrix need not be symmetric across classes.
- Backward correction: Backward correction multiplies the loss by T^-1 and is unbiased when T is non-singular.Its expected corrected loss equals the clean-data loss, so their minimizers coincide.
- Forward correction: Forward correction multiplies network predictions by T and preserves the clean-loss minimizer for proper composite losses when T is non-singular.For cross-entropy, the corrected loss compares noisy labels with predictions corrupted by T.
- Estimating noise: The end-to-end procedure estimates T from noisy class-probability predictions and then retrains the network with a corrected loss.The estimator can use an unlabeled feature sample, but assumes suitable instances and a network rich enough to model noisy-label probabilities accurately.
- Limitations: The correction procedures require knowing or estimating T, and estimation quality can deteriorate under difficult data conditions.The paper notes that high noise and few images per class can substantially reduce the gain from correction, while small estimation errors are often mild.
- Curvature under noise: With ReLU activations, label noise does not change the loss Hessian, and backward-corrected and uncorrected losses have identical Hessians for any T.The result concerns curvature and first-order convergence rates, not the locations of stationary points or minima; it also extends to piecewise-linear networks such as those using max-pooling.
5. Experiments
The experiments evaluate loss correction across multiple datasets, noise settings, and neural architectures, including recurrent and residual networks. Results indicate robust performance under label corruption, with estimated-noise correction generally close to correction using the true noise matrix.
- Labels are artificially corrupted with class-dependent transition matrices designed to mimic mistakes between similar classes.Examples include CAT →DOG and structured transitions among related CIFAR classes.
- Cross-entropy is compared with backward and forward loss corrections under known or estimated noise matrices, alongside several alternative robust losses.The comparisons include unhinged, sigmoid, Savage, and soft and hard bootstrapping losses.
- Estimated noise correction closely tracks the true transition matrix on the binary IMDB task.The paper attributes this to easier estimation in the binary setting.
- With low asymmetric noise, correction performance differs between simpler tasks and deeper, more difficult problems.Table 2 reports similar clean-data accuracies across losses, while noisy settings produce clearer differences.
- On Clothing1M, the best result is obtained by fine-tuning a network trained with forward correction, outperforming the prior result by more than 2 percent without time-consuming bootstrapping of the clean subset.The experiment uses 1M noisy images plus clean data for training, validation, and testing.
6. Discussion and Conclusion
The proposed corrections keep accuracy close to clean-data cross-entropy despite substantial corruption, although forward correction often performs better and noise estimation can limit robustness. The authors identify optimization effects and difficult estimation regimes as important practical considerations.
- Accuracy is consistently only few percent points away from training cross-entropy on clean data, while corruption can worsen cross-entropy performance by 40 percent or more.
- Forward correction often performs better than backward correction, despite theoretical guarantees for both methods in the infinite-data limit.Backward correction can produce large-magnitude linear combinations of losses, whereas forward correction keeps predictions within a probability distribution.
- Noise-estimation quality is central to robustness, with a median accuracy drop of only 10 points relative to using the true noise matrix.For CIFAR-100 under the highest-noise setting, estimation destroys most of the correction gain.
- The authors attribute the CIFAR-100 estimation failure to high noise and only 500 images per class, and report sensitivity to α.
- Future work includes incorporating structural priors such as low-rank T and determining whether instance-dependent noise can be handled.The authors also anticipate applications to pre-training with noisy Web data.