Source-linked AI summary
Learning Deep Networks from Noisy Labels with Dropout Regularization
Ishan Jindal, Matthew Nokleby, Xuewen Chen
TL;DR
Large datasets often contain unreliable labels, creating a need for deep-learning methods that account for label noise. The paper augments a base network with a softmax noise model, trains both jointly with dropout regularization, and reports state-of-the-art results on noisy CIFAR-10 and MNIST, often surpassing models with known noise statistics. The learned model consistently overestimates label-flip probabilities, and the implications of this behavior remain open for future research.
Problem
Large datasets may provide unreliable labels, while expressive deep networks can fit corrupted labels directly instead of learning an accurate classifier.
Method
The method augments a standard deep network with a softmax layer modeling label noise and jointly trains the classifier and noise model using dropout regularization.
Results
On noisy CIFAR-10 and MNIST, dropout generally outperforms the base, trace-regularized, bootstrapping, and often even true-noise models.
Takeaways & Limitations
Dropout regularization enables robust deep-network learning from corrupted labels and can exceed performance obtained when the noise statistics are known a priori.
Takeaways & Limitations
The learned noise model consistently overestimates label-flip probabilities, and the consequences and ideal training noise model remain topics for future research.
Abstract
from arXiv · showhide
Large datasets often have unreliable labels-such as those obtained from Amazon's Mechanical Turk or social media platforms-and classifiers trained on mislabeled datasets often exhibit poor performance. We present a simple, effective technique for accounting for label noise when training deep neural networks. We augment a standard deep network with a softmax layer that models the label noise statistics. Then, we train the deep network and noise model jointly via end-to-end stochastic gradient descent on the (perhaps mislabeled) dataset. The augmented model is overdetermined, so in order to encourage the learning of a non-trivial noise model, we apply dropout regularization to the weights of the noise model during training. Numerical experiments on noisy versions of the CIFAR-10 and MNIST datasets show that the proposed dropout technique outperforms state-of-the-art methods.
I. PROBLEM STATEMENT
The paper formalizes learning when only noisy labels are available, assuming label corruption depends on the true class rather than the image and is conditionally independent across samples. It represents this corruption with a column-stochastic noise matrix and evaluates synthetic uniform and non-uniform noise settings.
- The training set contains images paired with noisy labels because accurate labels are difficult to obtain at large scale.
- The noise model assumes each noisy label depends only on its true label, not on the corresponding image.
- Noisy labels are assumed conditionally independent across samples given their true labels.
- The conditional noise process is represented by a column-stochastic matrix Ψ ∈ R^C×C.
- Experiments synthesize noisy training labels from fixed distributions while leaving test labels unperturbed.
- The study considers uniform noise, where the true label is retained with probability 1−p and otherwise sampled uniformly, and non-uniform noise based on simplex-valued columns.
A. Learning Deep Networks with Noise Models
The method augments a base deep network with a noise model and trains both jointly so noisy-label supervision can produce a more accurate classifier. The base model alone is adequate under limited noise but fails when incorrect labels overwhelm training.
- A. Learning Deep Networks with Noise Models: The proposed pipeline jointly learns a base network and a noise model using stochastic gradient descent, then discards the noise model at test time.During training, the noise model denoises labels during backpropagation; test predictions use only the base model output.
- A. Learning Deep Networks with Noise Models: The base model uses either a three-layer ReLU convolutional network with pooling or a three-layer ReLU deep neural network.
- A. Learning Deep Networks with Noise Models: The base network maps its final-layer vector h through softmax to estimate the clean class-label distribution.
- A. Learning Deep Networks with Noise Models: Treating noisy labels as true labels can work under mild noise, but incorrect labels overwhelm the base model when noise becomes severe.
- A. Learning Deep Networks with Noise Models: With known noise statistics, a linear noise model can be attached to the base network and trained using noisy-label cross-entropy.
- A. Learning Deep Networks with Noise Models: When noise statistics are unknown, jointly estimating them risks a trivial noise model because expressive convolutional networks can fit noisy labels directly.
- A. Learning Deep Networks with Noise Models: Trace regularization encourages diffuse noise estimates but requires projecting the estimate onto stochastic matrices after every iteration.
II. DROPOUT REGULARIZATION
The method augments a base network with an unconstrained softmax noise model and applies dropout during training so the noise model captures label corruption rather than merely fitting noisy labels.
- Noise-model architecture: The proposed architecture adds a softmax layer with square, unconstrained weights W to model the noisy-label distribution.Its output g = σ(Wh) is interpreted as the probability distribution over the noisy label y′.
- Noise-model architecture: Because softmax implicitly normalizes conditional probabilities, W needs neither nonnegative entries nor an explicit normalization step.This removes the normalization operation required by the alternative parameterization.
- Dropout regularization: Dropout is applied to the base-model output σ(h) to prevent the base model from learning noisy labels directly.At each SGD step, outputs are multiplied by independent Bernoulli variables, thinning the network during training.
- Dropout regularization: Each dropout realization zeros some W-related contributions, forcing the corresponding probabilities toward a uniform baseline and encouraging a non-trivial noise model.The simulations report q = 0.1 as the best setting, corresponding to an aggressively sparse model.
- Training and inference: The noise model is used only during training to denoise labels for learning the base model, then disconnected at test time without model averaging.The base model alone classifies test samples after training.
III. EXPERIMENTAL RESULTS
The evaluation covers CIFAR-10 and MNIST under uniform and non-uniform noise, using both CNN and DNN base models.
- Datasets: Experiments evaluate the method on two datasets: CIFAR-10 and MNIST.
- Noise models: The experiments include two label-noise models: uniform and non-uniform.
- Base models: Both CNN and DNN base models are used in the evaluation.
A. CIFAR Images
Experiments on CIFAR-10 compare dropout-regularized noise modeling with base, known-noise, and trace-regularized approaches under uniform and non-uniform label noise. Dropout usually performs best under uniform noise, while non-uniform noise exposes a gap associated with learned models becoming too close to uniform.
- Experimental setup: CIFAR-10 experiments evaluate CNN classification under 30%, 50%, and 70% uniform and non-uniform label noise.The comparisons include base, true-noise, trace-regularized, and proposed dropout approaches.
- Uniform noise: In most uniform-noise cases, dropout achieves the best performance, exceeding the true noise model except at 50% noise.The true noise model assumes Ψ is known a priori.
- Uniform noise: Learned uniform-noise matrices are approximately uniform and diagonally dominant but pessimistically underestimate correct-label probabilities.Average diagonal values are 0.279, 0.345, and 0.447 for 30%, 50%, and 70% noise, respectively.
- Non-uniform noise: Under non-uniform noise, dropout performs better than or on par with the trace scheme but worse than the true noise model overall.The learned models become close to uniform despite the true model being non-uniform, which the authors hypothesize explains the performance gap.
B. MNIST Images
MNIST experiments evaluate CNN and DNN models under uniform and non-uniform label noise. Dropout is strongest under uniform noise, while non-uniform noise exposes a consistent performance gap linked to imperfect noise-model learning.
- Experimental setup: The experiments compare CNN and DNN architectures on MNIST corrupted by uniform and non-uniform label noise.The CNN results are reported in Tables III–IV, and the DNN results in Tables V–VI.
- Non-uniform noise: Dropout performs worse relative to the true noise model under non-uniform noise, especially at the 70% noise level.The paper attributes this to poor learning of non-uniform noise models and weaker natural clustering in MNIST than in CIFAR-10.
- Uniform noise: Dropout outperforms every compared scheme on uniform-noise MNIST except at the 70% noise level.The comparison includes the base model, true noise model, and bootstrapping for the DNN experiment.
- Comparison with prior work: At 50% noise, dropout achieves 2.83% MNIST classification error versus 7.83% reported for a prior approach.The comparison is indirect because the prior approach uses a pretrained CNN, whereas this work trains end-to-end with noisy labels.
IV. CONCLUSION AND FUTURE WORK
The paper concludes that dropout-regularized noise modeling enables deep networks to learn from corrupted labels and can outperform models with known noise statistics. It also identifies systematic overestimation of label-flip probabilities as an unresolved issue for future research.
- Conclusion: The method augments a standard deep network with a softmax layer modeling label noise and jointly trains both components with dropout regularization.The noise model is applied to the final softmax layer during training.
- Conclusion: On CIFAR-10 and MNIST, the approach achieves state-of-the-art performance and sometimes surpasses models whose label-noise statistics are known a priori.This is the paper’s central empirical conclusion.
- Future work: The learned noise model consistently overestimates label-flip probabilities, and the ideal training noise model remains an open research question.The authors interpret this as encouraging clustering of ambiguously labeled data rather than direct classification from noisy labels.