Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels

Pengfei Chen; Benben Liao; Guangyong Chen; Shengyu Zhang

Source-linked AI summary

Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels

Pengfei Chen, Benben Liao, Guangyong Chen, Shengyu Zhang

arXiv:1905.05040v1May 13, 2019cs.LGstat.ML

TL;DR

Noisy labels challenge DNN training because high-capacity networks can memorize corrupted annotations, while their effects on generalization lack a quantitative explanation. The paper characterizes test accuracy through noise ratios, uses cross-validation to identify cleaner samples, and combines them with Co-teaching. The resulting strategy achieves state-of-the-art performance under synthetic and real-world training noise.

Problem
The effects of noisy labels on DNN training and generalization lack a theoretical and quantitative explanation, despite noisy labels being widespread and DNNs able to memorize them.
Method
The paper quantitatively analyzes noisy-label generalization, randomly splits datasets for cross-validation to identify clean samples, and applies Co-teaching using those samples.
Results
The strategy achieves the best clean-test-set accuracy against state-of-the-art methods on synthetic CIFAR-10 and real-world WebVision noise.
Takeaways & Limitations
The analysis supports practical selection of cleaner samples and robust DNN training against noisy labels.
Takeaways & Limitations
The theoretical claim assumes i.i.d. observed inputs and a sufficiently high-capacity network.

Abstract

from arXiv · show

Noisy labels are ubiquitous in real-world datasets, which poses a challenge for robustly training deep neural networks (DNNs) as DNNs usually have the high capacity to memorize the noisy labels. In this paper, we find that the test accuracy can be quantitatively characterized in terms of the noise ratio in datasets. In particular, the test accuracy is a quadratic function of the noise ratio in the case of symmetric noise, which explains the experimental findings previously published. Based on our analysis, we apply cross-validation to randomly split noisy datasets, which identifies most samples that have correct labels. Then we adopt the Co-teaching strategy which takes full advantage of the identified samples to train DNNs robustly against noisy labels. Compared with extensive state-of-the-art methods, our strategy consistently improves the generalization performance of DNNs under both synthetic and real-world training noise.

1. Introduction

Noisy labels are common and costly to avoid, yet their effects on DNN generalization remain insufficiently explained. The paper provides a quantitative analysis and combines cross-validation with Co-teaching to identify cleaner samples and improve noisy-label training.

Motivation: Accurate labels are expensive to obtain, while crowdsourcing and online queries yield noisy labels that can reduce DNN generalization through overfitting.The paper motivates understanding noisy-label effects as a prerequisite for robust training.
Related work: Existing approaches estimate noise matrices, select or weight samples, or correct labels, but face inaccurate estimation, unreliable selection, or overfitting.Some methods also require prior class-distribution knowledge that is often unavailable in practice.
Contributions: The paper quantitatively relates DNN generalization performance to label noise and shows that test accuracy is quadratic in the noise ratio under symmetric noise.This analysis explains previously reported empirical dependence on noise ratio.
Contributions: Cross-validation is applied to randomly split noisy data to identify samples with correct labels, followed by Co-teaching to train DNNs against noisy labels.The strategy uses identified samples to address Co-teaching degradation at higher noise ratios and unavailable noise-ratio settings.
Evaluation: Experiments on synthetic CIFAR-10 noise and real-world WebVision noise achieve the best clean-test-set accuracy compared with state-of-the-art methods.WebVision contains 2.4 million images crawled from websites with real-world noisy labels.

2. Preliminaries

The paper models classification datasets with observed labels that may be corrupted relative to true labels. It represents this corruption using a noise transition matrix and randomly splits noisy data for cross-validation.

The dataset consists of input samples paired with observed labels in a c-class classification problem.
A noise transition matrix T represents the probability that a true class-i example receives observed label j.
Cross-validation randomly splits the noisy dataset into two parts for subsequent training and evaluation.

3. Understanding DNNs trained with noisy labels

The paper characterizes how noisy labels affect DNN generalization through the noise transition matrix, then uses noisy cross-validation to select samples whose labels agree with model predictions. It analyzes symmetric and asymmetric noise and reports that the resulting accuracy equation fits prior experiments.

DNNs can fit noisy or random training labels while still exhibiting large generalization error.
Under i.i.d. inputs and sufficiently high capacity, a network trained on one noisy split predicts classes on the other split according to the shared noise transition matrix T.
The prediction and test-label distributions match, including when the model is evaluated on a clean test set with true labels.
Test accuracy is defined as the proportion of test examples for which the prediction equals the observed label and is formulated per class under the shared-noise assumptions.
The analysis considers symmetric and asymmetric noise, with symmetric noise distributing corruption across incorrect classes and asymmetric noise targeting one incorrect class.
Noisy Cross-Validation trains on each half, selects correctly predicted samples from the other half, and unions the selected sets into S.
The test accuracy under symmetric and asymmetric noise can be quantified by the noise ratio ε, and Eq. (4) fits prior experimental accuracy results while enabling noise-ratio estimation.

4. Training DNNs against noisy labels

The method first identifies a lower-noise subset through noisy cross-validation, then uses iterative selection and a modified Co-teaching procedure to train robustly with additional candidate samples.

Noisy Cross-Validation: Noisy Cross-Validation identifies samples as clean when their observed labels match network predictions, while keeping them outside the training subset used for evaluation.
Selection Metrics: Label Precision measures the clean fraction of the selected subset, whereas Label Recall measures the fraction of all clean samples that it contains.The selected subset’s noise ratio is εS = 1 − LP.
Selection Guarantees: Under practical transition-matrix conditions, the selected subset has a smaller noise ratio than the original dataset.This follows from εS < ε when each diagonal transition probability is largest.
Iterative Selection: Iterative Noisy Cross-Validation expands the selected set across iterations while removing candidate samples with large categorical cross-entropy loss.The procedure maintains selected and candidate sets and outputs an estimated noise ratio.
Robust Training: The training procedure uses two networks that initially focus on selected samples, then incorporate candidates and exchange small-loss samples through Co-teaching.This design is intended to improve training stability and test accuracy while using more training data.

5. Experiments

Experiments verify the theory on synthetic CIFAR-10 noise, evaluate INCV’s clean-sample identification, and show robust training against synthetic and real-world noisy labels.

Behavior of DNNs trained with noisy labels: Experiments compare theoretical and empirical test accuracy, label precision, and label recall under symmetric and asymmetric CIFAR-10 noise.Cross-validation splits the noisy data into two halves, with ResNet-110 trained on one half and tested on the other.
Behavior of DNNs trained with noisy labels: The test accuracy is a quadratic function of the noise ratio under symmetric noise, and the experiments are consistent with the theoretical estimates.The same experiments also closely match the proposed formulas for label precision.
Behavior of DNNs trained with noisy labels: The trained model’s confusion matrix approximately matches the noise transition matrix, supporting Claim 1’s prediction behavior.This result is shown for ResNet-110 trained on CIFAR-10 with symmetric noise ratio 0.7.
Identifying more clean samples by the INCV: After four INCV iterations under symmetric noise ratio 0.5, about 90% of clean samples are selected while the selected set’s noise ratio falls to around 10%.Across experiments, label recall increases substantially with iteration while label precision slightly decreases.
Identifying more clean samples by the INCV: INCV also identifies confusing mislabeled examples in original CIFAR-10, indicating that noisy labels occur even in that dataset.The paper notes that these corrupted samples are rare and have negligible influence on training.
Training DNNs robustly against noisy labels: On manually corrupted CIFAR-10, the proposed method achieves the best test accuracy across all tested noise types and ratios, including symmetric noise ratio 0.8.On WebVision’s real-world noisy labels, it consistently outperforms other state-of-the-art methods on the reported validation sets.

6. Conclusion

The paper formalizes how noisy labels affect DNN generalization, develops INCV to identify clean samples, and combines it with Co-teaching for robust training.

The work initiates a formal study of DNNs trained with noisy labels through theoretical analysis and extensive experiments.
INCV randomly divides noisy datasets and uses cross-validation to identify clean samples, with theoretical guarantees.
Experiments show that INCV can identify most clean samples accurately.
The method uses Co-teaching to exploit identified samples for robust training against noisy labels.
Compared with extensive baselines, the method achieves state-of-the-art test accuracy on the clean test set.

Supplementary Materials: Understanding and Utilizing Deep Neural Networks

The supplementary material is titled “Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels.”

The supplementary material concerns understanding DNNs trained with noisy labels.
Its stated focus includes utilizing DNNs trained with noisy labels.
The supplementary material is associated with the paper’s broader study of DNNs and noisy labels.

A. Further details on experiments

Experiments use synthetic CIFAR-10 noise, high-capacity ResNets for theory verification, and INCV-based clean-sample selection with specified training settings.

CIFAR-10 labels are randomly corrupted according to a noise transition matrix to create synthetic noisy labels.
ResNet-110 is used to verify the theory under a worst-case setting where the model can memorize corrupted samples.
ResNet-32 is used in later experiments, including INCV-based clean-sample selection.
INCV uses four iterations and trains ResNet-32 for 50 epochs per iteration with Adam and a scheduled learning rate.
After clean-sample selection, robust training uses warm-up periods of 40 or 80 epochs depending on candidate-set size.
Baseline methods use their original hyperparameters and the same ResNet-32 training schedule for comparison.

A.2. WebVision

WebVision experiments evaluate real-world noisy labels using a 50-class subset, an Inception-ResNet-v2 model, and INCV followed by robust training.

WebVision contains 2.4 million web-crawled images with real-world noisy labels, while experiments use the first 50 Google-image classes.
The WebVision model is Inception-ResNet-v2 trained with standard resizing, cropping, flipping, and a batch size of 128.
The method runs INCV for two iterations, trains 50 epochs per iteration, and sets the removal ratio to 0.1.
For low-training-accuracy noise settings, confusion matrices compare normally trained ResNet-110 predictions with the noise transition matrix.
Figure examples show automatically identified WebVision noisy labels whose labeled concepts are obviously unreasonable.

B. More plots of the confusion matrix

Across multiple noise types and ratios, the test-set confusion matrix matches the noise transition matrix, including settings with very low training accuracy.

M ≈ T holds for asymmetric noise at ratio 0.4 and symmetric noise ratios 0.8, 0.9, and 1.0.For symmetric ratios 0.8, 0.9, and 1.0, the corresponding training accuracies are 0.40, 0.24, and 0.36.
The observed equality between the test confusion matrix and noise transition matrix verifies that DNNs can fit noisy training sets exactly and generalize in distribution.
Extremely low training accuracy does not contradict the paper’s formulations of generalization performance under noisy labels.

C. The INCV automatically identifies many noisy labels in the WebVision dataset

INCV selects cleaner samples from noisy data, identifies likely mislabeled WebVision examples, and is theoretically guaranteed to reduce the selected subset’s noise ratio under stated assumptions.

INCV selects most clean samples from WebVision and identifies samples very likely to have wrong labels.The selected clean samples support training a model achieving the best generalization performance on the clean validation set compared with state-of-the-art methods.
Under symmetric and asymmetric noise, Alg. 1 selects a subset with smaller noise ratio than the original dataset when each diagonal transition probability is largest.The condition is T_ii being the largest among T_ij for each class i.
For symmetric noise, the largest-diagonal condition requires ε/(c − 1) < 1 − ε.
For asymmetric noise, the largest-diagonal condition requires ε < 1 − ε.
The selected-set noise ratio satisfies εS < ε under the stated symmetric and asymmetric noise conditions.Here ε is the original dataset’s noise ratio and εS is the selected set’s noise ratio.