Source-linked AI summary
Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, Masashi Sugiyama
TL;DR
Noisy labels are difficult for high-capacity deep networks because estimating label noise is challenging and models eventually memorize incorrect labels. Co-teaching trains two networks that exchange small-loss instances, achieving superior robustness across noisy MNIST, CIFAR-10, and CIFAR-100 experiments. Its experimental schedule assumes the noise level is known, or inferred using validation sets when it is unavailable.
Problem
Estimating noise transition matrices is difficult, especially with many classes, while deep networks eventually overfit noisy labels.
Method
Co-teaching trains two networks that select small-loss instances in each mini-batch and update themselves using instances selected by their peers.
Results
Co-teaching is much more robust than state-of-the-art baselines under 45% noisy labels and remains superior to most baselines under 20% noise across noisy MNIST, CIFAR-10, and CIFAR-100.
Takeaways & Limitations
Cross-training on peer-selected small-loss instances supports robust deep-model training under extremely noisy supervision.
Takeaways & Limitations
The experimental setup assumes the noise level is known, although validation sets can be used to infer it when unavailable.
Abstract
from arXiv · showhide
Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.
1 Introduction
Noisy labels are common and particularly difficult for high-capacity deep networks, while existing approaches struggle to estimate noise accurately or prevent error accumulation. Co-teaching addresses this by having two networks exchange small-loss instances, and experiments show superior robustness across noisy benchmarks.
- Noisy labels corrupt ground-truth labels and degrade the robustness of learned models, especially deep neural networks.
- Estimating the noise transition matrix is difficult, particularly when the number of classes is large.
- MentorNet and Decoupling select training instances to reduce noise, but Decoupling cannot explicitly handle noisy labels within its disagreement area.
- Deep networks first memorize clean, easy patterns and later memorize noisy labels, causing poor generalization.
- Co-teaching trains two networks that select small-loss instances and teach them to each other for parameter updates.
- Under 45% noisy labels, Co-teaching is much more robust than state-of-the-art baselines, and under 20% noise it remains superior to most baselines.
2 Related literature
Related work includes statistical methods based on surrogate losses, noise-rate estimation, and probabilistic modeling, alongside deep-learning approaches for noisy-label learning. The Co-teaching algorithm operationalizes cross-updating through two networks and small-loss instance selection.
- Statistical learning methods: Statistical learning methods address noisy labels through surrogate loss, noise-rate estimation, and probabilistic modeling.
- Co-teaching algorithm: Co-teaching shuffles the noisy dataset, forms mini-batches, selects R(T)% small-loss instances in each network, and cross-updates the networks with peer-selected data.
- Statistical learning methods: Surrogate-loss methods include unbiased noise-corrected estimators and robust non-convex losses.
- Statistical learning methods: Noise-rate estimation methods use class-probability estimators based on order statistics or ROC-curve slopes.
- Statistical learning methods: Probabilistic modeling methods include two-coin models for noisy labels from multiple annotators and instance-dependent extensions.
3 Co-teaching meets noisy supervision
Co-teaching uses two networks to select and exchange small-loss instances in each mini-batch. Its design relies on early clean-pattern learning, gradual reduction of retained data, and differing network abilities to filter noise.
- Each network selects a proportion R(T) of small-loss instances from every mini-batch and supplies those instances to its peer.
- The method asks why dynamic small-loss sampling identifies clean instances and why two networks should cross-update parameters.
- Small-loss instances are more likely to be correctly labeled, making their use resistant to noisy labels when the classifier is reliable.
- Because networks initially learn clean patterns but later overfit noisy labels, R(T) starts large and becomes smaller over training.
- Two classifiers can reduce sensitivity to wrongly selected instances because they develop different decision boundaries and noise-filtering abilities.
- Unlike Co-training, Co-teaching uses a single feature view, exploits deep-network memorization, and targets noisy-label learning rather than semi-supervised learning.
4 Experiments
Experiments on noisy MNIST, CIFAR-10, and CIFAR-100 evaluate Co-teaching under symmetric and pair-flipping noise, including extremely noisy settings. Across the harder cases, Co-teaching generally best preserves test accuracy and selects cleaner instances than the compared methods.
- Experimental setup: Experiments use noisy versions of MNIST, CIFAR-10, and CIFAR-100, with manually corrupted labels generated by symmetric-flipping and pair-flipping transition matrices.The main experiments use noise rates of 45% and 50%, with an additional 20% low-noise setting.
- Experimental setup: The comparison includes Bootstrap, S-model, F-correction, Decoupling, MentorNet, and Co-teaching under a common CNN and optimizer setup.The experiments use a 9-layer CNN with Leaky-ReLU activations and Adam optimization.
- MNIST results: 87.53% test accuracy for Co-teaching versus 80.88% for MentorNet on MNIST Pair-45% makes Co-teaching the best method in that hardest case.Several competing methods cannot learn effectively in this setting, while Co-teaching remains substantially better than the second method.
- MNIST results: Co-teaching consistently achieves higher test accuracy than MentorNet on harder MNIST cases and is the best method for Pair-45% and Symmetry-50%.The accuracy curves show standard training first improves and then declines as networks memorize noisy labels.
- CIFAR-10 results: On CIFAR-10 Pair-45%, Co-teaching achieves at least 14% higher test accuracy than MentorNet, while outperforming methods on the harder noise cases.Co-teaching is comparable with F-correction in the easiest Symmetry-20% case, but is strongest in Pair-45% and Symmetry-50%.
- CIFAR-100 results: On CIFAR-100, the observations match the other datasets: Co-teaching is best on the harder noisy cases, while its label-selection behavior is explained through label precision curves.Label precision measures the fraction of clean labels among selected small-loss instances; higher precision indicates fewer noisy instances after selection.
5 Conclusion
The paper presents Co-teaching as a two-network paradigm that trains deep models robustly under noisy supervision, while identifying broader extensions and theoretical analysis as future work.
- Co-teaching cross-trains two networks on instances screened by the small-loss criterion.
- Simulated experiments show Co-teaching can train deep models robustly with extremely noisy supervision.
- Future work includes adapting Co-teaching to other weak supervisions and investigating theoretical guarantees and generalization analysis.
A Definition of noise
The noise transition matrix Q represents the conditional probability of observing a noisy label given the clean label, with n denoting the number of classes.
- n denotes the number of classes in the transition-matrix definition.
B.1 MNIST
The MNIST results track test accuracy and label precision across training epochs.
- MNIST results report test accuracy versus the number of epochs.
- MNIST results also report label precision versus the number of epochs.
B.2 CIFAR-10
The CIFAR-10 results track test accuracy and label precision across training epochs.
- CIFAR-10 results report test accuracy versus the number of epochs.
- CIFAR-10 results also report label precision versus the number of epochs.
B.3 CIFAR-100
Figure 9 presents CIFAR-100 results by tracking test accuracy and label precision across training epochs, with multiple baseline and Co-teaching variants listed for comparison.
- The comparison includes Standard, Bootstrap, S-model, F-correction, Decoupling-1, Decoupling-2, MentorNet, Co-teaching-1, and Co-teaching-2.
- Figure 9 plots test accuracy against the number of epochs on CIFAR-100.
- The figure also plots label precision against the number of epochs.