Source-linked AI summary
How does Disagreement Help Generalization against Label Corruption?
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W. Tsang, Masashi Sugiyama
TL;DR
Noisy labels challenge learning because deep networks can memorize corrupted supervision, while Co-teaching’s two networks can eventually reach consensus. The paper proposes Co-teaching+, which filters disagreement data, selects small-loss examples, and cross-updates peer networks; experiments report superior robustness across noisy-label benchmarks. Its scope includes the stated assumption that the noise rate is known, and the paper notes that disagreement updating alone cannot handle noisy labels.
Problem
Noisy labels degrade classifier accuracy, and Co-teaching’s networks can converge over training, creating a need to preserve useful divergence.
Method
Co-teaching+ keeps prediction-disagreement data, selects small-loss examples among them, and updates each network using its peer’s selected data.
Results
Co-teaching+ was more robust than many state-of-the-art methods, including Co-teaching, MentorNet, and F-correction, across simulated and real-world noisy datasets.
Takeaways & Limitations
Robust noisy-label training combines small-loss selection, cross-updating between networks, and maintaining divergence throughout training.
Takeaways & Limitations
The paper states that Update by Disagreement alone cannot handle noisy labels, and its experimental initialization assumes the noise rate τ is known.
Abstract
from arXiv · showhide
Learning with noisy labels is one of the hottest problems in weakly-supervised learning. Based on memorization effects of deep neural networks, training on small-loss instances becomes very promising for handling noisy labels. This fosters the state-of-the-art approach "Co-teaching" that cross-trains two deep neural networks using the small-loss trick. However, with the increase of epochs, two networks converge to a consensus and Co-teaching reduces to the self-training MentorNet. To tackle this issue, we propose a robust learning paradigm called Co-teaching+, which bridges the "Update by Disagreement" strategy with the original Co-teaching. First, two networks feed forward and predict all data, but keep prediction disagreement data only. Then, among such disagreement data, each network selects its small-loss data, but back propagates the small-loss data from its peer network and updates its own parameters. Empirical results on benchmark datasets demonstrate that Co-teaching+ is much superior to many state-of-the-art methods in the robustness of trained models.
1. Introduction
Noisy labels can degrade classifier accuracy, especially because deep networks may memorize corrupted labels. Co-teaching uses peer-selected small-loss examples, but its networks can converge; Co-teaching+ combines disagreement-based filtering with cross-updating to preserve divergence.
- Deep networks can fully memorize noisy labels, worsening the accuracy degradation caused by corrupted supervision.
- Small-loss training became a promising approach, including MentorNet and Co-teaching, for selecting examples believed to be cleaner.MentorNet uses an extra network for selection, whereas Co-teaching trains two networks symmetrically and exchanges selected examples.
- Co-teaching’s initially diverged networks gradually approach consensus, reducing its distinction from self-paced MentorNet.The consensus issue motivates maintaining or slowing the loss of divergence during training.
- Co-teaching+ first retains examples on which two networks disagree, then selects small-loss examples and cross-updates each network with its peer’s selections.This combines a disagreement-update step for data with a cross-update step for parameters.
- Experiments on simulated and real-world noisy datasets found Co-teaching+ more robust than several state-of-the-art methods, including Co-teaching and MentorNet.The evaluated datasets include noisy MNIST, CIFAR-10, CIFAR-100, NEWS, T-ImageNet, and three Open-sets.
- The paper identifies small-loss selection, cross-updating, and keeping the two networks diverged as three factors in robust noisy-label training.
2. Co-teaching+: Towards Training of Robust Deep Networks with Noisy Labels
Co-teaching+ combines disagreement-based filtering with Co-teaching’s small-loss and peer-update mechanisms to keep two networks divergent during training. Its schedule gradually reduces the retained small-loss data as networks begin fitting noisy labels.
- Co-teaching+ algorithm: Co-teaching+ first retains prediction-disagreement data, then each network selects small-loss instances and updates itself using its peer’s selected data.The algorithm combines a disagreement-update step with a cross-update step.
- Motivation: Disagreement-based updates maintain divergence by restricting further training to data on which the two classifiers disagree.This is intended to prevent Co-teaching from reducing to self-training MentorNet.
- Motivation: Co-teaching+ is motivated by complementary peer learning, in which diverged peers can provide more useful signals than identical peers.The paper illustrates this with students who have complementary strengths in different subjects.
- Training schedule: The small-loss retention rate decreases over epochs because deep networks fit clean data before gradually over-fitting noisy data.Co-teaching+ decreases λ(e) quickly early and then either holds it constant or decreases it more slowly after Ek epochs.
- Motivation: Co-teaching can converge to consensus and functionally reduce to self-training MentorNet as training epochs increase.The initial divergence comes from different parameter initializations, but it diminishes over training.
- Key factors: The paper identifies three factors for handling noisy labels: exploiting memorization, cross-updating two networks, and keeping them divergent.These factors correspond to the small-loss trick, simultaneous peer learning, and disagreement-based training.
3. Experiments on Simulated Noisy Datasets
Experiments evaluate Co-teaching+ on simulated noisy datasets using standard baselines, multiple architectures, and test accuracy over training. Across MNIST, CIFAR-10, CIFAR-100, NEWS, and T-ImageNet, Co-teaching+ generally remains more robust, especially as noise becomes harder.
- Experimental setup: Experiments cover MNIST, CIFAR-10, CIFAR-100, NEWS, and T-ImageNet, using simulated symmetry-flipping and pair-flipping label noise.The evaluation includes vision and text datasets, with benchmark methods implemented using default parameters.
- Experimental setup: The comparisons include MentorNet, Co-teaching, Decoupling, F-correction, and a standard network trained directly on noisy data.These baselines represent small-loss selection, peer updating, disagreement-based updating, transition-matrix correction, and direct noisy-label training.
- Results: Standard training first achieves high test accuracy and then declines as the network overfits noisy labels, revealing the memorization effect.This trend appears in the MNIST, CIFAR-10, CIFAR-100, and NEWS experiments.
- Results: Co-teaching+ outperforms the competing methods on the easiest CIFAR-10 and CIFAR-100 symmetry-noise cases.On CIFAR-10, it performs much better than all baselines under Symmetry-20%; on CIFAR-100, it and F-correction outperform Co-teaching, MentorNet, and Decoupling.
- Results: On T-ImageNet, Co-teaching+ is best in both symmetry cases, while Co-teaching and Co-teaching+ outperform other methods in the pair case.The comparison uses averaged and maximal test accuracy over the last 10 epochs.
4. Experiments on Real-world Noisy Datasets
The experiments evaluate Co-teaching+ on open-set noisy datasets and report stronger robustness than competing methods. They also identify maintaining disagreement between networks as a key ingredient of robust training.
- Dataset: Open-sets replace some CIFAR-10 training images with outside images while preserving labels and per-class image counts.The outside images come from CIFAR-100, ImageNet-32, and SVHN.
- Network & Optimizer & Initialization: The experiments use a six-convolutional-layer network with one fully connected layer, batch normalization, ReLU activations, max pooling, and SGD.
- Results on three Open-sets: Co-teaching+ outperforms other baselines on all three open-set noisy datasets with 40% open-set noise.The reported comparisons include Standard, Iterative, MentorNet, and Co-teaching.
- Results on three Open-sets: Co-teaching and Co-teaching+ outperform Iterative on CIFAR-100 and ImageNet-32 noise, while Co-teaching+ is significantly better than Iterative on SVHN noise.
- Reflection of results: Co-teaching+ is framed as cross-training two diverged deep networks with the small-loss trick, and network divergence is identified as a key ingredient of robustness.The paper links this conclusion to the result difference between Co-teaching and Co-teaching+.
5. Conclusion
The paper presents Co-teaching+ as a robust learning paradigm for training deep networks under noisy supervision. Its central procedure combines prediction-disagreement selection with peer cross-training on small-loss data.
- Co-teaching+ maintains two networks that identify prediction-disagreement data and cross-trains on samples screened by the small-loss criterion.
- The paper reports that Co-teaching+ trains deep models robustly with extremely noisy supervision beyond Co-teaching and MentorNet.
- The method is evaluated empirically to demonstrate robust training under noisy supervision.
A. Related literature
The related literature spans statistical learning and deep learning approaches to noisy-label learning. Statistical methods are organized into surrogate loss, noise-rate estimation, and probabilistic modeling.
- Statistical learning methods: Statistical learning approaches to noisy labels are categorized into surrogate loss, noise rate estimation, and probabilistic modeling.
- Statistical learning methods: Surrogate-loss methods include unbiased noise-corrected losses and robust non-convex losses.
- Statistical learning methods: Noise-rate estimation methods use class-probability estimators based on score order statistics or ROC-curve slopes.
- Statistical learning methods: Probabilistic modeling methods include two-coin models for noisy labels from multiple annotators and extensions with instance-dependent flipping probabilities.
B. Training details
Training schedules vary by dataset: Co-teaching+ starts immediately on MNIST and NEWS, while other datasets use Co-teaching warm-up periods before disagreement updates.
- Training schedule: Co-teaching+ begins training immediately on MNIST and NEWS but uses warm-up strategies on the other datasets.
- Training schedule: CIFAR-10 uses 20 epochs of Co-teaching warm-up, and CIFAR-100 uses 5 epochs before Co-teaching+ training.
- Training schedule: For T-ImageNet, disagreement updates begin in the middle of training.