Source-linked AI summary
Combating noisy labels by agreement: A joint training method with co-regularization
Hongxin Wei, Lei Feng, Xiangyu Chen, Bo An
TL;DR
Noisy labels are difficult because efficient annotation procedures can reduce deep-network performance. JoCoR jointly trains two networks with Co-Regularization and small-loss selection, and experiments across benchmark datasets report robustness superior to many state-of-the-art approaches.
Problem
Efficient annotation methods can produce noisy labels, which noticeably reduce deep neural network performance and complicate learning with weak supervision.
Method
JoCoR jointly trains two differently initialized networks with supervised and Co-Regularization losses, selecting small-loss examples to update both simultaneously.
Results
JoCoR is reported as more robust than many state-of-the-art approaches on MNIST, CIFAR-10, CIFAR-100, and Clothing1M.
Takeaways & Limitations
Ablation studies report that both Co-Regularization and Joint Training contribute to robust learning with slightly and extremely noisy supervision.
Takeaways & Limitations
Co-teaching+ can use very few examples under extremely high noise, while JoCoR’s loss weight is tuned using a clean validation set or a selected clean subset.
Abstract
from arXiv · showhide
Deep Learning with noisy labels is a practically challenging problem in weakly supervised learning. The state-of-the-art approaches "Decoupling" and "Co-teaching+" claim that the "disagreement" strategy is crucial for alleviating the problem of learning with noisy labels. In this paper, we start from a different perspective and propose a robust learning paradigm called JoCoR, which aims to reduce the diversity of two networks during training. Specifically, we first use two networks to make predictions on the same mini-batch data and calculate a joint loss with Co-Regularization for each training example. Then we select small-loss examples to update the parameters of both two networks simultaneously. Trained by the joint loss, these two networks would be more and more similar due to the effect of Co-Regularization. Extensive experimental results on corrupted data from benchmark datasets including MNIST, CIFAR-10, CIFAR-100 and Clothing1M demonstrate that JoCoR is superior to many state-of-the-art approaches for learning with noisy labels.
1. Introduction
Noisy labels arise because scalable annotation methods are efficient but often inaccurate, reducing deep neural network performance. JoCoR addresses this challenge by jointly training two networks with Co-Regularization and small-loss selection, and experiments report robustness across several noisy datasets.
- Crowdsourcing and online queries improve labeling efficiency but introduce unavoidable noisy labels.
- Noisy labels have been shown to noticeably decrease the performance of deep neural networks.
- JoCoR trains two networks with a joint loss combining supervised learning and Co-Regularization.
- JoCoR uses the joint loss to select small-loss examples, preventing biased-selection error flow from accumulating in one network.
- Experiments on MNIST, CIFAR-10, CIFAR-100, and Clothing1M report that JoCoR is more robust than many state-of-the-art approaches.
2. Related work
Prior noisy-label methods estimate transition matrices, select small-loss samples, or update models on prediction disagreements. JoCoR instead uses agreement-oriented joint training, while related methods can face challenges in transition estimation or use too few examples under extreme noise.
- Noise rate estimation: Noise-transition methods rely on estimating label corruption, but accurate transition-matrix estimation is challenging, especially with many classes.
- Small-loss selection: Small-loss selection treats low-loss samples as clean because networks tend to learn simple patterns before memorizing all samples.
- Disagreement: Decoupling updates two networks only on examples where their predictions disagree.
- Disagreement: Co-teaching+ sends small-loss examples with disagreeing predictions from each network to its peer network.
- Agreement: JoCoR trains two networks together with a joint loss that brings their predictions closer to the ground-truth labels and to each other.
- Disagreement: Co-teaching+ may use very few training examples per mini-batch under extremely high noise because it requires both small loss and prediction disagreement.
- Agreement: Agreement maximization in multi-view learning motivates JoCoR’s use of peer-network agreement for noisy-label learning.
3. The Proposed Approach
JoCoR trains two classifiers jointly with supervised and agreement-based losses, selects small-loss examples, and updates both networks simultaneously. The approach combines joint training, Co-Regularization, and small-loss selection to address noisy labels.
- Joint Training: JoCoR trains two classifiers with one joint loss that includes a regularization term reducing prediction divergence.The classifiers retain different parameters but are updated simultaneously.
- Joint Loss: The supervised component combines the classification losses of both networks, while the contrastive component provides Co-Regularization.For multi-class classification, the supervised component uses cross-entropy loss, and the contrastive term measures prediction agreement.
- Small-Loss Selection: The selection rate starts higher and decreases across epochs until reaching 1 −τ, reflecting the networks’ tendency to fit clean data first.The schedule controls how many small-loss examples are retained in each training epoch.
- Small-Loss Selection: JoCoR selects small-loss examples using the joint loss because low joint loss indicates that both networks can reach consensus on an instance.The selected examples are then averaged for backpropagation, and both networks are updated from that loss.
- Relations to Other Approaches: Unlike disagreement-based methods, JoCoR combines small-loss selection with joint training and agreement maximization through regularization.The comparison distinguishes JoCoR from Decoupling, Co-teaching, and Co-teaching+ by its update and regularization choices.
4. Experiments
Experiments on simulated and real-world noisy-label benchmarks compare JoCoR with state-of-the-art methods, using test accuracy and label precision to assess robustness. JoCoR consistently performs strongly across datasets, while ablations support the roles of Co-Regularization and Joint Training.
- Results on MNIST: JoCoR consistently achieves higher MNIST accuracy than all baselines across four noise settings.On the hardest Symmetry-80% case, JoCoR reaches the best average classification accuracy of 84.89%.
- Results on CIFAR-10: On CIFAR-10, JoCoR performs best in all four cases and outperforms competing approaches on both test accuracy and label precision.Co-teaching can lead early in one asymmetric-noise setting, but JoCoR surpasses it in later epochs.
- Results on Clothing1M: On Clothing1M, JoCoR improves final accuracy by +5.11 over Standard and by +1.28 over the best baseline method.The reported comparison distinguishes validation-selected best-epoch scores from scores at the end of training.
- Ablation study: Ablation studies show that Co-Regularization hinders memorization of noisy labels, while Joint Training provides a more efficient training paradigm.Joint-only is comparable to Co-teaching on test accuracy and better on label precision in the cited CIFAR-10 analysis.
5. Conclusion
JoCoR improves robustness to noisy labels by jointly training two classifiers with supervised and Co-Regularized losses, while selecting small-loss instances. Experiments across four datasets and ablations support the effectiveness of Co-Regularization and Joint Training.
- JoCoR trains two classifiers simultaneously with one joint loss combining supervised and Co-Regularized components.
- JoCoR selects small-loss instances in each mini-batch to update both networks using the joint loss.
- Experiments on MNIST, CIFAR-10, CIFAR-100 and Clothing1M demonstrate robust training under slightly and extremely noisy supervision.
- Ablation studies demonstrate the effectiveness of Co-Regularization and Joint Training.
A. Dataset
The experiments use four benchmark datasets for evaluating learning with noisy labels, with detailed dataset characteristics provided in supplementary materials and summarized in Table 6.
- Detailed characteristics of the datasets are shown in Table 6.
B. Network Architecture
The paper uses MLP and CNN network architectures, with the models used on MNIST, CIFAR-10 and CIFAR-100 documented in Table 7.
- The network architectures consist of MLP and CNN models.
- Table 7 lists the models used on MNIST, CIFAR-10 and CIFAR-100.
C. Parameter Sensitivity Analysis
The sensitivity analysis varies λ on MNIST with symmetry-50% noise to examine its effect on classifier divergence and performance. Larger λ reduces divergence, and λ = 0.95 gives the best reported performance.
- The analysis compares λ values of 0.05, 0.35, 0.65 and 0.95 on MNIST with symmetry-50% noise.
- Larger λ produces less divergence between the two JoCoR classifiers.
- As λ increases, JoCoR’s test accuracy improves, with λ = 0.95 achieving the best performance.
- Label precision follows the same increasing trend as λ.