Source-linked AI summary
Jo-SRC: A Contrastive Approach for Combating Noisy Labels
Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang, Zhenmin Tang
TL;DR
Noisy labels can be memorized by DNNs, degrading performance, while small-loss selection within mini-batches overlooks varying noise ratios and high-loss information. Jo-SRC performs global clean-sample selection, distinguishes ID/OOD samples through two-view consistency, and trains with a joint classification-consistency loss. It significantly outperforms state-of-the-art methods on synthetic and real-world noisy datasets.
Problem
DNNs can memorize noisy labels, while mini-batch small-loss selection overlooks fluctuating noise ratios and requires discarding other samples.
Method
Jo-SRC uses two image views, Jensen-Shannon divergence for global clean selection, prediction consistency for ID/OOD distinction, and a joint loss.
Results
Jo-SRC significantly outperforms state-of-the-art methods on synthetic and real-world noisy datasets.
Takeaways & Limitations
Jo-SRC selects clean, ID noisy, and OOD samples for subsequent learning while combining classification with consistency regularization.
Takeaways & Limitations
Noise-tolerant loss functions can fail when the noise ratio is high.
Abstract
from arXiv · showhide
Due to the memorization effect in Deep Neural Networks (DNNs), training with noisy labels usually results in inferior model performance. Existing state-of-the-art methods primarily adopt a sample selection strategy, which selects small-loss samples for subsequent training. However, prior literature tends to perform sample selection within each mini-batch, neglecting the imbalance of noise ratios in different mini-batches. Moreover, valuable knowledge within high-loss samples is wasted. To this end, we propose a noise-robust approach named Jo-SRC (Joint Sample Selection and Model Regularization based on Consistency). Specifically, we train the network in a contrastive learning manner. Predictions from two different views of each sample are used to estimate its "likelihood" of being clean or out-of-distribution. Furthermore, we propose a joint loss to advance the model generalization performance by introducing consistency regularization. Extensive experiments have validated the superiority of our approach over existing state-of-the-art methods.
1. Introduction
Noisy labels are difficult because DNNs can memorize them, while existing sample-selection methods use fixed within-mini-batch proportions and discard useful noisy samples. Jo-SRC addresses these issues with global selection, consistency-based ID/OOD handling, and a joint loss.
- Motivation: Web-scale images are easier to obtain but often carry noisy labels, which can cause DNNs to overfit and degrade performance.The problem is especially relevant where reliable annotation is costly or requires expertise.
- Limitations of prior methods: Existing methods select a human-defined proportion of small-loss samples within each mini-batch, despite fluctuating noise ratios across mini-batches.Global selection over an entire huge dataset is impractical when losses must be recorded for every sample.
- Jo-SRC: Jo-SRC trains with a joint loss combining cross-entropy and consistency terms to improve classification and generalization performance.The approach is motivated by self-supervised contrastive learning and uses two views of each image.
- Jo-SRC: Jo-SRC estimates clean-sample likelihood globally with Jensen-Shannon divergence and distinguishes ID from OOD noisy samples using prediction consistency.ID and OOD noisy samples are relabeled by a mean-teacher network before network updates.
- Results: Jo-SRC significantly outperforms state-of-the-art methods on both synthetic and real-world noisy datasets.The paper also reports extensive ablation studies validating the approach.
2. Related Works
Prior noisy-label research centers on loss correction and sample selection, but both approaches face practical or robustness challenges. Jo-SRC uses two augmented views, divergence-based clean selection, consistency-based OOD identification, and mean-teacher relabeling.
- Research directions: Existing noisy-label methods are categorized into loss correction and sample selection approaches.These represent the two main directions described in the related-work discussion.
- Loss Correction: Loss-correction methods may require estimating a noise transition matrix, which is difficult without prior knowledge or well-labeled data.Noise-tolerant loss functions can also fail when the noise ratio is high.
- Sample Selection: Sample-selection methods exploit DNN memorization by treating lower-loss samples as more likely to have clean labels.Examples include Co-teaching, which uses one network’s selected small-loss samples to train another.
- Jo-SRC framework: Jo-SRC augments each image into two views and predicts corresponding probability distributions for its selection modules.The framework includes overall training, clean-sample selection, and ID/OOD sample selection components.
- Jo-SRC framework: Jo-SRC uses Jensen-Shannon divergence to estimate clean likelihood, prediction disagreement to estimate OOD likelihood, and mean-teacher relabeling before optimization.Its final objective combines classification and consistency terms.
3. The Proposed Method
Jo-SRC globally selects clean samples using bounded Jensen-Shannon divergence, distinguishes ID and OOD noisy samples through prediction consistency, and reassigns labels before training. It combines these procedures with consistency regularization and dynamic thresholding to use noisy data more effectively.
- 3.1. Global clean sample selection: A sample is selected as clean when its clean likelihood Pclean(x) exceeds the threshold τclean.
- 3.1. Global clean sample selection: Jo-SRC measures clean-sample likelihood with Jensen-Shannon divergence between predicted probabilities and label distributions.Because JS divergence is bounded in [0, 1], it supports global rather than mini-batch-relative selection.
- 3.2. Out-of-distribution detection: For samples judged unclean, prediction disagreement between two augmented views identifies OOD noise, while consistency identifies ID noise.OOD samples have disagreement above τood; consistent predictions indicate ID noisy samples.
- 3.3. Label re-assignment: Jo-SRC preserves clean labels, uses mean-teacher pseudo-label distributions for ID samples, and assigns approximately uniform targets to OOD samples.The OOD target is smoothed with a scaling constant, empirically set to s = 10.
- 3.4. Consistency regularization: Consistency regularization encourages agreement for clean and ID samples while increasing prediction divergence for OOD samples.The auxiliary consistency loss also implicitly promotes self-supervised representation learning.
- 3.5. The overall framework: The clean-selection threshold increases during training, initially admitting more samples and later restricting clean status to improve label quality.The maximum threshold τm is empirically set to 0.95.
4. Experiments
Jo-SRC is evaluated on synthetic and real-world noisy-label datasets using test accuracy and sample-selection precision. Across these settings, it generally outperforms competing methods, while ablations examine the roles of clean selection, noisy-sample treatment, and consistency regularization.
- Experiment setup: Jo-SRC is evaluated on CIFAR100N-C, CIFAR80N-O, Clothing1M, and Food101N, with test accuracy and label precision as evaluation metrics.CIFAR100N-C and CIFAR80N-O are synthetic datasets; Clothing1M and Food101N represent real-world noisy-label settings.
- Synthetic noisy datasets: On CIFAR80N-O, Jo-SRC consistently outperforms other methods across noise settings, including the challenging Symmetry-80% case.Performance drops as noise becomes harder, but Jo-SRC remains comparatively effective; all approaches fail to fully combat massive noise in Symmetry-80%.
- Real-world noisy datasets: Jo-SRC achieves the best reported Clothing1M scores on both ResNet-18 and ResNet-50 backbones.The method improves over the existing state of the art by 1.48% with ResNet-18 and raises test accuracy from 74.76% to 75.93% with ResNet-50.
- Real-world noisy datasets: Jo-SRC achieves the best score on Food101N and outperforms DeepSelf by 1.55%.Food101N contains 310k training images in 101 food categories and a large proportion of noisy labels.
- Ablation study: Jo-SRC selects clean samples accurately and reliably, with selection precision increasing steadily during training under Symmetry-80% noise.Its selection strategy also distinguishes ID and OOD samples, as evaluated in Figure 4 and Table 5.
- Ablation study: Ablations identify clean-sample selection as most crucial, while appropriately treated noisy samples and consistency loss further improve generalization.Jo-SRC-C, Jo-SRC-CI, Jo-SRC-CIO, and the final Jo-SRC isolate these training components.
5. Conclusion
Jo-SRC addresses performance degradation from noisy labels by combining contrastive training, global clean-sample identification, consistency-based noise separation, and joint-loss regularization. Experiments on synthetic and real-world noisy datasets validated its superiority.
- Jo-SRC trains DNNs contrastively while identifying clean samples globally using JS divergence and distinguishing ID from OOD noisy samples through consistency.
- A joint loss combines classification with consistency regularization to improve performance and robustness during subsequent network learning.
- Experiments on both synthetic and real-world noisy datasets validated Jo-SRC’s superiority over existing methods.