Source-linked AI summary
Iterative Learning with Open-set Noisy Labels
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, Shu-Tao Xia
TL;DR
Noisy labels threaten CNN training, and existing closed-set assumptions fail when mislabeled samples belong to unseen classes. The paper proposes iterative detection, discriminative feature learning, and reweighting, reporting robust performance across open-set and closed-set noise settings. Its formulation leaves relationships between pairs of noisy samples undefined because their true classes are not contained in the training data.
Problem
Existing noisy-label methods commonly assume mislabeled samples have true classes among the known training classes, but open-set noise violates this assumption.
Method
An iterative framework combines noisy-label detection, Siamese contrastive feature learning, and reweighting based on detected-label confidence.
Results
The model outperforms state-of-the-art methods for open-set noise and is effective for closed-set noise across datasets of various scales.
Takeaways & Limitations
The framework robustly trains CNNs with open-set as well as closed-set noisy labels.
Takeaways & Limitations
Relationships between two noisy samples are left undefined because their true classes are not contained within the training data.
Abstract
from arXiv · showhide
Large-scale datasets possessing clean label annotations are crucial for training Convolutional Neural Networks (CNNs). However, labeling large-scale data can be very costly and error-prone, and even high-quality datasets are likely to contain noisy (incorrect) labels. Existing works usually employ a closed-set assumption, whereby the samples associated with noisy labels possess a true class contained within the set of known classes in the training data. However, such an assumption is too restrictive for many applications, since samples associated with noisy labels might in fact possess a true class that is not present in the training data. We refer to this more complex scenario as the \textbf{open-set noisy label} problem and show that it is nontrivial in order to make accurate predictions. To address this problem, we propose a novel iterative learning framework for training CNNs on datasets with open-set noisy labels. Our approach detects noisy labels and learns deep discriminative features in an iterative fashion. To benefit from the noisy label detection, we design a Siamese network to encourage clean labels and noisy labels to be dissimilar. A reweighting module is also applied to simultaneously emphasize the learning from clean labels and reduce the effect caused by noisy labels. Experiments on CIFAR-10, ImageNet and real-world noisy (web-search) datasets demonstrate that our proposed model can robustly train CNNs in the presence of a high proportion of open-set as well as closed-set noisy labels.
1. Introduction
The paper identifies open-set noisy labels as a practical challenge in which mislabeled samples may belong to classes absent from training data. It proposes an iterative framework that detects noisy labels, learns discriminative features, and reweights labels to robustly train CNNs.
- Large-scale labeling is costly and error-prone, while noisy labels can damage representation learning and prediction performance.
- Open-set noise occurs when a mislabeled sample’s true class is absent from the known training classes, unlike closed-set noise.
- Closed-set correction methods can be inaccurate for open-set noise because the true class may not exist in the dataset.
- The proposed framework iteratively detects noisy labels, separates clean and noisy representations with a Siamese network, and reweights their learning contributions.
- The paper identifies open-set noisy labels as a new representation-learning challenge and proposes a framework not dependent on an assumption of noise.
- The model significantly outperforms state-of-the-art methods for open-set noise and performs comparably or better under closed-set noise.
2. Related work
Prior noisy-label methods commonly correct losses or infer clean labels under a closed-set assumption. The paper distinguishes open-set samples and instead detects and separates them during representation learning.
- Some approaches remove samples with suspicious labels, but distinguishing inherently difficult samples from noisy ones is challenging.
- Noise-model methods represent symmetric, asymmetric, or more complex relationships among samples, true labels, and noisy labels.
- Loss-correction methods modify training objectives using estimated factors, added layers, or combinations of raw and inferred target labels.
- These approaches generally assume true labels belong to known training classes, limiting their applicability to open-set noise.
- The proposed model differs by iteratively detecting noisy samples and pulling them away from clean samples rather than removing or relabeling them.
3. Iterative learning framework
The framework iteratively detects noisy labels, learns discriminative representations, and reweights training contributions in a mutually reinforcing loop. It combines contrastive and reweighted softmax losses to separate noisy from clean samples while preserving useful learning signals.
- Framework overview: The framework contains iterative noisy-label detection, discriminative feature learning, and reweighting modules that jointly improve through a closed loop.Detected clean and noisy subsets guide representation learning, whose outputs in turn support later detection.
- Discriminative feature learning: The Siamese network uses contrastive loss to minimize distances between similar samples and maximize distances between dissimilar samples.Dissimilar pairs include samples from different classes and clean–noisy pairs; relationships between two noisy samples are left undefined.
- Reweighting: Reweighting assigns clean samples weight 1 and reduces noisy-sample weights according to their estimated noisiness.The factor γ = 1 − pcLOF decreases as detection becomes more accurate, while boundary samples retain weights close to clean samples.
- Joint optimization: The network is jointly optimized with reweighted softmax loss and contrastive loss, using η to control their trade-off.This objective integrates detection, representation learning, and reweighting into one training framework.
- Iterative noisy label detection: The detector uses pre-softmax representations and applies cumulative pcLOF iteratively to identify noisy samples without assuming an underlying data distribution.pcLOF converts cumulative LOF scores into probabilities, where values near 0 indicate clean samples and values near 1 indicate noisy samples.
4. Experiments
Experiments evaluate the framework on synthetic open-set and closed-set noise, module ablations, parameter sensitivity, varying noise rates, and real-world web-search data. Results show strong robustness, iterative detection and feature separation, module complementarity, and usefulness for webly supervised learning.
- Experimental setup: Experiments span CIFAR-10, ImageNet, and web-search datasets, covering small-scale, large-scale, and real-world noisy-label settings.The evaluation targets robustness across open-set and closed-set noise.
- Experimental setup: Open-set CIFAR-10 datasets replace training images with outside images while preserving class labels and per-class dataset size.Outside images come from CIFAR-100, ImageNet32, SVHN, or severely damaged CIFAR-10 images.
- Classification performance: At 40% open-set noise, the model outperforms baselines on both type I and type II CIFAR-10 noise.The reported comparison covers outside-dataset images and severely damaged images.
- Model interpretation: The true positive rate of detected noisy labels becomes more precise over iterations, while learned features increasingly separate noisy samples from clean samples.Visualizations use CIFAR-10+CIFAR-100 with 40% open-set noise.
- Module analysis via ablation experiments: Removing or replacing any of the three modules reduces accuracy, with especially significant decreases after removing discriminative feature learning or iterative noisy-label detection.The ablation results also identify reweighting as necessary because detected noisy samples may include clean samples near the decision boundary.
- Parameter and real-world analysis: The model remains best across open-set noise rates up to 50%, is not sensitive to η in a comparable range, and shows promise for webly supervised learning.The real-world evaluation uses approximately 1.2M web-search images with unknown noise type and rate, while learned features are assessed through CIFAR-100 classification.
5. Conclusions
The paper investigates open-set noisy labels and proposes an iterative framework combining detection, discriminative feature learning, and reweighting. Experiments show effectiveness for both open-set and closed-set noise across datasets of varying scales.
- The framework combines iterative noisy label detection, discriminative feature learning, and reweighting.These modules are designed to benefit from each other and improve jointly over iterations.
- The model outperforms state-of-the-art methods for open-set label noise.
- The approach is also effective for closed-set label noise.
- The evaluation covers datasets of various scales.