Source-linked AI summary
Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning
Qing Yu, Daiki Ikami, Go Irie, Kiyoharu Aizawa
TL;DR
The paper addresses open-set SSL, where limited labeled data is paired with unlabeled data containing OOD samples. It introduces a multi-task curriculum that alternates OOD-score and network updates while using selected ID samples for SSL, and reports state-of-the-art performance across open-set benchmarks. The method has a documented weakness when few outliers make Otsu thresholding unreliable.
Problem
Open-set SSL must classify ID samples when unlabeled data also contains OOD samples, while only limited labeled ID data is available.
Method
A multi-task curriculum alternates network-parameter and OOD-score updates, excluding high-score samples before SSL training on the remaining unlabeled and labeled data.
Results
The method achieves state-of-the-art performance on several open-set SSL benchmarks by detecting OOD samples with high accuracy.
Takeaways & Limitations
Joint OOD detection and ID classification can address open-set SSL without training the two tasks separately.
Takeaways & Limitations
When few outliers are present, Otsu thresholding can produce an incorrect threshold and fail to improve the baseline.
Abstract
from arXiv · showhide
Semi-supervised learning (SSL) has been proposed to leverage unlabeled data for training powerful models when only limited labeled data is available. While existing SSL methods assume that samples in the labeled and unlabeled data share the classes of their samples, we address a more complex novel scenario named open-set SSL, where out-of-distribution (OOD) samples are contained in unlabeled data. Instead of training an OOD detector and SSL separately, we propose a multi-task curriculum learning framework. First, to detect the OOD samples in unlabeled data, we estimate the probability of the sample belonging to OOD. We use a joint optimization framework, which updates the network parameters and the OOD score alternately. Simultaneously, to achieve high performance on the classification of in-distribution (ID) data, we select ID samples in unlabeled data having small OOD scores, and use these data with labeled data for training the deep neural networks to classify ID samples in a semi-supervised manner. We conduct several experiments, and our method achieves state-of-the-art results by successfully eliminating the effect of OOD samples.
1 Introduction
Open-set SSL addresses unlabeled data containing outliers outside the labeled classes, a setting where standard SSL assumptions break. The proposed multi-task curriculum framework jointly detects OOD samples and trains ID classification, achieving strong results across open-set settings.
- Existing SSL assumes labeled and unlabeled data share classes, but mismatched unlabeled classes can damage SSL performance.
- Open-set SSL contains OOD samples in unlabeled data even though they do not belong to any labeled-data class.
- Traditional OOD detectors are unsuitable for open-set SSL because they rely on many labeled ID samples, which SSL lacks.
- The framework alternates network-parameter and OOD-score updates, then uses low-score unlabeled samples with labeled data for SSL-based ID classification.
- The method achieves state-of-the-art performance across several open-set SSL tasks and successfully eliminates the effect of OOD samples in unlabeled data.
2 Related Work
Related work covers SSL methods based on augmentation consistency and prior OOD detectors that generally require abundant labeled ID data. Open-set SSL remains challenging because unlabeled OOD samples can substantially degrade existing SSL methods.
- 2.1 Semi-supervised Learning: Data augmentation artificially enlarges supervised training sets by modifying inputs while preserving their labels.
- 2.1 Semi-supervised Learning: Consistency regularization keeps predictions stable before and after augmentation, with π-model and Mean Teacher providing representative implementations.
- 2.1 Semi-supervised Learning: MixMatch combines low-entropy label guessing, MixUp, and π-model training for strong image-classification SSL performance.
- Existing SSL methods assume labeled and unlabeled samples share classes, and OOD samples can reduce performance below supervised learning in some cases.
- 2.2 OOD Detection: Prior OOD methods include softmax-based detection, ODIN, and openMax, but they require many labeled ID samples and do not use unlabeled data.
3 Method
The method jointly detects OOD samples and performs semi-supervised ID classification in open-set SSL, where unlabeled data may contain classes outside the labeled set. It alternates OOD-score refinement with network training and selects low-score unlabeled samples for SSL classification.
- Problem: Open-set SSL contains OOD samples among unlabeled data, making detection difficult when only limited labeled ID samples are available.The method targets classification of ID samples while eliminating the training effect of OOD samples.
- Framework: The proposed framework solves OOD detection and SSL simultaneously through multi-task curriculum learning.It combines OOD detection with ID classification in an end-to-end trainable network rather than training them separately.
- Training procedure: The framework’s training loop reassigns noisy unlabeled OOD scores from DNN outputs and uses those scores to select samples for the semi-supervised loss.The overview describes score reassignment, alternating updates each epoch, and score-based sample selection.
- Curriculum learning: Unlabeled samples with low OOD scores are selected and combined with labeled ID data for semi-supervised classification.The selected samples are used in the SSL loss; this paper uses MixMatch, while the formulation is applicable to general SSL losses.
- OOD detection: Labeled samples receive initial OOD scores of 0, unlabeled samples receive 1, and the network parameters and unlabeled OOD scores are alternately updated to refine these assignments.The OOD objective is treated as binary classification supervised by the current OOD scores, while alternating updates correct noisy scores.
4 Experiments
Experiments across diverse open-set SSL settings show that the proposed method improves classification while detecting and excluding OOD samples. It generally outperforms baselines, though performance gains disappear when unlabeled data contains very few outliers.
- 4 Experiments: Across diverse in- and out-of-distribution dataset pairs, the method outperforms current state-of-the-art methods by a considerable margin.Experiments use CIFAR-10 and SVHN as ID datasets, with multiple OOD datasets mixed into unlabeled data.
- 4.4 Results: The method significantly outperforms MixMatch without OOD detection on CIFAR-10 by eliminating the effect of OOD samples in unlabeled data.Synthetic Gaussian and Uniform outliers are more harmful than natural-image TIN and LSUN outliers, while the proposed detector enables more stable SSL performance.
- 4.4 Results: During training, the method initially converges more slowly but later improves continuously and is more stable than the baseline.The slower early convergence is attributed to simultaneously learning OOD detection through multi-task learning.
- 4.4 Results: The method maintains higher and more stable performance than the baseline on SVHN, where outliers have a smaller effect than on CIFAR-10.The authors associate the smaller outlier effect with SVHN being a comparatively easier classification task.
- 4.5 Ablation Studies: The method remains stable as the number of OOD samples increases, but it fails to improve the baseline when the unlabeled data contains very few outliers.The few-outlier failure is attributed to an imbalanced Otsu threshold and could be addressed by introducing a parameter controlling selected ID samples.
- 4.5 Ablation Studies: The proposed OOD detector significantly outperforms existing OOD detection methods in the challenging setting with only 250 labeled samples.Previous methods achieve AUROC below 50% in some cases, whereas a perfect detector has AUROC 1.
5 Conclusion
The paper proposes a multi-task curriculum for open-set SSL, jointly detecting OOD samples and using selected ID data for semi-supervised classification. Experiments show state-of-the-art performance by detecting OOD samples with high accuracy and excluding their effect.
- The proposed multi-task curriculum jointly detects OOD samples and trains ID classification when labeled data is limited and unlabeled data contains OOD samples.It alternates updates to network parameters and OOD scores, while selecting low-score unlabeled samples for semi-supervised training with labeled data.
- The method achieves state-of-the-art performance on several open-set semi-supervised benchmarks by detecting OOD samples with high accuracy.