Source-linked AI summary
USB: A Unified Semi-supervised Learning Benchmark for Classification
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, Yue Zhang
TL;DR
USB addresses limited cross-domain SSL evaluation and costly from-scratch training by benchmarking 14 algorithms on 15 CV, NLP, and Audio tasks with a lower-cost pre-training and fine-tuning protocol. Results show that SSL methods differ across domains, pre-training improves efficiency and generalization, and unlabeled data can sometimes hurt performance.
Problem
Existing SSL benchmarks are mostly limited to computer vision, while commonly training models from scratch is costly and environmentally unfriendly.
Method
USB benchmarks 14 SSL algorithms on 15 tasks across CV, NLP, and Audio, using pre-trained models, a modular codebase, and an extensible evaluation framework.
Results
SSL algorithm performance is not steady across domains, while pre-training accelerates training and improves generalization performance.
Takeaways & Limitations
Diverse multi-domain tasks are necessary for SSL evaluation, and unlabeled data should not be assumed to improve performance when labeled data is scarce.
Takeaways & Limitations
USB focuses on semi-supervised classification and excludes several SSL settings, including imbalanced, open-set, sequence-modeling, regression, detection, and clustering tasks.
Abstract
from arXiv · showhide
Semi-supervised learning (SSL) improves model generalization by leveraging massive unlabeled data to augment limited labeled samples. However, currently, popular SSL evaluation protocols are often constrained to computer vision (CV) tasks. In addition, previous work typically trains deep neural networks from scratch, which is time-consuming and environmentally unfriendly. To address the above issues, we construct a Unified SSL Benchmark (USB) for classification by selecting 15 diverse, challenging, and comprehensive tasks from CV, natural language processing (NLP), and audio processing (Audio), on which we systematically evaluate the dominant SSL methods, and also open-source a modular and extensible codebase for fair evaluation of these SSL methods. We further provide the pre-trained versions of the state-of-the-art neural models for CV tasks to make the cost affordable for further tuning. USB enables the evaluation of a single SSL algorithm on more tasks from multiple domains but with less cost. Specifically, on a single NVIDIA V100, only 39 GPU days are required to evaluate FixMatch on 15 tasks in USB while 335 GPU days (279 GPU days on 4 CV datasets except for ImageNet) are needed on 5 CV tasks with TorchSSL.
1 Introduction
USB addresses evaluation gaps in semi-supervised learning by spanning multiple domains and reducing the cost of experiments through pre-training and fine-tuning. It also provides implementations and resources for consistent, reproducible evaluation.
- 1 Introduction: Pre-trained Vision Transformers reduce CV training iterations by 80%, from 1,000k to 200k, without hurting performance.Most SSL algorithms achieve better performance with fewer iterations using pre-trained ViT models.
- 1 Introduction: 39 GPU days evaluate FixMatch on USB’s 15 datasets, compared with 335 GPU days for TorchSSL’s five computer-vision datasets.The comparison uses a single NVIDIA V100, with 279 GPU days for TorchSSL excluding ImageNet.
- 1 Introduction: The benchmark reports that diverse domains improve evaluation coverage, pre-training improves efficiency and generalization, and unlabeled data do not always improve performance.The findings are based on evaluations of 14 SSL algorithms.
- 1 Introduction: USB benchmarks semi-supervised classification across 15 tasks spanning computer vision, natural language processing, and audio.The benchmark is designed for fair and consistent evaluation across domains.
- 1 Introduction: USB implements 14 SSL algorithms and releases a modular, extensible codebase, configurations, documentation, and pre-trained models.These resources support reproduction and continued community development.
2 Related Work
Prior SSL benchmarks mainly cover computer-vision classification and train models from scratch, limiting domain diversity and increasing computational cost. USB is positioned as a broader, maintainable alternative.
- 2 Related Work: Existing benchmarks such as Realistic SSL evaluation and TorchSSL focus on computer-vision classification tasks.They cover three and five CV tasks, respectively, and are no longer maintained or updated.
- 2 Related Work: Previous benchmarks typically train models from scratch, making SSL evaluation computationally expensive and time-consuming.The related-work discussion identifies this cost as a motivation for alternative protocols.
- 2 Related Work: USB’s modular benchmark is intended to support continuously updated SSL algorithms and neural models through community development.The paper contrasts this goal with the maintenance status of earlier benchmarks.
3 Tasks and Datasets
USB selects 15 challenging datasets across computer vision, NLP, and audio, emphasizing domain diversity, room for improvement, and affordable training. Dataset construction preserves task-specific sampling and test-set conventions.
- 3 Tasks and Datasets: USB contains 15 datasets across computer vision, NLP, and audio, selected for diversity, challenge, and affordable training.The datasets are also required to permit research use under permissive licenses.
- 3. Tasks and Datasets: USB uses two labels-per-class settings for most datasets, while Semi-Aves and FSDNoisy18k retain long-tailed distributions.Validation and labeled-data construction differ by domain, and all test sets remain unchanged.
- 3 Tasks and Datasets: The CV suite includes CIFAR-100, STL-10, TissueMNIST, EuroSAT, and Semi-Aves, covering several image domains.CIFAR-10 and SVHN are excluded because leading SSL methods approach fully supervised performance on them.
- 3 Tasks and Datasets: The NLP suite includes IMDB, AG News, Yahoo! Answer, Amazon Review, and Yelp Review, with added fine-grained sentiment tasks.Labeled, unlabeled, and validation data are sampled from original training data while test sets remain unchanged.
- 3.3 Audio Tasks: The audio suite covers urban sound, human sound, and music through five challenging classification datasets.The paper presents this as the first systematic evaluation of SSL algorithms on audio tasks.
4 SSL Algorithms
USB implements 14 SSL algorithms built around consistency-based use of unlabeled data. Their differences lie mainly in how unlabeled examples are converted, filtered, aligned, or regularized.
- 4 SSL Algorithms: USB implements 14 SSL algorithms, including Π model, FixMatch, FlexMatch, AdaMatch, and SimMatch.These methods generally encourage invariant predictions under input perturbations.
- 4 SSL Algorithms: The implemented methods use cross-entropy on labeled data but differ in their treatment of unlabeled data.The paper describes pseudo-labeling, thresholding, distribution alignment, and consistency mechanisms as key differences.
5 Benchmark Results
USB evaluates SSL across CV, NLP, and Audio using standardized protocols, pretrained backbones, and cross-domain rankings. Results show domain-dependent algorithm performance, benefits from pretraining, and cases where SSL underperforms supervised training.
- Performance Comparisons: ReMixMatch ranks first on CV, while SimMatch ranks first in NLP and AdaMatch outperforms other algorithms in Audio.CRMatch performs well in CV and NLP but poorly in Audio, whereas SimMatch performs similarly to AdaMatch in Audio.
- Performance Comparisons: Adaptive thresholding methods such as FlexMatch and AdaMatch perform consistently well across CV and NLP, even without self-supervised loss.NLP rankings are not fully predictive of CV rankings; for example, SimMatch ranks first in NLP but fourth in CV.
- Performance Comparisons: Audio SSL performance trails fully supervised learning more than CV and NLP on most tasks, likely because raw waveforms contain noisy information harmful to semi-supervised training.The paper identifies Mel-spectrogram-based audio models as a future direction.
- Performance Comparisons: SSL rankings differ across CV, NLP, and Audio, with FixMatch, CoMatch, and CRMatch showing rank differences of at least four across domains.The results support evaluating SSL methods on diverse tasks because domain characteristics affect method choice.
- Effectiveness of Pre-training: Pretrained ViT makes SSL training more efficient and improves generalization, while the same ViT trained from scratch performs worst among evaluated backbones.Pretrained ViT features are reported as more separable, and several methods that barely converge from scratch achieve reasonable results with pretraining.
- Robustness: SSL does not consistently outperform supervised training when labeled data is scarce, while CRMatch, AdaMatch, and SimMatch are relatively robust.USB defines robust SSL as consistently better than the supervised training setting.
6 Codebase Structure of USB
USB organizes its reusable SSL functionality into four abstract layers, from shared training infrastructure to a public API. This structure supports algorithm reuse, extensibility, and reproducible application development.
- 6 Codebase Structure of USB: The core layer supplies common training functions, datasets, data loaders, models, and modifiable training hooks.
- 6 Codebase Structure of USB: The algorithm layer abstracts SSL algorithms over core components, improving code reuse and simplifying implementation of new methods.
- 6 Codebase Structure of USB: The extension layer supports imbalanced and open-set SSL applications and systematic ablation studies through inherited components.
- 6 Codebase Structure of USB: The API layer packages USB as SEMILEARN, enabling training and inference in a few lines with reproducible configuration files.
- 6 Codebase Structure of USB: USB uses core, algorithm, extension, and API layers to organize its benchmark codebase.The layers are arranged bottom-up and separate shared infrastructure, SSL algorithms, application extensions, and user-facing access.
7 Limitation
USB focuses on semi-supervised classification and does not yet cover several other SSL tasks, algorithm families, or distribution-shift settings.
- 7 Limitation: USB currently excludes SSL tasks beyond classification, including regression, detection, clustering, sequence modeling, and text generation.
- 7 Limitation: USB does not implement GAN- or graph-neural-network-based SSL algorithms and does not yet address domain adaptation or out-of-distribution generalization.
- 7 Limitation: The authors plan to extend USB with more tasks and algorithms in future iterations.
8 Conclusion
USB is designed for consistent, lower-cost SSL evaluation across multiple domains. Its results show that domain diversity and pre-training matter, while unlabeled data can hurt when labels are extremely scarce.
- 8 Conclusion: USB evaluates 14 SSL algorithms across 15 tasks and finds that performance is not exactly stable across CV, NLP, and Audio domains.
- 8 Conclusion: USB reduces evaluation cost while broadening coverage, supporting affordable and consistent comparison across multiple datasets and domains.The benchmark is intended as an open project that can be extended with more challenging tasks and algorithms.
- 8 Conclusion: The benchmark reports that pre-training can accelerate SSL training and improve generalization, whereas unlabeled data sometimes hurts performance when labeled data is extremely scarce.
- 8 Conclusion: A Pearson correlation of 0.87 between TorchSSL and USB CV mean error rates does not yield consistent final algorithm rankings.FlexMatch ranks best on USB, whereas AdaMatch ranks best on TorchSSL.
C Performance Results on ImageNet
The ImageNet evaluation tests several SSL algorithms with an MAE-pre-trained ViT-B under two label budgets. Its ranking differs from USB, with CoMatch and SimMatch outperforming FlexMatch.
- C Performance Results on ImageNet: The ImageNet study evaluates UDA, FixMatch, FlexMatch, CoMatch, and SimMatch with an MAE-pre-trained ViT-B.Experiments use 10 or 100 labels per class, corresponding to roughly 1% or 10% of ImageNet labels.
- C Performance Results on ImageNet: On ImageNet, CoMatch and SimMatch outperform FlexMatch, although FlexMatch is best on USB.CoMatch ranks ninth on USB but performs strongly on ImageNet, while UDA and FixMatch remain near the bottom.
- C Performance Results on ImageNet: ImageNet results show that SSL algorithm rankings can change substantially across benchmark settings.
D Results with Different Pre-trained Backbones
USB evaluates SSL algorithms across multiple pre-trained backbones and finds that backbone changes preserve broad ranking tiers while altering some within-tier orderings.
- D Results with Different Pre-trained Backbones: Backbone changes preserve the distinction between the higher- and lower-performing algorithm groups, but can alter relative ordering within the top group.This pattern is reported across tasks when different pre-trained backbones are used.
- D Results with Different Pre-trained Backbones: The backbone study evaluates CV models using fine-tuned pre-trained Swin-Transformer models on EuroSAT and Semi-Aves.
- D Results with Different Pre-trained Backbones: RoBERTa generally performs better than BERT on NLP tasks, although their performance difference is very close.
- D Results with Different Pre-trained Backbones: Audio backbone comparisons use HuBert and Wave2Vec2.0, whose main difference is their pre-training data and target design.Wave2Vec2.0 uses raw human voice data, whereas HuBert uses an improved discrete clustering target.
E.1 CV Tasks
This section covers USB’s CV datasets and the SSL algorithms evaluated on them, spanning standard consistency, pseudo-labeling, augmentation, and distribution-alignment approaches.
- E.1 CV Tasks: USB’s CV evaluation includes natural-image, satellite-image, medical-image, and imbalanced bird-classification datasets.The listed datasets include CIFAR-100, STL-10, EuroSAT, TissueMNIST, and Semi-Aves.
- E.1 CV Tasks: STL-10 provides 100,000 unlabeled samples, including classes beyond those represented in its labeled data.
- E.1 CV Tasks: The evaluated SSL methods include consistency-based, pseudo-labeling, adversarial-perturbation, Mixup, stronger-augmentation, and distribution-alignment approaches.These include the Π model, Pseudo Labeling, Mean Teacher, VAT, MixMatch, ReMixMatch, and FixMatch.
- E.1 CV Tasks: FixMatch filters weak-augmentation predictions with a high confidence threshold before using them as pseudo-labels for strongly augmented data.The passage gives 0.95 as an example threshold.
G.1 Setup for CV Tasks in USB
USB’s setup uses pre-trained models and domain-specific training configurations for CV, NLP, and Audio tasks, with tuned learning-rate and layer-decay settings reused across algorithms.
- G.1 Setup for CV Tasks in USB: CV tasks use pre-trained ViT models, with task-specific patch and image sizes chosen to match the datasets and model parameter scales.Released ViT models can overfit unless the image resolution is fixed to the pre-trained resolution, so the authors pre-train their own models on ImageNet-1K.
- G.1 Setup for CV Tasks in USB: CV training uses layer-wise learning-rate decay, cosine annealing, 204,800 total steps, 5,120 warm-up steps, and labeled and unlabeled batch sizes of 16.Learning rate and layer decay are tuned with FixMatch and then used for all SSL algorithms.
- G.1 Setup for CV Tasks in USB: NLP tasks use pre-trained BERT-Base with batch size 4, AdamW, weight decay 1e−4, cosine scheduling, and back-translation augmentation.Learning rate and layer decay are tuned using FixMatch before applying the configuration to other SSL algorithms.
- G.1 Setup for CV Tasks in USB: Audio tasks use Wav2Vec 2.0 and HuBert with batch size 8, 16,000-Hz sampling, AdamW, and weight decay 5e−4.Audio strong augmentation randomly selects two augmentations from an augmentation pool.