Source-linked AI summary
SUPERB: Speech processing Universal PERformance Benchmark
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee
TL;DR
Speech research lacks a shared benchmark for systematically comparing self-supervised representations across diverse tasks. SUPERB provides that benchmark and evaluates a frozen shared model with lightweight task-specific heads, finding competitive cross-task performance and large gains over FBANK in reported comparisons.
Problem
Speech SSL studies use different datasets and experimental setups, limiting systematic comparison across techniques and tasks.
Method
SUPERB benchmarks frozen shared pretrained models across diverse speech tasks using lightweight task-specialized prediction heads and limited labeled data.
Results
SSL representations show competitive performance across SUPERB tasks, with wav2vec 2.0 and HuBERT especially strong and outperforming FBANK by a large margin in reported comparisons.
Takeaways & Limitations
SUPERB offers a standard, comprehensive testbed for evaluating the generalizability and reusability of pretrained speech representations.
Abstract
from arXiv · showhide
Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.
1. Introduction
SUPERB addresses the lack of a shared speech benchmark by evaluating self-supervised representations across diverse tasks with limited labeled data. Its frozen-model, lightweight-head framework achieves competitive performance and substantially surpasses FBANK features.
- Motivation: SSL pretrains shared models on unlabeled data, then adapts them to downstream tasks with limited task-specific resources.This paradigm offers reusable representations and reduces repeated data, modeling, computation, and training costs.
- Motivation: Speech SSL studies span many tasks but use differing datasets and experimental setups, making comparisons and cross-technique insights difficult.The existing literature covers phoneme, speaker, emotion, ASR, translation, language understanding, voice conversion, and TTS tasks.
- Contribution: SUPERB introduces a broad benchmark to examine pretrained models across speech tasks using limited labeled data.The benchmark targets direct usability of pretrained models across popular tasks, unlike prior efforts with narrower coverage or no downstream training.
- Contribution: The proposed framework freezes a shared pretrained model and trains lightweight task-specific prediction heads for each downstream task.This design focuses on representation extraction and avoids the resource demands associated with full-model finetuning.
- Results: SSL representations show competitive performance against traditional supervised pipelines and outperform FBANK by a large margin.These findings support the feasibility of more generalizable and reusable pretrained speech models.
2. Speech processing Universal PERformance Benchmark
SUPERB is a comprehensive testbed for pretrained speech models, initially covering discriminative tasks across content, speaker, semantics, and paralinguistics. It uses conventional protocols, public datasets, and limited labeled data to assess generalizability.
- Scope: SUPERB initially focuses on discriminative speech tasks spanning content, speaker, semantics, and paralinguistics.Generative tasks such as voice conversion and source separation are planned for a later challenge.
- Design principles: The benchmark follows conventional evaluation protocols, publicly available datasets, and limited labeled data to test pretrained-model generalizability.Ten tasks are included in the initial release.
- Content: Content tasks include phoneme recognition, ASR, keyword spotting, and query-by-example spoken term detection.Their metrics include PER, WER, ACC, and MTWV, respectively.
- Speaker: Speaker tasks evaluate identification, verification, and diarization using classification, pairwise verification, and timestamped speaker prediction.The benchmark includes both known-speaker identification and verification settings where test speakers may be unseen during training.
- Semantics: Semantic tasks infer intent and slot information directly from raw audio rather than using a transcription-then-semantic pipeline.Intent classification uses ACC, while slot filling evaluates slot-type F1 and slot-value CER.
- Paralinguistics: Emotion recognition evaluates four balanced emotion classes—neutral, happy, sad, and angry—using five-fold cross-validation and accuracy.The benchmark drops unbalanced emotion classes before evaluation.
3. Framework: Universal Representation
The framework evaluates SSL representations by freezing pretrained models and training lightweight task-specific heads. It combines hidden states and uses simple downstream models tailored to task structure.
- Framework: The framework freezes pretrained-model parameters and feeds fixed representations into small task-specific prediction heads.The explicit constraint keeps downstream models lightweight and limits training resources across tasks.
- Pretrained models: The investigated SSL models span generative, discriminative, and multi-task learning approaches.Examples include APC and TERA, wav2vec 2.0 and HuBERT, and PASE+.
- Representation extraction: The framework weighted-sums multiple hidden states because the final layer is not always the best representation.Downstream hyperparameter search is also restricted to preserve comparability and simplicity.
- Simple downstream models: PR uses a frame-wise linear CTC model, while KS, SID, IC, and ER use mean pooling followed by linear classification.These five tasks provide direct linear-evaluation indicators of representation quality.
- Task-specific models: ASR uses a 2-layer 1024-unit BLSTM with character CTC, while QbE, SF, and ASV use DTW, an ASR reformulation, and an x-vector model, respectively.These task-specific heads preserve the framework’s lightweight downstream adaptation strategy.
4. Experiment
Experiments compare SSL representations across SUPERB tasks using released checkpoints and constrained evaluation settings. Results show strong but task-dependent gains, with wav2vec 2.0 and HuBERT especially competitive.
- Evaluation: The experiments evaluate representations using public checkpoints or code and report results across multiple downstream tasks.Table 2 collects the comparative results for the benchmarked SSL representations.
- Linear evaluation: For linear tasks, FBANK fails on every task, whereas SSL representations perform well with different specializations.Wav2vec 2.0 and HuBERT achieve large-margin advantages on PR and IC using only linear models.
- Content tasks: HuBERT ranks first on QbE with MTWV 0.074, while TIMIT PPG obtains MTWV 0.052.The result identifies HuBERT as a competitive QbE representation in this evaluation.
- Speaker tasks: HuBERT improves speaker diarization over FBANK from 9.56 to 5.10 without additional VoxCeleb2 data or augmentation.SSL representations are nevertheless not uniformly better than FBANK for real-world speaker verification and diarization tasks.
- Overall findings: Wav2vec 2.0 and HuBERT achieve highly competitive performance with only lightweight prediction heads trainable.The authors conclude that SSL representations can support more generalizable and reusable pretrained models, although generalization across all tasks is non-trivial.
5. Conclusion
SUPERB benchmarks SSL-pretrained models across speech-processing tasks and provides a simple framework using a frozen shared model with minimal architecture changes.
- SUPERB is presented as a challenge for generally benchmarking SSL-pretrained models’ capabilities in speech processing.
- The framework solves all SUPERB tasks with a frozen, shared pretrained model and achieves competitive performance with minimal architecture changes.
- The evaluation toolkit is open-sourced, while detailed challenge policies are to be released on the leaderboard website.