SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities

1 Introduction

• Self-supervised learning learns from inputs or their perturbations without labeled data, enabling use of large amounts of unlabeled data.Its learned representations transfer across tasks, and increasing model and data sizes has often improved downstream performance.Source: §1 · p. 1
• Existing studies vary in datasets, fine-tuning strategies, and task-specific architectures, while SUPERB standardizes comparisons across 10 speech tasks.SUPERB covers linguistic, shallow semantic, speaker, and prosodic characteristics.Source: §1 · p. 2
• SUPERB-SG adds five tasks—speech translation, out-of-domain ASR, voice conversion, speech separation, and speech enhancement—to evaluate semantic and generative capabilities.The tasks target representations of linguistic, semantic, and speaker characteristics.Source: §1 · p. 2
• The training illustration shows a trainable weighted sum of layer representations feeding the downstream model while the upstream model remains frozen.The dashed arrow indicates gradient flow during backpropagation.Source: Figure 1 · p. 2
• The authors aim for a more comprehensive unified speech benchmark and integrate SUPERB-SG tasks with SUPERB.They hope this will motivate more generalizable and reusable pretrained models.Source: §1 · p. 2

2 Related Work

• The authors frame extensive evaluation as a way to assess model performance and understand learned representation capabilities holistically.They note that SSL models show promising performance across various tasks.Source: §2 · p. 2
• SUPERB benchmarks SSL models on 10 speech tasks with task-specific heads over frozen models, but most tasks are simple classification or use shallow semantics.SUPERB-SG instead focuses on harder semantic and generative tasks.Source: §2 · p. 2
• LeBenchmark evaluates SSL models on three French semantic tasks, but considers only wav2vec 2.0 with different upstream architectures.The authors contrast this with their evaluation of diverse SSL models and broader analysis.Source: §2 · pp. 2–3
• The Zero Resource Speech Benchmark 2021 evaluates zero-shot spoken language modeling at four linguistic levels, while SUPERB-SG spans varied speech tasks.The cited benchmark focuses on a particular domain.Source: §2 · p. 3
• HEAR 2021 targets general-purpose audio representations across tasks beyond speech, whereas SUPERB-SG focuses on speech processing.HEAR includes sound event detection, speech commands, and pitch and chroma classification.Source: §2 · p. 3

3.1 Tasks and Datasets

• The benchmark freezes pretrained parameters and trains only task-specific heads to assess representation robustness and generalizability with limited compute.The section introduces five new tasks and their task-specific heads.Source: §3.1 · p. 3

3.1.1 Speech Translation

• Speech translation uses CoVoST2 English-to-German data and a three-layer Transformer encoder-decoder to assess SSL models’ semantic capability.The setup uses official dataset splits, removes samples containing “REMOVE,” and reports case-sensitive de-tokenized BLEU.Source: §3.1.1 · p. 3

3.1.2 Out-of-domain ASR

• Out-of-domain ASR covers cross-lingual Spanish, Mandarin, and Arabic plus spontaneous American English, extending beyond read English LibriSpeech.It uses WER except for Mandarin, where it uses CER, and averages error rates across subtasks.Source: §3.1.2 · p. 3

3.1.3 Voice Conversion

• Voice conversion uses an any-to-one setting to test speaker transferability and generalizability, training on 60 target-speaker utterances spanning five minutes.A target-speaker-dependent model reconstructs acoustic features, which a neural vocoder uses to synthesize converted speech.Source: §3.1.3 · p. 4

3.1.4 Speech Separation

• Speech separation evaluates SSL representations on two-speaker mixtures by predicting spectral masks and scoring separation with SI-SDRi.The task uses simulated LibriSpeech mixed with WHAM! noise and a frequency-domain BLSTM system.Source: §3.1.4 · p. 4

3.1.5 Speech Enhancement

• Speech enhancement tests generative capability under noise on Voicebank-DEMAND using PESQ and STOI metrics.A three-layer BLSTM predicts a spectral mask for the clean signal.Source: §3.1.5 · pp. 4–5

3.2 Self-supervised Models

• The evaluation includes 15 upstream SSL models spanning different architectures, sizes, and learning objectives, with Log Mel Filterbank as the baseline.Some upstream models use vector quantization, which the authors note can also provide signal compression.Source: §3.2 · p. 5

4 Experimental Setup

• Each experiment freezes upstream parameters, extracts every hidden layer’s frame-level representations, and trains a task-specific weighted sum and downstream model.All experiments use one run with the same seed.Source: §4 · p. 5

5.1 Main result

• HuBERT-Large is most competitive across downstream tasks, but no model outperforms all others on every task.Speech-to-text results favor wav2vec 2.0 and HuBERT while DeCoAR 2.0 shows some improvements; FBANK matches or exceeds some SSL models on generation, and task-specific leaders vary.Source: §5.1 · pp. 5–6

5.2 Correlation between tasks

• Across 11 downstream tasks, every task except speech separation (SS) and speech enhancement (SE) showed strong positive correlations; excluding SS and SE, all correlation coefficients exceeded 0.58.The reported clusters align with content-information, speaker/prosodic-feature, and signal-quality and speaker-transfer distinctions.Source: §5.2 · pp. 6–7

5.3 Robustness of SUPERB-SG

• Robustness tests across downstream architectures and training-data sizes support the benchmark’s conclusions, including with training data reduced to 5%.At 1% of training data, most SSL models fail on the three evaluated tasks because task-specific knowledge requires non-trivial supervision.Source: §5.3 · p. 7, §5.3.2 · pp. 8–9

5.3.1 Downstream model

• Upstream-model rankings remain almost fixed across downstream architecture sizes, with ranking changes tied to negligible score differences.The small architecture performs worse than default and the large one better; the reported results support relative performance being agnostic to downstream architecture.Source: §5.3.1 · pp. 7–8, Table 6 · p. 8

5.3.2 Training data size

• With 10% of training data, upstream rankings remain almost unchanged; at 5%, the SS ranking changes because Modified CPC drops, but excluding it leaves rankings fixed.At 1%, most SSL models fail on ST, OOD-ASR, and SS; the authors attribute this to insufficient task-specific knowledge and supervision.Source: §5.3.2 · p. 8, Table 8 · p. 9

• SUPERB-SG introduces five new tasks—speech translation, out-of-domain ASR, voice conversion, speech separation, and speech enhancement—to evaluate semantic and generative capabilities of SSL models.Source: §6 · p. 9
• The study evaluates 15 SSL models, analyzes task correlations, and tests benchmark robustness across downstream architectures and training-data sizes.Source: §6 · p. 9
• The authors open-source the code and release a challenge to encourage further SSL research in speech.Source: §6 · p. 9

• The authors state that the work adheres to the ACL code of ethics and provide a supporting checklist in Appendix B.Source: Ethics · p. 9

A Complete Out-of-domain ASR Results

• XLSR, pre-trained on 56,000 hours across 53 languages, achieves the best performance among upstream models on OOD-ASR; multilingual data improves these tasks.Source: A Complete Out-of-domain ASR Results · p. 13, Table 9 · p. 13

B Responsible NLP Research Checklist

• The appendix section states that it answers the responsible-NLP ethics questions.Source: B Responsible NLP Research Checklist · p. 13

• The authors locate discussion of frozen-upstream and simple task-specific-head constraints in the abstract and Section 3.Source: §B.1 · p. 13

B.2 Did you discuss any potential risks of your work?

• The authors identify unstable benchmark results as a potential risk discussed in Section 5.3.Source: §B.2 · p. 13

B.3 Do the abstract and introduction summarize the paper’s main claims?

• The abstract and Section 1 summarize the paper’s main claims.Source: §B.3 · p. 13

B.4 Did you use or create scientific artifacts?

• The work uses public datasets and pre-trained models described in Section 3.Source: §B.4 · p. 14

B.4.1 Did you cite the creators of artifacts you used?

• The authors state that they cited the creators of the artifacts used in Section 3.Source: §B.4.1 · p. 14

B.4.2 Did you discuss the license or terms for use and/or distribution of any

• Artifact licenses are indicated; the authors used upstream models’ official implementations and public APIs and followed dataset licenses.Source: §B.4.2 · p. 14

B.4 Did you use or create scientific artifacts?

• The authors report no data collection and say artifact properties and relevant dataset statistics were reported.Source: §B.4.2 · p. 14

B.5.1 Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used?

• Model parameter counts were reported in Section 5.3.1, while computational budgets and infrastructure were reported in Table 12.Source: §B.5 · p. 14

B.5.2 Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values?

• Hyperparameter search was not conducted uniformly; some settings came from official implementations or related work, while others were searched by the authors.The authors state that the hyperparameters used are publicly available.Source: §B.5.2 · p. 14

B.5.3 Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run?

• The authors state that descriptive statistics about results were indicated in Section 4.Source: §B.5.2 · p. 14

B.5.4 Did you report implementation, model, and parameter settings for existing packages?

• Implementation, model, and parameter settings for existing packages were reported in Section 3.Source: §B.5.2 · p. 14