Source-linked AI summary
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, Hung-yi Lee
TL;DR
Speech instruction tuning remains underexplored, and existing speech evaluations often focus on specific tasks or fixed task sets. Dynamic-SUPERB introduces a collaborative benchmark and five baseline approaches; the baselines perform reasonably on seen tasks but struggle on unseen ones.
Problem
Instruction tuning supports zero-shot generalization in language models, but its use in speech remains underexplored and existing speech evaluations focus on specific tasks or fixed task sets.
Source: §1 · p. 1Method
Dynamic-SUPERB is a collaborative, expandable benchmark with speech-and-instruction tasks, and the paper tests five baseline approaches trained with Dynamic-SUPERB-Train.
Source: §1 · p. 1, §2.4 · p. 2, §3 · p. 3Results
Baselines performed reasonably on seen tasks but declined on unseen tasks; combining speech representations with a text language model surpassed other methods.
Source: §1 · p. 2, §4.3 · p. 4Takeaways & Limitations
The authors release the materials and invite community contributions to expand evaluation of universal speech models.
Source: §2.1 · p. 2, §5 · p. 4The initial benchmark focuses on classification tasks and leaves generative tasks for future collaboration.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Text language models have shown remarkable zero-shot capability in generalizing to unseen tasks when provided with well-formulated instructions. However, existing studies in speech processing primarily focus on limited or specific tasks. Moreover, the lack of standardized benchmarks hinders a fair comparison across different approaches. Thus, we present Dynamic-SUPERB, a benchmark designed for building universal speech models capable of leveraging instruction tuning to perform multiple tasks in a zero-shot fashion. To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark. To initiate, Dynamic-SUPERB features 55 evaluation instances by combining 33 tasks and 22 datasets. This spans a broad spectrum of dimensions, providing a comprehensive platform for evaluation. Additionally, we propose several approaches to establish benchmark baselines. These include the utilization of speech models, text language models, and the multimodal encoder. Evaluation results indicate that while these baselines perform reasonably on seen tasks, they struggle with unseen ones. We release all materials to the public and welcome researchers to collaborate on the project, advancing technologies in the field together.
1. INTRODUCTION
Speech instruction tuning remains underexplored, while prior speech systems and benchmarks focus on specific tasks or fixed task sets. Dynamic-SUPERB introduces a collaborative, expandable benchmark and baseline approaches, which perform reasonably on seen tasks but struggle on unseen ones.
Motivation
- Speech instruction tuning remains largely unexplored, and existing speech systems and benchmarks focus on specific tasks or static task sets.Instruction tuning improves language models’ zero-shot handling of unseen tasks, but its application to speech is less explored.Source: §1 · p. 1
Benchmark contribution
- Dynamic-SUPERB is a collaborative benchmark that extends task variety through community contributions and initially spans six dimensions with a classification-task focus.The paper describes over 20 public English datasets and a review process for proposed tasks.Source: §1 · p. 1
Baseline approaches and results
- The authors propose five baseline approaches and create Dynamic-SUPERB-Train to assess generalization on seen and unseen tasks.The approaches include speech models, speech representations combined with a text language model, and an ASR-ChatGPT system.Source: §1 · p. 1
- Combining speech representations with a text language model surpassed the other baseline methods, while no single model dominated across all tasks.ASR-ChatGPT excelled on some semantic tasks but underperformed on speaker and paralinguistic tasks; speech models integrating text struggled with unseen tasks.Source: §1 · p. 2
2. DYNAMIC-SUPERB
Dynamic-SUPERB organizes speech evaluation around flexible task–dataset pairings and a community-expandable task set. Its initial format uses speech, text instructions, and textual labels, and its current scope is classification tasks.
2.1. Objectives
- The benchmark aims to evaluate universal speech models comprehensively and expand task variety through community contributions, unlike benchmarks with fixed task sets.It provides a submission pipeline and review guidelines for proposed tasks.Source: §2.1 · p. 2
2.2. Terminologies
- A task names an operation, while an instance is a specific pairing of that task with a dataset.A dataset can support multiple tasks, and a task can use multiple datasets.Source: §2.2 · p. 2
2.3. Tasks & datasets
- The initial benchmark covers classification tasks across six dimensions, and Dynamic-SUPERB-Train supplies baseline training data with some shared tasks but different datasets.The six dimensions are content, speaker, semantics, degradation, paralinguistics, and audio processing.Source: §2.3 · p. 2
2.4. Task format
- Each task combines speech utterances, text instructions, and textual labels; instructions include answer options and scoring requires the exact correct option.Tasks contain approximately 10 to 30 instruction variants, and labels are represented textually so models can respond generatively.Source: §2.4 · p. 2
2.5. Benchmark extendibility
- Instruction tuning lets one model handle newly integrated tasks without explicit task-specific fine-tuning, while community collaboration supports growth beyond the benchmark’s current coverage.The authors note that Dynamic-SUPERB may not yet encompass every potential speech-processing task.Source: §2.5 · pp. 2–3
3. BASELINE FRAMEWORKS
The section establishes five instruction-tuning baselines by adapting existing speech and language models. The approaches range from adding text processing to speech models to combining speech representations or transcripts with language models.
3. BASELINE FRAMEWORKS
- Five approaches adapt existing models as baselines, trained on Dynamic-SUPERB-Train and tested on Dynamic-SUPERB.The paper notes that no model was specifically designed for instruction tuning toward universal speech models.Source: §3 · p. 3
3.1. BERT-GSLM
- BERT-GSLM adds BERT text representations to GSLM and expands its output tokens to include text as well as speech.HuBERT and BERT remain fixed while the unit language model and projection layer are trainable.Source: §3.1 · p. 3
3.2. Whisper
- Whisper instruction tuning appends the instruction to decoder text input so it generates a response conditioned on speech features.Training uses Whisper-medium with both encoder and decoder updated.Source: §3.2 · p. 3
3.3. Multi-modal large language model
- ImageBind-LLM integrates utterance-level speech features with LLaMA through a zero-initialized injection mechanism.ImageBind is fixed during training, and only LLaMA adapter modules are updated.Source: §3.3 · p. 3
- Whisper-LLM replaces ImageBind with the Whisper encoder to capture temporal speech information, adding downsampling to shorten feature sequences.Source: §3.3 · p. 3
3.4. ASR-ChatGPT
- ASR-ChatGPT transcribes speech with Whisper, combines the transcript with the instruction, and prompts ChatGPT to answer in the required label format.The added prompt asks ChatGPT to return only one label from the provided options.Source: §3.4 · p. 3
4. RESULTS
The baselines perform better on seen tasks than unseen tasks, with some unseen-task scores falling below random when models produce answers outside the provided options. Performance also varies by dimension and instruction familiarity.
4.1. Seen tasks
- On seen tasks, Whisper excelled in CNT, Whisper-LLM led most other dimensions, and ASR-ChatGPT excelled in SEM but struggled on SPK, DEG, and PRL.GSLM performed poorly overall; ASR-ChatGPT's speech-characteristic weaknesses were attributed to information discarded during ASR.Source: §4.1 · p. 4
4.2. Seen & unseen instructions in seen tasks
- Unseen instructions had little effect in CNT, SPK, and SEM but reduced accuracy in DEG and PRL, where seen and unseen instructions were imbalanced.The authors argue that diverse instruction formulations require community contributions beyond what a small research group or ChatGPT alone can provide.Source: §4.2 · p. 4
4.3. Unseen tasks
- Unseen-task performance declined across all models, and BERT-GSLM reached 0.0% accuracy on unseen CNT tasks.BERT-GSLM and Whisper could fall below random by answering outside the provided options, while ImageBind-LLM and Whisper-LLM were around random.Source: §4.3 · p. 4
- The authors surmise that models often match instruction patterns rather than task semantics, while AUD results also reflect its absence from training and the speech–audio distinction.ImageBind-LLM performed better in AUD, attributed to pretraining on multiple modalities including audio.Source: §4.3 · p. 4
5. CONCLUSIONS
Dynamic-SUPERB evaluates five instruction-tuning approaches for speech across diverse dimensions. The approaches show encouraging results on seen tasks, while unseen-task performance points to a need for continued research.
- Five approaches are tested on Dynamic-SUPERB, with encouraging results on seen tasks.Source: §5 · p. 4
- The benchmark explores instruction tuning across diverse dimensions in speech processing.Source: §5 · p. 4
- Performance on unseen tasks highlights the need for continued research in speech instruction tuning.Source: §5 · p. 4
- The authors open-source the materials to lower barriers to reproduction, benchmarking, and analysis.Source: §5 · p. 4
- The authors invite researchers to join the community and advance research in the field.Source: §5 · p. 4