Source-linked AI summary
WavLLM: Towards Robust and Adaptive Speech Large Language Model
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
TL;DR
Speech LLMs still face limited generalization across prompts and difficulty handling complex auditory tasks. WavLLM combines dual encoders, curriculum learning, and prompt-aware LoRA adaptation, achieving strong performance across speech tasks and complex CoT settings. Its main stated boundaries are English-focused processing, no speech generation, and no autonomous CoT decomposition.
Problem
Speech LLMs struggle to generalize across unseen or complex instructions and lack speech Chain-of-Thought capability for complex tasks.
Method
WavLLM combines Whisper and WavLM dual encoders with two-stage curriculum learning and a prompt-aware LoRA adapter for speech instruction following.
Results
WavLLM achieves state-of-the-art performance across speech-related tasks with robust single-task and multi-task generalization and improved CoT processing.
Takeaways & Limitations
The model completes zero-shot English listening comprehension and performs better with CoT-based instructions than with one-shot non-CoT instructions.
Takeaways & Limitations
WavLLM focuses primarily on English, cannot generate speech, and cannot autonomously decompose complex one-shot tasks into CoT sub-tasks.
Abstract
from arXiv · showhide
The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{aka.ms/wavllm}.
1 Introduction
WavLLM addresses weak generalization to unseen prompts and complex speech tasks with dual speech encoders, curriculum learning, and prompt-aware adaptation. It achieves strong performance across speech benchmarks while supporting zero-shot listening comprehension and speech CoT tasks.
- 1 Introduction: Prior speech LLMs struggle with unseen or complex instructions because task performance is prompt-sensitive and speech CoT capability is limited.These limitations motivate methods that improve instruction generalization and complex-task reasoning.
- 1 Introduction: WavLLM trains progressively from mixed elementary single tasks to advanced multi-task instructions using curriculum learning and a prompt-aware LoRA adapter.The training covers tasks including ASR, ST, ER, and SV, with the adapter introduced for advanced multi-task training.
- 1 Introduction: Extensive evaluations show robust generalization across universal speech benchmarks, spoken question answering, English listening comprehension, multi-task instructions, and speech CoT.The evaluation includes both single-task and multi-task settings.
- 1 Introduction: WavLLM uses Whisper for semantic speech content and WavLM for acoustic information such as speaker identity.The dual-encoder design decouples different types of speech information for downstream speech processing.
- 1 Introduction: WavLLM demonstrates zero-shot English listening comprehension and stronger performance with speech CoT than with non-CoT instructions.The model completes the Gaokao listening task without specialized training according to the supplied paper context.
2 Related Work
Related work highlights the difficulty of executing multiple complex speech tasks under one instruction. WavLLM responds with curriculum learning, prompt-aware LoRA scaling, and diverse prompt training to improve generalization.
- 2 Related Work: Existing audio-language work includes SALMONN and Qwen-audio, which extend LLMs to generic hearing and broad audio-language task coverage.Qwen-audio is described as covering over 30 tasks and multiple audio types.
- 2 Related Work: Prior speech LLM research largely targets single speech tasks, leaving simultaneous completion of multiple complex tasks underexplored.Multi-task instructions can divide complex tasks into logical sub-tasks, but prior feasibility remained insufficiently studied.
- 2 Related Work: Initial experiments found weaker multi-task performance, prompt-dependent degradation, and a trade-off between LoRA scaling for multi-task and training-task performance.The reported prompt comparison shows a decline from seen to unseen or diverse prompts of 3.5% vs. 2.1%.
- 2 Related Work: WavLLM addresses these issues by progressing from simple to complex instructions, dynamically adjusting LoRA output by instruction, and training with diverse GPT-4-generated prompts.The approach is designed to improve generalization across task and prompt conditions.
3 Method
WavLLM combines dual speech encoders, modality adapters, LLaMA, and prompt-aware LoRA with two-stage curriculum learning. Training progresses from single-task speech processing to prompt-aware multi-task instruction training, supported by mixed evaluation benchmarks.
- Model Architecture: Whisper extracts semantic speech information while WavLM captures acoustic information, and their adapted representations are concatenated before entering the LLM.The architecture also uses temporal downsampling, bottleneck adapters, and a final linear projector to align encoder outputs.
- Model Architecture: Prompt-aware LoRA generates instruction-dependent scaling factors to modulate LLaMA’s adaptation across single-task and multi-task prompts.The prompt representation is processed by a down-up adapter with attention weights, producing LoRA scaling factors for the attention layers.
- Two-Stage Curriculum Learning: The first curriculum stage trains speech and text capabilities on mixed single-task data while optimizing modality adapters, a linear layer, and LoRA components.The training mix includes ASR, ST, SV, ER, instruction tuning, and GPT-generated spoken question answering.
- Two-Stage Curriculum Learning: The second stage combines elementary instructions into complex multi-task examples because repetitive speech-text fine-tuning can weaken multi-task instruction following.The prompt adapter is introduced in this stage to produce adaptive LoRA scaling factors for different instructions and tasks.
- Evaluation Setup: The evaluation covers single-task and multi-task benchmarks spanning ASR, ST, SV, ER, SQA, spoken-query question answering, and listening comprehension.Table 1 summarizes the evaluated tasks, datasets, and metrics, while the multi-task training data include combinations such as ASR+ST and ASR+SQA.
4 Experiments
WavLLM is evaluated on single-task and multi-task speech instructions, including ASR, ST, SV, ER, SQA, independent multi-task instructions, and CoT tasks. It achieves strong benchmark performance and shows gains from advanced training, the prompt adapter, and dual-encoder representations.
- Single-task Evaluation: WavLLM achieves state-of-the-art ASR WERs of 2.0% and 4.8% on LibriSpeech test-clean and test-other, respectively.
- Single-task Evaluation: The model surpasses the cascaded Whisper+LLaMA baseline and other open-source speech LLMs on zero-shot English listening comprehension, reaching 67.55% versus 25.50–59.30%.SQA evaluates both exact-option accuracy and semantically equivalent answers scored by GPT-4.
- Multi-task Evaluation: WavLLM consistently improves over one-shot non-CoT instructions and produces the best performance on CoT-task instructions.
- Analysis: Advanced multi-task training improves zero-shot independent-task instruction following and CoT performance while retaining comparable or better single-task performance.The advanced-training model significantly outperforms the mixed-training model on the long story-based CoT task.
- Analysis: The prompt adapter consistently outperforms a two-stage LoRA-only baseline across single-task and multi-task instructions.WavLM also improves speaker verification and reduces ASR WER by 13.04% and 11.11% on test-clean and test-other.
- Analysis: WavLLM shows no degradation on unseen ASR prompts and improves ST performance, whereas SALMONN declines on unseen or diverse prompts.The t-SNE visualization indicates overlap between seen and unseen ASR prompt clusters and discrimination among task prompts.
5 Conclusion
WavLLM combines dual speech encoders, curriculum learning, and a prompt adapter to support robust, generalized speech instruction following. Experiments report strong performance across speech tasks, including complex multi-task instructions and CoT processing.
- 5 Conclusion: WavLLM achieves state-of-the-art performance across speech-related tasks while generalizing to single-task and multi-task instructions.The model uses LLaMA-2-chat with Whisper and WavLM encoders, curriculum learning, and a prompt adapter.
Limitations
The paper identifies limitations in WavLLM’s adaptive reasoning, modality coverage, and safety. These constraints motivate future work on autonomous CoT decomposition, speech synthesis, multilingual extension, and adversarial robustness.
- Adaptive Use of CoT: WavLLM improves with CoT instructions but cannot autonomously decompose complex non-CoT tasks into subtasks.Future work targets deciding when decomposition is appropriate and applying CoT accordingly.
- Broader Applicability: WavLLM primarily focuses on English speech and does not generate speech, although multilingual extension is considered feasible.Speech synthesis is deferred to future research.
- Safety and Ethics: Continuous speech representations may increase vulnerability to adversarial attacks and undermine compliance with Harmless, Helpful, Honest criteria.The paper identifies this vulnerability as requiring further investigation.
A Some Examples of Training Data
The training-data examples combine speech transcripts with task-specific prompts and strict output requirements. They include conversational question answering and instructions that prohibit exposing speaker-label metadata.
- A. Some Examples of Training Data: The examples include short utterances, literary text, and conversational exchanges as source speech content.Examples include LibriSpeech material and a dialogue about workplace drug testing.
- A.3 The Prompts for Generating SQA Data by GPT-4: Training examples use conversational speech transcripts to generate one reasoning-based question and answer in a strict quoted format.The prompt requires the answer to be correct, clear, conversation-related, and free of speaker labels.
- A.3 The Prompts for Generating SQA Data by GPT-4: Speaker A and Speaker B are treated as transcript labels that must not appear in generated questions or answers.The prompt permits referring to speakers by alternatives such as the first or second speaker.
B Some Examples of Evaluation Data
The evaluation examples test WavLLM on mixed audio-text instructions, including prediction, translation, transcription, factual question answering, and response scoring. The model is trained with a speech-instruction template in which only target tokens contribute to loss.
- B. Some Examples of Evaluation Data: Evaluation instructions combine multiple tasks such as next-action prediction, French translation, transcription, and text-only factual answering.The tasks are presented sequentially for one audio segment and include both speech-dependent and unrelated instructions.
- C.2 II-task Scoring: The scoring prompt checks whether sequential instructions were answered correctly, completely, in English where required, and without repeating instructions.Choice answers must select exactly one option, while translation applies to the audio content rather than the instruction.
- B. Some Examples of Evaluation Data: WavLLM formats each example as speech features plus text instructions and target outputs, with training loss applied only to the target portion.The template uses speech encoder features in place of the speech field and removes end-of-sequence or beginning-of-sequence markers differently for input and target.
- B. Some Examples of Evaluation Data: A multi-round dialogue figure illustrates the evaluation setting’s conversational dimension.
F Training Data Details
The training data uses diverse instructions across tasks, with SQA targets generated by large language models.
- Instructions are diverse across all training tasks.
- SQA targets are generated using GPT-3.5, GPT-4, or LLaMA-2-chat.
G Implementation Details
WavLLM uses frozen speech encoders and a frozen LLaMA-2-chat backbone, with trainable modality adapters and LoRA parameters.
- Whisper-large-v2, WavLM-base, and LLaMA-2-chat-7B parameters are frozen.
- The model has 7.55 billion total parameters, including 76.6 million trainable parameters during first-stage training.
- The modality-adapter outputs use 80 ms time strides and dimension 2048, while LoRA rank R is 32.
H The Effect of Advanced Training for Single-tasks
Advanced training extends mixed single-task training with multi-task instructions formed by linking elementary task instructions using connectives.
- Advanced training is evaluated against mixed training on single-task instructions.Mixed training denotes the first stage, whereas advanced training denotes the second multi-task stage.
- Combined instructions link ASR and text-based instruction tasks with connectives such as “Initially” and “Then.”
- Evaluation-set creation follows a process similar to training-set creation.
- The ASR + text-based IT task uses ASR speech input and combines the respective instructions from both tasks.
J Some Case Study and Error Analysis
Error analysis identifies failures in distinguishing transcription from translation and in avoiding repeated responses.
- One error confuses “transcription” with “translation.”
- Another error repeats the same responses, a problem also observed in other multimodal large language models.