Source-linked AI summary
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, Xipeng Qiu
TL;DR
Existing speech-language models commonly use cascades that limit cross-modal knowledge transfer and omit aspects of speech understanding. SpeechGPT unifies speech and text with discrete representations, trains on SpeechInstruct through three stages, and demonstrates strong multimodal instruction-following and spoken-dialogue abilities, while retaining stated limitations in paralinguistic handling, response generation, and multi-turn dialogue.
Problem
Cascaded speech-language models limit intermodal knowledge transfer, while existing systems have incomplete cross-modal perception and generation and may lose paralinguistic information.
Method
SpeechGPT discretizes speech into tokens, adds them to the LLM vocabulary, trains with SpeechInstruct, and applies modality-adaptation, cross-modal, and chain-of-modality fine-tuning.
Results
SpeechGPT demonstrates strong instruction following across unimodal and cross-modal tasks and strong spoken-dialogue ability in human evaluations and case analyses.
Takeaways & Limitations
Combining discrete speech tokens with a language model is presented as a promising direction for handling multiple modalities within one model.
Takeaways & Limitations
SpeechGPT does not model paralinguistic information, requires a text response before speech generation, and cannot support multi-turn dialogues because of context-length limitations.
Abstract
from arXiv · showhide
Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence (AGI) and have garnered significant interest with the emergence of ChatGPT. However, current speech-language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-model content. With discrete speech representations, we first construct SpeechInstruct, a large-scale cross-modal speech instruction dataset. Additionally, we employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow multi-modal human instructions and highlight the potential of handling multiple modalities with one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.
1 Introduction
SpeechGPT addresses limitations of cascaded speech-language systems by unifying speech and text through discrete speech tokens. It combines cross-modal instruction data with staged training to support multimodal perception, generation, and instruction following.
- Current multimodal LLMs generally understand multimodal content but cannot spontaneously generate it, while continuous speech and images do not directly fit discrete-token LLM inputs.
- Cascaded speech-language systems limit intermodal knowledge transfer, lose paralinguistic information, and often synthesize speech without comprehending its semantics.
- SpeechGPT discretizes speech with a self-supervised speech model and expands the resulting tokens into the LLM vocabulary for intrinsic speech perception and generation.
- SpeechInstruct combines speech-text cross-modal instruction data with Chain-of-Modality data, where the model processes speech commands in text before producing speech responses.
- SpeechGPT uses modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning to improve speech comprehension, cross-modal capability, and modality alignment.
- Human evaluations and case analyses report strong instruction following across unimodal and cross-modal tasks, alongside spoken dialogue ability.
2 Related Work
Related work spans multimodal LLMs focused mainly on vision, discrete-representation spoken language models, and systems that connect expert speech models to LLMs.
- Multimodal LLMs predominantly focus on vision by feeding continuous representations from pretrained visual encoders into language models.
- Generative spoken language models use discrete self-supervised representations or audio codecs to model and synthesize speech, including textless speech generation.
- Speech-enabled LLM systems integrate expert speech models with LLMs to support direct speech interaction, task decomposition, and invocation of ASR or TTS models.
3 SpeechInstruct Construction
SpeechInstruct addresses limited public speech data and scarce speech-text task diversity by combining cross-modal instruction data with Chain-of-Modality instruction formats.
- SpeechInstruct contains Cross-Modal Instruction and Chain-of-Modality Instruction data to expand speech-text cross-modal instruction following.
- 9 million unit-text pairs are produced from Gigaspeech, Common Voice, and LibriSpeech after mHuBERT2 discretization and removal of repetitive adjacent units.
- GPT-4 generates 100 task descriptions for each ASR and TTS task through a zero-shot prompting process.
- Each formatted instruction combines a task description, discrete unit sequence, and transcription into an instruction triplet.
- A text-to-unit Transformer converts selected text instruction data into speech instruction data, addressing the shortage of speech-input and speech-output instructions.
- Chain-of-Modality instructions cover speech-to-speech, speech-to-text, text-to-speech, and text-to-text input-output formats.
4 SpeechGPT
SpeechGPT combines a discrete-unit extractor, LLaMA-based language model, and unit vocoder to process and generate speech alongside text. Its training expands the vocabulary for speech units and proceeds through modality adaptation, cross-modal instruction tuning, and chain-of-modality tuning.
- Model Structure: SpeechGPT uses a discrete unit extractor, large language model, and unit vocoder so the model can perceive multimodal inputs and generate multimodal outputs.The extractor converts speech into discrete units, while the vocoder decodes generated units into speech.
- Model Structure: HuBERT transforms speech into discrete units by encoding audio, clustering intermediate representations, and removing adjacent duplicate indices.The resulting sequence contains cluster indices from 0 to K−1.
- Model Structure: The LLM component is Meta AI LLaMA, whose parameter counts range from 7B to 65B and whose pretraining dataset contains 1.0 trillion tokens.LLaMA consists of an embedding layer, transformer blocks, and an LM head.
- Training: Training has three stages: modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning.The first stage trains on unpaired speech with next-token prediction; the second uses paired cross-modal data; the third applies LoRA to Chain-of-Modality Instruction.
- Training: SpeechGPT expands the original vocabulary with K speech-unit tokens and creates a corresponding embedding matrix while preserving the original word embeddings.The new vocabulary is the union of the original vocabulary and the added unit-token set.
5 Experiments
SpeechGPT is evaluated through human case studies on cross-modal instruction following and spoken dialogue, using datasets and a multi-stage training configuration. The reported cases show accurate task outputs and speech-based responses to speech instructions.
- Experimental Setup: SpeechGPT uses LibriLight, Gigaspeech, Common Voice, LibriSpeech, and moss-002-sft-data across its training stages.LibriLight provides 60K hours of unlabelled English audiobook speech for modality-adaptation pre-training.
- Experimental Setup: The model is trained with LLaMA-13B using 96 A100 GPUs for stages 1 and 2 and 8 A100 GPUs for stage 3.Stages 1, 2, and 3 run for 900, 2100, and 4200 steps, respectively.
- Evaluation: Human evaluation examines SpeechGPT’s cross-modal instruction following and spoken dialogue abilities through case studies.The evaluation covers two aspects: cross-modal instruction following and spoken dialogue.
- Main Results: Table 1 shows that SpeechGPT performs instructed tasks and generates accurate outputs for varied cross-modal inputs.The result is reported for diverse instructions and corresponding outputs.
- Main Results: Table 2 presents spoken dialogues in which SpeechGPT comprehends speech instructions, responds in speech, and follows Harmless, Helpful, Honest criteria.The table contains 10 spoken-dialogue cases.
6 Limitation
SpeechGPT has notable cross-modal and speech-dialogue abilities but remains limited in paralinguistic expression, response generation order, and multi-turn interaction.
- Limitations: SpeechGPT cannot generate responses in different emotional tones because it does not consider paralinguistic speech information.The limitation concerns paralinguistic information such as emotion.
- Limitations: SpeechGPT must generate a text-based response before producing a speech-based response.
- Limitations: SpeechGPT cannot support multi-turn dialogues because of its context length limitation.
7 Conclusion
The paper presents SpeechGPT as a multimodal language model that perceives and generates multimodal content, supported by SpeechInstruct and three-stage training. Its results support discrete speech tokens as a promising direction for cross-modal modeling.
- Conclusion: SpeechGPT is an inherent cross-modal multimodal language model capable of perceiving and generating multimodal contents.
- Conclusion: SpeechInstruct addresses scarcity of speech-domain instruction data with cross-modal instruction and spoken-dialogue data based on chain-of-modality.
- Conclusion: A three-stage training paradigm is used to improve SpeechGPT’s cross-modal performance.
- Conclusion: Experiments report promising results across various unimodal and cross-modal tasks.
- Conclusion: Combining discrete speech tokens with a language model is identified as a promising direction.
A Prompts to Generate Task Description
The appendix describes GPT-4-generated task instructions for automatic speech recognition and text-to-speech tasks, emphasizing diversity in wording, language, and task type.
- ASR: The ASR prompt requests 100 diverse English instructions for recognizing the content of speech.
- ASR: ASR instructions are designed to vary verbs, language, formats, and task types.The prompt combines questions with imperative instructions and asks for diverse wording.
- TTS: The TTS prompt requests 100 diverse task instructions concerning text-to-speech.
B Examples of Task Description
The task descriptions cover speech-to-text transcription and text-to-speech generation, with varied prompts for producing each modality.
- ASR tasks convert spoken words into written text through transcription-focused instructions.
- TTS tasks ask the model to read, recite, articulate, or otherwise render text as speech.
- The examples vary wording and delivery goals, including natural conversation, soft whispering, clear articulation, and emphasis.
C Chain-of-Modality Instructions Templates
The chain-of-modality templates organize instructions and responses across speech and text, using intermediate text representations when needed.
- Speech Instruction-Speech Response transcribes the speech instruction, derives a text response, and then produces speech.
- Speech Instruction-Text Response transcribes the speech instruction and returns the resulting response as text.
- Text Instruction-Speech Response derives a text response from a text instruction and speaks that response aloud.
- Text Instruction-Text Response processes a text instruction and returns a text response without a speech output.
D Hyperparameters
Table 3 is titled “SpeechGPT training hyperparameters.”
- Table 3 presents SpeechGPT training hyperparameters.
- The table is identified as part of the SpeechGPT training configuration.
- The supplied caption labels Table 3 as a hyperparameter table.