PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Speech Language Models: A Survey

Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, Irwin King

arXiv:2410.03751v4cs.CLcs.SDeess.AS

TL;DR

The survey addresses limitations in the straightforward ASR–LLM–TTS approach to speech interaction. It reviews SpeechLM architectures, training, applications, and evaluation, and concludes that their integrated speech-token approach can retain paralinguistic information while reducing latency and cumulative errors.

  • Problem

    The survey addresses limitations in the straightforward ASR–LLM–TTS approach to speech interaction.

    Source: §I · p. 1

  • Method

    The paper surveys SpeechLM architectures, training recipes, applications, evaluation methods, challenges, and future directions.

    Source: §I · p. 2

  • Results

    SpeechLMs retain paralinguistic information and integrate speech-token modeling, mitigating information loss, latency, and cumulative errors.

    Source: §I · p. 2

  • Takeaways & Limitations

    The survey aims to illuminate SpeechLM research and assist the research community.

    Source: §VIII · p. 17

Abstract

from arXiv · show

Large Language Models (LLMs) have recently garnered significant attention, primarily for their capabilities in text-based interactions. However, natural human interaction often relies on speech, necessitating a shift towards voice-based models. A straightforward approach to achieve this involves a pipeline of ``Automatic Speech Recognition (ASR) + LLM + Text-to-Speech (TTS)", where input speech is transcribed to text, processed by an LLM, and then converted back to speech. Despite being straightforward, this method suffers from inherent limitations, such as information loss during modality conversion, significant latency due to the complex pipeline, and error accumulation across the three stages. To address these issues, Speech Language Models (SpeechLMs) -- end-to-end models that generate speech without converting from text -- have emerged as a promising alternative. This survey paper provides the first comprehensive overview of recent methodologies for constructing SpeechLMs, detailing the key components of their architecture and the various training recipes integral to their development. Additionally, we systematically survey the various capabilities of SpeechLMs, categorize their evaluation metrics, and discuss the challenges and future research directions in this rapidly evolving field. The GitHub repository is available at https://github.com/dreamtheater123/Awesome-SpeechLM-Survey

I. INTRODUCTION

The introduction motivates speech-capable interaction and frames ASR–LLM–TTS as a practical but limited route from speech to response. It presents SpeechLMs as an end-to-end alternative and surveys their architectures, training, applications, evaluation, and open challenges.

I. INTRODUCTION

  • The sequential ASR–LLM–TTS pipeline can lose paralinguistic information, add latency, and accumulate errors across stages.Transcription errors can affect language generation, while text that cannot be synthesized can degrade TTS performance.Source: §I · p. 1
  • SpeechLMs encode speech into tokens, model them autoregressively, and synthesize speech, retaining paralinguistic information while reducing latency and accumulated errors.The figure emphasizes that the same content can be used across speech and text modalities, allowing any input modality to yield any output modality.Source: §I · pp. 1–2, Figure 1 · p. 2
  • SpeechLMs can support speaker and emotional nuance, real-time interaction with interruption, and communication in rare languages with more spoken than written content.The rare-language application is described as a potential of training directly on speech data.Source: §I · p. 2
  • The survey presents the first comprehensive SpeechLM overview, covering architectures, training recipes, applications, evaluation methods, challenges, and future directions.Its stated contributions include a taxonomy based on underlying components and training recipes, plus a classification system for evaluation methods.Source: §I · p. 2

II. PROBLEM FORMULATION

SpeechLMs are autoregressive foundation models that process and generate speech end to end, while also allowing text and cross-modal interactions. Their three-component design uses speech tokens so a language model can process audio and a vocoder can synthesize it.

II. PROBLEM FORMULATION

  • A SpeechLM is an autoregressive foundation model that processes and generates speech end to end, with contextual understanding for coherent sequence generation.It can also incorporate text for speech-in-text-out and text-in-speech-out interactions.Source: §II · p. 3
  • The framework represents input and output as multimodal sequences whose elements may be audio samples or text tokens.The input and output sequence lengths may each be zero or greater.Source: §II · p. 3

III. COMPONENTS IN SPEECHLM

  • A SpeechLM groups a speech tokenizer, language model, and token-to-speech synthesizer so the language model can autoregressively model speech tokens.The tokenizer converts waveforms to tokens; the synthesizer converts generated tokens back into audio.Source: §III · p. 3

A. Speech Tokenizer

  • Speech tokenizers encode continuous audio into discrete or continuous tokens, reducing dimensionality for autoregressive language-model processing.The survey groups them by which aspects of raw audio they focus on.Source: A. Speech Tokenizer · pp. 3–4, Figure 3 · p. 4

1) Semantic Understanding Objective:

  • Semantic-understanding tokenizers extract speech content and meaning, supporting tasks such as ASR.They typically use a speech encoder to produce continuous embeddings and a quantizer to convert them into discrete indexes; continuous embeddings may also serve as tokens.Source: 1) Semantic Understanding Objective · p. 4
  • Speech tokens may be used as tokenizer training labels, including masked-token reconstruction targets, or to train the subsequent language model.Both discrete indexes and continuous embeddings can be used for autoregressive modeling.Source: 1) Semantic Understanding Objective · p. 4, A. Speech Tokenizer · p. 4
  • Semantic tokenizer design choices include encoder and quantizer methods, training targets, and objectives such as contrastive loss, masked language modeling, text-injection loss, or denoising.The survey notes WavLM’s denoising objective can boost non-semantic tasks such as speaker verification and speech separation.Source: 1) Semantic Understanding Objective · p. 4

2) Acoustic Generation Objective:

Acoustic-generation tokenizers prioritize acoustic characteristics and use synthesis or reconstruction objectives. Mixed-objective tokenizers combine semantic and acoustic information, with existing approaches often distilling semantic information into acoustic-tokenizer architectures.

2) Acoustic Generation Objective:

  • Acoustic-generation tokenizers preserve acoustic characteristics over semantic content, making them suited to speech synthesis and resynthesis.Their encoder and quantizer produce tokens that a decoder reconstructs into a waveform.Source: 2) Acoustic Generation Objective · p. 4
  • Neural audio codecs commonly serve as acoustic tokenizers, compressing waveforms into compact discrete tokens that are reconstructed by a decoder.Their encoder-quantizer-decoder design commonly uses vector quantization or residual vector quantization.Source: 2) Acoustic Generation Objective · pp. 4–5

3) Mixed Objective:

  • Mixed-objective tokenizers seek to balance semantic understanding and acoustic generation, but their development is described as early-stage.Most existing approaches use acoustic-generation architectures and distill information from semantic tokenizers into them.Source: 3) Mixed Objective · p. 5
  • HuBERT iteratively refines speech tokens by updating its encoder and discretizer parameters, enabling increasingly meaningful speech representations.It uses masked prediction of tokens derived from clustered MFCC features.Source: 3) Mixed Objective · p. 5
  • Encodec uses multi-stage residual vector quantization to produce discrete acoustic tokens that retain high-fidelity audio information for downstream modeling.Its decoder reconstructs the waveform from quantized tokens.Source: 3) Mixed Objective · p. 6
  • SpeechTokenizer combines HuBERT-derived semantic information at the first RVQ stage with acoustic residual quantization in later stages.This hybrid is described as capturing both high-level semantic and low-level acoustic information.Source: 3) Mixed Objective · p. 6

B. Language Model

SpeechLMs commonly adapt TextLM architectures for autoregressive speech generation by replacing text-token embeddings with speech-token embeddings. Expanding the vocabulary to include both token types also allows a model to generate speech and text in one sequence.

B. Language Model

  • SpeechLMs commonly use transformer or decoder-only TextLM architectures for autoregressive speech generation.For discrete speech tokens, the text embedding and output matrices are replaced with speech-token matrices.Source: B. Language Model · p. 6
  • A common approach to joint text-and-speech modeling appends speech embeddings to the text embedding matrix, expanding the vocabulary.The resulting model can generate both modalities in a single sequence.Source: B. Language Model · p. 6
  • With continuous speech tokens, tokenizer embeddings are fed directly into the language model, leaving its architecture unchanged.Source: B. Language Model · p. 6

C. Token-to-Speech Synthesizer (Vocoder)

The vocoder converts generated speech tokens into audible waveforms, using either direct synthesis or an input-enhanced path depending on the token and vocoder.

C. Token-to-Speech Synthesizer (Vocoder)

  • The token-to-speech module, often called a vocoder, synthesizes generated speech tokens into audible waveforms.This reverses the speech tokenizer’s waveform-to-token process.Source: C. Token-to-Speech Synthesizer (Vocoder) · p. 6
  • Direct synthesis maps generated speech tokens straight to audio waveforms, while input-enhanced synthesis first transforms tokens into a continuous latent representation.Input-enhanced pipelines accommodate vocoders that require intermediate audio representations such as mel-spectrograms.Source: C. Token-to-Speech Synthesizer (Vocoder) · p. 6
  • Direct synthesis is generally simpler and faster, but pipeline choice depends on token type: acoustic tokens suit direct synthesis, while semantic tokens benefit from acoustic enhancement.Semantic tokens may lack fine acoustic detail, particularly at higher frequencies.Source: C. Token-to-Speech Synthesizer (Vocoder) · p. 6
  • Table II summarizes the architectural choices of speech tokenizer, language model, and vocoder in popular SpeechLMs.Its notation distinguishes absent or unreported components, inherited architectures, alternatives tested, and combined components.Source: Table II · p. 7

1) GAN-based Vocoder:

GAN-based vocoders pair a generator with a discriminator and use multiple training objectives; their architectures add inductive biases for waveform generation.

1) GAN-based Vocoder:

  • GAN is the most adopted vocoder architecture and is known for fast, high-fidelity speech synthesis.Its generator creates waveforms, while its discriminator evaluates generated audio against real samples.Source: 1) GAN-based Vocoder · p. 7
  • The typical GAN objective uses least-squares loss for both generator and discriminator.The generator and discriminator losses are presented as the fundamental training objective.Source: 1) GAN-based Vocoder · p. 7
  • Mel-spectrogram loss aligns generated and ground-truth mel-spectrograms with L1 distance, while feature matching aligns discriminator-encoded features with L1 distance.The mel-spectrogram objective is intended to improve generated-speech fidelity.Source: 1) GAN-based Vocoder · pp. 7–8
  • MelGAN uses dilated residual blocks and multi-scale discrimination; HiFi-GAN uses multi-period discrimination; Fre-GAN uses DWT; BigVGAN uses snake activation and anti-aliasing.These designs target long-range correlations, waveform periodicity, frequency bands, and high-frequency artifacts, respectively.Source: 1) GAN-based Vocoder · p. 8

2) GAN-based Neural Audio Codec:

The survey covers GAN-based neural audio codecs, less-common vocoder architectures, and the dimensions used to organize SpeechLM training recipes.

2) GAN-based Neural Audio Codec:

  • Polyak et al. use HiFi-GAN as a codec vocoder backbone and disentangle its inputs into semantic tokens, pitch tokens, and speaker embeddings.The design supports pitch- and speaker-related tasks such as voice conversion and F0 manipulation.Source: 2) GAN-based Neural Audio Codec · p. 8

3) Other Types of Vocoder:

  • Pure signal-processing vocoders use deterministic algorithms rather than deep learning, but introduce noticeable synthesis artifacts and are rarely used.Source: 3) Other Types of Vocoder · p. 8
  • Autoregressive vocoders generate one waveform sample at a time, supporting high-quality synthesis but requiring more time and computation than parallelized GAN-based vocoders.Source: 3) Other Types of Vocoder · p. 8
  • Flow-based vocoders sample audio in parallel and can enhance speed and quality, but typically require more training parameters and memory than GAN-based vocoders.They use invertible transformations to map a simple distribution, such as a Gaussian, to the audio-sample distribution.Source: 3) Other Types of Vocoder · p. 8
  • VAEs learn compressed latent representations that can reconstruct input data, but are seldom explored as vocoder architectures.Source: 3) Other Types of Vocoder · p. 8
  • Diffusion vocoders gradually add noise and learn to reverse that process for high-fidelity synthesis; DiffWave uses DDPM to synthesize audio.Source: 3) Other Types of Vocoder · p. 8
  • HiFi-GAN maps speech tokens to waveforms with a generator and uses multi-period and multi-scale discriminators during adversarial training.At inference, only the generator reconstructs speech waveforms.Source: 3) Other Types of Vocoder · p. 8

IV. TRAINING RECIPES

  • The training-recipe survey organizes SpeechLM methods by modeled features, training stages and their techniques, and speech-generation paradigms.Source: §IV · pp. 8–9

A. Features Modeled

SpeechLMs model speech through discrete or continuous features, with representation choices shaping expressiveness, audio quality, and training demands. The section also reviews language-model pre-training, including text–speech alignment and its trade-offs.

A. Features Modeled

  • SpeechLM features are either discrete tokens or continuous, unquantized representations, and the choice affects model capabilities and performance.Discrete tokens fit text-language-model token processing; continuous features can retain fine-grained speech detail but require more storage and may need changes to standard training pipelines.Source: A. Features Modeled · p. 9, 1) Discrete Features · p. 9, 2) Continuous Features · p. 9

1) Discrete Features:

  • Most SpeechLMs use semantic tokens, while added prosody or style tokens can supply expressive information that semantic tokens alone often miss.GSLM found HuBERT outperformed CPC and wav2vec 2.0 on speech resynthesis and generation; AudioPaLM found USM-v2 best among its tested tokenizers on ASR and speech translation.Source: 1) Discrete Features · p. 9
  • Semantic tokens support coherent content but can omit acoustic detail; acoustic tokens support high-fidelity audio but may be less accurate in content.Combining token types can lengthen sequences and increase modeling complexity, while mixed-token approaches are reported as promising.Source: 1) Discrete Features · p. 9

2) Continuous Features:

  • Continuous features include spectrograms and neural-network representations, which can preserve nuanced speech detail lost through discretization.Examples include framewise spectrogram prediction and representations extracted from frozen or jointly trained encoders.Source: 2) Continuous Features · p. 9

1) Language Model Pre-Training:

  • SpeechLM language-model pre-training typically predicts the next speech token autoregressively on large speech corpora, including ASR, TTS, translation, podcast, and dialogue data.Some corpora also pair speech with transcripts, which can help the model learn links between spoken and written language.Source: 1) Language Model Pre-Training · p. 10

B. Training Stages

  • Pre-training may start from random weights or adapt a pretrained TextLM; text–speech alignment can help, but its effects and inference trade-offs remain open.Alternating speech and text tokens performed best in SUTLM cross-modal evaluations; text-present inference may improve reasoning and reduce hallucinations while increasing latency, whereas text-independent inference is more efficient but may be less stable.Source: 1) Language Model Pre-Training · pp. 10–11

2) Language Model Instruction-Tuning:

The section reviews instruction-tuning data construction, post-alignment, and interaction capabilities beyond complete-turn speech generation. It describes streaming and full-duplex interaction as well as recognizing when to respond or remain silent.

2) Language Model Instruction-Tuning:

  • Instruction-tuning fine-tunes SpeechLMs to follow instructions and generalize across tasks, making instruction-following dataset construction a central focus.Source: 2) Language Model Instruction-Tuning · p. 11
  • SpeechGPT and SpeechGPT-Gen create instruction data in two stages: cross-modal instruction fine-tuning followed by chain-of-modality fine-tuning.The first stage adapts paired ASR and TTS data; the second converts text instruction data into speech-in/speech-out examples using TTS.Source: 2) Language Model Instruction-Tuning · p. 11
  • Llama-Omni synthesizes speech-style prompts and TextLM-generated responses into instruction-following speech pairs.Source: 2) Language Model Instruction-Tuning · p. 11
  • A model trained on COSMIC’s speech-QA dataset generalized to unseen tasks such as speech-to-text translation using in-context learning.Source: 2) Language Model Instruction-Tuning · p. 11

3) Language Model Post-Alignment:

  • Post-alignment targets safe, reliable behavior, with SpeechLM methods addressing semantic inconsistency and generated-speech acoustic quality.The survey describes post-alignment as under-explored and calls for research on SpeechLM-specific safety challenges.Source: 3) Language Model Post-Alignment · p. 11

C. Speech Interaction Paradigm

  • Streaming tokenizers and vocoders allow processing before complete speech encoding, enabling immediate, low-latency responses, but do not capture richer natural interaction patterns.Source: C. Speech Interaction Paradigm · p. 11
  • Full-duplex interaction supports user interruptions and concurrent listening and speaking by jointly modeling user and model audio streams.Approaches include separate dialogue transformers with cross-attention, paired next-token prediction, concatenated channels, and streaming listening-and-speaking embeddings.Source: C. Speech Interaction Paradigm · p. 12
  • Interactive Period Recognition identifies whether users are addressing the model, so it can respond during interaction and stay silent otherwise.MiniCPM-o 2.6 uses a VAD threshold; VITA learns to distinguish query speech from non-query audio and terminate responses with an end-of-sequence token.Source: C. Speech Interaction Paradigm · p. 12

V. DOWNSTREAM APPLICATIONS

SpeechLMs are presented as generative models for speech-only, text-only, and multimodal tasks, spanning semantic, speaker, and paralinguistic applications. This section details semantic tasks and their possible input and output modalities.

V. DOWNSTREAM APPLICATIONS

  • SpeechLM applications are grouped into semantic, speaker-related, and paralinguistic tasks, reflecting their ability to model semantic and paralinguistic information.Source: §V · p. 12

A. Semantic-Related Applications

  • Spoken dialogue supports speech-only or cross-modal exchanges, while speech translation can take text or speech as input and produce either format.Source: A. Semantic-Related Applications · p. 12
  • SpeechLMs can perform ASR, TTS, and keyword spotting, with keyword spotting offering a way to trigger voice-activated assistants.ASR maps speech to text; TTS maps text to speech, and keyword spotting identifies predefined words or phrases in continuous speech.Source: A. Semantic-Related Applications · p. 12
  • Intent classification maps speech to an intent, while slot filling extracts task-relevant information into predefined classes.The section describes text outputs as a natural format for these results because they are easier to parse and classify.Source: A. Semantic-Related Applications · pp. 12–13
  • Query-by-example spoken term detection lets users search a larger audio stream for a spoken term by supplying an example.Source: A. Semantic-Related Applications · p. 13

B. Speaker-Related Applications

The section surveys speaker-identity tasks and paralinguistic applications, including emotion recognition, speech separation, and controlled expressive generation. It also introduces automatic and human assessments for evaluating SpeechLMs.

B. Speaker-Related Applications

  • Speaker-related applications cover identifying, verifying, and diarizing speakers, as well as generating speech conditioned on a speaker’s voice.SpeechLMs may also distinguish speakers implicitly during multi-speaker conversations; voice-conditioned generation includes cloning and conversion.Source: B. Speaker-Related Applications · pp. 13–14

C. Paralinguistic Applications

  • Paralinguistic applications include recognizing emotion, separating mixed speech, and generating speech with specified characteristics such as style, pace, or singing.Source: C. Paralinguistic Applications · p. 14

VI. EVALUATIONS

  • SpeechLM evaluation uses automatic and human assessments across distinct evaluation aspects, with popular benchmarks summarized in Table VI.The table caption identifies input/output modality, audio, and text as the meanings of I/O, A, and T.Source: §VI · p. 14, Table VI · p. 14

A. Automatic (Objective) Evaluation

The survey organizes SpeechLM evaluation into automatic measures and human judgments, spanning task performance, representations, linguistic and paralinguistic behavior, generation, and real-time interaction.

A. Automatic (Objective) Evaluation

  • Between-speaker ABX evaluates phonetic category separation, while resynthesis WER or CER measures information loss from discretizing speech into tokens.Resynthesis encodes input speech into tokens, synthesizes it, then compares ASR results for the input and resynthesized speech.Source: A. Automatic (Objective) Evaluation · pp. 14–15
  • Lexical, syntactic, and semantic evaluations use sWUGGY, sBLIMP, and Spoken StoryCloze, respectively, comparing negative log-likelihoods of paired data.The paired choices test real-word discrimination, grammaticality, and story-ending selection.Source: A. Automatic (Objective) Evaluation · p. 15
  • Paralinguistic measures assess prosodic token correctness, consistency, and expressiveness, or sentiment preservation in generated text or speech.pGSLM uses min-MAE, Pearson correlation, and standard deviation; SPIRIT-LM uses a sentiment classifier.Source: A. Automatic (Objective) Evaluation · p. 15
  • Generation quality and diversity can be assessed across sampling temperatures with AUC on perplexity and VERT, or with ChatGPT judgments of ASR transcripts.VERT measures repeated k-grams in generated speech; the ChatGPT approach uses ASR output as input for quality and diversity assessment.Source: A. Automatic (Objective) Evaluation · p. 15
  • Real-time evaluation examines turn-taking naturalness and assistant usefulness, including silence during user speech and stopping when interrupted.Methods compare turn-taking event statistics with human dialogue patterns and assess reflective pauses and interruptions.Source: A. Automatic (Objective) Evaluation · p. 15
  • Downstream benchmarks cover speech and audio tasks, but many require text responses; VoxEval evaluates spoken question answering with a speech-output pipeline and varied input conditions.VoxEval also pioneers spoken math reasoning evaluation.Source: A. Automatic (Objective) Evaluation · pp. 15–16

B. Human (Subjective) Evaluation.

  • Human evaluation uses MOS ratings for perceived speech quality, with MMOS, PMOS, and SMOS targeting naturalness, prosody, and timbre similarity.Because recruiting listeners can be difficult, researchers also use trained naturalness-prediction and speaker-identification models.Source: B. Human (Subjective) Evaluation · p. 16

VII. CHALLENGES AND FUTURE DIRECTIONS

The survey identifies open challenges in SpeechLM components, training, latency, safety, and rare-language performance, and outlines directions for future work.

A. Understanding Different Component Choices

  • Existing component comparisons, focused mainly on speech tokenizers, are limited in scope and depth, leaving trade-offs among choices insufficiently understood.The survey calls for comprehensive comparisons to guide more efficient component selection.Source: A. Understanding Different Component… · p. 16

B. End-to-End Training

  • Separate training of SpeechLM components may hinder overall potential; the survey proposes investigating end-to-end training with gradients from vocoder output to tokenizer input.The proposed approach could support more coherent, contextually relevant, and high-fidelity outputs.Source: B. End-to-End Training · p. 16

C. Real-Time Speech Generation

  • A typical vocoder waits for all language-model output tokens, causing noticeable delay; proposed directions include chunked streaming or autonomous waveform generation.The survey describes real-time speech generation as underexplored and requiring further investigation.Source: C. Real-Time Speech Generation · p. 16

D. Safety Risks in SpeechLMs

  • SpeechLM safety is less investigated than TextLM safety, with concerns including harmful semantic or acoustic outputs and privacy risks from speaker or sensitive-trait inference.The survey highlights possible inferences about ethnicity or religious beliefs from insufficient acoustic information.Source: D. Safety Risks in SpeechLMs · p. 16

E. Performance on Rare Languages

VIII. CONCLUSIONS

  • The survey covers SpeechLM architectures, components, training recipes, downstream applications, evaluation methods, and future challenges.It aims to support research toward more powerful SpeechLMs.Source: §VIII · p. 17
Loading 2410.03751v4…