PaperlayerView exact revision on arXiv

Source-linked AI summary

On The Landscape of Spoken Language Models: A Comprehensive Survey

Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yi Lee, Karen Livescu, Shinji Watanabe

arXiv:2504.08528v2cs.CLcs.SDeess.AS

TL;DR

Truly universal speech processing systems remain an unmet goal amid diverse spoken language models and evaluation settings. This survey unifies recent literature by reviewing SLM definitions, components, architectures, training, evaluation, successes, limitations, and future directions.

  • Problem

    Truly universal speech processing systems remain an unmet goal, motivating clearer understanding of existing SLMs and their remaining challenges.

  • Method

    The survey reviews recent SLM literature and provides a unified definition of SLMs and their components while organizing work by architecture, training, and evaluation choices.

  • Results

    The survey provides a snapshot of the field’s evolution that places new models in context and reviews SLM successes and remaining limitations.

  • Takeaways & Limitations

    The survey identifies challenges and open questions that suggest directions for future research toward universal speech processing systems.

  • Takeaways & Limitations

    Public high-quality training data, especially for instruction tuning and chat-based training, is lacking, while design and training ablations remain insufficient.

Abstract

from arXiv · show

The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act as universal speech processing systems. This trend is similar to the progression toward universal language models that has taken place in the field of (text) natural language processing. SLMs include both "pure" language models of speech -- models of the distribution of tokenized speech sequences -- and models that combine speech encoders with text language models, often including both spoken and written input or output. Work in this area is very diverse, with a range of terminology and evaluation settings. This paper aims to contribute an improved understanding of SLMs via a unifying literature survey of recent work in the context of the evolution of the field. Our survey categorizes the work in this area by model architecture, training, and evaluation choices, and describes some key challenges and directions for future work.

1 Introduction

Spoken language processing is moving toward spoken language models intended to generalize across speech tasks, paralleling text language models while retaining diverse model types and evaluation settings. This survey organizes that landscape with unified terminology, component descriptions, and a review of successes and limitations.

  • Speech processing is progressing from task-specific systems toward spoken language models intended as universal speech processing systems.
  • SLMs encompass pure speech models, speech+text models, and speech-aware text language models with different distributions, inputs, and outputs.
  • The survey defines universal speech processing systems as models with spoken input and output that can address arbitrary spoken language tasks.
  • Many existing models do not yet provide both speech input and output or support a broad task range, but remain steps toward universality.
  • End-to-end SLMs can retain acoustic information and avoid some cascaded error compounding, whereas cascades remain strong baselines for many tasks.
  • The survey addresses inconsistent terminology and heterogeneous evaluations by unifying SLM definitions and organizing recent literature across modeling choices.

2 Overall architecture

SLMs provide a unified architecture that accepts speech and/or text and generates speech and/or text, using encoders, adapters, and an autoregressive sequence model. The formulation covers pure speech, speech-aware text, and jointly modeled speech+text systems.

  • SLMs take speech and/or text as input and generate speech and/or text, with at least one input or output modality being speech.
  • Speech waveforms are encoded into continuous vectors or discrete tokens before modality adaptation.The encoder maps a waveform of length T to representations of length L, which may be continuous or discrete.
  • The modality adapter aligns speech representations with the sequence model and can reduce their temporal length.It transforms representations to dimension d′, typically with L′ ≤ L, while addressing speech–text length disparities.
  • A sequence model typically generates outputs autoregressively, appending each generated representation before producing the next.Pure speech LMs generate continuous vectors or discrete tokens that are decoded into waveforms.
  • Speech-aware text LMs concatenate adapted speech and text representations, then generate text tokens sequentially.The concatenated representation has shape (L′ + N′) × d′, and generated tokens are appended to the text sequence.
  • Speech+text LMs jointly model hybrid speech and text representations for generation.The survey treats hybrid output strategies as a distinct formulation discussed alongside pure speech and speech-aware text models.

3 SLM components

SLM components transform waveforms into speech representations, adapt them for sequence modeling, and generate outputs using different tokenization and decoding choices. These choices trade efficiency, fidelity, linguistic content, and downstream suitability.

  • Speech encoders: Speech encoders extract meaningful representations from continuous waveforms using learned models or DSP feature extractors.Inputs may include mel filter banks, SSL or supervised hidden representations, and neural audio codec representations.
  • Speech encoders: SLMs commonly use discrete speech tokens, while speech-aware text LMs generally use continuous representations.Temporal compression such as deduplication or BPE may reduce sequence length after representation extraction or quantization.
  • Discrete tokens: Phonetic tokens are quantized speech representations that resemble phonetic units and reduce speaker-specific information.They support language-focused understanding and applications such as speech-to-speech translation, although “semantic tokens” is an alternative term the survey avoids.
  • Discrete tokens: Audio codec tokens capture detailed acoustic characteristics and originate from neural codecs designed for faithful audio reconstruction.A neural codec uses an encoder, vector quantization module, and decoder; intermediate discrete representations can serve as SLM tokens.
  • Discrete tokens: Residual vector quantization generates multiple hierarchical tokens per time step, requiring specialized SLM decoding strategies.The first level encodes most information, while later levels encode residual information.
  • Discrete tokens: Hybrid tokenization combines phonetic and acoustic information through distilled representations, linguistic integration, or language-model objectives.The survey identifies these approaches as an active research direction.
  • Discrete tokens: Tokenization balances token bit rate against token quality, which affects efficiency, generation quality, and downstream-task suitability.Benchmarks such as Codec-SUPERB and DASB evaluate different token types and codec quality.
  • Speech modality adapters: Modality adapters align separately developed speech encoders with sequence-model representations and are typically trained on downstream tasks or during pre-training.A comparison found that Q-Former outperformed CNN-based adapters on Dynamic-SUPERB.

3.3 Sequence Model

SLMs generate speech using hierarchical token strategies that balance granularity, quality, consistency, complexity, and latency. They also combine text and speech tokens through several alignment and padding designs, then decode speech representations into waveforms.

  • Hierarchical token generation: Speech generation commonly separates coarse phonetic or codec tokens from fine-grained codec tokens, reflecting their different granularities.Phonetic tokens can provide intermediate supervision for TTS and reduce modeling complexity compared with direct text-to-codec generation.
  • Hierarchical token generation: Four decoding strategies generate hierarchical tokens: coarse-first, interleaved, temporal-plus-depth, and delayed patterns.These strategies differ in whether granularity levels are staged, interleaved by time step, split across temporal and depth modules, or offset for look-ahead.
  • Hierarchical token generation: Multiple prediction stages improve audio quality and long-term consistency but increase decoding complexity and latency, limiting suitability for real-time dialogue.Tokenization methods that merge phonetic and codec information aim to reduce the number of layers and address this trade-off.
  • Text-speech hybrid generation: Hybrid text-speech generation must reconcile different sequence lengths and temporal scales while seeking synchronization.Representative designs use end padding, fixed padding between tokens, dynamically learned padding, or interleaving based on time-aligned data.
  • Speech decoding: Speech decoders convert continuous features, phonetic tokens, or codec tokens into waveforms using vocoders, unit-based vocoders, or codec decoders.Codec decoders can directly transform generated audio codec tokens into waveforms without additional training.

4 Training Strategies

The survey distinguishes SLM pre-training from post-training by whether training targets broad downstream speech capability. It organizes speech-specific pre-training, modality alignment, and instruction-oriented post-training strategies.

  • Training phase definitions: Pre-training excludes an explicit goal of enabling many downstream speech tasks, whereas post-training targets a more universal speech processing system.Text LM post-training performed before adding speech is counted as SLM pre-training because it does not target universal speech processing.
  • Pre-training strategies: Speech-specific pre-training includes pure speech modeling, joint speech-text modeling, continual modality or domain adaptation, and conditional-model initialization.Pure speech models autoregressively predict tokenized speech from unlabeled data, while joint models learn aligned speech-text distributions.
  • Modality alignment: Speech and text can be aligned implicitly through matching responses across modalities or explicitly by minimizing distances between speech features and transcript embeddings.Explicit alignment methods can keep text embeddings fixed while optimizing correspondence with speech features.
  • Post-training strategies: Post-training may begin with predefined multi-task learning using task specifiers before instruction tuning or other universal-task approaches.Instruction tuning trains on speech, an instruction in text or speech, and the corresponding ground-truth output.
  • Instruction-tuning data: Instruction-tuning data is created by converting task-specific datasets, generating speech question-answer data with LLMs, or applying TTS to textual instruction and conversation datasets.These sources provide varied task instructions, contexts, response styles, and output formats.

5 Survey of representative SLMs

Representative SLMs fall into pure speech LMs, speech+text LMs, and speech-aware text LMs, with substantial variation in architecture, training, and evaluation. The survey highlights unresolved coverage, reproducibility, and parameter-update trade-offs.

  • Reproducibility: Industry multimodal models broaden speech-language capabilities, but their configurations are undisclosed and rigorous evaluations have focused on limited speech tasks.The survey therefore identifies open-source, reproducible approaches as an important complement.
  • Pure speech LMs: Pure speech LMs model p(speech) from unlabeled tokenized speech and can incorporate pitch and duration to improve prosody modeling.They are the spoken analogue of generative text LMs and typically use discrete phonetic representations.
  • Speech+text LMs: Speech+text LMs jointly model p(text, speech), enabling understanding and generation across both modalities.Moshi is an open-source speech-in-speech-out dialogue system with real-time inference, using time-aligned text and discrete speech representations.
  • Speech-aware text LMs: Speech-aware text LMs combine speech encoders with text LMs, commonly using modality alignment and speech instruction tuning to support diverse tasks.They inherit linguistic and instruction-following capabilities from pretrained text LMs while adapting to speech inputs.
  • Representative designs: Speech-aware text LMs differ in task coverage and initialization, with examples spanning speech classification, unseen-task generalization, paralinguistic analysis, and alternative encoder or sequence architectures.WavPrompt was not evaluated on unseen tasks, whereas SALMONN generalized to unseen tasks after ASR, captioning, and broader speech-understanding training.
  • Open design issues: The field has not thoroughly explored how freezing or updating pretrained parameters trades off retained text knowledge against learning speech-specific information.This is especially relevant for tasks requiring non-textual information such as speech emotion.

6 Duplex speech dialogue

Natural speech dialogue is duplex rather than turn-based: users and SLMs may listen, speak, backchannel, and interrupt simultaneously. Proposed systems use dual channels or time multiplexing, but interactive evaluation remains difficult.

  • Duplex dialogue challenges: Full-duplex dialogue requires an SLM to detect when a user has finished, produce backchannels or non-verbal vocalizations, and handle interruptions.These interaction demands do not arise in text-based dialogue.
  • Dual-channel approach: Dual-channel systems continuously receive user input on a listening channel while directing generated speech and silence tokens through a speaking channel.They can track both incoming speech and the model’s own spoken output.
  • Dual-channel approach: Dual-channel designs require specialized architectures, including dual-tower transformers with cross-attention or modified transformer input structures.This departs from a typical autoregressive sequence-model architecture.
  • Time multiplexing: Time multiplexing uses one channel that switches between listening without generation and speaking autoregressively from the model’s own output.Its single-channel decoder-only structure can be initialized from a text LLM.
  • Time multiplexing: Time-multiplexed systems either alternate fixed-duration listening and speaking chunks or let the model trigger switching with special [speak] and [listen] tokens.The latter design predicts when to respond during listening and returns to listening after generating output.
  • Evaluation: Most benchmarks evaluate SLMs offline rather than interactively, making it challenging to measure whether duplex architectures achieve natural user interaction.The survey identifies interactive dialogue benchmarking as an ongoing community effort.

7 Benchmarking and Evaluating SLMs

SLM evaluation spans likelihood, generation, interaction, conversation, safety, bias, and security, because no single benchmark captures their diverse capabilities. Results reveal persistent trade-offs: cascades remain strong semantically, while end-to-end systems better preserve acoustic and conversational behavior but face latency, capacity, and training risks.

  • Likelihood-Based Evaluation: Speech tokenization substantially shapes sequence-modeling performance and should be evaluated jointly with the sequence model.HuBERT tokens outperform quantized log-mel spectrograms, while higher-compression speech tokenizers yield better speech-modeling performance.
  • Likelihood-Based Evaluation: SLM benchmarks measure complementary capabilities, and performance across content, sentiment, acoustic alignment, and conversational tasks is not necessarily correlated.Models can perform well on sentiment or text modeling yet remain near-random on acoustic-text alignment; content-modeling leaders are not always strongest on acoustic details.
  • Generative Metrics: Generative evaluation commonly transcribes generated speech and measures text-LM perplexity, while VERT combines perplexity with auto-BLEU diversity to reduce repetition sensitivity.Text LMs may assign high probability to repeated content, motivating a metric that balances quality and within-sentence diversity.
  • Generative Metrics: Cascaded ASR→LLM systems typically excel at semantic tasks but underperform end-to-end models on speaker-related and paralinguistic tasks.Jointly training the speech encoder with the LLM can improve instruction following and paralinguistic understanding, but risks catastrophic forgetting.
  • Evaluating Interactivity: Full-duplex cascades offer modularity and strong semantic modeling but often add latency and lose fine-grained conversational behavior, whereas end-to-end models improve turn-taking and backchanneling.Robust end-to-end full-duplex architectures remain challenging because they must model listening and speaking channels while preserving overall capacity.
  • Evaluating Interactivity: Latency is measured as the delay between the user stopping speech and the model beginning its spoken response, with reported values depending on system conditions.Studies also use user evaluations: a VAD-based cascade yielded turns better when interrupted, while the cascade and Moshi shared some behaviors.
  • Hallucination: Hallucination evaluations find that tested SLMs hallucinate more than a simple audio-captioning-plus-text-LLM cascade on questions about sound events and their sources.A proposed mitigation prompts the model to describe the audio first and then answer the instruction based on that description.
  • Safety and Bias: Toxicity evaluations generally analyze transcriptions of generated speech, so they assess verbal content rather than nonverbal or acoustic harms.Bias evaluation can additionally vary speaker characteristics such as gender and age, but regular benchmarking remains necessary as models mature.

8 Challenges and future work

Current SLMs remain far from universal speech processing systems, with unresolved challenges in architecture, training data, evaluation, openness, inclusiveness, and safety.

  • Current SLMs are not yet close to universal speech processing systems, motivating several categories of open questions and future research.
  • Model architecture: The optimal speech representation and the best method for combining speech and text remain unclear because alternatives have not been thoroughly compared.
  • Model architecture: Most current SLMs are large and slow, limiting their practicality for real-time and on-device settings.Efficiency challenges also arise when multiple pretrained components with different architectures and frame rates are combined.
  • Training: High-quality public training data is scarce, especially for instruction tuning and chat-based training, while dataset diversity complicates attribution of performance differences.Controlled ablations and scaling studies are also needed to separate design effects from training-data effects and understand scaling behavior.
  • Evaluation: Benchmark adoption remains limited, existing evaluations do not cover the full range of spoken-language tasks, and standardized speech-generation, latency, and turn-taking tests are needed.Dynamic-SUPERB Phase-2 includes 180 tasks, but speech encompasses text-related tasks plus speaker, accent, and prosody-specific tasks.
  • Open research: Few SLMs are fully open-source, making comprehensive comparisons between approaches virtually impossible.
  • Inclusiveness and safety: Future SLMs should serve a broader range of users, including diverse languages, dialects, and speech-related medical conditions, while addressing speech-specific safety concerns.

A Appendix

The appendix tables organize selected spoken language models and summarize their architectures, training and evaluation strategies, tasks, data, and findings.

  • Table 4 lists selected spoken language models and identifies their speech representations, encoders, and decoders.It distinguishes phonetic, acoustic, and continuous representations and references components not explicitly cited in the table.
  • Table 5 continues the listing of selected spoken language models and their associated architectural components.
  • SALM demonstrates in-context learning for speech-content tasks, while SALMONN reports generalization to unseen tasks and task overfitting.
  • Qwen-Audio and SpeechVerse scale training across diverse tasks, with SpeechVerse also studying inference strategies for unseen-task generalization.
  • DiscreteSLU combines speech encoders and uses discrete tokens for speech-aware text language modeling.
  • BLSP-Emo, DeSTA, Qwen2-Audio, Phi-4-Multimodal, Audio-Reasoner, and DeSTA2.5-Audio extend speech-language modeling through emotion understanding, alignment, multimodal routing, reasoning, and additional training data.
  • Table 6 summarizes representative training and evaluation strategies for speech-aware text language models, including tasks, training data, and key findings.
Loading 2504.08528v2…