PaperlayerView exact revision on arXiv

Source-linked AI summary

WavChat: A Survey of Spoken Dialogue Models

Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, Xiaoda Yang, Zehan Wang, Qian Yang, Jian Li, Yidi Jiang, Jingzhen He, Yunfei Chu, Jin Xu, Zhou Zhao

arXiv:2411.13577v2eess.AScs.CLcs.LGcs.MMcs.SD

TL;DR

Cascaded spoken dialogue systems can bring high latency, limited interactivity, and difficulty handling non-text information, while end-to-end systems aim to understand and generate speech directly. This survey organizes these systems and their core technologies, and reports progress in streaming and full-duplex interaction alongside remaining gaps in unified interaction and evaluation.

  • Problem

    Cascaded systems can have high latency, limited interactivity, and difficulty processing non-textual information, motivating approaches that address these challenges.

    Source: §1 · p. 2

  • Method

    The survey organizes spoken dialogue models into cascaded and end-to-end paradigms and reviews speech representations, training, streaming, duplex interaction, and data and evaluation.

    Source: §1 · p. 2, §7 · p. 42

  • Results

    Current systems show progress in streaming processing and full-duplex interaction, including reduced response latency and improved interruption handling and conversational naturalness.

    Source: §5.2.3 · p. 33

  • Takeaways & Limitations

    The survey identifies a need for frameworks that handle varied interactions seamlessly and for standardized benchmarks to evaluate interaction capabilities.

    Source: §5.2.3 · p. 33

    Current spoken dialogue systems still lack a unified system that handles all forms of interaction seamlessly.

    Source: §5.2.3 · p. 33

Abstract

from arXiv · show

Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at https://github.com/jishengpeng/WavChat.

1 Introduction

Modern spoken dialogue models extend speech interaction beyond text conversion, while the survey organizes systems by architecture and reviews the technologies underlying their capabilities. It also outlines limitations and future-research considerations across these technologies.

1 Introduction

  • Modern systems generate natural speech and handle acoustic features such as timbre, emotion, and style, alongside music and other speech-related representations.They are also described as offering realistic interaction and low-latency dialogue.Source: §1 · p. 2
  • The survey traces models from early systems to recent ones, orders them chronologically, and distinguishes cascaded from end-to-end architectures.Figure 1’s timeline is mainly based on technical-paper release dates; it omits some systems without published papers.Source: §1 · p. 2, Figure 1 · p. 3
  • Cascaded systems pass audio through ASR, an LLM, and TTS, whereas end-to-end systems directly comprehend and generate speech representations without text as the central intermediary.The survey notes that cascaded systems can have high latency, limited interactivity, and difficulty processing non-textual information.Source: §1 · p. 2
  • The survey reviews speech representations, training and modality alignment, streaming and duplex interaction, and dataset construction and evaluation, discussing limitations and future research.Its sections also cover model architectures and generation strategies.Source: §1 · p. 2

2 Overall

This section introduces spoken dialogue models by defining their scenarios and surveying current systems. It distinguishes cascaded models from end-to-end models.

2 Overall

  • The section defines spoken dialogue models by examining representative spoken-dialogue scenarios.Source: §2 · p. 3
  • It provides an overview of current spoken dialogue models and their development.Source: §2 · p. 3
  • The overview separates cascaded spoken dialogue models from end-to-end spoken dialogue models.Source: §2 · p. 3

2.1 Functions of Spoken Dialogue Systems

The survey presents spoken dialogue systems as requiring both text-model capabilities and speech- and audio-specific abilities. It describes four capability areas here, while noting that text intelligence is currently concentrated in daily scenarios.

2.1 Functions of Spoken Dialogue Systems

  • The survey frames intelligent spoken dialogue as more than speech-to-text interaction, encompassing nine capability areas from text and speech intelligence to streaming and multimodal use.Figure 2 classifies systems by whether the core language model directly understands and generates speech representations.Source: §2.1 · p. 3, Figure 2 · p. 4

2.1.1 Text Intelligence

  • Text intelligence means retaining text-based dialogue capabilities, such as answering requests, but current systems focus more on daily scenarios than complex mathematical reasoning.The latter warrants further exploration in unified multimodal dialogue models.Source: §2.1.1 · p. 4

2.1.2 Speech Intelligence

  • Speech intelligence includes understanding and generating acoustic information beyond words, including timbre, style, emotion, accents, and controllable voice or style.The survey describes adapting responses to vocal cues and generating speech in a prompted voice or designated style.Source: §2.1.2 · pp. 4, 6

2.1.3 Audio and Music Generation

  • Spoken dialogue systems may generate audio and music, including requested musical pieces, sound effects, or songs based on supplied lyrics and melody.Source: §2.1.3 · p. 6

2.1.4 Audio and Music Understanding

The survey extends spoken dialogue capabilities to audio understanding, multilingual speech, and long-form, context-dependent interaction. These capabilities include interpreting audio characteristics, handling different languages and accents, and adapting replies over multiple turns.

2.1.4 Audio and Music Understanding

  • Audio understanding includes identifying sound content and acoustic characteristics, such as a bird call or whether music sounds calm or energetic.The system may also create poetry or songs based on audio or music.Source: §2.1.4 · p. 6

2.1.5 Multilingual Capability

  • Multilingual capability includes translating spoken content and handling acoustic variation by responding in different languages and accents appropriate to the input.Source: §2.1.5 · p. 6

2.1.6 Context Learning

  • Context learning includes long audio inputs, extended speech outputs, multi-turn use of conversation history, and revising prior replies as requests evolve.The survey identifies long-form and multi-turn conversation as a performance benchmark.Source: §2.1.6 · pp. 6–7

2.1.7 Interaction Capability

Spoken dialogue systems support full-duplex interaction, allowing users and systems to interrupt one another in real time. Conversational fillers can also make exchanges more natural.

2.1.7 Interaction Capability

  • Full-duplex interaction lets the system respond before a user finishes and lets either participant interrupt in real time.A system can stop its current generation when the user interrupts, or interrupt the user to clarify intent.Source: §2.1.7 · p. 7
  • Systems can use fillers such as “okay,” “haha,” or “oh” to signal acknowledgment or agreement.Source: §2.1.7 · p. 7
  • Interaction requires the system to listen and speak simultaneously while responding dynamically to the conversation.Source: §2.1.7 · p. 7

2.1.8 Streaming Latency

Spoken dialogue models need chunk-based, causal processing to understand and generate audio in real time. First-token and average generation latency are key factors in responsiveness and usability.

2.1.8 Streaming Latency

  • Chunk-based streaming lets a model process and generate audio dynamically in real time rather than waiting for a full audio segment.Source: §2.1.8 · p. 8
  • Streaming requires causal processing based only on past information, without relying on future audio.Source: §2.1.8 · p. 8
  • First-token wait time and average generation latency are critical factors for conversational responsiveness and usability.Source: §2.1.8 · p. 8

2.1.9 Multimodal Capability

Multimodal spoken dialogue models process inputs such as video, images, and text while generating speech. The survey describes this capability as enabling systems to hear, see, and speak simultaneously.

2.1.9 Multimodal Capability

  • Multimodal dialogue models process video, images, and text inputs while generating intelligent speech responses.Source: §2.1.9 · p. 8
  • The capability allows a system to “hear, see, and speak” simultaneously.Source: §2.1.9 · p. 8
  • Gestures can improve generated responses, while multimodal inputs can deepen a system’s understanding of the physical world.Source: §2.1.9 · p. 8
  • The survey presents unified multimodal comprehension and generation as a future direction, with spoken dialogue as a foundational modality.Source: §2.1.9 · p. 8

2.2 Cascaded Spoken Dialogue Systems

Cascaded spoken dialogue systems have expanded from an ASR–LLM–TTS pipeline toward richer speech, audio, and multimodal capabilities. They remain cascaded when the LLM generates text first and a TTS module converts it to speech.

2.2 Cascaded Spoken Dialogue Systems

  • AudioGPT established a basic cascaded speech-to-speech pipeline: ASR converts speech to text, ChatGPT handles dialogue, and TTS converts text back to speech.The framework used speech as an input-output interface, while an open-source implementation added VAD to distinguish speech from silence and speakers.Source: §2.2 · p. 8
  • ParalinGPT and E-chat [228] combine conversational context, speech embeddings, and paralinguistic attributes to generate text responses; Spoken-LLM [128] uses Emotion2Vec [144] style vectors with Llama2-Chat.LoRA [80] fine-tuning enables text responses with specified stylistic attributes that can guide downstream TTS.Source: §2.2 · p. 8
  • Some cascaded systems [41] [34] [112] feed speech representations directly to an LLM using frozen speech encoders and trainable adapters, treating speech as a specialized text form.The approach extends the text model’s vocabulary so it can process speech and produce text responses.Source: §2.2 · p. 8
  • Cascaded models such as SALMONN [199] extend comprehension to music and other audio, while Qwen-audio [33] [34] supports tasks including ASR, translation, captioning, classification, and audio question answering.SALMONN bridges frozen Whisper and BEATs [28] encoders to an LLM via a Window-Level Q-Former.Source: §2.2 · p. 9
  • Even when models such as Llama3.1 include trainable TTS in the LLM decoder and support streaming, they remain cascaded if they generate text before speech.They do not directly generate speech representations within the LLM.Source: §2.2 · p. 9
  • VITA [61] and Baichuan-Omni [123] add image, audio, and video encoders or tokenizers to LLMs, enabling multimodal input comprehension and text responses.Source: §2.2 · p. 9
  • Cascaded systems pair dialogue models with expressive text-conditioned TTS, including systems that control generated speech style from the text output.Examples include VALL-E (X) [210] [251], CosyVoice [49], ParlerTTS [141], and style-controllable systems such as PromptTTS2 [119] and ControlSpeech [94].Source: §2.2 · p. 9

2.3 End-to-End Spoken Dialogue Systems

The survey classifies systems as end-to-end when their language models directly understand and generate speech representations, rather than generating only text. It traces approaches from early duplex systems to newer designs, which vary in representation, training, architecture, and generation strategy.

2.3 End-to-End Spoken Dialogue Systems

  • End-to-end systems are distinguished by whether the language model directly understands and generates speech representations; text-only generation is classified as cascaded.The survey notes that many systems align speech with text to leverage pretrained language models.Source: §2.3 · p. 9
  • dGSLM is identified as the earliest fully end-to-end dialogue system, using self- and cross-attention on dual-track data to simulate duplex interaction without text.The survey notes that it lacks LLM integration and basic textual intelligence despite its conversational interactivity.Source: §2.3 · p. 9
  • SpeechGPT [243] uses autoregressive speech and text tokens, while PSLM [155] shares weights before separate output projections and Spirit-LM [159] interleaves speech and text.Spirit-LM requires precise speech-text alignment; autoregressive text-then-speech generation can delay speech until text generation finishes.Source: §2.3 · pp. 9–10
  • Moshi [44] and Mini-Omni [223] generate text and speech tokens simultaneously, while Mini-Omni2 adds multimodal understanding and duplex functionality.Moshi models user speech and system responses in parallel streams; Mini-Omni uses padding and batch-parallel inference strategies.Source: §2.3 · p. 10
  • Other designs target real-time or text-free interaction: LLaMA-Omni [57] streams low-latency speech, Freeze-Omni transfers functions to encoder and decoder, and Omni-Flatten [247] trains duplex dialogue.IntrinsicVoice [249] uses GroupFormer to generate HuBERT tokens from LLM hidden states, while Omni-Flatten and SyncLLM [204] avoid text generation during inference.Source: §2.3 · p. 10

3 Representations of Spoken Dialogue Models

This section surveys spoken dialogue representations at the input and output, covering semantic and acoustic features, their trade-offs, and choices among hybrid, continuous, and discrete representations. It also reviews quantizer choices and text guidance, including latency costs and directions for speech-to-speech generation without text guidance.

3 Representations of Spoken Dialogue Models

  • Speech representations bridge speech with other modalities, and semantic tokens prioritize linguistic information while acoustic tokens emphasize compression and high-fidelity reconstruction.The survey organizes common representations from input and output perspectives and discusses their advantages, limitations, and future directions.Source: §3 · pp. 10–11

3.1 Speech Representations at the Inputs

  • Input systems use pretrained semantic encoders to support speech understanding and LLM alignment, while acoustic and hybrid tokenizers capture expressive detail for speech generation.WavLM uses masked speech denoising and prediction, and SpeechVerse downsamples its intermediate features; Mimi distills semantic information while using split-RVQ to avoid forcing acoustic information into semantic quantizer residuals.Source: §3.1 · pp. 11–14

3.2 Speech Representations at the Outputs

  • Output systems predict semantic units or acoustic tokens: semantic units typically need a vocoder, while acoustic tokens can be decoded directly into waveforms.Mimi’s eight codebooks at 12.5Hz require 100 autoregressive steps per second; Moshi’s RQ-Transformer separates temporal and codebook-depth modeling to handle longer sequences or greater depth.Source: §3.2 · pp. 14–15

3.3.1 Semantic Representation vs. Acoustic Representation

  • Semantic representations support LLM alignment and efficient speech encoding, but acoustic representations better preserve prosody, style, and naturalness and can connect directly to codec decoders.Semantic units can require separately trained vocoders or text-to-speech tools, affecting latency; hybrid approaches such as SpeechTokenizer and Mimi distill semantic information into acoustic tokens, and the survey calls for further unified modeling.Source: §3.3.1 · p. 16

3.3.2 Continuous Representation vs. Discrete Representation

  • The input-side choice between continuous features and discrete tokens remains unsettled: continuous inputs require adapters and alignment training, while discrete inputs shift training onto the LLM.For outputs, discrete features support codec decoding, acoustic control, and reuse in the next dialogue round; continuous features may require additional toolkits or vocoders, and their performance relative to discrete inputs remains an open question.Source: §3.3.2 · p. 17

3.3.3 Single-Layer Quantizer vs. Multi-Layer Quantizer

  • Multi-layer RVQ codecs introduce generation latency; delayed steps and parallel VQ generation mitigate it but remain less efficient than semantic representations.Mini-Omni and Mini-Omni 2 use delayed steps, while Moshi splits eight VQs for independent parallel generation.Source: §3.3.3 · p. 17
  • Single-VQ models achieve competitive reconstruction and generation, with WavTokenizer reaching 40Hz compression and potentially reducing feature-extraction and autoregressive-modeling burdens.Examples include WavTokenizer [92], Single-Codec [120], and BigCodec [225].Source: §3.3.3 · p. 17

3.3.4 With Text Guidance vs. Without Text Guidance

  • End-to-end systems commonly generate text representations before speech tokens or generate text and speech tokens in parallel, improving semantic consistency at the cost of response speed.Direct speech-to-speech mapping is challenging [223] [57], motivating text guidance in current systems [57] [249] [223] [44].Source: §3.3.4 · p. 17
  • Direct speech-to-speech generation remains promising, but retraining large spoken language models is constrained by scarce high-quality conversational data and requires multi-stage training.A proposed alternative is a codec that aligns text and speech during encoding, potentially enabling autoregressive speech modeling without text guidance.Source: §3.3.4 · pp. 17–18

4 Training Paradigm of Spoken Dialogue Model

Training spoken dialogue models must transfer speech input and output to text-based LLMs despite limited labeled speech data and speech’s less compact representation. The survey reviews architectural and training choices for preserving language abilities while supporting speech interaction, alignment, and lower-latency generation.

4.1 Architecture Paradigm about Modal Alignment of Speech and Text

  • Text-output-only systems map speech into an LLM but need external TTS; Chain-of-Modality generates text first, guiding speech content but delaying live responses.Text-output-only approaches may overlook emotion, prosody, and non-verbal cues; Chain-of-Modality delays speech until the full text response is complete.Source: §4.1 · pp. 18–20
  • Interleaved tokens, parallel text-speech generation, and speech-to-speech methods offer alternative alignment designs, with parallel generation reducing latency while sometimes compromising quality.Moshi models multiple streams for full-duplex dialogue; SyncLLM uses time chunking for real-time full-duplex interaction.Source: §4.1 · p. 20

4.2 Multi-stage Training strategy

  • Training commonly proceeds through text LLM pretraining, modality adaptation and alignment, supervised fine-tuning, and optional preference optimization to add speech while preserving existing capabilities.Most systems use pretrained LLMs as their text backbone rather than pretraining separately on text.Source: §4.2 · p. 20, §4.2.1 · p. 21

4.2.2 Modality Adaptation and Alignment Post-training

  • Adaptation strategies combine multimodal tasks, unlabeled speech, and speech-specific modules; Mini-Omni separates adapter training from LLM updates to preserve language capabilities.Other approaches include interleaved text-speech pretraining, speech-input/text-response training, and joint supervision of text and speech objectives.Source: §4.2.2 · pp. 21–22

4.2.3 Supervised Fine-tuning or Dialogue Dataset Fine-tuning

  • Dialogue fine-tuning uses instruction or spoken-dialogue data, with some systems training progressively toward half- or full-duplex interaction.Moshi’s dialogue data includes noise and overlap; SyncLLM combines mostly synthetic dialogue data with a smaller amount of real-world data.Source: §4.2.3 · p. 22

4.2.4 Preference Optimization and Reinforcement Learning

  • Preference optimization for spoken dialogue remains scarce; Align-SLM applies offline DPO with semantic-metric preferences and curriculum learning to refine textless speech models.The survey notes that preference optimization has been used for TTS but is virtually absent from spoken dialogue models.Source: §4.2.4 · pp. 22–23

4.3 Training Frameworks and Generation Strategies

  • Generation frameworks use streaming decoders, delayed parallel decoding, grouped speech-token prediction, hierarchical codebooks, or synchronized autoregressive streams to manage latency and long sequences.LLaMA-Omni streams speech alongside text; IntrinsicVoice predicts speech-token groups in one step, while Moshi hierarchically models multi-codebook audio tokens.Source: §4.3 · pp. 23–24

4.4.1 Text and Speech Modality Alignment

  • Paired speech-text post-training can cause catastrophic forgetting because paired data is imbalanced against large text pretraining corpora, potentially harming LLM text capabilities.The survey calls for precise parameter design and customized optimization to mitigate this issue.Source: §4.4.1 · p. 24

5 Streaming, Duplex, and Interaction

This section surveys streaming, duplex, and interaction designs in cascaded and end-to-end systems, focusing on progressive speech processing, real-time speaking and listening, turn-taking, and interruptions. It also covers reported latency and interruption results and notes the lack of a unified interaction system and standardized evaluation benchmarks.

5.1 Streaming Spoken Dialogue Models

  • Streaming requires both input understanding during speech and progressive output generation, so models can respond without waiting for complete input or intermediate states.End-to-end systems use causal convolution, causal attention, and queue management; some models use non-streaming Whisper encoders and reduce latency on the output side.Source: §5.1 · p. 26, §5.1.1 · pp. 26–27

5.1.1 Streaming End-to-End Spoken Dialogue Models

  • End-to-end streaming designs rely on causal convolution, causal attention, and queue management; some models use non-streaming Whisper encoders and compensate with lower-latency output.Causal convolution and attention restrict processing to current and past inputs, while queue management processes audio frames in order.Source: §5.1.1 · pp. 26–27

5.1.2 Streaming Cascaded Spoken Dialogue Models

  • Combining streaming U2++ Conformer ASR [220] and XTTS-v2 TTS [21] achieved the lowest latency and significantly improved interruption capabilities compared with the non-streaming alternatives.The comparison also included Whisper ASR and VITS TTS [110].Source: §5.1.2 · p. 27

5.2.1 Duplex Technology

  • Full-duplex systems listen while speaking and allow overlapping speech, backchannels, and interruptions, unlike half-duplex systems that require turn-taking and introduce pauses.Achieving fluid turn-taking and precise interaction timing remains challenging.Source: §5.2.1 · pp. 28–29

5.2.2 Interaction

  • Turn-taking involves cues, turn-end detection or prediction, and overlap; early VAD-based systems often mishandled backchannels, causing unnecessary interruptions and delays.Cascaded systems explicitly manage dialogue turns, whereas end-to-end systems learn interaction modeling from training data.Source: §5.2.2 · pp. 29, 31
  • Threefold lower response latency, responses within 500 milliseconds in over 50% of dialogues, and 96.7% interruption handling with 54.7% interruption accuracy were reported for a full-duplex streaming system.Its perception and action modules use streaming ASR and TTS, while a neural finite-state machine switches between speaking and listening states.Source: §5.2.2 · p. 30
  • dGSLM models dialogue directly from raw audio using a dual-tower Transformer with cross-attention over parallel speaker channels, capturing turn-taking, backchannels, silences, and interaction events.It uses HuBERT audio tokens and autoregressive prediction of token and duration, followed by audio reconstruction.Source: §5.2.2 · p. 31
  • Moshi combines a text language model, audio codec, and joint text-audio inner monologue in multi-stream streaming inference; its latency is 160 milliseconds theoretically and 200 milliseconds in practice.It is pretrained on unsupervised audio, fine-tuned on Fisher for overlapping speech and interruptions, then optimized on an instruction-tuning dataset.Source: §5.2.2 · p. 32
  • Parrot predicts paired tokens across separate speaker channels for streaming turn-taking, overlap, and interruptions, while SyncLLM alternates user and system audio chunks to synchronize responses.SyncLLM predicts user speech at each time step before generating the next system chunk and uses de-duplicated speech tokens and synchronization markers for real-time inference.Source: §5.2.2 · pp. 32–33
  • Mini-Omni2 uses irq and n-irq markers to stop generation and return to listening, while OmniFlatten progressively removes text streams to enable low-latency speech generation during user input.Mini-Omni2 trains interruption handling with command phrases in noisy audio; OmniFlatten progresses from half-duplex dialogue to pure speech-stream generation using block-by-block processing.Source: §5.2.2 · pp. 32–33
  • Freeze-Omni [214] predicts one of three chunk-level conversation states to decide whether to keep listening, provide immediate feedback, or generate a response.Voice activity detection initiates chunk-wise processing, and state predictions guide when the model continues listening or transitions to response generation.Source: §5.2.2 · p. 33

5.2.3 Discussions about streaming and interaction

  • Despite progress in latency, interruption handling, and conversational naturalness, current systems lack a unified approach to interaction and standardized benchmarks for comparing interaction capabilities.Future research could address both user interruptions and system interruptions and develop a unified evaluation benchmark.Source: §5.2.3 · p. 33

6 Training Resources and Evaluation

This section surveys staged training resources for spoken dialogue models, from text-language foundations through audio adaptation and conversational tuning, and reviews evaluation benchmarks across audio understanding, task-oriented interaction, reasoning, and response quality. It also covers differences in task coverage, input formats, and evaluation methods, as well as limited public test sets, metrics, and unresolved evaluation challenges.

6.1.1 Training resources about Text LLM Pre-training

  • Text-language-model pre-training supplies the vocabulary, grammar, contextual knowledge, and reasoning foundation on which most spoken dialogue systems are built.The survey notes that most systems use existing open-source text models, but does not examine this stage in detail.Source: §6.1.1 · p. 34

6.1.2 Training resources about Post-Train for Audio Modal Alignment

  • Audio modal adaptation adds encoders, decoders, and task or speaker tokens, using paired audio-text tasks to align speech understanding and generation with text.Common tasks include TTS, ASR, and audio captioning; training may use single-channel audio, word-level alignment, and balanced text-audio data to reduce forgetting.Source: §6.1.2 · p. 34

6.1.3 Training resources about Post-Train for Dual-Stream Dialogue Processing

  • Dual-stream training synthesizes filtered text dialogues into alternating user and assistant audio so models can listen while speaking without complex interaction modeling.Text is filtered for code, formulas, rare symbols, and samples over 200 words; varied user voices, pauses, interruptions, noise, and echo can make the synthesized data more realistic.Source: §6.1.3 · p. 35

6.1.4 Training resources about Enhancing Conversational Abilities and Instruction Tuning

  • Conversational tuning addresses non-overlapping synthetic dialogue and the lack of real-world speech-instruction datasets through dialogue and instruction-data synthesis.Real conversational speech can support interruption and backchannel modeling; synthetic instructions pair speech and text instructions with text and speech responses, while generated dialogues can include role-play, errors, and brief turns.Source: §6.1.4 · pp. 35–36

6.2 Evaluation

  • Evaluation lacks public test sets, comprehensive metrics, and established benchmarks, and the survey proposes eleven factors spanning speech, text, interaction, latency, multimodality, and safety.The framework also includes audio and music understanding and generation, multilingual capability, and context learning.Source: §6.2 · p. 36

6.2.1 Common Evaluation

  • Common evaluation separates text intelligence, speech quality, and streaming latency, using accuracy or similarity metrics, subjective ratings, error rates, first-token wait time, and real-time factor.Text evaluations distinguish closed-ended accuracy metrics from open-ended text-similarity metrics; speech quality uses MOS for clarity and naturalness and WER or CER for word errors.Source: §6.2.1 · pp. 36–38

6.2.2 Advanced Evaluation

  • Advanced evaluation covers paralinguistic speech understanding and controllable speech generation, while audio and music tasks use objective, similarity, or subjective measures suited to their outputs.Speech understanding can assess emotion, age, or accent and response relevance; audio classification and event detection use accuracy or mAP, while open-ended captions and creative writing use similarity or subjective judgments. Audio and music generation can use MOS and similarity measures.Source: §6.2.2 · pp. 38–39
  • Advanced evaluation also considers context learning, interruption handling, multilingual and multimodal dialogue, and security, but online understanding, visual grounding, and speech defenses remain underexplored.The section also describes language identification with human assessment for conversational multilingual ability and notes that speech security benchmarks need further development.Source: §6.2.2 · pp. 39–40

6.3 Benchmark

  • Conventional evaluation uses public datasets and metrics to assess generalizability and task-specific performance across content recognition, speaker modeling, semantic understanding, and emotion recognition.Tasks include phoneme recognition, ASR, keyword spotting, spoken term detection, speaker identification and verification, diarization, intent classification, and slot filling.Source: §6.3 · p. 41
  • AudioBench [206] covers speech, audio-scene, and voice understanding across eight tasks and 26 datasets, including seven newly developed datasets.It emphasizes instruction-following conditioned on audio, including recognition, environmental sound interpretation, and paralinguistic feature extraction.Source: §6.3 · p. 41
  • AIR-Bench [235] combines 19 audio tasks and over 19,000 foundation questions with more than 2,000 open-ended chat questions, using audio mixing and GPT-4-based evaluation.Its tasks span speech recognition, acoustic scenes, music, and complex real-world audio queries.Source: §6.3 · p. 41
  • SpokenWOZ evaluates spoken task-oriented dialogue under incremental processing, disfluencies, incomplete utterances, and ASR noise, with cross-turn and reasoning slot-detection metrics.It includes multi-domain human-to-human dialogues with over 200,000 utterances and 249 hours of audio, tested through textual and auditory inputs.Source: §6.3 · p. 41
  • SD-Eval [7] assesses spoken understanding and response generation through content, paralinguistic cues, and environmental context across four subtasks and 7,303 utterances.Its emotion, accent, age, and environment tasks span 8.76 hours of audio.Source: §6.3 · p. 41
  • SuperCLUE evaluates voice interaction, general capabilities, scenario applications, and response speed, but is not open source and focuses on Mandarin ability.Metrics include interruption recognition, speech naturalness, memory, retrieval, reasoning, multilingual translation, scenario performance, and latency.Source: §6.3 · p. 41
  • MMAU [183] assesses 27 reasoning and information-extraction tasks across speech, sound, and music, including temporal reasoning, speaker roles, emotion, eco-acoustics, phonemic stress, and melody.It also tests consistency across difficulty levels and robustness in real-world audio scenarios.Source: §6.3 · pp. 41–42

7 Conclusion

The survey organizes spoken dialogue models and reviews the technologies, resources, and evaluation approaches used to study them. It also identifies areas, including semantic and acoustic representations, where established paradigms remain lacking.

7 Conclusion

  • The survey categorizes spoken dialogue models into cascaded and end-to-end paradigms.It also reviews research on these models systematically.Source: §7 · p. 42
  • It classifies speech representations from input and output perspectives, covering semantic and acoustic representations.Source: §7 · p. 42
  • It reviews five modalities of alignment, multi-stage training strategies, model architectures, and generation paradigms.Source: §7 · p. 42
  • It analyzes streaming input and output alongside duplex interaction technologies.Source: §7 · p. 42
  • It compiles training resources, evaluation metrics, and benchmarks, and addresses evaluation across intelligence levels and scenarios.The survey notes that semantic and acoustic representations still lack well-established paradigms and includes discussion of open issues at each section’s end.Source: §7 · p. 42

A Resources about Music and Sound Datasets

The appendix lists music and non-speech sound datasets covering environmental sounds, music, and emotional sounds, as resources for developing voice dialogue systems.

A Resources about Music and Sound Datasets

B Open-source Spoken Dialogue Models

The appendix provides a comprehensive list of publicly available, open-source spoken dialogue models in Table 5.

B Open-source Spoken Dialogue Models

C Open-source Codec Models

This section presents a comprehensive list of publicly available, open-source codec models. Table 6 includes model URLs.

C Open-source Codec Models

Related papers

Loading 2411.13577v2…