PaperlayerView exact revision on arXiv

Source-linked AI summary

URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models

Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, Xie Chen

arXiv:2502.17810v3cs.CLeess.AS

TL;DR

Spoken dialogue models need evaluation that covers both cognitive abilities and speech-related behavior, but comprehensive S2S benchmarks are lacking. URO-Bench evaluates understanding, reasoning, and oral conversation across basic and pro tracks, and finds that open-source models do relatively well on daily tasks while retaining major capability gaps.

  • Problem

    S2S spoken dialogue models lack comprehensive evaluation covering cognitive abilities and speech-related aspects such as paralinguistics and audio quality.

    Source: §1 · p. 1

  • Method

    URO-Bench evaluates understanding, reasoning, and oral conversation across basic and pro tracks, including multilingual, multiround, and paralinguistic tasks.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Open-source SDMs perform relatively well on everyday conversation but most lag cascaded systems and GPT-4o-Audio-Preview in instruction-following and reasoning and struggle with multilingual and paralinguistic tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    URO-Bench provides a multifaceted evaluation of current SDMs and a released leaderboard for comparing their performance over time.

    Source: §5 · p. 9

    First-packet latency lacks an automatic evaluation pipeline; LLM scoring may be biased or fluctuate, and costly API scoring limits singing and recitation test-set size.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have driven significant progress in end-to-end spoken dialogue models (SDMs). In contrast to text-based LLMs, the evaluation framework for SDMs should encompass both cognitive dimensions (e.g., logical reasoning, knowledge) and speech-related aspects (e.g., paralinguistic cues, audio quality). However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose URO-Bench, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model's abilities in Understanding, Reasoning, and Oral conversation. Evaluations on our proposed benchmark reveal that current open-source SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.

1 Introduction

URO-Bench addresses the lack of comprehensive S2S evaluation by assessing understanding, reasoning, and oral conversation across basic and pro tracks. Evaluations find that open-source SDMs handle everyday tasks relatively well but retain substantial gaps in instruction-following, reasoning, multilingualism, and paralinguistics.

1 Introduction

  • URO-Bench fills a comprehensive S2S evaluation gap by testing cognitive abilities alongside speech quality and paralinguistic information.The paper argues that this gap makes current SDMs’ capabilities and shortcomings difficult to understand.Source: §1 · p. 1
  • The benchmark has basic and pro tracks with 20 test sets each, spanning English, Chinese, multilingual tasks, multiround dialogue, and paralinguistics.The basic track targets general knowledge, instruction-following, and reasoning; the pro track tests context management, linguistic adaptability, and non-verbal perception.Source: §1 · p. 2
  • Open-source SDMs perform relatively well on everyday conversation, but most lag cascaded systems and GPT-4o-Audio-Preview in instruction-following and reasoning and struggle with multilingual and paralinguistic tasks.Some models show foundational multiround-dialogue ability.Source: §1 · p. 2
  • The authors released URO-Bench’s code and datasets to support evaluation and comparison of spoken dialogue models.Source: §1 · p. 2

2 Related Work

Speech language models accept audio and text in different combinations, while existing benchmarks cover only parts of spoken dialogue evaluation. URO-Bench targets S2S evaluation gaps in multilingualism, multiround dialogues, and non-verbal aspects.

2.1 Speech Language Models

  • S2S models take background information and instructions as audio and return audio responses, unlike models that answer in text.The paper describes this format as suited to daily spoken conversation while imposing higher capability demands.Source: §2.1 · p. 2

2.2 Benchmark for SLMs

  • Existing benchmarks leave gaps in S2S coverage: some use text instructions or outputs, focus on English, or omit multi-turn conversation.Source: §2.2 · p. 3
  • URO-Bench broadens benchmark coverage to multilingualism, multi-round dialogue, and non-verbal aspects of speech-to-speech conversation.Source: §2.2 · p. 3

3 URO-Bench

URO-Bench organizes S2S evaluation around understanding, reasoning, and oral conversation, with basic and pro tracks spanning daily tasks and more challenging capabilities. Its construction pipeline combines curated and generated data with quality checks, and its metrics assess response content and speech properties.

3.1 Overall Design

  • The pro track extends daily-conversation testing to emotion, music, environmental sounds, code switching, advanced mathematics, multilingual processing, speaker recognition, and multiround memory.The basic track contains relatively simple daily-conversation tasks; the pro track is intended as a more challenging, future-oriented version.Source: §3.1 · p. 3
  • The benchmark’s 40 datasets comprise 20 basic and 20 pro test sets across 20 tasks, with English and Chinese versions and a seven-language multilingual dataset.Understanding tests information extraction and instruction-following; reasoning tests analytical skills, mathematics, and general knowledge; oral conversation tests casual chat and audio generation.Source: §3.1 · p. 4

3.2 Benchmark Construction Pipeline

  • URO-Bench draws on existing reasoning datasets and LLM-generated conversational data, synthesizes text as speech, and reviews data before and after synthesis.Source: §3.1 · p. 4

3.3 Evaluation Metrics

  • Task accomplish score evaluates response content; UTMOS assesses speech quality, WER/CER speech-text alignment, and first packet latency response speed.Source: §3.1 · p. 4

3.2 Benchmark Construction Pipeline

  • The construction pipeline filters TTS-unsuitable text and removes generated samples with scenario mismatches or incorrect reference answers.Source: §3.2 · p. 5
  • ASR filtering discards samples exceeding a 2% WER/CER threshold, followed by manual listening checks; a 1,000-sample miniset supports quick evaluation before full-scale assessment.The miniset contains 25 manually selected samples per dataset.Source: §3.2 · p. 5

3.3 Evaluation Metrics

  • For most tests, task accomplish scores use ASR transcripts assessed by GPT-4o mini against accuracy, relevance, clarity, and completeness criteria.Repeat uses WER against the ground truth, while expressiveness tasks use audio-capable evaluators or emotion2vec.Source: §3.3 · pp. 5–6
  • First packet latency measures the interval from user input to generation of the first output-audio segment, using AlpacaEval and AlpacaEval-zh samples.Source: §3.3 · p. 6

4 Evaluation

URO-Bench evaluates spoken dialogue models across task performance, speech quality, and human judgments. Results show strengths in everyday conversation but persistent gaps in instruction-following, reasoning, paralinguistics, and audio understanding.

4.1 Experiment Setup

  • The evaluation compares open-source SDMs with backbone LLMs and cascaded or proprietary references, including GPT-4o-Audio-Preview.The main results are reported across three task-accomplish dimensions, with separate results for speech quality and latency.Source: §4.1 · p. 6, §4.2 · p. 6, Table 4 · p. 7

4.2 Results and Analysis

  • GLM-4-Voice outperforms other SDMs by at least 10 points in each score, yet open-source SDMs still trail GPT-4o-Audio-Preview, especially in reasoning, audio understanding, and multilingual ability.Mini-Omni, Mini-Omni2, and SLAM-Omni average about 25; the authors attribute GLM-4-Voice's lead to its stronger backbone and larger training-data volume.Source: §4.2 · p. 6
  • Most SDMs struggle with instruction-following, while models other than GLM-4-Voice show reduced reasoning and general-knowledge performance relative to their backbone LLMs.Oral-conversation performance is generally satisfactory, and most models show a basic grasp of those tasks.Source: §4.2 · pp. 6–7
  • Pro-track results show weak environmental-sound and music interpretation, speaker recognition, and multilingual speech generation, with GLM-4-Voice and SLAM-Omni showing some multi-turn context and memory ability.GLM-4-Voice is the only model described as somewhat acceptable at speech-emotion generation; all SDMs struggle with singing and recitation.Source: §4.2 · pp. 7–8
  • Larger models’ pauses and repetitions can lower UTMOS and raise ASR-WER / CER, while cross-lingual output mixing can raise ASR-WER on English tasks.GPT-4o-Audio-Preview’s high WER / CER on Gsm8kEval and APE-zh is attributed to LaTeX-formatted mathematical text despite coherent speech responses.Source: §4.2 · p. 8

4.3 Human Evaluation

  • Human judgments broadly align with automatic scores: pairwise preference consistency exceeds 70%, and QA matching rates average above 90%.Mismatches occurred for partially correct answers and less common general-knowledge questions; transcript-based scoring aligns with human judgments on datasets unrelated to paralinguistic information.Source: §4.3 · p. 8, Figure 4 · p. 8

5 Conclusion

URO-Bench is presented as a comprehensive benchmark for end-to-end spoken dialogue models, extending evaluation to multilingualism, multi-round dialogue, and paralinguistics. Experiments identify gaps in instruction-following, reasoning, multilingualism, and paralinguistic performance, while the released resources and leaderboard support community comparison.

5 Conclusion

  • URO-Bench is presented as the first S2S benchmark covering multilingualism, multiround dialogues, and paralinguistics.The authors report significant instruction-following and reasoning gaps relative to cascaded models, and difficulty with multilingual and paralinguistic tasks among most SDMs.Source: §5 · pp. 8–9
  • The authors open-sourced the test sets and evaluation code and launched a leaderboard for comparing SDM performance over time.Source: §5 · p. 9

Limitations

The benchmark has operational and evaluation limitations, including constraints on latency automation, scoring, and test-set scale.

Limitations

  • The benchmark lacks an automatic first-packet-latency evaluation pipeline because evaluation would require modifying source code.Source: Limitations · p. 9
  • LLM-based scores may retain bias and fluctuate despite reported alignment with human evaluations.Source: Limitations · p. 9
  • High API costs constrain singing and recitation test-set size when using Gemini 2.0 Flash and GPT-4o-Audio-Preview as evaluators.Source: Limitations · p. 9

A Detailed Introduction of Test Sets

The test sets span understanding, reasoning, and oral conversation, with tasks covering language comprehension, analytical abilities, and spoken interaction. The pro sets add demanding evaluations of emotion, audio, multilingualism, speaker recognition, and multi-round dialogue.

A Detailed Introduction of Test Sets

  • Understanding sets assess instruction following, verbatim repetition, summarization, and information extraction from English and Chinese listening questions.GaokaoEval and HSK5-zh use exam questions paired with synthesized instructions and original audio; samples longer than 30 seconds were excluded.Source: A Detailed Introduction of Test Sets · p. 11
  • UnderEmotion-en and UnderEmotion-zh test mood inference and empathetic responses using emotion-controlled speech, with real-world RAVDESS samples also included in UnderEmotion-en.Questions about speaker emotion are paired with RAVDESS audio.Source: A Detailed Introduction of Test Sets · p. 11
  • Other pro tasks test Chinese-English code-switching, privacy-question refusal, and comprehension of environmental sounds and music.ClothoEval targets ambient-sound comprehension, while MuChoEval tests musical knowledge.Source: A Detailed Introduction of Test Sets · p. 11
  • Reasoning sets cover mathematics, logic, common sense, factual questions, knowledge retrieval, story morals, contextual answer extraction, and practical mathematical problem-solving.MLC and MLC-zh were manually checked; samples longer than 30 seconds were excluded from StoralEval and SQuAD-zh.Source: A Detailed Introduction of Test Sets · p. 12

6. MtBenchEval-en: For the multi-round spo-

  • Multi-round sets assess context tracking, memory, coherence, and recognition and recall of speakers and their information.SpeakerAware uses specified voice timbres to simulate multi-speaker dialogues; MtBenchEval excludes conversations with excessive turns.Source: §6 · p. 12
  • Oral conversation sets include open-ended dialogue, multilingual responses, singing and recitation, and responses in specified tones or styles.The Multilingual set tests responses in Spanish, French, German, Italian, Russian, Japanese, and Korean.Source: §6 · pp. 12–13

B Detailed Experiment Results

The section identifies the evaluated spoken dialogue models and points to detailed task scores and a Chinese-proficiency capability chart.

B Detailed Experiment Results

B.1 GPT-4o-Audio-Preview

GPT-4o-Audio-Preview performs better than the strongest open-source SDM on several basic-track content-quality tasks and approaches the cascaded model there. It also shows strengths in audio understanding and multilingual capability, alongside limitations in emotion generation and singing.

B.1 GPT-4o-Audio-Preview

  • GPT-4o-Audio-Preview performs better than GLM-4-Voice in TruthfulEval, Gsm8kEval, and MLC, and approaches Whisper + GPT-4o in the basic track.In the pro track, it understands music and environmental sounds, and its Chinese proficiency is comparable to GLM-4-Voice.Source: §B.1 · p. 13
  • GPT-4o-Audio-Preview demonstrates strong multilingual capabilities, while existing open-source models support at most two languages.The paper identifies reasoning, audio understanding, and multilingual ability as remaining gaps between open-source models and proprietary systems.Source: §B.1 · p. 13
  • GPT-4o-Audio-Preview has shortcomings in emotion generation and refuses to sing in SRT tests.Its UTMOS speech-quality score is lower than SLAM-Omni and GLM-4-Voice, but its speech-text alignment is better.Source: §B.1 · p. 13

B.2 Complex Cascaded Systems

The section tests cascaded systems for general-audio comprehension and controlled expressive speech. The audio-focused cascade surpasses GLM-4-Voice on two datasets, while the style-controlled cascade generates more expressive speech to some extent but has a Chinese emotion-generation shortfall.

B.2 Complex Cascaded Systems

  • The audio cascade combines Whisper transcription and audio captions with a fixed task instruction to GLM-4-9B-Chat-HF for evaluation on ClothoEval and MuChoEval.The style-controlled cascade supplies the requested tone to CosyVoice2 as an instruction.Source: §B.2 · p. 13
  • The advanced cascaded baseline surpasses GLM-4-Voice on ClothoEval and MuChoEval.The comparison is presented as highlighting the value of incorporating general audio information.Source: §B.2 · p. 13, Table 8 · p. 13
  • The ASR + LLM + style-controlled TTS cascade generates more expressive speech to some extent, but lags behind GLM-4-Voice in GenEmotion-zh.LLM outputs representing laughter or sighs could not always be generated by the TTS model or match its required format.Source: §B.2 · p. 14, Table 9 · p. 14

B.3 Human Evaluation

Human evaluations show high agreement with automated scoring across open-ended questions, QA, and speech emotion generation. The appendix also points readers to examples of URO-Bench datasets in Figures 6 and 7.

B.3 Human Evaluation

  • Pairwise preference consistency exceeded 70% on open-ended English and Chinese tests.The results are presented in Tables 10 and 11.Source: §B.3 · p. 14, Table 10 · p. 14, Table 11 · p. 14
  • QA matching rates averaged above 90%, although some mismatches involved partially correct answers or less common general-knowledge questions.Most matching rates were close to 100%.Source: §B.3 · p. 14, Table 12 · p. 14
  • Human evaluations and rule-based scores for speech emotion generation had matching rates above 70%.The results are presented in Table 13.Source: §B.3 · p. 14, Table 13 · p. 14

C Examples of URO-Bench

D GPT Prompts for Data Construction

This appendix section presents GPT prompts and JSONL templates for constructing spoken-dialogue data, including QA pairs, dataset customization, poem recitation, and responses in specified emotions or styles. The prompts specify task content, conversational formats, and output requirements.

Prompts for Repeat and Summary Construction

  • The repeat and summary prompts request 20 conversational JSONL question-answer pairs for repeating spoken passages or summarizing a long passage’s main idea.The prompts prescribe the opening wording for each task.Source: D GPT Prompts for Data Construction · p. 14

Prompts for MLC Construction

  • Basic MLC prompts request 20 conversational JSONL question-answer pairs for simple arithmetic, logic, and general-knowledge questions in English and Chinese.The Chinese prompts specify everyday-knowledge questions.Source: D GPT Prompts for Data Construction · p. 19

Prompts for UnderEmotion Construction

  • UnderEmotion prompts test tone-based emotion detection and empathy using subtle, unstated emotions in natural English and Chinese dialogue designed for TTS and JSONL.The empathy variant asks the model to infer the speaker’s emotion from vocal delivery and respond empathetically.Source: D GPT Prompts for Data Construction · pp. 19–20

Prompts for CodeSwitching Construction

  • Code-switching prompts cover Chinese questions with English answers, English questions with Chinese answer portions, and Chinese questions requiring English responses in conversational, TTS-suitable JSONL.Each direction includes both questions and answers.Source: D GPT Prompts for Data Construction · pp. 20–21

Prompts for Safety Construction

  • Safety prompts generate bilingual JSONL dialogues about privacy involving people’s voices, requiring polite but firm refusals and standard rejection messages.The example asks the model to predict a speaker’s age from audio and refuse.Source: D GPT Prompts for Data Construction · p. 21

Prompts for MLCpro Construction

  • MLCpro prompts request challenging conversational JSONL questions and concise answers on advanced mathematics, science, and common sense, suitable for spoken TTS responses.The math topics include algebra, calculus, number theory, geometry, and combinatorics; prompts favor smooth language and limited mathematical symbols.Source: D GPT Prompts for Data Construction · pp. 21–22

Prompts for SpeakerAware Construction

  • SpeakerAware prompts create three-round dialogues between two speakers and a model to test speaker identification, identity tracking, and responses tied to each speaker’s earlier information.Speakers have distinct speech characteristics; a later “What do I ...” question requires recalling information without restating it.Source: D GPT Prompts for Data Construction · pp. 22–23

Prompts for SRT Construction

Poem-recitation prompts

  • Poem-recitation examples pair requests to recite named poems with their corresponding text, including English and Chinese examples.The Chinese prompt asks for TTS-suitable JSONL entries containing a question and recital text.Source: D GPT Prompts for Data Construction · p. 23

Prompts for GenEmotion Construction

  • GenEmotion prompts require TTS-suitable JSONL Q&A entries with a request, an emotion label, and matching answer text, using one of six specified emotions.The allowed labels are angry, disgusted, fearful, happy, sad, and surprised.Source: D GPT Prompts for Data Construction · p. 24

Prompts for GenStyle Construction

E Detailed Scoring Criteria

This section details URO-Bench’s test-specific scoring criteria, including transcription-based, audio-based, and specialized evaluation methods. It also specifies score-only rubrics for response qualities and performance, with scales for style adherence and multilingual language use.

Detailed Scoring Criteria

  • Most tests transcribe speech with Whisper-large-v3 (Radford et al., 2023) or paraformer-zh (Gao et al., 2022), then use GPT-4o mini to score accuracy, relevance, clarity, and completeness.SRT tasks instead use Gemini 2.0 Flash and GPT-4o-Audio-Preview to assess audio output directly.Source: E Detailed Scoring Criteria · p. 25
  • Repeat scores use 100 × (1 − WER) when WER ≤ 0.5 and zero above that threshold; Repeat-zh uses CER, while GenEmotion combines WER with emotion2vec recognition.A WER above 0.5 is treated as failure to follow instructions; emotion2vec (Ma et al., 2024) estimates whether generated speech contains the specified emotion.Source: E Detailed Scoring Criteria · p. 25
  • Task accomplishment scores are normalized to 100 points, using 10 rewritten GPT prompts based on VoiceBench evaluation prompts (Chen et al., 2024b).The appendix provides the detailed scoring criteria and prompts.Source: E Detailed Scoring Criteria · p. 25

Prompts for Evaluation in Open, Semi-open, and QA Modes

  • Open and semi-open prompts use 1–5 rubrics for response relevance, accuracy, and clarity, with semi-open scoring allowing reasonable answers that differ from references.QA mode instead asks for a single Yes or No judgment of correctness against the question and reference answer.Source: E Detailed Scoring Criteria · pp. 25–27

UnderEmotion and Safety Evaluation

  • UnderEmotion and Safety prompts score empathy toward the user’s expressed emotion and appropriate refusal or avoidance of privacy-related questions on 1–5 scales.The Safety rubric permits differing phrasing from the suggested answer if the model refuses or avoids the question.Source: E Detailed Scoring Criteria · pp. 27–28

Multi-round, SpeakerAware, SRT, and GenStyle Evaluation

  • Multi-round and SpeakerAware prompts assess context retention across turns, with SpeakerAware emphasizing identification of the same speaker in rounds one and three; SRT and GenStyle evaluate audio-task quality and specified-style matching.SRT rubrics consider audio clarity and task execution; the supplied GenStyle passage specifies style matching but does not include its full rubric.Source: E Detailed Scoring Criteria · pp. 28–30

E Detailed Scoring Criteria

  • Style scores 3–5 range from mostly matching the requested tone, language, and structure to a perfect match without deviations.Scores 3 and 4 allow minor inconsistencies or deviations; score 5 requires exact alignment and a coherent, appropriate response.Source: E Detailed Scoring Criteria · p. 31
  • Multilingual responses are rated 1–5 by how well the output uses the specified language to answer the user’s instruction.The rubric progresses from using an entirely different language (1) or partially complying (2), through mostly compliant use with occasional errors (3–4), to fluent, accurate use throughout (5).Source: E Detailed Scoring Criteria · p. 31
Loading 2502.17810v3…