PaperlayerView exact revision on arXiv

Source-linked AI summary

WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild

Linhao Zhang, Jian Zhang, Bokai Lei, Chuhan Wu, Aiwei Liu, Wei Jia, Xiao Zhou

arXiv:2506.21875v3cs.CL

TL;DR

End-to-end speech LLMs need benchmarks that assess generated speech and realistic spoken interactions rather than relying on adapted text-oriented evaluation. WildSpeech-Bench combines curated conversational queries, varied speakers and acoustic conditions, speech-specific challenges, and query-aware automatic evaluation. Experiments report that S2T dialogue performance does not translate to S2S, tested models are vulnerable to noise, and tailored evaluation aligns highly with human judgments.

  • Problem

    Existing benchmarks often adapt text-based or S2T evaluation and lack realistic queries and acoustic diversity, limiting comprehensive assessment of end-to-end speech interactions.

    Source: §1 · pp. 1–2

  • Method

    WildSpeech-Bench curates real-world conversational queries, varies speaker and acoustic conditions, adds speech-specific challenges, and uses automated evaluation with query-specific checklists.

    Source: §1 · p. 2, §2.2 · p. 5

  • Results

    Experiments find that S2T dialogue benchmark performance does not translate to S2S, all tested models are vulnerable to environmental noise and background speech, and query-specific evaluation aligns highly with human judgments.

    Source: §1 · p. 3

  • Takeaways & Limitations

    WildSpeech-Bench provides a specialized basis for assessing speech models across varied speech scenarios and query-specific criteria.

    Source: §1 · p. 3, §5 · p. 9

    The benchmark focuses on single-turn evaluation and prioritizes diagnostic depth over dataset volume.

    Source: §2.1 · p. 3, §2.3 · p. 6

Abstract

from arXiv · show

Recent multi-modal Large Language Models (LLMs) such as GPT-4o have demonstrated strong capabilities of direct speech interaction. However, the lack of specialized and comprehensive benchmarks for end-to-end speech LLM evaluation hinders optimizing the user experience of Audio LLMs in real-world applications. Existing evaluation methods often adapt text-based benchmarks, overlooking speech's unique characteristics and challenges, including prosody, homophones, stuttering, and differing user expectations. Here, we introduce the first comprehensive benchmark designed to systematically evaluate end-to-end speechLLMs in practical speech conversations. We systematically curate real-world chat data relevant to spoken scenarios, introduce diversity in speaker attributes and acoustic conditions, and augment the dataset with speech-specific phenomena. We further design a query-aware evaluation method to use customized evaluation checklists and prompts to enhance the accuracy of automatic evaluation. We conduct comprehensive testing and detailed analysis of various mainstream speech models, revealing significant differences in model performance across different speech scenarios. The use of query-aware evaluation further enables a finer-grained assessment under various speech-specific scenarios. Our benchmark can provide valuable insights for speech model development and evaluation.

1 INTRODUCTION

Existing benchmarks do not comprehensively assess end-to-end speech interactions because they often evaluate text rather than generated speech and use queries and audio that are unlike real-world speech. WildSpeech-Bench addresses these gaps with curated, acoustically varied queries and tailored evaluation; experiments find that S2T benchmark performance does not translate to S2S, tested models lack robustness to noise, and query-aware evaluation aligns well with human judgments.

1 INTRODUCTION

  • Existing benchmarks often evaluate text rather than generated speech and may favor verbose responses, despite spoken interactions prioritizing conciseness and efficiency.Some benchmarks assess models incapable of speech generation, placing them in a distinct research category.Source: §1 · pp. 1–2
  • Many existing datasets use text-derived or corpus queries and clean, homogeneous synthesized audio, limiting realistic assessment of user intent and acoustic robustness.Such query sources can yield declarative statements rather than authentic user queries; real-world applications require robustness to background noise.Source: §1 · p. 2
  • WildSpeech-Bench introduces 1,100 queries across five categories, curated from conversational data and varied by speaker attributes, acoustic conditions, and speech-specific phenomena.Challenges include meaning altered by prosody or pauses, near-homophones, and stuttering.Source: §1 · p. 2
  • The benchmark uses hand-crafted, query-specific evaluation prompts to assess speech clarity, conciseness, tone, content comprehensiveness, and accuracy.The prompts are tailored to challenging queries rather than relying only on generic rubrics.Source: §1 · p. 2
  • Strong performance on S2T dialogue benchmarks does not translate to S2S, all tested models show vulnerability to environmental noise and background speech, and query-specific evaluation aligns highly with human judgments.The reported noise vulnerability includes proprietary leaders such as GPT-4o.Source: §1 · p. 3

2 WILDSPEECH-BENCH

WildSpeech-Bench provides a single-turn benchmark built from curated real-world queries, diverse speech conditions, and automated evaluation. Its construction spans general conversational and paralinguistic queries, while its evaluation combines repeated ASR scoring, query-specific checklists, and acoustic-quality assessment.

2.1 DATASET CONSTRUCTION

  • WildSpeech-Bench focuses on single-turn evaluation, and query selection starts from real conversational data filtered for spoken interaction suitability.WildChat queries were filtered to English, non-toxic, under 50 words, and single-turn; 34,736 remained before further curation.Source: §2.1 · p. 3, §2.1.1 · p. 4

2.1.1 QUERY SELECTION

  • Query selection combines model-based commonality and difficulty screening with manual review, categorization, balancing, and human verification, yielding 1,000 general queries.The curation process removes redundant, multilingual, ambiguous, or unsuitable queries and balances four categories.Source: §2.1.1 · p. 4
  • The 100 Paralinguistic-Featured queries target pauses, stresses, tones, stutters, and near-homophones, using generated and manually sourced or crafted examples.Task-specific prompts were used with GPT-4o to generate queries.Source: §2.1.1 · p. 4

2.1.2 SPEECH GENERATION

  • Speech generation uses 16 voice profiles across genders and four age groups, human recordings for nuanced paralinguistic queries, and controlled noise augmentation.Noise includes background speech and environmental or incidental sounds; samples are manually checked for audibility.Source: §2.1.2 · pp. 4–5

2.2 PERFORMANCE EVALUATION

  • Automatic evaluation transcribes each response with Whisper-large-v3, scores three ASR runs with GPT-4o mini, and averages them while limiting response-length bias.Customized checklists assess open-ended outputs, and UTMOS evaluates acoustic quality.Source: §2.2 · pp. 5–6

2.3 DATA STATISTICS

  • The final benchmark contains 1,100 queries: 1,000 across four general interaction categories and 100 Paralinguistic-Featured queries, balanced across categories and themes.The design emphasizes diagnostic depth over sheer volume.Source: §2.3 · p. 6

3 EXPERIMENTAL RESULTS

WildSpeech-Bench results distinguish model performance across conversational and paralinguistic tasks, while noise tests, acoustic-quality assessments, and case studies expose additional differences and challenges. The experiments also compare automatic evaluation with human judgments and test components of the evaluation framework.

3.1 SETTINGS

  • The evaluated model set includes GLM-4-Voice, MiniCPM, Qwen-2.5-omni, Kimi-Audio, and GPT-4o-Audio, alongside a Whisper–Qwen2.5 7B–MeloTTS pipeline baseline.The baseline is expected to struggle on some paralinguistic-featured tasks because ASR can lose crucial paralinguistic information.Source: §3.1 · p. 6

3.2 MAIN RESULTS

  • GPT-4o-Audio outperforms other models in all categories; Qwen-2.5-omni leads open-source models overall but trails GLM-4-Voice and MiniCPM on paralinguistic queries.Table 2 covers Text Creation, Information Inquiry, Solution Request, Opinion Exchange, and Paralinguistic-Featured queries, including five PF subcategories.Source: §3.2 · pp. 6–7, Table 2 · p. 7
  • Kimi-Audio's strong speech-to-text benchmark performance does not transfer to speech output, while the naive pipeline performs comparably to GLM-4-Voice and MiniCPM except on stress and tone.The paper hypothesizes that multimodal generation's audio-text embedding fusion may contribute to Kimi-Audio's performance degradation.Source: §3.2 · p. 7
  • WildSpeech correlates more strongly with human scores than the VoiceBench baseline, and removing query-aware evaluation or a second ASR-evaluation round reduces correlation.A selected-subset comparison also found query-aware evaluation significantly contributed to evaluation accuracy and reliability.Source: Figure 4 · p. 7, §3.2 · p. 8

3.4 NOISE IMPACT

  • Both natural and human-voice noise significantly reduce existing models' performance, with human-voice noise having a particularly pronounced impact.Strong clean-speech performance does not necessarily transfer to noisy scenarios.Source: §3.4 · p. 8, Figure 5 · p. 8

3.5 AUDIO QUALITY

  • GPT-4o and Qwen2.5-mini achieve the best generated-speech acoustic quality, with less variation across task categories than content quality shows.The paper attributes the smaller category variation to acoustic characteristics not varying substantially across categories.Source: §3.5 · p. 8, Figure 6 · p. 8

3.6 CASE STUDY

  • Two cases illustrate semantic misinterpretation of an album title and difficulty distinguishing meanings conveyed by prosodic pauses despite identical sentence text.The paper identifies both semantic misinterpretation and nuanced paralinguistic interpretation as challenges for current models.Source: §3.6 · pp. 8–9, Figure 7 · p. 9

4 RELATED WORK

Prior speech benchmarks are described as limited in their support for realistic, end-to-end speech interaction and evaluation of expressive speech generation. The related work discusses text-derived query sets, text-based scoring, and benchmarks that add acoustic diversity without evaluating full speech-to-speech interaction.

4 RELATED WORK

  • Existing benchmarks remain underdeveloped for evaluating end-to-end SpeechLLMs, despite the shift from traditional pipelines toward real-time speech interaction.The paper frames these benchmark issues as limiting field progress.Source: §4 · p. 9
  • Web Questions- and TriviaQA-derived sets use exact character matching or GPT-based scoring, while AIR-Bench and SD-Eval explore acoustic diversity but not full speech-to-speech interaction or expressive generation.VoiceBench also adapts text benchmarks through TTS and evaluates text output, while using queries whose distribution differs from speech-model queries.Source: §4 · p. 9

5 CONCLUSION

The conclusion presents WildSpeech-Bench as a benchmark built from realistic, diverse speech-interaction data and speech-specific challenges. It also reports customized evaluation criteria for more nuanced and reliable model assessment, and the paper discloses limited LLM use for manuscript polishing.

5 CONCLUSION

  • WildSpeech-Bench combines realistic and diverse data, controlled speaker and acoustic variations, and speech-specific challenges to reflect real-world speech interactions.The authors also report that customized evaluation criteria enable more nuanced and reliable assessment of model performance.Source: §5 · pp. 9–10

A LARGE LANGUAGE MODEL (LLM) USAGE STATEMENT

  • The authors used an LLM after the initial draft to improve grammar and polish, not to generate research ideas, experimental designs, or scientific content.They state that scientific contributions, analyses, and conclusions are theirs.Source: A LARGE LANGUAGE MODEL (LLM) USAGE… · p. 14

B HUMAN EVALUATION DETAILS

Human evaluation used four experienced annotators for initial scoring and two quality-control specialists to review annotations, with discussion resolving ambiguous or complex cases.

B HUMAN EVALUATION DETAILS

  • Four annotators scored responses before two quality-control specialists reviewed all annotations for consistency and accuracy.Ambiguous or complex cases were resolved through discussion between annotators and the QC group; all used the same guidelines as automatic evaluation.Source: B HUMAN EVALUATION DETAILS · p. 14

C BENCHMARK QUERY COMPARISON

WildSpeech-Bench uses authentic user queries and varied non-linguistic noises to represent conversational goals and challenging acoustic conditions.

C BENCHMARK QUERY COMPARISON

  • WildSpeech-Bench uses real user queries curated from WildChat, unlike SD-Eval’s dataset-derived queries and AIR-Bench’s synthetic text-based queries.The authors say this better reflects authentic conversational goals and natural linguistic diversity.Source: C BENCHMARK QUERY COMPARISON · p. 14

D NATURAL NOISE DETAILS

  • Five noise categories—animals, natural soundscapes, human non-speech, interior/domestic, and exterior/urban—are sampled to reflect their commonality in daily life.The benchmark adds these non-linguistic noises to audio queries for analysis under varied acoustic conditions.Source: D NATURAL NOISE DETAILS · p. 14

E ASR FOR MCQ RESULTS

The section identifies pitfalls in automated speech evaluation and reports a dual-modal performance drop for Kimi-Audio. It describes a response-evaluation prompt and scoring scale.

E ASR FOR MCQ RESULTS

  • ASR transcription errors can mark clear, correct spoken MCQ answers wrong, conflating speech-model quality with evaluator errors.Table 5 illustrates this with the answers “A. microorganisms” and “C. a town.”Source: E ASR FOR MCQ RESULTS · pp. 14–15, Table 5 · p. 16

F EVALUATION PROMPT

  • The evaluation prompt asks judges to analyze user intent, assess response strengths and weaknesses, and assign a score from 1 to 10.It directs evaluation from the perspective of voice communication, using a checklist as reference rather than a standard answer.Source: F EVALUATION PROMPT · pp. 16–17

G PERFORMANCE DEGRADATION IN DUAL-MODAL GENERATION

  • 3.97 in text-only mode fell to 3.00 for Kimi-Audio’s textual component in dual-modal generation on VoiceBench CommonEval.The authors report that the drop affects text generation as well as speech synthesis.Source: G PERFORMANCE DEGRADATION IN… · p. 17

H EVALUATION CHECKLIST

The query-aware checklist structures evaluation around criteria tailored to each request, aiming to reduce ambiguity and improve score consistency.

H EVALUATION CHECKLIST

  • The checklist targets factual accuracy, expected answer structure, and critical nuances such as punctuation-driven semantic shifts.Its criteria are tailored to each query rather than serving as a single holistic judgment.Source: H EVALUATION CHECKLIST · pp. 17–18
  • The authors report that query-aware criteria reduce ambiguity and make scores a more accurate, consistent, and fine-grained measure of model capabilities.Table 7 is identified as a query evaluation checklist.Source: H EVALUATION CHECKLIST · p. 18

I LIMITATIONS AND FUTURE WORK

The authors flag a potential long-term risk that benchmark optimization may not track real-world conversational performance.

I LIMITATIONS AND FUTURE WORK

  • Standardized checklist criteria could incentivize developers to optimize for the benchmark rather than general conversational ability.The authors describe this as a potential “teaching to the test” risk.Source: I LIMITATIONS AND FUTURE WORK · p. 18
  • WildSpeech-Bench scores might rise without a proportional improvement in real-world performance.This is presented as a potential consequence, not a demonstrated outcome.Source: I LIMITATIONS AND FUTURE WORK · p. 18
  • Addressing this potential negative externality is identified as an area for future benchmarking research.The stated concern follows from incentives created by widespread adoption of standardized benchmarks.Source: I LIMITATIONS AND FUTURE WORK · p. 18

J.1 FOCUS ON SINGLE-TURN EVALUATION

WildSpeech-Bench establishes a single-turn evaluation foundation because unresolved problems in evaluating speech quality, acoustic robustness, and query-aware relevance would confound multi-turn results. Its 1,100 manually reviewed queries and custom checklists prioritize diagnostic depth over dataset size.

J.1 FOCUS ON SINGLE-TURN EVALUATION

  • WildSpeech-Bench focuses on single-turn evaluation as a foundation for future, more robust multi-turn benchmarks.The benchmark’s design rationale is that meaningful multi-turn evaluation cannot be built on flawed single-turn foundations.Source: §J.1 · p. 20
  • Single-turn evaluation addresses unresolved challenges in direct speech-quality assessment, acoustic-noise robustness, and query-aware relevance before extending evaluation to multi-turn settings.The authors argue that building multi-turn benchmarks before resolving these issues would compound evaluation problems and produce unreliable, confounded results.Source: §J.1 · p. 20

J.2 DATASET SCALE

  • 1,100 manually reviewed queries are paired with custom checklists for query-aware evaluation, prioritizing diagnostic depth over sheer volume.The authors describe this as the first speech benchmark, to their knowledge, with query-aware evaluation; the approach increases annotation and verification workload.Source: §J.2 · p. 20
Loading 2506.21875v3…