Source-linked AI summary
VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models
Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, Irwin King
TL;DR
Existing QA benchmarks do not evaluate SLM knowledge understanding through end-to-end speech interactions or account for varied input audio conditions. VoxEval is a speech-based benchmark that also tests spoken mathematical reasoning; evaluations reveal low accuracy, sensitivity to audio variation, and difficulty with reasoning tasks.
Problem
Existing QA benchmarks fall short of evaluating SLM knowledge understanding end-to-end in speech and across varied input audio conditions.
Source: §1 · p. 1Method
VoxEval evaluates SLM knowledge understanding with speech questions and answers, varied input audio conditions, and spoken math problems.
Source: §1 · p. 2Results
Recent SLMs show low accuracies, often below random guessing, sensitivity to input audio variation, and difficulty with mathematical reasoning.
Source: §5 · p. 9Takeaways & Limitations
The results point to a need for further research on SLM knowledge understanding, robustness, and reasoning in real-world conversational settings.
Source: §5 · p. 9VoxEval excludes fairness, toxicity, and hallucination evaluation, and synthetic TTS speech may not capture natural speech variability.
Source: §6 · p. 9
Abstract
from arXiv · showhide
With the rising need for speech-based interaction models, end-to-end Spoken Language Models (SLMs) have emerged as a promising solution. While these models require comprehensive world knowledge for meaningful and reliable human interactions, existing question-answering (QA) benchmarks fall short in evaluating SLMs' knowledge understanding due to their inability to support end-to-end speech evaluation and account for varied input audio conditions. To address these limitations, we present VoxEval, a novel SpeechQA benchmark that assesses SLMs' knowledge understanding through pure speech interactions. Our benchmark 1) uniquely maintains speech format for both inputs and outputs, 2) evaluates model robustness across diverse input audio conditions, and 3) pioneers the assessment of complex tasks like mathematical reasoning in spoken format. Systematic evaluation demonstrates that VoxEval presents significant challenges to current SLMs, revealing their sensitivity to varying audio conditions and highlighting the need to enhance reasoning capabilities in future development. We hope this benchmark could guide the advancement of more sophisticated and reliable SLMs. VoxEval dataset is available at: https://github.com/dreamtheater123/VoxEval
1 Introduction
End-to-end Spoken Language Models need world knowledge for meaningful interactions, but existing benchmarks do not fully evaluate speech-to-speech knowledge QA across varied audio conditions. VoxEval addresses these gaps and evaluates spoken mathematical reasoning, finding current models struggle with the benchmark.
1 Introduction
- Existing benchmarks lack end-to-end speech evaluation and do not account for varied input audio conditions.Speech-to-text and text-response setups misalign with speech-to-speech interaction; real-world audio also varies in timbre, speaking style, and quality.Source: §1 · pp. 1–2
- VoxEval keeps both questions and answers in audio and tests robustness using varied input audio conditions.The same question is presented under different audio conditions.Source: §1 · p. 2
- VoxEval introduces spoken math problems and answers to assess SLMs’ mathematical reasoning in speech format.The benchmark uses Chain-of-Thought prompts to assess step-by-step problem solving.Source: §1 · p. 2
- VoxEval is challenging for current SLMs, which the authors report perform poorly on the benchmark.The evaluation covers several recent SLMs.Source: §1 · p. 2
2 Related Work
Related work distinguishes speech-understanding models from end-to-end speech interaction models, reviews speech evaluation benchmarks, and situates VoxEval among text-based knowledge benchmarks.
2.1 Speech Large Language Models
- SpeechLLMs can understand and/or generate speech; S2TLLMs use speech input but retain text output, whereas end-to-end SLMs hear and speak.End-to-end SLMs typically use a tokenizer, language model, and vocoder.Source: §2.1 · pp. 2–3
2.2 SpeechLLM Evaluation Benchmarks
- Existing SpeechLLM benchmarks assess language, paralinguistic modeling, or downstream audio understanding, while VoxEval emphasizes knowledge comprehension, mathematics, and varied scenarios.VoiceBench evaluates various audio conditions but uses textual output rather than end-to-end speech output.Source: §2.2 · p. 3
2.3 Knowledge Understanding of LLMs
- Text-based knowledge benchmarks span commonsense, science, school- and college-level subject knowledge, and human-level tasks.Examples include CommonsenseQA, OpenBookQA, MMLU, MMLU-Pro, and AGIEval.Source: §2.3 · p. 3
3 VoxEval
VoxEval builds a speech-based knowledge-understanding benchmark from MMLU and evaluates SLM robustness across varied audio inputs. It also introduces spoken math problems to assess math-solving abilities.
3.1 Data Construction
- VoxEval contains 13,938 unique SpeechQA pairs, built by converting MMLU questions into speech and excluding high school computer science because its code snippets are unsuitable for verbal evaluation.MMLU’s subjects span STEM, social sciences, and humanities, supporting broad knowledge-understanding evaluation.Source: §3.1 · pp. 3–4
3.2 Various Input Conditions
- Robustness is defined as maintaining consistent knowledge-understanding QA accuracy when audio characteristics vary but input semantic content remains unchanged.The benchmark varies speakers, speaking styles, and audio quality.Source: §3.2 · p. 4
3.2.1 Different Speakers
- Six OpenAI TTS voices—Alloy, Echo, Fable, Nova, Onyx, and Shimmer—provide variation in speaker properties such as gender and accent.Source: §3.2.1 · p. 4
3.2.2 Different Speaking Styles
- Linguistic variation covers fillers, mispronunciations, disfluencies, false starts and corrections, and language proficiency; only question text is altered, not answer choices.GPT-4o modifies question text before it is converted to speech; paralinguistic variations include pitch shift and speed change.Source: §3.2.2 · pp. 4–5
3.2.3 Different Audio Qualities
- Audio-quality conditions include randomly applied noise types and environmental effects such as room impulse response, filters, bitcrush, gain, clipping, and equalization.The listed noise types include Gaussian noise, colored noise, background music, and short interruption noise; augmentations use speech generated with the Alloy speaker.Source: §3.2.3 · p. 5
3.3 Handle Math Expressions and More
- VoxEval pioneers spoken math-solving evaluation, using few-shot examples and GPT-4o to convert written problems into spoken forms before TTS synthesis.The conversion handles Arabic numerals, units, operators, and brackets because current TTS systems struggle to verbalize mathematical expressions accurately.Source: §3.3 · p. 5
4 Experiments
Experiments evaluate five SLMs on VoxEval across audio conditions, knowledge domains, and mathematical reasoning tasks. Results show weak overall performance and sensitivity to input conditions, modality, and reasoning prompts.
4.2 Results
- Most evaluated SLMs fail to surpass the 25% random-guessing baseline on VoxEval; only GLM-4-Voice exceeds it.SpeechGPT’s performance is near 0%, which the authors attribute as likely due to direct speech-to-speech generation without chain-of-modality.Source: Table 5 · p. 6, §4.2 · pp. 6–7
4.1 Experimental Setups
- Five open-source SLMs are evaluated using speaker-specific MMLU few-shot examples, ASR transcription, and string-matched answer accuracy.Inputs cover speakers, speaking styles, and audio qualities; speech is truncated to 80 seconds for models with input constraints.Source: §4.1 · p. 6
4.2 Results
- SPIRIT-LM’s accuracy varies by more than 20% across speech/text input-output combinations, highlighting inconsistent behavior across configurations.SPIRIT-LM is the only evaluated model that supports both speech and text for input and output during inference.Source: Table 6 · p. 6, §4.2 · p. 7
- Across subject groups, different models lead in different areas: TWIST in others, Moshi in STEM, and GLM-4-Voice in social science.With alloy input, SPIRIT-LM and GLM-4-Voice show approximately 4% and 10% score differences, respectively, across groups.Source: §4.2 · p. 7, Table 7 · p. 7
- Speaking-style and audio-quality variations can reduce performance by up to 6%, while speaker-voice changes have little effect.Pitch shifts pose the greatest challenge for most models; environment acoustics and linguistic variations have minimal impact.Source: §4.2 · pp. 7–8, Figure 2 · p. 8
- Individual-subject performance gaps often exceed 10%, although overall differences are typically within 1–2%; GLM-4-Voice shows minimal relative variation.TWIST and Moshi tend to be less stable across audio conditions.Source: §4.2 · p. 8
- Models perform similarly across math difficulty levels, and GLM-4-Voice is the only model that consistently excels at elementary math.The authors interpret the pattern as suggesting that most SLMs are unlikely to possess basic math reasoning abilities.Source: Table 8 · p. 7, §4.2 · pp. 8–9
- Chain-of-Thought prompting reduces performance across tested models, while longer examples usually have little effect; TWIST struggles without audio truncation.The math evaluation uses speech-format questions and examples, with cut and uncut audio settings.Source: §4.2 · pp. 8–9
5 Conclusion
VoxEval evaluates end-to-end SLM knowledge understanding across diverse audio conditions. Evaluations find low accuracy, audio-condition sensitivity, and difficulty with mathematical reasoning.
5 Conclusion
- Recent SLMs often score below random guessing on VoxEval and show sensitivity to audio variation and difficulty with mathematical problem-solving.The authors conclude that further research is needed to improve SLM knowledge understanding, robustness, and reasoning capabilities in real-world conversational settings.Source: §5 · p. 9
6 Limitations
VoxEval’s scope and synthetic-speech construction constrain what it assesses. The paper also reports checks of its automated answer-extraction pipeline.
6 Limitations
- VoxEval assesses knowledge understanding, not fairness, toxicity, or hallucination, and synthetic TTS may not capture natural speech’s emotional, dialectal, and spontaneous variation.The authors say those excluded concerns require separate evaluation frameworks.Source: §6 · p. 9
A Accuracy of the Answer Extraction Pipeline
- The answer-extraction pipeline correctly extracted 29 of 30 responses; its single error followed ASR failure on a rapid, unclear SPIRIT-LM answer.The authors report low WER and CER across the three tested models and attribute the error to unclear model speech rather than a systematic pipeline limitation.Source: A Accuracy of the Answer Extraction… · pp. 11–12
B Comparison between Different Speech Interaction Models
The comparison evaluates GLM-4-Voice against cascaded open-source systems and a closed-source API model on 12 selected subjects. Cascaded models outperform SLMs, while the API model performs best.
B Comparison between Different Speech Interaction Models
- The comparison includes two Whisper-plus-TLM cascades and the closed-source gpt-4o-audio-preview API model, alongside GLM-4-Voice.GLM-4-Voice was selected as the SLM because it performed best in the main experiments.Source: B Comparison between Different Speech… · p. 12
- Cascaded models outperform GLM-4-Voice, and gpt-4o-audio-preview achieves the highest performance in this comparison.The comparison uses three subjects from each of four groups and the first 100 QA pairs per subject.Source: B Comparison between Different Speech… · p. 12
C Further Analysis of the In-context Learning Capabilities of SLMs with and without CoT
In-context examples improve GLM-4-Voice’s mathematical performance, while the benefit of Chain-of-Thought (CoT) prompting depends on problem complexity and can trade off with instruction-following.
C Further Analysis of the In-context Learning Capabilities of SLMs with and without CoT
- 1-shot examples outperform 0-shot baselines for both CoT and non-CoT mathematical reasoning.The experiments evaluate GLM-4-Voice across three mathematical domains.Source: C Further Analysis of the In-context… · p. 12
- CoT outperforms non-CoT in elementary mathematics, but its advantage diminishes in high school and college mathematics.The reported reasoning benefit does not extend to the more complex domains.Source: C Further Analysis of the In-context… · pp. 12–13
- On challenging problems, 1-shot CoT underperforms 1-shot non-CoT, suggesting a trade-off between reasoning and instruction-following.The authors suggest explicit reasoning may impair answer formatting and selection when problems exceed the model’s reasoning capacity.Source: C Further Analysis of the In-context… · p. 13
D Statistical Test Results across Different Input Audio Conditions
Statistical tests find that input audio conditions affect all evaluated SLMs except SpeechGPT, with sensitivity to speaking rate and pitch varying across models. The section also describes the comparisons and recording-condition transformations used.
D Statistical Test Results across Different Input Audio Conditions
- The analysis compares 56 subject accuracy scores across 11 audio conditions using Friedman tests followed by pairwise Wilcoxon signed-rank tests at a 5% Type I error rate.The pairwise tests use Bonferroni correction.Source: D Statistical Test Results across… · p. 13
- Input audio conditions significantly affect every evaluated SLM except SpeechGPT, whose accuracy is near zero across conditions.Pairwise tests with Bonferroni correction were conducted on the other four models; Figure 3 visualizes the comparisons.Source: D Statistical Test Results across… · p. 13, Figure 3 · p. 14
- GLM-4-Voice, Moshi, and SPIRIT-LM show significant performance differences under Speed and Pitch conditions.The reported pattern indicates sensitivity to speaking-rate and pitch variations, with effects varying by model.Source: D Statistical Test Results across… · p. 13, Figure 3 · p. 14
E Explanation for Various Speech Recording Conditions
- Table 14 describes input audio conditions categorized as other environment acoustics and their audio-augmentation transformation settings.Figure 3 presents Wilcoxon signed-rank test results for the four SLMs with significant Friedman-test differences.Source: E Explanation for Various Speech… · p. 13, Figure 3 · p. 14
F Additional Experimental Settings
The experiments use the models’ default hyperparameters and run each experiment once. The case study presents randomly selected Alloy-setting responses intended to represent each SLM’s characteristic answer style.
F Additional Experimental Settings
- SpeechGPT, TWIST, SPIRIT-LM, and Moshi have 7B parameters, while GLM-4-Voice has 9B parameters; SPIRIT-LM uses its base version.All models use default implementation hyperparameters, and each experiment is performed once.Source: F Additional Experimental Settings · p. 14
G Case Study for SLM Responses
- The case study randomly selects Alloy-setting responses to represent each SLM’s characteristic answer style.Table 15 pairs example responses with the authors’ comments.Source: G Case Study for SLM Responses · p. 14
H String Matching vs. LLM Extraction
The section contrasts string matching with LLM-based extraction of spoken answers, noting that extraction can change accuracy in edge cases. It also provides prompts for converting math questions and choices and extracting final answers.
H String Matching vs. LLM Extraction
- LLM-based answer extraction yields lower accuracy than string matching in most cases, primarily because of edge cases.When an SLM implies option D without explicitly identifying it as correct, string matching can count D as correct while the LLM returns “None!”Source: H String Matching vs. LLM Extraction · p. 14
- The prompt figures document GPT-4o conversion of math questions and answer choices, plus extraction of final answers from SLM outputs.The question, choice, and extraction prompts appear in Figures 4, 5, and 6, respectively.Source: Figure 4 · p. 17, Figure 5 · p. 18, Figure 6 · p. 19