Source-linked AI summary
VoiceBench: Benchmarking LLM-Based Voice Assistants
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, Haizhou Li
TL;DR
LLM-based voice assistants lack a benchmark that evaluates speech interaction beyond ASR and clean spoken question answering. VoiceBench addresses this gap with real and synthetic spoken instructions covering speaker, environmental, and content variation, and finds substantial weaknesses in current systems and evaluation protocols.
Problem
Existing evaluations emphasize ASR or general knowledge with clean speech, lacking comprehensive assessment of voice assistants under realistic speaker, environmental, and content variations.
Method
VoiceBench is a comprehensive benchmark using real and synthetic spoken instructions to evaluate knowledge, instruction following, safety, and diverse real-world variations.
Results
VoiceBench evaluations reveal limitations in current protocols and a significant overall and robustness performance gap between end-to-end assistants and ASR-plus-LLM pipeline models.
Takeaways & Limitations
VoiceBench identifies vulnerabilities across speaker, environmental, and content factors and highlights areas for improving spoken-instruction processing and voice-assistant robustness.
Takeaways & Limitations
Future work must extend evaluation to speech-based responses and more diverse, realistic evaluation data.
Abstract
from arXiv · showhide
Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering a significantly improved user experience compared to traditional text-based interactions. However, the absence of benchmarks designed to evaluate these speech interaction capabilities has hindered progress of LLM-based voice assistants development. Current evaluations focus primarily on automatic speech recognition (ASR) or general knowledge evaluation with clean speeches, neglecting the more intricate, real-world scenarios that involve diverse speaker characteristics, environmental and content factors. To address this, we introduce VoiceBench, the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants. VoiceBench also includes both real and synthetic spoken instructions that incorporate the above three key real-world variations. Extensive experiments reveal the limitations of current LLM-based voice assistant models and offer valuable insights for future research and development in this field.
1 Introduction
VoiceBench addresses the lack of holistic evaluation for LLM-based voice assistants by benchmarking capabilities across real-world speaker, environmental, and content variations. Its evaluation reveals weaknesses in current protocols and a performance gap between end-to-end and pipeline systems.
- Motivation: Existing evaluations mainly emphasize ASR or clean spoken question answering, leaving system capabilities insufficiently assessed in realistic conditions.The introduction identifies this narrow scope as a limitation of current evaluation practice.
- Benchmark design: The benchmark uses both real and synthetic spoken instructions to assess general knowledge, instruction following, and safety.Its synthetic data covers diverse speaker properties, environmental effects, and spoken-language content variations.
- Findings: Extensive experiments identify weaknesses in current evaluation protocols and a substantial performance and robustness gap between end-to-end assistants and ASR-plus-LLM pipelines.The reported gap motivates further advancement of end-to-end voice assistants.
- Novel benchmark: VoiceBench is presented as the first comprehensive benchmark for multi-faceted evaluation of LLM-based voice assistants.It evaluates general knowledge, instruction-following, and safety capabilities.
- Real-world scenarios: VoiceBench tests speaker, environmental, and content variations that are underrepresented in existing evaluations.Examples include age, accent, pitch, signal distortion, echo, far-field conditions, grammar errors, mispronunciations, and disfluencies.
2 Background
Voice assistants differ from audio-analysis systems because they must directly respond to spoken questions, motivating evaluation beyond established audio benchmarks. VoiceBench extends text-LLM evaluation priorities to voice assistants by measuring knowledge, instruction following, safety, and real-world challenges.
- Pipeline and end-to-end models: Pipeline voice assistants transcribe speech with ASR before passing the resulting text to an LLM for response generation.The background notes that transcription can lose information and reduce efficiency.
- Application setting: Voice assistant applications require direct responses to spoken questions, unlike audio-analysis systems that answer text instructions about audio contexts.This distinction separates voice-assistant evaluation from several established audio-analysis benchmarks.
- Evaluation gap: Existing voice-assistant evaluations focus on ASR or general knowledge with clean speech and therefore do not comprehensively assess model capabilities.The background characterizes these evaluations as limited in scope.
- VoiceBench scope: VoiceBench evaluates general knowledge, instruction-following ability, and safety while incorporating real-world challenges faced by spoken interactions.This design is motivated by the broader criteria commonly used for text-based LLM evaluation.
3 VoiceBench
VoiceBench evaluates voice assistants across general capabilities and robustness to speaker, environmental, and content variations. It combines spoken instruction datasets, multiple assistant architectures, and text-versus-speech comparisons to expose substantial weaknesses in current systems.
- Benchmark scope: VoiceBench evaluates general knowledge, instruction following, and safety alongside robustness across speaker, environmental, and content variations.Its framework has separate capability and robustness components.
- Dataset construction: VoiceBench covers open-ended, reference-based, and multiple-choice knowledge questions, short speech-suitable instruction-following tasks, and harmful prompts requiring refusal.Knowledge datasets include AlpacaEval, CommonEval, OpenBookQA, and MMSU; IFEval supplies instruction following and AdvBench supplies safety evaluation.
- Dataset construction: The benchmark combines real and synthetic spoken instructions, including real CommonEval and SD-QA speech and synthetic AlpacaEval, IFEval, and AdvBench speech.Instructions are kept relatively short for voice interaction, while other experiments use the helpful_base and vicuna AlpacaEval subsets.
- Experiment setup: The evaluation compares open-source end-to-end assistants, two ASR-plus-LLM pipelines, and GPT-4o-Audio using response-content scores rather than speech-output or transcription quality.AlpacaEval and CommonEval use GPT scores from 1 to 5, while SD-QA uses human-labeled references.
- Results: Naive pipelines outperform all open-source end-to-end models by more than 20 points on spoken instructions, while GPT-4o-Audio nearly matches its pipeline counterpart.The result indicates that strong ASR-plus-LLM systems remain substantially ahead of current open-source end-to-end assistants.
- Results: End-to-end assistants show a speech-processing gap exceeding 35 points in VITA, and several models, including Mini-Omni, answer malicious speech prompts instead of refusing.The speech gap is especially pronounced in multiple-choice QA, where most end-to-end models approach random-guessing performance.
4 Speaker Variations
VoiceBench evaluates speaker variation across speaking speed, age, volume, pitch, and accent using controlled synthetic and real speech. Performance is generally stable across moderate variations but degrades for extreme speeds and lower-resource accents.
- Method: The evaluation varies speaking speed, speaker age, volume, pitch, and accent using controlled TTS, speech prompts, and real or synthetic accent samples.Accent evaluation includes real SD-QA speech and synthetic AlpacaEval data.
- Results: Speaker age, volume, and pitch usually have limited impact, except at exceptionally low volume and for VITA processing child or high-pitched speech.Most assistants maintain consistent performance across wide variation ranges.
- Results: End-to-end models degrade substantially below 0.5x or above 1.5x speaking speed, while the naive pipeline remains stable from 0.25x to 2.0x.Speaking speed has a relatively greater influence on end-to-end models than the other tested speaker features.
- Results: Accent variation affects low-resource Indian English and Philippines accents more than high-resource Australian, UK, and US accents.The degradation is more pronounced for real accent data than synthetic data.
- Results: Mini-Omini performs worst and Native best under accent variation, while VITA shows larger cross-accent differences than other assistants.The authors hypothesize that speech-encoder choice influences robustness and generalization.
5 Environmental Variations
The benchmark tests voice assistants under noisy real-world conditions, including far-field speech, distortion, reverberation, packet loss, babble, and white noise. The naive pipeline is more resilient than end-to-end models, potentially because transcription provides implicit denoising.
- Motivation: The evaluation targets noisy scenarios common in homes, vehicles, and public spaces, addressing limited specificity in existing evaluations.Tested conditions include far-field speech, signal distortion, reverberation, packet loss, and noise interference.
- Method: Far-field speech is simulated with low-pass filtering, distortion with clipping, reverberation by varying RT60, and background noise with white and babble noise.These transformations model weakened high frequencies, amplitude clipping, reflected speech, and steady or dynamic interference.
- Results: All models with Whisper speech encoders show similar resilience across environmental conditions, but the naive pipeline outperforms the others.VITA shows significantly lower performance in the reported environmental evaluation.
- Results: End-to-end models are more susceptible to noise because they process audio directly, whereas the naive cascade transcribes speech before passing text to the LLM.The intermediate transcription may provide an implicit denoising step that improves input clarity.
6 Content Variations
VoiceBench examines natural spoken-content errors that clean evaluations often overlook, including disfluencies, mispronunciations, and grammatical errors. Pipeline models are most robust, with assistants tolerating grammar errors better than mispronunciations and high transcription error rates.
- Method: The content-variation analysis covers mispronunciations, grammatical errors, and disfluencies such as repairs and repetitions.These errors are presented as common features of natural speech that can affect assistant performance.
- Method: Table 5 reports response scores and performance-degradation percentages after injecting content noise.Table 6 provides examples of different content variations.
- Results: Naive pipeline models show the best robustness when handling content errors.This result is reported across the evaluated content-error conditions.
- Results: Assistants tolerate grammatical errors strongly but are much more vulnerable to mispronunciations.Mispronunciations often produce incorrectly recognized words and higher WER, which can alter intended meaning.
- Results: LLMs also tolerate grammatical errors better than high WER, because grammatical mistakes are common in written text whereas incorrect word recognition is not.The passage offers this contrast as a likely explanation for the differing resilience.
7 Conclusion
The paper introduces VoiceBench as a comprehensive benchmark using real and synthetic spoken instructions to assess voice-assistant capabilities. Its findings expose performance gaps and robustness vulnerabilities, while future work targets speech responses and more diverse evaluation data.
- Conclusion: VoiceBench is presented as the first comprehensive multi-facet benchmark using real and synthetic spoken instructions.The benchmark assesses voice-assistant capabilities across speaker, environmental, and content-related factors.
- Conclusion: The results show a significant performance gap between end-to-end models and straightforward pipeline models in processing spoken instructions.The conclusion frames this gap as motivating further advancement in spoken-instruction processing.
- Conclusion: Future work includes protocols for speech-based responses and more diverse, realistic evaluation data.These directions extend the benchmark beyond its current evaluation scope.
A Generation Prompts
The generation prompt normalizes raw text into a form suitable for a Text-to-Speech system. It specifies the transformations to apply and requires outputting only the normalized instruction.
- The prompt instructs the system to normalize raw text for a Text-to-Speech (TTS) engine.The input is provided through an instruction placeholder.
- Normalization expands abbreviations, acronyms, contractions, numbers, dates, times, and measurement units into spoken forms.
- The prompt also converts special characters, including symbols and newline or hyphen characters, into words.
- The system must output only the normalized instruction.
B Evaluation Prompts
The evaluation prompts assess model responses against questions, reference answers, or transcribed speech interactions. They use binary correctness judgments or a 1-to-5 quality scale and require score-only outputs.
- The automatic evaluation prompt presents a question, reference answer, and candidate answer for correctness assessment.
- The binary evaluator instructs the model to output only “Yes” or “No” when judging whether a candidate answer is correct.
- AlpacaEval & CommonEval: For speech interaction evaluation, models receive transcribed user instructions and response transcriptions, which are rated on a 1-to-5 scale.
- AlpacaEval & CommonEval: A score of 1 denotes largely irrelevant, incorrect, or nonresponsive output, while 2 denotes partial relevance with limited accuracy or completeness.
C Evaluation results with different TTS models
Voice assistants perform best on synthetic speech generated by Google TTS, and their rankings remain consistent across different synthetic datasets. These findings support using Google TTS synthetic data for benchmarking voice assistants.
- Voice assistants demonstrate their best overall performance on data produced by Google TTS.The passage attributes this outcome to the superior quality of Google’s text-to-speech system.
- Model rankings remain consistent across different synthetic datasets, supporting the reliability and validity of Google TTS synthetic data for benchmarking.
- Table 7 reports AlpacaEval performance generated with different TTS systems and distinguishes male and female voices.M denotes a male voice and F denotes a female voice.