Source-linked AI summary
AudioBench: A Universal Benchmark for Audio Large Language Models
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, Nancy F. Chen
TL;DR
AudioLLM evaluation lacks systematic, audio-specific coverage and comparable datasets across instruction-following tasks. AudioBench addresses this gap with a broad benchmark spanning speech, environmental sounds, and paralinguistic understanding; across five evaluated models, no single system consistently outperformed the others.
Problem
AudioLLM evaluations use substantially different datasets and limited task scopes, while existing multimodal benchmarks cannot directly measure audio understanding with audio context.
Method
AudioBench evaluates instruction-following AudioLLMs across 8 tasks and 26 datasets, covering speech understanding, audio scene understanding, and voice understanding.
Results
No single model consistently outperforms the others across all tasks and evaluation criteria.
Takeaways & Limitations
The benchmark identifies substantial opportunities for improving AudioLLM capabilities and supports analysis of data selection, metric design, and evaluation pipelines.
Takeaways & Limitations
AudioBench exclusively includes English datasets, leaving multilingual and code-switching capabilities for future expansion.
Abstract
from arXiv · showhide
We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments.
1 Introduction
AudioBench addresses fragmented and incomplete evaluation of instruction-following AudioLLMs with a broad benchmark spanning tasks, datasets, prompts, audio lengths, and evaluation methods. Results across five models show that no single system performs consistently well across all criteria.
- 1 Introduction: Existing AudioLLM evaluations use substantially different datasets and limited task scopes, making comparisons less justified.For example, Qwen-Audio-Chat and SALMONN share only two evaluation datasets.
- 1 Introduction: AudioBench covers 8 tasks and 26 datasets, including 7 newly adapted or collected datasets addressing gaps in existing resources.
- 1 Introduction: The benchmark varies prompt templates and audio lengths from seconds to minutes to test instruction compatibility, robustness, and longer-sequence performance.The test suite contains more than 400 hours of audio across over 100,000 samples.
- 1 Introduction: No single model consistently outperforms the others across all tasks and datasets.The benchmark evaluates five models, including both end-to-end and cascaded systems.
2 Related Work
Prior multimodal benchmarks richly evaluate text and vision, but they do not provide a comprehensive audio-centered assessment. AudioBench responds to this gap by targeting speech, environmental sound, and paralinguistic understanding.
- 2 Related Work: AudioLLM evaluation lacks a comprehensive benchmark that systematically compares audio interpretation across different tasks.
- 2 Related Work: Text-LLM benchmarks cover reasoning, knowledge, safety, and multilingual capability, while Vision-LLM benchmarks cover multidisciplinary understanding, perception, mathematics, and video.
- 2 Related Work: Video and vision benchmarks cannot directly evaluate audio understanding because they predominantly rely on visual inputs, with audio serving only as a supplementary feature.
- 2 Related Work: Existing AudioLLMs typically focus on subsets of tasks, including speech recognition or non-speech audio, rather than broad audio understanding.
3 AudioLLM Benchmark (AudioBench)
AudioBench evaluates instruction-following AudioLLMs across speech, environmental audio, and paralinguistic understanding using varied datasets, tasks, instructions, and metrics.
- Core Concept: AudioBench covers speech understanding, audio scene understanding, and voice understanding through compiled existing datasets and newly developed datasets.The benchmark is organized around semantic speech content, non-human environmental sounds, and human-related paralinguistic information such as emotion, accent, and gender.
- Evaluation Setup: AudioBench evaluates open-ended AudioLLM outputs mainly with model-as-judge scoring, while ASR uses WER and audio captioning additionally uses METEOR.The model-as-judge scores are rescaled to a 100-point scale for comparison.
- Speech Understanding: ASR evaluates speech-to-text conversion across nine datasets, including three long-form datasets, while current models struggle with audio exceeding ten minutes.Long recordings are segmented and reassembled when models do not support the original length.
- Audio Scene and Voice Understanding: AudioBench includes environmental audio question answering and paralinguistic tasks such as emotion, accent, and gender recognition.The benchmark uses refined and newly developed audio question-answering datasets alongside voice-understanding evaluations.
4 Results and Analysis
AudioBench evaluates five AudioLLMs across diverse tasks, revealing complementary strengths rather than a universally strongest model. Results also show sensitivity to prompt wording, difficulties with long-form ASR, and differences between direct AudioLLMs and cascaded systems.
- 4.1 Main Results: No single model consistently outperforms the others across all 26 datasets and tasks.The benchmark compares four AudioLLMs and the Whisper+Llama3 cascade.
- 4.1 Main Results: All AudioLLMs struggle with long-form ASR, while Qwen1/2-Audio and WavLLM show robust ASR capabilities overall.The authors relate long-form difficulty to limited training audio lengths and smaller pretraining datasets than task-specific systems such as Whisper.
- 4.1 Main Results: Whisper+Llama3 performs best on speech-intensive SQA and SI tasks, where transcription and language reasoning complement each other.The cascade relies on Whisper for recognition and Llama3 for reasoning over verbal content.
- 4.1 Main Results: AudioLLMs generally outperform cascaded models on paralinguistic and non-verbal-sound tasks, although their results remain imperfect.Cascade models particularly struggle with non-verbal content.
- 4.1 Main Results: WavLLM excels on SQA but underperforms in audio scene and vocal understanding because its training lacks non-spoken scenarios.The result suggests that instruction-following does not readily generalize without aligned training samples.
- 4.2 Robustness Queries: SALMONN is highly prompt-sensitive, whereas Qwen-Audio remains stable across three ASR prompt templates.SALMONN can match Qwen-Audio with suitable prompts but may perform phoneme recognition or speech translation under another template, harming WER.
- 4.3 Model-as-Judge Comparison: Llama-3-70B-Instruct correlates most strongly with GPT-4-as-a-judge, exceeding 0.85 across all three evaluation datasets.The authors therefore use it as their open-source judging model, while noting that model-as-judge grading remains unresolved.
5 Conclusion
AudioBench is presented as a comprehensive benchmark for evaluating AudioLLMs across diverse audio-understanding capabilities. The authors position its benchmark design and analysis as a basis for future progress in multimodal foundation models.
- 5 Conclusion: AudioBench provides a comprehensive AudioLLM benchmark spanning 8 tasks and 26 datasets.The paper presents the benchmark as its central contribution and discusses data selection, metric design, and evaluation procedures.
- 5 Conclusion: The authors anticipate that AudioBench will stimulate further progress in multimodal foundation models.
Limitations
AudioBench has scope and evaluation limitations involving language coverage, free-style generation metrics, and deployment efficiency.
- AudioBench currently includes only English datasets, leaving multilingual and code-switching speech evaluation for future iterations.The authors identify multilingual capabilities and code-switching as important extensions.
- Free-style generation remains difficult to evaluate because traditional metrics perform poorly in zero-shot scenarios and response correctness is complex.The paper calls for robust metrics or judge models tailored to audio inputs.
- AudioLLMs’ large model sizes lead to longer inference times, while AudioBench primarily evaluates accuracy rather than efficiency or deployment conditions.Future evaluations should consider inference speed and deployment environments.
A.1 Discussion on Dataset
The dataset discussion describes newly curated resources spanning listening comprehension, dialogue understanding, public speeches, spoken instructions, and caption-based audio question answering.
- CN-College-Listen: CN-College-Listen converts English college-exam listening questions into open-ended audio question-answer items and combines 271 collected triples with 2,000 additional questions.The dataset covers monologues and dialogues and requires precise capture of audio content.
- DREAM-TTS: DREAM-TTS converts human-annotated text dialogue questions and answers into spoken dialogue using text-to-speech while retaining the original question-answer structure.Gender information is preserved to better mimic real-life scenarios.
- Public-SG-SpeechQA: Public-SG-SpeechQA uses four Singapore public speeches, segmented by topic transitions, with generated question-answer pairs reviewed by humans.The speeches range from 20 minutes to one hour.
- OpenHermes-Audio and ALPACA-Audio: Text-based instruction data were converted into spoken instructions with text-to-speech, then human reviewers filtered approximately 90% of samples and retained 10%.Review focused on readability, naturalness, and reference-answer correctness.
- WavCaps QA and AudioCaps QA: WavCaps QA and AudioCaps QA generate question-answer pairs from captions and use human annotation to refine questions and revise answers.The process produced over 300 questions for each dataset.
- The section also describes scoring templates for model-as-judge evaluation, including 0–5 or 1–5 ranges and rescaling to 100 points.Binary tasks use 0 for incorrect and 1 for correct.
A.3 Comparison with AIR-Bench
AudioBench and AIR-Bench both evaluate audio-based instruction following, but differ in dataset coverage and evaluation preferences.
- AudioBench adds new datasets alongside ASR, speech question-answering, and speech instruction datasets to address gaps in existing evaluation coverage.The passage contrasts this coverage with AIR-Bench’s inclusion of music datasets.
- AudioBench provides an expandable foundation across speech understanding, audio scene understanding, and voice understanding.These three aspects define its principal evaluation scope relative to the comparison discussed.
A.4 Why SUPERB and Dynamic-SUPERB is not Ideal
SUPERB and Dynamic-SUPERB are not ideal fits for this benchmark’s goal of evaluating general AudioLLM instruction following. AudioBench instead focuses on AudioLLMs with dedicated datasets and evaluation pipelines.
- SUPERB evaluates self-supervised speech encoders through a supervised fine-tuning step, rather than directly targeting general AudioLLM instruction following.Its design differs from AudioBench’s audio-query-to-answer evaluation setup.
- Dynamic-SUPERB supports zero-shot instruction-following measurement but is an open crowdsourced collection without a specific AudioLLM focus.AudioBench is introduced to address this focus and provide dedicated evaluation pipelines.
B Research Outlook
The research outlook identifies longer-context, multi-round, multilingual, and speech-generation capabilities as important directions for AudioLLMs. It also emphasizes more natural speech output with human-like intonation and rhythm.
- Long Audio Processing and Understanding: Long-audio processing could broaden AudioLLM applications to meeting summarization and sequential event understanding.Current benchmarks primarily use clips within minutes, while text-based LLMs have made rapid progress on long-sequence processing.
- Multi-round Query Handling: Multi-round query handling remains limited in open-source models, constraining dynamic interactions involving different modalities.The outlook specifically highlights queries that may involve images or audio.
- Multilingual Capabilities, Code-Switching, and Dialects: Multilingual, code-switching, and dialect capabilities are needed for broader linguistic and cultural applicability.The stated scope includes handling multiple languages, code-switching, and various dialects.
- Speech Generation: More sophisticated speech generation is sought to support natural, engaging human-computer interactions.The proposed improvements include speech that conveys information while mimicking human-like intonations and rhythms.