AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

Qian Yang; Jin Xu; Wenrui Liu; Yunfei Chu; Ziyue Jiang; Xiaohuan Zhou; Yichong Leng; Yuanjun Lv; Zhou Zhao; Chang Zhou; Jingren Zhou

Source-linked AI summary

AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, Jingren Zhou

arXiv:2402.07729v2Feb 12, 2024eess.AScs.CLcs.LGcs.SD

TL;DR

Existing LALM benchmarks emphasize fundamental audio tasks and lack standardized evaluation of open-ended audio-centered interaction. AIR-Bench addresses this gap with a hierarchical benchmark and GPT-4-based evaluation framework spanning diverse audio types, and its experiments reveal limited audio understanding or instruction-following capabilities in existing models.

Problem
Standardized benchmarks for LALMs’ generative instruction-following and open-ended audio interaction are lacking, limiting evaluation beyond fundamental tasks.
Method
AIR-Bench combines 19 foundation audio tasks with over 19k single-choice questions, over 2k open-ended chat questions, audio mixing, and GPT-4-based hypothesis evaluation.
Results
Evaluation of 9 LALMs shows that existing models have limited audio understanding or instruction-following capabilities, leaving significant room for improvement.
Takeaways & Limitations
AIR-Bench provides a broad generative evaluation framework for LALMs across speech, natural sounds, music, and complex audio interactions.
Takeaways & Limitations
AIR-Bench excludes multiple-audio comparison tasks and multiturn dialogues with multiple audio inputs, and its evaluation depends on GPT-4 availability and access.

Abstract

from arXiv · show

Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as Automatic Speech Recognition (ASR), and lack an assessment of the open-ended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (\textbf{A}udio \textbf{I}nst\textbf{R}uction \textbf{Bench}mark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: \textit{foundation} and \textit{chat} benchmarks. The former consists of 19 tasks with approximately 19k single-choice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended question-and-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research.

1 Introduction

AIR-Bench addresses the lack of standardized evaluation for generative instruction-following in audio-language models. It introduces a hierarchical benchmark spanning diverse audio and combines foundation testing, chat evaluation, audio mixing, and GPT-4-based scoring.

Motivation: Previous LALMs mainly target specific fundamental tasks, while standardized evaluation of generative instruction-following remains absent.This makes fair comparisons and tracking model limitations difficult.
Benchmark design: AIR-Bench evaluates audio understanding across human speech, natural sounds, and music through foundation and chat benchmarks.The foundation benchmark contains 19 audio tasks and over 19k single-choice questions; the chat benchmark contains over 2k open-ended audio questions.
Benchmark design: The benchmark introduces audio mixing with loudness control and temporal dislocation to increase audio complexity.The strategy aims to make mixed audio more closely resemble real-world scenarios.
Evaluation: AIR-Bench uses a unified, objective, and reproducible framework to evaluate generative hypotheses.GPT-4 extends questions and choices, then evaluates foundation correctness or chat hypotheses using reference answers.
Experiments: The authors benchmarked 9 LALMs to assess their capabilities.The evaluation is intended to expose limitations in audio understanding and instruction following.

2 Related Work

Earlier audio benchmarks largely evaluate task-specific or constrained abilities, while instruction-following evaluation remains narrower. AIR-Bench extends coverage across audio types and evaluates open-ended generative interaction.

Task-specific benchmarks: Speech benchmarks commonly evaluate automatic speech recognition, while sound and music benchmarks target captioning or question answering.Examples include LibriSpeech, Common Voice, Clotho, AudioCaps, MusicCaps, and MUSIC-AVQA.
Task-specific benchmarks: Many existing audio question-answering benchmarks use highly constrained answer formats.These formats simplify evaluation but do not directly assess free-form generation.
Instruction-following benchmarks: SUPERB and HEAR provide broad evaluation for self-supervised audio models, but they are not dedicated to LALM instruction following.Dynamic-SUPERB is dedicated to instruction following but focuses on human speech and excludes open-ended dialogue generation.
AIR-Bench: AIR-Bench is presented as the first large-scale generative evaluation benchmark for LALMs across varied audio types.Its design targets both foundational abilities and high-level instruction-following capabilities.
AIR-Bench: AIR-Bench covers speech, natural sounds, music, and mixed audio in hierarchical foundation and chat benchmarks.Its foundation dimension uses single-choice questions, whereas its chat dimension uses open-ended question-and-answer evaluation.

3 AIR-Bench

AIR-Bench is a hierarchical benchmark for evaluating LALMs across diverse audio signals, foundational subtasks, and open-ended instruction-following interactions. It uses direct answer generation and GPT-4-based reference evaluation to assess model hypotheses.

Benchmark structure: AIR-Bench evaluates speech, natural sounds, music, and mixed audio through foundation and chat benchmarks.The foundation benchmark targets individual capabilities, while the chat benchmark targets perceiving, reasoning, and interacting through open-ended audio questions.
Foundation benchmark: The foundation benchmark contains 19 audio tasks using single-choice questions to assess individual foundational capabilities.Its task coverage includes speech, music, and sound, with examples such as emotion recognition, acoustic scene classification, and music question answering.
Chat benchmark: The chat benchmark contains over 2k samples across speech, sound, music, and mixed audio for open-ended instruction-following evaluation.Mixed audio combines multiple audio types to increase signal complexity and approximate real-world audio scenarios.
Audio mixing strategy: AIR-Bench increases mixed-audio complexity by controlling the relative loudness and temporal relationship of two audio clips.These controls provide Louder and Ahead meta-information for the resulting mixtures.
Evaluation strategy: Both benchmarks treat model responses as generated hypotheses rather than teacher-forced comparisons against reference-answer probabilities.The unified framework supplies audio meta-information and questions to GPT-4, which evaluates each hypothesis against a reference answer.
Evaluation strategy: Foundation hypotheses receive binary correctness scores, whereas chat hypotheses receive 1-to-10 scores for usefulness, relevance, accuracy, and comprehensiveness.For chat evaluation, the GPT-4-generated answer is used as a scoring reference rather than as ground truth; scores are averaged after swapping hypothesis and reference positions to reduce position bias.

4 Experiments

AIR-Bench experiments compare multiple LALMs across foundation and chat benchmarks, while testing GPT-4-based evaluation against human judgments and positional bias. Results show strong model differences, format-sensitive matching, and high human–GPT-4 agreement for chat preferences.

Models: The experiments evaluate open-source or publicly accessible LALMs, including SpeechGPT, BLSP, SALMONN, Qwen-Audio-Chat, Qwen-Audio Turbo, PandaGPT, Macaw-LLM, and NExT-GPT.Models are evaluated using their latest publicly available checkpoints and default decoding strategies; Whisper plus GPT-4 is also used as a baseline.
Main Results: GPT-4 evaluation raises every model’s foundation matching success rate to 100%, overcoming varied output formats that make exact choice extraction difficult.Exact matching works particularly well for BLSP and SALMONN, but other models often express a choice as a semantically equivalent full sentence.
Main Results: Qwen-Audio-Chat and Qwen-Audio Turbo outperform other models across speech, sound, and music in the foundation benchmark.PandaGPT and SALMONN are reported as the next strongest models in that benchmark.
Main Results: Qwen-Audio Turbo achieves the highest average chat score, followed by SALMONN at 6.11 and Qwen-Audio-Chat at 6.08.SALMONN performs best on mixed-audio understanding, while Whisper plus GPT-4 obtains the highest score when chat evaluation focuses primarily on speech transcription.
Human Evaluation: Chat pairwise preference consistency between GPT-4 and human judgments exceeds 70% on 200 questions evaluated by three native English speakers.Foundation consistency is assessed using 400 randomly selected questions and three native English speakers, with Qwen-Audio-Chat as the representative model.
Human Evaluation: Swapping hypothesis and reference positions reveals clear GPT-4 scoring bias, motivating two evaluations whose scores are averaged.The second scoring is designed to mitigate bias caused by positional placement.

5 Conclusion

The conclusion presents AIR-Bench as a broad generative benchmark for audio-language models and outlines continued leaderboard maintenance for consistent community comparison.

Conclusion: AIR-Bench combines 19 audio tasks with over 19k foundation questions and over 2k open-ended chat questions covering speech, natural sounds, and music.The benchmark uses a standardized, objective, and reproducible framework to assess LALM-generated hypotheses.
Conclusion: The authors plan to maintain a leaderboard so the community can compare model performance consistently over time.The leaderboard is intended as an ongoing platform for accessing and comparing results.

6 Limitations

AIR-Bench does not yet cover multiple-audio comparison tasks or multiturn dialogues with multiple audio inputs, and its evaluation depends on GPT-4 availability and pricing.

Limitations: AIR-Bench excludes multiple-audio comparison tasks, such as music coherence assessment, from both foundation and chat benchmarks.It also does not evaluate multiturn dialogues that may involve multiple audio inputs.
Limitations: The evaluation framework depends on GPT-4, whose future accessibility, openness, and pricing are external factors requiring possible alternative evaluators.The authors state that alternatives would need to be explored if GPT-4 becomes closed-source or more expensive.

7 Ethical Considerations

AIR-Bench constructs question-and-answer data from public datasets and uses automated evaluation, but dataset and scoring biases may cause differences from human judgment.

Ethical Considerations: The benchmark uses publicly available datasets and automated methods to create and evaluate question-and-answer data more efficiently than manual evaluation.The authors follow the licenses and usage guidelines of the original open-source materials when generating related data.
Ethical Considerations: Automated evaluation may introduce dataset or scoring-algorithm biases, so its outcomes should be treated as a general benchmark rather than a definitive measure.The authors caution that automated results can differ from human judgment.

A Detailed Results of Foundation Benchmark

The foundation benchmark evaluates models across tasks with different choice structures, so random-choice accuracy varies by task type.

50% is the theoretical random-selection accuracy for Speaker Gender Recognition and Synthesized Voice Detection.These are binary-choice tasks.
All other foundation-benchmark tasks require selecting from four options.The corresponding random-choice baseline is described as higher-order than the binary-task baseline, but the supplied passage is truncated before its value.

B GPT Prompts for the Chat benchmark

The chat benchmark uses engineered GPT prompts both to generate reasoning-oriented question-answer pairs and to score model responses. Prompt wording and ordering materially affect evaluation reliability.

B GPT Prompts for the Chat benchmark: Figure 5 separates prompts for generating reasoning-related question-answer pairs from prompts for scoring chat performance.The upper section handles question-answer generation, while the lower section assesses model responses.
B GPT Prompts for the Chat benchmark: GPT-4 is instructed to generate questions and reference answers while excluding audio data that is less amenable to question formulation.The process specifies the desired question type before generation.
B GPT Prompts for the Chat benchmark: GPT-4 scores both reference answers and model responses so reference answers remain a comparison standard.The scoring prompt takes a multifaceted approach rather than evaluating model responses in isolation.
C Prompts Engineering for GPT Scoring: The paper demonstrates iterative adjustment of prompts used to assess chat-performance scores.
C Prompts Engineering for GPT Scoring: Removing helpfulness, relevance, accuracy, and comprehensiveness descriptions reduces scoring precision and promotes score normalization.The authors report this pattern across multiple tests.
C Prompts Engineering for GPT Scoring: Reordering the detailed audio description, question, or answer can turn correct evaluations incorrect.The reported sensitivity concerns section permutation, not minor punctuation or grammatical errors.
C Prompts Engineering for GPT Scoring: Absolutely correct and absolutely incorrect responses can both receive scores around 5 after prompt-position changes.The passage reports scores as low as 5 for correct answers and around 5 for incorrect responses.

D Examples of the Foundation Benchmark

The appendix provides examples of foundation-benchmark questions and illustrates how evaluated models respond on both foundation and chat benchmarks.

Table 5 presents example questions and data for each foundation-benchmark task.
Figure 6 shows foundation-benchmark responses from nine models alongside the question and audio metadata.The metadata is not provided to evaluated models, which receive only the audio and question.
Figure 7 provides an example of responses from various models on the chat benchmark.

F Details in Human Evaluation

Human evaluation compares GPT-4 judgments with crowd-worker judgments for both benchmarks, using audio playback and benchmark-specific assessment tasks. The analysis also examines alignment across audio types.

Three native English-speaking judges reviewed each pair of GPT-4 and human evaluations in a quiet headphone-based listening environment.The procedure included sample feedback after testing.
400 foundation-benchmark questions were sampled, with evaluators judging whether GPT-3.5 Turbo and GPT-4 Turbo responses were accurate.The foundation evaluation instructions are shown in Figure 8.
200 chat dialogues were sampled from Qwen-Audio-Chat, SALMONN, BLSP, and GPT-4, with evaluators selecting the superior or equivalent model response.The chat evaluation instructions are shown in Figure 9.
Speech accounts for 42% of sampled chat QAs, followed by Sound at 22%, Mixed Audio at 20%, and Music at 16%.The analysis groups human-judgment alignment by audio type.
Music and Mixed Audio QAs generally show slightly higher GPT-human alignment than Sound and Speech QAs.The paper associates lower alignment in Sound and Speech with situational and reasoning questions that are more challenging for GPT evaluation.