MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

S Sakshi; Utkarsh Tyagi; Sonal Kumar; Ashish Seth; Ramaneswaran Selvakumar; Oriol Nieto; Ramani Duraiswami; Sreyan Ghosh; Dinesh Manocha

Source-linked AI summary

MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha

arXiv:2410.19168v1Oct 24, 2024eess.AScs.AIcs.CLcs.SD

TL;DR

Audio-language models lack comprehensive evaluation of advanced perception, knowledge use, and complex reasoning across speech, sounds, and music. MMAU addresses this gap with a 10,000-question, expert-annotated benchmark spanning 27 skills and evaluates 18 models, whose performance demonstrates substantial difficulty. The benchmark’s current scope is limited by its multiple-choice format and other author-identified gaps.

Problem
Existing audio benchmarks lack comprehensive evaluation of audio-language models’ advanced perception, knowledge extraction, and complex reasoning abilities.
Method
MMAU constructs a 10,000-question benchmark across speech, sounds, and music, using expert-annotated questions covering 27 tasks and skills.
Results
The top-performing LALM achieves only 53% accuracy, while the best cascaded captioning + LLM approach reaches 59%.
Takeaways & Limitations
MMAU provides a rigorous test of advanced audio perception and domain-specific reasoning that exposes significant challenges for current models.
Takeaways & Limitations
MMAU currently focuses on multiple-choice tasks and does not evaluate open-ended generation; future work also targets mixed skill types, annotation bias, and broader task coverage.

Abstract

from arXiv · show

The ability to comprehend audio--which includes speech, non-speech sounds, and music--is crucial for AI agents to interact effectively with the world. We present MMAU, a novel benchmark designed to evaluate multimodal audio understanding models on tasks requiring expert-level knowledge and complex reasoning. MMAU comprises 10k carefully curated audio clips paired with human-annotated natural language questions and answers spanning speech, environmental sounds, and music. It includes information extraction and reasoning questions, requiring models to demonstrate 27 distinct skills across unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes advanced perception and reasoning with domain-specific knowledge, challenging models to tackle tasks akin to those faced by experts. We assess 18 open-source and proprietary (Large) Audio-Language Models, demonstrating the significant challenges posed by MMAU. Notably, even the most advanced Gemini Pro v1.5 achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio achieves only 52.50%, highlighting considerable room for improvement. We believe MMAU will drive the audio and multimodal research community to develop more advanced audio understanding models capable of solving complex audio tasks.

1 INTRODUCTION

MMAU addresses the lack of comprehensive benchmarks for audio-language models that test advanced perception, knowledge use, and complex reasoning. It introduces a broad, expert-annotated evaluation spanning speech, sounds, and music.

Current audio benchmarks often assess foundational understanding but lack comprehensive evaluation of audio-language models’ perception and reasoning abilities.
MMAU introduces over 10,000 expertly annotated audio-question-response pairs across speech, sound, and music domains.
The benchmark requires 27 distinct skills through information extraction and reasoning questions involving domain knowledge and advanced perception.
MMAU includes challenging tasks such as multi-speaker role mapping, emotional shift detection, and temporal acoustic event analysis.
The benchmark evaluates 18 open-source and proprietary models, with leading systems achieving only about 53% accuracy.

2 RELATED WORK

Prior work developed audio-language encoders and audio benchmarks, while broader multimodal evaluation increasingly targets expert knowledge and advanced reasoning. These developments motivate more comprehensive audio-focused evaluation.

Audio-Language Models: Audio-language research has advanced cross-modal representation learning through encoders such as AudioCLIP, CLAP, and CompA.
Audio Benchmarks: Recent multimodal benchmarks increasingly evaluate expert-level knowledge and advanced reasoning across areas such as subject knowledge, safety, multilingualism, and multidisciplinary understanding.

3 THE MMAU BENCHMARK

MMAU is a comprehensive benchmark for advanced audio comprehension, knowledge extraction, and complex reasoning across speech, sounds, and music. It uses expert-curated tasks and a multi-stage construction process to provide broad and deep evaluation beyond foundational audio understanding.

3.1 OVERVIEW OF MMAU: MMAU evaluates expert-level multimodal reasoning and knowledge retrieval using curated audio clips paired with domain-expert questions and answers.
3.1 OVERVIEW OF MMAU: MMAU spans speech, sounds, and music with 27 tasks comprising 16 reasoning tasks and 11 information extraction tasks.
3.1 OVERVIEW OF MMAU: The benchmark contains 10,000 multiple-choice questions, including a 1,000-question test-mini set for hyperparameter tuning and a main test set.
3.2 DATA CURATION AND ANNOTATION: The seven-step curation pipeline includes source selection, task curation, expert annotation, filtering, option augmentation, expert review, and finalization.
3.2 DATA CURATION AND ANNOTATION: The final benchmark balances all 27 task types and provides equal coverage of speech, sound, and music.
3.3 COMPARISON WITH OTHER BENCHMARKS: Unlike foundational audio benchmarks, MMAU combines audio perception with knowledge extraction and complex reasoning, covering all three major audio domains.

4 EXPERIMENTAL SETUP

The evaluation compares diverse audio-language and text-only models on MMAU across sound, speech, and music, using micro-averaged accuracy and response processing to extract selected answers.

The study compares open-source, open-access, and proprietary audio-language models, including models specialized in music, sound, or speech and models spanning multiple audio domains.
Results are reported across sound, speech, and music domains, with human evaluation provided for the MMAU test-mini split and training-data coverage marked by modality.
The benchmark evaluation includes text-only GPT-4o and Llama 3 8B Instruct models, with additional tests using audio captions to assess whether external audio information improves performance.
Micro-averaged accuracy is computed from multiple-choice responses after regular-expression processing and string matching map model outputs to the provided options.

5 RESULTS AND DISCUSSION

MMAU is challenging for current models, with performance shaped by audio reliance, caption quality, skill type, and error patterns. The results identify perceptual understanding as a major weakness while showing that detailed captions can improve text-only model performance.

5.1 MAIN RESULTS: The top-performing LALM achieves only 53% accuracy, while the best cascaded captioning plus LLM approach reaches 59%, compared with 82% human performance.
5.1 MAIN RESULTS: Qwen2 performs almost on par with Gemini-Pro, differing by only 0.47% in average performance, whereas GAMA trails Gemini-Pro by 21%.
5.1 MAIN RESULTS: Models trained across speech, sounds, and music, including Qwen2-Audio, LTU-AS, and Gemini, show strong overall performance.
5.2 ARE LALMS REALLY LISTENING?: Replacing audio with Gaussian noise leaves MuLLaMa and SALMONN largely unchanged but causes significant performance drops for GAMA, Qwen2-Instruct, and Gemini Pro v1.5.
5.3 CAN CAPTIONS BRIDGE THE GAP FOR TEXT-ONLY MODELS?: Strong, detailed captions help bridge the audio-understanding gap for text-only models, with GPT4o achieving 59% accuracy.
5.4 ANALYSIS BY SKILL AND DIFFICULTY: Gemini Pro performs well on some skills across difficulty levels but struggles with Temporal Reasoning regardless of difficulty.
5.5 PINPOINTING LALM WEAKNESSES: WHERE ARE THEY FALLING SHORT?: Perceptual errors dominate 500-instance error analyses, accounting for 55% of Qwen2-Audio-Instruct errors and 64% of Gemini Pro v1.5 errors.

6 CONCLUSION, LIMITATIONS AND FUTURE WORK

The paper presents MMAU as a large-scale benchmark for advanced multimodal audio understanding and reasoning, and reports substantial challenges for current models. Future iterations will address task coverage, annotation bias, and evaluation scope.

6 CONCLUSION: MMAU evaluates 27 distinct audio-understanding and reasoning skills through diverse tasks designed to resemble expert-level challenges.
6 CONCLUSION: Across evaluations of 18 open-source and proprietary LALMs, the overall best model achieves only 59% accuracy on MMAU.
6 LIMITATIONS AND FUTURE WORK: Future versions will combine information-extraction and reasoning skills, refine annotations to reduce potential bias, add open-ended tasks, and broaden task coverage.

B ADDITIONAL RESULTS

MMAU evaluates audio-language encoders using hypothesis matching and finds that their apparent reasoning strength is limited by lexical matching. Expert annotation and filtering support the benchmark’s curated question-answer pairs.

Evaluation Strategy: GPT-4o converts each question-choice pair into a hypothesis, and the encoder selects the best option by cosine similarity between audio and hypothesis embeddings.Performance is measured using microaccuracy across all data points.
Results: ALEs perform well in this encoder-tailored evaluation setup despite their encoder-only architecture.The authors note that this resembles findings where ALEs outperform LALMs in deductive reasoning.
Result Analysis: ALEs’ deductive-reasoning advantage reflects bag-of-words behavior rather than robust compositional reasoning.MS-CLAP struggles when counter-options contain the same words in a different order.
Difficulty Analysis: Performance declines on hard tasks after slightly better results on medium tasks, indicating difficulty with more intricate audio reasoning.The authors associate this trend with potential limitations in reasoning or understanding complex audio cues.
Annotation: Expert annotators from sound, speech, and music domains annotated and filtered approximately 11,000 audio-question pairs before final curation.The annotation guidelines required complete listening, valid English multiple-choice questions, and task, difficulty, and dataset tags.

C.4 HUMAN EVALUATION

The paper situates MMAU among audio-language and music-language models spanning general audio understanding, reasoning, captioning, and retrieval. The listed systems use varied encoders, training strategies, and language-model backbones.

Audio-Language Models: Audio Flamingo supports in-context learning, retrieval-augmented generation, and multi-turn dialogue for open- and closed-ended audio tasks.It is described as achieving state-of-the-art results across several audio understanding and few-shot learning tasks.
Audio-Language Models: Qwen-Audio and Qwen2-Audio support diverse audio types and tasks, with Qwen2-Audio generating textual outputs from audio and text inputs.Qwen2-Audio is reported to perform strongly across speech, sound, music, and mixed-audio subsets.
Audio-Language Models: General audio-language models such as LTU, LTU-AS, SALMONN, and Pengi target combinations of perception, reasoning, comprehension, and language generation.Their reported applications include classification, captioning, translation, speaker or keyword-related tasks, and broader audio understanding.
Music-Language Models: Music-language models including MusiLingo, MU-LLaMa, and M2UGen focus on music question answering, captioning, understanding, editing, or generation.The models align music representations with text or support multimodal music-generation tasks.
Audio-Language Encoders: GAMA integrates an LLM with multiple audio representations and reports state-of-the-art performance across 16 datasets and four tasks.MS-CLAP, CompA-CLAP, LAION-CLAP, and ReCLAP instead use contrastive audio-text representations, with CompA-CLAP specifically targeting compositional reasoning.

E DATASET DETAILS

MMAU pools speech, sound, and music resources while using expert annotation tools and structured task labels. Its benchmark design emphasizes difficult reasoning and information-extraction skills beyond prior audio benchmarks.

Data Sources: MMAU draws real data from diverse resources including AudioSet, AudioSet Strong, VoxCeleb, MELD, MusicCaps, MusicBench, MTG-Jamendo, SDD, GuitarSet, and MUSDB18.These sources cover audio events, speech, emotion, music description, tagging, transcription, and source separation.
Annotation: The annotation tool presents annotators with audio, questions, options, and answers so they can accept or reject each question-answer pair.The workflow includes expert annotation, answer verification, and filtering of plausible samples.
Benchmark Comparison: MMAU is presented as more difficult than prior audio benchmarks because its questions require complex reasoning ability.The comparison is summarized in Table 7.
Task Coverage: MMAU covers sound, speech, and music through task examples involving event recognition, causal reasoning, speaker counting, acoustic-source inference, and environmental interpretation.The examples require interpreting multiple sounds, identifying sources, and inferring settings or events.
Task Coverage: Additional tasks test sarcasm, dissonant emotion, historical and cultural reasoning, temporal ordering, duration, melody, and background instruments.These tasks connect audio perception with conversational, cultural, temporal, and musical knowledge.

I FAILURE CASES

The failure analysis illustrates errors across sound, speech, and music tasks, including source confusion, event misidentification, counting mistakes, and incorrect conversational or musical reasoning. The examples are organized around failures of the Qwen2-Audio-Instruct model.

Failure Overview: Qwen2-Audio-Instruct failure cases span sound, speech, and music categories with sub-category information.The table presents representative model failures rather than aggregate performance metrics.
Sound Failures: The benchmark tests event recognition by contrasting similar possibilities such as an alarm clock, handbell, bell tower, and repeatedly pressed doorbell.These options require distinguishing closely related sound events.
Sound Failures: Sound failures include confusing rain patterns, dripping water, instruments, mechanisms, bells, and other acoustic sources.The examples distinguish among ambient events, source identities, and event interpretations.
Failure Overview: One documented response is labeled “Nine One (incorrect reasoning),” illustrating an explicit reasoning error in the failure examples.The surrounding layout pairs domain questions with answer and model-response information.
Speech Failures: Speech failures include counting stressed words, retrieving a speaker’s name, identifying who issued an instruction, and counting conversation participants.These examples require phonemic, conversational, and multi-speaker analysis.
Music Failures: Music-related failures include estimating tempo, identifying melody or lead instruments, and applying historical or cultural knowledge.The examples ask for tempo in bpm, instrument roles, and musician associations.

J BENCHMARK EVALUATION

The benchmark evaluation uses blinded domain-expert ratings to assess the difficulty of existing benchmarks. Experts reviewed audio-question samples under fixed guidelines, and aggregated feedback appears in Table 2.

Experts rated each existing benchmark’s question-solving difficulty on a 1-to-5 scale.The evaluation used 1,000 randomly selected samples per benchmark, or the entire benchmark when it contained fewer than 1,000 examples.
Domain experts listened to the audio and answered corresponding questions using fixed guidelines covering question breadth and type.
Benchmark names were concealed before evaluation to support unbiased difficulty scoring.Before assigning scores, experts summarized their evaluation in one or two sentences.
Aggregated expert feedback and difficulty scores were reported in Table 2.

K ADDITIONAL DETAILS ON ERROR TYPES

This section categorizes MMAU errors, illustrates representative question-prediction failures, and documents prompts used to construct contrastive options, captions, hypotheses, and transcription-based or premise-formulated tasks.

Error Types: MMAU distinguishes knowledge, reasoning, formatting, and answer-key errors in model predictions.Examples include lacking frequency knowledge, incorrect reasoning about sound, logical-reasoning difficulty, incorrect answer formatting, and a wrong answer key.
Error Types: A frequency question illustrates a knowledge error when the model understands the audio but lacks the instrument’s specific frequency information.The example asks for the typical frequency range of a bass and includes ranges from 20 Hz to 200 Hz, 40 Hz to 400 Hz, 400 Hz to 4 kHz, and 4 kHz to 40 kHz.
Error Types: A weather question illustrates incorrect reasoning about sound when the model predicts “Humid” among choices including Windy, Calm, and Rainy.
Error Types: A focus-identification question illustrates logical-reasoning difficulty by asking whether the audio centers on speech, breathing, continuous music, or singing with music.
Error Types: Table 10 supplies additional error-type details and examples based on Gemini Pro v1.5 predictions.
Prompting and Construction Details: The appendix provides prompts for generating six contrastive options, accurate speech transcriptions, premise statements, captions, and hypotheses from question-choice pairs.The examples include reformulating “Does the audio contain any melody?” and “It’s hard to tell” into a single premise statement.