PaperlayerView exact revision on arXiv

Source-linked AI summary

MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence

Sonal Kumar, Šimon Sedláček, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeonggon Ryu, Lichang Chen, Maxim Plička, Miroslav Hlaváček, William Fineas Ellingwood, Sathvik Udupa, Siyuan Hou, Allison Ferner, Sara Barahona, Cecilia Bolaños, Satish Rahi, Laura Herrera-Alarcón, Satvik Dixit, Siddhi Patil, Soham Deshmukh, Lasha Koroshinadze, Yao Liu, Leibny Paola Garcia Perera, Eleni Zanou, Themos Stafylakis, Joon Son Chung, David Harwath, Chao Zhang, Dinesh Manocha, Alicia Lozano-Diez, Santosh Kesiraju, Sreyan Ghosh, Ramani Duraiswami

arXiv:2508.13992v1eess.AScs.SD

TL;DR

Comprehensive evaluation of audio intelligence remains underserved despite its importance for generally intelligent AI systems. MMAU-Pro addresses this gap with a broad, expert-annotated benchmark spanning diverse audio skills and challenging reasoning settings, and evaluations show that leading models still struggle, especially on complex multi-audio, spatial, and open-ended tasks.

  • Problem

    Audio intelligence evaluation remains underserved, while progress toward artificial general intelligence is incomplete without strong capabilities across diverse and complex audio.

    Source: Introduction · p. 1, Audio Benchmarks · p. 3

  • Method

    MMAU-Pro is a human-involved benchmark-construction pipeline with 5,305 expert-annotated question–answer pairs covering 49 skills across speech, sounds, music, mixtures, long audio, spatial reasoning, and multi-audio analysis.

    Source: Introduction · p. 2, Data Curation, Annotation and Validation · p. 3

  • Results

    Leading models achieve only moderate accuracy on single-modality tasks, typically 20–50% on mixed modalities, rarely surpass 40% on spatial audio, and remain below 30% on multi-audio reasoning.

    Source: Results and Discussion · p. 7

  • Takeaways & Limitations

    Multi-audio, spatial reasoning, free-form answering, instruction following, and nuanced temporal, prosodic, and emotional understanding remain clear directions for model and benchmark development.

    Source: Results and Discussion · p. 7

    The benchmark does not explore all dimensions of audio processing and reasoning ability, and fully benchmarking human auditory capabilities remains a work in progress.

    Source: Conclusion, Limitations and Future Work · p. 10

Abstract

from arXiv · show

Audio comprehension-including speech, non-speech sounds, and music-is essential for achieving human-level intelligence. Consequently, AI agents must demonstrate holistic audio understanding to qualify as generally intelligent. However, evaluating auditory intelligence comprehensively remains challenging. To address this gap, we introduce MMAU-Pro, the most comprehensive and rigorously curated benchmark for assessing audio intelligence in AI systems. MMAU-Pro contains 5,305 instances, where each instance has one or more audios paired with human expert-generated question-answer pairs, spanning speech, sound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro evaluates auditory intelligence across 49 unique skills and multiple complex dimensions, including long-form audio comprehension, spatial audio reasoning, multi-audio understanding, among others. All questions are meticulously designed to require deliberate multi-hop reasoning, including both multiple-choice and open-ended response formats. Importantly, audio data is sourced directly ``from the wild" rather than from existing datasets with known distributions. We evaluate 22 leading open-source and proprietary multimodal AI models, revealing significant limitations: even state-of-the-art models such as Gemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy, respectively, approaching random performance in multiple categories. Our extensive analysis highlights specific shortcomings and provides novel insights, offering actionable perspectives for the community to enhance future AI systems' progression toward audio general intelligence. The benchmark and code is available at https://sonalkum.github.io/mmau-pro.

Introduction

MMAU-Pro addresses the limited coverage of auditory-intelligence evaluation with a broad benchmark spanning core audio domains, mixtures, and challenging reasoning settings. It combines expert annotation, real-world audio, diverse skills, and model evaluation to expose persistent weaknesses in current systems.

  • MMAU-Pro targets an underserved gap in evaluating audio intelligence as a component of general intelligence.Source: Introduction · p. 1
  • Questions require deliberate multi-hop reasoning and include both multiple-choice and open-ended formats, while audio is drawn from the wild.Source: Introduction · p. 2
  • The benchmark contains 5,305 expert-annotated question–answer pairs spanning 49 auditory-intelligence skills.Source: Introduction · pp. 1–2
  • It covers speech, environmental sounds, music, mixtures, long-form audio up to ten minutes, multi-audio and spatial reasoning, voice-chat comprehension, and instruction following.Source: Introduction · p. 2, Figure 1 · p. 2
  • 59.2% accuracy for Gemini 2.5 Flash and 51.7% for Audio Flamingo 3 show that leading models still face substantial challenges.Source: Introduction · p. 2

Related Work

Prior audio benchmarks established broad and specialized evaluation settings but remain shallow, fragmented, or limited in scale, data sources, and task coverage. MMAU-Pro addresses these gaps by jointly testing underexplored dimensions relevant to real-world audio intelligence.

  • Earlier benchmarks evaluated speech, sounds, music, reasoning, spoken language, instruction following, or hallucinations, but generally focused on narrower settings.Source: Large Audio Language Models · p. 2, Audio Benchmarks · p. 3
  • Existing evaluations remained shallow and fragmented despite some progress on multi-audio and spatial-audio tasks.Source: Audio Benchmarks · p. 3
  • No existing benchmark systematically combined long-form audio up to ten minutes, multi-audio, spatial, open-ended, instruction-following, and multicultural scenarios.Source: Audio Benchmarks · p. 3
  • MMAU-Pro was designed to provide comprehensive coverage of these underexplored dimensions for real-world audio general intelligence.Source: Audio Benchmarks · p. 3

The MMAU-Pro Benchmark

MMAU-Pro is a human-curated benchmark built to test diverse audio skills and realistic reasoning conditions beyond conventional short, single-audio multiple-choice evaluation. Its design combines expert validation, in-the-wild recordings, long-duration categories, open-ended responses, and specialized audio scenarios.

Benchmark Design

Evaluation Formats

Data Curation, Annotation and Validation

Task Coverage

Experimental Setup

The evaluation covers large audio-language models, cascaded captioning or transcription systems, and both multiple-choice and open-ended outputs. It uses embedding similarity for answer-choice selection and an LLM judge for open-ended response quality.

  • Large Audio-Language Models are evaluated on long- and short-form reasoning, spatial understanding, multicultural music, and multi-audio comparisons.Source: Experimental Setup · p. 6
  • Cascaded systems first generate captions for sound and music or transcripts for speech, then pass these with questions to text-only LLMs.Source: Experimental Setup · p. 6
  • For multiple-choice questions, the system selects the answer whose pretrained-transformer embedding is most similar to the model’s output embedding.Source: Experimental Setup · p. 6
  • Open-ended responses are judged on correctness, relevance, completeness, clarity, and an overall score, with 1–5 scores converted to percentages.Source: Experimental Setup · p. 6

Results and Discussion

MMAU-Pro exposes persistent weaknesses in audio-language models as tasks move beyond single-domain understanding toward mixed, spatial, multi-audio, open-ended, and instruction-following settings. Performance also varies substantially by skill, question phrasing, musical culture, and whether models can rely on acoustic evidence.

Cross-modal performance

  • 30–60% accuracy is typical on core Sound, Music, and Speech tasks, while Qwen2.5-Omni-7B rarely exceeds 65% on Music or Speech.These results indicate that foundational audio understanding remains challenging even for strong open-source models.Source: Results and Discussion · p. 7
  • 20–50% accuracy is typical on mixed-modality tasks, and top models rarely surpass 40% on spatial audio understanding.The reported degradation suggests difficulty fusing information across multiple audio streams.Source: Results and Discussion · p. 7

Reasoning and instruction following

  • 71.7% is Gemini 2.5 Flash’s score on voice-chat reasoning and STEM knowledge inference, compared with 60% for Qwen2.5-Omni-7B; smaller or less instruction-tuned models often fall below 50%.Most models nevertheless score between 25% and 60% on these tasks.Source: Results and Discussion · p. 7

Benchmark sensitivity

  • Up to 5% gains from LLM-generated question rephrasing show that Phi4-MM-Instruct and Qwen2.5-Omni-3B are sensitive to surface-level language cues.Phi4-MM-Instruct rises from 37.5% on originals to 41.3% on GPT4o rewrites, while Qwen2.5-Omni-3B rises from 43.4% to 48.6%.Source: Effect of question rephrasing · p. 8
  • 52.2% to 30.6% is Qwen2.5-Omni-7B’s drop when clean audio is replaced with noise, while AF3 improves from 47.2% noise-only to 51.7% with actual audio.The clean-versus-noise gap indicates that benchmark performance depends on acoustic input rather than language priors alone.Source: Are LALMs listening? · p. 9

Conclusion, Limitations and Future Work

MMAU-Pro broadens holistic audio-intelligence evaluation with diverse, demanding tasks, while showing that leading models still struggle across categories. The authors acknowledge incomplete coverage and outline extensions for languages, environments, interaction, instruction following, and culturally grounded reasoning.

Conclusion

Limitations

Future Work

  • Future work will expand languages and low-resource acoustic environments, add interactive streaming tasks, refine instruction-following evaluation, and improve paralinguistic and culturally grounded reasoning metrics.Source: Conclusion, Limitations and Future Work · p. 10

A Curating the Audio Set

The audio set combines in-the-wild recordings from everyday videos, spoken-content sources, sound effects, and full musical tracks across cultures and genres.

A Curating the Audio Set

B Annotation Guidelines

Annotation procedures require audio-only solvability, complete listening, valid audio inputs, English questions, and task labeling, while rephrasing prompts preserve meaning without adding context.

B Annotation Guidelines

Prompt used for open-ended Evaluation

Open-ended responses are evaluated by an expert-evaluator prompt that scores factual accuracy, question alignment, coverage, clarity, and an overall average in a fixed format.

Prompt used for open-ended Evaluation

D Open-ended QA Evaluation Ablations

The evaluation compares LLM-as-a-judge scores with human correctness ratings for open-ended audio question-answer responses on MMAU and MMAR.

E Skill wise examples and their definitions

The appendix presents examples and brief definitions for MMAU-Pro skills across music, sound, and speech, covering both perceptual and reasoning tasks.

Skill-wise examples and definitions

Music skills

  • Music skills are organized into perceptual and reasoning categories with domains, definitions, and example questions.Table 7 focuses on multiple-choice perceptual music questions, while Table 8 covers multi-step reasoning in music.Source: Table 7 · p. 16, Table 8 · p. 16

Sound skills

  • Sound skills are divided into perceptual and reasoning categories with domains, definitions, and example questions.Tables 9 and 10 describe sound perception tasks and deep reasoning over sound, respectively.Source: Table 9 · p. 17, Table 10 · p. 17

Speech skills

  • Speech skills include separate perceptual and reasoning tables containing domains, definitions, and example questions.Table 11 addresses speech perception, whereas Table 12 addresses speech reasoning tasks.Source: Table 11 · p. 17, Table 12 · p. 18

Related papers

Loading 2508.13992v1…