Source-linked AI summary
MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, Zhikang Niu, Tianrui Wang, Yuping Wang, Yuxuan Wang, Yihao Wu, Guanrou Yang, Jianwei Yu, Ruibin Yuan, Zhisheng Zheng, Ziya Zhou, Haina Zhu, Wei Xue, Emmanouil Benetos, Kai Yu, Eng-Siong Chng, Xie Chen
TL;DR
Existing audio benchmarks provide limited evidence about deep reasoning across diverse, mixed-modality scenarios. MMAR addresses this gap with a 1,000-question, expert-curated benchmark and evaluates 30 audio-capable models, finding that open-source LALMs remain near random guessing while reasoning-enhanced and closed-source models perform more strongly.
Problem
Existing benchmarks primarily assess surface-level audio understanding and provide limited evaluation of deep reasoning across diverse audio domains.
Method
MMAR constructs 1,000 human-curated, multi-step audio reasoning questions spanning mixed modalities and four hierarchical reasoning layers, then benchmarks 30 audio-capable models.
Results
Open-source LALMs perform only marginally above random guessing, whereas reasoning-enhanced and closed-source models show significantly stronger performance.
Takeaways & Limitations
MMAR exposes substantial limitations in current audio understanding and reasoning capabilities and provides insights for developing stronger LARMs.
Takeaways & Limitations
A small subset of questions remains answerable from text priors alone, and the dataset is limited to 1,000 examples because annotation is difficult.
Abstract
from arXiv · showhide
We introduce MMAR, a new benchmark designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR comprises 1,000 meticulously curated audio-question-answer triplets, collected from real-world internet videos and refined through iterative error corrections and quality checks to ensure high quality. Unlike existing benchmarks that are limited to specific domains of sound, music, or speech, MMAR extends them to a broad spectrum of real-world audio scenarios, including mixed-modality combinations of sound, music, and speech. Each question in MMAR is hierarchically categorized across four reasoning layers: Signal, Perception, Semantic, and Cultural, with additional sub-categories within each layer to reflect task diversity and complexity. To further foster research in this area, we annotate every question with a Chain-of-Thought (CoT) rationale to promote future advancements in audio reasoning. Each item in the benchmark demands multi-step deep reasoning beyond surface-level understanding. Moreover, a part of the questions requires graduate-level perceptual and domain-specific knowledge, elevating the benchmark's difficulty and depth. We evaluate MMAR using a broad set of models, including Large Audio-Language Models (LALMs), Large Audio Reasoning Models (LARMs), Omni Language Models (OLMs), Large Language Models (LLMs), and Large Reasoning Models (LRMs), with audio caption inputs. The performance of these models on MMAR highlights the benchmark's challenging nature, and our analysis further reveals critical limitations of understanding and reasoning capabilities among current models. We hope MMAR will serve as a catalyst for future advances in this important but little-explored area.
1 Introduction
MMAR addresses the limited evaluation of deep audio reasoning by introducing a broad, carefully curated benchmark for multi-step reasoning across diverse audio inputs. Its model evaluation shows substantial capability gaps, especially among open-source systems.
- Motivation and benchmark: MMAR evaluates Audio-Language Models on multi-step reasoning across sound, speech, music, and mixed-modality audio.The benchmark targets expert-level perceptual understanding, logical inference, and contextual or domain-specific knowledge.
- Benchmark design: Each question is organized into Signal, Perception, Semantic, and Cultural reasoning layers with fine-grained sub-categories and a manually labeled Chain-of-Thought rationale.The taxonomy was co-developed through Human–LLM collaboration.
- Benchmark construction: The benchmark construction uses expert authorship, multi-stage refinement, and rigorous quality control to produce reliable annotations.These procedures are presented as central to MMAR’s quality.
- Evaluation and findings: 30 audio-capable models are evaluated across LALMs, LARMs, OLMs, LLMs, and LRMs using audio caption inputs.The evaluation covers 24 open-source and 6 closed-source models.
- Evaluation and findings: Open-source LALMs struggle substantially, while closed-source and reasoning-enhanced models perform more strongly on MMAR.The analysis motivates development of stronger open-source Large Audio Reasoning Models.
2 Related Work
Prior audio benchmarks cover sound, music, speech, and broader audio understanding, but generally provide limited evaluation of deep reasoning. MMAR extends this landscape with real-world mixed-modality audio and a comprehensive reasoning-oriented design.
- Existing benchmarks: Existing benchmarks span sound, music, speech, and multiple audio domains, but primarily assess surface-level understanding.Examples include Clotho-AQA, MusicBench, LibriSQA, AudioBench, AIR-Bench, and MMAU.
- MMAR’s distinction: MMAR focuses on deep reasoning, an important but underexplored component of audio understanding.Its design is explicitly multi-dimensional rather than limited to a single audio domain.
- Mixed-modality coverage: MMAR includes naturally occurring mixtures of sound, speech, and music, unlike benchmarks restricted to unimodal domains or artificially combined clips.The benchmark targets complex interactions present in real-world audio scenarios.
- MMAR’s distinction: MMAR combines mixed modalities, real-world audio, deep reasoning challenges, and stronger data-quality standards.These features are presented as advances over existing benchmark construction.
- Comparison dimensions: Table 1 compares benchmarks by domain coverage, task scope, evaluation paradigm, and data origin.Its notation distinguishes artificially synthesized data from in-the-wild or expert-handcrafted material.
3 The MMAR Benchmark
MMAR is a 1,000-question benchmark built around broad real-world audio coverage and a four-layer reasoning taxonomy. Its expert-led curation pipeline combines human annotation, LLM assistance, audio processing, and iterative quality inspection.
- 3.1 Overview: MMAR contains 1,000 handcrafted audio reasoning tasks requiring multi-step inference, with some demanding challenging perceptual skills and domain-specific knowledge.Domain experts developed, refined, and validated the questions.
- 3.1 Overview: The benchmark covers seven audio domains, including sound, music, speech, and four mixed combinations.The mixed categories are mix-sound-music, mix-sound-speech, mix-music-speech, and mix-sound-music-speech.
- 3.1 Overview: Its hierarchical taxonomy progresses from concrete to abstract through Signal, Perception, Semantic, and Cultural layers.Each layer contains multiple sub-categories representing varied reasoning tasks.
- 3.2 Data Curation Pipeline: The five-stage curation pipeline comprises brainstorming, taxonomy construction, heuristic human annotation, raw-data preparation, and quality inspection.The process is summarized in Figure 3.
- 3.2 Data Curation Pipeline: Experts brainstorm questions, while LLMs organize insights and collaborate with experts to construct the hierarchical taxonomy.This establishes task sub-categories before data collection and annotation.
- 3.2 Data Curation Pipeline: Annotators locate relevant internet videos and label metadata including timestamps, questions, answers, CoTs, modalities, categories, and spoken language.Audio clips are then crawled, trimmed, and processed, while CoTs and distractor options are refined or generated.
- 3.2 Data Curation Pipeline: Independent authorship, correction, review, and iterative inspection provide the benchmark’s dual quality guarantee.Questions failing inspection are subject to further revision before inclusion.
4 Experimental Setup
The experiments compare five categories of audio-capable models under a multiple-choice evaluation protocol. Accuracy is determined by matching each model’s selected option against the ground-truth answer.
- Model categories: The study evaluates LALMs, LARMs, OLMs, caption-based LLMs, and caption-based LRMs.LLMs and LRMs reason over captions generated from audio.
- Evaluation protocol: MMAR uses classification accuracy because every task is formulated as a multiple-choice question.The evaluation inputs audio, the question, and answer choices to each model.
- Evaluation protocol: Prediction correctness is determined by regular expressions and string matching against ground-truth answers.This follows the approach used by MMAU.
5 Experimental Results
MMAR is highly challenging across seven audio domains, with open-source models generally near random-guessing performance and a substantial gap separating them from closed-source systems. Explicit reasoning models perform better than models without such capabilities.
- Evaluation setup: MMAR evaluates five model categories across seven single- and mixed-modality domains using accuracy on multiple-choice questions.The evaluated categories are LALMs, LARMs, OLMs, LLMs, and LRMs with audio captions.
- Overall difficulty: Below 60% average accuracy, Qwen-2.5-Omni (7B) remains the strongest open-source model, while none of the open-source LALMs significantly outperform random guessing.Significance is assessed with a Bonferroni-corrected threshold.
- Model comparison: 65.6% accuracy is achieved by closed-source Gemini 2.0 Flash, exceeding Qwen2.5-Omni (7B) at 56.7% and all cascaded setups.The comparison indicates a substantial open-source versus closed-source performance gap.
- Reasoning capability: Explicit reasoning models consistently outperform counterparts without explicit reasoning in both end-to-end and cascaded architectures.Audio-Reasoner surpasses Qwen2-Audio models, and Caption + DeepSeek-R1 outperforms Caption + DeepSeek-V3.
- Implications: The results motivate further innovation in audio-language reasoning, particularly for open-source systems where the performance gap remains substantial.MMAR emphasizes multi-step reasoning and rich audio content beyond traditional audio question answering.
6 Discussion
Analysis across reasoning layers, noise replacement, cascaded systems, and failures shows that MMAR probes both audio perception and reasoning. Models use the audio signal, while performance benefits independently from stronger perception and reasoning components.
- 6.1 Reasoning hierarchies: All representative models perform best on the Semantic layer and worst on the Signal layer, although random guessing is highest on Signal.Figure 5a compares a representative LALM, LARM, and OLM across the four hierarchical layers.
- 6.2 Comparison with Noise Input: Replacing audio with noise causes a substantial performance drop even for Qwen2-Audio-Instruct, indicating that models rely on audio rather than only textual or statistical biases.The experiment tests both audio utilization and the role of language priors.
- 6.3 Comparison on Cascaded Models: Improving either the audio captioning front-end or the language model increases cascaded-model accuracy, showing cumulative contributions from perception and reasoning abilities.Table 3 compares Qwen2-Audio-Instruct and Qwen-2.5-Omni front-ends with GPT-4o and OpenAI o1.
- 6.4 Error Analysis: 37% of 100 failed Audio-Reasoner predictions are perceptual errors, including difficulty distinguishing environmental sounds, identifying musical structures, and interpreting speech.Reasoning errors account for 20%, while knowledge gaps account for 9%.
7 Conclusion
MMAR is presented as a 1,000-question benchmark for deep audio reasoning across real-world, mixed-modality tasks and four hierarchical layers. Evaluation of 30 models finds substantial difficulty, with stronger performance from reasoning-augmented and closed-source systems.
- Benchmark contribution: MMAR contains 1,000 human-curated questions spanning four hierarchical reasoning layers, with detailed chains of thought for interpretability and future research.The benchmark covers real-world, mixed-modality, and multidisciplinary audio tasks.
- Conclusion: Evaluation of 30 audio-capable models across five categories shows that MMAR is substantially more challenging than existing benchmarks.Open-source LALMs perform only marginally above random guessing.
- Conclusion: Reasoning-augmented and closed-source models show significantly stronger performance, while ablations and error analysis reveal limitations in current models.The authors position MMAR as a rigorous benchmark for advancing audio reasoning.
A Task Layer Definitions
MMAR organizes audio reasoning into four layers, progressing from raw acoustic analysis to perceptual interpretation, semantic understanding, and culturally grounded knowledge. Its examples span low-level signal judgments, perceptual and semantic inferences, and domain-specific musical evaluation.
- Signal Layer: Signal tasks analyze raw acoustic features such as frequency, amplitude, duration, rhythm, and silence with minimal semantic context.
- Perception Layer: Perception tasks interpret sound types, paralinguistic features, and musical instrumentation beyond raw signal analysis.
- Semantic Layer: Semantic tasks infer meaning or intent from speech, sound events, and complex audio scenes.
- Cultural Layer: Cultural tasks integrate audio with social norms, world knowledge, contextual information, or domain-specific expertise.
- Examples require multi-step judgments, including Mandarin-tone counting, spatial inference, event reconstruction, and comparative piano-performance assessment.The piano example favors the fourth passage because it combines complex rhythms, arpeggiated chords, and fast note clusters.
B.1 Speech and Sound Tasks
Speech and sound annotations were produced by academically trained researchers with relevant research experience and, for many team members, publication experience in leading venues. Music tasks additionally used researchers and conservatory-trained experts to provide specialized perceptual and domain knowledge.
- Speech and sound annotation, correction, and quality assurance personnel all held at least bachelor’s degrees and relevant research experience.Over half were enrolled in or had completed Ph.D. programs, and each had at least one year of relevant research experience.
- Most speech and sound team members had publications in venues including ICASSP, INTERSPEECH, NeurIPS, and ICML.The paper links this background to understanding both signal and semantic aspects of audio content.
- Music annotation combined music-technology researchers with graduate students and alumni from prestigious music conservatories.The music experts contributed academic and artistic expertise in music perception and domain knowledge.
C Data Annotation Platform
MMAR’s annotation process used real-world short audio clips, structured authoring requirements, and multi-stage platform-based review. Instructions emphasized audio-grounded, objective, unambiguous, and genuinely multi-step questions rather than simple recognition.
- Data Annotation Platform: The annotation platform supported structured editing, version control, and multi-stage review of questions, choices, answers, reasoning chains, and metadata.Figure 7 shows a snapshot of this correction and quality-inspection platform.
- Source and Duration Requirements: All audio was sourced from real-world internet videos, with links and timestamps preserved, and each clip was limited to 30 seconds.
- Language and Expression: Drafting could occur in Chinese, but questions were finalized in English where appropriate and generally used role-based descriptors instead of specific names.
- Audio-Grounded Design: Questions had to depend on audio alone, avoid language-prior answers, and exclude intuitive examples such as identifying a dog from barking.
- Question and Answer Quality: Answers were required to be concise, objective or subjectively universal, and limited to one valid interpretation.Annotators were also instructed to avoid overly broad or unclear formulations.
- Reasoning Requirements: Each question required at least two reasoning cues, including one audio cue, and multi-step reasoning beyond direct captioning or ASR.The guidelines encouraged predictive, counterfactual, evaluative, planning, and domain-specific questions.
E Benchmarking Candidates
MMAR benchmarking spans specialized audio models, omni-modal models, general language models, and reasoning models, alongside prior audio systems with varied architectures and training strategies. Across representative comparisons, MMAR is substantially harder than MMAU, with accuracy dropping for every model shown.
- The benchmark includes LALMs, LARMs, OLMs, LLMs, and LRMs, covering systems for general audio, speech, music, and reasoning-focused tasks.The candidate set includes Audio Flamingo, SALMONN, LTU variants, MusiLingo, Qwen-Audio, Audio-Reasoner, and omni-modal models.
- Prior candidates differ in architecture and supervision, including dual encoders, audio frontends, Q-Formers, continuous audio tokens, instruction tuning, and chain-of-thought data.
- The evaluated candidate pool also contains compact, open, proprietary, and omni-modal systems supporting audio through native input or audio captions.
- Every representative model experiences a significant accuracy drop from MMAU test-mini to MMAR.
- Qwen-2.5-Omni 7B exceeds 65% accuracy on MMAU but falls below 60% on MMAR.
G Error Analysis
MMAR error analysis identifies perceptual, reasoning, knowledge, and other failures, with perceptual errors the most frequent and instruction-following failures also substantial. The benchmark’s evaluation uses multiple-choice accuracy, while its limited size and residual text-prior bias constrain coverage and interpretation.
- Failure Types: Perceptual errors account for 37% of failures, including misclassified sounds, missed repeated events, and confusion about fine-grained musical elements.Models also struggle with speech perception, global auditory context, polyphonic disambiguation, and detailed music information retrieval.
- Failure Types: Lack of knowledge causes 9% of failures when models perceive sounds but lack commonsense, cultural, or scientific understanding needed for interpretation.Examples include confusing instruments, misinterpreting character relationships, and failing to connect physical events with scientific consequences.
- Failure Types: Reasoning errors comprise 20% of failures, involving causal structure, temporal dynamics, discourse logic, emotion attribution, sarcasm, coreference, and quantitative estimation.These errors occur when models perceive relevant cues but misinterpret their relationships or implications.
- Failure Types: Other errors account for 34% of failures, including instruction misinterpretation, incorrect option selection, generation collapse, and choice-formatting mistakes.These failures expose fragility in instruction following, constrained decoding, and long-horizon reasoning.
- Evaluation: MMAR evaluates models with classification accuracy by matching predicted choices against ground-truth answers for multiple-choice questions.The supplied benchmark-results table captions identify evaluations with audio captioning models and noise-replaced audio, but provide no numerical results here.
- Limitations: The benchmark contains 1,000 examples and may include a small subset answerable from text priors alone, limiting coverage and leaving residual bias despite three-stage review.The authors plan expansion, particularly in underrepresented sub-categories and modalities.