Source-linked AI summary
The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agents
Ziyang Ma, Ruiyang Xu, Yinghao Ma, Chao-Han Huck Yang, Bohan Li, Jaeyeon Kim, Jin Xu, Jinyu Li, Carlos Busso, Kai Yu, Eng Siong Chng, Xie Chen
TL;DR
Audio reasoning benchmarks largely emphasize final answers rather than the quality of intermediate reasoning. This paper presents the Interspeech 2026 Audio Reasoning Challenge with MMAR-Rubrics and separate model and agent tracks, finding that agent systems currently lead while single models rapidly advance.
Problem
Existing audio reasoning benchmarks focus primarily on final-answer accuracy, leaving the factuality, logic, and completeness of intermediate reasoning insufficiently evaluated.
Method
The challenge introduces MMAR-Rubrics, an instance-level rubric protocol, and evaluates end-to-end models alongside multimodal, tool-using agents.
Results
Agent systems generally outperform end-to-end systems in reasoning quality and final-answer accuracy; the top agent scored 69.83% on Rubrics versus 65.29% for the top single model.
Takeaways & Limitations
Iterative tool use and cross-modal analysis characterize strong agents, while reinforcement learning and data pipelines are advancing internalized reasoning in single models.
Abstract
from arXiv · showhide
Recent Large Audio Language Models (LALMs) excel in understanding but often lack transparent reasoning. To address this "black-box" limitation, we organized the Audio Reasoning Challenge at Interspeech 2026, the first shared task dedicated to evaluating Chain-of-Thought (CoT) quality in the audio domain. The challenge introduced MMAR-Rubrics, a novel instance-level protocol assessing the factuality and logic of reasoning chains. Featured Single Model and Agent tracks, the competition attracting 156 teams from 18 countries and regions. Results show agent systems currently lead in reasoning quality, utilizing iterative tool orchestration and cross-modal analysis. Besides, single models are rapidly advancing via reinforcement learning and sophisticated data pipeline. We details the challenge design, methodology, and a comprehensive analysis of state-of-the-art systems, providing new insights for explainable audio intelligence.
1. Introduction
The challenge targets a gap in audio reasoning evaluation by shifting attention from final-answer accuracy to the quality of intermediate reasoning. It introduces MMAR-Rubrics and dual tracks for evaluating end-to-end models and tool-using agents.
- Motivation: Existing audio reasoning benchmarks emphasize final-answer accuracy while overlooking whether intermediate reasoning is factual, logical, and complete.This limitation can leave sound reasoning indistinguishable from spurious paths that happen to produce correct answers.
- Challenge Design: The Audio Reasoning Challenge at Interspeech 2026 evaluates Chain-of-Thought quality in audio reasoning systems.The challenge is positioned as a step toward transparent audio intelligence.
- Challenge Design: The challenge separates end-to-end Large Audio Reasoning Models from multimodal, tool-using, self-verification agents.These tracks accommodate different architectural approaches to audio reasoning.
- Evaluation Protocol: MMAR-Rubrics uses instance-level rubric-based evaluation to assess reasoning-path factuality, logic, and completeness more reliably.The protocol is designed to address instability in prior LLM-as-a-judge evaluation methods.
- Participation: 156 registrations came from 18 countries and regions, with 23 Single Model Track and 24 Agent Track teams submitting to the final leaderboard.Participation indicates broad interest in robust audio reasoning capabilities.
- Resources: The challenge releases MMAR-Rubrics data, evaluation scripts, and technical reports to support future research on explainable audio intelligence.These open resources are listed among the challenge’s main contributions.
2. Related Work
Audio reasoning research spans end-to-end models and agents, while benchmarks cover specialized and holistic skills. However, these benchmarks primarily assess final answers, leaving standardized evaluation of reasoning paths underdeveloped.
- System Paradigms: Audio reasoning systems generally use either end-to-end models that internalize reasoning or agents that coordinate specialized audio tools.This distinction structures current approaches to audio reasoning.
- End-to-End Models: Model-based audio reasoning has progressed from prompting explicit Chain-of-Thought to supervised fine-tuning and reinforcement-learning alignment.The research trajectory moves from elicitation toward increasingly internalized reasoning capabilities.
- Audio Agents: Audio agents decompose reasoning into planning, perception, and integration, often bridging audio to text or structured representations.Recent agents also use dynamic refinement and iterative tool-querying loops.
- Benchmark Scope: Benchmarks evaluate specialized skills such as speech deduction, mathematical reasoning, temporal-spatial understanding, and physical sound dynamics.These tasks target reasoning beyond simple auditory perception.
- Benchmark Scope: Other benchmarks assess holistic auditory intelligence across speech, music, and environmental sounds, including taxonomies spanning signal, perception, semantic, and cultural reasoning.Some evaluations include expert-level tasks and dozens of distinct skills.
- Unresolved Gap: Despite their diversity, existing benchmarks predominantly use final-answer accuracy and lack standardized protocols for evaluating reasoning-path factuality, logic, and completeness.This outcome-oriented focus does not reveal whether a correct answer came from sound logic or spurious correlations.
3. Challenge Designs
The challenge formulates audio reasoning as generating an explicit reasoning chain followed by an answer, while requiring correct execution for valid reasoning. It introduces MMAR-Rubrics and two complementary tracks to evaluate reasoning across end-to-end models and tool-using agents.
- Task Formulation: Given an audio signal and language query, systems must generate an explicit reasoning chain followed by a final answer.
- Task Formulation: Correct final answers are a prerequisite for receiving credit for reasoning quality, preventing incorrect conclusions supported by hallucinated reasoning.
- Instance-Level Rubric-Based Evaluation: MMAR-Rubrics decomposes each predicted reasoning path into verifiable atomic criteria derived from human-annotated reasoning, judged as binary outcomes and averaged.
- Instance-Level Rubric-Based Evaluation: Gemini-2.5-Pro generates five instance-specific criteria, while GPT-4o evaluates each criterion and supplies a brief textual justification.
- Instance-Level Rubric-Based Evaluation: MMAR-Rubrics is evaluated against a universal 5-point system-level rubric using reliability and human-alignment comparisons.
- Challenge Tracks: The challenge separates intrinsic single-model reasoning from agent reasoning that orchestrates multiple models and tools under distinct constraints.
4. Challenge Results
The challenge compared intrinsic single-model reasoning with tool-using agent systems under a rubric-based evaluation. Agent systems generally led in reasoning quality, while single-model entries advanced through reinforcement learning and specialized data pipelines.
- Participation and Evaluation: 156 teams registered from 18 countries, with 23 Single Model Track and 24 Agent Track teams submitting valid preliminary results.The preliminary stage used 500 questions, while the final stage used the complete 1,000-question MMAR benchmark.
- Overall Performance: Agent systems generally outperformed end-to-end systems in reasoning quality and final answer accuracy.The comparison was reported on the final leaderboard across both tracks.
- Overall Performance: 69.83% was the top Agent Track Rubrics score, compared with 65.29% for the top Single Model Track entry.The reasoning-quality gap was more pronounced than the final-accuracy gap.
- Single Model Track: The leading single-model team used two-stage reinforcement learning with hard negatives and boundary cases, using semantic similarity of CoT paths as the reward signal.GRPO was used in both reinforcement-learning stages.
- Single Model Track: Single-model teams also explored training-free audio attention scaling and Question-to-Reasoning data synthesis with verification-oriented filtering.The attention modification increased emphasis on audio token spans, while the data pipeline rejected reverse-engineered CoT.
- Agent Track: The leading agent integrated over 40 specialized audio tools and iteratively re-queried audio when confidence was low.Other leading agents used spectrogram analysis with a consistency split or multi-agent debate to support numerical analysis and consensus.
5. Conclusion
The challenge shifts audio-intelligence evaluation from final-answer accuracy toward reasoning-process rigor through MMAR-Rubrics. Its analysis contrasts iterative tool use and visual-spectral bridging in agents with reinforcement learning and data pipelines for internalized reasoning in single models.
- Conclusion: MMAR-Rubrics provides a stable, instance-level metric for quantifying audio Chain-of-Thought quality.The challenge evaluates reasoning-process rigor rather than simple accuracy alone.
- Conclusion: Agent systems demonstrated iterative tool use and visual-spectral bridging, while single models pursued internalized reasoning through reinforcement learning and meticulous data pipelines.The challenge releases related resources to accelerate trustworthy, explainable audio-model development.