PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Yupei Li, Qiyang Sun, Mohamed Mady, Chenxi Wang, Zhengwei Gong, Berrak Sisman, Björn Schller

arXiv:2609.09681v1cs.SD

TL;DR

Accuracy-based benchmarks do not reveal whether LALMs genuinely reason over audio, and text-only evaluation misses perceptual grounding. ARIA-Rubrics uses CoT and six complementary metrics to assess audio reasoning; evaluations identify three reasoning modes and report alignment with human judgments.

  • Problem

    Accuracy-based evaluation cannot distinguish genuine reasoning from statistical pattern exploitation, while text-only metrics miss perceptual grounding in audio reasoning.

    Source: §1 · pp. 1–2

  • Method

    ARIA-Rubrics uses CoT to expose reasoning and six complementary metrics to assess acoustic perception, reasoning progressiveness, and conclusion consistency.

    Source: §1 · p. 2

  • Results

    Across nine models and two benchmarks, ARIA-Rubrics identifies Reasoning, Scaffolding, and Decoration modes and shows strong correlation with human judgments.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ARIA-Rubrics provides a diagnostic tool for assessing LALM reasoning beyond accuracy-based evaluation.

    Source: §5 · p. 9

    The evaluation covers only two closed-source models; broader proprietary-model coverage would provide a more comprehensive view of reasoning modes.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.

1 Introduction

The introduction argues that accuracy-based evaluations cannot establish whether LALMs genuinely reason over audio, motivating process-level assessment that accounts for acoustic grounding. It presents ARIA-Rubrics and reports evaluation across models and benchmarks, with reliability checked against human judgments.

1 Introduction

  • Accuracy and task-completion scores cannot distinguish genuine reasoning from statistical pattern exploitation, despite the need to integrate acoustic modalities into LALM reasoning.Source: §1 · p. 1
  • Process-level evaluation can expose failure modes and guide improvement; human annotation is costly at scale, while LLM judges are opaque and limited in reliability (Shojaee et al., 2026; Yang et al., 2025; Huang et al., 2025; Bi et al., 2026).Source: §1 · pp. 1–2
  • Text-only metrics miss perceptual grounding, so they cannot detect fabricated or inaccurate acoustic observations underlying later reasoning (Golovneva et al., 2023).Source: §1 · p. 2
  • ARIA-Rubrics uses CoT to expose intermediate reasoning and combines six metrics across acoustic perception, reasoning progressiveness, and conclusion consistency.Automatic metrics, lightweight model scoring, and targeted LLM judgments aim to balance transparency with effectiveness.Source: §1 · p. 2
  • Experiments across nine models and two benchmarks identify Reasoning, Scaffolding, and Decoration modes, while human evaluation shows strong correlation with ARIA-Rubrics.Source: §1 · p. 2

2 Related work

Prior LALM benchmarks assess answer correctness but not reasoning quality, while existing audio- and text-reasoning evaluation frameworks leave gaps in scalability, transparency, or acoustic grounding.

2 Related work

  • Audio benchmarks including MMAU, MMAR, AudioBench, MUSE, and AIR-Bench cover diverse tasks but rely on accuracy or task completion rather than reasoning quality.Source: §2 · p. 2
  • Prior audio frameworks trade off scalability and transparency: MMAR-Rubrics relies on proprietary models and human-annotated CoTs, CAFE uses black-box LLM judgments, and OCRA depends on human annotators (Ma et al., 2026a; Mao et al., 2026; Sedláček et al., 2025).Source: §2 · p. 2
  • Text-oriented CoT metrics assess coherence, informativeness, or faithfulness, but lack perceptual grounding needed to detect audio hallucinations (Golovneva et al., 2023).Source: §2 · p. 3

3 ARIA-Rubrics framework

ARIA-Rubrics evaluates observable reasoning chains rather than final answers alone, using structured CoT prompting and six complementary metrics. Their mean score profiles support diagnosis of three reasoning modes.

3.1 Overview; 3.2 CoT prompt design

  • A four-step CoT prompt externalizes Perception, Analysis, Reasoning, and Answer before six metrics score the chain and combine into an overall ARIA score.The prompt design draws on LLM-generated CoT patterns; sanity checks found no huge accuracy difference in most cases with versus without the prompt.Source: §3.1 · p. 3, §3.2 · p. 3

3.3.1 Acoustic Grounding Score (M1)

  • Acoustic Grounding Score (M1) uses CLAP to assess whether perceived sound entities are grounded in the audio, detecting fabricated or misidentified events; no entities yields zero.The metric is intended to prevent models from inflating scores by listing many sound entities.Source: §3.3.1 · pp. 3–4

3.3.2 Inter-step Coherence Score (M2)

  • Inter-step Coherence Score (M2) measures semantic coherence across Perception, Analysis, and Reasoning by comparing consecutive-step embeddings; an absent slot yields zero.The comparisons are Perception–Analysis and Analysis–Reasoning.Source: §3.3.2 · p. 4

3.3.3 Reasoning Chain Coverage (M3)

  • Reasoning Chain Coverage (M3) averages substantiveness scores for Perception, Analysis, and Reasoning with a deterministic Answer-presence score, assigning zero to absent slots.A few-shot Qwen2.5-1.5B-Instruct scorer rates the first three slots from 0 to 1; a non-empty Answer receives 1.0.Source: §3.3.3 · pp. 4–5

3.3.4 Reasoning Step Causality (M4)

  • Reasoning Step Causality (M4) estimates each step’s information gain from reductions in conditional perplexity for the correct answer, then applies a sigmoid transformation.The evaluator is Qwen2.5-1.5B-Instruct, avoiding reliance on additional pretrained models.Source: §3.3.4 · p. 5

3.3.5 Reasoning Progress Score (M5)

  • Reasoning Progress Score (M5) uses NLI entailment scores to assess whether each step follows from the previous one; absent slots receive zero.The classifier is Roberta-large-mnli.Source: §3.3.5 · p. 5

3.3.6 Audio Lexical Density (M6)

  • Audio Lexical Density (M6) averages each reasoning-chain word’s maximum similarity to a 417-word AudioSet ontology, approximating audio relevance and potentially penalizing irrelevant verbosity.The authors describe the metric as an approximation and hypothesize that more audio-related vocabulary correlates with more meaningful inference.Source: §3.3.6 · p. 5

3.4 Overall discussion

  • The six metrics are unweighted-averaged and designed to constrain gaming across dimensions; score profiles distinguish Reasoning, Scaffolding, and Decoration modes.High accuracy with low ARIA indicates Decoration; high ARIA with high accuracy indicates Reasoning, while Scaffolding is associated mainly with low M2, M4, and M5.Source: §3.4 · pp. 5–6

4 Experiments and Results

Across two benchmarks and nine LALMs, experiments test whether ARIA scores align with human judgments, distinguish reasoning modes, and remain consistent across datasets.

4 Experiments and Results

  • ARIA scores diverge from accuracy: AudSemThinker answers correctly despite weak acoustic grounding, while Qwen2.5-Omni reasons well but misses an answer due to a phonetics knowledge gap.These cases illustrate diagnostic information that accuracy-based evaluation misses.Source: §4 · p. 6, Table 3 · p. 8
  • ARIA rankings largely agree with DeepSeek- and Gemini-based judgments while offering more interpretable improvement directions and a more efficient, structured evaluation.The authors contrast ARIA with the computationally and token-intensive LLM-based evaluation.Source: §4 · p. 6, Table 2 · p. 7
  • Model identity explains 91.9–96.4% of score variance, dataset identity at most 1.7%, and Kendall’s W of 0.933–0.992 indicates consistent cross-benchmark rankings.ICC(2,1) also confirms instance-level cross-dataset agreement.Source: Figure 2 · p. 7, §4 · p. 7, Figure 3 · p. 8
  • Perturbation-derived mode assignments align with ARIA rankings: Reasoning models rank highest, Scaffolding models midrange, and Decoration models lowest.Reasoning models resist misleading CoTs and reground answers in audio; Decoration models show no causal effect of CoT on the final answer.Source: §4 · p. 6, Table 4 · p. 8

Component’s (M1-M6) metrics’ correlation

  • M4 and M5 correlate most strongly (r = 0.951), while M2 and M3 also correlate highly (r = 0.916); M6 is negatively correlated with M2 and M3.The negative correlations are r = −0.578 and r = −0.633, respectively; the authors interpret M6 as capturing a complementary dimension.Source: Component’s (M1-M6) metrics’ correlation · pp. 7–8, Figure 4 · p. 8
  • Closed-source models show lower variance across M2, M3, and M6 and achieve the highest ARIA scores on both benchmarks; open-source models show higher variance, especially on M2 and M3.GPT-4o-audio and Gemini 2.5 Flash score 0.5796 on MMAR and 0.5801 on MMAU-mini; Qwen2.5-Omni is closest among open-source models.Source: Component’s (M1-M6) metrics’ correlation · pp. 8–9
  • Open-source models are uniformly weak on M1, Decoration-mode models are weak on M2 and M3, and M5 is lowest across nearly all models.The authors identify perceptual grounding, joint improvement of reasoning substance and coherence, and process-level supervision as directions for improvement.Source: Component’s (M1-M6) metrics’ correlation · p. 9
  • ρ = 0.671 (p = 0.006) for the full ARIA score’s correlation with human evaluations, exceeding every individual metric and tested metric group.The human study used 45 annotators to rate 450 instances, with three independent ratings per instance.Source: Table 5 · p. 9, Component’s (M1-M6) metrics’ correlation · p. 9

5 Conclusion

ARIA-Rubrics evaluates LALM reasoning quality with six complementary metrics, revealing three reasoning modes across nine models and two benchmarks. The authors report validation through CoT perturbation and alignment with human evaluation.

5 Conclusion

  • Six complementary metrics reveal three reasoning modes across nine models on two benchmarks.The authors describe ARIA-Rubrics as a diagnostic tool for moving beyond accuracy-based assessment.Source: §5 · p. 9
  • CoT perturbation and alignment with human evaluation confirm the reported validity of ARIA-Rubrics.Source: §5 · p. 9
  • The authors identify future work in extending evaluation to open-ended tasks and improving LALMs’ perceptual grounding.Source: §5 · p. 9

Limitations

The authors identify limitations in scoring-model capacity, human-evaluation scale, and the breadth of closed-source model coverage.

Limitations

  • Lightweight scoring models in M3–M5 support accessibility and reproducibility, but more capable closed-source models may improve scoring precision.The authors note that funding constraints limit use of those models.Source: Limitations · p. 9
  • Human evaluation covers 90 sampled instances and 9 annotators, and larger-scale validation with more diverse annotators and audio types needs additional resources.Source: Limitations · p. 10
  • Resource constraints limit evaluation to two closed-source models; adding systems such as Gemini 2.0 and Claude would broaden the reasoning-mode landscape assessed.Source: Limitations · p. 10

Ethical considerations

The study reports that its audio data come from publicly available benchmarks and that it collected no personally identifiable information.

Ethical considerations

A Prompt design

The appendix shows the structured CoT prompt template and several prompt variants used in the study. These include changes to reasoning structure, hints, answer content, and perception information.

A.1 CoT prompt to externalization LALMs’ reasoning process

A.2 Other CoT prompt for models to reason tasks

  • The other CoT prompt requires an explicit answer for accuracy while leaving other restrictions open to minimize interference with reasoning.Source: §A.2 · p. 12
  • The NoStruct variant replaces explicit reasoning structure with a sequential instruction to attend to sounds, infer their meaning, and reason toward an answer.Source: §A.2 · p. 13
  • RandHints adds audio-focused hints, with three hints randomly sampled in each experiment.Source: §A.2 · p. 13
  • AnsSwap supplies a structured reasoning chain ending in a randomly swapped wrong answer before asking for a final answer.Source: §A.2 · p. 13
  • The Irrelevant variant provides a generic, ambiguous-sounding reasoning passage before requesting a final answer.Source: §A.2 · p. 13
  • WrongPerc supplies an incorrect perception description of rain, thunder, and wind before asking the model to answer.Source: §A.2 · p. 13

B Additional results

Across MMAU and MMAR, CoT prompting generally matches or improves free-prompt performance, while the appendix also documents the RCC prompt and an LLM-judge rubric. Perturbation results support CoT as an externalization of reasoning for capable models.

B.1 CoT prompt versus free clean prompt

  • CoT prompting generally matches or improves baseline performance on both benchmarks, with clear gains for Qwen2Omni and GPT-4o-audio and no increase in no-answer rate.R1-AQA and AudSemThinker are notable exceptions; their weaker CoT results are discussed in light of answer-only responses and CoT-based fine-tuning, respectively.Source: §B.1 · p. 14
  • Non-adversarial prompt variations produce answer-change rates consistent with baseline performance, while WrongPerception and AnsSwap meaningfully change answers for Reasoning-mode models.The authors interpret these results as evidence that CoT externalizes reasoning rather than merely rationalizing answers post hoc for capable models.Source: §B.1 · p. 14

B.2 RCC score prompt

  • The RCC evaluation prompt is provided in Figure 6.The figure is identified as the prompt template used to evaluate RCC.Source: Figure 6 · p. 14, §B.2 · p. 14

B.1 CoT prompt versus free clean prompt

  • The comparison covers nine models on MMAU and MMAR under CoT and free-prompt settings, reporting accuracy and no-answer rate.Table 7 reports accuracy and no-answer rate for MMAR perturbation settings.Source: §B.1 · p. 14, Table 6 · p. 15, Table 7 · p. 15

B.3 LLM-as-a-judge

  • The LLM-as-a-judge rubric scores correctness, logical consistency, evidence grounding, hallucination, and conciseness, plus an overall judgment from 0 to 1.The judge receives the question, choices, ground-truth answer, chain of thought, and final answer.Source: §B.3 · pp. 14–15

B.4 Model reasoning modes analysis

The appendix probes reasoning modes through a single MMAR sample and examples of model outputs. The examples include unsupported acoustic descriptions and cases where a chain of thought conflicts with its answer.

B.4 Model reasoning modes analysis

  • On one MMAR sample whose ground truth is Parrot, the perturbation experiment compares unstructured decoration CoT with scaffolding CoT that points toward Owl.The prompt asks the model to answer using the audio and the inserted reasoning, and invites step-by-step reasoning.Source: §B.4 · p. 15
  • Audio-Reasoner’s original answer is Rooster, while example continuations give unsupported alternatives, including Dog and Tiger.One continuation describes ambiguous animal and background sounds before answering Dog; another identifies a low-pitched hoot as Tiger.Source: §B.4 · p. 16
  • AudSemThinker’s original response is N/A, while an example continuation argues for Parrot after describing an ambiguous animal-like sound and indoor setting.A separate continuation describes the hooting as characteristic of a Tiger but concludes Parrot.Source: §B.4 · p. 16

[ANSWER]: Owl

  • The cited original answers are Parrot for Gemini 2.5 Flash, Owl for Gemma-3n-E2B, Parrot for GPT-4o-audio, Rooster for Phi-4, Owl for Qwen2-Audio, and Parrot for Qwen2.5-Omni.Source: §B.4 · pp. 16–17, [ANSWER]: Owl · p. 17

B.5 Pre-trained model selection for M3-M5

The appendix tests whether metric rankings depend on scorer backbones, documents the evaluation setup, and provides supporting materials for the ontology, answer extraction, and human rating examples.

B.5 Pre-trained model selection for M3-M5

  • M3 scores show no significant differences across the tested model sizes or Qwen and Gemma architectures, suggesting larger models are unnecessary for this textual comprehension metric.Table 8 compares Qwen2.5-1.5B, Qwen2.5-32B, and Gemma-3n-E2B.Source: §B.5 · p. 17, Table 8 · p. 18
  • M4 rankings remain consistent across scorer backbones, supporting robustness of the perplexity-based information-gain computation to scorer choice.The comparisons use SmolLM2-1.7B and TinyLlama-1.1B alongside the default scorer.Source: §B.5 · p. 17, Table 9 · p. 18
  • M5 rankings remain largely consistent across the tested NLI backbones, and Roberta-large-mnli is retained as default for its stronger reported NLI performance.The alternatives are distilbert-base-uncased-mnli and nli-MiniLM2-L6-H768.Source: §B.5 · p. 17, Table 10 · p. 19

C Filtered AudioSet ontology

  • The filtered AudioSet ontology lists sound labels grouped into music, human sounds, animal sounds, natural sounds, and sounds of things.Examples include instruments, speech and physiological sounds, birds, weather, water, vehicles, and household objects.Source: C Filtered AudioSet ontology · p. 17, §C.1 · p. 17, §C.2 · p. 18

D Hyperparameters

  • The experiments use pretrained Hugging Face checkpoints without training models and run on an A6000 GPU.Other important or non-default settings are listed in Table 11.Source: D Hyperparameters · p. 19, Table 11 · p. 20

E Answer extraction process from LLM generated contents

  • Answer extraction checks the last [ANSWER] tag, applies exact then substring choice matching, and finally maps a standalone choice letter to an index.Responses that fail the stages are marked unanswered and incorrect; a regex also extracts embedded answer letters in longer sentences.Source: E Answer extraction process from LLM… · p. 20

F.1 Participants demographics

  • The human-rating examples use a five-point scale and avoid matching annotator criteria exactly to ARIA metrics; the nine annotators are experienced audio or NLP researchers.One example notes that a chain can have largely correct perception and reasoning yet contradict its final answer.Source: §F.2 · pp. 20–21, §F.1 · p. 20

G M1 rational

The M1 rationale distinguishes accurate audio-entity identification from simply using generic audio vocabulary. Adversarial manipulation results further support M1 as an entity-grounding measure.

G M1 rational

  • Generic audio-word frequency does not consistently track M1 scores, and such terms can still name valid entities, so M1 is not primarily driven by vocabulary frequency.M1 requires accurate identification of audio entities.Source: G M1 rational · p. 21, Table 12 · p. 21
  • Superficial entity manipulation cannot easily produce a high M1 score; higher scores still require more accurate audio-entity identification.The authors use this result to support M1's validity as an audio entity-grounding measure.Source: G M1 rational · p. 22, Table 13 · p. 22

Related papers

Loading 2609.09681v1…