PaperlayerView exact revision on arXiv

Source-linked AI summary

AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning

Siminfar Samakoush Galougah, Rishie Raj, Sanjoy Chowdhury, Sayan Nag, Ramani Duraiswami

arXiv:2508.07470v2cs.CV

TL;DR

Prior audio-visual benchmarks focus on final answers without assessing reasoning traces, making it difficult to distinguish genuine comprehension from correct answers supported by flawed reasoning or hallucinations. AURA evaluates reasoning across six fine-grained audio-visual tasks and uses AuraScore to score perceptual grounding and logical validity separately. For most tasks, the best-performing model (Qwen2.5-Omni 7B [Xu et al. 2025b]) usually scores ∼80–90% in accuracy and ∼20–40% in reasoning inference; tempo/AV-synchronization and pitch/timbre tasks score poorly across metrics.

  • Problem

    Prior audio-visual benchmarks focus on final answers without assessing reasoning traces, making it difficult to distinguish genuine comprehension from correct answers supported by flawed reasoning or hallucinations.

    Source: §1 · p. 2

  • Method

    AURA evaluates reasoning across six fine-grained audio-visual tasks and uses AuraScore to score perceptual grounding and logical validity separately.

    Source: §1 · p. 2

  • Results

    For most tasks, the best-performing model (Qwen2.5-Omni 7B [Xu et al. 2025b]) usually scores ∼80–90% in accuracy and ∼20–40% in reasoning inference; tempo/AV-synchronization and pitch/timbre tasks score poorly across metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results show a gap between answer accuracy and reasoning scores, supporting assessment of models beyond simple accuracy.

    Source: §1 · p. 2

    AURA covers six tasks and could expand to more reasoning types and diverse video content; its Factual Consistency assessment relies on proprietary GPT-4o, raising reproducibility and accessibility challenges.

    Source: §6 · p. 7

Abstract

from arXiv · show

Current audio-visual (AV) benchmarks focus on final answer accuracy, overlooking the underlying reasoning process. This makes it difficult to distinguish genuine comprehension from correct answers derived through flawed reasoning or hallucinations. To address this, we introduce AURA (Audio-visual Understanding and Reasoning Assessment), a benchmark for evaluating the cross-modal reasoning capabilities of Audio-Visual Large Language Models (AV-LLMs) and Omni-modal Language Models (OLMs). AURA includes questions across six challenging cognitive domains, such as causality, timbre and pitch, tempo and AV synchronization, unanswerability, implicit distractions, and skill profiling, explicitly designed to be unanswerable from a single modality. This forces models to construct a valid logical path grounded in both audio and video, setting AURA apart from AV datasets that allow uni-modal shortcuts. To assess reasoning traces, we propose a novel metric, AuraScore, which addresses the lack of robust tools for evaluating reasoning fidelity. It decomposes reasoning into two aspects: (i) Factual Consistency - whether reasoning is grounded in perceptual evidence, and (ii) Core Inference - the logical validity of each reasoning step. Evaluations of SOTA models on AURA reveal a critical reasoning gap: although models achieve high accuracy (up to 92% on some tasks), their Factual Consistency and Core Inference scores fall below 45%. This discrepancy highlights that models often arrive at correct answers through flawed logic, underscoring the need for our benchmark and paving the way for more robust multimodal evaluation.

1 Introduction

AURA addresses gaps in prior audio-visual benchmarks by assessing fine-grained attributes and the reasoning behind answers. Its benchmark and AuraScore reveal that high answer accuracy can coexist with weak reasoning scores.

1 Introduction

  • AURA evaluates reasoning traces across six cross-modal tasks focused on advanced, fine-grained audio-visual attributes.The benchmark includes areas such as performer skill profiling, tempo and synchronization, and timbre recognition.Source: §1 · p. 2
  • AuraScore separates reasoning evaluation into perceptual grounding (Factual Consistency) and the logical validity of each step (Core Inference).The metric is presented as a way to scrutinize reasoning faithfulness beyond surface-level accuracy.Source: §1 · p. 2
  • The best-performing model (Qwen2.5-Omni 7B [Xu et al. 2025b] for most tasks) usually scored ∼80–90% in accuracy but ∼20–40% in reasoning inference; tempo/AV-synchronization and pitch/timbre tasks performed poorly across all metrics.The results support assessing models beyond simple accuracy.Source: §1 · p. 2
  • AURA’s automated QA pipeline uses fine-grained audio and video captions to generate complex question-answer pairs and can adapt as captioning models evolve.The paper also compares prior benchmarks by their coverage of cross-modal dependency, reasoning-trace evaluation, and fine-grained audio and visual attributes.Source: §1 · p. 3, Table 1 · p. 3

2 Related Work

Prior multimodal work developed reasoning methods and audio-visual benchmarks, but systematic evaluation of reasoning faithfulness in complex audio-visual tasks remains open.

2 Related Work

  • Chain-of-Thought, Self-Consistency, Tree of Thoughts, and multimodal chain-of-thought extend stepwise or multi-path reasoning, while faithfulness of generated explanations remains a challenge.Prior studies cited in the paper report plausible explanations that may not reflect the model’s inference process.Source: §2 · p. 3
  • Existing audio-visual benchmarks assess modality capabilities, but the paper identifies gaps in evaluating causal reasoning, synchronization, and implicit cross-modal contradictions.The related work also notes prior efforts connecting answer accuracy with clue-based or audio-visual-temporal grounding.Source: §2 · p. 3

3 AURA

AURA comprises more than 1,600 question-answer pairs across six cognitive task categories and uses an automated pipeline to generate and validate them.

3.1 Task Categories and Benchmark Statistics

  • More than 1,600 question-answer pairs span six tasks, covering cross-modal causality, unanswerability, timbre/pitch, tempo/synchronization, performer skill profiling, and implicit distractions.The tasks target distinct aspects of audio-visual understanding, including fine-grained auditory perception and spatial attention.Source: §3.1 · pp. 3–4
  • Figure 2 summarizes question counts by task, video-duration distribution, and sample contributions by source.The accompanying text directs readers to the appendix for further implementation details.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 QA Generation Pipeline

  • The QA pipeline annotates videos with AuroraCap visual captions, FusionAudio audio-event captions, and Whisper transcripts, then prompts GPT-4o with task-specific instructions and examples to generate QA pairs and gold reasoning.Scripts parse the structured output, validate it, and append each question to a JSONL evaluation dataset.Source: §3.2 · p. 4

4 Evaluation and Metrics

AuraScore supplements answer accuracy with separate measures of factual grounding and logical validity, using an LLM-based factual comparison and an NLI-based inference score.

4 Evaluation and Metrics

  • AuraScore reports Answer Accuracy alongside separate scores for factual grounding (FCS) and logical validity (CIS), because correct answers can arise from unfaithful reasoning.FCS checks whether generated reasoning is grounded in the audio-visual context, while CIS isolates logical soundness from factual precision.Source: §4 · p. 4
  • FCS is the fraction of ground-truth reasoning facts correctly and completely represented in the generated reasoning, scored from 0.0 to 1.0.GPT-4o compares factual elements such as subjects, objects, attributes, and locations in the generated and ground-truth traces.Source: §4 · p. 4
  • CIS sanitizes generated and ground-truth traces into abstract logical claims, then scores the entailment probability from the ground-truth claim to the generated claim.A pretrained cross-encoder NLI model supplies the 0.0-to-1.0 score; sanitization removes grounded details already assessed by FCS.Source: §4 · pp. 4–5
  • Together, FCS and CIS assess whether models perceive audio-visual content accurately and reason coherently about it.The two scores distinguish factual alignment from inference ability.Source: §4 · p. 5

5 Experiments, Results and Analysis

The experiments evaluate selected open- and closed-source models with accuracy, factual-consistency, and core-inference measures across AURA’s six tasks. Results show that leaders in answer accuracy do not always lead in reasoning quality.

5.1 Model Selection and Evaluation Scores

  • The evaluation selects open-source and closed-source AV-LLMs and OLMs, extracts selected answers from their raw reasoning outputs, and scores accuracy, factual consistency, and core inference.AuraScore returns zero when the answer is incorrect; otherwise, FCS compares atomic facts and CIS uses NLI-based scoring against ground-truth reasoning.Source: §5.1 · p. 5

5.2 Performance Analysis of the Models

  • Across tasks, leaders in accuracy and reasoning measures can differ, as Table 2 reports ACC, FCS, and CIS and the task results show distinct metric leaders.For example, Qwen2.5-Omni-7B leads accuracy on unanswerability, while Ola leads FCS and CIS; on causal reasoning, Qwen2.5-Omni-7B leads accuracy and CIS, while Reka leads FCS.Source: Table 2 · p. 6, §5.2 · p. 7
  • Gemini 2.0 Flash leads timbre-and-pitch reasoning in FCS (47.5%) and CIS (37.09%), while Qwen2.5-Omni-7B matches its accuracy but scores lower on both reasoning measures.The authors note that correct predictions may therefore arise from shortcut learning or shallow heuristics rather than sound reasoning.Source: §5.2 · p. 6
  • Tempo and synchronization are difficult across models: Gemini 2.0 Flash leads accuracy (55% ACC), but Gemini 2.5 Flash leads FCS (30.95%) and CIS (25.78%).The authors report low absolute scores across metrics in this task.Source: §5.2 · p. 6
  • In performer skill profiling, Gemini 2.5 Flash has the highest accuracy (84.45%) and FCS (52.00%), while Gemini 2.0 Flash leads CIS (33.39%).The task requires integrating subtle visual and auditory cues to distinguish novice from expert performers.Source: §5.2 · p. 6
  • For cross-modal causal reasoning, Qwen2.5-Omni-7B leads accuracy (80%) and CIS (32.3%), while Reka leads FCS (46.34%).The results indicate a trade-off between answer accuracy and reasoning depth in this task.Source: §5.2 · p. 7
  • On implicit distractions, Qwen2.5-Omni-7B leads FCS (67.95%) and CIS (45.23%), while Gemini 2.5 Flash has the highest accuracy (84.55%).The task tests resolving ambiguous referents amid similar distractor objects.Source: §5.2 · p. 7
  • For unanswerability, Qwen2.5-Omni-7B has the highest accuracy (92%), while Ola leads FCS (73.46%) and CIS (61.10%).The task asks models to recognize when multimodal content contradicts a question and justify rejecting it.Source: §5.2 · p. 7

6 Limitations and Future Work

The authors identify limits in both AURA’s task and video coverage and AuraScore’s evaluator. They propose broadening the benchmark and developing more reproducible, accessible evaluators.

6 Limitations and Future Work

  • AURA covers six cognitive tasks, but its scope could expand to additional reasoning types and more diverse video content.The authors suggest long-form narratives and skills such as sports or surgery as areas for assessing generalizability.Source: §6 · p. 7
  • AuraScore’s Factual Consistency assessment depends on proprietary GPT-4o, whose stochasticity, potential biases, and limited accessibility challenge reproducibility and adoption.The authors propose open-source evaluators tailored to assess reasoning-trace faithfulness.Source: §6 · p. 7
  • The authors propose scaling AURA’s automated, modular QA pipeline with additional task categories and videos from varied sources.They identify these expansions as ways to improve model robustness and generalization.Source: §6 · p. 7

7 Conclusion

The conclusion presents AURA as a benchmark for cross-modal AV reasoning across six fine-grained cognitive tasks and AuraScore as a way to evaluate reasoning ability. Together, they provide a foundation for holistic AV evaluation and insights into model behavior.

7 Conclusion

  • AURA enables evaluation of state-of-the-art models’ cross-modal AV reasoning across six fine-grained cognitive tasks.The authors describe the benchmark as a foundation for holistic AV evaluation.Source: §7 · p. 7
  • The benchmark uses an automated approach that the authors say supports dataset scalability.Source: §7 · p. 7
  • AuraScore combines Factual Consistency and Core Inference to evaluate model reasoning and reveal differences in factual alignment and inference ability.Source: §7 · p. 7

A Additional Details on AURA

AURA’s design requires cross-modal reasoning, targets advanced cognitive abilities, and uses a modular automated pipeline for scalable QA generation. Its task categories probe distinct forms of audio-visual reasoning, from causality and acoustic properties to synchronization, skill, and distraction handling.

A.1 Design Principles

  • Each AURA question is designed to be unanswerable from audio or video alone, requiring models to integrate clues from both streams.Source: §A.1 · p. 8
  • The benchmark adds tasks for nuanced abilities such as timbre and pitch discrimination, audio-visual causality, desynchronization detection, and factual-inconsistency identification.Source: §A.1 · p. 8
  • A modular, prompt-driven pipeline uses GPT-4o to generate task-specific QA pairs, supporting consistency, scalability, and distractor creation.Source: §A.1 · p. 8

A.2 Construction of Task Categories

  • Cross-modal causal reasoning asks models to connect audio events to visual outcomes or vice versa; unanswerability asks them to reject false premises contradicted by audio-visual cues.The unanswerability questions place false premises and distracting choices in different modalities and require an explanation of why the question cannot be answered.Source: §A.2 · p. 8
  • Timbre/pitch reasoning requires distinguishing sound sources’ acoustic properties and linking a target sound to a visual attribute, using videos from MUSIC-AVQA.Source: §A.2 · p. 8
  • Tempo/AV synchronization analysis tests whether streams are aligned using both synchronized and deliberately misaligned musical-performance clips.Source: §A.2 · p. 8
  • Performer Skill Profiling asks models to combine visual cues such as posture and finger fluidity with auditory qualities such as rhythmic stability and tonal clarity.The task uses curated novice and expert performance videos sourced under Creative Commons licenses or with creator permission.Source: §A.2 · p. 8
  • Implicit Distractions uses split-screen performances to test whether models answer about a specified target object while ignoring a similar distractor.The task uses duet performances from the MUSIC dataset [Zhao et al. 2018].Source: §A.2 · p. 8

A.3 Benchmark Statistics

AURA contains more than 1,600 QA pairs across six cognitive task categories, with CR and UANS the largest. The benchmark draws on task-specific video sources and uses structured prompts in its QA and reasoning-score pipeline.

A.3 Benchmark Statistics

  • More than 1,600 QA pairs cover six categories, with Cross-Modal Causal Reasoning and Unanswerability each contributing approximately 400 questions.The authors describe the distribution as well-balanced.Source: §A.3 · p. 8
  • FineVideo supplies 45.7% of videos for CR and UANS, while MUSIC-AVQA supplies 34.3% for TPR and TSA.The clips are 20 seconds for CR and 40 seconds for TPR and TSA.Source: §A.3 · p. 9
  • The ID task uses vertically stacked duet clips from The Sound of Pixels, while PSP uses concatenated novice and expert performances from a custom licensed YouTube set.The two sources account for 8.6% and 11.4% of the videos, respectively; PSP clips are 30–40 seconds.Source: §A.3 · p. 9

B GPT/Model prompts

  • Structured GPT-4o prompts with core rules and in-context examples guide QA generation and calculation of Factual Consistency and Core Inference scores.The described examples include an Unanswerability QA-generation prompt and a prompt for factual-consistency scoring.Source: B GPT/Model prompts · p. 9

C Implementation Details

Models process full videos natively for evaluation, without extracting multiple frames. Their raw responses are compared with ground-truth reasoning generated by AURA’s automated QA pipeline.

C Implementation Details

  • Models process full videos natively, without multiple-frame extraction, and their raw responses are compared with ground-truth reasoning from AURA’s automated QA-generation pipeline.Open-source models followed their repository instructions and were evaluated on one RTXA5000 GPU; closed-source models were accessed through official APIs.Source: C Implementation Details · p. 9

D More Qualitative Examples

The qualitative examples show that correct answers can coexist with ungrounded or incomplete reasoning across AURA tasks. They also situate AURA alongside prior work in multimodal learning and AV question answering.

D More Qualitative Examples

  • Among 11 tested models, only Gemini 2.5 Flash got both the Performer Skill Profiling answer and reasoning right; the reported top performers across examples were Gemini 2.0 Flash, Gemini 2.5 Flash, and Qwen 7B.Other models often named the expert performer without explaining why that performer demonstrated greater skill.Source: D More Qualitative Examples · pp. 9–10
  • In the Implicit Distractions example, models often omitted spatial-attention evidence, and some showed weak temporal context when the queried shirt pattern appeared only in the video’s second half.The authors note that correct answers may reflect easier questions or video data rather than spatial grounding.Source: D More Qualitative Examples · p. 10
  • For Unanswerability, most models invented a motorcycle-racing-history topic absent from the speech, while Qwen2.5-Omni 7B and Ola answered correctly but gave absent or incorrect reasoning.The authors describe the false claim as potentially biased by available facts.Source: D More Qualitative Examples · p. 11
  • For a misaligned sample with audio shifted by approximately 5 seconds, almost every model misclassified the audio and visual cues as synchronized.The authors report very low accuracy across models on Tempo/AV-Synchronization Analysis.Source: D More Qualitative Examples · p. 11
  • In the timbre example, the xylophone—not the usually more resonant violin—had the more resonant timbre, and Qwen2.5-Omni models gave the right answer without identifying the performer or instrument in their reasoning.The authors suspect the correct answer may therefore be hallucinated; Gemini 2.5 Flash was the only model that did not fail in some manner in this example.Source: D More Qualitative Examples · p. 12
  • For the chest-press example, most models answered correctly about the person’s focused expression, but only Gemini 2.5 Flash and OneLLM gave the causal reasoning correctly.Other traces described concentration or a workout without explicitly connecting the exercise to the expression.Source: D More Qualitative Examples · pp. 12–13

E.1 Multi-modal Learning

  • The multimodal-learning discussion surveys work on cross-modal generation, audio-visual representation, multimodal LLMs, and cross-modal integration, emphasizing dynamic, embodied perception for general-purpose systems.The cited passage presents this as a collective direction in prior research.Source: §E.1 · pp. 13–14

E.2 Audio-Visual Question Answering

  • Prior AVQA work includes open-ended answering, bias mitigation, object-aware reasoning, MLLM integration, procedural inference, and long-form temporal understanding; the authors describe these directions as complementary to AURA’s structured evaluation.The cited examples include CAT, AutoEval-Video, and ProViQ.Source: §E.2 · p. 14
Loading 2508.07470v2…