PaperlayerView exact revision on arXiv

Source-linked AI summary

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi, Ziqi Ye, Xiaorong Guo, Peixuan Li, Zhuoran Sun, Ziyang Gong, Peiyuan Zhang, Mingxin Liu, Shuran Ma, Hongbo Liu, Xiaomin Yu, Yu Tian, Wentao Wang, Zhenjie Yang, Jian Hu, Yunhang Shen, Haoyu Cao, Xue Yang

arXiv:2610.28444v2cs.CV

TL;DR

Answer-only audio-visual benchmarks cannot establish whether correct answers use evidence from both modalities, and text or unimodal shortcuts can undermine their reliability. EvaOmni evaluates answers together with supporting evidence intervals across a human-annotated benchmark. Its evaluation finds that answer accuracy can exceed grounded performance, including for the strongest reported model.

  • Problem

    Answer-only MCQ evaluations cannot distinguish evidence-supported answers from lucky guesses or textual and unimodal shortcuts, limiting assessment of joint audio-visual grounding.

    Source: §1 · pp. 2, 4

  • Method

    EvaOmni pairs human-annotated audio-visual questions with required supporting evidence intervals and evaluates answer correctness alongside evidence coverage and temporal localization.

    Source: §1 · pp. 2–3, §3.5 · p. 7

  • Results

    79.4% accuracy, 70.8% C.G., and 35.1% B.G. were achieved overall by Gemini3.7-Flash, indicating weaker grounded performance than answer accuracy.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    EvaOmni’s findings underscore the value of evaluating whether models localize supporting evidence, not only whether they answer correctly.

    Source: §1 · pp. 3–4

    The 1,027 questions come from 392 videos lasting 1–12 minutes, so results do not establish generalization to substantially longer videos, subtitle-rich content, sparse-event scenes, or all linguistic and cultural contexts.

    Source: §E.1 · p. 34

Abstract

from arXiv · show

Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.

1 Introduction

EvaOmni addresses weaknesses in answer-only audio-visual benchmarks with an evidence-grounded benchmark and evaluation protocol. Its evaluation finds that strong answer accuracy can coexist with substantially weaker evidence grounding.

1 Introduction

  • Answer-only multiple-choice scoring cannot distinguish evidence-supported answers from lucky guesses or answers reached through textual or unimodal shortcuts.Questions and answer options may allow answers based on common knowledge, audio alone, or vision alone.Source: §1 · pp. 2, 4
  • EvaOmni contains 392 videos and 1,027 human-annotated question-answer pairs covering 15 capabilities across three levels of reasoning and evidence complexity.The levels progress from locating evidence, to combining complementary cues, to complex evidence distributions or multi-step cross-modal reasoning.Source: §1 · pp. 2, 4
  • EvaOmni requires models to return audio and visual evidence intervals alongside answers, using expert annotation, cross-checking, and model-assisted filtering to support question quality.These procedures are designed to reduce annotation errors and text or unimodal shortcuts.Source: §1 · p. 2, Figure 2 · p. 3
  • Coverage Grounded Score measures identified annotated evidence, while Bottleneck Grounded Score penalizes inaccurate temporal localization in either modality; both incorporate answer correctness.The evaluation covers seven open-weight and 17 proprietary Omni LLMs.Source: §1 · pp. 3–4
  • 79.4% accuracy, 70.8% C.G., and 35.1% B.G. were achieved overall by Gemini3.7-Flash, showing a gap between correct answers and fully grounded performance.The introduction also reports that human experts achieve 94.3% B.G., compared with 35.1% for Gemini3.7-Flash.Source: §1 · pp. 3–4

2 Related Work

Prior work includes Omni LLM systems and benchmarks for joint cross-modal reasoning, but existing evaluation leaves important questions about evidence grounding and objectivity unresolved.

2 Related Work

  • Omni LLM work spans shared multimodal representations, LLM-connected representations, omni-modal generation, streaming speech, and full-duplex audio-video-text interaction.The passage identifies ImageBind, PandaGPT, NExT-GPT, GPT-4o, Gemini, Qwen, and SeedRealtime as examples.Source: §2 · p. 4
  • DailyOmni covers everyday scenarios, WorldSense emphasizes audio-visual understanding in real-world videos, and OmniVideoBench requires reasoning over both modalities.OmniBench uses static images, while SocialOmni focuses on social interaction rather than video question answering.Source: §2 · p. 4
  • Existing benchmarks do not jointly score answer correctness and localization of supporting audio-visual evidence, while answer-only MCQ scores can credit unsupported correct choices.Open-ended evaluation is also described as insufficiently objective and convenient because it relies on judge models.Source: §2 · p. 4

3 EvaOmni: A New Evidence-Driven Paradigm for Omni-Modal Evaluation

EvaOmni pairs answer scoring with audio-visual evidence annotation and localization, addressing the inability of answer-only MCQ scores to establish joint grounding.

3.1 Rethinking Audio-Visual Reasoning Benchmarks

  • Answer-only MCQ scoring cannot establish joint audio-visual grounding; Table 1 compares benchmark designs by shortcut checks, evidence annotation, grounded evaluation, and task taxonomies.Source: §3.1 · p. 5, Table 1 · p. 5

3.2 Introducing EvaOmni

  • EvaOmni requires experts to mark supporting audio and visual intervals and models to return timestamped evidence with answers; its capabilities span retrieval and grounding, basic understanding, and complex reasoning.Source: §3.2 · pp. 5–6

3.3 Data Collection

  • The benchmark uses recent 1–12-minute videos from open platforms, spanning real-world scenarios and excluding hard-coded subtitles or prominent textual overlays to limit OCR shortcuts.Source: §3.3 · p. 6

3.4 Annotation Pipeline

  • Experts create joint audio-visual questions and mark evidence; human cross-validation and model-assisted text-only, audio-only, and visual-only screening identify and revise potential shortcuts.Disputed pairs receive independent review by two additional experts and are discarded unless both approve them.Source: §3.4 · p. 6

3.5 Evaluation Metrics

  • C.G. weights averaged audio and visual evidence-hit rates by answer correctness, counting annotated segments with any positive-duration overlap.B.G. weights answer correctness by the lower audio/visual temporal IoU, so strong localization in one modality cannot compensate for weak localization in the other; overly broad intervals are not directly penalized by C.G.Source: §3.5 · pp. 6–7

4 Experiments

The experiments evaluate 24 Omni LLMs on EvaOmni and compare answer accuracy with evidence coverage and temporal grounding. Results show substantial grounding gaps, especially on complex tasks, and modality ablations and error analyses further characterize those gaps.

4.1 Evaluated Models

  • The evaluation covers 24 models—7 open-weight and 17 proprietary—on 1,027 questions from 392 videos; input frames are sampled at 1.0 FPS and resized to 480 pixels on the longer side.Open-weight models outside the MiMo family use one eight-H20-GPU server; MiMo and proprietary models use official provider APIs.Source: §4.1 · p. 7, §4.2 · p. 7

4.3 Main Findings

  • Gemini3.7-Flash scores 79.4% Acc., 70.8% C.G., and 35.1% B.G., versus MiMo-V2.5-310B-A15B’s 56.8%, 48.4%, and 20.5%, respectively.Human experts score 96.8% Acc., 96.0% C.G., and 94.3% B.G.; Gemini3.7-Flash trails by 17.4, 25.2, and 59.2 percentage points, respectively.Source: §4.3 · pp. 7, 9, Table 2 · p. 8
  • Level 3 produces the lowest C.G. for all 24 models and the lowest B.G. for all 24, while accuracy does not follow the same decline.Gemini3.7-Flash rises from 73.6% to 79.1% accuracy from Level 2 to Level 3, while C.G. falls from 65.3% to 49.1% and B.G. from 29.2% to 12.3%.Source: §4.3 · pp. 9–10
  • Doubao-Seed-2.1-Lite has the best cost efficiency across Acc., C.G., and B.G., while Gemini3.7-Flash has the highest Overall scores among the selected models.The reported costs per normalized score are $0.0188, $0.0208, and $0.0491 for Doubao, versus $0.0545, $0.0611, and $0.1232 for Gemini3.7-Flash.Source: Table 3 · p. 9, §4.3 · p. 10

4.4 Further Analysis

  • Native audio outperforms ASR in the evaluated MiMo-V2.5 settings, and removing either audio or video substantially reduces its answer and grounding scores.With video, Audio beats ASR by 8.1, 10.3, and 8.9 percentage points in Acc., C.G., and B.G.; either modality alone yields 0.0% B.G.Source: §4.4 · p. 10, Table 4 · p. 10
  • Adding ASR to video improves Qwen3VL-8B and Qwen3VL-A3B accuracy and C.G., but their B.G. remains 1.9% and 0.8%, respectively.Under Video+ASR, MiMo-V2.5 reaches 11.6% B.G.; the authors note transcripts do not ensure effective evidence integration and precise localization.Source: §4.4 · pp. 10–11
  • Answer correctness and dual-modal grounding fail asymmetrically: 48.1%–79.9% of correct answers fail the both-IoUs-at-least-0.5 criterion, while 12.2%–32.6% of grounding-passing predictions are wrong.For Gemini3.7-Flash, 79.4% of answers are correct but only 41.8% meet both grounding criteria.Source: Table 5 · p. 10, §4.4 · p. 11

5 Conclusion

EvaOmni combines answer correctness with evidence localization to evaluate audio-visual reasoning. The evaluation identifies precise temporal grounding as a particularly clear remaining capability gap.

5 Conclusion

  • EvaOmni requires models to identify supporting evidence in addition to selecting the correct answer, and its Grounded Scores assess evidence localization.Coverage Grounded Score rewards evidence coverage, while Bottleneck Grounded Score prevents strong localization in one modality from masking failure in the other.Source: §5 · p. 11
  • The benchmark contains 1,027 questions across three hierarchical levels and is built through annotation and quality-control procedures.The authors report releasing the data, annotations, and evaluation framework.Source: §5 · p. 11
  • Large-scale evaluation identifies precise temporal grounding as the capability current Omni LLMs most clearly lack.The authors describe EvaOmni as a reproducible, interpretable basis for measuring progress.Source: §5 · p. 11

A Dataset Construction and Annotation Details

The dataset construction process filters videos for suitable audio-visual evidence and distractor design, then uses expert annotation, human review, and model-assisted shortcut screening. EvaOmni’s questions are organized across three levels and fine-grained capabilities.

A.1 Video Collection and Filtering

  • Videos are selected to reduce potential contamination and textual shortcuts, retain sufficient audio-visual cues, and support plausible distractors.The collection is limited to 1–12-minute videos; videos with hard-coded subtitles or inadequate event diversity are excluded.Source: §A.1 · pp. 14–15

A.2 Annotation Principles and Schema

  • Five expert annotators separately mark timestamped audio and visual evidence, and Level 3 multi-hop questions also receive step-by-step reasoning chains.Questions must require joint audio-visual use, with distractors based on cues present in the video and plausible alternative explanations.Source: §A.2 · p. 15

A.3 Human Cross-Validation

  • Human cross-validation checks capability alignment, shortcut susceptibility, and evidence quality; approximately 82% of initially submitted pairs pass overall.Disputed pairs are reviewed by two additional experts and discarded unless both approve them.Source: §A.3 · p. 15

A.4 Model-Assisted Shortcut Screening

  • After human validation, text-only, audio-only, and vision-only model checks flag questions that can be answered with plausible rationales without all modalities; flagged pairs are revised and screened again.The resulting benchmark contains 1,027 curated question-answer pairs.Source: §A.4 · pp. 15–16

A.5 EvaOmni Structure

  • EvaOmni contains 392 videos and 1,027 questions across three levels, with the supplied capability descriptions including temporal alignment, event binding, and entity binding.The supplied Level 3 definitions include long-range reasoning over evidence at least four minutes apart and multi-evidence chains of at least five temporally separated pieces.Source: §A.5 · p. 16
  • The supplied capability definitions cover causal reasoning (127 questions) about event causes, emotional intelligence (87), intent understanding (56), and atmosphere understanding (28), with the latter three inferred from audio-visual cues.Level 2 contains 438 questions, and Level 3 contains 153 questions.Source: §A.5 · p. 16

B Experimental Details

Model inputs are standardized by frame sampling and resizing, while inference hardware and access routes vary by model family. Human experts provide a reference baseline under a viewing protocol that differs from model inference.

B.1 Input Preprocessing and Inference Setup

  • Videos are resized to 480 pixels on the longer side and sampled uniformly at 1.0 FPS, with corresponding audio supplied to models.The 1–12-minute duration range keeps these inputs within the evaluated models’ context windows.Source: §B.1 · p. 16
  • Most open-weight models run on a single eight-H20-GPU server, whereas MiMo and proprietary models are evaluated through official APIs.The setup uses model-card-recommended frameworks and parameter settings for open-weight models outside the MiMo family.Source: §B.1 · p. 16

B.2 Human Expert Baseline

  • Independent human participants answer the same questions and report supporting audio and visual intervals, but may pause and replay videos.The authors treat human scores as a reference baseline rather than a strictly matched comparison because replay changes the viewing protocol.Source: §B.2 · p. 17

C Prompt Templates

D.1 Shortcut Evaluation and Case Studies

The shortcut evaluation tests whether questions can be answered repeatedly without full audio-visual input. The results are diagnostic of potential vulnerability, not direct estimates of flawed questions or benchmark exploitability.

D.1 Shortcut Evaluation and Case Studies

  • Four input conditions use two models each, and only questions answered correctly across all four runs count in the Intersection measure.The conditions are video-only, audio-only, text-only, and audio-video; the repeated-run intersection aims to reduce chance successes and individual-model biases, but does not eliminate them.Source: §D.1 · p. 17
  • Repeatedly correct answers under unimodal or text-only inputs indicate potential answer-only evaluation vulnerability, but do not establish a flawed question or shortcut exploitation.The reported rates are not the proportion of flawed questions or the probability that a benchmark can be exploited.Source: §D.1 · p. 17
  • Shortcut examples cover two text-only, two visual-only, and two audio-only cases from each of WorldSense, OmniVideoBench, and Daily-Omni.The appendix provides prompt examples for audio-video, ASR-only, video-plus-ASR, and single-modality or text-only inputs.Source: §D.1 · p. 17, Figure 7 · p. 18, Figure 8 · p. 18

D.2 Additional Grounded Evaluation Results

Additional metrics combine answer correctness with average audio-visual localization, complementing the main grounded scores. The results show declining performance at harder levels and sensitivity to stricter localization thresholds.

D.2 Additional Grounded Evaluation Results

  • The evaluation defines the average of audio and visual temporal IoUs per sample, then reports set-level results as percentages.The supplied passage also introduces the indicator function used in the set-level definition.Source: §D.2 · p. 31
  • Mean continuously combines answer score and average evidence localization, while Thr@τ counts all samples that are correct and meet the average-IoU threshold.Unlike B.G., average-IoU thresholds permit stronger localization in one modality to compensate for weaker localization in the other.Source: §D.2 · p. 31
  • Gemini3.7-Flash leads overall Mean at 43.7%, while MiMo-V2.5 leads the updated open-weight models at 26.3%, a 17.4-point gap.Mean combines answer score with average audio and visual IoU; unlike B.G., average IoU allows stronger localization in one modality to offset weaker localization in the other.Source: §D.2 · p. 31
  • Mean falls from Level 1 to Level 2 to Level 3 for all 20 models; Gemini3.7-Flash scores 57.0%, 37.6%, and 23.5%.Because these scores also incorporate answer correctness, the decline cannot be attributed entirely to localization errors.Source: §D.2 · p. 31
  • Gemini3.7-Flash scores 63.7%, 49.9%, and 26.8% at average-IoU thresholds 0.3, 0.5, and 0.7, respectively.Its overall Mean lead does not hold at every threshold: Gemini3.8-Flash scores 27.8% versus 26.8% at Thr@0.7.Source: §D.2 · p. 31

D.3 Details of Failure Mode Analysis

Failure-mode analyses separate answer selection from dual-modal localization and examine how results vary by modality, threshold, and capability. Even strong answer accuracy can coincide with weak evidence grounding.

D.3 Details of Failure Mode Analysis

  • The seven-model analysis matches available annotations by question ID and verifies each record’s video ID, question, options, level, and reference answer.For scoring, temporal IoU is computed between unions of predicted and annotated intervals; failed, malformed, or truncated outputs receive zero scores and remain in the denominator.Source: §D.3 · pp. 31–32
  • Dual-modal grounding at τ requires both audio and visual IoU to meet the threshold, and Table 5 reports four outcomes as percentages of all questions.Localization scores are computed separately from answer correctness.Source: §D.3 · p. 32
  • Six of seven models have higher mean audio than visual IoU; Qwen3.8-Omni-Flash shows the reverse, with 27.4% audio and 34.3% visual IoU.Among the six models, audio-versus-visual differences range from 0.7 percentage points for Qwen3.5-Omni-Plus to 3.7 for Gemini3.5-Flash.Source: §D.3 · p. 32
  • Across seven models, 47.8%–79.8% of correct answers fail dual-modal localization at τ = 0.5.For Gemini3.7-Flash, 47.8% of correct answers fail grounding, while 12.2% of grounding-passing predictions answer incorrectly.Source: §D.3 · p. 32
  • For Gemini3.7-Flash, correct answers failing the dual-modal grounding threshold rise from 32.9% at τ = 0.3 to 78.2% at τ = 0.7.The corresponding proportion at τ = 0.5 is 47.8%; these threshold-dependent proportions are not direct evidence that answers were guessed.Source: §D.3 · p. 32

D.4 Analysis Across Fine-Grained Capabilities

  • On multi-hop reasoning, Gemini3.7-Flash reaches 80.7% accuracy but 10.6% B.G.; on multi-evidence chains, it reaches 82.9% accuracy but 15.6% B.G.Across three Level 1 alignment tasks, it achieves 82.3%–88.8% accuracy and 47.5%–49.9% B.G.; Figure 31 compares both models across 15 capabilities.Source: §D.4 · p. 33, Figure 31 · p. 33

D.5 More Results

The appendix tests evidence hints and open-ended judging as additional evaluation settings. Hints improve answer accuracy in two models, while judge choice affects absolute open-ended scores.

D.5 More Results

  • Hints raise MiMoV2.5 accuracy from 56.8% to 81.6% and Qwen3.8Omni-Flash accuracy from 58.2% to 85.1%.The hint provides annotated evidence time segments and descriptions alongside the audio/video and question options, but does not enable either model to answer every question correctly.Source: §D.5 · p. 33, Table 8 · p. 34
  • DeepSeekV4.1-flash-1017 yields higher Acc., C.G., and B.G. than Qwen3.8Flash for both evaluated models in the open-ended comparison.The Qwen3.8Omni-Flash accuracy advantage over MiMoV2.5 remains under both judges, with gaps of 3.9 and 6.1 percentage points.Source: §D.5 · p. 33, Table 9 · p. 34
  • Judge selection changes absolute open-ended scores, especially answer accuracy, while model rankings remain stable for accuracy and C.G. but differ for B.G.EvaOmni combines MCQ and evidence supervision to balance evaluation objectivity and interpretability.Source: §D.5 · p. 33

E.1 Limitations

EvaOmni’s dataset scope constrains which settings its results cover; future extensions propose broader video, scene, language, and task coverage.

E.1 Limitations

  • EvaOmni’s 392 videos span 1–12 minutes, so results do not establish generalization to substantially longer videos or other excluded content and contexts.The exclusions include subtitle-rich content, sparse-event scenes, and all linguistic and cultural contexts.Source: §E.1 · p. 34
  • The three capability levels describe task requirements, but do not guarantee increasing answer accuracy across different question sets.Source: §E.1 · p. 34

E.2 Future Work

  • Future extensions propose longer videos, broader linguistic and cultural contexts, sparser audio-visual cues, subtitle-rich content, and open-ended question answering.The proposals retain checks for both modalities and would test findings without answer-option cues.Source: §E.2 · p. 34

Related papers

Loading 2610.28444v2…