PaperlayerView exact revision on arXiv

Source-linked AI summary

JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation

Jianghan Chao, Jianzhang Gao, Wenhui Tan, Yuchong Sun, Ruihua Song, Liyun Ru

arXiv:2512.12772v2cs.MMcs.CV

TL;DR

Existing benchmarks do not consistently provide strict audio-visual correlation, diverse audio types, and multi-scene reasoning coverage. JointAVBench combines these dimensions in a benchmark with a semi-automated question-generation pipeline; Omni-LLMs outperform uni-modal models, but the best overall accuracy is 65.3%.

  • Problem

    Existing benchmarks have gaps in strict audio-visual correlation, audio-type coverage, or multi-scene reasoning, limiting comprehensive evaluation of joint audio-visual understanding.

    Source: §1 · p. 1

  • Method

    JointAVBench uses a taxonomy of cognitive dimensions, audio types, and scene spans, with a semi-automated pipeline and human verification to create questions requiring integrated visual and audio understanding.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    65.3% average accuracy was achieved by the best-performing model, and Omni-LLMs consistently outperformed Video-LLMs and Audio-LLMs.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    The results point to a need for more powerful omni-modal models with enhanced audio-visual fusion capabilities.

    Source: §5 · p. 10

    The dataset is derived exclusively from SF20K, which may bias its data distribution.

    Source: D LIMITATIONS · p. 20

Abstract

from arXiv · show

Understanding videos inherently requires reasoning over both visual and auditory information. To properly evaluate Omni-Large Language Models (Omni-LLMs), which are capable of processing multi-modal information including vision and audio, an effective benchmark must comprehensively cover three key aspects: (1) multi-modal dependency (i.e., questions that cannot be answered using vision or audio alone), (2) diverse audio information types (e.g., speech, sound events), and (3) varying scene spans. However, existing datasets fall short in one or more of these dimensions, limiting strict and comprehensive evaluation. To address this gap, we introduce JointAVBench, a novel benchmark with strict audio-video correlation, spanning five cognitive dimensions, four audio information types (speech, sound events, music, vocal traits), and three scene spans (single-, cross-, and full-scene). Given the high cost of manual annotation, we propose an automated pipeline that leverages state-of-the-art vision-LLMs, audio-LLMs, and general-purpose LLMs to synthesize questions and answers that strictly require joint audio-visual understanding. We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset. Results show that even the best-performing Omni-LLM achieves an average accuracy of only 65.3\%, outperforming uni-modal baselines but revealing substantial room for improvement, especially in cross-scene reasoning.

1 INTRODUCTION

JointAVBench addresses gaps in strict audio-visual correlation, audio-type coverage, and multi-scene reasoning with a benchmark spanning cognitive dimensions, audio types, and scene spans. Its semi-automated pipeline produces questions requiring joint audio-visual reasoning, and evaluations show current Omni-LLMs outperform uni-modal counterparts while remaining uneven across tasks.

1 INTRODUCTION

  • Existing benchmarks often lack strict audio-visual correlation, diverse audio coverage, or complex multi-scene reasoning, despite Omni-LLMs targeting joint audio-visual input.Joint reasoning across scenes combines visual and auditory cues from distinct temporal and spatial contexts.Source: §1 · p. 1
  • JointAVBench organizes 15 strictly audio-visual tasks across five cognitive dimensions, four audio types, and single-, cross-, and full-scene spans.The dimensions include temporal, plot, and long-form reasoning; audio types include speech, sound events, music, and vocal traits.Source: §1 · p. 2
  • A three-stage semi-automated pipeline generates multimodal captions, synthesizes questions requiring joint reasoning, and checks quality before human filtering.The resulting multiple-choice benchmark contains 2,853 samples.Source: §1 · p. 2
  • Omni-LLMs outperform Video-LLMs and Audio-LLMs, but model capabilities vary across audio types and decline as scene complexity increases.The evaluation also identifies substantial room for improvement in audio-visual reasoning.Source: §1 · p. 2
  • The paper contributes a systematic benchmark, a question-generation pipeline, and an evaluation of current models’ joint audio-visual reasoning limitations.These contributions are presented as supporting development of integrated audio-visual reasoning models.Source: §1 · p. 3

2 RELATED WORKS

Prior multimodal work expanded from visual-text and audio-text integration toward video modeling and joint audio-visual reasoning. Existing audio-visual benchmarks vary in audio coverage, video inclusion, and strict modality correlation; JointAVBench targets diverse audio and multilevel scenes.

2.1 MULTIMODAL LARGE LANGUAGE MODELS

  • Multimodal research progressed from visual-text and audio-text fusion to video-text modeling and efforts to interleave audio, video, and text for joint reasoning.The passage frames strict audio-visual correlations in question-answer datasets as necessary for evaluating this joint reasoning.Source: §2.1 · p. 3

2.2 AUDIO-VISUAL BENCHMARKS

  • Earlier benchmarks limited audio types or lacked strict audio-visual correlation, while others used mostly images and audio; WorldSense lacks strict correlation and emphasizes visual tasks.JointAVBench instead targets diverse audio signal types and multilevel scenes; Table 2 identifies its task categories and audio-type abbreviations.Source: §2.2 · p. 4, Table 2 · p. 4

3 JOINTAVBENCH

JointAVBench is designed to evaluate joint audio-visual reasoning through tasks requiring both modalities, spanning cognitive abilities, audio types, and scene complexity. Its semi-automated construction pipeline combines captioning, QA generation, verification, and human review to produce a 2,853-question benchmark.

3.1 BENCHMARK REQUIREMENTS

  • JointAVBench defines 15 tasks requiring integration of audio and video, organized across five cognitive dimensions, four audio types, and three scene complexities.The dimensions include temporal, spatial, emotional, plot, and long-form reasoning; audio types include speech, vocal traits, sound events, and music.Source: §3.1 · pp. 4–5

3.2 BENCHMARK CONSTRUCTION

  • The three-stage construction pipeline generates omni-modal captions, creates QA pairs, and applies quality control.Scene identification groups semantically consistent scenes before visual and audio caption generation.Source: Figure 2 · p. 5, §3.2 · p. 5, §3.2.1 · p. 5

3.2.2 STAGE 2: QA PAIR CREATION

  • QA generation uses task-specific templates where needed and supplies only the modality and scene descriptions required for each task.This is intended to avoid interference from extraneous modalities and scenes.Source: §3.2.2 · p. 6

3.2.3 STAGE 3: QUALITY CONTROL

  • Quality control checks whether each QA pair requires both modalities and whether its answer follows from the question’s context.Task-specific checks assess sequence order, ambiguity, and whether sound-event or music information can be inferred visually.Source: §3.2.3 · p. 6
  • The pipeline creates three plausible but incorrect distractors for each verified QA pair.Distractors can misdirect by replacing a sound source or confusing details.Source: §3.2.3 · p. 6

3.3 HUMAN VERIFICATION

  • Human review retained 2,853 of 3,974 generated questions, a 71.8% retention rate, followed by answer-label refinement without changing benchmark size.Annotators rated answer correctness, information correctness, audio-visual dependency, and question difficulty.Source: §3.3 · pp. 6–7

3.4 BENCHMARK STATISTICS

  • 2,853 manually verified multiple-choice questions span all scene levels and audio types, with an average video duration of 97.2 seconds.Single-scene, multi-scene, and full-scene tasks mainly use videos under 1 minute, 1–10 minutes, and over 10 minutes, respectively.Source: §3.4 · p. 7

4 EXPERIMENT

Experiments compare Omni-, Video-, and Audio-LLMs under accuracy-based, standardized evaluation, finding that Omni-LLMs lead overall but remain uneven across audio types and scene complexity. Analyses further identify weaknesses in emotion, spatial, and increasingly complex multi-scene reasoning.

4.1 EXPERIMENT SETUP

  • Models are evaluated by accuracy across Omni-, Video-, and Audio-LLM groups, with open-source models generally selected near 7B parameters and all models given 32 frames.Open-source and closed-source models use official code or APIs with default configurations; inputs contain only the question text, and deterministic inference yields stable results across runs.Source: §4.1 · p. 8

4.2 RESULTS AND FINDINGS

  • 65.3% average accuracy is the best-performing model’s result, while Omni-LLMs consistently outperform Video-LLMs and Audio-LLMs overall.Gemini2.5-Pro is the overall leader; Qwen2.5-Omni remains competitive with strong open-source video and audio baselines across most tasks.Source: §4.2 · p. 8
  • Models perform better on sound-event and music tasks than on speech and vocal-trait tasks, with SPL, SPER, and MPO among the worst-performing tasks.The authors suggest stronger visual correspondence may help sound-event and music tasks, while limited training data on vocal traits may contribute to the weaker results.Source: §4.2 · p. 8, Figure 4 · p. 9
  • Multi-scene tasks usually score below single-scene tasks, while full-scene tasks often do better; Figure 5 presents results across scene types.Speech-based single-scene tasks such as STL and VSSR perform well, whereas multi-scene speech tasks such as MPO and PTG fare worse; full-scene tasks focus on global narratives.Source: §4.2 · p. 8, Figure 5 · p. 9
  • Omni-LLMs generally lead in 11 of 15 tasks but trail single-modality models on emotion tasks and Video-LLMs on spatial tasks.The authors attribute the spatial weakness to reliance on video spatial information without effective integration of complementary audio cues.Source: §4.2 · p. 8, Figure 6 · p. 9
  • Accuracy on multi-scene tasks drops by approximately 20% when scene counts rise from 0–20 to over 60 scenes.The authors identify increasing scene number as a performance challenge for current MLLMs.Source: Figure 7 · p. 9, §4.2 · p. 9
  • For most models, tasks where audio-visual performance exceeds the best single-modality score outnumber tasks where it falls below the worst single-modality score.The authors report that the former count rises and the latter falls with overall model performance; Qwen2.5-Omni’s dual-modality results significantly surpass its single-modality results.Source: §4.2 · pp. 9–10, Table 4 · p. 10

5 CONCLUSION

The conclusion presents JointAVBench as a benchmark for joint audiovisual reasoning and reports that even its best-performing models reach only 65.3% accuracy.

5 CONCLUSION

  • JointAVBench evaluates joint audiovisual reasoning with a hierarchical taxonomy and an automated generation pipeline, and its questions require visual information and a specific audio type.Human verification is also used to ensure benchmark quality.Source: §5 · p. 10
  • Even the best-performing models achieve only 65.3% accuracy, and the authors call for more powerful omni-modal models with enhanced audio-visual fusion.Source: §5 · p. 10

A.1 TASK DEFINITION

The appendix defines JointAVBench’s task taxonomy and describes dataset statistics, human review, and a post-verification answer-label audit.

A.1 TASK DEFINITION

  • The benchmark defines 15 tasks, with question templates designed to capture critical visual cues and temporal relationships in single- and multi-scene tasks.Table 5 is identified as the source of detailed task descriptions and examples.Source: §A.1 · p. 15, Table 5 · p. 15

A.2 MORE STATISTICS

  • The appendix reports QA-pair distributions, question-length characteristics, and a comparison indicating more samples per task than WorldSense.It also reports frequent visual descriptors and auditory terms in captions and QA pairs; the provided passage does not specify the numerical counts.Source: §A.2 · pp. 15–16, Figure 8 · p. 16

A.3 MANUAL CHECK

  • Annotators rate generated MCQs and correct cases where the designated answer appears among the distractors, discarding 17.4% of data for incorrect answers.Other data are filtered using three additional judging criteria, leaving 2,853 MCQs.Source: §A.3 · pp. 16–17

A.4 POST-AUDIT ANSWER REFINEMENT

  • A post-audit uses an external LLM’s answer and the human-verified answer in majority voting to refine likely label inconsistencies without changing benchmark size.When no majority agreement is reached, the original human-verified label is retained; reported experiments are recomputed using audited labels.Source: §A.4 · p. 17

A.5 ANNOTATION INFORMATION

  • The annotation team consists of professional crowdsourced annotators with university degrees, English proficiency, and qualification-test experience.Source: §A.5 · p. 17

A.6 EXPERIMENT DETAILS

The appendix details experimental settings, benchmark examples, multimodal caption generation, QA construction, quality checks, and distractor generation.

A.6 EXPERIMENT DETAILS

  • Reported results use the audited benchmark; open-source models are generally selected at 7B parameters and videos are capped at 32 frames.GPT-4o receives video frames and question text but no timestamps; answer options are shuffled to balance correct-answer prefixes.Source: §A.6 · p. 17

A.7 CASES

  • Benchmark case examples show task names, video frames, questions, options, and correct answers.Source: §A.7 · p. 17

B.1 VIDEO CAPTION GENERATION

  • Visual captions are generated with Qwen2.5-VL at one frame per second for each identified scene, regardless of scene quality.The captioning prompt is designed to capture static and dynamic scene information.Source: §B.1 · pp. 17–18

B.2 AUDIO CAPTION GENERATION

  • Audio descriptions are generated separately for vocal traits, sound events, and music, while Whisper-v3 is used for dialogue transcription and timestamps.The pipeline post-processes sound-event and music descriptions to separate them, then refines audio captions to distinguish background music from sound events, align vocal traits with dialogue, and remove redundancy.Source: §B.2 · p. 18

B.3 QA PAIR GENERATION

  • QA generation uses high-quality scenes for single- and multi-scene tasks but all scene descriptions for full-scene tasks, and checks multi-scene questions for cross-scene information intervals.The model also generates a brief answer justification.Source: §B.3 · p. 18

B.4 QUALITY CONTROL

  • General QA checks assess dual-modality clues, format, logical support, and speculation, while task-specific checks vary by task.The sequence check differs slightly between VSSR and MPO, and the ambiguity check differs between SPL and SOER.Source: §B.4 · p. 18

B.5 DISTRACTOR GENERATION

  • Distractors are generated after QA filtering because their creation requires additional video information, with error types used to diversify options.Examples of error types include incorrect details and temporal or spatial misplacement.Source: §B.5 · pp. 18–19

C.1 EVALUATION OF AV CORRELATION FOR PREVIOUS WORKS

The authors estimate audio-visual correlation in prior benchmarks with Qwen-2.5 and compare its judgments with human assessments. The automatic score corresponds closely to the human scores, supporting its use as an indicator of joint audio-visual reasoning.

C.1 EVALUATION OF AV CORRELATION FOR PREVIOUS WORKS

  • Qwen-2.5 judged 1,000 QA pairs per dataset using the modality-check prompt to estimate the proportion requiring both audio and visual information.The authors also recruited volunteers to assess 100 randomly sampled items per benchmark.Source: §C.1 · p. 19
  • Human volunteers assessed 100 randomly sampled items per benchmark to verify whether questions required both auditory and visual information.Source: §C.1 · p. 19
  • The automatically calculated score corresponded closely to human scores and was presented as an indicator of joint audio-visual reasoning ability.Source: §C.1 · p. 19

C.2 ERROR STUDY

The error study identifies weaknesses in aligning vocal traits with visual information and in temporal reasoning. Examples include using speech and vocal traits to infer spatial information, arranging storyline details, and grounding events in long videos.

C.2 ERROR STUDY

  • Examples show models failing to align vocal traits with visual information, including when inferring spatial information from speech and vocal traits.The authors characterize this alignment ability as low in current models.Source: §C.2 · p. 19
  • A storyline example tests whether models can understand the sequence of events and arrange detailed information.Source: §C.2 · p. 19
  • A separate long-video example tests temporal grounding, and the authors identify temporal relationships in audio-visual scenarios as an area for future work.Source: §C.2 · p. 20

D LIMITATIONS

The paper notes limitations in JointAVBench’s data source, taxonomy, evaluation coverage, and automated construction, alongside broader impacts and safeguards. It also declares LLM use and specifies the dataset’s license.

D LIMITATIONS

  • JointAVBench derives exclusively from SF20K, which may bias its data distribution, and its taxonomy may not cover every dimension of joint audio-visual reasoning.Experiments cover selected representative MLLMs rather than an exhaustive set; automated construction also risks subtle factual errors, weak modality dependency, or partially plausible answers despite manual verification and an answer-label audit.Source: D LIMITATIONS · p. 20

E BROADER IMPACTS

  • The dataset may support human-like agents, video-understanding tools, and assistive software, while raising privacy and copyright concerns.Source: E BROADER IMPACTS · p. 20

F DECLARATION OF LLM USAGE

G SAFEGUARDS

  • The authors used pre-filtered safe SF20K content and the official Qwen API’s safety mechanisms to screen prompts and responses.Source: G SAFEGUARDS · p. 20

H LICENSE

  • JointAVBench is released under CC BY-NC-SA 4.0, whose terms subsequent research must follow.Source: H LICENSE · p. 20

Related papers

Loading 2512.12772v2…