PaperlayerView exact revision on arXiv

Source-linked AI summary

AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?

Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, Xiangyu Yue

arXiv:2412.02611v1cs.CVcs.AIcs.CLcs.MMcs.SDeess.AS

TL;DR

The paper asks whether MLLMs truly understand audio-visual information, amid gaps in prior benchmarks’ assessment of basic listening and broader audio-visual coverage. It introduces DeafTest and AV-Odyssey, and finds current models still struggle with listening and audio-visual integration.

  • Problem

    Prior audio-visual benchmarks overlook basic listening and have limited audio-attribute and domain coverage, leaving MLLMs’ audio-visual understanding insufficiently assessed.

    Source: §1 · p. 2

  • Method

    The paper pairs four-task DeafTest with AV-Odyssey, a 4,555-question, 26-task multiple-choice benchmark combining text, image or video, and audio inputs.

    Source: §1 · pp. 2–3, §3.2 · p. 4

  • Results

    Current MLLMs fall short on complex audio-visual integration; all models score below 65% on DeafTest loudness comparison.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors intend AV-Odyssey to inform future dataset collection and model development.

    Source: §5 · p. 9

    For four-choice AV-Odyssey questions, below-random performance indicates ineffective task handling, and gaps between two below-random models are meaningless.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini 1.5 Pro, and Reka Core, have expanded their capabilities to include vision and audio modalities. While these models demonstrate impressive performance across a wide range of audio-visual applications, our proposed DeafTest reveals that MLLMs often struggle with simple tasks humans find trivial: 1) determining which of two sounds is louder, and 2) determining which of two sounds has a higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a comprehensive audio-visual benchmark designed to assess whether those MLLMs can truly understand the audio-visual information. This benchmark encompasses 4,555 carefully crafted problems, each incorporating text, visual, and audio components. To successfully infer answers, models must effectively leverage clues from both visual and audio inputs. To ensure precise and objective evaluation of MLLM responses, we have structured the questions as multiple-choice, eliminating the need for human evaluation or LLM-assisted assessment. We benchmark a series of closed-source and open-source models and summarize the observations. By revealing the limitations of current models, we aim to provide useful insight for future dataset collection and model development.

1. Introduction

The paper asks whether MLLMs truly understand audio-visual information, noting gaps in prior benchmarks’ coverage of basic listening and broader evaluation dimensions. It introduces DeafTest and AV-Odyssey, and reports that current models still struggle with basic listening and complex audio-visual integration.

1. Introduction

  • Earlier audio-visual benchmarks overlook basic listening ability and cover limited audio attributes and application domains, motivating DeafTest and AV-Odyssey.The paper frames these gaps as limiting evaluation of MLLMs’ audio-visual understanding.Source: §1 · p. 2
  • On DeafTest, all evaluated models score below 65% on loudness comparison and none above 60% on pitch; Gemini 1.5 Pro leads sound counting at 81%, versus humans’ 100%.Counting clips contain monotonous sounds clearly separated by silent intervals.Source: §1 · p. 2
  • The dataset encompasses 4,555 carefully selected questions spanning 26 distinct tasks, with each question constructed to include three components: text, image/video, and an audio clip.Source: §1 · p. 3
  • The benchmark filters questions that single-modal approaches could easily answer, spans sound attributes and application domains, and uses multiple-choice evaluation.Its stated scope includes timbre, tone, spatial characteristics, and temporal dynamics, alongside domains such as music, daily life, and transportation.Source: §1 · p. 3
  • Current MLLMs fall short on complex audio-visual integration, and audio-caption-vision training does not effectively combine modalities.Error analysis identifies audio-input perceptual errors as the major cause of audio-visual inference errors.Source: §1 · p. 3

2. Related Work

The related work traces the development of models that add visual and audio capabilities to language models, then reviews benchmarks for evaluating them. It identifies limited coverage of audio attributes, domains, and interleaved inputs in prior audio-visual benchmarks.

2. Related Work

  • Large language models perform across diverse textual domains, catalyzing vision-language and multimodal large language models that extend capabilities to visual comprehension and, with audio integration, accommodate audio inputs.Source: §2 · p. 3
  • Prior benchmarks emphasize visual comprehension and mathematical reasoning, while existing audio-visual benchmarks have limits in task breadth and coverage.Music-AVQA [33] is limited to music, and AVQA primarily tests daily-life applications.Source: §2 · p. 3
  • AV-Odyssey is presented as spanning a wide range of audio attributes and application domains to assess audio-visual comprehension.The passage contrasts this scope with prior benchmarks’ restricted attributes and domains.Source: §2 · pp. 3–4

3. Method

The method pairs DeafTest’s low-level listening tasks with AV-Odyssey, a broad multiple-choice benchmark designed to assess audio-visual integration. The benchmark spans diverse sound attributes and domains, uses interleaved multimodal inputs, and combines automated and expert-led question construction with filtering.

3.1. DeafTest Tasks

  • DeafTest tests sound counting, loudness, pitch, and duration; its clips vary the target sound property to assess basic auditory perception.Loudness comparisons use clips sampled from 30–60 dB and 70–100 dB; duration comparisons use 1–3 s and 4–6 s clips.Source: §3.1 · p. 4
  • The pitch task compares two clips sampled at 100–500 Hz and 1,000–2,000 Hz, with human checks that the pitch difference is easy to hear.The benchmark also randomly switches the order of the clips in the loudness and pitch tasks.Source: §3.1 · p. 4
  • Gemini 1.5 Pro exceeds 80% accuracy on sound counting and duration, while no tested model exceeds 65% on loudness or pitch comparison.The authors interpret the results as evidence that MLLMs fall short in basic listening ability, which might hinder audio-visual integration.Source: §3.1 · p. 4

3.2. Overview of AV-Odyssey Bench

  • AV-Odyssey uses four-option questions, interleaved text, image/video, and audio inputs, and covers 26 tasks across 10 domains and 7 audio attributes.Its tasks are intended to assess audio-visual integration across diverse sound attributes and application domains.Source: §3.2 · pp. 4–5
  • Task categories cover timbre matching, tone and meme understanding, melody matching, spatial inference, and temporal matching, including singer and bird-species recognition.Examples include matching music to dance, estimating object distance or 3D angle, and identifying an action’s start and end times.Source: §3.2 · p. 5
  • Hallucination tasks probe audio-visual reasoning, while intricacy tasks require integrated analysis across visual elements and multiple audio attributes.Action prediction, for example, combines visual elements with audio attributes such as timbre and timing.Source: §3.2 · p. 5

3.3. Data Curation Process

  • Question construction spans sound attributes and application domains, using public audio datasets with corresponding online visuals for many concept-matching questions and expert-authored questions for others.Human experts post-evaluate and filter low-quality questions.Source: §3.3 · pp. 5–6
  • Quality control simplifies question context and filters questions solvable by tested vision or audio models; 2.54% were filtered because all tested models could solve them.The filtering used VLMs and audio LLMs to identify questions answerable without the intended multimodal evidence.Source: §3.3 · pp. 6–7

4. Experiment

The experiments benchmarked 18 MLLMs in zero-shot settings, using caption-based workarounds for GPT-4o's unavailable joint-input API. Results show weak AV-Odyssey performance and identify audio perception as the leading error source.

4.1. Models

  • The evaluation covered 18 MLLMs in zero-shot settings; GPT-4o was tested through separate audio-caption and visual-caption methods because joint image, video, and audio API access was unavailable.For the audio-caption method, audio captions were supplied alongside text and images or video; the visual-caption method supplied visual captions alongside text and audio.Source: §4 · p. 7, §4.1 · p. 7

4.2. Main Result Analysis

  • 34.5% accuracy was achieved by the top-performing model, GPT-4o audio caption, on AV-Odyssey.Most existing MLLMs scored only marginally above the 25% random-guessing baseline.Source: §4.2 · p. 7
  • 27.4% and 27.2% accuracy for PandaGPT and OneLLM respectively were only marginally below GPT-4o audio caption at 34.5% and Gemini 1.5 at 30.8%.Their results were comparable to some closed-source models, including Gemini 1.5 Flash 8B and Reka Core.Source: §4.2 · p. 8
  • GPT-4o audio caption outperformed GPT-4o visual caption on AV-Odyssey.The authors suggest this may reflect greater information loss in visual captions.Source: §4.2 · p. 8
  • The audio-text-vision training pipeline used by several open-source MLLMs was insufficient to bridge audio and vision and learn audio-visual integration.The cited models include OneLLM, Unified-IO-2, VideoLLaMA2, and NExT-GPT.Source: §4.2 · p. 8

4.3. Error Analysis

  • Human experts annotated four randomly selected error instances per task, yielding 104 Gemini 1.5 Pro cases shown in Figure 5.Detailed case analyses are provided in the Appendix.Source: §4.3 · p. 8, Figure 5 · p. 8
  • 81% of Gemini 1.5 Pro errors were perception errors: 63% audio, 10% vision, and 8% text; Figure 5 shows the 104-case distribution.The authors identify audio perception as the main bottleneck in audio-visual integration.Source: §4.3 · p. 8, Figure 5 · p. 8
  • 13% of errors involved incorrect reasoning despite successful information extraction, while 6% were other errors, primarily rejections.Some rejections followed content being mistakenly flagged for security reasons.Source: §4.3 · p. 8

5. Conclusion

The paper introduces AV-Odyssey Bench to evaluate MLLMs' understanding of audio-visual information. Its 4,555 multiple-choice problems require models to integrate visual and audio cues, and the authors report limitations in current models.

5. Conclusion

  • AV-Odyssey Bench contains 4,555 multiple-choice problems designed to test integration of visual and audio cues.The benchmark evaluates MLLMs' understanding of audio-visual information.Source: §5 · pp. 8–9
  • Benchmarking closed-source and open-source models revealed limitations in their effective understanding of audio-visual inputs.Source: §5 · p. 9
  • The authors intend the benchmark to support community work toward more capable audio-visual understanding.They describe it as a resource for advancing MLLMs.Source: §5 · p. 9

A. Data Distribution

The appendix describes AV-Odyssey Bench's task distribution and reports that the benchmark covers 26 tasks.

A. Data Distribution

B. Breakdown Results

The appendix provides model evaluation breakdowns for Timbre and several other AV-Odyssey task parts.

B. Breakdown Results

List of Case Study Figures

The figure list compiles sampled error cases across recognition, music, spatial-temporal, and action-reasoning tasks.

Recognition

Music and media

Spatial and temporal tasks

Reasoning and other recognition tasks

Loading 2412.02611v1…