PaperlayerView exact revision on arXiv

Source-linked AI summary

MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

Yuxuan Fan, Gyusik Seo, Jing Hao, Jaemin Cho, Mohit Bansal, Jaehong Yoon

arXiv:2606.30026v1cs.CVcs.AI

TL;DR

Existing video benchmarks mainly ask what is happening in a scene, leaving underexplored whether MLLMs can infer intent behind creative decisions. MUSEBENCH uses expert video essays and a four-phase pipeline to build 4,016 questions across four audiovisual-art categories. In zero-shot evaluation, the strongest MLLM reaches 48.29% accuracy, compared with 87.18% for human experts.

  • Problem

    Existing video benchmarks primarily assess what happens in a scene rather than whether MLLMs can infer the intent behind creative decisions.

    Source: §1 · p. 2

  • Method

    MUSEBENCH uses video essays and a four-phase human-in-the-loop pipeline to construct 4,016 expert-validated questions across four audiovisual-art categories.

    Source: §5 · p. 10

  • Results

    48.29% accuracy for the strongest MLLM versus 87.18% for human experts; models also lag on game arts and recover only the most salient multi-select option.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The findings motivate treating audiovisual arts as more than specialized video understanding and incorporating richer artistic and cultural supervision into training.

    Source: §4.2 · p. 8

    MUSEBENCH covers four art categories and relies primarily on video essays, which may not capture all artistic diversity and vary in availability across art forms and languages.

    Source: A Limitation and Future Work · p. 15

Abstract

from arXiv · show

Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements (e.g., fear amplified through claustrophobic framing, or grief conveyed through silence and lingering close-ups). True artistic understanding extends beyond recognizing what is depicted to reasoning about why it is expressed through particular creative choices. Despite the strong progress of multimodal large language models (MLLMs), this critical aspect of artistic understanding remains underexplored, as existing benchmarks largely measure perceptual recognition while overlooking reasoning about creative intent. To address this gap, we introduce Musebench, a comprehensive benchmark designed to evaluate MLLMs on nuanced artistic understanding. It comprises 4,016 questions spanning cinematic arts, static visual arts, stage performing arts, and game arts, distilled from over 10K candidate video essays that pair professional commentary with visual demonstration. To capture the open-ended nature of artistic analysis at scale, the benchmark combines single-select and variable-option multi-select questions. All questions are generated and refined through a four-phase iterative pipeline combining shortcut filtering, adversarial distractors, and expert validation. Comprehensive zero-shot evaluation of 28 state-of-the-art MLLMs reveals that even the best-performing model achieves only 48.29% accuracy, substantially below human expert performance of 87.18%, exposing a significant gap in current models' creative domain expertise.

1 Introduction

MuseBench evaluates whether MLLMs can reason about artistic intent—not just recognize depicted content—using a benchmark built for the interpretive plurality of audiovisual arts. In zero-shot evaluation, the best model substantially underperforms human experts.

Motivation

  • Artistic understanding requires inferring why creators use visual and auditory choices to convey intent, beyond identifying what appears on screen.Existing video benchmarks primarily test scene content with a single correct option, leaving intent-level understanding underexplored [63] [44] [42].Source: §1 · pp. 1–2

Benchmark construction

  • A four-phase pipeline turns over 10,000 video essays into visually grounded intent-level questions, using adversarial distractors to suppress transcript-based shortcuts.Video essays pair professional commentary with on-screen demonstrations; distractors target technical misreading, oversimplification, factual error, and conceptual confusion.Source: §1 · pp. 2–3

Evaluation protocol

  • Chance-Adjusted Accuracy normalizes single-select scores across option counts, while set-based F1 and exact match assess multi-select judgments and partial agreement.Most evaluated models show higher precision than recall, and even the strongest systems have a sizable multi-select F1–exact-match gap.Source: §1 · p. 3

Evaluation findings

  • 48.29% accuracy was the best result among 28 zero-shot MLLMs, versus 87.18% accuracy for human experts.The gap exposes an artistic-understanding weakness that existing benchmarks focused on surface content can obscure.Source: §1 · p. 3
  • Models lag on game arts across tiers, recover only the most salient multi-select option, and gain little from adaptive key-frame selection.The authors identify stylistic vocabulary and cultural priors, rather than temporal localization, as the bottleneck.Source: §1 · p. 3

Benchmark design

  • MuseBench covers four audiovisual-art categories and 11 sub-domains, combining single- and multi-select questions with variable option counts to represent interpretive plurality.Single-select items target the most precise interpretation, while multi-select items test whether models identify the full set of valid analytical dimensions.Source: §1 · p. 3

2 Related Work

Related work advances video-capable MLLMs and benchmarks, but largely targets general, factual, or academic understanding; MuseBench addresses underexplored interpretive expertise in the audiovisual arts.

Multimodal Large Language Models for Video Understanding

  • Prior MLLMs improve video processing through sparse encoding, retrieval, and temporal training methods,, [43], [60], [61], [6], [46], [66], but leave audiovisual-arts expertise untested.Existing systems are developed and evaluated mainly on everyday activities, open-domain QA, or academic lectures.Source: §2 · pp. 3–4

Benchmarks for Video Understanding

  • Video benchmarks expanded from short-clip QA [63] to story-level and temporal reasoning [19] [28], multimodal breadth [13], long-video scale [52] [44], and expert STEM knowledge [18] [42].Despite this breadth, existing benchmarks largely center on general activities, factual comprehension, or academic STEM knowledge.Source: §2 · p. 4
  • AV-Odyssey Bench [14] probes pitch and loudness; MuseBench shifts audiovisual evaluation toward artistic intent, including cinematographic technique, composition, and performance craft.This extends inquiry from low-level audiovisual perception toward interpreting artistic intent.Source: §2 · p. 4

3 MuseBench Construction

MuseBench turns expert video essays into an expert-validated benchmark of intent-level audiovisual arts understanding, spanning four art categories and using distinct question formats and metrics. Its construction combines aligned audiovisual evidence, iterative quality review, and benchmark-wide validation.

3.1 Video Essays as a Knowledge Source

  • Expert video essays provide dense explanations of why artistic techniques create effects, align commentary with audiovisual evidence, and cover creative-arts domains underrepresented in existing video benchmarks [42].This supports intent-level questions about why creative choices were made rather than only what occurs in a scene.Source: §3.1 · p. 4

3.2 Evaluation Scope and Question Design

  • A hierarchical taxonomy covers four art categories and 11 sub-domains, paired with single-select questions with one correct answer and multi-select questions with 2 to 4 correct answers.Single-select probes choosing an interpretation; multi-select tests selecting the full set of valid interpretations without over-claiming, and neither format reveals the number of correct answers.Source: §3.2 · pp. 4–5

3.3 Data Collection and Question-Answer Annotation

  • The four-phase pipeline segments essays, captions clips using aligned transcripts, generates 3 to 5 candidate questions per video, then adds plausible distractors.Questions must be answerable from narrator-removed evidence clips, and correct answers are formulated before distractors; captions are construction resources only and are not shown to evaluated models. Distractors target technical misreads, oversimplification, factual errors, and conceptual confusion, with options shuffled and proper nouns prohibited.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

3.4 Quality Review

  • Around 9% of generated QA pairs were flagged as incorrect, while expert ratings of 90 samples exceeded 4.0 across dimensions and achieved Gwet AC2 = 0.855.Four experts rated items on a 0–5 scale; per-category averages ranged from 4.03 to 4.53 with a standard deviation of 0.41. Every retained item was manually verified, and the review loop updated prompts and regenerated items to address failures.Source: §3.4 · p. 6, Figure 4 · p. 6

3.5 Evaluation Metrics

  • Chance-Adjusted Accuracy normalizes single-select scores against each item’s guessing probability, while multi-select uses set-based precision, recall, F1, and secondary exact-match accuracy.CAA makes random guessing score 0 and a fully correct response score 1 regardless of option count; negative aggregate CAA indicates worse-than-chance performance.Source: §3.5 · p. 7

3.6 Benchmark Statistics

  • 4,016 expert-validated questions span four art categories and 11 sub-domains, distilled from over 10,000 candidate video essays.Each retained video contributes 3 to 5 questions, and each question offers 4 to 8 options.Source: §3.6 · p. 7

4 Experiments

Zero-shot evaluations show that MUSEBENCH remains challenging for current MLLMs, with distinct weaknesses across art categories and multi-select reasoning. Analyses further find limited gains from adaptive frame selection, benefits from combining audio and video, and a pronounced first-option bias among open-source models.

4.1 Experimental Setup

  • Evaluations cover 28 representative MLLMs plus five dynamic key-frame-selection models, using sampled video frames and audio or transcripts with prompts specifying single- or multi-select format.Frames are sampled at 1 fps unless that exceeds a model’s supported frame count; native audio-input models receive the full narrator-removed audiovisual clip.Source: §4.1 · p. 7

4.2 Main Results

  • Frontier proprietary MLLMs lead but remain far from saturation, while video-specialized models show no decisive advantage and smaller general-purpose models can match or surpass them.The reported bottleneck is stylistic vocabulary, cultural priors, and grounded inference, not formatting, evaluation noise, or temporal localization.Source: §4.2 · pp. 7–8, Table 1 · p. 8
  • Models that perform well on cinematic, static visual, and stage performing arts drop markedly on game arts, while open-source models show pronounced category specialization.The authors suggest limited representation of interactive visuals, real-time camera control, and stylized rendering in web-scale corpora as a likely explanation, and recommend per-axis reporting.Source: §4.2 · p. 8, Figure 5 · p. 9

4.3 In-Depth Analysis

  • Five adaptive key-frame-selection models score 14.42–20.51 ACC, at or below the lower end of the video-specialized tier and 7–13 points behind its strongest models.The analysis attributes the limited gains to artistic vocabulary and cultural priors rather than failure to locate salient frames.Source: §4.3 · pp. 8–9
  • On multi-select questions, F1 substantially exceeds EM and precision exceeds recall for most models, indicating that models often recover salient correct options but under-predict valid alternatives.The authors caution that F1 can reward partial alignment while EM better distinguishes models that capture multi-faceted reasoning from those offering only one plausible label.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Adding video produces the largest single modality gain for VideoLLaMA2 and Qwen2.5-Omni-7B, audio alone changes scores negligibly, and combining audio with video adds a further gain.Source: §4.3 · p. 9, Table 2 · p. 9
  • On 1,407 single-select items with at least five choices, position A receives 30.9% of predictions versus 15.9% of gold labels, while positions E–H receive 22.2% versus 33.0%.Open-source models predict A at 34.9%, 19.0 points above its gold share; proprietary models predict A at 22.4%, 6.5 points above baseline. Because answer positions are randomized, the pattern indicates an intrinsic positional prior.Source: §4.3 · pp. 9–10, Figure 6 · p. 9

5 Conclusion

MUSEBENCH evaluates MLLMs’ audiovisual arts understanding with expert-validated questions derived from video essays. Zero-shot results reveal a large gap from human experts and recurring failures across categories and question types.

5 Conclusion

  • 48.29% accuracy for the strongest of 28 zero-shot MLLMs, versus 87.18% for human experts, reveals a substantial creative-understanding gap.The evaluation covers four art categories.Source: §5 · p. 10
  • Models lag sharply on game arts, recover only the most salient option in multi-select pairs, and gain little from adaptive key-frame selection, pointing to stylistic-vocabulary and cultural-prior bottlenecks.The conclusion contrasts these bottlenecks with temporal localization.Source: §5 · p. 10
  • MUSEBENCH provides 4,016 expert-validated questions across four art categories, built from video essays through a four-phase pipeline with iterative human review.The categories are cinematic arts, static visual arts, stage performing arts, and game arts.Source: §5 · p. 10
  • The authors release MUSEBENCH to support continued progress toward MLLMs with authentic creative-domain understanding.Source: §5 · p. 10

Appendices · A Limitation and Future Work · B Benchmark Details

MuseBench currently covers four art categories using video essays, which may not represent the full diversity of creative works and are unevenly available across art forms and languages. Future work proposes broader sources and art forms, open-ended evaluation, and arts-specific fine-tuning.

A Limitation and Future Work

B.1 Comparison with Existing Benchmarks · B.2 Category and Sub-domain Definitions

MUSEBENCH combines expert audiovisual-arts analysis with visual dependency, while its taxonomy organizes 4,016 questions across four art categories and 11 sub-domains. These sections distinguish it from existing video benchmarks and define the scope of each category.

B.1 Comparison with Existing Benchmarks

  • Existing benchmarks largely focus on everyday clips or factual QA; Video-MMMU [18] targets STEM lecture learning, not artistic interpretation or joint subtitle-and-audio reasoning.The section also notes that existing benchmarks do not enforce domain expertise or control visual dependency in question design.Source: §B.1 · p. 15
  • MUSEBENCH is the only benchmark in Table 3 combining open-domain scope, subtitles and audio, domain expertise, and visual dependency.Its taxonomy and narrator-removed audiovisual clips paired with timestamped expert commentary support these capabilities.Source: §B.1 · p. 15, Table 3 · p. 15

B.2 Category and Sub-domain Definitions

  • The benchmark organizes 4,016 questions into four art categories and 11 sub-domains grounded in creative disciplines and topics analyzed by expert video essayists.The taxonomy is hierarchical, with each top-level category partitioned into fine-grained sub-domains.Source: §B.2 · p. 15
  • Cinematic Arts covers moving-image works, probing how filmmakers translate dramatic intent into image and sound through cinematography, mise-en-scène, editing and pacing, and sound.These sub-domains encompass camera and lighting choices; staging and spatial composition; cuts, transitions, and rhythm; and audio, silence, and sound bridges.Source: §B.2 · p. 16
  • Static Visual Arts addresses still images and artifacts through Fine Art and Photography, emphasizing compositional and material reasoning rather than temporal structure.Fine Art includes composition, color, perspective, brushwork, chiaroscuro, and material technique; Photography includes exposure, lensing, framing, and documentary intent.Source: §B.2 · p. 16
  • Stage Performing Arts examines live meaning shaped by performer presence, staged space, and time-bound reception across Performance, Stage Design, and Theatrical Lighting.The sub-domains cover performers’ vocal and physical communication, stage spatial organization, and lighting choices that direct attention and shape mood.Source: §B.2 · p. 16
  • Game Arts covers audiovisual craft alongside interactive systems and player agency through CG and Interactive Visuals.CG includes rendered art styles, cinematics, shading, lighting, and post-processing; Interactive Visuals include level design, environmental storytelling, HUD, navigation cues, and gameplay-conditioned camera and animation.Source: §B.2 · p. 16

B.3 Dataset Statistics … C.8 Failure Taxonomy

The appendix reports MUSEBENCH’s statistics and vocabulary, then details its source-to-question construction pipeline and iterative human review. The review eliminated the listed failures before release while also identifying broader issues addressed through prompt refinement.

B.3 Dataset Statistics; B.4 Benchmark Vocabulary Overview

  • Table 4 consolidates global, source-data, and question-format statistics, while Figure 7 visualizes the benchmark’s dominant artistic vocabulary.The vocabulary display removes function words and generic analytical fillers; prominent terms include emotional, composition, color, spatial, narrative, character, and audience.Source: §B.3 · p. 16, Table 4 · p. 16, Figure 7 · p. 17

C Construction Details

  • Construction proceeds from curated video essays through transcription, clip segmentation and captioning, question generation, distractor synthesis, and final iterative review.The appendix follows the main text’s per-video phases and documents both prompt revision and final-stage human evaluation.Source: C Construction Details · p. 17

C.1 Source Curation

  • Source curation used four LLM-controlled discovery stages, admitted relevant candidates at confidence ≥0.55, expanded exhausted keywords, and then applied human vetting.The fallback keyword lists cover 273 keywords across four categories; the category vocabularies use different structures, including thirteen thematic blocks for Stage Performing Arts.Source: §C.1 · pp. 17–19

C.2 Transcription; C.3 Phase A: Clip Segmentation

  • Whisper-Large-v3 [35] transcripts supply full text to question generation and timestamped narration to captioning, while videos are split into non-overlapping 10-second clips.Videos are capped at 1,800 seconds, bounding clip and QA counts; stable clip indices support captioning, clip matching, and relevant-clip references.Source: §C.2 · p. 19, §C.3 · pp. 19, 21

C.4 Phase B: Clip Captioning

  • Keye-VL-1.5 [59] captions each clip at one frame per second using aligned narration, prior-clip context, and category-specific guidance.Captions describe visual attributes such as color, composition, motion, and scene context; they support construction and review but are not shown to evaluated models.Source: §C.4 · p. 21

C.5 Phase C: Question Generation; C.6 Phase D: Distractor Generation

  • The QA generator creates 3 to 5 candidate pairs per video, about 30% multi-select, and attaches contiguous relevant-clip ranges; distractor generation adds 3 to 7 plausible options.Multi-select questions have 2 to 4 independent correct answer points. Distractors use seven strategies, including scope error, temporal confusion, and partial truth, added during iterative review.Source: §C.5 · p. 21, §C.6 · p. 23, Figure 15 · p. 23

C.7 Quality Review Loop

  • The category-specific review loop used expert-tagged pilot items to revise generation and source-vetting rules, and Game Arts exited after three consecutive rounds raised no new failure type.Game Arts revisions removed abstract stems, then title-recognition shortcuts, by banning proper nouns and requiring stems to begin with observable clip evidence.Source: §C.7 · pp. 23, 25, Figure 17 · p. 25

C.8 Failure Taxonomy

  • None of the eight tagged failures remain in the released benchmark, and the audit identified seven systemic issues addressed through prompt-level refinement.CRITICAL and HIGH issues were removed by replacing affected QA pairs; MEDIUM and LOW issues were addressed through stronger prompts and programmatic post-hoc alignment. The systemic issues included weak option discriminability, imprecise targeting of artistic intent, and inconsistent visual-evidence requirements.Source: §C.8 · pp. 25–26, Table 6 · p. 26

C.9 Human Evaluation Details · D Detailed Evaluation Metrics

Human evaluation used a clip-and-question interface and domain experts who rated benchmark items on four dimensions using anchored 0–5 Likert scales.

C.9 Human Evaluation Details

  • A web interface let domain experts play each clip, read its question and options, and score the item on four Likert dimensions.The interface is shown in Fig. 18.Source: §C.9 · p. 27
  • Four domain experts were assigned by training to arts categories, with two independent raters per category and their scores averaged for category means.Drama, Film and Literature raters covered Stage Performing Arts and Cinematic Arts; Computational Media and Arts raters covered Game Arts and Static Visual Arts.Source: §C.9 · p. 27
  • Each item received a 0–5 Likert score on four dimensions, with definitions and anchor descriptions shown beside each slider.Source: §C.9 · p. 27
  • Holistic Quality assessed whether an item exemplified a high-quality benchmark question, from fundamentally flawed to publication-ready.A score of 3 meant acceptable with minor edits.Source: §C.9 · p. 27
  • Visual Necessity assessed whether answering required watching the content, distinguishing specific indispensable on-screen cues from text- or frame-sufficient cues and absent audiovisual anchors.The anchors ranged from 1 for no audiovisual anchor to 5 when the stem required watching a specific cue.Source: §C.9 · p. 27
  • Mechanistic Trace assessed whether the correct option linked a craft technique to an effect and intent, rather than offering only vague effects or aesthetic labels.A score of 5 required a specific craft object and causal explanation.Source: §C.9 · p. 27
  • Answer Integrity assessed whether the options contained a unique supported answer and suitable distractors, or a closed correct subset for multi-select items.Lower anchors covered weak distractors and wrong, duplicated, or missing correct options.Source: §C.9 · p. 27
  • Before scoring, raters completed a calibration walkthrough using three pre-selected pairs per category and were told to reserve scores of 0 and 5 for unambiguous cases.Source: §C.9 · p. 27

D.1 Single-Select Evaluation: Chance-Adjusted Accuracy · D.2 Multi-Select Evaluation: Precision, Recall, and F1

The evaluation uses chance-adjusted accuracy to compare single-select performance across varying option counts, and set-based precision, recall, and F1 to diagnose multi-select predictions beyond exact-match accuracy.

D.1 Single-Select Evaluation: Chance-Adjusted Accuracy

  • Chance-adjusted accuracy (CAA) measures single-select performance relative to random guessing because option counts—and therefore random-guess baselines—vary across questions.This normalization makes equal raw accuracy on a question with more options reflect stronger performance.Source: §D.1 · p. 27
  • CAA assigns 1 to a correct answer, has expected score 0 under uniform random guessing, and gives negative scores for worse-than-chance performance.The per-question setup identifies the number of options and whether the prediction matches the ground truth.Source: §D.1 · p. 27

D.2 Multi-Select Evaluation: Precision, Recall, and F1

  • For multi-select questions, precision, recall, and F1 supplement exact-match evaluation, which treats near-complete and wholly incorrect answer sets alike.The evaluation defines true positives as selected correct options, false positives as selected incorrect options, and false negatives as missed correct options.Source: §D.2 · p. 28
  • The multi-select evaluation averages per-question precision, recall, and F1 for macro scores, while micro scores first aggregate counts across questions.Per-question true-positive, false-positive, and false-negative counts are based on intersections and differences between predicted and ground-truth answer sets.Source: §D.2 · pp. 28–29
  • The benchmark reports overall accuracy and CAA for single-select questions, and precision, recall, and F1 for multi-select questions across art categories.The table caption identifies these evaluation metrics and says it marks best and second-best results, without supplying the values here.Source: Table 7 · p. 29
  • Precision, recall, and F1 characterize multi-select predictions by measuring incorrect distractor selection, missed valid perspectives, and the balance between those behaviors.These set-based metrics provide a more informative diagnosis of model behavior than exact-match accuracy.Source: §D.2 · p. 29

E Complete Experimental Results · F Broader Impact · G Additional Examples of MUSEBENCH

The expanded results show category-level accessibility differences and a precision-over-recall pattern, with Video-R1 as an exception; the benchmark also has potential research and applied uses, and its examples illustrate intent-level reasoning across four arts categories.

E Complete Experimental Results

  • Open-source MLLMs were evaluated on one NVIDIA A800 cluster, proprietary systems through official APIs, and all systems zero-shot on the full test set in one pass.Source: E Complete Experimental Results · p. 29
  • Stage and Cinematic Arts lead per-category ACC, while Game Arts trails by 15 to 25 points for nearly every system.Claude-4.6-Opus scores 57.77% Stage ACC and 50.20% Cinematic ACC; Game Arts scores 32.84% for Claude, 30.11% for GPT-5.4, 31.28% for Doubao-Seed-1.8-Pro, and 29.62% for Qwen3.5-397B-A17B.Source: E Complete Experimental Results · p. 29
  • Precision exceeds recall in cited multi-select examples, including GPT-5.4 Cinematic (77.52% vs. 71.05%), Claude Cinematic (79.60% vs. 70.68%), and LLaVA-OneVision-7B Static (40.99% vs. 32.98%).Source: E Complete Experimental Results · p. 29
  • Video-R1 reverses the category-level precision–recall pattern, with recall above precision in Cinematic, Static, Stage, and Game Arts.Its recall is 61.21% Cinematic, 65.63% Static, 62.37% Stage, and 51.12% Game.Source: E Complete Experimental Results · p. 29

F Broader Impact

  • MUSEBENCH may guide richer cultural and stylistic training, and serve as a standardized probe for arts education, accessibility, and content-analysis applications.The passage names audio description for visually impaired audiences and pipelines for archivists, educators, and independent critics.Source: F Broader Impact · p. 30

G Additional Examples of MUSEBENCH

Related papers

Loading 2606.30026v1…