MMVU: Measuring Expert-Level Multi-Discipline Video Understanding

1 INTRODUCTION

• Existing specialized-domain benchmarks mainly assess text or images, leaving expert-level reasoning over videos—rich in temporal dynamics, procedural knowledge, and interactions—under-evaluated.In the example of a chemical reaction video, understanding reaction stages from visual cues such as color changes or precipitates requires chemical knowledge as well as visual recognition.Source: §1 · p. 2
• MMVU contains 3,000 expert-annotated QA examples from 1,529 specialized-domain videos, spanning 27 subjects across four disciplines.Textbook-guided annotation has experts identify concepts, source relevant videos, and create questions requiring domain knowledge and expert-level reasoning; examples include rationales and relevant knowledge, with quality controls.Source: §1 · p. 2
• o1 achieved the highest performance among 32 tested models; GPT-4o scored 66.7% versus human experts’ 86.8% in the open-book setting.The evaluation covered models from 17 organizations. Chain-of-thought reasoning generally enhanced performance over direct answer generation, and human-reviewed error analyses and case studies examined model limitations.Source: §1 · p. 2

Video Understanding Benchmark

• Existing video benchmarks emphasize general-purpose comprehension and natural scenes, leaving specialized-domain videos requiring both visual perception and domain expertise insufficiently assessed.The gap is especially consequential in healthcare, engineering, and science.Source: §2 · pp. 2–3

Multi-discipline Evaluation Benchmark

• Textual and multimodal multidisciplinary benchmarks have advanced expert reasoning evaluation, but multimodal efforts remain largely static-image-based and video expert reasoning underexplored.Creating a high-quality multidisciplinary video benchmark is harder because suitable resources, such as textbooks or exam questions, are scarce.Source: §2 · p. 3
• MMWorld [He et al., 2024] includes videos across disciplines, but only 39.5% of examples require domain expertise and 76.4% are GPT-4V-generated; MMVU adds expert rationales and domain knowledge per example.These annotations enable more nuanced assessment of expert-level reasoning than benchmarks that provide only ground-truth answers.Source: §2 · p. 3

3.1 PRELIMINARY SETUP

• A study of 133 college and graduate students informed selection of 27 subjects across four disciplines for expert-level video understanding.The study gathered example questions and feedback; authors selected subjects best aligned with the benchmark’s construction goals.Source: §3.1 · p. 4

3.2 TEXTBOOK-GUIDED QA EXAMPLE ANNOTATION

• Textbook-guided annotation links key concepts to suitable videos, then produces questions requiring domain knowledge to reason over video content.Videos are selected to be vision-intensive, CC-licensed, without audio tracks or excessive on-screen text; annotations include relevant timestamps, distractors, domain knowledge, and step-by-step rationales. Restricting videos to CC licenses can limit coverage of subjects such as sports.Source: §3.2 · pp. 4–5, §3.3 · p. 5

3.3 DATA QUALITY CONTROL

• Expert review revised 523 examples and excluded 72; 83.0% of initial examples met design criteria without revisions.Annotators were compensated by time spent; annotation averaged 20 minutes 17 seconds per example and validation 4 minutes 12 seconds.Source: §3.3 · p. 6

3.4 MMVU BENCHMARK ANALYSIS

• MMVU contains 3,000 examples, split into 1,000 validation and 2,000 hidden test examples for model development and standard evaluation.The test set is reserved to prevent contamination, and the authors describe plans for a public online evaluation platform and leaderboard.Source: §3.4 · p. 6, Table 2 · p. 6
• Human accuracy rose from 49.7% closed-book to 86.8% open-book and 95.3% after oracle revision across 120 questions.Five participants answered 30 test questions per discipline; the open-book phase allowed external resources, and oracle revision used ground-truth domain knowledge and self-sourced resources.Source: §3.4 · pp. 6–7

4.1 EXPERIMENT SETUP

• The evaluation covers 16 open-source and eight proprietary model series, using direct-answer and CoT prompts; GPT-4o assesses answer accuracy.Models without native video support receive the maximum number of images that fits their context window.Source: §4.1 · p. 7

4.2 MAIN FINDINGS

• GPT-4o reaches 66.7% accuracy with CoT, below the 86.8% accuracy achieved by human experts in an open-book setting.The result illustrates the remaining gap in expert-level video reasoning on MMVU.Source: §4.2 · p. 7, Table 3 · p. 8
• Open-source models still lag proprietary models overall, but Qwen2-VL-72B and DeepSeek-VL2 exceed human benchmarks in closed-book settings and approach leading proprietary models.Source: §4.2 · p. 8
• CoT generally improves performance, but gains vary: Claude 3.5 Sonnet improves by 11.0%, whereas GPT-4o shows only marginal improvement.Source: §4.2 · pp. 8–9, Figure 4 · p. 9
• o1 and Gemini 2.0 Flash Thinking achieve the top two MMVU results, demonstrating the effectiveness of System-2 thinking and long CoT.The authors point to increased test-time compute and long CoT as ways to enhance expert-level video reasoning.Source: §4.2 · p. 9

4.3 QUALITATIVE ANALYSIS

• Human analysis of sampled model errors finds visual-perception errors (18%) and domain-knowledge misuse in visual perception (20%).The analysis covers 50 validation-set error cases for each of four top-performing models.Source: §4.3 · p. 9
• Domain-knowledge misuse in reasoning (27%) and heavy reliance on textual information (20%) are also primary error types.Textual reliance includes evaluating multiple-choice options without using video content.Source: §4.3 · pp. 9–10
• The remaining primary errors are logical reasoning inconsistencies (6%) and other errors (9%), including refusal, output-limit, repetition, or math-computation problems.Source: §4.3 · p. 10

5 CONCLUSION

• o1 achieves the highest performance among 32 evaluated models, approaching human expert-level proficiency, while a notable gap remains between other models and human experts.Models using CoT reasoning consistently outperform models that generate final answers directly.Source: §5 · p. 10
• MMVU is a high-quality benchmark built from scratch by human experts using a textbook-guided pipeline to capture breadth of knowledge and depth of reasoning.The paper reports comprehensive error analyses and case studies identifying persistent challenges in specialized-domain video understanding.Source: §5 · p. 10

A.1 ANNOTATOR BIOGRAPHY

• The 73 annotators are English-fluent and affiliated with universities ranked in the 2024 QS Global Rankings Top 500.Their biographies are provided in Table 4, with identities hidden for confidentiality.Source: §A.1 · p. 24, Table 4 · p. 24

A.2 TEXTBOOK FOR EACH SUBJECT

• Subject-specific textbooks were selected by expert annotators as authoritative references, supporting MMVU’s textbook-guided annotation pipeline.Tables 6–9 list textbooks and corresponding example counts for Science, Engineering, Healthcare, and Humanities and Social Science.Source: §A.2 · p. 26, Table 6 · p. 26, Table 7 · p. 27

A.3 ANNOTATION GUIDELINE AND INTERFACE

• The annotation interface checks submitted YouTube videos for a Creative Commons license and blocks submissions that fail the requirement.Annotators enter a video URL and select a question type in Step 1.Source: Figure 6 · p. 29
• Step 2 of the annotation interface supports both multiple-choice and open-ended question annotation.The interface is described as based on Turkle (HLTCOE@JHU, 2024), an open-source clone of Amazon’s Mechanical Turk.Source: Figure 7 · p. 29, §A.3 · p. 29, Figure 8 · p. 30

A.4 VALIDATION GUIDELINE AND INTERFACE

• Each example receives expert review by an author or top-performing annotator to check accuracy and alignment with benchmark criteria.Validators request revisions and a second review when feasible, or may discard low-quality examples unlikely to meet criteria through revision.Source: §A.4 · p. 31, Figure 9 · p. 31

A.5 DATA ANNOTATION AND VALIDATION PAYMENT

• Annotators received $6 per hour, plus $2 per completed annotation and $0.40 per validated example; average annotation and validation times were 20 minutes 17 seconds and 4 minutes 12 seconds.The stated pay structure was designed to make wages competitive with teaching assistant payments at annotators’ universities.Source: §A.5 · p. 31
• Annotators were paid according to time spent rather than examples completed, and screen recordings supported checks of reported time, productivity, distractions, and task duration.The authors said this approach was intended to prevent rushed work and support dataset quality.Source: §A.5 · p. 31
• To reduce pressure, annotators were advised to limit work to 10 QA annotations or 50 QA validations per day.The annotation and validation process spanned three months.Source: §A.5 · p. 31

B.1 CONFIGURATION OF EVALUATED MODELS

• Inference uses each model’s official default vision settings, temperature 1.0, and a 1024-token output limit, except Gemini-2-Flash-Thinking uses 8192 tokens.Inferences are reproducible on a workstation with two NVIDIA A100-80G GPUs.Source: §B.1 · p. 32
• Table 10 lists evaluated models, their sources, and default input-frame counts for multi-image models, selected from 2, 4, 8, 16, or 32 within context limits.“HF” denotes Hugging Face.Source: Table 10 · p. 32

B.2 CHAIN-OF-THOUGHT AND DIRECT ANSWER PROMPTS

• The study uses separate CoT-reasoning and direct-answer prompts for multiple-choice questions; the CoT and direct-answer prompts are adopted from MMMU-Pro (Yue et al., 2024b).Figure 10 shows the CoT prompt and Figure 12 the direct-answer prompt.Source: Figure 10 · p. 33, Figure 12 · p. 33, §B.2 · p. 33
• The study also provides distinct CoT-reasoning and direct-answer prompts for open-ended questions.Figures 11 and 13 show the respective prompts.Source: Figure 11 · p. 33, §B.2 · p. 33, Figure 13 · p. 34

B.3 PROMPTS FOR ACCURACY EVALUATION

• Figure 14 presents an evaluation prompt for assessing multiple-choice QA accuracy.Source: Figure 14 · p. 35
• Figure 15 presents an evaluation prompt for assessing open-ended QA accuracy.Source: Figure 15 · p. 35

C.1 COMPARISON BETWEEN COT REASONING AND DIRECT ANSWERING

• Figure 16 compares model performance between CoT reasoning and direct answering on the validation set.The supplied caption gives no scores or comparison outcome.Source: Figure 16 · p. 36

C.2 ERROR CASE ANALYSIS: VISUAL PERCEPTION ERROR

• Figure 17 presents an error case of Thermodynamics.Source: Figure 17 · p. 37
• Figure 18 presents an error case of Electromagnetism.Source: Figure 18 · p. 38
• Figure 19 presents an error case of Art.Source: Figure 19 · p. 39

• Figure 20 presents an error case in computer science.Source: Figure 20 · p. 40
• Figure 21 presents an error case in electrical engineering.Source: Figure 21 · p. 41
• Figure 22 presents an error case in pharmacy.Source: Figure 22 · p. 42

C.4 ERROR CASE ANALYSIS: MISUSE OR LACK DOMAIN KNOWLEDGE IN REASONING

• Figures 23–25 present error cases in computer science, biology, and chemistry, respectively.The captions identify the examples’ disciplines but provide no further information about the reasoning errors.Source: Figure 23 · p. 43, Figure 24 · p. 44, Figure 25 · p. 45

C.5 ERROR CASE ANALYSIS: HEAVY RELIANCE ON TEXTUAL INFORMATION

• Figure 26 presents a Clinical Medicine error case in the heavy-reliance-on-textual-information category.Source: Figure 26 · p. 46
• Figure 27 presents a Management error case in the heavy-reliance-on-textual-information category.Source: Figure 27 · p. 47

C.6 ERROR CASE ANALYSIS: LOGICAL REASONING ERROR

• Figure 28 presents a Mechanical Engineering error case in the logical-reasoning-error category.Source: Figure 28 · p. 48
• Figure 29 presents a Clinical Medicine error case in the logical-reasoning-error category.Source: Figure 29 · p. 49