PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, Gaoang Wang

arXiv:2504.14693v2cs.CVcs.AI

TL;DR

Knowledge-intensive, reasoning-heavy lecture comprehension remains insufficiently evaluated, despite advances in video LMMs. Video-MMLU benchmarks perception and reasoning in multi-discipline lectures and evaluates more than 90 models. The authors report that proprietary and open-source LMMs perform poorly on lecture understanding.

  • Problem

    Knowledge-intensive and reasoning-heavy lecture videos remain insufficiently tested, although they require perception and domain-specific reasoning across dense visual material.

    Source: §1 · p. 2

  • Method

    Video-MMLU evaluates lecture understanding with detailed captions for visual perception and question-answer pairs for reasoning, alongside analysis of visual tokens and base LLMs.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Existing models generally achieve only 10% to 50% accuracy on Video-MMLU, and both proprietary and open-source LMMs perform poorly.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    The authors' analysis of visual-token strategies and base LLM architectures provides insights for future research on lecture understanding.

    Source: §5 · p. 12

Abstract

from arXiv · show

Recent advancements in language multimodal models (LMMs) for video have demonstrated their potential for understanding video content, yet the task of comprehending multi-discipline lectures remains largely unexplored. We introduce Video-MMLU, a massive benchmark designed to evaluate the capabilities of LMMs in understanding Multi-Discipline Lectures. We evaluate over 90 open-source and proprietary models, ranging from 0.5B to 40B parameters. Our results highlight the limitations of current models in addressing the cognitive challenges presented by these lectures, especially in tasks requiring both perception and reasoning. Additionally, we explore how the number of visual tokens and the large language models influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension.

1. Introduction

Video-MMLU addresses the limited evaluation of knowledge-intensive and reasoning-heavy lecture videos with a benchmark spanning lecture collection, annotation, and dataset statistics. Its construction pairs detailed captions for visual perception with reasoning questions, and the paper reports that current models generally perform poorly.

1. Introduction

  • Video-MMLU targets theorem demonstrations and problem-solving, addressing a gap in knowledge-intensive, reasoning-heavy lecture evaluation that static-image benchmarks do not cover.Such lectures combine dense text, equations, and visual demonstrations with perception and domain-specific reasoning.Source: §1 · p. 2
  • Existing models generally achieve only 10% to 50% accuracy on Video-MMLU, which tests lecture understanding across mathematics, physics, and chemistry.The benchmark evaluates more than 90 proprietary and open-source models.Source: §1 · p. 2

2.1. Large Multimodal Models for Video

  • Video-LMM research spans short- and long-video understanding, captioning, token reduction, and video reasoning, using encoder, compression, and context-window strategies.The cited work also explores audio inputs and hierarchical narration for video understanding.Source: §2.1 · p. 3

2.2. Benchmarks for Video Understanding

  • Video-MMLU extends video benchmarks toward lecture comprehension across disciplines, assessing detailed perception and reasoning through fine-grained captioning and complex QA.Prior benchmarks include longer videos, temporal reasoning, streaming, detailed descriptions, and specialized scenarios.Source: §2.2 · pp. 3–4

3.1. Video Collection and Processing

  • The collection contains 1,065 videos from 10 YouTube channels, capped at four minutes and focused on theorem demonstrations and problem-solving.Videos lacking English subtitles, overly static videos, and videos requiring strong domain-specific prior knowledge are excluded; keyframes are sampled at creator-specific rates, typically every one to five seconds.Source: §3.1 · p. 4

3.2. Annotations Construction Pipeline

  • Each video receives a detailed caption for visual-perception evaluation and 15 questions for reasoning; captions combine video structure, frame-level details, and subtitle-checked formula and number refinements.Claude-3.5-sonnet generates reasoning QA pairs with answers capped at 15 words; captions are manually reviewed for hallucinations and omitted visual elements.Source: §3.2 · pp. 4–6

3.3. Datasets Statistics

  • Video-MMLU contains 1,065 captions averaging 489 words and 15,746 reasoning QA pairs; surface and reasoning question embeddings have a Jensen-Shannon distance of 0.668.Videos average 109 seconds, with 26 keyframes per video; mathematics accounts for 90.3%, physics 3.6%, and chemistry 6.1%.Source: §3.3 · p. 6

4. Evaluation of Video-MMLU

Video-MMLU evaluates 87 models under shared video-input and scoring procedures, then compares their lecture-comprehension performance and examines model and visual-token choices. Models generally struggle, with proprietary models outperforming open-source models and reasoning-heavy tasks exposing particular challenges.

4.1. Models and Evaluation Strategies

  • The evaluation covers 87 models—3 proprietary, 78 open-source LMMs, and 6 vision-blind baselines—and uses shared frame sampling with Qwen2.5-72B as an evaluation assistant.Image-based LMMs receive four frames per video, reduced to two for LLaVA-NeXT-Vicuna and XComposer, which fail to produce valid outputs with four.Source: §4.1 · pp. 6–7

4.2. Leaderboard

  • Video captioning correlates weakly with model size (r = 0.18), versus video QA (r = 0.40), while size-related performance gains remain non-linear and inconsistent.The authors propose possible overfitting to open-world training data as one explanation for weaker generalization to multi-discipline lectures.Source: Figure 4 · p. 8, Figure 5 · p. 8, §4.2 · p. 10
  • Accuracy is mostly 10% to 50%: both proprietary and open-source models perform poorly, although proprietary models consistently outperform open-source models across tasks.Claude-3.5-sonnet achieves the highest performance across all tasks.Source: Table 2 · p. 7, §4.2 · p. 7, Table 3 · p. 8
  • Models perform better on physics and chemistry than mathematics, where lectures use more formulas and dynamic visualizations and models struggle to infer logical relationships from animation.The authors suggest current models rely more on textual information in frames than on interpreting animations.Source: §4.2 · p. 9
  • Under the same architecture, video LMM ShareGPT4V-13B underperforms image LMM LLaVA-1.5-Vicuna-13B despite additional video training.The authors suggest that limited OCR and visual-knowledge reasoning in video-text training data may contribute to this result.Source: §4.2 · p. 9

4.3. Variation Analysis

  • 16–300 visual tokens per frame offers an efficiency-performance balance; PVC-8B at 64 tokens improves 24.7% over InternVL2-8B, while 2-token LLaMA-VID suffers drops.Even the strongest compressed model, PVC-8B, trails MiniCPM-o-2.6; the authors point to difficulty preserving formulas and numbers.Source: Figure 10 · p. 11, §4.3 · p. 12

4.2. Leaderboard

  • Larger Qwen2.5 and InternVL2.5 LLM backbones improve lecture understanding, but gains diminish to a 6.7% improvement at the largest scale.The comparison uses InternVL2.5 models with Qwen2.5 backbones and vision-blind Qwen2.5 models.Source: §4.2 · pp. 10–11
  • Qwen2.5- and InternLM2.5-based LMMs and MoE-based LLMs perform strongly and relatively evenly, while ultrasmall SmolLM2, Fuyu, Vicuna, and LLaMA2-based models struggle.The authors highlight reasoning-focused LLMs such as QwQ-32B as a direction for lecture understanding.Source: §4.2 · p. 11

5. Conclusion

Video-MMLU is a large-scale video benchmark for evaluating multimodal perception and reasoning in multi-discipline lecture comprehension. The paper reports that models still struggle and analyzes visual-token strategies and LLM architectures.

  • Video-MMLU is a large-scale video-based benchmark designed to evaluate LMM perception and reasoning in multi-discipline lecture comprehension.Source: §5 · p. 12
  • Both proprietary and open-source LMMs perform poorly on the benchmark, underscoring challenges in lecture understanding.Source: §5 · p. 12
  • The paper analyzes visual-token strategies and base LLM architectures to provide insights for future research.Source: §5 · p. 12

Supplementary Material

The supplementary material collects related work, dataset and evaluation details, prompt templates, and additional analyses of model size and visual-token reduction.

  • The supplement includes a literature review and additional details on dataset construction and statistics.Source: Supplementary Material · p. 27
  • It provides further evaluation-strategy details and question-answering judgment prompt templates for reasoning-based and surface questions.Source: Supplementary Material · p. 27
  • Additional sections cover question-answer generation and caption-answer extraction prompts, plus analyses of model size and visual-token reduction.Source: Supplementary Material · p. 27

S1. Related Works

The supplementary passages review visual-token compression and describe parts of Video-MMLU construction, including caption-based question-answer generation and its prompt examples.

S1.1. Visual Token Compression

  • Visual-token compression methods include pooling, downsampling, convolution, joint training, and training-free selection or merging based on similarity, query relevance, or information content.The review also mentions methods for grounding and long-form video processing.Source: S1.1. Visual Token Compression · p. 27

S2. More Details About Video-MMLU Construction

S3. Question-answer Pairs Generation Prompt Template of Video-MMLU

  • Claude-3.5-sonnet generates short question-answer pairs from structured detailed captions, with prompt examples showing questions about lecture visuals and mathematical content.The prompt asks for open-ended questions and short answers, followed by manual checking and refinement.Source: S3. Question-answer Pairs Generation… · pp. 27–29

S4. Predicted Answer Extraction Prompt Template

The template uses Claude-3.5-sonnet to extract predicted answers from generated captions, answering questions based only on detailed video or image descriptions. It directs the model to respond briefly.

S4. Predicted Answer Extraction Prompt Template

S5. More details about Video-MMLU Statistics

Video-MMLU videos are collected from YouTube, primarily from ten creators. The supplied figures show word distributions for detailed captions and reasoning question-answer pairs.

S5. More details about Video-MMLU Statistics

S6. More Details about the Evaluation Strategies

The evaluation strategies specify proprietary model versions and use distinct prompt instructions for visual question answering and video captioning. They also identify the assistant used for LLM evaluation and acceleration.

S6. More Details about the Evaluation Strategies

S7. Correctness Evaluation for Detailed Captioning Prompt Template

The procedure uses Qwen2.5-72B to judge predicted answers against correct answers, returning a binary judgment and a relative score. It applies stricter matching to OCR questions than to other questions.

S7. Correctness Evaluation for Detailed Captioning Prompt Template

S8. Correctness Evaluation for Reasoning QA Prompt Template

The reasoning-QA evaluation uses an LLM judge to assess answer correctness and assign a relative score. Its prompt requires essential concepts and details from the correct answer, while allowing differences in phrasing.

S8. Correctness Evaluation for Reasoning QA Prompt Template

  • Qwen2.5-72B judges predicted answers for correctness and assigns a relative score from 0 to 5.The judge receives the question, correct answer, and prediction, and returns a True/False judgment with the score.Source: S8. Correctness Evaluation for… · p. 33
  • A prediction is incorrect if it omits an essential concept or example, alters key information, or adds unrelated information.Minor phrasing differences are acceptable when the key meaning and concepts are preserved.Source: S8. Correctness Evaluation for… · p. 33
  • The output format is a Python dictionary string with a yes/no prediction and an integer score, as illustrated by the RNA-polymerase example.The example response is {’pred’: ’yes’, ’score’: 5}.Source: S8. Correctness Evaluation for… · p. 33

S9. More Analysis About the Model Size and Overall Performance

Larger models often perform better on Video-MMLU, but size alone does not explain performance: scaling is uneven, and proprietary models outperform open-source models. Mid-sized models can also surpass larger ones.

S9. More Analysis About the Model Size and Overall Performance

  • The supplementary table presents results for visual-token compression models on Video-MMLU.The supplied table caption gives its scope but no cell values.Source: Table J1 · p. 34
  • Proprietary models significantly outperform open-source models, while larger models generally do better but the scaling trend is not strictly linear.The figure relates model size to average video-captioning and question-answering performance; some mid-sized models around 8B–13B parameters may outperform larger ones.Source: S9. More Analysis About the Model Size… · pp. 33–34, Figure I4 · p. 34
  • InternVL2.5 improves consistently as model size increases, whereas some other models show fluctuating gains.The reported examples of fluctuating gains are [80] [117].Source: S9. More Analysis About the Model Size… · p. 34

S10. More Analysis about the Visual token Reduction

Token reduction can substantially compress video inputs, but its effects vary across tasks and disciplines. Moderate merging can be effective, while reasoning-heavy and time-dependent tasks remain more sensitive to compression.

S10. More Analysis about the Visual token Reduction

  • The token-reduction analysis varies retained visual tokens per frame from 49 to 730 to test effects on captioning and reasoning-based QA.Token merging is described as a core AuroraCap [21] design for reducing visual-token redundancy.Source: S10. More Analysis about the Visual… · pp. 34–35
  • Most models retain over 80% of peak performance with only 20%–40% of visual tokens, although performance is not strictly monotonic.The analysis covers detailed captioning and reasoning-based QA across disciplines.Source: S10. More Analysis about the Visual… · p. 35, Figure J5 · p. 35
  • Captioning stays relatively stable, especially in mathematics and chemistry, while QA fluctuates sharply in physics and chemistry and is more token-sensitive.The findings suggest static cues such as formulas are more resilient than dynamic visual elements.Source: S10. More Analysis about the Visual… · p. 35
  • Moderate token merging can improve efficiency, but excessive compression loses information, particularly for reasoning-heavy tasks.AuroraCap [21] focuses on spatial token merging without explicitly addressing temporal dependencies, motivating additional strategies for time-dependent visual reasoning and task-specific compression.Source: S10. More Analysis about the Visual… · pp. 35–36
  • Across most token ratios, AuroraCap [21] QA scores exceed captioning scores, with the largest gap at mid-range ratios of 0.4–0.6.At ratios below 0.2, both QA and captioning performance drop sharply.Source: S10. More Analysis about the Visual… · p. 36, Figure J6 · p. 36
  • The task gap varies by discipline: mathematics shows a relatively small gap, while physics QA starts lower at high token counts and surpasses captioning as fewer tokens remain.Chemistry shows the widest gap favoring QA, particularly at mid-range ratios; reasoning benefits from retaining structured visual elements and textual annotations.Source: S10. More Analysis about the Visual… · pp. 36–37
  • For an Arithmetic Mean-Root Mean Square Inequality demonstration, AuroraCap [21] captions become progressively shorter as the visual-token kept ratio increases.The analysis presents captions at different token kept ratios for this video.Source: S10. More Analysis about the Visual… · p. 37
Loading 2504.14693v2…