Source-linked AI summary
Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, Ziwei Liu
TL;DR
Existing video benchmarks do not systematically measure how LMMs acquire knowledge from educational videos. Video-MMMU addresses this gap with a multidisciplinary benchmark organized around perception, comprehension, and adaptation, and evaluations reveal progressively declining model performance and smaller knowledge gains than humans.
Problem
Existing video benchmarks lack systematic evaluation of LMMs’ ability to acquire knowledge from videos, despite video’s role in progressing from information perception to practical application.
Method
Video-MMMU evaluates LMM knowledge acquisition using 300 expert-level educational videos, stage-aligned questions across Perception, Comprehension, and Adaptation, and the ∆knowledge metric.
Results
Models show progressively lower performance from perception through comprehension to adaptation, while ∆knowledge is 33.1% for humans, 15.6% for GPT-4o, and 11.4% for Claude-3.5-Sonnet.
Takeaways & Limitations
Video-MMMU indicates that current LMMs struggle particularly to learn and apply knowledge from videos in novel practical scenarios.
Takeaways & Limitations
The benchmark’s comparison highlights a fundamental limitation of current LMMs: they struggle to understand, remember, and apply information from video content as humans do.
Abstract
from arXiv · showhide
Humans acquire knowledge through three cognitive stages: perceiving information, comprehending knowledge, and adapting knowledge to solve novel problems. Videos serve as an effective medium for this learning process, facilitating a progression through these cognitive stages. However, existing video benchmarks fail to systematically evaluate the knowledge acquisition capabilities in Large Multimodal Models (LMMs). To address this gap, we introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across six disciplines, evaluating knowledge acquisition through stage-aligned question-answer pairs: Perception, Comprehension, and Adaptation. A proposed knowledge gain metric, Δknowledge, quantifies improvement in performance after video viewing. Evaluation of LMMs reveals a steep decline in performance as cognitive demands increase and highlights a significant gap between human and model knowledge acquisition, underscoring the need for methods to enhance LMMs' capability to learn and adapt from videos.
1. Introduction
Video-MMMU addresses the lack of systematic evaluation of how LMMs acquire knowledge from educational videos across perception, comprehension, and adaptation. Its evaluations show that model performance declines with deeper cognitive demands and that model knowledge gains remain below human gains.
- Video-MMMU evaluates three stages of knowledge acquisition: perceiving information, comprehending knowledge, and adapting knowledge to solve novel problems.The framework aligns video-based learning with three cognitive stages described in Bloom’s taxonomy.
- 300 expert-level videos span six professional disciplines, and each video includes question-answer pairs aligned with the three knowledge-acquisition stages.The disciplines are Art, Business, Science, Medicine, Humanities, and Engineering.
- Model performance decreases progressively from perception to comprehension and then adaptation as cognitive demands increase.Models perform relatively better on perception tasks, with notable declines on comprehension and further declines on adaptation.
- ∆knowledge shows that humans achieve ∆knowledge = 33.1% after viewing videos, compared with GPT-4o: ∆knowledge = 15.6% and Claude-3.5-Sonnet: ∆knowledge = 11.4%.The metric measures performance improvement on practical exam questions after video learning.
- The results identify current LMM video learning as a challenge and motivate research on improving how models acquire and use video-based information.
- Existing video benchmarks mainly assess visual understanding, temporal reasoning, captioning, multitasking, and domain understanding rather than knowledge acquisition from videos.
3. Video-MMMU Dataset
Video-MMMU is a multidisciplinary dataset of college-level educational videos designed to test progressively deeper knowledge acquisition. It combines curated concept-introduction and problem-solving videos with stage-aligned questions and multi-stage quality assurance.
- Video-MMMU covers 30 subjects across six professional disciplines: Art, Business, Medicine, Science, Humanities, and Engineering.The benchmark is designed to evaluate knowledge acquisition from educational videos across these domains.
- The dataset contains 300 college-level educational videos curated through topic selection, candidate-video search, and three-tier quality assurance.Domain experts selected assessment topics, GPT-4o generated search queries and assessed technical depth, and annotators filtered irrelevant or poor-quality videos.
- Video Types: Concept-introduction videos explain factual knowledge, fundamental concepts, and theories, while problem-solving videos demonstrate step-by-step solutions.Problem-solving videos particularly emphasize systematic reasoning and detailed calculations in STEM disciplines.
- Question Design: Questions are annotated across Perception, Comprehension, and Adaptation, assessing progressively deeper levels of knowledge acquisition.Perception questions include extracting formulas, data points, charts, and handwritten notes; Adaptation requires adjusting methods for new problems.
- Annotation Process: Each video receives three questions aligned with the cognitive tracks, with cross-checking, model-assisted refinement, and expert review supporting annotation quality.Multiple-choice questions contain 10 options to increase assessment rigor.
- Video Content: The videos average 506.2 seconds and provide dense domain-specific information through lectures, tutorials, demonstrations, charts, diagrams, and handwritten explanations.
4. Experiments
Experiments evaluate LMMs and human experts across perception, comprehension, and adaptation tracks, finding progressively weaker performance as cognitive demands rise. Performance also varies by discipline, and audio transcripts improve perception and comprehension but reduce adaptation performance.
- Performance by Track: Human experts outperform models across all three tracks, with both groups declining from perception to comprehension to adaptation.Claude achieves the highest model scores but remains below human performance.
- Performance by Track: Most models exceed 50% accuracy on perception, while adaptation is the most difficult track and usually falls below 50%.Comprehension scores also decline relative to perception, especially for open-source models, which typically drop 10 ∼20%.
- Performance by Discipline: Models perform better in Art and Humanities than in Science, Engineering, Business, and Medicine.The latter disciplines require quantitative reasoning and interpretation of detailed technical visuals, including diagrams and handwritten notes.
- Impact of Audio Transcript: Audio transcripts improve overall performance, with the largest enhancement in comprehension and additional gains in perception.These gains suggest audio supports video content understanding and information extraction.
- Impact of Audio Transcript: Audio transcripts decrease adaptation performance, contrasting with their improvements on perception and comprehension.The results indicate a trade-off between immediate understanding and adapting knowledge to novel scenarios.
5. Knowledge Acquisition in Adaptation Track
The Adaptation track measures knowledge gain after video viewing and shows that models often acquire information imperfectly: gains are offset by answer reversals, with method adaptation remaining a major challenge.
- Knowledge Gain Metric: Δknowledge measures performance improvement on practical Adaptation questions after watching the video.It is defined as the difference between post-video and pre-video accuracy, with normalization accounting for baseline difficulty levels.
- Human-Model Gap: 33.1% human Δknowledge versus 15.6% for GPT-4o demonstrates a substantial human-model knowledge acquisition gap.Some models even show negative Δknowledge after video exposure.
- Response Changes: 29.5% Wrong-to-Right performance for Gemini-1.5-Pro shows models can acquire some video knowledge, but 55.0% Right-to-Wrong for LongVA offsets such gains.The results indicate difficulty maintaining initially correct answers while processing new video information.
- Response Changes: Humans achieve a 40.4% Wrong-to-Right Rate and a 10.7% Right-to-Wrong Rate, indicating stronger integration of new knowledge while preserving prior knowledge.These rates compare answers before and after video viewing.
- Error Analysis: Method Adaptation errors account for 64% of Claude-3.5-Sonnet’s sampled Adaptation-track errors, despite correct recall and understanding of demonstrated methods.The reported example involves applying DFS principles from a simple tree to a more complex graph with cycles.
- Error Analysis: The findings identify a gap between comprehension and adaptation, with novel practical application remaining challenging for current models.Question misreading and method selection errors also limit successful knowledge application.
6. Conclusion
Video-MMMU evaluates video-based knowledge acquisition across Perception, Comprehension, and Adaptation, using Δknowledge to expose a human–model performance gap, especially in applying knowledge to novel practical scenarios.
- Video-MMMU evaluates LMM knowledge acquisition from videos across Perception, Comprehension, and Adaptation.
- Δknowledge reveals a gap between human experts and models, particularly in adapting acquired knowledge to novel practical scenarios.
- The benchmark motivates future research on improving LMMs’ ability to learn and apply knowledge from video content.
7. Subjects by Discipline
Table 3 categorizes the benchmark’s subjects under six disciplines.
- Table 3 organizes Video-MMMU subjects by discipline.
8. Additional Knowledge Acquisition Experiment Results
The additional experiment examines knowledge changes after video viewing, including answer transitions and the practical challenges models face when acquiring and applying new information.
- Table 4 breaks down questions transitioning from Wrong-to-Right and Right-to-Wrong, together with their corresponding rates.
- The Δknowledge experiment reveals a human–model gap in learning new information from videos.
- Models are not yet capable of effectively acquiring new knowledge from video and applying it in practice.
- Figure 8 presents the prompt used for the Adaptation track.
- Future improvements must address models’ ability to understand, remember, and apply information from video content.
- The adaptation track appends each question’s image to the end of its video.
10. Prompt for Determining the Helpfulness of Audio
The appendix describes audio-helpfulness analysis for video-question pairs and points to prompts and an annotation-pipeline illustration supporting benchmark construction.
- 10. Prompt for Determining the Helpfulness of Audio: Gemini 1.5 Pro analyzes every video-question pair for potential audio usefulness.
- 10. Prompt for Determining the Helpfulness of Audio: The audio analysis identifies whether audio might help solve each question.
- 10. Prompt for Determining the Helpfulness of Audio: The analysis is intended to benefit future LMMs with audio-processing capabilities.
- 10. Prompt for Determining the Helpfulness of Audio: Figure 9 contains the prompt used to determine audio helpfulness.
- 11. Annotation Pipeline: Figure 10 illustrates the pipeline for video collection and question-answer annotation.
- 11. Annotation Pipeline: The annotation pipeline covers both video collection and QA annotation.
12. More Error Analysis
The paper analyzes representative model errors across the Adaptation, Perception, and Comprehension tracks. It also references additional knowledge-acquisition results reported with Delta (%) values.
- Adaptation: Adaptation errors include method-selection, question-misreading, and method-adaptation failures across Claude-3.5-Sonnet and GPT-4o.Figures 11–14 provide representative cases for these error types.
- Additional Results: Additional knowledge-acquisition experiment results are reported with Delta (%) values.The supplied passage identifies these results as Table 4.
- Perception and Comprehension: Perception and Comprehension each receive two representative error-case analyses.The cases are presented in Figures 15–18 with detailed analyses of the observed errors.
13. Wrong-to-Right Case Analysis
The paper examines cases where models successfully learn from video content and then answer Adaptation questions correctly. It organizes this analysis alongside the dataset curation pipeline, error cases, and an audio-helpfulness evaluation prompt.
- Wrong-to-Right Examples: Wrong-to-Right examples show models successfully learning from video content to correctly solve Adaptation questions.The analysis includes three Claude-3.5-Sonnet examples and one GPT-4o example.
- Model Examples: Claude-3.5-Sonnet Wrong-to-Right examples appear in Figures 19–21, while GPT-4o has an example in Figure 22.Each case study analyzes how the model successfully adapted knowledge.
- Audio Evaluation: Figure 9 provides a prompt for determining whether audio is helpful for answering a video question.The prompt evaluates whether a question requires audio information or can be answered using visual information alone.
- Dataset Curation: The dataset curation pipeline is illustrated separately in Figure 10.The supplied passage identifies the figure as an illustration of dataset curation.