Source-linked AI summary
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig
TL;DR
MMMU-Pro addresses whether multimodal benchmarks measure genuine understanding or allow models to exploit shortcuts. It filters text-solvable questions, expands answer choices, and adds vision-only inputs with embedded questions. Across tested models, performance falls 16.8% to 26.9% versus MMMU, while CoT generally helps and OCR prompts usually have little effect.
Problem
Existing MMMU results leave open whether models genuinely comprehend multimodal content or exploit shortcuts and statistical patterns.
Method
MMMU-Pro filters text-only-solvable questions, expands candidate options, and presents some questions as embedded text within screenshots or photos.
Results
16.8% to 26.9%: all tested models showed significant performance drops on MMMU-Pro relative to MMMU; CoT generally improved results, while OCR prompts usually had little effect.
Takeaways & Limitations
MMMU-Pro provides a more rigorous benchmark that exposes current multimodal models’ limitations and highlights vision-text integration and reasoning as research directions.
Takeaways & Limitations
The dataset may retain statistical shortcuts, covers predefined disciplines and formats, does not fully capture human perception, and relies on approximated human performance.
Abstract
from arXiv · showhide
This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.
1 Introduction
MMMU-Pro is introduced to test whether multimodal models genuinely integrate visual and textual information rather than exploit shortcuts. Its construction filters text-solvable questions, expands answer choices, and embeds questions in images; performance drops substantially relative to MMMU.
- Motivation: MMMU results raise whether models use genuine comprehension or exploit textual shortcuts and statistical patterns.This matters because superficial cues may lead to overestimated capabilities and unpredictable failures in novel scenarios.
- Contribution: MMMU-Pro targets more rigorous assessment of multimodal understanding and reasoning across academic disciplines.Its design responds to text-only solvability, limited multiple-choice options, and insufficiently integrated modality processing.
- Construction: The benchmark filters text-answerable questions, augments candidate options, and introduces vision-only questions embedded in screenshots or photos.The three-step construction is designed to reduce shortcut use and require joint visual-textual processing.
- Vision-only setting: Vision-only input tests simultaneous visual reading and interpretation, reflecting how people process screenshots, photos, diagrams, and interfaces.The setting is motivated by both human cognition and natural user interactions with AI systems.
- Results: 16.8% to 26.9%: all tested models showed performance decreases on MMMU-Pro compared with MMMU.CoT generally improved performance, but its benefits varied across models and settings.
- Analysis: Explicit OCR prompts generally had little effect, while embedded text increased visual-input complexity beyond simple text extraction.Models must recognize text and relate it to visual elements and the question.
2 MMMU-Pro: A More Robust Version of MMMU
MMMU-Pro strengthens MMMU by addressing text-only solvability and answer-option shortcuts. It filters questions, expands candidate choices, and adds a vision-only setting to reduce guessing and increase multimodal difficulty.
- MMMU benchmark: MMMU evaluates college-level multimodal tasks across six disciplines, 30 subjects, and 183 subfields using 11.5K image-text questions.Each question contains four multiple-choice options and diverse visual formats including charts, diagrams, maps, and chemical structures.
- Benchmark issues: Text-only LLMs can answer some MMMU questions because images are irrelevant or candidate options expose exploitable shortcuts.This undermines the requirement for visual input in those cases.
- Filtering Questions: The construction filters questions answerable by text-only models and samples 1,800 remaining questions evenly across 30 subjects.The filtering procedure uses repeated answers from four open-source LLMs to identify questions for exclusion.
- Augmenting Candidate Options: Candidate options increase from four to ten, with human experts and GPT-4o supporting augmentation and validation.The expanded option space makes guessing based on answer choices more difficult.
- Effect on text-only models: The two construction steps significantly reduce text-only models’ accuracy when guessing answers.The reduction is reported for the filtered and option-augmented question sets.
- Vision-only Setting: The vision-only setting presents questions embedded in screenshots or photos without separately supplied text.Annotators vary backgrounds, fonts, and font sizes to represent real-world visual conditions.
3 Experiments
MMMU-Pro evaluates a broad set of proprietary and open-source multimodal models under increasingly difficult input and option settings. Results show substantial overall performance declines, varied CoT effects across models and disciplines, and a weak relationship between OCR accuracy and multimodal reasoning.
- Experimental Setup: The evaluation covers proprietary and open-source multimodal models across standard, augmented-option, and vision-only settings.The reported overall MMMU-Pro score averages the 10-option standard and vision-only settings; original MMMU validation performance is included for comparison.
- Overall Results: 10.7%: GPT-4o’s accuracy fell from 64.7% to 54.0% when candidate options increased from 4 to 10.The option increase was intended to reduce the likelihood of guessing from the candidate set.
- Overall Results: 16.8%–26.9%: models’ performance decreased relative to MMMU (Val), including an 18.9% decline for Gemini 1.5 Pro and a 26.9% decline for VILA-1.5-40B.The reductions are presented as evidence that MMMU-Pro mitigates shortcuts and guessing strategies available in the original benchmark.
- Impact of CoT Prompting: CoT generally improved performance, but gains varied by model and discipline, with GPT-4o improving 14.49% in Tech and Engineering and 8.22% in Science.Benefits were smaller for LLaVA-OneVision 72B and could become negative, including a 17.12% decline for that model in Art and Design.
- OCR and Vision Reasoning: OCR prompts produced minimal performance changes, while high OCR accuracy did not consistently predict strong MMMU-Pro Vision reasoning.LLaVA-OneVision-72B matched InternVL2-Llama3-76B and GPT-4o mini in OCR accuracy but lagged in Vision performance; GPT-4o performed strongly in both areas yet still dropped relative to MMMU.
4 Guide for Future Model Training
The paper identifies training directions for improving multimodal models, including scaling backbones, strengthening feature fusion, and generating text-rich images for reasoning.
- Scaling of LLM Backbones: Larger LLM backbones consistently enhance perception and reasoning capabilities on MMMU-Pro.GPT-4o outperforms GPT-4o mini, while LlavaOneVision-72B achieves better results than smaller counterparts.
- Feature Fusion: Effective feature fusion is critical for bridging multimodal gaps.
- CoT Data Generation: CoT prompting benefits reasoning-heavy domains, but its impact varies across disciplines and can be detrimental in Art and Design.Tech and Engineering and Business show notable improvements, motivating more diverse reasoning-intensive CoT data and domain-tailored strategies.
- Text-Rich Image Generation in Reasoning Scenarios: Strong OCR accuracy and traditional benchmark reasoning performance do not always transfer to MMMU-Pro Vision.The paper suggests that limited training data containing text-rich images in reasoning-intensive contexts may contribute to this gap.
5 Related Work
Prior multimodal models have advanced substantially, but existing benchmarks remain insufficient for evaluating comprehensive multimodal understanding and can permit text-only shortcuts. MMMU-Pro addresses these limitations with harder, more realistic evaluation settings.
- Multimodal Large Language Models: Recent multimodal models, including GPT-4V, GPT-4o, Gemini, and Claude-3.5, demonstrate strong vision-language performance.
- MLLM Benchmarks: Earlier benchmarks such as VQA, OK-VQA, and MSCOCO no longer suffice to evaluate the full spectrum of LMM capabilities.Newer benchmarks expanded multimodal and reasoning evaluation, but MMMU still allowed some text-only shortcuts.
- MLLM Benchmarks: MMMU-Pro removes text-only answerable questions, expands candidate options, and adds a vision-only input setting.These changes are intended to better reflect real-world multimodal scenarios.
6 Conclusion
MMMU-Pro is presented as a stronger benchmark that exposes substantial limitations in current multimodal systems and points to gaps in consistent cross-setting performance, vision-text integration, and reasoning.
- 6 Conclusion: MMMU-Pro exposes significant performance drops across all tested systems relative to MMMU.
- 6 Conclusion: Future model development should target consistent performance across standard and vision-only settings.
- 6 Conclusion: The benchmark highlights the need for better vision-text integration on complex mixed-format inputs.
- 6 Conclusion: MMMU-Pro also motivates reasoning techniques suited to its heightened question complexity.
- 6 Conclusion: The benchmark excludes sensitive content and focuses assessment on multimodal capabilities without introducing bias.
- Limitations: MMMU-Pro remains limited by possible statistical shortcuts, predefined disciplines and formats, incomplete human-perception coverage, and approximated human performance.
B Approximating Human Expert Performance
The paper approximates MMMU-Pro human performance from existing MMMU evaluations to avoid new expert-assessment costs, while using staged validation to improve augmented answer options.
- Approximating Human Expert Performance: Existing MMMU human evaluation data are used to approximate human performance on MMMU-Pro instead of conducting new evaluations.The approach is motivated by the time and expense of rigorous human assessment.
- Approximating Human Expert Performance: The approximation relies on unchanged core question content and difficulty between MMMU and MMMU-Pro.Original experts also documented problem-solving processes, reducing the likelihood of random guessing.
- Approximating Human Expert Performance: The method categorizes 577 MMMU-Pro questions using evaluations from 90 original human experts according to solution-process availability and answer correctness.
- Approximating Human Expert Performance: A conservative lower-bound estimate combines correctly solved questions with detailed solutions and the proportion of correctly guessed questions without them.
- Candidate Option Augmentation: Automated option generation and filtering are followed by two rounds of human review to improve the quality and reliability of expanded candidates.Reviewers assess diversity, logical distinction, ambiguity, and residual inconsistencies.
D Analysis of CoT’s Impact
Figure 9 compares Chain-of-Thought and direct accuracy across subcategories and major domains for GPT-4o and LLaVA-OneVision 72B.
- CoT and direct accuracy are compared across subcategories within major domains for GPT-4o and LLaVA-OneVision 72B.
E Experimental Setup of Vision Encoder Impact
This section evaluates vision-encoder effects using a controlled Cambrian-1 setup and presents comparisons across input and option settings.
- Experimental Setup: Cambrian-1 experiments fix the training data and Llama 3.1 8B to isolate the impact of different vision encoders.The setup follows Cambrian-1’s methodology and interpolates visual features to a fixed number of tokens.
- Input Settings: Figure 10 compares GPT-4o responses between Standard and Vision Input settings.
- Augmented Options: Figure 11 compares GPT-4o responses with and without augmented options.
- Output Comparisons: Figure 12 highlights descriptions in orange and analyses in light red when comparing GPT-4o outputs across settings.
- Output Comparisons: Figure 13 likewise highlights descriptions in orange and analyses in light red across different GPT-4o input settings.
- Appendix Examples: The appendix provides different input-setting examples for Art and Design, including Art, Art Theory, Design, and Music.
- Appendix Examples: The appendix also provides different input-setting examples for Business subfields including Accounting and Economics.
Business: Finance
The Finance examples compare standard and vision-setting solutions for internal-rate-of-return questions, while adjacent examples report model outputs on other disciplines.
- Business: Finance: The finance ground truth is Project A: 22% and Project B: 23.37%.
- Business: Finance: GPT-4o’s standard-setting analysis uses cash-flow schedules and the NPV-equals-zero definition of IRR.
- Business: Finance: The standard-setting calculation reports IRR = 22 for Project A before continuing with Project B.
- Business: Finance: The vision-setting analysis formulates IRR calculations from the cash flows of Projects A and B.
- Business: Finance: The vision-setting result reports IRR = 28.69 and selects answer D.
- Business: Finance: The analysis states that both settings require correctly extracting cash flows.
- Other Examples: Additional passages show standard-setting outputs and ground truths for unemployment, embryonic development, and circuit-voltage questions.
J.25 Tech and Engineering: Architecture and Engineering
The Architecture and Engineering section includes appendix input-setting examples and examples of circuit and heat-engine reasoning outcomes.
- Architecture and Engineering: Figure 38 provides an example of a different input setting in Architecture and Engineering.
- Appendix Examples: Figures 39–43 provide different input-setting examples for Computer Science, Electronics, Energy and Power, Materials, and Mechanical Engineering.
- Electronics: The circuit example’s ground truth is (B) -8v -10v 20v.
- Electronics: The analysis states that the vision setting correctly identifies voltage drops and the expected answer, while the standard setting is not characterized here as correct.
- Electronics: The standard-setting circuit solution analyzes initial inductor conditions and currents after the switch is thrown.
- Electronics: A later circuit calculation reports an answer of J after considering intermediate differences and possible errors.
- Energy and Power: For the heat-engine question, both Vision and Standard settings attempt first-law and second-law efficiencies but produce incorrect results.