PaperlayerView exact revision on arXiv

Source-linked AI summary

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig

arXiv:2409.02813v3cs.CLcs.CV

TL;DR

Do current benchmark results reflect deep, multifaceted understanding, or can models exploit shortcuts and statistical patterns? MMMU-Pro filters text-answerable questions, expands answer options, and adds vision-only inputs; performance drops by 16.8% to 26.9% across tested models compared with MMMU.

  • Problem

    The paper asks whether benchmark results reflect deep, multifaceted understanding of diverse subjects or whether models can exploit shortcuts and statistical patterns.

    Source: §1 · p. 1

  • Method

    MMMU-Pro filters questions answerable by text-only models, augments candidate options, and adds a vision-only setting with questions embedded in images.

    Source: §1 · p. 1

  • Results

    Performance drops across all tested models on MMMU-Pro versus MMMU, with decreases ranging from 16.8% to 26.9%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MMMU-Pro highlights research directions in cross-setting consistency, vision-text integration, and reasoning for more complex questions.

    Source: §6 · p. 9

    The dataset’s scope is limited to predefined disciplines and question formats.

    Source: Limitations · p. 10

Abstract

from arXiv · show

This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.

MMMU Team · 1 Introduction

MMMU-Pro is designed to test multimodal understanding more rigorously by filtering text-solvable questions, expanding answer options, and embedding questions in images. The introduction reports lower performance than on MMMU and examines how CoT and OCR prompts affect results.

1 Introduction

  • GPT-4o achieved 69.1% accuracy on college-level MMMU questions integrating text and images, amid broader MLLM progress on multimodal reasoning tasks.Source: §1 · p. 1
  • The introduction questions whether benchmark success reflects genuine subject understanding or shortcuts, warning that superficial cues may lead to capability overestimates and failures in novel scenarios.Source: §1 · p. 1
  • MMMU-Pro addresses text-only solvability, limited multiple-choice options, and modality integration through text-solvability filtering, option augmentation, and vision-only question images.Source: §1 · p. 1
  • The vision-only setting tests models’ ability to read and see simultaneously, reflecting tasks such as interpreting scientific diagrams and navigating graphical interfaces.Source: §1 · pp. 1–2
  • 16.8% to 26.9% decreases across tested models versus the original MMMU; CoT generally improves performance, with benefits varying by model and setting.Source: §1 · p. 2
  • Explicit OCR prompts do not significantly affect performance for most models, but OCR alone is insufficient for vision-only challenges involving text context and visual relationships.Source: §1 · p. 2

2 MMMU-Pro: A More Robust Version of MMMU

MMMU-Pro strengthens a college-level multimodal benchmark by filtering questions answerable without images, expanding answer options, and testing questions embedded in screenshots or photos. These changes target text-only shortcuts while evaluating models on integrated visual-textual input.

2.1 Revisiting the MMMU Benchmark

  • MMMU contains 11.5K college-level multimodal questions across six disciplines, 30 subjects, and 183 subfields, each pairing an image with four answer options.Its images span 30 types, including charts, diagrams, maps, and chemical structures.Source: §2.1 · p. 2
  • Text-only LLMs can answer some MMMU questions through image-independent wording or shortcuts and correlations in candidate options.These issues motivate a more robust benchmark.Source: §2.1 · p. 2

2.2 Methods

  • MMMU-Pro filters text-answerable questions and expands options from four to ten, yielding 1,730 questions after image-relevance review.Four open-source LLMs answer without images across ten trials; questions are excluded when at least three models answer correctly across most trials. Experts, GPT-4o, and Claude 3.5 help generate, filter, and review the expanded options.Source: §2.2 · pp. 2–3
  • The filtering and option-augmentation steps significantly reduce text-only model accuracy on MMMU question sets.The supplied passage gives no accuracy values.Source: §2.2 · p. 3
  • MMMU-Pro adds a vision-only setting that presents questions embedded in screenshots or photos, producing 1,730 screenshot/photo questions alongside 1,730 standard-format questions.Annotators vary backgrounds, fonts, and font sizes to simulate real-world visual contexts; the setting reflects users’ tendency to capture text and images together.Source: §2.2 · p. 3

3 Experiments

MMMU-Pro tests multimodal models across standard and vision-only settings; compared with MMMU, its results show that more options and image-embedded questions make performance substantially harder to sustain.

3.1 Experimental Setups

  • The evaluation covers proprietary and open-source multimodal models in standard four-option, augmented ten-option, and vision-only settings, with overall MMMU-Pro scores averaging the latter two.Direct and CoT prompts were evaluated, with the higher score reported in the overall results; standard four-option and original MMMU validation results serve as comparison points.Source: §3.1 · p. 4

Approximating Human Expert Performance.

  • The authors approximate human expert performance on MMMU-Pro using original MMMU human results, reasoning that question content and difficulty remain unchanged and experts integrate visual and textual information.For responses lacking detailed solution processes, they randomly select an augmented candidate and recalculate accuracy; they expect human performance to remain similar in the vision-only format.Source: Approximating Human Expert Performance · pp. 4–5

3.2 Overall Results

  • 18.9%, 16.8%, and 26.9% drops from MMMU (Val) were reported for Gemini 1.5 Pro (0801), Claude 3.5 Sonnet, and VILA-1.5-40B, respectively.Increasing options from 4 to 10 reduced GPT4o (0513) from 64.7% to 54.0%; switching from 10-option standard to vision-only reduced GPT4o (0513) by 4.3% and LLaVA-OneVision-72B by 14.0%.Source: Table 1 · p. 5, §3.2 · pp. 5–6

3.3 Impact of CoT Prompting

  • CoT generally improved performance, but its benefits varied by model and discipline and could be negative when instruction-following or response formatting faltered.Claude 3.5 Sonnet rose from 42.7% to 55.0% in the standard setting; GPT4o gained 14.49% in Tech and Engineering and 8.22% in Science, while LLaVA-OneVision 72B declined 17.12% in Art and Design. VILA1.5-40B also showed a significant drop, which the authors associate with instruction-following and response-format problems.Source: §3.3 · p. 6

3.4 Does OCR Help in the Vision Setting?

  • OCR prompts did not significantly alter vision-setting performance, and OCR accuracy alone did not reliably indicate multimodal reasoning ability.LLaVA-OneVision-72B matched InternVL2-Llama3-76B and GPT-4o mini in OCR accuracy but lagged in MMMU-Pro Vision performance.Source: Table 2 · p. 7, §3.4 · p. 7, Figure 6 · p. 7

3.5 Qualitative Analysis

  • Qualitative cases reveal errors from confusing similar options, increased visual-text integration load, over-weighting visual cues, and context switching.Some models extracted image text correctly but still answered incorrectly, while switching between a textual description and a geometric figure led to misreading the feasible region.Source: §3.5 · p. 7

3.6 Error Analysis

  • Reasoning errors accounted for 46% of GPT-4o’s 60 vision-setting errors, up from 26% in the original MMMU; text recognition was not the primary perception bottleneck.The analysis instead identifies integration and interpretation of visual and textual information as key challenges.Source: Figure 7 · p. 8, §3.6 · p. 8

3.7 Response Length Comparison

  • GPT-4o produced shorter vision-setting responses, with more descriptive than analytical tokens, which the authors suggest may reflect visual-processing demands.Qwen2-72B-Instruct classified GPT-4o’s responses into descriptive and analytical sentences for this comparison.Source: Figure 8 · p. 8, §3.7 · p. 8

4 Guide for Future Model Training

The section identifies scaling, stronger visual representation learning, and improved vision–text integration as directions for multimodal model development. It also recommends more diverse reasoning-focused CoT data and training with text-rich images in reasoning contexts.

Scaling of LLM Backbones

  • Scaling LLM backbones consistently improves perception and reasoning, with larger GPT-4o, LlavaOneVision, and InternVL2 models outperforming their smaller counterparts.The section presents scaling as a critical factor for improving multimodal understanding and reasoning.Source: §4 · p. 8

More Capable Vision Encoders that Highlights Visual Representation Learning

  • Self-supervised DINOv2 ViT-G-14 performs better on MMMU-Pro Vision than language-supervised Siglip ViT-SO400M-14, which performs well on MMMU Val but struggles on MMMU-Pro Vision.The authors suggest enhancing visual feature learning while exploring ways to integrate language-based and self-supervised training objectives.Source: §4 · p. 8, Table 3 · p. 9

Better Integration of Vision and Text Modalities

  • Better cross-modal attention and effective feature fusion are needed because current architectures often struggle with tasks requiring deep vision–text understanding.The section frames improved integration of visual and textual information as a key challenge for multimodal models.Source: §4 · p. 8

CoT Data Generation

  • CoT improves performance in Tech and Engineering and Business but remains weak or detrimental in Art and Design, motivating diverse reasoning-focused data and domain-tailored strategies.The section also recommends leveraging inference-compute concepts (Welleck et al., 2024) to improve CoT generalization across reasoning tasks.Source: §4 · p. 9

Text-Rich Image Generation in Reasoning Scenarios

  • Strong OCR and traditional-benchmark reasoning do not guarantee success on MMMU-Pro Vision, motivating scalable training data with text-rich images in reasoning-intensive contexts.The authors developed a tool that embeds questions and images into screenshots using JSON inputs, based on the MMMU-Pro Vision human annotation process.Source: §4 · p. 9

5 Related Work

Recent multimodal models have advanced, but evaluating their capabilities remains challenging. Existing benchmarks often miss expert-level knowledge and complex reasoning, and MMMU still allows text-only shortcuts; MMMU-Pro addresses these gaps with a more demanding design.

Multimodal Large Language Models

  • Advanced multimodal models, including GPT-4V, GPT-4o, Gemini (Team et al., 2023), and Claude-3.5, perform strongly, but evaluating them remains challenging.The field has also developed instruction-following models inspired by large language models.Source: §5 · p. 9

MLLM Benchmarks

  • Existing benchmarks often fall short on expert-level domain knowledge and complex reasoning, while MMMU (Yue et al., 2024) still permits text-only answer shortcuts (Zhang et al., 2024b).Earlier benchmarks such as VQA (Goyal et al., 2017), OK-VQA (Marino et al., 2019), and MSCOCO (Lin et al., 2014) no longer suffice for evaluating the full spectrum of LMM capabilities.Source: §5 · p. 9
  • MMMU-Pro removes text-only-answerable questions, expands candidate options, and adds vision-only input to better reflect real-world multimodal scenarios.It is introduced to address the limitations of existing benchmarks and MMMU.Source: §5 · p. 9

6 Conclusion … Evaluation Prompts: Split Response Task

MMMU-Pro is presented as a stronger benchmark that exposes limitations in current multimodal models and points to research on cross-setting consistency, vision-text integration, and reasoning. The accompanying evaluation prompts specify OCR, direct-answer, CoT, OCR-only, and response-splitting tasks.

6 Conclusion

  • MMMU-Pro exposes significant performance drops across tested systems and motivates work on cross-setting consistency, vision-text integration, and reasoning.The conclusion also describes the benchmark as stronger than MMMU; its ethical statement says it excludes sensitive content, aims for unbiased assessment, and encourages transparent reporting and research on societal impacts.Source: §6 · p. 9, Ethical Statement · p. 10

Limitations

  • The dataset may retain exploitable statistical shortcuts, cover only predefined disciplines and formats, incompletely model human perception, and rely on approximated rather than directly evaluated human performance.The paper notes that the benchmark filters text-only-solvable questions and adds a vision-only setting, but these design choices do not remove the stated limitations.Source: Limitations · p. 10

Evaluation Prompts: OCR Prompt

  • The OCR prompt asks the model to transcribe the question in the image and solve it, while the no-OCR prompt asks it to answer the pictured multiple-choice question directly.Both prompts request step-by-step thinking and a final answer in letter format.Source: Evaluation Prompts: OCR Prompt · p. 15

Evaluation Prompts: Direct vs CoT

Evaluation Prompts: OCR Task

  • The OCR task prompt instructs the model to extract the full question and answer choices, exclude associated-image text and question numbers, and not solve the question.Source: Evaluation Prompts: OCR Task · p. 15

Evaluation Prompts: Split Response Task

  • The split-response task separates an answer into a question description and an answer analysis, preserving all input content without adding anything.It requests JSON fields named description_of_question and analysis_of_answer, with the description providing context and the analysis giving a complete explanation or solution.Source: Evaluation Prompts: Split Response Task · p. 15

B Approximating Human Expert Performance · C Ensuring Quality and Diversity of Expanded Options

The authors estimate human performance on MMMU-Pro from existing MMMU evaluations rather than conducting costly new evaluations, and use automated filtering plus human review to validate expanded answer options.

B Approximating Human Expert Performance

  • The approximation assumes MMMU-Pro preserves the original questions' content and difficulty, detailed solution records reduce guessing, and experts integrate visual and textual information across formats.For questions without detailed solutions, the method simulates random selection from the expanded options.Source: B Approximating Human Expert Performance · p. 16
  • The estimate draws on 577 MMMU validation questions and evaluations by 90 human experts, categorized by solution-process documentation and answer correctness.Table 5 summarizes the counts for questions with and without detailed solutions.Source: B Approximating Human Expert Performance · p. 16, Table 5 · p. 17
  • The lower-bound estimate combines correctly solved questions with detailed solution processes and the proportion of correct guesses among questions lacking them.This produces a conservative performance estimate.Source: B Approximating Human Expert Performance · p. 16
  • The authors provide a reasonable approximation of human performance on MMMU-Pro from original MMMU evaluations, avoiding the cost of new expert evaluations.The estimates are reported across disciplines at low, medium, and high performance levels.Source: B Approximating Human Expert Performance · p. 16, Table 4 · p. 16

C Ensuring Quality and Diversity of Expanded Options

  • GPT-4o generated expanded options, and Claude 3.5 filtered candidates that were contextually irrelevant or logically inconsistent before human review.The preliminary filtering reduced reviewers' workload.Source: C Ensuring Quality and Diversity of… · p. 18
  • Reviewers first checked each question's options for diversity, logical distinctness, and ambiguity, then two additional experts cross-validated each question and its options.Reviewers corrected flaws or created replacement options; the second round checked for residual errors and inconsistencies.Source: C Ensuring Quality and Diversity of… · p. 18
  • Combining automated methods with multi-stage human validation improved the quality, robustness, and reliability of expanded options.The process was designed to ensure options aligned with the benchmark's intended challenges.Source: C Ensuring Quality and Diversity of… · p. 18

D Analysis of CoT’s Impact

The section compares CoT and direct accuracy across subcategories within major domains for GPT-4o and LLaVA-OneVision 72B.

D Analysis of CoT’s Impact

  • Figure 9 compares CoT and direct accuracy across subcategories within major domains for GPT-4o and LLaVA-OneVision 72B.Source: Figure 9 · p. 19

E Experimental Setup of Vision Encoder Impact … J.7 Business: Finance

The appendices describe a controlled vision-encoder evaluation and provide comparisons and qualitative examples across input settings, augmented options, disciplines, and fields.

E Experimental Setup of Vision Encoder Impact

F Comparison of GPT-4o’s responses between Standard and Vision Input settings

  • Figure 10 compares GPT-4o’s responses between Standard and Vision Input settings.Source: Figure 10 · p. 20

G CoT vs. Direct Acc: Model Differences Across Disciplines

  • Table 6 compares CoT and direct accuracy for two representative models across disciplines in the Vision Input setting.It defines the difference as CoT accuracy minus direct accuracy; the supplied passage does not provide cell values.Source: Table 6 · p. 21

H Comparison With and Without Augmented Options

I Comparison of Model Outputs Across Different Input Modes

  • In the elliptical-orbit example, GPT-4o’s Standard Setting response identifies H, the sun might be at point B, as correct; the supplied ground-truth label is also H, though its text says point A.Source: I Comparison of Model Outputs Across… · p. 22

J.1 Art and Design: Art; J.2 Art and Design: Art Theory; J.3 Art and Design: Design; J.4 Art and Design: Music; J.5 Business: Accounting; J.6 Business: Economics

Business: Finance … J.24 Tech and Engineering: Agriculture

The appendix presents different input-setting examples across business, science, health, humanities, and engineering, alongside sample answers for finance, management, and biology questions.

Business: Finance

  • Ground truth gives Project A an IRR of 22% and Project B an IRR of 23.37%.In the vision-setting finance example, GPT-4o answers D and reports an IRR of 28.69.Source: Figure 20 · p. 30

Business: Manage

  • For the unemployment-chart question, GPT-4o identifies September as the month with the smallest difference, about 2,500.The response compares monthly differences from June through October.Source: Business: Manage · p. 31, Figure 21 · p. 31

J.9 Business: Marketing

  • The appendix includes a different input-setting example for Business: Marketing.The supplied passage identifies the example but does not provide its question or result.Source: Figure 22 · p. 32

Science: Biology

  • For the embryo question, GPT-4o selects B, while ground truth labels C: segmentation, type of symmetry, and head-tail orientation.The model response says symmetry is not explicitly visible and excludes it from its answer.Source: Science: Biology · p. 33, Figure 23 · p. 33

J.11 Science: Chemistry; J.12 Science: Geography; J.13 Science: Math; J.14 Science: Physics; J.15 Health and Medicine: Basic Medical Science; J.16 Health and Medicine: Clinical Medicine

  • Different input-setting examples are shown for Science: Chemistry, Geography, Math, and Physics, and Health and Medicine: Basic Medical Science and Clinical Medicine.The supplied captions identify these examples but provide no task results.Source: Figure 24 · p. 34, Figure 25 · p. 35, Figure 26 · p. 36

J.17 Health and Medicine: Diagnostics and Laboratory Medicine; J.18 Health and Medicine: Pharmacy; J.19 Health and Medicine: Public Health; J.20 Humanities and Social Science: History

  • Different input-setting examples are shown for Health and Medicine: Diagnostics and Laboratory Medicine, Pharmacy, and Public Health, and Humanities and Social Science: History.The captions identify the fields but do not report task results.Source: Figure 30 · p. 40, Figure 31 · p. 41, Figure 32 · p. 42

J.21 Humanities and Social Science: Literature; Humanities and Social Science: Sociology; J.23 Humanities and Social Science: Psychology

J.24 Tech and Engineering: Agriculture

  • The appendix also shows a different input-setting example for Tech and Engineering: Agriculture.The supplied caption does not provide a task result.Source: Figure 37 · p. 47

J.25 Tech and Engineering: Architecture and Engineering … J.30 Tech and Engineering: Mechanical Engineering

The appendix illustrates different input settings across Tech and Engineering subfields. In its Electronics example, the Vision Setting reaches the correct answer while the Standard Setting does not; in Energy and Power, both settings produce incorrect results.

J.25, J.26, J.29, and J.30 Tech and Engineering

  • Figures 38, 39, 42, and 43 show examples of different input settings in Architecture and Engineering, Computer Science, Materials, and Mechanical Engineering, respectively.The examples are identified in the appendix captions.Source: Figure 38 · p. 48, Figure 39 · p. 49, Figure 42 · p. 52

J.27 Tech and Engineering: Electronics

  • In Electronics, the Vision Setting correctly identifies the voltage drops and answer, while the Standard Setting's analysis is incorrect; the ground truth is (B) -8v -10v 20v.Figure 40 illustrates a different input setting for this subfield.Source: Tech and Engineering: Electronics · p. 50, Figure 40 · p. 50

J.28 Tech and Engineering: Energy and Power

Related papers

Loading 2409.02813v3…