PaperlayerView exact revision on arXiv

Source-linked AI summary

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen

arXiv:2311.16502v4cs.CLcs.AIcs.CV

TL;DR

Existing college-level discipline benchmarks use text-only questions, although experts can solve multimodal problems. MMMU evaluates college-level multimodal reasoning and knowledge, and its evaluation finds current models substantially challenged, with GPT-4V at 55.7% accuracy.

  • Problem

    Existing college-level discipline benchmarks such as MMLU [25] and AGIEval [92] use text-only questions, despite experts being capable of solving multimodal problems.

    Source: §1 · p. 2

  • Method

    MMMU assembles 11.5K college-sourced multimodal questions across six disciplines to assess perception, subject knowledge, and reasoning.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    MMMU poses significant challenges to current models, with GPT-4V achieving 55.7% accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MMMU is intended to help monitor progress toward Expert AGI.

    Source: §1 · p. 3

    MMMU is not a sufficient test for Expert AGI because its scores lack a direct mapping to skilled-adult performance and college exams do not cover all AGI tasks.

    Source: §1 · p. 3, §6 · p. 8

Abstract

from arXiv · show

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 14 open-source LMMs as well as the proprietary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.

1. Introduction

MMMU is introduced as a benchmark for college-level, multidisciplinary multimodal understanding and reasoning, intended to help assess progress toward Expert AGI while exposing substantial model limitations.

Introduction

  • Expert AGI is framed as reaching at least the 90th percentile of skilled adults across a broad range of tasks, making college-level assessments a natural starting point.MMMU is not a sufficient Expert AGI test because its scores lack a direct mapping to that threshold and college exams do not encompass all AGI tasks; the authors argue strong MMMU performance should nevertheless be necessary.Source: §1 · pp. 1–3
  • Existing multimodal benchmarks mostly emphasize commonsense or daily knowledge, while ScienceQA [50] is largely elementary-to-middle-school level despite its disciplinary breadth.The authors argue this leaves a gap in benchmarking expert-level domain knowledge and advanced reasoning; text-only college-exam benchmarks such as MMLU [25] and AGIEval [92] do not cover multimodal problems.Source: §1 · p. 2
  • MMMU contains 11.5K questions from college exams, quizzes, and textbooks across six disciplines, 30 subjects, and 183 subfields, with varied image formats and interleaved text-image inputs.Many questions require expert-level reasoning, such as applying Fourier Transform or Equilibrium Theory; the sampled examples illustrate the expert knowledge and reasoning needed to interpret the questions and images.Source: §1 · p. 2, Figure 2 · p. 2
  • GPT-4V achieves 55.7% accuracy on MMMU, while the highest-performing open-source models BLIP2-FLAN-T5-XXL and LLaVA-1.5 achieve approximately 34%.The authors describe this as a pronounced performance disparity and substantial room for improvement.Source: §1 · pp. 2–3
  • In 150 GPT-4V error cases, 35% are perceptual, 29% reflect lack of knowledge, and 26% stem from reasoning flaws.Models perform better in Art & Design and Humanities & Social Science than in Business, Science, Health & Medicine, and Tech & Engineering, where visual data is more complex and reasoning demands are greater.Source: §1 · p. 3
  • OCR- or caption-augmented LLMs show no notable improvement, and the authors attribute this to MMMU’s demand for deeper joint image-text interpretation.The reported discipline pattern contrasts higher performance in Art & Design and Humanities & Social Science with lower performance in Business, Science, Health & Medicine, and Tech & Engineering.Source: §1 · p. 3

2. Related Work

Prior multimodal models advanced pretraining, instruction tuning, and in-context learning, while existing evaluations often emphasized basic perception; MMMU targets harder expert-level reasoning across 30 subjects.

Multimodal Pre-Training

  • Early multimodal pretraining used Faster RCNN features in models such as LXMERT [71] and UNITER [10], while later CLIP [66] and ALIGN [29] trained ViT representations from scratch on large web datasets.These models succeeded on VQA and captioning tasks that require less knowledge and reasoning.Source: §2 · p. 3

Multimodal Instruction Tuning

  • Following instruction-tuned LLMs FLAN-T5 [14] and Vicuna, LLaVA [44] [45] and MiniGPT-4 [94] used open resources to improve LMM instruction following, with later work enhancing visual instruction data quantity and quality.Examples of later models include LLaMA-Adapter [20] [88], mPlug-OWL [81] [82], SVIT [89], LRV-Instruction [43], and InstructBLIP [16].Source: §2 · p. 3

LMM Benchmarks

  • MMMU addresses benchmark gaps with difficult expert-level problems across 30 subjects, requiring nuanced perception, domain knowledge, and step-by-step reasoning beyond basic holistic evaluations or math-only MathVista.The related work also notes GAIA’s concurrent collection of 466 questions testing reasoning, multimodality handling, or tool use.Source: §2 · p. 3

Multimodal In-Context Learning

  • Flamingo [2] and OpenFlamingo [4], Otter [34], M3IT [36], MetaVL [56], Sparkles [26], and MMICL [90] explored multimodal in-context learning with interleaved text-image examples.The passage credits these models with contributing to multimodal training and instruction-following advances.Source: §2 · p. 3

3. The MMMU Benchmark

MMMU is a college-level multimodal benchmark spanning many disciplines and image types, designed to test perception, subject knowledge, and reasoning. Its curation and comparisons emphasize the need for expert visual understanding and deliberate, domain-informed reasoning.

3.1. Overview of MMMU

  • MMMU contains 11.5K questions across 30 subjects, six disciplines, and over 183 subfields, split into few-shot development, validation, and test sets.The few-shot development set has five questions per subject, validation has approximately 900 questions, and the test set has 10.5K questions.Source: §3.1 · pp. 3–4
  • MMMU tasks require expert-level visual perception and deliberate reasoning with subject-specific knowledge across heterogeneous image types.The benchmark measures perception, knowledge, and reasoning, requiring models to understand text and images and apply domain knowledge to derive solutions.Source: §3.1 · p. 4

3.2. Data Curation Process

  • The collection yielded 13K questions, with approximately 10% of very easy problems excluded during curation.Over 50 university-student annotators collected questions from textbooks and online resources, and authors checked for duplicates, formatting and typos, and difficulty.Source: §3.2 · p. 4, §3.1 · p. 4

3.3. Comparisons with Existing Benchmarks

  • MMMU covers college-level knowledge and 30 image formats, while prior benchmarks typically emphasize daily knowledge, common sense, and simpler reasoning.Its formats include diagrams, tables, charts, chemical structures, medical images, and music sheets.Source: §3.3 · p. 4, Figure 3 · p. 5

4. Experiments

MMMU experiments compare zero-shot language and multimodal models, text-only baselines, and human experts. Results show a substantial gap between current models and human expertise, with performance varying by model, discipline, image type, and difficulty.

4. Experiments

  • Experiments use zero-shot prompts, micro-averaged accuracy, and rule-based answer extraction, with invalid responses counted incorrect for open questions and randomized for multiple-choice.The evaluation also includes Random Choice and subject-specific Frequent Choice baselines; selected models have few-shot results reported in the appendix.Source: §4 · p. 5, §4.1 · p. 6

4.2. Main Results

  • On the MMMU benchmark, the best human expert achieves 88.6% validation accuracy, significantly outperforming all models reported in Table 2, demonstrating a remaining gap between human expertise and current model performance.The comparison is detailed in Table 2.Source: §4.2 · p. 6, Table 2 · p. 6

4.3. Analysis on Images Types and Difficulties

  • GPT-4V consistently outperforms other models across image types, while all models score very low on geometric shapes, music sheets, and chemical structures.Open-source models perform relatively strongly on photos and paintings, image types more frequently seen during training; results on uncommon types indicate poor generalization.Source: Figure 4 · p. 7, §4.3 · p. 7

4.2. Main Results

  • LLaVA-1.6-34B and InternVL-Chat-V1.2 reach 44.7% and 46.2% test accuracy, narrowing the gap with proprietary models.Earlier open-source models BLIP2-FLAN-T5-XXL and LLaVA-1.5 reach approximately 34% accuracy, significantly below GPT-4V.Source: §4.2 · p. 7
  • OCR and captioning do not significantly improve text-only model performance; models perform relatively better in Art & Design and Humanities, and worse in reasoning-heavy disciplines.Lower-performing disciplines include Science, Health & Medicine, and Technology & Engineering.Source: §4.2 · p. 7

4.3. Analysis on Images Types and Difficulties

  • GPT-4V achieves 76.1% success on Easy questions and 55.6% on Medium questions; its advantage over other models almost disappears on Hard questions.The passage attributes this pattern as a possible limitation in handling expert-level queries.Source: Table 3 · p. 7, §4.3 · p. 7

5. Error Analysis and Future Work

An expert review of 150 randomly sampled GPT-4V errors finds that perception, specialized knowledge, and reasoning are the main failure sources. The analysis points to better visual grounding, domain knowledge, reasoning, and language–vision integration as priorities for future work.

5. Error Analysis and Future Work

  • Perceptual failures include elementary visual misinterpretations and a tendency to prioritize textual interpretations over visual evidence.The analysis distinguishes basic perceptual errors from domain-specific perceptual errors, which it classifies as lack of knowledge; one example favors a text-based reading of “imperialism” over a cartoon’s visual narrative.Source: §5 · pp. 7–8
  • Perceptual Errors (35%), Lack of Knowledge (29%), Reasoning Errors (26%), Textual Understanding Error (6%), Rejection to Answer (3%), Annotation Error (2%), and Answer Extraction Error (1%) make up the annotated errors.Expert annotators examined 150 randomly sampled GPT-4V errors; Figure 5 shows their distribution.Source: §5 · pp. 7–8, Figure 5 · p. 8
  • Insufficient specialized knowledge can impair both domain-specific visual interpretation and reasoning, motivating more diverse domain-specific training data.Examples include misinterpreting double circles as accept states in Deterministic Finite Automata and flawed reasoning in a medical context.Source: §5 · p. 8
  • Reasoning errors persist even when GPT-4V interprets text and images correctly and recalls relevant knowledge, including cases where it omits a necessary mathematical step.The analysis identifies improved reasoning capability as critical for addressing these shortcomings.Source: §5 · p. 8
  • Future research should address language–vision interplay, visual grounding, and complex reasoning involving lengthy chains or extensive calculations.Language may aid explainable visual understanding but also induce hallucinations; grounding specific elements in images remains challenging even for GPT-4V.Source: §5 · p. 8

6. Conclusion · A. Subject Distribution · B. Breakdown Results on Different Subjects

MMMU evaluates perceptual and reasoning abilities across professional domains as a benchmark toward Expert AGI; the authors argue strong performance should be a necessary criterion despite limitations.

6. Conclusion

  • MMMU is presented as a comprehensive benchmark for evaluating basic perception and complex reasoning across professional domains, aligned with skilled adults’ expectations.Source: §6 · p. 8
  • The authors argue that strong performance on MMMU should be a necessary criterion for an Expert AGI system.Source: §6 · p. 8
  • The authors note that manual curation may introduce biases and college-level subject coverage may be insufficient for testing Expert AGI [57].Source: §6 · p. 8
  • MMMU combines multiple-choice and concise open-ended questions to assess diverse subjects while addressing open-ended evaluation challenges.Source: §6 · p. 8
  • The authors cite performance from over 30 models and human experts as evidence of MMMU’s challenging nature.Source: §6 · p. 8

A. Subject Distribution

  • Figure 7 describes MMMU as containing 11.5K multimodal questions across six broad disciplines, 30 subjects, and 183 subfields.Source: Figure 7 · p. 14

B. Breakdown Results on Different Subjects

B.1. Main Results … Agriculture: Plant Physiology

This section surveys MMMU validation and test results by overall and discipline-specific scope, and presents subject-level case studies through multimodal question examples across disciplines. The examples include correct and erroneous responses, including a perceptual error in identifying a soybean disease.

B.1. Main Results

  • Table 4 presents overall results for different models on the MMMU validation and test sets.Its caption notes that best-in-category results are bolded, second-best results are underlined, and some results were provided by authors; it gives no values.Source: Table 4 · p. 15

B.2. Art & Design

  • Table 5 presents Art & Design results for different models on the MMMU validation and test sets.The caption indicates best and second-best results by category but supplies no scores.Source: Table 5 · p. 16

B.3. Business

  • Table 6 presents Business results for different models on the MMMU validation and test sets.The caption indicates best and second-best results by category but supplies no scores.Source: Table 6 · p. 17

B.4. Science

  • Table 7 presents Science results for different models on the MMMU validation and test sets.The caption indicates best and second-best results by category but supplies no scores.Source: Table 7 · p. 18

B.5. Health & Medicine

  • Table 8 presents Health & Medicine results for different models on the MMMU validation and test sets.The caption indicates best and second-best results by category but supplies no scores.Source: Table 8 · p. 19

B.6. Humanities & Social Science

  • Table 9 presents Humanities & Social Science results for different models on the MMMU validation and test sets.The caption indicates best and second-best results by category but supplies no scores.Source: Table 9 · p. 20

B.7. Tech & Engineering

  • Table 10 presents Tech & Engineering results for different models on the MMMU validation and test sets.The caption indicates best and second-best results by category but supplies no scores.Source: Table 10 · p. 21

List of Case Study Figures

  • The case-study figure index organizes examples by subject and lists categories including correct case, perception, lack of knowledge, reasoning, and other.The index includes entries for subjects across the benchmark, but the supplied passages do not provide aggregate error frequencies.Source: Table 11 · p. 24, List of Case Study Figures · p. 24

Medical examples

History examples

Examples across disciplines

  • Figure captions label examples as perceptual, textual-understanding, reasoning, or lack-of-knowledge errors, alongside cases marked correct.Examples include a comparative-literature item labeled a perceptual error and a clinical-psychology item also labeled a perceptual error.Source: Figure 44 · p. 61, Figure 45 · p. 62, Figure 48 · p. 65

Psychology examples

Agriculture and pharmacy examples

Agriculture: Plant Physiology

  • A sample Agriculture plant physiology perceptual error asks which disease causes a brown pit in soy and is favored by drought.The options are stalk rot, stem rust, brown stem rot, and charcoal rot; the passages do not identify the correct answer.Source: Figure 78 · p. 95, Agriculture: Plant Physiology · p. 95

Architecture and Engineering: Surveying and Mapping

The section features a surveying-and-mapping problem that asks students to determine photograph scale and air-base length from aerial-photography conditions.

Architecture and Engineering: Surveying and Mapping

  • A surveying-and-mapping question asks for photograph scale and air-base length given aerial-photography and terrain conditions.The setup specifies a 200 mm focal length, 1,500 m flying height, 60% overlap, level flight without drift, and flat ground 250 m above mean sea level.Source: Architecture and Engineering:… · p. 96

Architecture and Engineering: Structural Engineering

The section presents a Structural Engineering problem asking for the horizontal displacement of joint B under a 240-kN load.

Architecture and Engineering: Structural Engineering

  • A sample problem asks for joint B’s horizontal displacement under a 240-kN load, given bar area and Young’s modulus, with four answer choices.The choices are 2.657, 3.657, 4.657, and 5.657 mm; the supplied passage does not identify the correct answer.Source: Architecture and Engineering:… · p. 97, Figure 80 · p. 97

Architecture and Engineering: Civil Engineering

The Civil Engineering example asks a model to fit the Greenshields model to time-lapse highway data and determine jam density; the cited failure is categorized as a reasoning error.

Architecture and Engineering: Civil Engineering

Computer Science: Data Structure and Algorithm … E. Distributions of Image Types

MMMU spans 30 subjects and 183 subfields, with questions using 30 image types; the supplied examples illustrate its technical breadth, while the appendix summarizes image-type diversity.

Computer Science: Data Structure and Algorithm; Computer Science: Operating System; Computer Science: Compiler Principle

Electronics: Signal Processing

Energy and Power: Fluid Mechanics

Materials: Materials Science; Materials: Mechanics of Materials; Materials: Control System

Mechanical Engineering: Engineering Dynamics

E. Distributions of Image Types

  • MMMU’s 11.5K questions use 30 image types, with the distribution spanning categories from advertisements to diagrams.The distribution is shown as a horizontal bar chart; the supplied passages do not provide category-level counts.Source: Figure 95 · p. 112, E. Distributions of Image Types · p. 112

D. Subfields of Different Subjects

F. Results on Different Image Types

The section reports selected models’ performance across 30 image types, noting that a single image may have multiple types.

F. Results on Different Image Types

G. Few-shot Results

OpenFlamingo and Otter show weak few-shot performance on MMMU, with performance generally declining as more in-context examples are introduced.

G. Few-shot Results

  • The few-shot results suggest the data samples may be too hard for these models to understand the underlying patterns or context.The dev set was used as the in-context learning examples.Source: G. Few-shot Results · p. 115
  • OpenFlamingo falls from 0.263 at 0-shot to 0.256 at 1-shot and 3-shot, while Otter declines to 0.276 at 1-shot and 0.258 at 3-shot and 5-shot.OpenFlamingo shows a slight increase at 5-shot; the reported trend suggests existing open-source models have very weak few-shot learning ability.Source: Table 14 · p. 115, G. Few-shot Results · p. 115

H. Data Annotation Protocol … J. Version Change Log

The appendices specify how MMMU questions are collected, annotated, formatted, reviewed, and ethically handled, then document author contributions and successive benchmark updates.

H.1. Data Collection; H.2. General Guidelines

  • Questions should come from diverse permitted sources and include images; they must be English, college-level, unambiguous, and classified and annotated consistently.The protocol allows multiple-choice and short open-ended formats, and asks annotators to avoid questions with very long answers.Source: §H.1 · p. 116, §H.2 · p. 116

H.3. Data Format and Structure

  • Each sample is stored as a separate JSON file with question, answer, difficulty, and optional explanation fields, while image paths are embedded where needed.The protocol specifies standard naming conventions for question JSON and image files.Source: §H.3 · p. 116

H.4. Quality Control and Validation; H.5. Handling Ambiguities

  • A secondary team reviews annotations and random audits check consistency; unclear instances are flagged and resolved collaboratively in team meetings..Source: §H.4 · p. 116, §H.5 · p. 116

H.6. Ethical Considerations

  • Annotators must follow copyright and licensing rules, avoid sources that prohibit copying or redistribution, and exclude questions containing private information..Source: §H.6 · p. 116

H.7. Data Contamination Considerations

  • To reduce contamination risk, annotators should favor questions whose answers require finding information in less obvious locations and synthesizing challenging sources..Source: §H.7 · p. 117

H.8. Example Questions

  • The appendix provides annotated multiple-choice and open-ended examples and encourages multi-image questions with careful image-file referencing.Figure 98 illustrates a multi-image question and its JSON representation.Source: §H.8 · p. 117, Figure 98 · p. 118

I. Author Contribution Statement

  • All authors contributed to data collection, annotation, and validation; Yue led the project, Ni co-led curation, Zhang evaluated models, and Zheng worked on baselines and annotation tools.All authors also contributed to case studies and error analysis, annotation discussions, feedback, and proofreading.Source: §I · pp. 117–118

J. Version Change Log

  • The version log records added model results, revised overall tables, human results, renamed models, and corrections to figures and typos.It also notes Gemini results from reports and, Reka, Claude 3, and GPT-4o.Source: J. Version Change Log · p. 119

Related papers

Loading 2311.16502v4…