PaperlayerView exact revision on arXiv

Source-linked AI summary

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen

arXiv:2311.16502v4cs.CLcs.AIcs.CV

TL;DR

Existing multimodal benchmarks do not adequately test college-level, domain-specific perception and reasoning across disciplines. MMMU addresses this gap with 11.5K multimodal questions, yet GPT-4V achieves only 55.7% accuracy, highlighting the benchmark’s difficulty.

  • Problem

    Existing benchmarks largely use text-only questions, limiting evaluation of multimodal models against college-level, discipline-specific expert tasks.

  • Method

    MMMU constructs a college-level benchmark of 11.5K multimodal questions spanning six disciplines, 30 subjects, and 183 subfields to measure perception, knowledge, and reasoning.

  • Results

    MMMU poses significant challenges to current models; GPT-4V achieves only 55.7% accuracy.

  • Takeaways & Limitations

    MMMU provides a comprehensive benchmark for assessing multimodal perception and complex reasoning across professional domains relevant to Expert AGI.

  • Takeaways & Limitations

    Manual curation may introduce bias, and college-level subjects alone may not suffice to test Expert AGI.

Abstract

from arXiv · show

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 14 open-source LMMs as well as the proprietary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.

1. Introduction

MMMU is introduced as a college-level, multi-discipline multimodal benchmark for monitoring progress toward Expert AGI. Its 11.5K questions span broad domains and reveal substantial performance gaps and challenges for current multimodal models.

  • Motivation: Expert AGI motivates college-level exam benchmarks because skilled adults solve specialized tasks, while existing MMLU and AGIEval consider only text-based questions.The authors frame multimodal evaluation as necessary because human experts can solve problems involving images and text.
  • Benchmark: MMMU contains 11.5K multimodal questions from exams, quizzes, and textbooks across six disciplines, 30 subjects, and 183 subfields.The benchmark targets college-level multimodal understanding and reasoning with broad disciplinary coverage.
  • Findings: 55.7% accuracy is achieved by GPT-4V on MMMU, demonstrating substantial room for improvement.The evaluation includes 28 open-source LMMs and advanced proprietary models.
  • Findings: Approximately 34% accuracy is achieved by the highest-performing open-source models, including BLIP2-FLAN-T5-XXL and LLaVA-1.5, creating a pronounced disparity with GPT-4V.The passage explicitly identifies a performance gap between open-source LMMs and GPT-4V.
  • Findings: OCR or generated-caption augmentation yields no notable improvement, indicating that MMMU requires deeper joint interpretation of images and text.Performance is higher in Art & Design and Humanities & Social Science, but lower in Business, Science, Health & Medicine, and Tech & Engineering.
  • Limitations: MMMU is intended to advance multimodal foundation models and monitor Expert AGI, but it is not a sufficient Expert AGI test because its scores lack a direct mapping to skilled-adult performance.The authors also note that college exams are not the only tasks an AGI should tackle.

2. Related Work

Related work traces multimodal language models from early vision-language pre-training systems toward improved visual instruction data and multimodal in-context learning with interleaved text-image examples.

  • Multimodal Pre-Training: Early universal vision-language models jointly encoded vision and language in fusion architectures for many multimodal tasks.Examples include LXMERT, UNITER, VinVL, Oscar, VilBert, and VLP.
  • Multimodal Pre-Training: Subsequent research improved the quantity and quality of visual instruction data for multimodal language models.Representative models include LLaMA-Adapter, mPlug-OWL, SVIT, LRV-Instruction, and InstructBLIP.
  • Multimodal In-Context Learning: Another major research direction studied multimodal in-context learning with interleaved text and image examples.This direction includes Flamingo, OpenFlamingo, Otter, M3IT, MetaVL, and Sparkles.

3. The MMMU Benchmark

MMMU is a college-level multimodal benchmark spanning 30 subjects, six disciplines, and over 183 subfields, with 11.5K questions designed to test perception, knowledge, and reasoning. Its heterogeneous visual inputs and subject-specific demands emphasize expert-level perception combined with deliberate reasoning.

  • Benchmark scope: MMMU covers 30 subjects across six disciplines and over 183 subfields to assess expert-level multimodal understanding across diverse tasks.The disciplines include Art & Design, Business, Health & Medicine, Science, Humanities & Social Science, and Tech & Engineering.
  • Benchmark scope: 11.5K questions are divided into a few-shot development set, an approximately 900-question validation set, and a 10.5K-question test set.The development set contains five questions per subject, while the validation set supports hyperparameter selection.
  • Evaluation targets: MMMU measures three essential LMM skills: perception, knowledge, and reasoning.The benchmark evaluates whether models can perceive and understand multimodal inputs while applying relevant knowledge and reasoning.
  • Evaluation targets: MMMU requires expert-level visual perception and deliberate reasoning with subject-specific knowledge across heterogeneous image types.Its tasks involve processing varied visual inputs and using domain-specific knowledge rather than relying only on general visual understanding.
  • Benchmark construction: 30 image formats, including diagrams, tables, charts, chemical structures, photos, paintings, geometric shapes, music sheets, and medical images, broaden MMMU’s visual coverage beyond prior benchmarks.Subject selection prioritized areas where visual inputs are commonly used and provide valuable information.

4. Experiments

Experiments evaluate closed- and open-source LLMs and LMMs in a zero-shot setting using micro-averaged accuracy and rule-based response processing. MMMU remains highly challenging: human experts and GPT-4V substantially outperform most models, while open-source systems improve but struggle with uncommon image types and harder questions.

  • Evaluation Setup: Models are evaluated zero-shot with default or engineered prompts, without fine-tuning or few-shot demonstrations.The evaluation includes both closed- and open-source LLMs and LMMs.
  • Evaluation: Micro-averaged accuracy is computed through systematic, rule-based pipelines that robustly extract answers from open and multiple-choice responses.Random Choice and Frequent Choice provide reference baselines.
  • Overall Results: 88.6% is the best human expert validation accuracy, significantly exceeding all reported models and exposing a persistent expertise gap.The comparison is reported on the MMMU validation set.
  • Overall Results: 44.7% and 46.2% are the test accuracies of LLaVA-1.6-34B and InternVL-Chat-V1.2, respectively, narrowing the gap with proprietary models despite earlier open-source results near 34%.GPT-4V remains the leading model in the comparison.
  • Image Types and Difficulty: 76.1% is GPT-4V’s success rate on Easy questions, while all models score very low on uncommon image types such as geometric shapes, music sheets, and chemical structures.Open-source models perform relatively strongly on photos and paintings, but generalize poorly to less common visual categories.

5. Error Analysis and Future Work

The section analyzes 150 randomly sampled GPT-4V errors through expert annotation to identify root causes and guide future improvements. Findings highlight imbalanced language–vision interaction, domain-specific knowledge gaps, grounding difficulties, and broader challenges in perception, reasoning, and multimodal understanding.

  • Error-analysis method: 150 randomly sampled GPT-4V errors were examined by expert annotators to identify the root causes of mispredictions.The analysis was intended to clarify the model’s capabilities and limitations and guide future design and training improvements.
  • Future work: Language–vision interplay remains double-edged: language can make visual understanding more explainable but can also induce hallucinations.The summary identifies grounding, visual perception, knowledge representation, reasoning, and multimodal joint understanding as areas for further research.
  • Future work: Grounding and referring to specific elements within visual inputs remain challenging for GPT-4V.These difficulties are presented as part of the broader challenges posed by MMMU.

6. Conclusion … G. Few-shot Results

MMMU is presented as a comprehensive benchmark for Expert AGI that evaluates perception and reasoning across professional domains, while acknowledging curation bias and limits of college-level coverage. The supplied material also identifies subject-specific breakdown results as part of the benchmark’s reported analyses.

  • 6. Conclusion: MMMU evaluates basic perception and complex reasoning across various professional domains.The benchmark is framed as aligning with the expectations of skilled adults in these fields.
  • 6. Conclusion: MMMU’s manual curation may introduce bias, and college-level subjects may not fully test Expert AGI.The paper nevertheless argues that strong MMMU performance should be necessary for an Expert AGI system.
  • 6. Conclusion: The benchmark’s challenge is supported by results from over 30 models and human experts.The supplied passage presents these results as evidence of MMMU’s challenging nature.
  • B. Breakdown Results on Different Subjects: The reported analyses include a main-results section and breakdowns across Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.The supplied contents passage lists these subject-level breakdowns but provides no numerical results.

H. Data Annotation Protocol … B. Breakdown Results on Different Subjects

The appendix outlines a dedicated data annotation protocol and reports MMMU’s subject coverage and model breakdowns. MMMU comprises 11.5K multimodal questions spanning six disciplines, 30 subjects, and 183 subfields.

  • H. Data Annotation Protocol: The data annotation protocol includes data collection, general guidelines, data format and structure, quality control, and validation.These components are listed as subsections of the protocol.
  • H. Data Annotation Protocol: The annotation protocol also addresses ambiguities, ethical considerations, data contamination, and example questions.These topics appear as additional protocol subsections.
  • I. Author Contribution Statement: The supplied passages identify the appendix sections but provide no substantive author-contribution or version-change details.Only the listed section titles are available for those merged sections.
  • J. Version Change Log: The supplied passages identify the appendix sections but provide no substantive author-contribution or version-change details.Only the listed section titles are available for those merged sections.
  • A. Subject Distribution: 11.5K multimodal questions span six broad disciplines, 30 subjects, and 183 subfields.This establishes the benchmark’s breadth across disciplines and subject areas.
  • B. Breakdown Results on Different Subjects: The reported subject distribution contextualizes model evaluation across a benchmark organized into six disciplines, 30 subjects, and 183 subfields.This connects the benchmark’s structural coverage to the appendix’s subject-level breakdown framing.
  • B. Breakdown Results on Different Subjects: The appendix presents main results and breakdown results for different models across each discipline and subject.The supplied passage describes the scope of the breakdown analysis without reporting numerical results.

B.1. Main Results … Agriculture: Plant Physiology

The paper reports MMMU results across validation and test sets for six disciplines, then illustrates representative correct and error cases spanning domain-specific multimodal questions. The case studies cover perception, knowledge, reasoning, and rejection-related outcomes across numerous subjects and subfields.

  • B.7. Tech & Engineering: The benchmark’s technical coverage extends to technology and engineering, while the provided tables and case-study materials emphasize heterogeneous multimodal evaluation across subjects and image-based tasks.Table 10 is identified as the Tech & Engineering results table; the supplied case materials also include maps, graphs, diagrams, tables, and other visual question formats.
  • C. Case Study: The case-study index organizes examples by subject and error category, including Correct Case, Perception, Lack of Knowledge, Reasoning, Other, and Reject to Answer.The index explicitly lists subject-level examples and associates case-study figures with error categories.
  • Art & Design: Art and design cases span visual arts, photography, fine arts, art criticism, art history, industrial design, music, and design history, with both correct and error examples.Examples include interpreting themes, identifying a panning blur, analyzing architectural devices, and reading musical intervals.
  • Business: Business cases cover accounting, economics, finance, management, and marketing through image-based questions involving financial statements, markets, auctions, costs, ethics, and statistical tests.The supplied examples include price-weighted index returns, retained earnings, equilibrium interest rates, auction receipts, portfolio choice, product costing, and hypothesis-test selection.
  • Health & Medicine: Health and medicine cases range from physiology and pathology to radiology, neuropathology, ophthalmic pathology, and heart disease, with examples of correct, perceptual-error, knowledge-gap, and rejection cases.Questions include cardiovascular pathways, newborn diagnosis, clinical pathology, imaging-based diagnosis, lesion localization, and intraocular tumors.
  • Humanities & Social Science: Humanities and social-science examples include history, literature, sociology, and psychology, while the supplied psychology case describes meditation and breathing as possible inhibitors of the stress response.The passage specifically presents a Clinical Psychology question concerning stress reduction and the hypothalamic-pituitary-adrenal axis.

Architecture and Engineering: Surveying and Mapping

The section includes a surveying-and-mapping problem that tests aerial-photography geometry under specified flight, camera, overlap, print, and terrain conditions. It asks for the photograph’s scale and air base length.

  • Surveying and Mapping: Aerial-photography parameters include a 200 mm focal length, 1500 m flying height, 60% overlap, and 250 mm × 250 mm prints.Both exposures share the same flying height, with a balanced flight and no drift.
  • Surveying and Mapping: The problem specifies flat ground 250 m above mean sea level as the terrain condition for determining the requested quantities.The terrain and flight assumptions define the geometric setup of the calculation.
  • Surveying and Mapping: The required outputs are the photograph’s scale and the length of the air base.The passage provides the question and answer-option heading but does not state the numerical solutions.

Architecture and Engineering: Structural Engineering

The section presents a structural-engineering problem asking models to compute the horizontal displacement of joint B under a 240-kN load, given bar properties and answer choices.

  • Structural Engineering: 240-kN load: The task asks for the horizontal displacement of joint B shown in Figure P8.10.The problem is framed as a structural-engineering computation.
  • Structural Engineering: 2400 mm^2 area and 200 GPa modulus: These properties are specified for all bars.The stated parameters apply uniformly across the bar members.
  • Structural Engineering: 2.657 mm, 3.657 mm, 4.657 mm, and 5.657 mm: The multiple-choice displacement options are listed as A through D.The alternatives are presented in increasing numerical order.

Architecture and Engineering: Civil Engineering

The Civil Engineering subsection includes a highway traffic-flow question requiring regression analysis with the Greenshields model to determine jam density. The question presents three closely spaced answer choices from 110 to 112 vehicles per mile.

  • Civil Engineering: A highway time-lapse dataset is used to fit the Greenshields model through regression analysis and determine jam density.The task is framed as a civil-engineering traffic-flow problem based on photographic observations.
  • Civil Engineering: The answer choices are (A) kj=110 veh/mi, (B) kj=111 veh/mi, and (C) kj=112 veh/mi.The options differ by one vehicle per mile across the listed range.
  • Civil Engineering: The problem requires interpreting time-lapse highway observations rather than relying on a purely textual description.An image accompanies the question’s stated data source.

Computer Science: Data Structure and Algorithm … E. Distributions of Image Types

The appendix illustrates MMMU’s breadth through representative questions and correct or erroneous cases spanning computer science, electronics, energy, materials, and mechanical engineering. It also reports coverage of 183 subfields across 30 subjects and 30 diverse image types in 11.5K questions.

  • Computer Science: Data Structure and Algorithm; Computer Science: Operating System; Computer Science: Compiler Principle: Computer science examples cover Dijkstra shortest paths, interleaved transaction processing, and minimal-DFA recognition, with figures labeled as correct or perceptual-error cases.The examples span data structures and algorithms, operating systems, and compiler principles.
  • Electronics: Signal Processing: Electronics signal-processing examples ask for Laplace-transform circuit formulations and current magnitudes in unbalanced three-phase systems, including correct and rejected-answer cases.The circuit formulation provides four candidate expressions, while the three-phase problem supplies voltage and impedance values.
  • Energy and Power: Fluid Mechanics; Energy and Power: Thermodynamics: Energy-and-power examples include fitting helium viscosity data to a power law and determining heat loss or gain during polytropic helium compression.The compression problem specifies initial and final pressures, temperatures, and volume, with candidate heat values including −5.553 kJ, −6.553 kJ, and −7.553 kJ.
  • Materials: Materials Science; Materials: Mechanics of Materials: Materials examples ask which metal is least reactive from standard electrode potentials and calculate average shear strains at two deformed-plate corners.The shear-strain options pair 0.0292 rad and 0.0125 rad in different combinations for corners A and B.
  • Materials: Control System: A control-system example asks for steady-state errors under the test input 25u(t), with candidate answers 0, 1, and −1.The associated figure is labeled a sample correct case.
  • Mechanical Engineering: Engineering Dynamics: Mechanical-engineering dynamics examples determine the velocity of two connected spheres on a smooth circular guide and the supporting force near the bottom.Another projectile-motion problem asks for the minimum horizontal velocity needed for a rock to clear an obstruction, with candidates 18.014 m/s, 24.014 m/s, and 28.014 m/s.
  • D. Subfields of Different Subjects: MMMU contains 183 subfields across 30 subjects.This appendix explicitly lists the subfields of each subject in Table 12.
  • E. Distributions of Image Types: The 11.5K MMMU questions use 30 image types, whose distribution is shown with a horizontal bar chart ranging from advertisements to diagrams.The section presents image-type counts across the dataset.

F. Results on Different Image Types

This section reports selected-model performance across 30 different image types, noting that a single image may belong to multiple types.

  • Evaluation scope: The evaluation covers selected models across 30 different image types.Results are reported in Table 13.
  • Evaluation scope: A single image may have multiple image types in the evaluation.This qualification accompanies Table 13’s results.

G. Few-shot Results … J. Version Change Log

The appendix reports weak few-shot performance for OpenFlamingo and Otter, documents a rigorous multimodal data-annotation protocol, summarizes author contributions, and records benchmark version changes. The protocol emphasizes college-level image-based questions, quality control, ethical sourcing, contamination resistance, and structured examples.

  • G. Few-shot Results: 0.263 to 0.256: OpenFlamingo declines from 0-shot to 1-shot and 3-shot, while Otter drops to 0.276 in 1-shot and 0.258 in 3-shot and 5-shot.OpenFlamingo shows a slight increase at 5-shot, but overall trends indicate weak few-shot ability in existing open-source models.
  • H. Data Annotation Protocol; H.1. Data Collection: The annotation protocol collects college-level multimodal questions from free online resources, quizzes, textbooks, and study materials while requiring diverse sources and copyright compliance.Questions include images such as diagrams, charts, and photographs, and may use multiple-choice or open-ended formats.
  • H.2. General Guidelines; H.3. Data Format and Structure: Annotations must be accurate, consistent, academically rigorous, English-written, unambiguous, college-level, image-containing, and fully documented with question, answer, image, difficulty, and explanation fields.The structured JSON format stores question type, question text, answer options when applicable, correct answer, difficulty, and explanation.
  • H.4. Quality Control and Validation; H.5. Handling Ambiguities: Secondary review teams and random audits verify annotation quality and guideline adherence, while ambiguous instances undergo collaborative review to establish standardized decisions.This combines the documented quality-control and ambiguity-handling procedures.
  • H.6. Ethical Considerations: Copyright and licensing compliance, privacy-law adherence, ethical data handling, and avoidance of private information are mandatory annotation considerations.The protocol also avoids sources prohibiting copying or redistribution.
  • H.7. Data Contamination Considerations; H.8. Example Questions: Annotators are instructed to reduce benchmark contamination by selecting questions whose answers are hidden in less obvious locations, including separate documents or textbook conclusions.Annotated-question examples cover multiple-choice and open-ended formats, and annotators are encouraged to collect questions containing multiple images.
  • I. Author Contribution Statement: All authors contributed to data collection, annotation, validation, case studies, error analysis, discussion, feedback, and proofreading, with authors contributing 1/3 of MMMU examples.Specific roles included project leadership, data curation, evaluation, baseline testing, interface development, figure preparation, conceptualization, writing, and strategic guidance.
  • I. Author Contribution Statement; J. Version Change Log: Version changes added model results, author contributions, updated prompts and evaluation outputs, corrected naming and typos, adjusted tables, and added validation-set results across successive releases.The log covers changes from V1 through V4, including Qwen-VL-PLUS, SPHINX, Gemini variants, updated model names, and additional validation results.
Loading 2311.16502v4…