Source-linked AI summary
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, Ran He
TL;DR
MLLM research lacks a comprehensive evaluation that quantitatively reflects diverse multimodal abilities while limiting data leakage and prompt-engineering effects. The paper introduces MME, a manually constructed benchmark with concise yes-or-no instructions covering 14 perception and cognition subtasks, and evaluates 30 advanced MLLMs. Results show substantial room for improvement and expose problems that can guide subsequent model optimization.
Problem
Existing evaluations do not comprehensively quantify MLLM performance because public datasets risk data leakage, open-ended evaluations are limited, and many benchmarks assess only one aspect.
Method
MME manually constructs instruction-answer pairs for 14 perception and cognition subtasks and uses concise unified instructions with quantitative accuracy and accuracy+ metrics.
Results
30 advanced MLLMs show clear performance discrepancies on MME, and the experiments indicate substantial room for improvement.
Takeaways & Limitations
MME provides a comprehensive evaluation framework and summarizes experimentally exposed problems as guidance for the evolution of MLLMs.
Takeaways & Limitations
The current MME version uses relatively simple samples, with 20 images and 40 instruction-answer pairs, because some capabilities remain in their infancy.
Abstract
from arXiv · showhide
Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fairly compare MLLMs, instead of struggling in prompt engineering. Besides, with such an instruction, we can also easily carry out quantitative statistics. A total of 30 advanced MLLMs are comprehensively evaluated on our MME, which not only suggests that existing MLLMs still have a large room for improvement, but also reveals the potential directions for the subsequent model optimization. The data are released at the project page https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation.
1 Introduction
MLLMs show impressive multimodal abilities, but existing evaluation approaches do not comprehensively or reliably quantify performance. MME addresses this gap with a 14-subtask benchmark spanning perception and cognition, manually designed concise instructions, and broad evaluation of 30 models.
- Motivation: Existing evaluations struggle to comprehensively measure MLLM performance because public datasets risk leakage, open-ended evaluations are limited, and many benchmarks cover only one capability.These limitations motivate a unified benchmark with broader coverage and controlled evaluation conditions.
- Benchmark design: MME evaluates both perception and cognition across 14 subtasks, including object recognition, OCR, commonsense reasoning, calculation, translation, and code reasoning.Perception covers coarse- and fine-grained recognition plus OCR, while cognition combines visual information with language-model knowledge for more complex answers.
- Benchmark design: All instruction-answer pairs are manually constructed, with public datasets used only for images rather than their original annotations, reducing the risk of data leakage.The benchmark also uses real photographs and generated images.
- Evaluation protocol: Concise unified instructions ask models to answer yes or no, enabling objective quantitative statistics and fairer comparison without extensive prompt engineering.The authors report that multiple-choice instructions may exceed current MLLMs' ability to follow complex instructions.
- Evaluation: 30 advanced MLLMs are evaluated through overall perception and cognition leaderboards plus 14 individual-subtask leaderboards.The leaderboards combine accuracy and accuracy+ scores, with full scores of 2000 for perception, 800 for cognition, and 200 for each subtask.
- Findings: The benchmark reveals clear performance discrepancies among evaluated MLLMs and summarizes exposed problems to guide subsequent model improvement.The paper presents this analysis as evidence that existing MLLMs still have substantial room for improvement.
2 MME Evaluation Suite
MME evaluates multimodal perception and cognition through concise yes/no instructions, manually designed instruction-answer pairs, and 14 subtasks. Its suite spans object and fine-grained recognition, OCR, and image-based reasoning tasks, while limiting some tasks to relatively simple examples.
- Evaluation Metric: MME uses paired yes/no questions per image, with one ground-truth answer of each type, to support quantitative evaluation.Accuracy is computed per question, while accuracy+ requires both questions for an image to be answered correctly.
- Evaluation Metric: Accuracy and accuracy+ have random baselines of 50% and 25%, respectively, and each subtask score sums the two metrics.The full perception and cognition scores are 2000 and 800, respectively.
- Perception Tasks: The suite covers 10 perception subtasks spanning coarse-grained recognition, fine-grained recognition, and OCR.Fine-grained recognition includes movie posters, celebrities, scenes, landmarks, and artworks.
- Data Construction: All evaluation instructions and answers are manually designed rather than directly using public dataset annotations.The benchmark uses public images in some tasks but avoids directly relying on their original annotations.
- Perception Tasks: OCR is included as a foundational capability, but this version uses 20 images and 40 instruction-answer pairs containing relatively simple samples.The paper states that MLLMs are still in their infancy for this capability.
- Cognition Tasks: The cognition suite tests logical reasoning after image perception, requiring instruction following, visual understanding, and knowledge retrieval.It includes commonsense reasoning, numerical calculation, text translation, and code reasoning.
3 Experiments
The experiments evaluate 30 advanced MLLMs across the benchmark and compare six top models across 14 subtasks. Results show uneven strengths across perception tasks, weak performance on several cognition tasks, and substantial variation between models.
- Experimental Setup: 30 advanced MLLMs are evaluated on the MME benchmark.The evaluated set includes models such as GPT-4V, Qwen-VL-Chat, LLaVA, Lion, WeMM, and SPHINX.
- Perception: Perception results show different models leading object existence, count, position, color, and fine-grained recognition subtasks.Object existence reaches a highest score of 195, while leaders vary across object attributes and recognition categories.
- Error Analysis: Figure 4 illustrates common answer-format and perception problems using ground-truth yes/no labels and generated responses.The figure notation uses [Y]/[N] for ground truth and [R] for the generated answer.
- Perception: Object position is the weakest of the four coarse-grained subtasks, indicating limited sensitivity to positional information.The result is reported across the evaluated MLLMs.
- Cognition: Cognition results show GPT-4V leading overall, with a score of 142.14 in commonsense reasoning and 170 in code reasoning.GPT-4V, Lion, and WeMM receive the gold, silver, and bronze positions across cognition tasks.
- Cognition: None of the highest scores in commonsense reasoning, numerical calculation, or text translation exceeds 150.The paper presents this pattern as evidence that these cognition capabilities still have substantial room for improvement.
4 Analysis
MME analysis identifies four recurring failure modes in MLLMs: instruction-following failures, weak perception, broken reasoning, and object hallucination. These problems can produce incorrect answers even when models partially understand the image or compute the right result.
- Instruction-following failures occur when models answer freely instead of obeying the required yes-or-no format.Responses lacking “yes” or “no” at the beginning are judged incorrect.
- Perception errors include miscounting objects and misreading text, with small instruction changes sometimes producing contradictory results.The examples involve banana counting and character recognition on the same images.
- Reasoning failures arise when models recognize relevant image facts or calculate correctly but still produce the wrong final answer.The paper describes these cases as breaks in the reasoning chain and suggests that chain-of-thought prompts may help.
- Object hallucination occurs when models accept descriptions of objects absent from the image and answer yes.Constantly answering yes yields about 50% accuracy and about 0 accuracy+ in the reported tables.
5 Conclusion
The paper presents MME as the first MLLM evaluation benchmark with four design characteristics and evaluates 30 advanced MLLMs. Results show substantial room for improvement, while the benchmark’s capability coverage remains incomplete.
- MME is presented as the first MLLM evaluation benchmark spanning task type, data source, instruction design, and quantitative statistics.
- 30 advanced MLLMs are evaluated on MME, and the experimental results show substantial room for improvement.
- MME still needs broader capability coverage, including more scenarios requiring reasoning.