PaperlayerView exact revision on arXiv

Source-linked AI summary

MMBench: Is Your Multi-modal Model an All-around Player?

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin

arXiv:2307.06281v5cs.CVcs.CL

TL;DR

Existing VLM evaluation lacks comprehensive quantitative comparability, while common metrics can produce false negatives and limited fine-grained analysis. MMBench is a bilingual, multi-ability benchmark using CircularEval and LLM-assisted answer extraction; its evaluations show CircularEval reveals larger performance gaps than VanillaEval.

  • Problem

    Existing VLM evaluations lack comprehensive quantitative comparability, and common metrics can produce false negatives and lack fine-grained ability analysis.

    Source: §1 · p. 2

  • Method

    MMBench provides a multiple-choice benchmark covering 20 fine-grained abilities and uses CircularEval plus GPT-4 to extract choices from free-form predictions.

    Source: §1 · p. 3, §4.3 · p. 8

  • Results

    CircularEval reveals larger performance gaps than VanillaEval; LLaVA-v1.5-13B's advantage over its 7B counterpart is 4.7% Top-1 accuracy versus 2.1% under VanillaEval.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    MMBench provides a systematic objective benchmark for evaluating VLMs across multiple abilities and yielding robust, reliable evaluation results.

    Source: §6 · p. 12

    Across evaluated VLMs, accuracy varies substantially by ability category, with weaknesses in low-level visual features and structured inputs such as tables, diagrams, and layouts.

    Source: §5.3 · p. 11

Abstract

from arXiv · show

Large vision-language models (VLMs) have recently achieved remarkable progress, exhibiting impressive multimodal perception and reasoning abilities. However, effectively evaluating these large VLMs remains a major challenge, hindering future development in this domain. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but lack fine-grained ability assessment and robust evaluation metrics. Meanwhile, subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, which is not scalable and may display significant bias. In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of the following key features: 1. MMBench is meticulously curated with well-designed quality control schemes, surpassing existing similar benchmarks in terms of the number and variety of evaluation questions and abilities; 2. MMBench introduces a rigorous CircularEval strategy and incorporates large language models to convert free-form predictions into pre-defined choices, which helps to yield accurate evaluation results for models with limited instruction-following capabilities. 3. MMBench incorporates multiple-choice questions in both English and Chinese versions, enabling an apples-to-apples comparison of VLMs' performance under a bilingual context. To summarize, MMBench is a systematically designed objective benchmark for a robust and holistic evaluation of vision-language models. We hope MMBench will assist the research community in better evaluating their models and facilitate future progress in this area. The evalutation code of MMBench has been integrated into VLMEvalKit: https://github.com/open-compass/VLMEvalKit.

1 Introduction

MMBench proposes a systematic objective benchmark for assessing diverse vision-language model abilities, combining fine-grained multiple-choice coverage with methods intended to improve answer matching and evaluation robustness.

  • Public benchmarks can falsely penalize semantically equivalent answers through exact-match metrics and provide limited fine-grained insight when focused on specific tasks.For example, the passage notes that “bicycle” can be marked incorrect when the reference answer is “bike.”Source: §1 · p. 2
  • Human evaluation can account for wording differences and compare multiple facets, but it is biased, costly, difficult to reproduce, and hard to scale.Small evaluation datasets may also produce statistical instability, while collecting more data demands substantial human labor.Source: §1 · pp. 2–3
  • GPT-4 choice matching aligns with human assessments in 91.5% of cases, while CircularEval and choice extraction target robustness and weak instruction-following capability.GPT-4 matches model predictions to candidate choices, including predictions that do not directly output choice labels.Source: §1 · p. 3
  • MMBench evaluates 21 well-known vision-language models across architectures and scales, reporting ability-dimension performance to support model comparison and future optimization.Source: §1 · p. 3
  • MMBench curates 3,217 questions across 20 fine-grained skills, with over 125 questions per ability dimension to support balanced assessment.The benchmark is described as containing over 3,000 questions, with ability dimensions kept roughly equal in size.Source: §1 · p. 3

2 Related Work

Prior multimodal datasets and evaluations cover specialized tasks or offer limited breadth, motivating MMBench’s comprehensive benchmark and evaluation standards. Related models span varied vision-language architectures and instruction-tuning approaches.

2.1 Multimodal Datasets

  • Captioning, VQA, and task-specific datasets use differing metrics, while broader evaluations OwlEval [49] and MME [17] remain limited in size or scale.The paper presents these limitations as obstacles to comprehensive assessment across VLM capabilities.Source: §2.1 · p. 3
  • MMBench introduces a multimodal benchmark and evaluation standards designed to make results stable and accurate.Source: §2.1 · p. 4

2.2 Multimodal Models

  • Recent multimodal models use approaches including gated cross-attention, BLIP-2’s Q-former, instruction tuning, visual-language projection, and LoRA, alongside publicly available proprietary-model APIs.The paper evaluates these models on MMBench and offers insights for future multimodal research.Source: §2.2 · p. 4

3 The construction of MMBench

MMBench combines a hierarchical taxonomy of multimodal abilities with manually collected, quality-controlled multiple-choice questions and a human-verified Chinese version. Its dataset is distributed across ability categories and divided into public development and confidential test subsets.

3.1 The Hierachical Ability Taxonomy of MMBench

  • MMBench organizes abilities under perception and reasoning, with six level-2 and 20 level-3 dimensions.The taxonomy is intended to assess fine-grained abilities; detailed definitions are provided in the Appendix.Source: §3.1 · p. 4

3.2 Data Collection and Quality Control

  • Volunteers manually collect image-based multiple-choice questions for each level-3 ability, drawing on public datasets and the Internet.More than 80% of questions come from the Internet; remaining images come from public-dataset validation sets, with questions self-constructed.Source: §3.2 · pp. 4–5
  • MMBench filters text-answerable questions through LLM majority voting and checks potentially incorrect samples using VLM answers, followed by manual review.Questions are flagged as text-answerable when more than half of the LLMs answer correctly using text alone; samples are potentially problematic when all selected VLMs fail.Source: §3.2 · pp. 5–6
  • MMBench-CN translates questions and choices with GPT-4 and has humans verify the translations, enabling comparison across English and Chinese contexts.Proper nouns, symbols, and code are not translated.Source: §3.2 · p. 6

3.3 MMBench Statistics

  • 3,217 samples span 20 level-3 abilities, with at least 125 samples per category and an effort to keep category counts even.The dataset is split 4:6 into a public development subset with ground-truth answers and a test subset whose answers are confidential and evaluated via a server.Source: §3.3 · p. 6

4 Evaluation Strategy

MMBench combines LLM-assisted choice extraction with CircularEval to evaluate VLMs robustly while managing free-form answers and the bias of multiple-choice guessing. The evaluation tests GPT-4’s extraction accuracy against human annotations and requires consistent success across shuffled-choice passes.

4.1 LLM-involved Choice Extraction

  • The extraction pipeline first uses heuristic matching for a choice label, then asks an LLM—GPT-4 by default—to align unparsed predictions with an option or return pseudo-choice Z if none fits.A prediction is scored correct when the extracted label matches the ground-truth label.Source: §4.1 · p. 7

4.2 LLM as the Choice Extractor: A Feasibility Analysis

  • VLM instruction-following varies widely, and VisualGLM had the lowest heuristic matching success rate at 65%; strong format-following did not necessarily mean strong multimodal understanding.OpenFlamingo v2 [4] had top instruction-following capability but performed among the worst on MMBench.Source: §4.2 · p. 7, Table 1 · p. 7
  • GPT-4 achieved 91.5% human alignment and was selected as the choice extractor; GPT-3.5-Turbo and Qwen-Max reached around 85%, while InternLM2-7B reached 87%.The comparison used about 420 hard examples that could not be parsed by heuristic matching; volunteers manually extracted choices from a 10% sample. Differences among top extractors had little effect on VLM quantitative performance.Source: Figure 5 · p. 7, §4.2 · p. 8

4.3 CircularEval Strategy

  • CircularEval tests each question once per choice by circularly shifting the options and answer, counting it correct only if the VLM succeeds on every pass.A failed pass ends testing early, making practical cost less than N× while reducing effects of random guessing and choice preferences.Source: Figure 6 · p. 7, §4.3 · p. 8

5 Evaluation Results

Models were evaluated zero-shot with a shared prompt, open-ended predictions, and GPT-4-0125 choice extraction; CircularEval was adopted as the default. Results compare overall and fine-grained performance on MMBench and MMBench-CN.

5.2 Main Results

  • 4.7% versus 2.1% Top-1 accuracy separates LLaVA-v1.5-13B from its 7B counterpart under CircularEval and VanillaEval, respectively; OpenFlamingo v2 falls from 36.7% to 2.6%.CircularEval also produces roughly 10% Top-1 drops for state-of-the-art proprietary models and is used as the default evaluation paradigm.Source: §5.2 · pp. 8–9, Table 2 · p. 9
  • 8.3%, 1.5%, and 3.5% are the overall Top-1 gains from 13B over 7B variants in MiniGPT, InstructBLIP, and LLaVA v1.5, respectively.With vision backbone and multimodal corpus held constant in the LLaVA series, switching from Vicuna-v1.5 to InternLM2-20B improves all L-2 abilities, especially reasoning tasks.Source: §5.2 · p. 9
  • InternLM-XComposer2 is the strongest open-source model, while GPT-4v and Qwen-VL-Max lead overall; MiniCPM-V exceeds 60% Top-1 accuracy at ≤3B parameters.The reported overall ranking is for MMBench-test.Source: §5.2 · p. 9, Table 3 · p. 10

5.3 Fine-grained Analysis

  • 2.4% is the maximum estimated MMBench-test accuracy effect of moderation; GPT-4v, Gemini-Pro-V, and Qwen-VL-Max reject 1.8%, 1.6%, and 0.1% of cases, respectively.Among GPT-4v's rejected questions, 74% concern celebrity recognition; no obvious rejection pattern is reported for Gemini-Pro-V.Source: Table 4 · p. 10, §5.3 · p. 10, Figure 8 · p. 10

5.2 Main Results

  • For models above 70% Top-1 on English MMBench, the largest EN-CN performance gap is 2%; InternLM-XComposer2 drops less than 1% on MMBench-CN.Most models score lower on MMBench-CN, except OpenFlamingo v2, VisualGLM, and Qwen-VL-Plus; the authors suggest bilingual capability and balanced cross-language training may help explain the strong results.Source: §5.2 · p. 9, Figure 7 · p. 11

5.3 Fine-grained Analysis

  • Proprietary models outperform LLaVA-InternLM2-20B on structured image-text understanding and tasks requiring external knowledge, but show no advantage on other perception or reasoning capabilities.The comparison covers three proprietary VLMs against the top-performing model trained only on open-source datasets.Source: Figure 9 · p. 11, §5.3 · p. 11
  • Across evaluated VLMs, the hardest cases expose weaknesses in low-level visual features, structured inputs such as tables and diagrams, and 2D or 3D inter-object spatial relations.Figure 10 examples come from the four L-3 abilities with the lowest maximum accuracy across models; all evaluated VLMs answered the displayed examples incorrectly under CircularEval.Source: §5.3 · p. 11, Figure 10 · p. 12

6 Conclusion

MMBench is an objective benchmark for evaluating vision-language models, combining broad multiple-choice coverage with stricter evaluation and methods to make results more reliable.

6 Conclusion

  • MMBench provides over 3,000 multiple-choice questions across 20 ability dimensions for objective VLM evaluation.The benchmark comprehensively evaluates over 20 mainstream VLMs spanning different architectures and parameter sizes.Source: §6 · p. 12
  • CircularEval is stricter than vanilla one-pass evaluation and yields reliable results at an affordable cost.The paper introduces CircularEval to produce robust and reliable evaluations.Source: §6 · p. 12
  • To improve accuracy for VLMs with limited instruction-following ability, MMBench uses LLMs to extract choices from model predictions.The evaluations of more than 20 mainstream VLMs provide insights for future improvements.Source: §6 · p. 12

A More Details about the Data … Logic Reasoning

The appendix defines MMBench’s leaf abilities, illustrates them with data samples, and enumerates the data sources used to construct the benchmark. Its taxonomy spans coarse and fine-grained perception and several forms of reasoning.

A More Details about the Data

  • The appendix introduces its leaf-ability definitions alongside visual samples and enumerates the data sources used to construct MMBench.Figure 11 presents Coarse Perception data samples.Source: A More Details about the Data · p. 13

A.1 Definition about Each Leaf Ability

Coarse Perception

  • Coarse Perception covers image style, scene, emotion, quality, and topic.The appendix also shows data samples for this capability.Source: Coarse Perception · pp. 13–14

Fine-grained Perception (single-instance)

  • Fine-grained single-instance perception covers object localization, attribute recognition, celebrity recognition, and OCR.Localization includes position, coordinates, object count, and orientation; attributes include texture, shape, appearance, emotion, and category.Source: Fine-grained Perception… · p. 14

Fine-grained Perception (cross-instance)

  • Fine-grained cross-instance perception covers spatial relationships, attribute comparisons, and human action recognition.Action recognition includes pose motion and human-object or human-human interaction.Source: Fine-grained Perception (cross-instance) · p. 14

Attribute Reasoning

  • Attribute Reasoning predicts objects’ physical properties and functions, and infers a person’s identity from appearance.Examples include predicting that a broom sweeps and inferring occupation from clothing and appearance.Source: Attribute Reasoning · p. 16

Relation Reasoning

  • Relation Reasoning includes social, physical, and nature relationships.Examples include interpersonal relations, spatial relations and connections between objects, and natural relations such as predation or symbiosis.Source: Relation Reasoning · p. 16

Logic Reasoning

  • Logic Reasoning includes structured image-text understanding and prediction of future events or changes.Examples include parsing chart trends or code in images, and predicting physical phenomena, events, or emotional changes.Source: Logic Reasoning · p. 16

A.2 Data Sources of MMBench · B More Details on MMBench Construction

MMBench draws most of its data from the Internet and uses layered quality controls to identify text-only questions and flawed samples. Its construction details also describe applying those controls to other benchmarks and manually verifying its Chinese translations.

A.2 Data Sources of MMBench

  • 80% of MMBench data comes from the Internet and 20% from validation sets of public datasets.The sources cover images, questions, choices, and answers.Source: §A.2 · p. 17
  • Table 5 catalogs MMBench sources for images, questions, choices, and answers, distinguishing fully customized items from customized-or-selected components.“Customize” means all question, choice, and answer components are constructed by the authors; “Customize & selection” means they are constructed or selected from the original dataset.Source: Table 5 · p. 17

B More Details on MMBench Construction

C More Details on LLM-based Choice Extraction

This section examines failure modes in heuristic answer matching, details the LLM-based choice-extraction prompt, and reports that alternative extractors yield similar MMBench-dev results. It also tests LLM-based semantic matching on existing benchmarks and describes the Chinese translation instructions.

Failure Cases of Heuristic Matching

  • Heuristic matching can fail when GPT-4v declines or cannot answer, paraphrases the correct choice, or includes multiple choice labels in its response.Figure 20 includes an example where GPT-4v cannot identify a band and another where it answers with wording matching the correct option.Source: C More Details on LLM-based Choice… · pp. 18–19, Figure 20 · p. 19

B.1 MMBench-CN Translation

  • The MMBench-CN translation instructions require translating only English question and option text into Chinese while preserving meaning and returning JSON.They also specify retaining proper names, symbols, code, and people’s names in English.Source: C More Details on LLM-based Choice… · p. 19

LLM-based Semantic Matching

  • ChatGPT-based matching follows the same performance trend as exact matching on GQA, OK-VQA, and Text-VQA, with especially close results on GQA.The evaluation randomly sampled 1,000 test examples per benchmark; MiniGPT-4 examples show ChatGPT matching slightly different answers from the ground truth where exact matching fails.Source: C More Details on LLM-based Choice… · pp. 19–20

C.1 Prompt for Choice Extraction

Performance Evaluated with Other Choice Extractors

  • VisualGLM’s scores vary by around 1.4% across choice extractors, while top proprietary VLMs such as GPT-4v and Gemini-Pro-V vary by at most 0.3%.The authors report that changing extractors does not lead to significantly different evaluation results; VLMs with heuristic-matching success rates above 99% are omitted.Source: C More Details on LLM-based Choice… · p. 18, Table 6 · p. 20

D Evaluation Settings and Results

This section describes detailed MMBench evaluation settings and presents additional evaluation results; the main paper reports test-split results for MMBench and MMBench-CN.

D Evaluation Settings and Results

  • The main paper reports test-split results for MMBench and MMBench-CN, while this section adds detailed evaluation settings and further results.Source: D Evaluation Settings and Results · p. 20
  • Table 7 compares LLM-based matching with exact matching on VQA benchmarks, reporting exact-match accuracy and average GPT similarity scores from gpt-3.5-turbo-0613.Each GPT score is an integer from 1 to 5 indicating answer similarity to the ground truth.Source: Table 7 · p. 21
  • Table 8 shows MiniGPT-4 GQA answers that differ from ground truth but can be matched by ChatGPT-based evaluation.Source: Table 8 · p. 21

D.1 Evaluation Settings · D.2 Model Settings

MMBench evaluation results use conventional zero-shot settings, while the appendix also reports attempts at few-shot and chain-of-thought evaluation and documents the evaluated open-source models.

D.1 Evaluation Settings

  • Unless stated otherwise, all reported results use conventional zero-shot evaluation.The appendix provides the prompt template used for zero-shot inference.Source: §D.1 · p. 21
  • Few-shot evaluation was attempted, but it produced no encouraging results.This is reported alongside the paper’s conventional zero-shot evaluation setting.Source: §D.1 · p. 21
  • Chain-of-thought evaluation was attempted, but it produced no encouraging results.This is reported alongside the paper’s conventional zero-shot evaluation setting.Source: §D.1 · p. 21
  • Figure 22 presents the prompt template adopted for zero-shot inference on MMBench.The evaluation-settings passage introduces this prompt as the one used to evaluate a VLM under zero-shot settings.Source: Figure 22 · p. 21, §D.1 · p. 21

D.2 Model Settings

  • Table 9 provides details of all open-source models evaluated in MMBench.It also includes several additional models that do not fit in the main article.Source: §D.2 · p. 21
  • Table 9 includes additional evaluated open-source models that do not fit in the main article.The appendix identifies these models as part of its model-setting details.Source: §D.2 · p. 21

D.3 More Results

This section presents detailed MMBench and MMBench-CN evaluations of 30 VLMs, with L-2 CircularEval results spanning development and test sets in both languages.

D.3 More Results

  • The detailed evaluation covers 30 VLMs, including models omitted from the main paper because of space limitations.Results for each L-3 ability are provided in a separate supplementary-materials sheet.Source: §D.3 · p. 21
  • CircularEval results cover L-2 abilities on the MMBench-dev, MMBench-test, MMBench-CN-dev, and MMBench-CN-test sets.Open-source models marked with * incorporate in-house data in training.Source: Table 10 · p. 23, Table 11 · p. 24, Table 12 · p. 25

Related papers

Loading 2307.06281v5…