PaperlayerView exact revision on arXiv

Source-linked AI summary

MMBench: Is Your Multi-modal Model an All-around Player?

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin

arXiv:2307.06281v5cs.CVcs.CL

TL;DR

Existing VLM benchmarks provide limited fine-grained, robust, and scalable evaluation of multimodal abilities. MMBench addresses this gap with a systematically designed objective benchmark, CircularEval, and LLM-based choice extraction; its evaluations provide insights for future VLM improvement.

  • Problem

    Existing benchmarks lack fine-grained ability assessment and robust metrics, while human-based evaluation is not scalable and may be biased.

  • Method

    MMBench uses over 3,000 multiple-choice questions covering 20 ability dimensions, CircularEval, and LLM-based extraction of choices from free-form predictions.

  • Results

    MMBench evaluations of over 20 mainstream VLMs provide insights into model performance and future improvement across architectures and parameter sizes.

  • Takeaways & Limitations

    MMBench offers a robust and holistic objective framework for evaluating diverse vision-language model abilities.

  • Takeaways & Limitations

    Existing VLMs remain limited in recognizing low-level visual features and understanding structured inputs such as tables, diagrams, and layouts.

Abstract

from arXiv · show

Large vision-language models (VLMs) have recently achieved remarkable progress, exhibiting impressive multimodal perception and reasoning abilities. However, effectively evaluating these large VLMs remains a major challenge, hindering future development in this domain. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but lack fine-grained ability assessment and robust evaluation metrics. Meanwhile, subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, which is not scalable and may display significant bias. In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of the following key features: 1. MMBench is meticulously curated with well-designed quality control schemes, surpassing existing similar benchmarks in terms of the number and variety of evaluation questions and abilities; 2. MMBench introduces a rigorous CircularEval strategy and incorporates large language models to convert free-form predictions into pre-defined choices, which helps to yield accurate evaluation results for models with limited instruction-following capabilities. 3. MMBench incorporates multiple-choice questions in both English and Chinese versions, enabling an apples-to-apples comparison of VLMs' performance under a bilingual context. To summarize, MMBench is a systematically designed objective benchmark for a robust and holistic evaluation of vision-language models. We hope MMBench will assist the research community in better evaluating their models and facilitate future progress in this area. The evalutation code of MMBench has been integrated into VLMEvalKit: https://github.com/open-compass/VLMEvalKit.

1 Introduction

MMBench addresses limitations in existing objective and subjective VLM evaluation by combining a broad, curated multiple-choice benchmark with robust choice extraction and evaluation procedures.

  • The benchmark evaluates models across diverse multimodal perception and reasoning abilities rather than relying primarily on qualitative examples.Its motivation follows recent progress in large language and vision-language models, alongside the need for comprehensive quantitative comparison.
  • Existing objective benchmarks can produce false negatives and lack fine-grained analysis when evaluating advanced vision-language models.Exact-match metrics may score semantically equivalent answers differently, while existing datasets do not comprehensively assess model abilities.
  • Subjective evaluations offer broader assessment but are difficult to reproduce because human judgments are biased and evaluations do not scale efficiently.Repeated annotation is expensive, and robust evaluation may require collecting more data and human labor.
  • MMBench provides over 3,000 multiple-choice questions covering 20 ability dimensions, with roughly equal question counts across dimensions.The benchmark is designed to support balanced assessment of abilities such as object localization and social reasoning.
  • MMBench uses CircularEval and GPT-4-based choice matching to improve evaluation for models that output free-form answers or exhibit choice preferences.CircularEval evaluates each question under circularly shifted answer choices, while GPT-4 maps predictions to candidate labels.

2 Related Work

Prior multimodal research developed datasets and architectures for evaluating or improving vision-language models, while MMBench introduces a benchmark with stability and accuracy-oriented evaluation standards.

  • Multimodal Datasets: Existing multimodal datasets support tasks such as image captioning and visual question answering with human-generated annotations.COCO Caption, Nocaps, and Flickr30k provide image captions and corresponding text-description tasks.
  • Evaluation Benchmarks: MMBench presents a multimodal benchmark together with evaluation standards intended to improve the stability and accuracy of evaluation results.The benchmark is positioned as an evaluation contribution alongside prior multimodal datasets and model developments.
  • Multimodal Models: Recent multimodal models build on large language models and use architectures that connect visual features with language representations.Examples include Flamingo’s gated cross-attention blocks and BLIP-2’s Querying Transformer.

3 The construction of MMBench

MMBench is constructed as a bilingual, hierarchical benchmark with rigorous quality control for evaluating diverse vision-language abilities. It combines manually collected multiple-choice questions, balanced ability coverage, filtering strategies, and separate development and test splits.

  • Hierarchical Ability Taxonomy: MMBench organizes evaluation into a hierarchical taxonomy with perception and reasoning at level 1, six level-2 dimensions, and 20 level-3 abilities.The taxonomy is designed around fine-grained multimodal perception and reasoning capabilities.
  • Data Collection: The benchmark collects multiple-choice questions for each level-3 ability from public datasets and Internet sources, with manually gathered images, questions, choices, and answers.More than 80% of questions come from the Internet; remaining images come from public-dataset validation sets when available, while questions are self-constructed.
  • Quality Control: Quality control filters samples answerable from text alone and samples containing flawed questions, choices, or answers through majority voting, model checking, and manual review.Questions that all evaluated VLMs answer incorrectly are treated as potentially problematic and manually checked before exclusion when they are wrong.
  • Bilingual Benchmark: MMBench-CN translates questions and choices into Chinese with GPT-4 and human verification, enabling direct comparison of VLM performance across English and Chinese contexts.Proper nouns, symbols, and code are excluded from translation.
  • Data Statistics: MMBench contains 3,217 samples across 20 level-3 abilities, maintaining at least 125 samples per ability to support balanced evaluation.The dataset aims to keep problem counts roughly even across abilities.
  • Data Splits: The dataset is split into development and test subsets at a 4:6 ratio, releasing development answers publicly while keeping test answers confidential through server-based evaluation.The test subset releases data samples but requires prediction submission to obtain evaluation results.

4 Evaluation Strategy

MMBench combines LLM-based choice extraction with CircularEval to handle free-form VLM outputs and reduce evaluation bias from answer-order preferences. Its evaluation strategy validates extractor quality against human annotations and requires success across circularly shifted passes.

  • LLM-involved Choice Extraction: Many VLMs produce free-form answers despite multiple-choice prompts, and proprietary models plus many open-source models often fail heuristic output matching.Some models exceed 99% heuristic matching success, but these are not listed in the reported statistics.
  • LLM-involved Choice Extraction: LLM-based choice extraction converts free-form VLM predictions into answer labels when heuristic matching fails, including a pseudo-choice Z for predictions unlike all options.The process first attempts heuristic extraction, then provides the question, choices, and prediction to an LLM for alignment.
  • LLM-involved Choice Extraction: GPT-4 achieves a 91.5% human alignment rate, while GPT-3.5-Turbo and Qwen-Max achieve around 85%.InternLM2-7B reaches an 87% alignment rate, and gpt-4-0125 is adopted as the choice extractor.
  • CircularEval Strategy: CircularEval tests each question N times with circularly shifted choices and counts it correct only when the VLM succeeds in every pass.If a VLM fails during a pass, later passes need not be executed, making the practical cost lower than N×.
  • CircularEval Strategy: CircularEval addresses approximately 25% random-guessing accuracy in four-choice questions and bias toward predicting a particular choice.The strategy is designed to trade off evaluation robustness against computational cost.

5 Evaluation Results

MMBench evaluates VLMs with CircularEval across overall and fine-grained abilities, revealing stronger robustness requirements, model-family effects, bilingual performance patterns, proprietary-model advantages in selected tasks, and persistent hard cases.

  • Evaluation Strategy: CircularEval generally lowers accuracy but widens model differences; LLaVA-v1.5-13B’s gap over 7B increases from 2.1% to 4.7% Top-1 accuracy.OpenFlamingo v2 drops from 36.7% to 2.6%, while proprietary VLMs suffer approximately 10% Top-1 accuracy drops.
  • Overall Results: InternLM-XComposer2 achieves the strongest open-source overall performance, while GPT-4v and Qwen-VL-Max remain stronger closed-source models.MiniCPM-V exceeds 60% Top-1 accuracy despite having no more than 3B parameters; OpenFlamingo v2 performs at random level.
  • Model Analysis: Changing the language model improves performance across L-2 abilities, especially reasoning; 13B variants exceed 7B counterparts by 8.3%, 1.5%, and 3.5% overall Top-1 accuracy in three model series.The comparisons concern MiniGPT, InstructBLIP, and LLaVA v1.5, respectively.
  • Bilingual Results: Most VLMs perform worse on MMBench-CN, but models scoring above 70% in English show at most a 2% gap; InternLM-XComposer2 drops by less than 1%.The paper associates these patterns with language-model bilingual ability and the balance of multilingual multimodal training data.
  • Fine-grained Analysis: Content moderation affects proprietary-model accuracy by up to 2.4% under an upper-bound recalculation, with rejection rates of 1.8%, 1.6%, and 0.1% for GPT-4v, Gemini-Pro-V, and Qwen-VL-Max.The paper reports that 74% of GPT-4v’s rejected questions concern celebrity recognition.
  • Fine-grained Analysis: Proprietary VLMs outperform open-source models on structured image-text understanding and tasks requiring external knowledge, but show no advantage on other perception or reasoning tasks.The comparison uses three proprietary VLMs and LLaVA-InternLM2-20B.
  • Fine-grained Analysis: MMBench identifies weaknesses in low-level visual features, structured inputs such as tables and diagrams, and inter-object spatial relationships.These hard cases are drawn from L-3 capabilities with the lowest maximum accuracy across evaluated VLMs.

6 Conclusion

MMBench is an objective benchmark with over 3,000 multiple-choice questions spanning 20 ability dimensions, combining CircularEval and LLM-based choice extraction for robust VLM evaluation.

  • Conclusion: MMBench contains over 3,000 multiple-choice questions covering 20 ability dimensions for objective VLM evaluation.The benchmark is designed to evaluate models across different architectures and parameter sizes.
  • Conclusion: CircularEval provides stricter evaluation than one-pass inference, while LLM-based choice extraction supports models with limited instruction-following ability.The paper presents these mechanisms as producing reliable evaluation results at affordable cost.
  • Conclusion: Comprehensive evaluation of more than 20 mainstream VLMs yields insights intended to guide future model improvements.

A More Details about the Data

The appendix details MMBench’s fine-grained ability taxonomy through sample questions and visualizations spanning perception, reasoning, relations, and structured image-text understanding.

  • Coarse Perception: Coarse Perception covers image style, scene, emotion, quality, and topic, with samples visualized in Figure 11.Image quality includes properties such as blur, brightness, darkness, and contrast.
  • Fine-grained Perception (single-instance): Fine-grained single-instance perception includes object localization, attribute recognition, celebrity recognition, and OCR.These abilities address positions, object counts, visual attributes, known entities, and text or formulas in images.
  • Fine-grained Perception (cross-instance): Fine-grained cross-instance perception covers spatial relationships, attribute comparison, and action recognition across objects or people.Examples include relative position, comparing shape or color, and recognizing poses or interactions.
  • Attribute Reasoning: Attribute Reasoning includes physical-property, function, and identity reasoning about objects and people.Examples include predicting an object’s use, material-related properties, or a person’s identity.
  • Relation Reasoning: Relation Reasoning covers social, physical, and nature relations, including interpersonal ties, spatial connections, predation, and symbiosis.The sample questions ask about family relationships, relative proximity, and ecological interactions.
  • Logic Reasoning: Logic Reasoning includes structured image-text understanding and future prediction, with examples visualized in Figure 16.Structured inputs include charts, code, and other image-text layouts.

A.2 Data Sources of MMBench

MMBench draws most data from the Internet and the remainder from public-dataset validation sets, then applies systematic quality control and manual verification, including for Chinese translation.

  • Data Sources: MMBench obtains 80% of its data from the Internet and 20% from validation sets of public datasets.Table 5 lists the sources for images, questions, choices, and answers.
  • Quality Control: Text-only filtering uses GPT-4, Gemini-Pro, and Qwen-Max under CircularEval, removing manually verified questions that multiple LLMs answer correctly without images.Questions answered correctly by more than two LLMs are manually checked and removed when unqualified.
  • Quality Control: Wrong-question filtering uses three proprietary and two open-source VLMs, followed by manual checks when none answers correctly under CircularEval.The procedure targets ambiguous questions, repeated options, and incorrect answers.
  • Quality Control: The quality-control paradigm also detects unqualified samples in other multimodal benchmarks such as MME and SEEDBench.
  • MMBench-CN Translation: GPT-4 translates MMBench into Chinese using instructions and in-context examples, and all translations receive manual verification.The translated benchmark is MMBench-CN.

C More Details on LLM-based Choice Extraction

MMBench uses LLM-based choice extraction to map free-form answers to predefined options, addressing failures of heuristic matching. Different extractors produce broadly consistent evaluation results, while LLM matching can recognize semantically equivalent answers.

  • Failure Cases of Heuristic Matching: Heuristic matching can fail when VLMs reject questions, answer with different wording, or include multiple-choice labels.GPT-4v exhibits these failure patterns during exact matching.
  • The Prompt for LLM-based Choice Extraction: In-context examples are included in the extraction prompt to improve the adopted LLM’s instruction-following capability.
  • Performance Evaluated with Other Choice Extractors: Different choice extractors do not produce significantly different MMBench-dev results; the largest range is around 1.4% for VisualGLM and at most 0.3% for top proprietary VLMs.The comparison includes GPT-4, GPT-3.5-Turbo, and InternLM2-7B, while models with over 99% heuristic-matching success are skipped.
  • The Prompt for LLM-based Choice Extraction: The choice-extraction prompt asks an LLM to select the option most similar in literal meaning, or output Z when no option matches.The prompt prohibits external inference and requires a single uppercase choice label.
  • MMBench-CN Choice Extraction: Chinese choice extraction uses a Chinese translation of the prompt adopted for MMBench.
  • LLM-based Semantic Matching: GPT-3.5-Turbo-based semantic matching can match slightly different MiniGPT-4 answers that exact matching fails to recognize.The similarity score ranges from 1, completely wrong, to 5, completely correct.

D Evaluation Settings and Results

The supplementary evaluation reports CircularEval results and examines LLM-based matching against exact matching on VQA benchmarks. It also documents MiniGPT-4 answers that ChatGPT-based evaluation can match despite differences from ground truth.

  • Evaluation Settings and Results: The evaluation compares LLM-based matching with exact matching on VQA benchmarks using accuracy and average GPT similarity scores.GPT-3.5-Turbo-0613 assigns each answer an integer similarity score from 1 to 5.
  • Evaluation Settings and Results: ChatGPT-based evaluation matches MiniGPT-4 GQA answers that differ from the ground-truth answer.

D.1 Evaluation Settings

MMBench evaluations use a conventional zero-shot setting, with a prompt template provided for inference. The supplementary material also lists the evaluated open-source models.

  • Evaluation Settings: MMBench results generally follow the conventional zero-shot evaluation setting.Few-shot and chain-of-thought evaluations were also attempted, but no encouraging results were observed.
  • Evaluation Settings: The zero-shot evaluation uses the prompt template shown for VLM inference.
  • Evaluation Settings: The supplementary material provides details of all evaluated open-source models, including additional models omitted from the main article due to space limitations.

D.3 More Results

The supplementary material reports detailed MMBench and MMBench-CN results for 30 VLMs, including CircularEval evaluations across development and test splits. It also provides a table of evaluated open-source models.

  • D.3 More Results: Detailed results are reported for 30 different VLMs on MMBench and MMBench-CN.Separate supplementary sheets provide results for each L-3 ability.
  • Evaluated Models: The evaluated open-source VLMs are documented in a dedicated supplementary table.
  • MMBench Results: CircularEval results are provided for the MMBench-dev and MMBench-test sets across L-2 abilities.Open-source models marked with an asterisk incorporate in-house data during model training.
  • MMBench-CN Results: CircularEval results are also provided for the MMBench-CN-dev and MMBench-CN-test sets across L-2 abilities.The tables identify open-source models that incorporate in-house training data.
Loading 2307.06281v5…