Source-linked AI summary
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, Lijuan Wang
TL;DR
MM-Vet addresses the lack of systematic evaluation for complicated multimodal tasks and the difficulty of scoring diverse open-ended responses. It defines integrated VL-capability tasks and uses an LLM-based evaluator to benchmark LMMs, with GPT-4V reaching 67.7% while top methods remain around 68%, indicating substantial room for improvement.
Problem
Existing benchmarks do not systematically evaluate complicated multimodal tasks, while diverse answer styles and question types complicate unified metric design.
Method
MM-Vet defines six core VL capabilities and 16 integrated tasks, then uses a few-shot LLM-based evaluator to score open-ended outputs across question types and answer styles.
Results
67.7%: GPT-4V surpasses open-sourced LMMs and LLM-based multimodal agents on MM-Vet by substantial margins.
Takeaways & Limitations
Per-capability analysis provides insights beyond overall ranking into the relative strengths and weaknesses of different LMM system paradigms and models.
Takeaways & Limitations
MM-Vet focuses on image-text input and text output, and its GPT-4 evaluator incurs usage fees because GPT-4 currently best matches human grades.
Abstract
from arXiv · showhide
We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes. Rapid model advancements pose challenges to evaluation benchmark development. Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking. To this end, we present MM-Vet, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination. For evaluation metrics, we propose an LLM-based evaluator for open-ended outputs. The evaluator enables the evaluation across different question types and answer styles, resulting in a unified scoring metric. We evaluate representative LMMs on MM-Vet, providing insights into the capabilities of different LMM system paradigms and models.
1. Introduction
MM-Vet addresses the difficulty of evaluating complicated multimodal tasks by organizing them around integrated VL capabilities and using an LLM-based evaluator for diverse open-ended outputs. It benchmarks representative LMMs to reveal capability-specific strengths and weaknesses beyond overall ranking.
- Existing VL benchmarks mainly test one or two capabilities, leaving complicated multimodal tasks and their relationships insufficiently evaluated.
- Diverse answer formats and task-specific evaluation priorities make a single metric difficult: math may require one word, whereas essay writing requires lengthy, high-quality text.
- MM-Vet benchmarks end-to-end LMMs, tool-using systems, and industry solutions to expose relative strengths and weaknesses across model paradigms.
- MM-Vet defines six core VL capabilities and 16 emergent tasks that integrate these capabilities for quantitative evaluation.
- The proposed LLM-based evaluator unifies scoring across question types and answer styles while evaluating factual correctness and text quality.
2. Related work
Prior VL benchmarks target specialized capabilities, while newer evaluation studies broaden coverage but generally emphasize comprehensive samples or overall rankings. MM-Vet instead focuses on integrated capabilities and unified open-ended evaluation.
- Classic VL benchmarks focus on specialized abilities such as recognition, image description, scene text understanding, commonsense reasoning, and outside knowledge.
- MM-Vet differs from concurrent comprehensive benchmarks by defining and studying integrated VL capabilities, enabling insights beyond overall model ranking.
- Its open-ended LLM-based evaluator handles different answer styles and question types without requiring binary or multiple-choice answers.
- Figure 2 separates the proportions of individual capabilities from capability integrations: individual proportions can exceed 100%, whereas integration proportions sum to 100%.
3. MM-Vet
MM-Vet constructs open-ended multimodal questions requiring combinations of six VL capabilities and evaluates responses with a few-shot GPT-4 scoring procedure. The benchmark aggregates sample scores overall and by capability or integration.
- Core capabilities: The six capabilities are recognition, knowledge, OCR, spatial awareness, language generation, and math.
- Benchmark motivation: MM-Vet targets realistic tasks requiring integrated capabilities, such as visual-joke explanation, document-math solving, and image-based exam questions.
- Data collection: The dataset contains 200 images and 218 open-ended questions paired with ground truths, with 155 annotations produced by the authors and 50 answers collected from the Internet.
- LLM-based evaluator: GPT-4 evaluates each response from the question, ground truth, and model output using few-shot examples covering short and long answers and partial correctness.
- Scoring: Each sample receives a score from 0 to 1, while capability and integration scores average the relevant sample scores and multiply by 100%.
4. Evaluation results
MM-Vet evaluates end-to-end tuned and tool-using multimodal systems across core capabilities and their integrations. Results show strong specialization: tool-using systems lead OCR, spatial awareness, and math, while tuned models lead some recognition-focused categories.
- Experiment settings: MM-Vet evaluates end-to-end tuned LMMs and LLM-tool-using methods across core capabilities and capability integrations.The reported systems include OpenFlamingo, BLIP-2, LLaVA, MiniGPT-4, LLaMA-Adapter V2, Otter, InstructBLIP, MM-ReAct, and Transformers Agent.
- Core capabilities: 39.2%: LLaVA-13B (LLaMA-2) achieves the best overall recognition capability performance.It improves 8.3% over LLaVA-13B (Vicuna-13B), which the authors associate with its vision model and stronger language model.
- Core capabilities: 65.7%: MM-ReAct-GPT4 performs best in OCR, while LLaVA-13B (LLaMA-2) leads end-to-end tuned models at 22.7%.MM-ReAct-GPT4 uses an external OCR model as a tool.
- Core capabilities: 31.4%: LLaMA-Adapter v2-7B leads knowledge, while MM-ReAct-GPT4 reaches 29.0% using GPT-4 and external knowledge tools.The knowledge category spans joke understanding and encyclopedia-style questions.
- Core capabilities: 56.8%: MM-ReAct-GPT4 leads spatial awareness, supported by dense captioning and OCR tools that provide coordinate information.Among end-to-end tuned models, LLaVA-13B (V1.3, 336px) performs best at 31.3%.
- Capability integrations: MM-ReAct-GPT4 consistently outperforms other models in math, while integrated-task results show tool and model choices affect specific capability combinations.The authors attribute its math performance to the PAL math tool and report category-specific leaders including LLaMA-Adapterv2-7B, InstructBLIP, and LLaVA variants.
4.3. Result discussion
MM-Vet analyses how foundation models, vision components, language models, and tuning data relate to integrated multimodal performance. GPT-4V achieves 67.7%, substantially exceeding open-sourced LMMs and multimodal agents.
- Vision: Vision-encoder superiority remains unresolved because comprehensive ablation studies are unavailable.With Vicuna-7B, InstructBLIP-8B excels at recognition, whereas LLaVA-7B performs particularly well on OCR.
- Language: Stronger language models generally improve LMM performance, except InstructBLIP-8B outperforming its 14B counterpart.
- Tuning data: More tuning data can enhance performance, as InstructBLIP-8B uses 26 public datasets and scores higher than BLIP-2-12B.
- Comparison with GPT-4V: 67.7%: GPT-4V surpasses open-sourced LMMs and LLM-based multimodal agents by substantial margins.
- Capability analysis: Specialized tools benefit OCR and math, while recognition and language generation require stronger vision and language modules, respectively.
4.4. Effectiveness analysis of LLM-based evaluation
The analysis tests LLM-based evaluation against human judgments and prompt variations. GPT-4 provides the most reliable evaluator among the compared systems, with all seven few-shot examples producing the closest alignment.
- Evaluator comparison: 0.273: Keyword matching has a high average discrepancy from human scores, making it unsuitable for open-ended MM-Vet answers.The discrepancy is measured as the average absolute difference between evaluator and human scores.
- Evaluator comparison: GPT-4 outperforms the compared open-source evaluators, while Gemini and Claude remain substantially less aligned with human scores.
- Evaluator combination: GPT-4 alone performs best among combined evaluator configurations, so MM-Vet uses GPT-4 (0613) for evaluation.
- Prompt ablation: All seven few-shot grading examples yield the closest alignment with human evaluations.The criterion is the lowest average discrepancy ∆.
- Overall findings: GPT-4V outperforms existing open-sourced methods, while MM-ReAct-GPT-4 ranks second with effective external tools.
- Overall findings: Top-performing methods achieve only around 68% on MM-Vet, whose full score is 100%.
5. Conclusion and Limitation
MM-Vet benchmarks integrated vision-language capabilities using a multimodal dataset and an LLM-based evaluator. The evaluation shows that even GPT-4V scores only around 68%, while the benchmark remains limited to image-text inputs and text outputs and relies on GPT-4 evaluation.
- Conclusion: MM-Vet evaluates integrated VL capabilities with a dataset requiring multiple capabilities and an LLM-based evaluator for open-ended outputs.
- Conclusion: Around 68%: Even GPT-4V achieves only this score on MM-Vet, indicating substantial room for improving integrated VL capabilities.
- Limitations: MM-Vet covers image-text inputs and text outputs, excluding other modalities accepted by fewer popular LMMs.
- Limitations: GPT-4 is required for evaluation because current alternatives do not match human grades well, creating usage fees.
Impact Statement
MM-Vet provides a benchmark for assessing existing LMM capabilities and supporting future advances in multimodal model development.
- MM-Vet assesses existing LMM capabilities and establishes groundwork for future advancements in the field.
A. Model details
The evaluated model details are provided in Table 11.
- Table 11 contains the details of the models evaluated in MM-Vet.
- Readers can consult Table 11 for the evaluated models’ specifications.
B. Comparison with Bard
On the Bard set, Bard leads overall and across several capabilities, while MM-ReAct-GPT-4 leads the remaining capabilities and integrations. The comparison also indicates a gap between these systems and open-source end-to-end models such as LLaVA.
- 53.5% overall, Bard’s highest score on the Bard set, exceeds the evaluated alternatives overall.Bard also achieves the highest scores in three of six capabilities and seven of fifteen capability integrations.
- MM-ReAct-GPT-4 leads the remaining three of six capabilities and nine of fifteen capability integrations.
- MM-ReAct-GPT-4 performs better in OCR, spatial awareness, and math, highlighting the potential benefit of specialized external tools.
- The Bard-set results are reported in Tables 9 and 10, with capability-level and integration-level evaluations.
- Open-source end-to-end models such as LLaVA still show a considerable gap from the compared systems.
C. Examples of capability integrations
MM-Vet examples span simple and complex multimodal questions, illustrating integrations of recognition, OCR, knowledge, generation, spatial awareness, and math. The examples include memes, charts, tables, scene text, spatial queries, and visual arithmetic tasks.
- Organization: The examples are organized across tables documenting samples that require different capability integrations.
- Meme examples: Meme questions combine recognition, OCR, knowledge, and language generation to explain humor or identify appropriate usage contexts.
- Chart and table examples: Chart and table prompts require OCR, language generation, and spatial awareness to summarize trends and compare quantitative features.
- Recognition and spatial examples: Scene-text and spatial questions ask models to read labels, locate objects, identify parking spaces, or determine relative positions.
- Chart and table examples: The underground-railway example compares networks by age, route length, and annual passenger volume across six cities.
- Integrated examples: Other examples integrate recognition, OCR, spatial awareness, and math, including predicting board content or calculating gasoline quantity.
D. GPT-4V prediction examples
The GPT-4V examples illustrate both successful and unsuccessful predictions across multimodal tasks. Reported examples include meme explanation and long-form table summarization, with the meme response receiving a perfect score.
- GPT-4V examples include both success and failure cases across the benchmark’s multimodal tasks.
- 1.0, GPT-4V’s score for explaining the procrastination meme, reflects a successful multimodal prediction.
- GPT-4V explains the meme’s procrastination theme and connects the delayed schedule with Kermit’s relaxed attitude.
- A further example asks GPT-4V to summarize an underground-railway table in at least 150 words.