PaperlayerView exact revision on arXiv

Source-linked AI summary

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, Wenqi Shao

arXiv:2404.16006v1cs.CV

TL;DR

Task breadth is a fundamental criterion for levels of AGI, making broad evaluation relevant to assessing LVLM progress toward multitask AGI. MMT-Bench evaluates LVLMs across 162 multimodal tasks, and results reveal significant challenges for current models, including GPT-4V's 62.0/100 overall score.

  • Problem

    Task breadth across domains is a fundamental criterion for levels of AGI, making broad evaluation relevant to assessing LVLM progress toward multitask AGI.

    Source: §1 · p. 1, §6 · p. 9

  • Method

    MMT-Bench is a curated benchmark of 31,325 multiple-choice questions across 162 multimodal tasks, with a task-map analysis of LVLMs.

    Source: §6 · p. 9

  • Results

    Evaluations found significant challenges for current LVLMs, with GPT-4V scoring 62.0/100 overall across all subtasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MMT-Bench's task-map analysis allows prediction of LVLM performance on a new task.

    Source: §6 · p. 9

    Curated data and task selection may bias assessment and only partially capture real-world scenarios, limiting insight into LVLM capabilities across domains and populations.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises $31,325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $30$ LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.

1. Introduction

MMT-Bench is designed to address the gap between LVLM progress and evaluation breadth, which matters for assessing models’ ability to perform across diverse tasks. Its broad task map supports evaluation of multimodal multitask capability and discovery of in- and out-of-domain tasks.

1. Introduction

  • Prior benchmarks offer limited task coverage and test rudimentary capabilities, while MathVista (Lu et al., 2023) and MMMU (Yue et al., 2023a) focus on scientific-diagram questions.The introduction argues that this narrow coverage cannot meet the task-breadth requirement for evaluating multitask AGI.Source: §1 · p. 1
  • MMT-Bench contains 32K questions spanning 32 meta-tasks and 162 subtasks, 13 image types, and scenarios including vehicle driving, GUI navigation, and embodied AI.Its 14 multimodal capabilities include recognition, localization, reasoning, OCR, counting, 3D perception, and temporal understanding.Source: §1 · p. 2, Figure 1 · p. 2
  • InternVL-Chat led MMT-Bench, surpassing QWen-VL-Plus, GPT-4V, and GeminiProVision; GPT-4V scored 62.0/100 across all subtasks and 55.6/100 excluding visual-recognition subtasks.The authors report that 30 publicly available LVLMs were assessed under various input modes.Source: §1 · p. 2
  • Error analyses found perception, reasoning, and knowledge errors prevalent among leading models; taxonomy analysis found strength in in-domain recognition and description but weakness in out-of-domain localization and pixel perception.The analyses cover 162 multimodal tasks.Source: §1 · p. 2
  • BLIP2, despite lacking instruction tuning, outperformed most models tuned on millions of instruction-following examples; performance also improved from 7B to 13B for both LLaVA variants and with InternLM2 upgrades.The authors suggest instruction tuning on some tasks may hurt generalization to others.Source: §1 · pp. 2–3
  • Specific prompting methods improved some multi-image, coordinate-related, and visual-referring tasks, but most models did not improve with visual prompting.The authors identify visual prompting as a potential area for future enhancement.Source: §1 · pp. 2–3

2. Related Work

Related work frames MMT-Bench as a broad evaluation of LVLMs, addressing limited task coverage in existing benchmarks and drawing on task-map approaches to analyze multitask capabilities.

LVLM

LVLM Evaluation

  • Existing holistic LVLM benchmarks cover limited multimodal skills, while MMT-Bench evaluates broad reasoning requiring expert knowledge, visual recognition, localization, reasoning, and planning.The paper describes MMT-Bench as challenging for current state-of-the-art LVLMs.Source: §2 · p. 3

Multitask Analysis

  • Taskonomy models visual-task relationships through transfer learning (Zamir et al., 2018), Task2Vec uses Fisher information as task vectors (Achille et al., 2019), and this paper uses its collected task data to map LVLM performance and identify challenging tasks.Source: §2 · pp. 3–4

3. MMT-Bench

MMT-Bench organizes multimodal evaluation into a broad hierarchy of tasks and builds its multiple-choice visual questions through a curated data-collection pipeline. Its 31,325 questions span 32 meta-tasks, 162 subtasks, and diverse image types and capabilities.

3.1. Tasks

  • The benchmark defines 32 meta-tasks through brainstorming, then selects 162 subtasks for basic capability coverage, challenge to current LVLMs, and public test-sample availability.The 162 subtasks are reported as 3.8 times the number in TinyLVLM-eHub.Source: §3.1 · p. 4

3.2. Data Collection

  • Each task’s metadata combines images with annotations for question generation, and sampling caps tasks at 200 samples while keeping dataset sample counts equal.Questions have up to eight choices and are generated using manually designed rules or ChatGPT prompts.Source: §3.2 · p. 4
  • 31,325 questions span 13 input image types, 32 meta-tasks, 162 subtasks, and 14 capabilities across scenarios including GUI navigation and document understanding.The benchmark tests skills such as visual recognition, localization, reasoning, OCR, counting, 3D perception, and temporal understanding.Source: §3.2 · p. 5

4. Experiments

Experiments evaluate 30 LVLMs on MMT-Bench and examine overall accuracy, prompting strategies, and error distributions. The results reveal capability gaps across tasks and highlight effects of model choice and prompting.

4.1. Evaluation Details

  • Accuracy is the primary metric, with responses extracted by checking option letters, matching answer content, then using ChatGPT, and assigning Z if extraction fails.The evaluation covers 30 representative LVLMs varying in parameter counts, vision encoders, and LLMs.Source: §4.1 · p. 5

4.2. Overall Evaluation

  • Most LVLMs perform best on Visual Recognition and Visual Captioning but struggle with fine-grained perception and complex reasoning; larger or upgraded LLMs improve LLaVA performance.BLIP2 also outperforms instruction-tuned LLaVA models in MMT-Bench’s close-set settings, despite lacking instruction-following training.Source: §4.2 · p. 5
  • InternVL-Chat, GPT-4V, and GeminiProVision reach 63.4%, 62.0%, and 61.6% accuracy, respectively, while MMT-Bench still exposes substantial capability gaps.GeminiProVision’s accuracy drops to 55.1% when Visual Recognition, where it scores 84.7%, is excluded.Source: §4.2 · p. 5, Table 2 · p. 6

4.3. Specific Task and Prompt Methods Analysis

  • GeminiProVision achieves only 53.8% accuracy on multi-image tasks, while multi-image prompting improves most evaluated models, including its face-retrieval score from 30.5 to 92.5.The analysis covers 28 tasks that usually require multiple images; for single-image LVLMs, images were processed individually and their visual embeddings concatenated, or images were combined into one for single-image prompting.Source: §4.3 · pp. 6–7, Figure 3 · p. 7
  • Visual prompting substantially underperforms text-based bounding-box prompts and combined visual-text prompts, likely because most LVLMs lack visual-prompting data during supervised fine-tuning.The comparison uses 14 subtasks involving visual referring prompts.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4. Error Analysis

  • Perception errors are most common across the three analyzed models, while reasoning errors are second most common.GPT-4V’s perception error rate is 51%, versus 76.9% for GeminiProVision and 67.2% for InternVL; reasoning error rates are 9.94%, 10.4%, and 14.8%, respectively. Knowledge errors range from 6.99% to 9.0%; GPT-4V has higher lack-of-capability and refusal rates (19% and 6.11%), while InternVL has the highest instruction-following error rate (6.64%), compared with 2.99% for GPT-4V and 1.14% for GeminiProVision.Source: §4.4 · pp. 7–8

5. Taskonomy Analysis

MMT-Bench’s task map supports systematic analysis of relationships among multimodal subtasks. The analysis finds that task proximity corresponds to more consistent LVLM rankings and helps distinguish model strengths from persistent capability gaps.

5.1. Analytical Tools

  • The task map represents 162 subtasks with task vectors derived by LoRA-finetuning QwenVL-Chat, and uses Kendall’s tau to measure ranking correlation between LVLM performance across tasks.The task vectors are constructed from adapted multi-choice VQA samples; LoRA reduces their length from 9.6B to 3.5M.Source: §5.1 · p. 8

5.2. Findings on Task Map

  • LVLM rankings become more consistent as tasks get closer, suggesting performance on a new task can be predicted when it is near an MMT-Bench subtask.As the proximity threshold decreases, average Kendall’s tau increases.Source: §5.2 · pp. 8–9, Table 3 · p. 8, §5.1 · p. 8
  • Clusters 8, 9, and 11 have low accuracy and ranking correlation, revealing weaknesses in fine-grained localization and detection, GUI tasks, and special data structures such as tables.The analysis identifies these clusters as out-of-domain task regions.Source: §5.2 · p. 9, Table 4 · p. 9
  • Clusters 2, 3, and 10 have relatively high accuracy and ranking correlation, reflecting strengths in high-level visual comprehension, recognition of specialized images such as medical images, and image description.These clusters are identified as in-domain tasks that most current LVLMs can handle correctly.Source: §5.2 · p. 9

6. Conclusion and Discussion

MMT-Bench provides broad multitask evaluation of LVLMs and taskonomy analysis, while the authors acknowledge limits in its coverage and potential evaluation biases. They intend it to track progress toward multitask AGI, not to determine whether AGI has been achieved.

  • MMT-Bench contains 31,325 curated multiple-choice questions across 162 multimodal tasks, and its evaluation reveals significant challenges for current LVLMs.Source: §6 · p. 9
  • Taskonomy analysis on MMT-Bench’s task map can help predict performance on new tasks; the benchmark is intended to track progress toward multitask AGI, not certify its achievement.The authors argue strong MMT-Bench performance should be necessary for multitask AGI, while acknowledging the benchmark cannot include all multimodal tasks and will continue to expand it.Source: §6 · p. 9

Broader Impact

  • MMT-Bench may drive LVLM advances for applications including visual dialogue, video analysis, and document understanding, while raising ethical and societal considerations.Source: §6 · p. 9
  • Curated data and uneven subtasks may bias meta-task averages, while limited scenario and demographic coverage and reliance on aggregate metrics can obscure domain or group disparities.The authors plan to collect more multimodal tasks for unbiased evaluation.Source: §6 · pp. 9–10

A. Task Map · B. Hierarchical Structure of MMT-Bench · C. Task Abbreviations

The task map separates LVLM strengths in visual recognition, specialized-image understanding, and image description from weaknesses in localization, GUI navigation, and structured-image handling. The appendices also catalog the benchmark’s task hierarchy and define abbreviations used in the paper.

A. Task Map

  • The task map is hierarchically clustered into 12 groups, with task names listed for each cluster.The paper analyzes cluster results alongside model performance on corresponding tasks.Source: A. Task Map · p. 13
  • Clusters 8 and 11 expose weaknesses in fine-grained localization and spatial cognition, as well as recognizing special images and converting them into structured text.Cluster 11 also requires robust OCR and knowledge of structures such as code or tables.Source: A. Task Map · pp. 13–14
  • Cluster 9 centers on GUI navigation, where LVLMs need stronger visual understanding, object localization, and mobile-device expertise.GUI tasks are described as a novel task type requiring further model optimization (Yang et al., 2023c).Source: A. Task Map · p. 14
  • Clusters 2 and 3 show strong performance on visual recognition, including specialized images and tasks requiring domain knowledge such as medicine and emotion.Cluster 2 covers relatively simple recognition, while cluster 3 extends to sophisticated visual understanding; both have high accuracy and large ranking correlation τ.Source: A. Task Map · p. 14
  • Cluster 10 shows good LVLM performance on image-description tasks, consistent with models being tuned on massive image-text pairs.The task-map visualization presents model performance across the 12 categories formed by clustering.Source: Figure A1 · p. 13, A. Task Map · p. 14

B. Hierarchical Structure of MMT-Bench

C. Task Abbreviations

  • Table A1 provides abbreviations used throughout the paper to condense its discussion of the many benchmark tasks and models.Source: C. Task Abbreviations · p. 14

D. More Experimental Details … E. Pixel Coordinates vs. Normalized Coordinates

MMT-Bench’s supplementary details document model and prompting setups, while coordinate-format experiments show normalized coordinates are preferred by most models and reveal a gap for GeminiProVision against leading open-source LVLMs. The evaluation covers 19 detection-related tasks across localization, pixel-level perception, and visual captioning.

D.1. LVLMs Model Details

  • Evaluation follows OpenCompass’ protocol, with LLaVA-v1.5-7B taking 12 minutes on 8 GPUs and InternVL-Chat-V1.2-34B taking 79 minutes and around 80GB of memory.The paper’s open-source codebase supports multi-GPU distributed inference to accelerate evaluation.Source: §D.1 · p. 14
  • Table A6 summarizes evaluated LVLM parameter sizes, visual encoders, and LLMs.Inference time varies by model size; the codebase supports multi-GPU distributed inference.Source: §D.1 · p. 14

D.2. Multi-Images Prompt Experimental Details

  • For 28 multi-image tasks, Tables A7–A10 provide task names corresponding to abbreviations and designed prompt examples for single-image and multi-image LVLMs.The evaluation process follows OpenCompass’ protocol.Source: §D.1 · p. 14, §D.2 · p. 15

D.3. Visual Referring Prompting Experimental Details

  • Visual prompting efficacy is compared with alternative strategies across 14 tasks, including keypoint detection, segmentation, tracking, and counting.The evaluation process follows OpenCompass’ protocol.Source: §D.1 · p. 14, §D.3 · p. 15

E. Pixel Coordinates vs. Normalized Coordinates

  • Most models prefer normalized coordinates, while GeminiProVision trails detection-trained open-source LVLMs BLIP2 and XComposer2 across the coordinate-format analysis.The comparison spans 19 detection-related tasks across Localization, Pixel-level Perception, and Visual Captioning; the preference is attributed to normalized-coordinate training instruction templates.Source: E. Pixel Coordinates vs. Normalized… · p. 16
  • The coordinate-format analysis covers 19 detection-related tasks spanning Localization, Pixel-level Perception, and Visual Captioning.Tasks include point tracking, depth estimation, object detection, referring detection, reason segmentation, and image dense captioning.Source: E. Pixel Coordinates vs. Normalized… · p. 16

F. Analysis on Images Types and Capabilities · G. Case Study

The analysis finds reasoning on scientific diagrams and Raven’s Progressive Matrices difficult for LVLMs, while GeminiProVision leads most capabilities, especially retrieval and multi-image analysis. Case studies illustrate capability gaps, refusals, and instruction-following errors across benchmark tasks.

F. Analysis on Images Types and Capabilities

  • LVLMs struggle with scientific diagrams and Raven’s Progressive Matrices, which require complex reasoning that current models do not handle well.The comparison covers 20 LVLMs across 13 visual input types.Source: F. Analysis on Images Types and… · p. 16, Figure A3 · p. 28
  • GeminiProVision outperforms other open-source LVLMs across most capabilities, especially retrieval and multi-image analysis, among the 20 models compared across 14 capabilities.The paper attributes this strength to support for multi-image mode and powerful generalization, and identifies multi-image and video understanding as directions for open-source models.Source: F. Analysis on Images Types and… · p. 16, Figure A4 · p. 29

G. Case Study

H. Comparison of MMT-Bench with Other Benchmarks on OCR-Related Tasks … I.3. Statistics of Image and Video in MMT-Bench

MMT-Bench broadens OCR evaluation with longer, more varied text and adds diverse visual data, while its construction uses structured metadata, challenging options, and visually grounded questions. The details describe its metadata workflow and question design.

H. Comparison of MMT-Bench with Other Benchmarks on OCR-Related Tasks

  • MMT-Bench OCR samples average over 5 times MME’s and over 2 times MMBench’s word and token counts, with more long samples and varied text lengths.The benchmark adds font, handwriting, handwritten-formula, document-question-answering, and chart-question-answering tasks beyond direct text output.Source: H. Comparison of MMT-Bench with Other… · p. 63, Table A12 · p. 63

I.1. Metadata

  • Metadata dictionaries include shared keys for source, task, question, answer, visual prompt, and required capabilities, plus sample-specific keys such as category spaces or referring expressions.The category space is retained for animal-recognition choices, while referring expressions are preserved in reasoning segmentation.Source: §I.1 · p. 63, Table A13 · p. 63
  • Metadata collection extracts keys from source datasets with scripts, while co-authors define templates, visual prompts, and capabilities, then manually check samples.The check covers 50% of samples, up to 100 per dataset, with emphasis on sample-related keys.Source: §I.1 · p. 63

I.2. Prepare the Answer and Options

  • GPT-4V generates most distractors using task-specific 2-shot or 3-shot prompts, with human review; Python perturbations generate options for counting, OCR, and object detection.Distractors can be semantically similar or visually ambiguous, and option word lengths have a standard deviation of 1.1.Source: §I.2 · pp. 63–64
  • LVLMs gain 18.1%–20.2% with visual input, while models without it perform near the 31.7% frequency-based and 28.5% random-guessing baselines.Question templates avoid embedding specific answers, and complex options discourage solving from prior knowledge or option correlations.Source: §I.2 · p. 64
  • Domain-proficient co-authors use simple, unambiguous question templates and clear options; longer localization questions specify the task, answer format, and detection conditions.The localization format may include [x, y, x, y] and category conditions such as the 80 COCO classes.Source: §I.2 · p. 64

I.3. Statistics of Image and Video in MMT-Bench

  • MMT-Bench contains 25,732 single images, 3,800 image pairs (14,800 images), and 1,793 videos (10,572 frames), exceeding cited benchmarks in several visual-data counts.It has the most single images, more image pairs than MMMU, and more videos and frames than MVBench and Seed-bench.Source: §I.3 · p. 64

J. OpenCompass’ Protocol · K. Computaional Resources

OpenCompass’ multi-step protocol reliably extracts LVLM answer selections while handling unclear responses and refusals. Evaluation resources vary substantially across models, with open-source distributed inference and API evaluation involving different time and cost profiles.

J. OpenCompass’ Protocol

  • Step 1 extraction succeeds for over 87% of most models, while BLIP-2, InternVL-Chat-V1.2-34B, and LLaVA-1.5-7B exceed 99%.Success rates improve in subsequent steps.Source: J. OpenCompass’ Protocol · p. 65, Table A17 · p. 66
  • Some models refuse 0.875% to 10.136% of questions; OpenCompass assigns these cases option Z to distinguish them from valid answers.The passages suggest refusals are likely related to sensitive questions.Source: J. OpenCompass’ Protocol · p. 65
  • OpenCompass was selected for convenience, ease of use, and scalability, with its multi-step extraction handling unclear responses and refusals.The passage describes the protocol as highly effective at extracting model selections.Source: J. OpenCompass’ Protocol · p. 65
  • MME’s yes-or-no format can bias models and is resource-intensive, while MMBench has high testing overhead that limits suitability at MMT-Bench’s scale.Source: J. OpenCompass’ Protocol · p. 65
  • SeedBench’s candidate-option log-likelihood avoids requiring direct option answers, but cannot be used with API-based models.Source: J. OpenCompass’ Protocol · pp. 65–66

K. Computaional Resources

L. Detailed Main Results

Tables A19–A37 provide detailed accuracy results across the benchmark’s subtasks. The overview passage says the results cover 20 models and 162 subtasks, while the table captions describe results for 30 LVLMs.

Related papers

Loading 2404.16006v1…