Source-linked AI summary
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, Wenqi Shao
TL;DR
Existing LVLM benchmarks provide limited coverage of multimodal tasks and often test rudimentary capabilities, motivating broader evaluation. MMT-Bench addresses this gap with a large, diverse multitask benchmark and task-map analysis; evaluations reveal substantial challenges and distinct error and domain patterns in current LVLMs.
Problem
Existing multimodal benchmarks cover limited tasks and rudimentary capabilities, providing insufficient coverage for evaluating LVLM development toward multitask AGI.
Method
MMT-Bench curates 31,325 multi-choice visual questions across 32 meta-tasks and 162 subtasks, evaluates LVLMs with standardized accuracy, and uses a task map for taxonomy analysis.
Results
Current LVLMs face significant challenges on MMT-Bench; GPT-4V scores 62.0/100 overall, while top models predominantly show perception, reasoning, and knowledge errors.
Takeaways & Limitations
MMT-Bench supports measuring progress toward multitask AGI and identifying in-domain and out-of-domain tasks for current LVLMs.
Takeaways & Limitations
MMT-Bench relies on curated data, averages subtasks within meta-tasks with unequal subtask counts, and may only partially represent real-world diversity.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises $31,325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $30$ LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.
1. Introduction
Existing LVLM benchmarks cover limited multimodal tasks and rudimentary capabilities, while MMT-Bench evaluates broader multitask understanding across diverse tasks, inputs, and scenarios. Results show substantial challenges for current models, alongside task-specific prompting and scaling effects.
- Existing benchmarks cover limited multimodal tasks and mainly test rudimentary capabilities such as visual recognition and text-scarce OCR.
- MMT-Bench contains 32K multi-choice visual questions spanning 32 meta-tasks and 162 subtasks across 13 image types and diverse multimodal scenarios.It tests capabilities including recognition, localization, reasoning, OCR, counting, 3D perception, and temporal understanding.
- MMT-Bench evaluates LVLMs across vehicle driving, GUI navigation, embodied AI, and other scenarios requiring varied multimodal capabilities.
- GPT-4V achieves 62.0/100 overall and 55.6/100 excluding visual recognition subtasks, indicating substantial room for improvement toward multitask AGI.
- InternVL-chat leads MMT-Bench among the compared models, surpassing QWen-VL-Plus, GPT-4V, and GeminiProVision.
- Taskonomy analysis finds strong performance on visual recognition and description but weaker performance on localization and pixel perception.
- Performance improves with larger model size for LLaVA variants, while visual prompting helps certain multi-image and coordinate-related tasks but not most models overall.
2. Related Work
LVLM research has expanded beyond single-task evaluation, motivating broader multimodal assessment and task-relationship analysis. Prior evaluation benchmarks and multitask-analysis methods provide foundations but leave comprehensive multimodal coverage unresolved.
- LVLMs increasingly address multimodal understanding and generation by using large language models as backbones.
- Existing single-task benchmarks are insufficient for comprehensively evaluating LVLMs’ broad reasoning capabilities.
- Multitask Analysis: Taskonomy models visual-task relationships through transfer learning, while Task2Vec represents tasks using Fisher information vectors for meta-learning.
- Multitask Analysis: MMT-Bench uses extensive task data to evaluate LVLMs on a task map and identify challenging tasks.
3. MMT-Bench
MMT-Bench is constructed through a hierarchical task taxonomy and a standardized data-collection pipeline. The resulting benchmark combines broad multimodal coverage with curated, uniformly formatted multiple-choice visual questions.
- Hierarchical Task Structure: The benchmark organizes multimodal understanding into 32 deduplicated meta-tasks and subtasks selected for basic capability coverage and task importance.
- Datasets Search: Data collection searches related datasets, evaluates their suitability, and retains publicly accessible sources for each subtask.
- Metadata Construction: Metadata unifies images, task information, capability annotations, and visual-prompt types for downstream question generation.
- Metadata Construction: Each task is capped at 200 randomly sampled examples, with equal sample counts across datasets for evaluation efficiency.
- Question and Answer Generation: Questions contain up to eight choices and are generated from metadata using manually designed rules or ChatGPT prompts, depending on the task.
- Dataset Statistics: 31,325 questions span 13 image types, 32 meta-tasks, and 162 subtasks across diverse multimodal scenarios.
4. Experiments
MMT-Bench evaluates 30 LVLMs across 32 meta-tasks using accuracy and varied prompting settings, revealing substantial performance challenges and task-specific prompting effects. Error analysis further identifies perception and reasoning as prominent failure modes.
- Overall Evaluation: 30 LVLMs are evaluated across 32 meta-tasks using multi-choice accuracy and response-extraction procedures that fall back to ChatGPT when needed.The benchmark reports overall and meta-task scores, with Random Choice and Frequent Choice as baselines.
- Overall Evaluation: 63.4%, 62.0%, and 61.6% accuracy are achieved by InternVL-Chat, GPT-4V, and GeminiProVision, respectively, indicating substantial room for improvement.GeminiProVision’s score drops to 55.1% when visual recognition is excluded, despite 84.7% accuracy on visual recognition.
- Specific Task and Prompt Methods Analysis: Multi-image tasks remain difficult, with GeminiProVision achieving only 53.8% best accuracy, while multi-image prompting improves its face-retrieval accuracy from 30.5 to 92.5.The comparison covers tasks commonly requiring multiple images, including image retrieval and video captioning.
- Specific Task and Prompt Methods Analysis: Visual prompting substantially trails normalized-coordinate, pixel-coordinate, and combined visual-text prompting across 14 visual-referring subtasks.The authors mainly attribute this disparity to limited visual-prompting data during supervised fine-tuning.
- Error Analysis: Perception errors are most common across GPT-4V, GeminiProVision, and InternVL, while reasoning errors are the second most prevalent category.GPT-4V has a 51% perception-error rate versus 76.9% for GeminiProVision and 67.2% for InternVL; reasoning errors are 14.8%, 10.4%, and 9.94%, respectively.
- Error Analysis: GPT-4V has higher lack-of-capability and refusal-to-answer error rates, whereas InternVL most often fails to follow instructions.The reported rates are 19% and 6.11% for GPT-4V’s two categories, and 6.64% instruction-following errors for InternVL.
5. Taskonomy Analysis
MMT-Bench supports taskonomy analysis by representing subtasks on a task map and relating task distance to model-ranking consistency. Clustering then distinguishes task regions where current LVLMs struggle from regions they handle relatively well.
- Analytical Tools: MMT-Bench’s task map systematically interprets multimodal capabilities through relationships among subtasks.The analysis uses task vectors based on fine-tuned weight variation from an initial probing model.
- Analytical Tools: Kendall’s tau measures whether LVLM performance rankings remain consistent between pairs of subtasks, with τ = 1 indicating completely consistent rankings.Task distance is defined using cosine distance between task vectors.
- Findings on Task Map: As the distance threshold δ decreases, average Kendall’s tau τδ increases, showing more consistent model rankings for closer tasks.The authors therefore use nearby MMT-Bench subtasks to predict performance on a new task.
- Findings on Task Map: Hierarchical clustering groups the 162 subtasks into 12 clusters to identify out-of-domain regions where current models struggle.Clusters are assessed using multimodal accuracy and ranking correlation.
- Findings on Task Map: Clusters 8, 9, and 11 show low accuracy and ranking correlation, corresponding to weaknesses in fine-grained visual cognition, localization, and detection.These clusters also contain tasks involving positional and spatial relationships.
- Findings on Task Map: Clusters 2, 3, and 10 show relatively high accuracy and ranking correlation, indicating in-domain strengths in high-level visual comprehension and recognition.These strengths extend to specialized images such as medical images and image description.
6. Conclusion and Discussion
MMT-Bench is introduced as a comprehensive benchmark for multimodal multitask understanding, but the authors caution that it cannot by itself determine whether multitask AGI has been achieved. Its curated-task design also carries risks of biased aggregation, incomplete real-world coverage, demographic imbalance, and oversimplified performance measurement.
- Contribution and scope: 31,325 multi-choice questions covering 162 multimodal tasks form MMT-Bench, which evaluates LVLM progress toward multitask AGI.The authors present the benchmark as comprehensive while acknowledging that no benchmark can include all multimodal tasks.
- Contribution and scope: MMT-Bench supports task-map analysis that can predict performance on a new task and reveal in- and out-of-domain capabilities.
- Limitations: Averaging subtasks within each meta-task may bias assessment because meta-tasks contain different numbers of subtasks.
- Limitations: Task selection may only partially represent real-world diversity, while data collection may overrepresent particular demographics or contexts.
- Limitations: Overall scores and task-specific accuracies may oversimplify evaluation and obscure performance differences across groups or domains.
A. Task Map
The task map is hierarchically clustered into 12 groups to identify task regions where LVLMs perform relatively well or poorly. Models show strengths in visual recognition and description, but weaknesses in localization, GUI navigation, special-image understanding, and structured-text conversion.
- Task-map clustering: 12 hierarchical clusters organize the task map and associate each cluster with its constituent tasks and model performance.The analysis uses hierarchical clustering with the number of clusters set to 12.
- Out-of-domain tasks: Clusters 8, 9, and 11 have low multimodal accuracy and ranking correlation τ, identifying out-of-domain task groups.
- Out-of-domain tasks: Cluster 8 exposes weak fine-grained visual cognition and spatial understanding through detection, tracking, and localization tasks.
- Out-of-domain tasks: Cluster 9 centers on GUI navigation, combining visual understanding, object localization, and expert knowledge of mobile-device operation.
- Out-of-domain tasks: Cluster 11 reveals shortcomings in recognizing special images and converting them into structured text requiring OCR and domain knowledge.
- In-domain tasks: Clusters 2, 3, and 10 show relatively high accuracy and ranking correlation τ, covering recognition, specialized-domain understanding, and visual description.
D. More Experimental Details
The supplementary experiments document the evaluated LVLM architectures, multi-image task protocols, visual-prompting comparisons, and 19 detection-related tasks used for additional analysis.
- Evaluation setup: Table A1 provides abbreviations for terms used throughout the paper.
- Input protocols: 28 multi-image tasks are documented with task names and prompt examples for single-image and multi-image LVLMs.
- Prompting analysis: Visual prompting is compared with alternative prompting strategies across 14 distinct multimodal tasks.
- Detection analysis: The detection analysis spans 19 tasks, including tracking, localization, depth estimation, object detection, and reasoning segmentation.
F. Analysis on Images Types and Capabilities
The appendix analyzes LVLM performance across visual input types and multimodal capabilities. Scientific diagrams and reasoning-intensive inputs remain difficult, while GeminiProVision is especially strong in retrieval and multi-image analysis.
- Visual input types: 20 LVLMs are compared across 13 visual input types, with scientific diagrams posing particular difficulty because they require complex reasoning.
- Multimodal capabilities: 20 LVLMs are compared across 14 capabilities, with GeminiProVision especially outperforming open-source LVLMs in retrieval and multi-image analysis.
- Appendix documentation: Tables A3–A5 specify sample numbers, visual input types, and evaluated capabilities for all 162 subtasks, while Table A6 describes the 30 evaluated model architectures.
- Task coverage: The task inventory includes visual recognition, captioning, emotion, temporal understanding, document understanding, action recognition, and 3D tasks.
- Task coverage: Document and visual tasks cover chart and table understanding, clock reading, document VQA, and structured-text recognition.
- Task coverage: The appendix also includes anomaly detection, image evaluation, depth estimation, matching, deepfake detection, translation, keypoint detection, OCR, segmentation, and tracking.
- Task coverage: Localization, visual grounding, cross-image matching, GUI navigation, captioning, hallucination, relation reasoning, counting, medical understanding, and autonomous driving are represented among the subtasks.
G. Case Study
The case study illustrates MMT-Bench’s breadth across visual recognition, localization, pixel perception, OCR, prompting, retrieval, reasoning, detection, translation, and summarization, while documenting several LVLM error types.
- Case-study organization: The case-study index organizes examples by meta-task and associated LVLM error categories.The examples cover 32 meta-tasks overall, with the index linking cases to error categories.
- Error categories: LVLMs may lack the capability to solve corresponding tasks, with examples including camouflaged object detection and sign-language recognition.The paper notes this error particularly for GPT-4V, which may answer honestly when unable to perform a task.
- Error categories: LVLMs may refuse to answer anthropocentric or sensitive questions, as illustrated by the case-study examples.This category is reported for models such as GPT-4V and Gemini.
- Error categories: LVLMs may fail to follow instructions by misunderstanding specified conditions or generating an image caption instead of addressing the instruction.The cited examples concern visual prompt understanding and visual summary tasks.
- Representative tasks: The case studies span landmark recognition, camouflaged-object localization, image matting, handwritten OCR, retrieval, counting, keypoint detection, action recognition, hallucination, anomaly detection, jigsaw solving, and visual summary.Each example presents answer choices with green marking the correct answer and red marking an incorrect answer.
H. Comparison of MMT-Bench with Other Benchmarks on OCR-Related Tasks
MMT-Bench addresses limited and text-scarce OCR evaluation by using broader, visually grounded question construction and diverse multimodal data coverage.
- OCR coverage: Previous OCR benchmarks contain relatively short texts with limited variation and often test direct text output from scenes or cropped images.The comparison is reported using OCR-related samples from multiple evaluation benchmarks.
- Dataset construction: MMT-Bench uses uniform metadata containing image paths, sources, subtask and meta-task names, question templates, answers, prompt types, and required capabilities.Additional sample-specific keys can preserve category spaces or referring expressions for question generation.
- Dataset construction: Metadata construction combines scripted extraction of dataset keys, predefined templates and prompts, and manual annotation by co-authors.The process uses dataset-level shared characteristics to define task-specific information.
- Option construction: Wrong options are generated or perturbed to create difficult distractors, including visually confusing alternatives for classification, captioning, counting, OCR, and detection.GPT-4V, in-context prompting, manual review, and direct perturbations are used for different task types.
- Visual grounding: MMT-Bench questions are designed to require visual information, while text-only systems perform near frequency-based or random guessing without visual input.The benchmark also uses simple, unambiguous templates to reduce excessive dependence on language understanding.
- Visual coverage: MMT-Bench contains 25,732 single images, 3,800 image pairs, and 1,793 videos, covering diverse visual data types.The benchmark has more single images than compared benchmarks and more videos and video frames than MVBench and Seed-bench.
J. OpenCompass’ Protocol
The OpenCompass multi-choice protocol is selected for convenient, scalable answer extraction, but its multi-step option-answering approach cannot be applied to API-based models.
- Protocol behavior: OpenCompass’ protocol extracts model selections through multiple steps, including checking for option letters and subsequent response processing.The protocol’s step success rates vary across LVLMs but are generally high.
- Protocol behavior: Most models achieve success rates above 87% at Step 1, with further improvements in later steps.BLIP-2, InternVL-Chat-V1.2-34B, and LLaVA-1.5-7B are highlighted for especially clear and structured responses.
- Protocol choice: OpenCompass is chosen for its convenience, ease of use, scalability, and effective extraction of model selections.Several models achieve success rates over 99% under the protocol.
- Protocol limitation: The multi-step option-answering approach cannot be applied to API-based models.This is identified as a limitation of the evaluation protocol.
- Resource consumption: Evaluation resource use varies substantially: LLaVA-v1.5-7B takes 12 minutes on 8 GPUs, whereas InternVL-Chat-V1.2-34B takes 79 minutes and about 80GB of memory.The open-source codebase supports multi-GPU distributed inference.
L. Detailed Main Results
Detailed results report accuracy across 30 LVLMs and all 162 subtasks, with separate tables covering the benchmark’s broad task categories.
- Results scope: Accuracy is the evaluation metric for 20 models reported across all 162 subtasks in Tables A19–A37.The tables provide detailed performance comparisons across the benchmark’s task categories.
- Task-level results: Tables A19–A22 cover visual grounding, document understanding, action recognition, localization, and visual recognition.Visual recognition results are distributed across multiple tables.
- Task-level results: Tables A23–A28 cover additional visual recognition, GUI, OCR, image-to-image translation, temporal understanding, relation reasoning, discipline knowledge, IQ testing, and embodied AI.These tables extend the detailed results beyond basic recognition tasks.
- Task-level results: Tables A29–A33 cover emotion quotient testing, visual illusion, meme understanding, counting, hallucination, retrieval, visual prompting, anomaly detection, keypoint detection, commonsense reasoning, visual code, and image evaluation.The results span reasoning, perception, and prompt-related capabilities.
- Task-level results: Tables A34–A37 cover pixel-level perception, multiple-image analysis, 3D and medical understanding, cross-image matching, visual summary, and autonomous driving.These categories include fine-grained perception, multimodal comparison, and applied scenarios.