PaperlayerView exact revision on arXiv

Source-linked AI summary

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Junjun He, Yu Qiao

arXiv:2408.03361v7eess.IVcs.CV

TL;DR

Existing medical benchmarks do not fully address diverse clinical evaluation needs, including varied modalities and perceptual granularities. GMAI-MMBench organizes 284 datasets into customizable, multi-granularity VQA evaluations, and its assessment finds uneven LVLM performance, with GPT-4o reaching 53.96% accuracy.

  • Problem

    Existing benchmarks leave unclear how well LVLMs meet diverse clinical demands and are limited in scope, clinical relevance, customization, or perceptual granularity.

    Source: Introduction · p. 2

  • Method

    GMAI-MMBench organizes 284 datasets across 38 modalities, 18 clinical VQA tasks, 18 departments, and four granularities into customizable evaluations using a lexical tree.

    Source: Introduction · pp. 3, 5

  • Results

    Performance is uneven across clinical tasks, departments, and perceptual granularities; even GPT-4o achieves only 53.96% accuracy.

    Source: Introduction · p. 4

  • Takeaways & Limitations

    The evaluation provides insights into LVLM strengths and weaknesses for accommodating real-world clinical applications.

    Source: Introduction · p. 5

    Manual curation may introduce bias, and a focus solely on medical subjects may not fully meet the stated criteria for general medical AI.

    Source: Conclusion · p. 11

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI.

Introduction

The introduction argues that existing benchmarks do not fully match diverse real-clinical evaluation needs. GMAI-MMBench addresses this with categorized, multi-granularity coverage, while evaluation shows substantial LVLM shortcomings.

Introduction

  • Existing benchmarks either focus on limited modalities and tasks or lack clinical relevance and categorization for customized evaluation.The introduction notes that some benchmarks draw on academic papers, textbooks, or specific databases rather than sources close to clinical practice.Source: Introduction · p. 3
  • Its lexical tree categorizes cases by clinical task, department, and modality so users can select cases for specific evaluations, such as oncology tasks.Source: Introduction · pp. 3–4
  • The benchmark evaluates perception from local masks or bounding boxes to entire images, a capability relevant to detection, segmentation, classification, and interactive LVLMs.Source: Introduction · p. 4
  • 53.96% accuracy is GPT-4o’s score, while MedDr and DeepSeek-VL-7B reach approximately 44%, surpassing Claude3-Opus and Qwen-VL-Max but trailing GPT-4o.The evaluation covers publicly available and proprietary LVLMs, including general-purpose and medical-specific models.Source: Introduction · p. 4
  • Most medical-specific models fall below general LVLMs’ approximately 30% accuracy except MedDr at 43.69%, and box-level annotation consistently has the worst accuracy.Performance is also unbalanced across clinical VQA tasks, departments, and perceptual granularities.Source: Introduction · p. 4
  • Reported performance bottlenecks include perceptual errors, lack of medical knowledge, irrelevant responses, and refusals prompted by safety protocols.Source: Introduction · p. 5
  • GMAI-MMBench draws on 284 datasets and covers 38 modalities, 18 clinical VQA tasks, 18 departments, and four perceptual granularities.The datasets include image-label pairs sourced from hospitals and annotated by professional doctors.Source: Introduction · pp. 3, 5

GMAI-MMBench

GMAI-MMBench is built from curated medical datasets and organized to support customized evaluation across clinical tasks, departments, modalities, and perceptual granularities. Its construction pipeline generates, validates, and selects image-based VQA pairs.

Overview

  • The benchmark combines 284 public and hospital datasets with a pipeline that produces approximately 26K questions across perceptual granularities.The overview describes the data sources and pipeline; the construction process finalized 25,831 QA pairs, with 4,550 for validation and 21,281 for testing.Source: Overview · pp. 5–6, Benchmark Construction · p. 7

Benchmark Construction

  • Dataset standardization converts 2D/3D images to 2D RGB and normalizes labels by expanding abbreviations, unifying synonyms, and merging left/right distinctions.The label changes target understanding and reasoning rather than directional judgment.Source: Benchmark Construction · p. 6
  • A lexical tree categorizes cases by 18 clinical VQA tasks, 18 departments, and four perceptual granularities, with labels assigned by three annotators and merged by voting.The categorized tree supports customized evaluation of specific LVLM abilities.Source: Benchmark Construction · p. 6
  • Questions generated with GPT-4o are manually reviewed to include modality, task, and granularity information without answerable image-free hints.The modality is replaced with a placeholder in the standardized question template.Source: Benchmark Construction · p. 6
  • Global-view and local-view questions use different option pools, and QA pairs are manually checked for completeness, answer correctness, and image quality before balanced selection.The final selection targets 30 cases per answer, choosing diverse images when more than 30 are available.Source: Benchmark Construction · pp. 6–7

Experiments

The experiments evaluate LVLMs in a zero-shot setting using single-choice and multiple-choice metrics. The benchmark includes models from medical-specific, open-source, and proprietary families.

Experiment setup

  • Models were evaluated zero-shot, without example cues or prior task-specific examples, to test generalization and comprehension on novel problems.The evaluation used NVIDIA A100 GPUs with 80GB of memory.Source: Experiment setup · p. 7

Models

  • The evaluation tested medical-specific, open-source, and proprietary LVLMs, demonstrating 29 of 50 evaluated models in the main text.Versions with approximately 7 billion parameters were selected for testing, and the latest, largest, best-performing available checkpoint was used by default.Source: Models · p. 7

Metrics

  • Single-choice performance uses macro-averaged accuracy, while multiple-choice evaluation reports accuracy and recall based on correct matches relative to prediction length and ground-truth option length.Outputs without clearly followed answer-selection instructions could be parsed using ChatGPT-3.5-turbo-0613; unextractable answers counted as errors.Source: Metrics · pp. 7–8
  • Figure 4 reports single-choice results across contour-, mask-, image-, and box-level perceptual granularities.The figure compares different models across those four levels.Source: Figure 4 · p. 8
  • Table 2 presents single-choice results across clinical VQA tasks, marking each category’s best model in bold and second-best model with underlining.Full task names are listed in supplementary Table 5.Source: Table 2 · p. 8

Results

The evaluation finds substantial performance gaps across models and medical tasks, alongside imbalances in task, department, and perceptual-type performance. The case studies identify recurring response and perception errors, and the authors note limitations in benchmark scope and curation.

Analysis

  • GPT-4o reaches only around 54% accuracy on GMAI-MMBench, which the authors say falls short of clinical requirements.The paper concludes that current medical LVLMs need significant improvement.Source: Analysis · p. 8
  • Leading open-source models MedDr and DeepSeek-VL-7B surpass Claude3 Opus and Qwen-VL-Max and approach GPT-4V, though they remain less accurate than GPT-4o.The authors characterize open-source medical models as gradually catching up to top-performing commercial models.Source: Analysis · pp. 8–9
  • Disease Diagnosis and Nervous Tissue perform best, while Severity Grading and Attribute Recognition often fall below random baseline; Infectious Diseases and Neurosurgery lead departments, unlike General Surgery and Obstetrics and Gynecology.The authors describe persistent task gaps and department-level specialization bias.Source: Analysis · p. 9, Table 3 · p. 9
  • Contour-level performance is slightly better than mask-level, and both exceed image-level, while box-level perception performs worst, even below image-level.The paper identifies sensitivity to perceptual type as a robustness concern for interactive applications.Source: Analysis · p. 9
  • Medical-specific models significantly underperform general models overall, while MedDr surpasses its InternVL base model and is presented as an example of stronger dataset construction.The authors attribute weak results to instruction-following and medical-question difficulties, and note the need to improve online medical instruction-tuning data quality.Source: Analysis · p. 10
  • All models score below 55% accuracy on multiple-choice questions, and their accuracy and recall patterns differ from single-choice performance.CogVLM-Chat exceeds 90% recall, while InternVL-Chat-V1.5 and Qwen-VL-Max perform poorly on multiple-choice questions despite strong single-choice results.Source: Figure 5 · p. 9, Analysis · p. 10

Case Study

  • Case studies identify question misunderstanding, perceptual errors, lack of domain knowledge, irrelevant responses, and refusals as recurring error types.Examples include describing or repeating options instead of choosing, mistaking an esophagus for a spine, and misidentifying a stain and stained structure.Source: Case Study · pp. 10–11
  • Manual curation may introduce bias, and focusing solely on medical subjects may not fully meet the paper’s definition of general medical AI.The authors plan to integrate human evaluations to compare model capabilities with expert performance.Source: Conclusion · p. 11

Supplementary Materials

The supplementary background describes general-purpose LVLMs and medical-specialized models as two model groups relevant to healthcare applications.

A.1 Large Vision-Language Model(LVLMs)

  • Prominent proprietary general-purpose models include GPT-4o, Claude3-opus, and Qwen-max, while open-source examples include InternVL, LLAVA, DeepSeek, CogVLM, and others.The paper notes that open-source models are evolving through accessibility and collaborative development.Source: §A.1 · p. 29
  • Researchers fine-tune large models on domain-specific medical data to address specialized medical tasks.LLaVA-Med is described as derived from LLAVA, and MedDr as based on the InternLM framework.Source: §A.1 · p. 29
  • The paper presents medical-specialized models as contributing to healthcare LVLM applications and development in the medical domain.It identifies LLaVA-Med and MedDr as examples.Source: §A.1 · p. 29

A.2 Benchmarks

Medical benchmarks range from specialized assessments to broad, general-purpose evaluations, but the paper identifies gaps in generalizability, task and modality coverage, and data granularity. It argues that more comprehensive benchmarks are needed to support LVLM development in healthcare.

A.2 Benchmarks

  • Specialized benchmarks assess targeted modalities or disciplines, but their limited generalizability constrains broader applicability.Examples include radiology benchmarks VQA-RAD, SLAKE, and RadBench, and pathology benchmarks PathVQA [96] and PathMMU [238].Source: §A.2 · p. 30
  • General-purpose benchmarks span multiple medical domains, yet often have shortcomings in task volume, modality coverage, data distribution, and granularity.Examples include MMMU, OminimedVQA, and MMT-Bench [260].Source: §A.2 · p. 30
  • The paper frames benchmark development as necessary for LVLM progress in healthcare, contrasting specialized benchmarks’ narrow reach with general-purpose benchmarks’ coverage gaps.It calls for more comprehensive and robust benchmarks to bridge these gaps.Source: §A.2 · p. 30

B Dataset Details

GMAI-MMBench draws on datasets from public sources and hospitals, with the inventory documenting dataset, modality, dimensionality, task type, and case-count information.

B Dataset Details

  • 284 datasets spanning 38 medical image modalities form GMAI-MMBench, including 268 public datasets and 16 datasets shared by hospitals with ethical approval.The inventory describes each dataset’s name, sub-dataset count, modality, data dimension, task type, and case count.Source: B Dataset Details · p. 30
  • The dataset inventory distinguishes 2D and 3D data and uses Cls, MCls, Seg, and Det for classification, multi-label classification, segmentation, and detection.The count denotes cases used in GMAI-MMBench.Source: Table 4 · p. 30

T1 MRI, FLAIR MRI

  • Examples in the inventory include FLAIR MRI and T1 MRI datasets, as well as datasets spanning multiple imaging modalities and task types.BraTS entries include FLAIR MRI segmentation datasets; CHAOS includes T2-weighted MRI and T1-weighted data.Source: B Dataset Details · p. 31

C.1 Data Statistics

The supplementary material reports label distributions across clinical tasks, departments, and perceptual granularities, and describes a lexical tree for tailoring evaluation subsets. Users can select categories and filter questions to assess models on focused combinations.

C.1 Data Statistics

  • Disease Diagnosis accounts for 51.6% of clinical VQA tasks, Ophthalmolog for 11.3% of departments, and Image Level for 49.2% of perceptual granularities.Severity Grading is 9.1%, and Mask Level and Contour Level are each 22.0%.Source: §C.1 · p. 36, Figure 6 · p. 36

C.2 Lexical Tree

  • The method supports independent testing across departments, modalities, clinical tasks, and their combinations, with keyword refinement enabling evaluation of disease-diagnosis tasks within ophthalmology fundus photographs.For this example, keywords from the department–fundus photography and clinical tasks–disease diagnosis sections are intersected.Source: §C.2 · p. 39
  • The lexical tree is presented as a framework for focused model assessment across medical domains and user-selected evaluation needs.The authors state that users can test other departments, modalities, tasks, and combinations.Source: §C.2 · pp. 37, 39
  • For a customized evaluation, users select a department and modality, filter the question list by keywords, then assess models’ accuracy on the filtered questions.In the example, the selected categories are ophthalmology and fundus photography; the correct answers are hidden during evaluation.Source: §C.2 · p. 38, Figure 8 · p. 39

D Evaluation

The paper evaluates medical-specific, open-source general, and closed-source API LVLMs using a standardized prompting and answer-scoring process. For single-choice questions, it defines accuracy using the count of correct answers and the total question count.

D Evaluation

  • The evaluation includes medical-specific, open-source general, and closed-source API LVLMs, using VLMEvalKit and MultiModality-Arena for testing.Models receive the prompt in Table 8 and are expected to provide option-only answers; ChatGPT-3.5-turbo-0613 extracts answers when needed, and unextractable outputs count as errors.Source: D Evaluation · p. 40, Table 8 · p. 40
  • The evaluation prompt requests option-only answers, with fallback extraction for outputs that do not clearly select an answer or option letter.Outputs that cannot be parsed into an answer are treated as errors.Source: D Evaluation · p. 40

D.1 Evaluation Metric for Single-choice Questions

  • For single-choice questions, ACC is calculated using ncorrect, the number of correctly answered questions, and nquestions, the total number of questions.The passage introduces the calculation after defining these counts.Source: §D.1 · p. 40

D.2 Evaluation Metric for Multiple-choice Questions

The evaluation section describes how multiple-choice predictions are assessed and reports that 50 LVLMs were evaluated. A table lists the evaluated models’ architectures.

D.2 Evaluation Metric for Multiple-choice Questions

  • The multiple-choice evaluation counts correct predictions within each ground-truth case and uses prediction and ground-truth option lengths to calculate its metrics.The passage names the correct-match count as nmatch but does not provide the full metric formulas.Source: §D.2 · p. 40

D.3 Evaluated Models

  • Table 9 lists the model architectures of the 50 evaluated LVLMs.Source: Table 9 · p. 41
  • 50 LVLMs were evaluated on GMAI-MMBench.Source: §D.3 · p. 41

E Results

The results section presents complete quantitative results and then analyzes representative model outputs in a case study.

E Results

  • The section first presents complete quantitative experimental results.Source: E Results · p. 41
  • It then analyzes 53 representative examples of model outputs in a case study.Source: E Results · p. 41
  • The section pairs quantitative evaluation with case-based analysis of model outputs.Source: E Results · p. 41

E.1 Quantitative Results

The quantitative results are organized across clinical VQA tasks, departments, and perceptual granularities, while a figure illustrates the case-study process from input to output.

E.1 Quantitative Results

  • Table 10 reports single-choice results across clinical VQA tasks for 50 LVLMs.The caption indicates that the best and second-best models in each category are marked.Source: §E.1 · p. 41, Table 10 · p. 42
  • Table 11 reports single-choice results across departments for 50 LVLMs.The caption indicates that the best and second-best models in each category are marked.Source: §E.1 · p. 41, Table 11 · p. 43
  • Table 12 reports single-choice results across perceptual granularities for 50 LVLMs.The caption indicates that the best and second-best models in each category are marked.Source: §E.1 · p. 41, Table 12 · p. 44
  • Figure 9 illustrates the case study’s logical process from input to output.Source: Figure 9 · p. 45

E.2 Case Study

The case study examines 53 VQA pairs and categorizes model outputs, finding different error patterns in proprietary and specialized medical models. Its observations point to needs for stronger image performance in general models and language training in specialized models.

E.2 Case Study

  • Question misunderstanding can lead a model to describe image details instead of answering the requested task.The passage gives an example involving identification of a surgical process.Source: §E.2 · p. 45
  • Perceptual errors include missing critical image details or misinterpreting image content needed to answer the question.Examples include GPT-4o overlooking a surgical tool and Claude3-Opus misidentifying content in a mask.Source: §E.2 · p. 45
  • Models can recognize the image and question yet err on specialized questions, as when GPT-4o failed to distinguish carcinoma in situ from invasive carcinoma.Source: §E.2 · p. 45
  • Irrelevant responses fail to address the question, a problem especially noticeable in open-source models.One example is RadFM returning only a reference paper.Source: §E.2 · p. 45
  • The study also classifies refusals to answer, which proprietary models may produce under safety protocols for sensitive questions or requests for medical advice.Source: §E.2 · p. 45
  • Across 53 sampled VQA pairs, proprietary models rarely struggled with question understanding, while their errors mainly involved perception and domain knowledge.The sample spans different clinical VQA tasks, departments, and perceptual granularities.Source: §E.2 · p. 45
  • The authors conclude that general models need better performance on specialized medical images, while specialized medical models need further language training or fine-tuning.The authors note that improving general models may require more medical training data.Source: §E.2 · pp. 45–46
Loading 2408.03361v7…