Source-linked AI summary
OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, Ping Luo
TL;DR
Medical LVLMs lack comprehensive evaluation on diverse real medical images spanning modalities and anatomical regions. The paper introduces OmniMedVQA, a large-scale benchmark assembled from medical datasets and evaluates general-domain and specialized models. Results show that existing LVLMs struggle, with medical-specialized models exhibiting inferior performance to general-domain models overall.
Problem
Comprehensive evaluation of medical LVLMs is limited because existing benchmarks often contain few images, limited modalities, and restricted anatomical coverage.
Method
The paper constructs OmniMedVQA by converting medical classification datasets into multiple-choice VQA items covering diverse real medical images, modalities, and anatomical regions.
Results
Medical-specialized LVLMs exhibit inferior performance to general-domain models, while existing LVLMs struggle with the benchmark's medical VQA problems.
Takeaways & Limitations
OmniMedVQA reveals shortcomings in current LVLMs for understanding real medical images and provides a comprehensive benchmark for biomedical evaluation.
Takeaways & Limitations
The prefix-based score measures the likelihood of each option being regarded as correct rather than the LVLM's directly generated response.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in various multimodal tasks. However, their potential in the medical domain remains largely unexplored. A significant challenge arises from the scarcity of diverse medical images spanning various modalities and anatomical regions, which is essential in real-world medical applications. To solve this problem, in this paper, we introduce OmniMedVQA, a novel comprehensive medical Visual Question Answering (VQA) benchmark. This benchmark is collected from 73 different medical datasets, including 12 different modalities and covering more than 20 distinct anatomical regions. Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs. Through our extensive experiments, we have found that existing LVLMs struggle to address these medical VQA problems effectively. Moreover, what surprises us is that medical-specialized LVLMs even exhibit inferior performance to those general-domain models, calling for a more versatile and robust LVLM in the biomedical field. The evaluation results not only reveal the current limitations of LVLM in understanding real medical images but also highlight our dataset's significance. Our code with dataset are available at https://github.com/OpenGVLab/Multi-Modality-Arena.
1. Introduction
The paper introduces OmniMedVQA to address limited comprehensive evaluation of medical LVLMs using diverse real medical images. It evaluates general and medical-specialized LVLMs and reports persistent performance challenges.
- Existing medical LVLM evaluations lack comprehensive coverage of real medical images, limiting understanding of model applicability and performance.
- The dataset converts medical classification datasets into VQA format and adds incorrect options to create multiple-choice question-answer items.
- The benchmark contains 118,010 images and 127,995 test items collected from authentic medical scenarios.
- Medical-specialized LVLMs perform better on some modalities but do not consistently outperform general models across all modalities, while existing models remain unsatisfactory overall.
- OmniMedVQA is a large-scale medical VQA benchmark spanning 12 modalities and more than 20 anatomical regions.
- The evaluation covers 12 LVLMs, including 8 general-domain models and 4 medical-specialized models.
2. Related Work
Related work develops general-domain and medical-specialized LVLMs alongside Medical VQA datasets. The paper identifies limited dataset scale, modality diversity, anatomy coverage, and comprehensive model evaluation as open deficiencies.
- Large Vision-Language Models: General-domain LVLMs align visual features with language models through architectures such as Flamingo's cross-attention and BLIP2's Q-former.
- Large Vision-Language Models: Medical-specialized LVLMs include Med-Flamingo, LLaVA-Med, MedVInT, and RadFM, using medical multimodal data, instruction tuning, or radiologic images.
- Medical VQA Dataset: Existing Medical VQA datasets such as VQA-RAD, SLAKE, Path-VQA, and VQA-Med each contain fewer than 5K images.
- Medical VQA Dataset: Prior datasets also provide limited modality diversity and anatomical coverage, restricting comprehensive evaluation and further application.
- Medical VQA Dataset: OmniMedVQA addresses these gaps with real medical images across 12 modalities and almost every human anatomical region.
3. Dataset Collection
OmniMedVQA is constructed from diverse real medical classification datasets converted into balanced VQA items spanning modalities, anatomies, and question types.
- 73 medical classification datasets provide images spanning 12 imaging modalities and more than 20 human anatomical regions.
- QA templates convert classification categories and attributes such as modality and anatomy into question-answer pairs.Examples target diagnosis, modality recognition, and anatomy localization.
- Five question types evaluate modality recognition, anatomy identification, disease diagnosis, lesion grading, and other biological attributes.
- Inverse Proportional Sampling assigns smaller sample ratios to templates with more associated images, balancing categories and reducing duplicate-question bias.
- The resulting dataset contains 118,010 images and 127,995 QA-items across 12 modalities and more than 20 anatomical regions.
4. Evaluation Method
The evaluation turns each medical VQA item into a multiple-choice prompt and scores models by comparing generated responses or option likelihoods with the ground-truth answer.
- Each image, question, and candidate options are combined into a prompt before the LVLM generates a response.
- The final option is selected by choosing the candidate with the greatest similarity to the generated response, then comparing it with the ground truth.
- The prefix-based score computes each candidate option’s likelihood after visual features are prefixed into text embeddings supplied to the language model.
- The candidate with the highest prefix-based score becomes the predicted answer, and VQA accuracy is computed against the ground truth.
- The prefix-based score is not equivalent to direct LVLM responses but measures the likelihood of each option being regarded as correct.
5. Experiment
The experiments evaluate 12 representative LVLMs on OmniMedVQA using Question-answering and Prefix-based scores. Results show that the benchmark is highly challenging, with general-domain BLIP2 achieving the strongest average performance while medical-specialized models remain inconsistent across modalities.
- 5.1. Experimental Details: 12 representative LVLMs are evaluated using separate Question-answering and Prefix-based scores.The models include eight general-domain and four medical-domain LVLMs.
- 5.2. Overall Performance: Most LVLMs only slightly surpass random-guess performance on OmniMedVQA.Table 5 reports results across five question types using both evaluation scores.
- 5.2. Overall Performance: BLIP2 achieves the best average performance across all tasks, surpassing every evaluated medical-domain LVLM by a large margin.The reported comparison covers the tested general-purpose and medical-domain models.
- 5.2. Overall Performance: MedVInT and Med-Flamingo obtain the highest overall accuracy among evaluated medical LVLMs, outperforming many general-purpose models except BLIP2 and InstructBLIP.The paper attributes this result to extensive injected medical knowledge, including textbooks and image-caption pairs.
- 5.2. Overall Performance: Medical instruction tuning improves LLaVA's biomedical performance, but LLaVA-Med records the worst performance among the evaluated medical LVLMs.The comparison links LLaVA-Med's weaker result to limited and textually generated instruction data and an unsuitable pretrained model.
- 5.3. Analysis in terms of modalities: Medical LVLMs perform relatively well on CT and MRI but fail to show consistent superiority on modalities resembling general-domain images.RadFM performs best on CT and X-Ray tasks and competitively on MR, while the paper calls for high-quality data across additional modalities.
6. Conclusion
The conclusion presents OmniMedVQA as a large-scale medical VQA benchmark for evaluating LVLMs across diverse real-world medical imagery. Its broad coverage supports comprehensive model evaluation.
- 6. Conclusion: OmniMedVQA contains 118,010 images and 127,995 question-answer items across 12 modalities and more than 20 anatomical regions.The benchmark is designed to support thorough evaluation of different LVLMs.
7. The details of involved datasets
The benchmark combines numerous medical datasets converted into VQA format, with dataset metadata and access conditions documented. It includes both open-access and restricted-access resources, plus a self-constructed 3D modality dataset.
- 7. The details of involved datasets: OmniMedVQA is constructed by converting numerous medical datasets into VQA format and documenting modality, image, question-answer, and access information.Tables 8 and 9 list the included datasets and their metadata.
- 7. The details of involved datasets: The self-constructed 3D Modality dataset combines data from 17 medical datasets to create questions about 3D medical imaging.The listed sources include multiple stroke, lesion-segmentation, cardiac, and brain-tumor datasets.
- 7. The details of involved datasets: 42 datasets are completely open access, while 31 restricted-access datasets provide evaluation questions and instructions for associating them with original images.Researchers must download restricted original datasets to reconstruct the corresponding image-question pairs.
- 7. The details of involved datasets: Results on completely open-access datasets are provided to facilitate benchmark establishment and experimental analysis without downloading every restricted dataset.These results are reported in Tables 11–13.
8. The distribution of our dataset
The dataset's class distributions for modality recognition, anatomy identification, and disease diagnosis remain balanced, indicating no significant bias in these categories. The paper attributes this balance to its sampling process.
- 8. The distribution of our dataset: Modality recognition, anatomy identification, and disease diagnosis classes show no significant distribution bias in OmniMedVQA.The distributions are illustrated in Figure 4 and described as balanced.
9. The details of modalities
OmniMedVQA incorporates data from 12 medical imaging modalities and illustrates modality-specific images alongside their QA items.
- 12 different modalities are represented in OmniMedVQA.Figures 5 and 6 show modality-specific images together with corresponding QA items.
10. The details of multi-choice questions
OmniMedVQA constructs multiple-choice QA pairs using two to four candidate options, with question formats varying by option count.
- 2 to 4 candidate options are provided for each multiple-choice question.Incorrect options are generated for every item to construct the question-answer pairs.
- Questions with two options are framed as Yes/No selections.
- Questions with three options predominantly address Lesion Grading, which judges disease severity.