PaperlayerView exact revision on arXiv

Source-linked AI summary

PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering

Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, Weidi Xie

arXiv:2305.10415v6cs.CV

TL;DR

MedVQA needs models that can interpret medical images while answering open-ended questions beyond fixed answer sets. The paper introduces MedVInT, aligns a pre-trained vision encoder with a large language model, and constructs the PMC-VQA dataset for training. MedVInT outperforms existing models on public benchmarks, while the proposed challenging test set shows that substantial development remains.

  • Problem

    MedVQA requires accurate answers to text-based questions about medical images, but language-focused LLMs cannot directly interpret visual modalities and existing methods use limited answer bases.

  • Method

    The paper reframes MedVQA as generative learning, aligns a pre-trained vision encoder with a large language model, and constructs PMC-VQA through a scalable dataset pipeline.

  • Results

    MedVInT achieves over 80% accuracy on multi-choice selection and outperforms existing models on public MedVQA benchmarks.

  • Takeaways & Limitations

    PMC-VQA provides a large, diverse resource for MedVQA, while the challenging benchmark supports more demanding evaluation of generative methods.

  • Takeaways & Limitations

    PMC-VQA may differ from clinical practice because academic papers selectively present typical cases or annotated images, making its data simpler than clinical scenarios.

Abstract

from arXiv · show

Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery by leveraging artificial intelligence to interpret and answer questions based on medical images. In this study, we reframe the problem of MedVQA as a generation task that naturally follows the human-machine interaction and propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model. We establish a scalable pipeline to construct a large-scale medical visual question-answering dataset, named PMC-VQA, which contains 227k VQA pairs of 149k images that cover various modalities or diseases. We train the proposed model on PMC-VQA and then fine-tune it on multiple public benchmarks, e.g., VQA-RAD, SLAKE, and Image-Clef-2019, significantly outperforming existing MedVQA models in generating relevant, accurate free-form answers. In addition, we propose a test set that has undergone manual verification, which is significantly more challenging, serving to better monitor the development of generative MedVQA methods. To facilitate comprehensive evaluation and comparison, we have maintained a leaderboard at https://paperswithcode.com/paper/pmc-vqa-visual-instruction-tuning-for-medical, offering a centralized resource for tracking progress and benchmarking state-of-the-art approaches. The PMC-VQA dataset emerges as a vital resource for the field of research, and the MedVInT presents a significant breakthrough in the area of MedVQA.

1 Introduction

MedVQA requires models to interpret medical images while answering diverse clinical questions, but existing systems are limited by visual blindness, finite answer vocabularies, and small datasets. The paper addresses these gaps with generative visual instruction tuning, PMC-VQA, and a challenging benchmark.

  • LLMs excel at medical language but remain essentially blind to visual modalities, limiting their use for accurate medical visual question answering.
  • Existing MedVQA methods typically use retrieval, contrastive, or classification objectives with limited answer bases, restricting them to predefined outcomes.
  • 227k VQA pairs across 149k images form PMC-VQA, a scalable dataset covering diverse modalities and diseases and surpassing existing datasets in size and diversity.The dataset includes 80% radiological images.
  • MedVInT was trained on PMC-VQA and fine-tuned on public benchmarks, outperforming existing models by a large margin and exceeding 80% accuracy on multi-choice selection.The evaluated benchmarks include VQA-RAD, SLAKE, and ImageClef-VQA-2019.
  • The paper introduces a manually verified challenging test set because even state-of-the-art models continue to struggle on difficult MedVQA evaluation.
  • The proposed MedVInT reframes MedVQA as generative learning by aligning a pre-trained vision encoder with a large language model through visual instruction tuning.

2 Results

The study analyzes PMC-VQA’s diverse images, questions, and answers, then evaluates MedVInT on public benchmarks and a manually checked PMC-VQA test set. MedVInT surpasses prior methods on several public benchmarks, while the challenging test set exposes substantial remaining difficulty.

  • 2.1 Data Analysis: PMC-VQA contains 227k image-question pairs spanning diverse modalities, diseases, question types, and difficulty levels.Questions range from basic modality or anatomy identification to specialized questions involving compound figures.
  • 2.1 Data Analysis: Most PMC-VQA questions contain 5–15 words, while most answers are around 5 words and primarily describe positions, modalities, or anatomical regions.The answer options are distributed unevenly: A 24.07%, B 30.87%, C 29.09%, and D 15.97%.
  • 2.2 Evaluation on Public Benchmarks: MedVInT-TE and MedVInT-TD surpass previous best-performing methods on VQA-RAD and SLAKE, with open-ended accuracy improving from 67.2% to 73.7% on VQA-RAD and from 81.9% to 88.2% on SLAKE.For close-ended questions, accuracy improves from 84.0% to 86.8%.
  • 2.2 Evaluation on Public Benchmarks: Pre-training on PMC-VQA improves open-ended accuracy by approximately 16% on VQA-RAD and 4% on SLAKE for MedVInT-TE relative to the same architecture trained from scratch.Similar enhancements are observed for MedVInT-TD.
  • 2.3 Evaluation on PMC-VQA: The manually checked PMC-VQA-test evaluates open-ended and multiple-choice performance, where language-only models achieve near-guessing accuracy and multimodal understanding remains necessary.LLaMA reaches 27.2% in Blanking and 30.8% in Choice, while GPT-4 reaches 21.1% and 25.7%, respectively; around 30% of questions have “B” answers.
  • 2.4 Evaluation of Visual Backbone Performance: MedVInT remains competitive across DermaMNIST, PneumoniaMNIST, and BreastMNIST, despite pre-training on 177K images versus PMC-CLIP’s 1.6M image-caption pairs.MedVInT-TE achieves the best performance on DermaMNIST and the second-best performance on PneumoniaMNIST and BreastMNIST.

3 Discussion

The discussion positions MedVInT and PMC-VQA as advances for generative MedVQA, while emphasizing benchmark difficulty, broader adoption, and limitations in clinical realism and evaluation.

  • PMC-VQA resource: PMC-VQA provides 227k VQA pairs across 149k images and supports training models that outperform comparable systems without PMC-VQA pre-training.The dataset covers diverse medical modalities and diseases and was constructed through a scalable pipeline.
  • Benchmark difficulty: General visual-language models such as BLIP-2 and Flamingo struggle on PMC-VQA, underscoring the dataset’s biomedical difficulty.The discussion attributes this challenge to the strong biomedical relevance of the questions and images.
  • MedVInT performance: MedVInT achieves state-of-the-art performance on existing generative MedVQA benchmarks, with specialized medical visual backbones improving results.The discussion also reports different training styles favoring longer-answer VQA-RAD or shorter-response SLAKE questions.
  • Benchmark difficulty: PMC-VQA-test is a more rigorous benchmark because even state-of-the-art medical multimodal models perform poorly, including near-random multi-choice accuracy for PMC-CLIP.The benchmark includes blanking and multiple-choice evaluation tasks.
  • Impact: Since its public release, PMC-VQA has supported numerous generative medical models and influenced dataset-construction and prompting methods.The discussion cites adoption in systems including MathVista, RadFM, Qilin-Med-VL, CheXagent, and Quilt-LLaVA.
  • Limitations: PMC-VQA may contain distribution bias because academic-paper images often depict typical cases or annotated slices that are simpler than clinical scenarios.The authors regard the dataset as useful for training, while identifying clinical realism as a testing boundary.

4 Method

The method constructs PMC-VQA through a scalable dataset pipeline and trains MedVInT as an open-ended generative MedVQA model. The pipeline combines automated generation and filtering with manual test-set verification, while the model aligns visual and textual information for free-form answer generation.

  • PMC-VQA Dataset: The pipeline starts from PMC-OA medical figure-caption data and uses a scalable flow to construct PMC-VQA.The source includes 381K image-caption pairs from the medical figure collection stage, without subfigure auto-separation.
  • PMC-VQA Dataset: ChatGPT generates five question-answer pairs per image caption, each with one correct and three randomized incorrect options.Pairs based on captions that are too short or produce dummy outputs are discarded before further filtering.
  • PMC-VQA Dataset: Text-only LLaMA filtering removes questions answerable from biomedical knowledge without requiring the image.Questions answered correctly in at least three of five shuffled-choice inferences are dismissed.
  • PMC-VQA Dataset: 81.77% accuracy was achieved by a question-classification filter trained to identify whether questions are answerable from images alone.The classifier was trained and evaluated using 1,752 and 440 manually labeled pairs, respectively, producing 226,946 pairs for 149,075 images.
  • PMC-VQA Dataset: A manually verified test set contains 2,000 samples selected from an initial 50,000-pair test set with image-disjoint training and testing data.Verification checks image-question relevance, distractor complexity, and image quality; over 80% of cases were retained.
  • Generative MedVQA Model: MedVInT reframes MedVQA as open-ended natural-language generation rather than classification over a fixed answer set.Its training objective maximizes the probability of the ground-truth answer token by token, using visual and textual encoders with a multimodal decoder; two variants support encoder- and decoder-based language models.

4.3 Datasets and Backbones

The evaluation uses three public MedVQA benchmarks and compares MedVInT with generative and established medical visual-question-answering models. Experiments use medical or general-purpose language and vision backbones, with accuracy and BLEU-based evaluation for supported task formats.

  • Evaluation Datasets: MedVInT is evaluated on VQA-RAD, SLAKE, and ImageClef-VQA-2019 using their official dataset splits.The benchmarks contain 315, 642, and 4,200 images, respectively, with questions spanning radiology, bilingual medical VQA, modality, plane, organ system, and abnormality.
  • Evaluation Protocol: The datasets support both multiple-choice and open-ended answering, with accuracy used for option selection and BLEU plus accuracy for open-ended evaluation.PMC-VQA has over 100K possible answers, motivating direct answer generation without input options for the blank setting.
  • Baselines: Comparisons include Open-Flamingo and BLIP-2 alongside Hanlin, MEVF-BAN, CPRD-BAN, M3AE, and PMC-CLIP.Open-Flamingo and BLIP-2 are evaluated using released checkpoints, while several medical baselines use reported benchmark results under fine-tuning settings.
  • Backbones: PMC-LLaMA or PMC-LLaMA-ENC and PMC-CLIP are the default language and vision backbones because prior work found them more suitable for medical data.The study also examines LLaMA, PubMedBERT, and encoder variants of LLaMA and PMC-LLaMA.
  • Training Setup: Training uses AdamW with learning rate 2e-5, context length 512, batch size 128, and eight NVIDIA A100 GPUs.Deepspeed, Automatic Mixed Precision, and gradient checkpointing are used to improve training speed.
  • Evaluation Protocol: Generative-model accuracy is computed by matching generated outputs with answer options using difflib.SequenceMatcher.This evaluation is described as more difficult than retrieval-based evaluation because generated outputs have an unlimited space.

5 Conclusion

The paper reframes MedVQA as generation by aligning a pre-trained vision encoder with a large language model and introduces PMC-VQA to support training. MedVInT achieves state-of-the-art performance on existing MedVQA datasets, while PMC-VQA provides a benchmark resource for method evaluation.

  • Conclusion: MedVInT aligns visual information from a pre-trained vision encoder with a large language model for generative MedVQA.The approach is intended to mirror human-machine interaction by treating MedVQA as a generation task.
  • Conclusion: PMC-VQA contains 227k VQA pairs across 149k images spanning diverse medical modalities and diseases.The dataset is introduced to facilitate model training and provide a comprehensive MedVQA resource.
  • Conclusion: MedVInT delivers state-of-the-art performance on existing MedVQA datasets and supports comparison among methods.The conclusion characterizes PMC-VQA as a benchmark resource for evaluating different approaches in the field.

7 Data Availability

PMC-VQA is made available under a CC BY-SA license, with source-paper identifiers and licensing information provided for the papers used to construct it.

  • Data Availability: PMC-VQA is available on Hugging Face under a CC BY-SA license for developing medical generative-based VQA models.The source papers come from the Commercial Use Allowed PMC Open Access split and are licensed under CC0 or CC BY.

A.1 Data Analysis

The data analysis examines question and answer word-length distributions. Most questions contain 5–15 words, while most answers are approximately five words.

  • Most questions range from 5 to 15 words.The distribution is reported as percentages of questions and answers by word length.
  • Most answers are around 5 words.
  • The analysis compares word-length distributions for questions and answers.

A.2 Evaluation on Original Split Test Set

On the original randomly split test set, performance declined slightly relative to PMC-VQA-test, with only a minimal reduction reported for the choice task.

  • Performance experienced a slight decline compared with PMC-VQA-test.The original test set was more extensive but did not undergo additional manual verification.
  • 40.3 to 39.2: choice-task accuracy decreased on the original test set.
  • The minimal variation underscores the dataset’s reported high quality and robustness.

A.3 Ablation Study

The ablation study compares MLP-based and transformer-based projection modules across open-ended and multiple-choice tasks, finding comparable performance across variants.

  • MLP-based projection uses a two-layer Multilayer Perceptron.
  • Transformer-based projection uses a 12-layer transformer decoder with learnable query vectors.
  • Comparable performance was observed across projection-module variants and evaluated tasks.The comparison covers both open-ended and multiple-choice tasks.

A.4 Fail Case Study

The fail-case study examines hallucinations in out-of-scope and nonsensical cases. MedVInT may produce unstructured report-generation responses and cannot reliably reject questions whose requested finding does not match the image.

  • Fail cases: For out-of-scope report generation, MedVInT may fail to produce a structured radiology report.The example asks for a report explaining a brain MRI image.
  • Fail cases: For nonsensical questions, the model cannot refuse to answer or highlight the mismatch.The cited example asks about lung nodules in an abdomen CT image.
  • Fail cases: Some out-of-scope or nonsensical cases nevertheless receive reasonable answers.
Loading 2305.10415v6…