PaperlayerView exact revision on arXiv

Source-linked AI summary

HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale

Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, Benyou Wang

arXiv:2406.19280v4cs.CVcs.AIcs.CLcs.LG

TL;DR

Medical MLLMs remain limited by scarce, noisy medical vision-text data, despite PubMed’s large de-identified image-text resources. The paper filters PubMed data and uses GPT-4V for unblinded multimodal reformatting, constructing PubMedVision with 1.3 million VQA samples. PubMedVision improves medical multimodal performance and data quality, and supports the 34B HuatuoGPT-Vision model.

  • Problem

    Medical MLLMs lack sufficient high-quality medical visual knowledge, while PubMed-based training is limited by noisy descriptions and prior blinded text-only reformulation.

  • Method

    The paper filters PubMed image-text pairs and uses GPT-4V as an unblinded reformatter to generate better-aligned medical VQA data.

  • Results

    PubMedVision contains 1.3 million medical VQA samples and significantly improves MLLM medical multimodal capabilities, including an 11.7% overall-accuracy increase in medical VQA benchmarks.

  • Takeaways & Limitations

    PubMedVision provides a large-scale, higher-quality resource for medical visual alignment, while HuatuoGPT-Vision achieves superior performance among open-source medical MLLMs.

Abstract

from arXiv · show

The rapid development of multimodal large language models (MLLMs), such as GPT-4V, has led to significant advancements. However, these models still face challenges in medical multimodal capabilities due to limitations in the quantity and quality of medical vision-text data, stemming from data privacy concerns and high annotation costs. While pioneering approaches utilize PubMed's large-scale, de-identified medical image-text pairs to address these limitations, they still fall short due to inherent data noise. To tackle this, we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in an 'unblinded' capacity to denoise and reformat the data, resulting in the creation of the PubMedVision dataset with 1.3 million medical VQA samples. Our validation demonstrates that: (1) PubMedVision can significantly enhance the medical multimodal capabilities of current MLLMs, showing significant improvement in benchmarks including the MMMU Health & Medicine track; (2) manual checks by medical experts and empirical results validate the superior data quality of our dataset compared to other data construction methods. Using PubMedVision, we train a 34B medical MLLM HuatuoGPT-Vision, which shows superior performance in medical multimodal scenarios among open-source MLLMs.

1 Introduction

Medical MLLMs lack sufficient high-quality, domain-specific visual knowledge, while PubMed-scale approaches are undermined by noisy or blinded text reformulation. The paper addresses this with unblinded MLLM reformatting, producing PubMedVision and reporting stronger medical multimodal performance and data quality.

  • Medical MLLMs show limited medical performance because high-quality medical vision-text data are difficult to scale amid privacy and annotation constraints.Existing small datasets are high quality but difficult to expand, while medical data also raise privacy and licensing concerns.
  • PubMed offers large, de-identified medical image-text resources, but surrounding text often contains noise that harms multimodal training.Prior PubMed-trained models performed poorly against general MLLMs on medical multimodal tasks, which the paper attributes to data noise.
  • LLaVA-Med’s blinded text-only generation can miss image information and produced only 56K medical VQA entries.Because text-only LLMs cannot perceive the images, their generated answers may be misinterpreted or irrelevant.
  • The paper uses GPT-4V as an unblinded reformatter for 914,960 refined medical images and associated text, denoising and realigning PubMed data.The approach contrasts with prior blinded reformatting and generates more aligned medical VQA data.
  • PubMedVision contains 1.3 million medical VQA entries, and expert checks and experiments support its superior data quality and improved medical multimodal capabilities.The paper reports notable improvement on benchmarks including MMMU Health & Medicine and identifies LLaVA-v1.5-LLaMA-3-8B as the strongest open-source MLLM with PubMedVision.

2 Medical Visual Alignment in MLLMs

Existing medical VQA resources are small or vulnerable to misalignment, while PubMed captions may not accurately describe images. The paper motivates multimodal reformatting that uses both visual information and contextual text to improve alignment.

  • Existing Medical VQA Data: Early medical VQA datasets contain fewer than 20K entries and focus primarily on radiology, limiting scale and diversity.
  • Existing Medical VQA Data: PubMed-based datasets scale image-text alignment by rewriting contextual text into VQA, but blinded LLMs can misinterpret or misalign image descriptions.
  • Medical Visual Alignment: Current MLLMs typically adapt text-only LLMs with visual encoders, making image-knowledge injection central to medical visual alignment.
  • Data Reformatting Approaches: MLLM-Reformatted combines visual information with contextual cues, unlike text-only LLM-Reformatted and image-only GPT4v-Distill approaches.
  • Case Analysis: MLLM-Reformatted produces more accurate and thorough descriptions, whereas LLM-Reformatted misinterprets three sub-images and GPT4v-Distill generates factually incorrect descriptions.
  • Dataset Construction: Figure 2 depicts the construction process of the PubMedVision dataset.

3 PubMedVision

PubMedVision filters PubMed image-text data and uses unblinded MLLMs to generate aligned medical VQA at scale. The pipeline produces diverse image coverage and 1.3 million samples from 914,960 refined medical images.

  • 3.1 Data Collection: 914,960 medical images and associated contextual text were filtered for PubMedVision, covering diverse medical modalities and image regions.Figure 3 categorizes a random sample of 500 images by body parts and imaging modalities.
  • 3.2 Data Reformatting with MLLMs: MLLMs receive medical images and contextual descriptions to generate an overall description, an image-specific question, and its answer.The prompts incorporate medical images, scenario prompts, captions, and inline mentions.
  • 3.2 Data Reformatting with MLLMs: Alignment VQA combines a predefined question with the generated image description to support image-to-text alignment.The predefined question is sampled from a set of alignment questions.
  • 3.2 Data Reformatting with MLLMs: Instruction-Tuning VQA uses MLLM-generated image-specific questions and answers, with eight scenarios diversifying the questions.Scenario settings are randomly sampled into the synthetic prompt.
  • 3.2 Data Reformatting with MLLMs: 647,031 Alignment VQA and 647,031 Instruction-Tuning VQA samples yield 1.3 million PubMedVision data points.GPT-4V is used as the MLLM to synthesize both VQA types.

4 Experiment

The experiments evaluate PubMedVision across medical VQA, multimodal, and traditional medical imaging benchmarks, while comparing models and data-construction methods. Results show improved medical multimodal performance, broader applicability, and stronger data quality for MLLM-reformatted data.

  • 4.1 Experiment Settings: The experimental comparison includes the LLaVA-v1.5-LLaMA3-8B baseline, LLaVA_Med, and PubMedVision-enhanced training configurations.The baseline uses only LLaVA-1.5 data, while comparison models add medical datasets.
  • 4.1 Experiment Settings: The evaluation uses medical VQA, MMMU Health & Medicine, and OmniMedVQA benchmarks to assess medical multimodal capabilities.The medical VQA evaluation includes VQA-RAD, SLAKE, PathVQA, and PMC-VQA; OmniMedVQA covers 42 traditional imaging datasets.
  • 4.2 Experiment 1: Effectiveness of PubMedVision: PubMedVision increases overall medical VQA accuracy by 11.7% and significantly outperforms the earlier LLaVA_Med dataset.Adding medical multimodal data to LLaVA-v1.5-LLaMA3-8B substantially improves medical image understanding.
  • 4.2 Experiment 1: Effectiveness of PubMedVision: LLaVA-v1.5-LLaMA3-8B with PubMedVision surpasses other models on the MMMU Health & Medicine test set, approaching LLaVA-v1.6-34B performance.The result supports PubMedVision’s effectiveness for medical image alignment.
  • 4.3 Experiment 2: Data Quality of PubMedVision: MLLM-Reformatted data outperforms alternative datasets at the same volume, while full PubMedVision also outperforms Native-Captions.Expert evaluation finds MLLM-Reformatted strongest across completeness, usefulness, accuracy, and relevance.

5 Related Works

Related work develops general MLLM architectures, medical MLLMs, and medical VQA datasets for image-text alignment. Existing medical datasets remain limited in size, diversity, or alignment, motivating larger and better-aligned resources.

  • Multimodal Large Language Models: Flamingo, BLIP2, InstructBLIP, and LLaVA develop mechanisms for integrating visual features with language models and instruction-following data.Their approaches include cross-attention, Q-formers, and multimodal instruction tuning.
  • Medical VQA Datasets: Dataset comparisons distinguish native captions from reformatted data and identify Native Caption of PubMedVision as used in both training phases.The 60K comparison dataset is added only during the second training stage.
  • Medical MLLMs: Medical MLLMs such as Med-Flamingo and LLaVA-Med adapt multimodal modeling and PubMed-derived data to medical image understanding.LLaVA-Med filters PubMed image-text pairs and uses LLM-synthesized VQA data.
  • Medical VQA Datasets: Early medical VQA datasets contain fewer than 20K samples and focus primarily on radiology modalities.VQA-RAD, SLAKE, and Path-VQA are identified as early datasets with limited size and diversity.
  • Medical VQA Datasets: PubMed-based datasets increase scale but can suffer misinterpretation and misalignment when LLMs reformat contextual text without seeing the images.The comparison highlights the limitation of blinded data reformatting.

6 Conclusion

The study refines PubMed medical image-text data with MLLM-powered reformatting to construct PubMedVision, a large-scale, high-quality medical multimodal dataset. Experiments show that PubMedVision improves MLLM multimodal capabilities and suggests that PubMed’s main challenge is data quality.

  • 6 Conclusion: PubMedVision combines refined PubMed image-text data with MLLM-powered reformatting to create a large-scale, high-quality medical multimodal dataset.The study first refines numerous medical image-text pairs and then applies MLLM-powered reformatting.
  • 6 Conclusion: PubMedVision significantly boosts MLLM multimodal capabilities, producing marked improvements on medical benchmarks.The conclusion reports broad benchmark improvements rather than a result limited to a single task.
  • 6 Conclusion: PubMed has substantial potential for advancing medical multimodal capabilities, but poor descriptions and non-medical images make data quality the key challenge.This conclusion connects the dataset results to the broader role of PubMed data.
  • 6 Conclusion: PubMedVision is intended to support future development of medical MLLMs.

A More Experiments

Additional experiments evaluate PubMedVision on downstream VQA benchmarks and the MMMU Health & Medicine validation set. The results show broad task improvements and indicate that an 8B LLaVA model with PubMedVision can approach the medical-task effects of a 34B version.

  • VQA Benchmarks: PubMedVision significantly improves performance across all four evaluated VQA downstream tasks after fine-tuning.The benchmark models were fine-tuned on each task’s built-in training set for two epochs.
  • MMMU Validation: On MMMU validation, LLaVA-v1.6-34B has the strongest overall performance, while LLaVA-v1.5-LLaMA3-8B with PubMedVision performs better than its official test-set result.The passage distinguishes validation-set performance from the official MMMU test-set comparison.
  • MMMU Validation: PubMedVision enables the 8B LLaVA version to achieve effects comparable to the 34B version in medical applications.
  • MMMU Validation: The MMMU Health & Medicine track contains five categories: Basic Medical Science, Clinical Medicine, Diagnostics and Laboratory Medicine, Pharmacy, and Public Health.The table labels these categories BMS, CM, DLM, P, and PH, respectively.

B Data Pipline

The data pipeline integrates several PubMed image-text sources and filters them to retain informative medical image-text pairs. It uses textual medical-term filtering, image-quality and medical-content filtering, and deduplication.

  • Data Collection: The pipeline integrates LLaVA-Med PMC data, PMC-Inline, and PMC-OA to build a comprehensive PubMed image-text collection.The source collections contain 514K, 11M, and 1M items, respectively.
  • Contextual Text Filtering: Captions with fewer than five medical terms are filtered out using a refined medical lexicon derived from the SPECIALIST Lexicon and GPT-4.This step is designed to retain captions that are sufficiently informative.
  • Image Filtering: Images below 336x336 pixels and chart images are excluded before a CLIP-based classifier removes non-medical images.The classifier is trained from 1K manually labeled images and 10K MLLM-synthesized labels, reaching 91% validation accuracy.
  • Deduplication: The pipeline also includes deduplication after image and text-based filtering.The supplied passage introduces deduplication as the next pipeline stage.

C Question Set of Alignment VQA

Alignment VQA samples are generated from image descriptions and predefined single- or multi-image question sets, while Instruction-Tuning VQA uses ten balanced medical interaction scenarios. These scenarios cover patient, clinician, educational, and evaluation contexts.

  • Alignment VQA: Alignment VQA samples pair a generated image description with a question sampled from a predefined question set.Single-image questions are used for one image, and multi-image questions for multiple images.
  • Alignment VQA: Single-image and multi-image question sets express the same meaning through different natural-language formulations.
  • Instruction-Tuning VQA: Instruction-Tuning VQA uses ten preset scenarios to broaden medical topic coverage and improve instruction-following diversity.The sampling strategy also prevents overconcentration or absence of particular scenarios.
  • Instruction-Tuning VQA: The scenarios include standard Q&A, AI-assisted doctor consultation, patient assistance, family-doctor communication, and difficult-patient diagnosis discussions.
  • Instruction-Tuning VQA: Additional scenarios cover doctor-to-doctor discussion, evaluator assessment, intern-specialist teaching, teacher-student diagnosis, and senior-doctor intern questioning.

E Prompts for Evaluation

Evaluation uses a unified template, with Figure 18 identified as the evaluation prompt.

  • A unified template was used during evaluation.
  • Figure 18 is labeled as the prompt for evaluation.
  • The evaluation prompt is presented as a figure in the paper.

F Comparison of Methods for Constructing Multimodal Datasets

The paper compares multimodal dataset-construction methods by standardizing captions into VQA and evaluating their accuracy, while documenting dataset limitations and scoring guidance.

  • Comparison of Methods for Constructing Multimodal Datasets: Four caption-construction methods are compared by combining standardized captions with a medical-image description query to form VQA data.
  • Comparison of Methods for Constructing Multimodal Datasets: The LLM-reformatted prompt asks for an answer from textual image context without access to the actual image.
  • Comparison of Methods for Constructing Multimodal Datasets: Dataset scoring rates whether descriptions accurately reflect medical image features on a 1-to-5 accuracy scale.
  • Limitations: PubMedVision may contain hallucinations or inaccuracies because GPT-4V generated the dataset, and its instruction-tuning VQA covers 10 predefined scenarios.
Loading 2406.19280v4…