HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale

• Medical MLLMs show limited medical performance because high-quality medical vision-text data are difficult to scale amid privacy and annotation constraints.Existing small datasets are high quality but difficult to expand, while medical data also raise privacy and licensing concerns.Source: §1 · p. 1
• PubMed offers large, de-identified medical image-text resources, but surrounding text often contains noise that harms multimodal training.Prior PubMed-trained models performed poorly against general MLLMs on medical multimodal tasks, which the paper attributes to data noise.Source: §1 · p. 1
• LLaVA-Med’s blinded text-only generation can miss image information and produced only 56K medical VQA entries.Because text-only LLMs cannot perceive the images, their generated answers may be misinterpreted or irrelevant.Source: §1 · p. 2
• The paper uses GPT-4V as an unblinded reformatter for 914,960 refined medical images and associated text, denoising and realigning PubMed data.The approach contrasts with prior blinded reformatting and generates more aligned medical VQA data.Source: §1 · p. 2
• PubMedVision contains 1.3 million medical VQA entries, and expert checks and experiments support its superior data quality and improved medical multimodal capabilities.The paper reports notable improvement on benchmarks including MMMU Health & Medicine and identifies LLaVA-v1.5-LLaMA-3-8B as the strongest open-source MLLM with PubMedVision.Source: §1 · p. 2

Existing Medical VQA Data

• Early medical VQA datasets contain fewer than 20K entries and focus primarily on radiology, limiting scale and diversity.Source: §2.1 · p. 2
• PubMed-based datasets scale image-text alignment by rewriting contextual text into VQA, but blinded LLMs can misinterpret or misalign image descriptions.Source: §2.1 · p. 2, §2.2 · p. 3

Medical Visual Alignment

• Current MLLMs typically adapt text-only LLMs with visual encoders, making image-knowledge injection central to medical visual alignment.Source: §2.2 · p. 3

Data Reformatting Approaches

• MLLM-Reformatted combines visual information with contextual cues, unlike text-only LLM-Reformatted and image-only GPT4v-Distill approaches.Source: §2.2 · p. 3

Case Analysis

• MLLM-Reformatted produces more accurate and thorough descriptions, whereas LLM-Reformatted misinterprets three sub-images and GPT4v-Distill generates factually incorrect descriptions.Source: §2.2 · p. 3

Dataset Construction

• Figure 2 depicts the construction process of the PubMedVision dataset.Source: Figure 2 · p. 4

3.1 Data Collection

• 914,960 medical images and associated contextual text were filtered for PubMedVision, covering diverse medical modalities and image regions.Figure 3 categorizes a random sample of 500 images by body parts and imaging modalities.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Data Reformatting with MLLMs

• MLLMs receive medical images and contextual descriptions to generate an overall description, an image-specific question, and its answer.The prompts incorporate medical images, scenario prompts, captions, and inline mentions.Source: §3.2 · p. 5, Figure 4 · p. 5
• Alignment VQA combines a predefined question with the generated image description to support image-to-text alignment.The predefined question is sampled from a set of alignment questions.Source: §3.2 · p. 5
• Instruction-Tuning VQA uses MLLM-generated image-specific questions and answers, with eight scenarios diversifying the questions.Scenario settings are randomly sampled into the synthetic prompt.Source: §3.2 · p. 5
• 647,031 Alignment VQA and 647,031 Instruction-Tuning VQA samples yield 1.3 million PubMedVision data points.GPT-4V is used as the MLLM to synthesize both VQA types.Source: §3.2 · p. 5

4.1 Experiment Settings

• The experimental comparison includes the LLaVA-v1.5-LLaMA3-8B baseline, LLaVA_Med, and PubMedVision-enhanced training configurations.The baseline uses only LLaVA-1.5 data, while comparison models add medical datasets.Source: §4.1 · pp. 5–6
• The evaluation uses medical VQA, MMMU Health & Medicine, and OmniMedVQA benchmarks to assess medical multimodal capabilities.The medical VQA evaluation includes VQA-RAD, SLAKE, PathVQA, and PMC-VQA; OmniMedVQA covers 42 traditional imaging datasets.Source: §4.1 · p. 6

4.2 Experiment 1: Effectiveness of PubMedVision

• PubMedVision increases overall medical VQA accuracy by 11.7% and significantly outperforms the earlier LLaVA_Med dataset.Adding medical multimodal data to LLaVA-v1.5-LLaMA3-8B substantially improves medical image understanding.Source: §4.2 · p. 6
• LLaVA-v1.5-LLaMA3-8B with PubMedVision surpasses other models on the MMMU Health & Medicine test set, approaching LLaVA-v1.6-34B performance.The result supports PubMedVision’s effectiveness for medical image alignment.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Experiment 2: Data Quality of PubMedVision

• MLLM-Reformatted data outperforms alternative datasets at the same volume, while full PubMedVision also outperforms Native-Captions.Expert evaluation finds MLLM-Reformatted strongest across completeness, usefulness, accuracy, and relevance.Source: §4.3 · p. 8

Multimodal Large Language Models

• Flamingo, BLIP2, InstructBLIP, and LLaVA develop mechanisms for integrating visual features with language models and instruction-following data.Their approaches include cross-attention, Q-formers, and multimodal instruction tuning.Source: §5 · pp. 8–9

Medical VQA Datasets

• Dataset comparisons distinguish native captions from reformatted data and identify Native Caption of PubMedVision as used in both training phases.The 60K comparison dataset is added only during the second training stage.Source: Table 6 · p. 9

Medical MLLMs

• Medical MLLMs such as Med-Flamingo and LLaVA-Med adapt multimodal modeling and PubMed-derived data to medical image understanding.LLaVA-Med filters PubMed image-text pairs and uses LLM-synthesized VQA data.Source: §5 · p. 9

Medical VQA Datasets

• Early medical VQA datasets contain fewer than 20K samples and focus primarily on radiology modalities.VQA-RAD, SLAKE, and Path-VQA are identified as early datasets with limited size and diversity.Source: §5 · p. 9
• PubMed-based datasets increase scale but can suffer misinterpretation and misalignment when LLMs reformat contextual text without seeing the images.The comparison highlights the limitation of blinded data reformatting.Source: §5 · p. 9

6 Conclusion

• PubMedVision combines refined PubMed image-text data with MLLM-powered reformatting to create a large-scale, high-quality medical multimodal dataset.The study first refines numerous medical image-text pairs and then applies MLLM-powered reformatting.Source: §6 · p. 9
• PubMedVision significantly boosts MLLM multimodal capabilities, producing marked improvements on medical benchmarks.The conclusion reports broad benchmark improvements rather than a result limited to a single task.Source: §6 · p. 9
• PubMed has substantial potential for advancing medical multimodal capabilities, but poor descriptions and non-medical images make data quality the key challenge.This conclusion connects the dataset results to the broader role of PubMed data.Source: §6 · p. 9
• PubMedVision is intended to support future development of medical MLLMs.Source: §6 · p. 9

VQA Benchmarks

• PubMedVision significantly improves performance across all four evaluated VQA downstream tasks after fine-tuning.The benchmark models were fine-tuned on each task’s built-in training set for two epochs.Source: A More Experiments · p. 13, Table 7 · p. 13

MMMU Validation

• On MMMU validation, LLaVA-v1.6-34B has the strongest overall performance, while LLaVA-v1.5-LLaMA3-8B with PubMedVision performs better than its official test-set result.The passage distinguishes validation-set performance from the official MMMU test-set comparison.Source: A More Experiments · p. 13
• PubMedVision enables the 8B LLaVA version to achieve effects comparable to the 34B version in medical applications.Source: A More Experiments · p. 13
• The MMMU Health & Medicine track contains five categories: Basic Medical Science, Clinical Medicine, Diagnostics and Laboratory Medicine, Pharmacy, and Public Health.The table labels these categories BMS, CM, DLM, P, and PH, respectively.Source: Table 8 · p. 13

Data Collection

• The pipeline integrates LLaVA-Med PMC data, PMC-Inline, and PMC-OA to build a comprehensive PubMed image-text collection.The source collections contain 514K, 11M, and 1M items, respectively.Source: B Data Pipline · p. 13

Contextual Text Filtering

• Captions with fewer than five medical terms are filtered out using a refined medical lexicon derived from the SPECIALIST Lexicon and GPT-4.This step is designed to retain captions that are sufficiently informative.Source: B Data Pipline · p. 13

Image Filtering

• Images below 336x336 pixels and chart images are excluded before a CLIP-based classifier removes non-medical images.The classifier is trained from 1K manually labeled images and 10K MLLM-synthesized labels, reaching 91% validation accuracy.Source: B Data Pipline · p. 14

Deduplication

• The pipeline also includes deduplication after image and text-based filtering.The supplied passage introduces deduplication as the next pipeline stage.Source: B Data Pipline · p. 14

Alignment VQA

• Alignment VQA samples pair a generated image description with a question sampled from a predefined question set.Single-image questions are used for one image, and multi-image questions for multiple images.Source: C Question Set of Alignment VQA · p. 14
• Single-image and multi-image question sets express the same meaning through different natural-language formulations.Source: Figure 6 · p. 14, Figure 7 · p. 14

Instruction-Tuning VQA

• Instruction-Tuning VQA uses ten preset scenarios to broaden medical topic coverage and improve instruction-following diversity.The sampling strategy also prevents overconcentration or absence of particular scenarios.Source: D Prompts for different QA scenarios · p. 15
• The scenarios include standard Q&A, AI-assisted doctor consultation, patient assistance, family-doctor communication, and difficult-patient diagnosis discussions.Source: Figure 8 · p. 15, Figure 9 · p. 15, Figure 10 · p. 15
• Additional scenarios cover doctor-to-doctor discussion, evaluator assessment, intern-specialist teaching, teacher-student diagnosis, and senior-doctor intern questioning.Source: Figure 13 · p. 16, Figure 14 · p. 16, Figure 15 · p. 17

• A unified template was used during evaluation.Source: E Prompts for Evaluation · p. 17
• Figure 18 is labeled as the prompt for evaluation.Source: Figure 18 · p. 17
• The evaluation prompt is presented as a figure in the paper.Source: Figure 18 · p. 17

Comparison of Methods for Constructing Multimodal Datasets

• Four caption-construction methods are compared by combining standardized captions with a medical-image description query to form VQA data.Source: F Comparison of Methods for… · p. 18
• The LLM-reformatted prompt asks for an answer from textual image context without access to the actual image.Source: Figure 19 · p. 18
• Dataset scoring rates whether descriptions accurately reflect medical image features on a 1-to-5 accuracy scale.Source: Figure 20 · p. 19

Limitations

• PubMedVision may contain hallucinations or inaccuracies because GPT-4V generated the dataset, and its instruction-tuning VQA covers 10 predefined scenarios.Source: H Limiations · p. 19, I Ethical Statement · p. 19