Source-linked AI summary
MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
Hyunjae Kim, Dain Kim, Pan Xiao, Serina S. Applebaum, Younjoon Chung, Xuguang Ai, Yu Yin, Roy Jiang, Yuexi Du, Yawen Wei, Yiming Kong, Tuo Guo, Zhiyuan Cao, Mengmeng Du, Yuelei Fu, Yan Hu, Rui Shi, Gui Yang, Kevin W. Jin, Yuntian Liu, Yuxuan Tian, Jonathan Marquez, Zhen Chen, Sheng Zhang, Hoifung Poon, Hua Xu, Jaewoo Kang, Qingyu Chen
TL;DR
Medical multimodal foundation models need high-fidelity data to support clinicians’ integration of diverse information streams. MedPMC automatically curates aligned medical image-text data from biomedical literature, and its trained model improved average zero-shot AUC by 7.1 percentage points across 26 benchmarks in 11 specialties.
Problem
Medical multimodal foundation models require reusable data resources for integrating the diverse information streams used in clinical care.
Method
MedPMC automatically transforms permissively licensed biomedical literature into high-fidelity medical image-text data through modular curation stages, including panel-level alignment.
Results
A MedPMC-trained model improved average zero-shot AUC by 7.1 percentage points over BMC-CLIP across 26 benchmarks spanning 11 specialties.
Takeaways & Limitations
High-fidelity, reproducibly curated literature data supported effective medical multimodal model development and consistent gains across medical domains and downstream tasks.
Takeaways & Limitations
MedPMC focuses on image-text data and does not capture broader clinical modalities such as laboratory values, longitudinal records, and real-world workflows.
Abstract
from arXiv · showhide
Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
1 Introduction
MedPMC addresses the lack of large-scale, high-quality, publicly accessible multimodal medical data by converting permissively licensed biomedical literature into continuously evolving infrastructure. Its curated data and models improve medical visual representation, multimodal reasoning, and clinical retrieval across specialties and downstream tasks.
- Motivation and challenge: Clinical care integrates images, text, laboratory values, and other complementary data streams, but medical AI remains largely single-modality and single-task because suitable multimodal data are scarce.The needed data must be publicly accessible, shareable across institutions, and suitable for reproducible research.
- Motivation and challenge: Biomedical literature is difficult to repurpose because it contains clinically irrelevant visual content and figures requiring substantial processing before multimodal training.Examples of less useful visual content include graphs, charts, schematics, flow diagrams, and molecular illustrations.
- MedPMC framework: MedPMC is an automated, systematic framework that curates high-fidelity medical multimodal data from permissively licensed literature as reusable, continuously evolving infrastructure.The framework supports development of contrastive vision-language encoders and multimodal large language models.
- MedPMC framework: Its corpus spans pathology, radiology, ophthalmology, dermatology, and endoscopy, including slit-lamp, skin, gross pathology, endoscopy, angiography, PET/SPECT, and fMRI imagery.MedPMC is designed for sustainable expansion rather than one-time dataset construction, with modular processing stages.
- Evaluation and release: 26 public benchmarks spanning 11 specialties show MedPMC-CLIP achieves state-of-the-art performance, while clinical validation uses 10,524 Yale New Haven Health System dermatology images.The framework, curated corpus, benchmark resources, and pretrained model checkpoints are publicly released to support reuse and reproducibility.
2 Results
MedPMC applies a modular, five-stage pipeline to permissively licensed PMC literature, producing high-fidelity medical image-text data while filtering for clinically and translationally relevant visual evidence. Component evaluations and downstream experiments showed strong curation performance and improved multimodal medical question answering.
- Framework and dataset: 6.1M PMC articles were processed through a five-stage framework that yielded 11M medical image-text pairs.The pipeline transforms permissively licensed biomedical literature into curated data for multimodal model training.
- Framework and dataset: 7.3M medically relevant subfigure-subcaption pairs remained after decomposing 3.1M multi-panel figures into 29M subfigures and filtering non-medical content.The final classifier addresses mixtures of medical and non-medical elements within multi-panel figures.
- Component evaluation: mAP = 89.8 was achieved for figure separation, while caption separation and alignment reached F1 = 81.4 and ROUGE-L = 85.3, and medical figure classification reached F1 = 96.5.The caption model outperformed the PMC-OA pipeline and GPT-4T zero-shot on both reported caption metrics.
- Data quality: 4.7% of MedPMC images were categorized as non-medical, compared with 80.3% in BIOMEDICA.Sampled BIOMEDICA images included 42% statistical figures, graphs, and charts, whereas MedPMC was predominantly medical.
- Downstream model performance: 16.9 percentage points was the observed MedPMC-CLIP improvement on OmniMedVQA, compared with 1.9 percentage points on MMMU.The MMMU estimate had a 95% CI of −4.1 to 8.0 percentage points, while OmniMedVQA had a 95% CI of 14.9–18.7 percentage points.
- Clinical similarity: 0.239 was the median nearest-neighbor cosine distance between YNHHS clinical images and MedPMC in DINOv2 embedding space, lower than Fitzpatrick17k, SCIN, and DermNet.The corresponding distances were 0.262 for Fitzpatrick17k, 0.270 for SCIN, and 0.299 for DermNet.
3 Discussion
MedPMC is presented as reusable, updatable infrastructure that converts biomedical literature into medically relevant, panel-aligned supervision for medical multimodal models. The discussion emphasizes complementary clinical value, broad applications, and limitations inherited from literature-derived data.
- Data curation as reusable infrastructure for medical AI: MedPMC treats data curation as a core engineering problem, using a modular pipeline to address non-medical content, compound figures, weak alignment, and dataset staleness.The framework is designed as scalable infrastructure built from permissively licensed biomedical literature.
- Relationship to existing PMC-derived resources: 80% of sampled BIOMEDICA figures were non-medical, compared with 4.7% for MedPMC.MedPMC emphasizes fidelity, reproducibility, and updatability rather than scale alone.
- Relationship to existing PMC-derived resources: 81.4 F1 and 85.3 ROUGE-L were achieved by jointly separating captions and aligning subfigures with a supervised 4B-parameter MLLM.The model considers all panels together, avoiding approaches whose inference costs scale separately with panel count.
- Complementary strengths of MedPMC and clinical data: Literature-derived data complements rather than replaces clinical datasets by offering broader scale, context, and coverage across diseases, specialties, modalities, rare conditions, and emerging technologies.Clinical datasets remain closest to deployment because they capture patient populations, local protocols, workflows, and case mix.
- Complementary strengths of MedPMC and clinical data: MedPMC-CLIP transferred from biomedical publications to morphology-guided retrieval on internal patient dermatology data, supporting clinically motivated downstream use.The embedding-space analysis suggested broader overlap between MedPMC dermatology images and internal clinical dermatology images than several public dermatology datasets.
- Potential applications: MedPMC can support vision-language encoders, MLLM visual backbones, specialty-specific models, and filtered subsets for rare diseases, surgical imaging, pathology, skin photography, and specific modalities.Compound figures also provide supervision for related-image relationships such as progression, treatment comparisons, multi-view imaging, and radiology-pathology pairs.
- Limitations and Future Work: MedPMC remains limited by its image-text focus, variable captions, incomplete visual descriptions, publication selection bias, incomplete architectural coverage, and evaluations that do not establish deployment readiness.The authors frame it as broad pretraining supervision requiring additional clinical evaluation, adaptation, and clinician-in-the-loop work.
4 Methods
MedPMC trained specialized models for each curation stage, combining aligned existing datasets with newly constructed data where prior resources lacked scale, domain coverage, or representative PMC layouts. The framework used held-out component test sets and contamination-aware evaluations spanning public benchmarks, medical question answering, and independent clinical transfer.
- Pipeline training: MedPMC trained specialized models for each pipeline stage, augmenting existing datasets with newly constructed data when coverage or representativeness was insufficient.Dataset statistics for each component were reported separately.
- Initial screening: 39,360 sampled figures trained the initial screening classifier, which combined captions with inline article references and used PubMedBERT for binary medical-relevance classification.The model was fine-tuned with batch size 64 and learning rate 3e-5.
- Figure processing: 16,775 ImageCLEFmed samples plus 39,360 synthetic examples trained binary multi-panel detection, while YOLOv10 separated subfigures using the ImageCLEFmed separation dataset without synthetic annotations.Synthetic annotations did not further improve validation performance for separation.
- Validation and evaluation: Dedicated test sets were fully held out from development, and evaluation included architecture-matched public benchmarks, medical QA, and 10,524 independent Yale New Haven Health System dermatology images.The design also excluded four literature-curated OmniMedVQA subsets because publication overlap was possible, while acknowledging that provenance cannot guarantee image-level independence.
Author Contributions Statement
The study was conceived and designed by H.K. and Q.C., with teams contributing to dataset curation, pipeline development, annotation-interface configuration, annotation, and finalized-dataset evaluation.
- H.K. and Q.C. conceived and designed the study.
- H.K. and D.K. curated pretraining datasets, while H.K., D.K., P.X., and Y.Y. developed the data curation pipeline.
- H.K., X.A., Y.W., and Y.K. configured the manual annotation interface and measured annotator disagreement.
- Multiple contributors performed data annotation for pipeline evaluation, and P.X., S.S.A., R.J., Y.D., and J.M. manually evaluated the finalized dataset.
Data Availability
MedPMC releases its corpus, benchmarks, pretrained checkpoints, and metadata publicly through Hugging Face with license-aware records and versioned snapshots. The dermatology patient data used for clinical transfer evaluation remain restricted because of privacy, governance, and IRB requirements.
- Public resources: The MedPMC corpus, component benchmarks, pretrained checkpoints, and associated metadata are publicly available through its Hugging Face collection.Released records include source identifiers and license metadata for license-aware filtering, redistribution, and downstream model development.
- Versioning: Each MedPMC release is versioned by article cutoff date, source-license filters, component checkpoints, and processing configuration.This enables reproduction of specific corpus snapshots and future updates as new literature becomes available.
- Restricted clinical data: Yale New Haven Health System dermatology patient data are not publicly available because they contain patient-derived clinical data subject to privacy, governance, and IRB restrictions.Aggregate evaluation results and processing procedures are described in the manuscript.
MedPMC BIOMEDICA
This section presents manual image-distribution evaluation for MedPMC and BIOMEDICA, alongside an overview of 26 image-classification benchmarks.
- Image distribution analysis: Manual evaluation of randomly sampled MedPMC and BIOMEDICA images analyzed their distribution across categories.The five most frequent categories were highlighted.
- Benchmark overview: The study included 26 image-classification benchmarks.An overview of these benchmarks is provided in Extended Data Table 4.