Source-linked AI summary
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P. Lungren, Tristan Naumann, Sheng Wang, Hoifung Poon
TL;DR
Biomedical vision-language pretraining is constrained by limited, private, and insufficiently diverse image-text datasets. The paper introduces PMC-15M and BiomedCLIP, a domain-adapted CLIP model trained on this corpus. BiomedCLIP achieves state-of-the-art results across biomedical tasks, including outperforming radiology-specific models on radiology evaluation.
Problem
Existing biomedical vision-language datasets are often private, small, and insufficiently diverse, limiting accessible large-scale pretraining data.
Method
The paper extracts 15 million scientific image-text pairs into public PMC-15M and pretrains BiomedCLIP with biomedical adaptations including PubMedBERT.
Results
BiomedCLIP establishes state-of-the-art performance across retrieval, image classification, and visual question answering, including outperforming BioViL on RSNA.
Takeaways & Limitations
Large-scale, diverse biomedical pretraining supports a fully open-access foundation model that performs across diverse biomedical image tasks.
Takeaways & Limitations
PMC-15M omits in-line figure references and contains many composite figures that could be split for finer-grained modeling.
Abstract
from arXiv · showhide
Biomedical data is inherently multimodal, comprising physical measurements and natural language narratives. A generalist biomedical AI model needs to simultaneously process different modalities of data, including text and images. Therefore, training an effective generalist biomedical model requires high-quality multimodal data, such as parallel image-text pairs. Here, we present PMC-15M, a novel dataset that is two orders of magnitude larger than existing biomedical multimodal datasets such as MIMIC-CXR, and spans a diverse range of biomedical image types. PMC-15M contains 15 million biomedical image-text pairs collected from 4.4 million scientific articles. Based on PMC-15M, we have pretrained BiomedCLIP, a multimodal foundation model, with domain-specific adaptations tailored to biomedical vision-language processing. We conducted extensive experiments and ablation studies on standard biomedical imaging tasks from retrieval to classification to visual question-answering (VQA). BiomedCLIP achieved new state-of-the-art results in a wide range of standard datasets, substantially outperforming prior approaches. Intriguingly, by large-scale pretraining on diverse biomedical image types, BiomedCLIP even outperforms state-of-the-art radiology-specific models such as BioViL in radiology-specific tasks such as RSNA pneumonia detection. In summary, BiomedCLIP is a fully open-access foundation model that achieves state-of-the-art performance on various biomedical tasks, paving the way for transformative multimodal biomedical discovery and applications. We release our models at https://aka.ms/biomedclip to facilitate future research in multimodal biomedical AI.
Introduction
Biomedical vision-language models need large, high-quality image-text data, but existing biomedical datasets are often private, small, and insufficiently diverse. PMC-15M addresses these limitations with 15 million public pairs spanning thirty biomedical image types.
- Motivation: Biomedical data combines physical measurements with natural-language narratives, motivating multimodal models that jointly process images and text.Multimodal learning can leverage cross-modal correspondence and fusion to reduce curation bottlenecks and uncover signals latent in individual modalities.
- Limitations of prior data: Existing biomedical image-text datasets are limited by privacy restrictions, small scale, and inadequate diversity.Reported datasets range from 7k to 377k pairs, including MIMIC-CXR, CheXpert, ARCH, and ROCO.
- PMC-15M: 15 million image-text pairs make PMC-15M two orders of magnitude larger than prior large biomedical datasets such as MIMIC-CXR.The dataset is collected from public scientific papers and avoids privacy issues associated with private biomedical data.
- PMC-15M: PMC-15M spans thirty major biomedical image types, providing broad coverage for biomedical research and clinical practice.Its scientific-publication origin supports open accessibility while its breadth targets diverse biomedical applications.
- BiomedCLIP: BiomedCLIP is pretrained on PMC-15M and evaluated across retrieval, zero-shot classification, and medical visual question answering.The model uses biomedical domain adaptations and substantially outperforms prior approaches across these applications.
Results
The paper constructs PMC-15M from PubMed Central figures and captions, adapts CLIP for biomedical data, and evaluates it across retrieval, classification, and VQA. BiomedCLIP shows strong retrieval and classification performance, improves radiology and VQA results, and can retrieve proxies for proprietary data.
- Dataset: PMC-15M contains 15,282,336 image-caption pairs extracted from 4.4 million publicly available PubMed Central articles.The extraction process matches figure files with captions and records article provenance identifiers.
- Dataset: Biomedical images and captions are larger than standard CLIP defaults, motivating specialized image resolutions and longer text contexts.The paper also creates PMC-Fine-Grained-46M by splitting scientific figures into individual panels to probe image-class diversity.
- Modeling: BiomedCLIP adapts CLIP with PubMedBERT, a biomedical tokenizer, expanded context size, and biomedical image processing.CLIP aligns paired image and text embeddings using cosine similarity and an InfoNCE loss.
- Cross-modal retrieval: BiomedCLIP’s top-5 retrieval contains the correct result over 77% of the time among more than 700 thousand candidates.Its top-1 retrieval accuracy exceeds 56%, while PubMedCLIP performs worse than general-domain CLIP in the reported comparison.
- Image classification: BiomedCLIP outperforms radiology-specific BioViL on RSNA classification and exceeds fully supervised BioViL using only 10% of labeled data.The authors attribute this result to large, diverse pretraining rather than greater radiology-specific image volume.
- Visual question answering: BiomedCLIP improves over PubMedCLIP on both VQ-RAD and SLAKE, with about one-point and six-point increases in overall accuracy, respectively.It achieves comparable performance to Med-PaLM M and BiomedGPT-B, while LLaVA-Med records the highest VQA performance on both datasets.
- Visual question answering: BiomedCLIP correctly answers two challenging VQA examples that prior state-of-the-art models failed, while identifying the relevant organ in a third.The authors connect these gains to the large number of parallel image-caption pairs in PMC-15M.
- Privacy-preserving analysis: BiomedCLIP retrieves PMC-15M examples that agree substantially better with proprietary radiology labels than PLIP and CLIP on most reported measures.On lung opacity and atelectasis, the method reaches recall values of 88.80 and 95.04, respectively, but performs worse than CLIP on Cardiomegaly.
Discussion
BiomedCLIP builds on biomedical vision-language pretraining by scaling beyond CXR-focused datasets to diverse image types and establishes strong performance across biomedical tasks, while the authors identify several avenues for improvement.
- Discussion: 15 million image-caption pairs make this the largest reported biomedical vision-language pretraining study, spanning an extremely diverse range of biomedical images.
- Discussion: BiomedCLIP establishes new state-of-the-art performance on cross-modal retrieval, image classification, and visual question answering across eight standard biomedical datasets.
- Discussion: The authors report that large-scale pretraining on extremely diverse data is effective for biomedical vision-language processing.
- Discussion: Prior biomedical vision-language work largely focuses on chest X-rays with limited training data, whereas this study evaluates broad biomedical image coverage.
- Discussion: Current limitations include excluding in-line references, retaining composite figures, and not yet using PMC-Fine-Grained-46M to enhance pretraining.
Methods
The study constructs and evaluates biomedical vision-language data and models using scientific articles, embedding-based retrieval, broad image-classification benchmarks, and multimodal VQA frameworks.
- Data processing: PMC articles are parsed to extract figure captions, figure references, and article provenance into structured records.
- Data processing: PMC-Fine-Grained-46M refines PMC-15M by splitting compound figures into sub-figures and adding in-line article references as image-text sources.
- Retrieval: Retrieval uses pretrained CLIP vision and text encoders, cosine similarity, and approximate nearest-neighbor search to rank captions for figure embeddings.
- Baselines: The study compares BiomedCLIP with general-domain, radiology-specific, pathology, multimodal, and instruction-tuned biomedical models.
- Biomedical image classification: Image classification uses ELEVATER across zero-shot, few-shot, and full-shot settings, including PatchCamelyon, LC25000, TCGA-TIL, and RSNA.
- Medical Visual Question Answering: VQA uses METER for transformer-based co-attention fusion and evaluates VQA-RAD and SLAKE with accuracy and token-level F1.
- Clinical image retrieval: A proprietary chest-X-ray retrieval analysis compares BiomedCLIP, PLIP, and CLIP using label agreement measured by F1 and Recall.
Data availability
The authors plan to provide scripts for reproducing PMC-15M and PMC-Fine-Grained-46M from PubMed Central Open-Access data.
- Scripts for reproducing PMC-15M and PMC-Fine-Grained-46M from PMC-OA will be available at the BiomedCLIP release URL.
Code availability
The BiomedCLIP release includes model weights, source code, and implementation guidance for independent replication.
- BiomedCLIP will provide model weights, pretraining, fine-tuning, and inference code, together with detailed replication steps.
Task Dataset Metric Description Data size
The study evaluates biomedical vision-language models across retrieval, classification, and visual question answering, using task-specific metrics and dataset-size conventions. Figures also compare privacy-preserving analysis based on retrieved public biomedical data.
- Data size: Dataset-size entries use image-caption-pair counts for retrieval and image counts for image classification and VQA.This convention is specified for train, development, and test splits.
- Cross-modal retrieval: Cross-modal retrieval evaluates text-to-image and image-to-text performance using Recall@1, Recall@5, and Recall@10.BiomedCLIP and alternative text encoders are compared, alongside qualitative top-four retrieval examples.
- Image classification: Image classification reports AUROC for TCGA-TIL and accuracy for the other standard datasets, with both zero-shot and linear-probing evaluations.The classification comparisons include five zero-shot datasets and fine-tuned PCam and RSNA results.
- Medical VQA: Medical VQA reports accuracy and token-level F1 on VQA-RAD and SLAKE under open-ended and closed-ended settings.The figure also presents challenging VQA-RAD examples comparing model answers.
- Privacy-preserving analysis: Privacy-preserving analysis compares CheXbert-derived labels from proprietary Providence radiology data with labels from the most similar PMC-15M data using recall and F1.The comparison includes retrieved image examples and uses captions and clinical reports to assess similarity.
Supplementary Note
The supplementary studies examine CLIP’s biomedical adaptations across text encoders, vision transformers, image sizes, batch schedules, and initialization choices. They identify PubMedBERT, larger ViT models, 224-pixel inputs, and a 4k batch size as the selected configuration for BiomedCLIP.
- CLIP objective: CLIP learns a shared image-text embedding space by increasing matched-pair cosine similarity and decreasing non-pair similarity through symmetric InfoNCE loss.The loss uses both image-to-text and text-to-image matching within batches.
- Text encoder: Biomedical adaptation replaces GPT-2 and BPE with PubMedBERT and WordPiece to better represent biomedical text.The text-side ablation evaluates these domain-specific changes on PMC-15M validation data.
- Vision encoder: Larger ViT models improve validation performance, leading the study to use ViT-B/16 in subsequent experiments.The comparison covers ViT-Small, ViT-Medium, and ViT-Base with PubMedBERT and a 256-token context.
- Vision initialization: ImageNet initialization offers more stable downstream performance despite no validation advantage over random initialization.The selected vision encoder is ViT-B/16 with ImageNet-pretrained weights.
- Image size: Using 384-pixel images improves validation classification over 224 pixels but can reduce downstream performance, especially on PCam.Upsampling PCam’s 96×96 images may introduce noise, motivating the choice of 224-pixel inputs.
- Batch size: Increasing batch size improves validation performance, but gains do not transfer to downstream evaluation after batch size 4k.The study therefore trains BiomedCLIP with a 4k batch size rather than increasing it to 64k.
- Final configuration: The final configuration combines PMC-15M pretraining with PubMedBERT, a larger vision transformer, and higher image resolution for generally stronger validation performance.The supplementary comparison reports that large-scale or continual PMC-15M pretraining is consistently helpful.
- Implementation: The implementation uses OpenCLIP with distributed contrastive training, gradient checkpointing, mixed precision, and sharded contrastive loss.Experiments use up to 16 NVIDIA A100 or V100 GPUs, while sharding reduces redundant similarity computations.