Source-linked AI summary
PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, Weidi Xie
TL;DR
Biomedical foundation-model development lags because expert annotation and privacy constraints limit data availability. This paper constructs PMC-OA from PubMed Central figures and captions, pretrains PMC-CLIP, and reports state-of-the-art performance across biomedical retrieval, classification, and VQA tasks, including an 8.1% I2T Rank@10 improvement.
Problem
Biomedical foundation-model development lags behind because data availability is limited by annotation expertise requirements and privacy concerns.
Method
The paper automatically constructs PMC-OA from PubMed Central figures and captions with subfigure-subcaption alignment, then pretrains PMC-CLIP using image-text contrastive and masked language-modeling objectives.
Results
PMC-CLIP achieves state-of-the-art performance across biomedical image-text retrieval, image classification, and VQA, outperforming prior image-text retrieval methods by 8.1% on I2T Rank@10.
Takeaways & Limitations
PMC-OA is an automatically constructed biomedical image-caption dataset 8× larger than previous datasets, and its pipeline can be continuously updated.
Abstract
from arXiv · showhide
Foundation models trained on large-scale dataset gain a recent surge in CV and NLP. In contrast, development in biomedical domain lags far behind due to data scarcity. To address this issue, we build and release PMC-OA, a biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral's OpenAccess subset, which is 8 times larger than before. PMC-OA covers diverse modalities or diseases, with majority of the image-caption samples aligned at finer-grained level, i.e., subfigure and subcaption. While pretraining a CLIP-style model on PMC-OA, our model named PMC-CLIP achieves state-of-the-art results on various downstream tasks, including image-text retrieval on ROCO, MedMNIST image classification, Medical VQA, i.e. +8.1% R@10 on image-text retrieval, +3.9% accuracy on image classification.
1 Introduction
Biomedical foundation-model development is constrained by limited annotated and privacy-safe data. This paper introduces PMC-OA, an automatically constructed large-scale image-caption dataset, and PMC-CLIP, a biomedical vision-language model evaluated across downstream tasks.
- Biomedical foundation-model development lags because biomedical data require expert annotation and raise privacy concerns.
- The pipeline crawls figures and captions from PubMed Central, whose publications are expert-examined and whose figures are anonymized and de-identified.
- PMC-OA contains 1.65M image-text pairs generated through medical figure collection, subfigure separation, and subcaption separation and alignment.
- PMC-CLIP is pretrained on PMC-OA with image-text contrastive loss and masked language modeling, then evaluated on retrieval, classification, and medical VQA.
- PMC-OA is 8× larger than previous biomedical image-caption datasets and can be continuously updated through the automatic pipeline.
- PMC-CLIP achieves state-of-the-art performance across various downstream biomedical tasks, surpassing previous methods significantly.
2 The PMC-OA Dataset
PMC-OA is constructed from PubMed Central through a three-stage pipeline that separates figures and captions into finer-grained image-text pairs. The resulting dataset is large, diverse across diagnostic procedures and diseases, and approximately gender-balanced.
- Dataset Collection: The PMC-OA collection procedure has three stages: medical figure collection, subfigure separation, and subcaption separation and alignment.
- Medical Figure Collection: 2,478,267 papers yielded 12,211,907 figure-caption pairs before medical-figure filtering.
- Medical Figure Collection: 381,096 medical figures remained after caption keyword filtering and scientific-figure classification.
- Subfigure Separation: mAP@0.5 of 0.94 was achieved by the subfigure detector on its test set, with precision 0.93 and recall 0.94 at confidence threshold 0.7.
- Subfigure Separation: 1,646,592 subfigures were obtained from 378,717 compound figures, averaging 4.3 subfigures per caption before subcaption alignment.
- Subcaption Separation & Alignment: 1,003,911 subfigure-subcaption pairs were finally aligned with 73% test alignment accuracy, alongside 642,681 subfigure-caption pairs.
- Dataset Overview: PMC-OA covers common and rare diagnostic procedures and supports elaborate disease identification through free-form text.
- Dataset Overview: PMC-OA is approximately gender-balanced, with 54% males, while its statistics indicate diagnostic-procedure diversity, disease coverage, and population fairness.
3 Visual-language Pre-training
PMC-CLIP is a biomedical vision-language model trained with image-text contrastive learning and masked language modeling, combining both objectives through a weighted total loss.
- 3.1 Architecture: PMC-CLIP trains a biomedical vision-language model on paired images and captions using image and text encoders.The model encodes each image-caption pair separately with ResNet-based visual and BERT-based text encoders.
- 3.2 Training Objectives: Image-text contrastive learning matches corresponding visual and text representations using cross-modality similarities across the batch.The loss combines image-to-text and text-to-image cross-entropy terms with one-hot matching labels.
- 3.2 Training Objectives: Masked language modeling reconstructs randomly masked text tokens from contextual content and visual cues.The method masks words with probability 15% and uses a fusion transformer to predict the masked token.
- 3.2 Training Objectives: The final training objective is a weighted sum of image-text contrastive and masked-language-modeling losses.The MLM loss weight λ is set to 0.5 by default.
4 Experiment Settings
The experiments compare biomedical datasets and evaluate PMC-CLIP on retrieval, classification, and visual question answering using task-specific datasets, metrics, and training settings.
- Experiment scope: The experiments cover dataset comparisons, downstream task descriptions, and implementation details.These settings are introduced as the basis for evaluating the model across multiple biomedical vision-language tasks.
- Compared datasets: ROCO provides 81K PubMed image-caption samples, while MedICaT extends this to 217K pairs with 75% compound figures.MIMIC-CXR contains 377,110 image-report pairs describing findings from doctors.
- Image-text retrieval: Retrieval is evaluated on 2,000 ROCO test-set image-text pairs after explicitly checking duplication with the training data.The evaluation follows prior work and resembles zero-shot evaluation.
- Classification: MedMNIST supplies 12 image-classification tasks spanning biomedical modalities such as colon pathology, dermatoscopy, and retinal OCT.The model is finetuned for these downstream classification tasks.
- Visual question answering: VQA evaluation uses the official splits of VQA-RAD and SLAKE, containing 315 and 642 images, respectively.Questions are categorized as close-ended or open-ended, and most pretrained model parameters are retained during adaptation.
- Metrics: Accuracy and AUC measure classification, Recall@K with K=1,5,10 measures retrieval, and Accuracy measures VQA.These metrics define the reported evaluation outcomes for the three downstream tasks.
- Implementation details: The implementation uses ResNet50, PubMedBERT, four fusion-transformer layers, 224 × 224 images, AdamW with lr = 1 × 10^-4, and batch size 128.Training runs for 100 epochs with the first 10 epochs used for warm-up on GeForce RTX 3090 GPUs.
5 Result
Experiments evaluate PMC-OA and PMC-CLIP across retrieval, classification, VQA, and pretraining-objective ablations. Results report improvements over prior datasets and methods, with visual-text objectives producing the strongest ablation outcome.
- 5.1 PMC-OA surpasses SOTA large-scale biomedical dataset: PMC-OA substantially improves retrieval performance when used for PMC-CLIP pretraining instead of alternative datasets.The comparison evaluates retrieval on the ROCO test set.
- 5.2 PMC-CLIP achieves SOTA across downstream tasks: 8.1% on I2T Rank@10: PMC-CLIP surpasses the previous state of the art on zero-shot ROCO image-text retrieval.The authors note that PMC-OA does not contain ROCO data, unlike some compared methods.
- 5.2 PMC-CLIP achieves SOTA across downstream tasks: PMC-CLIP obtains consistently higher results on the reported MedMNIST classification sub-tests than compared methods.The paper presents three of twelve sub-tests and reports gains over training from scratch with ResNet.
- 5.2 PMC-CLIP achieves SOTA across downstream tasks: PMC-CLIP surpasses M3AE in 5 out of 6 reported VQA results.The evaluation covers VQA-RAD and SLAKE.
- 5.3 Ablation Study: ITC contributes most of the pretraining performance, while MLM with visual features produces the best results in the ablation study.Text-only MLM acts as a regularization term, whereas visual cues support finer-grained image-caption correlations.
6 Conclusion
The paper presents PMC-OA, a large-scale biomedical image-caption dataset, and trains PMC-CLIP on it. PMC-CLIP achieves state-of-the-art performance across image-text retrieval, image classification, and visual question answering.
- PMC-OA collects image-caption pairs from abundant scientific documents in the biomedical domain.
- PMC-CLIP is a CLIP-style model trained on PMC-OA.
- PMC-CLIP achieves state-of-the-art performance across image-text retrieval, image classification, and visual question answering.