PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents

• Biomedical foundation-model development lags because biomedical data require expert annotation and raise privacy concerns.Source: §1 · p. 1
• The pipeline crawls figures and captions from PubMed Central, whose publications are expert-examined and whose figures are anonymized and de-identified.Source: §1 · pp. 1–2
• PMC-OA contains 1.65M image-text pairs generated through medical figure collection, subfigure separation, and subcaption separation and alignment.Source: §1 · p. 2
• PMC-CLIP is pretrained on PMC-OA with image-text contrastive loss and masked language modeling, then evaluated on retrieval, classification, and medical VQA.Source: §1 · p. 2
• PMC-OA is 8× larger than previous biomedical image-caption datasets and can be continuously updated through the automatic pipeline.Source: §1 · p. 2
• PMC-CLIP achieves state-of-the-art performance across various downstream biomedical tasks, surpassing previous methods significantly.Source: §1 · p. 2

Dataset Collection

• The PMC-OA collection procedure has three stages: medical figure collection, subfigure separation, and subcaption separation and alignment.Source: §2.1 · p. 3

Medical Figure Collection

• 2,478,267 papers yielded 12,211,907 figure-caption pairs before medical-figure filtering.Source: §2.1 · p. 3
• 381,096 medical figures remained after caption keyword filtering and scientific-figure classification.Source: §2.1 · p. 4

Subfigure Separation

• mAP@0.5 of 0.94 was achieved by the subfigure detector on its test set, with precision 0.93 and recall 0.94 at confidence threshold 0.7.Source: §2.1 · p. 4
• 1,646,592 subfigures were obtained from 378,717 compound figures, averaging 4.3 subfigures per caption before subcaption alignment.Source: §2.1 · p. 4

Subcaption Separation & Alignment

• 1,003,911 subfigure-subcaption pairs were finally aligned with 73% test alignment accuracy, alongside 642,681 subfigure-caption pairs.Source: §2.1 · p. 4

Dataset Overview

• PMC-OA covers common and rare diagnostic procedures and supports elaborate disease identification through free-form text.Source: §2.2 · p. 5
• PMC-OA is approximately gender-balanced, with 54% males, while its statistics indicate diagnostic-procedure diversity, disease coverage, and population fairness.Source: §2.2 · pp. 5–6

3.1 Architecture

• PMC-CLIP trains a biomedical vision-language model on paired images and captions using image and text encoders.The model encodes each image-caption pair separately with ResNet-based visual and BERT-based text encoders.Source: §3 · p. 6, §3.1 · p. 6

3.2 Training Objectives

• Image-text contrastive learning matches corresponding visual and text representations using cross-modality similarities across the batch.The loss combines image-to-text and text-to-image cross-entropy terms with one-hot matching labels.Source: §3.2 · pp. 6–7
• Masked language modeling reconstructs randomly masked text tokens from contextual content and visual cues.The method masks words with probability 15% and uses a fusion transformer to predict the masked token.Source: §3.2 · p. 7
• The final training objective is a weighted sum of image-text contrastive and masked-language-modeling losses.The MLM loss weight λ is set to 0.5 by default.Source: §3.2 · p. 7

Experiment scope

• The experiments cover dataset comparisons, downstream task descriptions, and implementation details.These settings are introduced as the basis for evaluating the model across multiple biomedical vision-language tasks.Source: §4 · p. 7

Compared datasets

• ROCO provides 81K PubMed image-caption samples, while MedICaT extends this to 217K pairs with 75% compound figures.MIMIC-CXR contains 377,110 image-report pairs describing findings from doctors.Source: §4.1 · p. 7

Image-text retrieval

• Retrieval is evaluated on 2,000 ROCO test-set image-text pairs after explicitly checking duplication with the training data.The evaluation follows prior work and resembles zero-shot evaluation.Source: §4.2 · p. 7

Classification

• MedMNIST supplies 12 image-classification tasks spanning biomedical modalities such as colon pathology, dermatoscopy, and retinal OCT.The model is finetuned for these downstream classification tasks.Source: §4.2 · p. 7

Visual question answering

• VQA evaluation uses the official splits of VQA-RAD and SLAKE, containing 315 and 642 images, respectively.Questions are categorized as close-ended or open-ended, and most pretrained model parameters are retained during adaptation.Source: §4.2 · p. 8

Metrics

• Accuracy and AUC measure classification, Recall@K with K=1,5,10 measures retrieval, and Accuracy measures VQA.These metrics define the reported evaluation outcomes for the three downstream tasks.Source: §4.2 · p. 8

Implementation details

• The implementation uses ResNet50, PubMedBERT, four fusion-transformer layers, 224 × 224 images, AdamW with lr = 1 × 10^-4, and batch size 128.Training runs for 100 epochs with the first 10 epochs used for warm-up on GeForce RTX 3090 GPUs.Source: §4.3 · p. 8

5.1 PMC-OA surpasses SOTA large-scale biomedical dataset

• PMC-OA substantially improves retrieval performance when used for PMC-CLIP pretraining instead of alternative datasets.The comparison evaluates retrieval on the ROCO test set.Source: §5.1 · p. 8, Table 1 · p. 9

5.2 PMC-CLIP achieves SOTA across downstream tasks

• 8.1% on I2T Rank@10: PMC-CLIP surpasses the previous state of the art on zero-shot ROCO image-text retrieval.The authors note that PMC-OA does not contain ROCO data, unlike some compared methods.Source: §5.2 · p. 9, Table 2 · p. 9
• PMC-CLIP obtains consistently higher results on the reported MedMNIST classification sub-tests than compared methods.The paper presents three of twelve sub-tests and reports gains over training from scratch with ResNet.Source: §5.2 · p. 9, Table 3 · p. 9
• PMC-CLIP surpasses M3AE in 5 out of 6 reported VQA results.The evaluation covers VQA-RAD and SLAKE.Source: §5.2 · p. 10, Table 4 · p. 10

5.3 Ablation Study

• ITC contributes most of the pretraining performance, while MLM with visual features produces the best results in the ablation study.Text-only MLM acts as a regularization term, whereas visual cues support finer-grained image-caption correlations.Source: §5.3 · p. 10, Table 5 · p. 10

• PMC-OA collects image-caption pairs from abundant scientific documents in the biomedical domain.Source: §6 · p. 10
• PMC-CLIP is a CLIP-style model trained on PMC-OA.Source: §6 · p. 10
• PMC-CLIP achieves state-of-the-art performance across image-text retrieval, image classification, and visual question answering.Source: §6 · p. 10