Source-linked AI summary
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, Jimeng Sun
TL;DR
Medical vision-text pre-training lacks sufficient paired data and suffers false negatives when semantically similar unpaired reports are treated as negatives. MedCLIP decouples images and texts and applies medical-knowledge-guided semantic matching, outperforming prior methods across zero-shot prediction, supervised classification, and image-text retrieval while achieving strong data efficiency.
Problem
Medical vision-text pre-training has far fewer paired examples than general-domain CLIP and suffers false negatives because unpaired reports may share symptoms or findings.
Method
MedCLIP decouples images and texts to exploit unpaired datasets combinatorially and uses medical semantic similarity in a soft matching loss.
Results
MedCLIP outperforms baselines on zero-shot prediction, supervised classification, and image-text retrieval, with over 10% average ACC improvement and over 2% retrieval-precision improvement.
Takeaways & Limitations
MedCLIP achieves strong pre-training data efficiency, winning over the state-of-the-art baseline by 1% ACC with around 10× fewer data.
Takeaways & Limitations
Semantic supervision can fail when semantic tags are incorrect or negation and uncertainty phrases are missed.
Abstract
from arXiv · showhide
Existing vision-text contrastive learning like CLIP aims to match the paired image and caption embeddings while pushing others apart, which improves representation transferability and supports zero-shot prediction. However, medical image-text datasets are orders of magnitude below the general images and captions from the internet. Moreover, previous methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this paper, we decouple images and texts for multimodal contrastive learning thus scaling the usable training data in a combinatorial magnitude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using around 200K data). Our code is available at https://github.com/RyanWangZf/MedCLIP.
1 Introduction
Medical vision-text pre-training is constrained by scarce paired data and subtle clinical semantics, while existing methods also waste unpaired datasets and introduce false negatives. MedCLIP addresses these challenges by decoupling images and texts and using medical knowledge for semantic matching, achieving strong data efficiency and downstream performance.
- Motivation: Medical image-text datasets are orders of magnitude smaller than CLIP’s 400M general-domain image-text pairs, while clinical distinctions are more subtle and fine-grained.The paper highlights pneumonia versus consolidation as an example of fine-grained medical semantics.
- Limitations of prior work: Existing paired-data methods leave many medical image-only and text-only datasets unused.ConVIRT and GLoRIA require paired medical images and reports, limiting usable training data.
- Limitations of prior work: Unpaired reports can describe the same symptoms as an anchor image, so treating them as negatives introduces noisy supervision and confuses the model.This is the false-negative problem identified for prior contrastive methods.
- MedCLIP: MedCLIP decouples images and texts for contrastive learning, expanding training data in a combinatorial manner.The approach uses unpaired image and text datasets rather than restricting pre-training to paired samples.
- MedCLIP: MedCLIP uses medical semantic similarity as supervision through a soft semantic matching loss to eliminate false negatives.This knowledge-guided loss is intended to capture subtle yet crucial medical meanings.
- Results: Over 10% average prediction ACC improvement was reported for zero-shot prediction and supervised image classification, alongside over 2% higher retrieval precision.The evaluation covered four public datasets and compared MedCLIP with baselines.
2 Related Works
Prior vision-text representation and retrieval methods largely target general-domain paired data, whereas medical image-text methods remain constrained by paired datasets and false-negative noise. MedCLIP is positioned as addressing both limitations.
- General-domain methods: General-domain vision-text representation methods learn from paired images and captions, while many retrieval methods use attention, object detection, or contrastive learning.These approaches rely on abundant general-domain web images and captions.
- Medical vision-text learning: Existing medical image-text representation methods use paired images and texts, leaving them exposed to the medical domain’s limited-data challenge.The related-work discussion identifies paired-data dependence as a shared limitation.
- Medical vision-text learning: Medical contrastive methods also suffer false-negative noise when noise contrastive estimation performs instance discrimination.The paper links this noise to undermined representation quality.
3 Method
MedCLIP combines separate vision and text encoders with knowledge-driven semantic matching, allowing unpaired medical data to form many training pairs while weighting relationships by medical similarity.
- MedCLIP uses knowledge extraction, vision and text encoders, and a semantic matching loss as its three main components.
- Vision and text projection heads map their embeddings into the same dimension P for contrastive learning.
- Decoupling n paired samples yields (n+m)×(n+h) image-text pairs by combining separately sampled images and texts.Previous methods use only n paired samples; with 2 paired samples and 3 additional images and sentences, MedCLIP forms 25 pairs.
- The semantic matching loss uses medical similarity as soft supervision rather than treating all positive samples equally.Image-to-text and text-to-image soft targets are normalized with softmax, while predicted similarities come from cosine similarities between normalized embeddings.
- Medical entities and image labels are aligned through external knowledge to construct semantic similarities between arbitrary image-text combinations.Reports are split into sentences, entities are extracted with MetaMap, and image classes are mapped to UMLS concepts.
4 Experiments
The experiments evaluate MedCLIP on four X-ray datasets across zero-shot recognition, knowledge-driven supervision, fine-tuned classification, retrieval, and embedding quality.
- Four X-ray datasets are used to evaluate MedCLIP across five research questions.The questions cover zero-shot recognition, semantic matching supervision, fine-tuned classification and label efficiency, cross-modal retrieval, and embedding quality.
- CheXpert provides 14 observation labels from Stanford Hospital, with CheXpert-5x200 sampling 200 exclusively positive images for five evaluation tasks.
- MIMIC-CXR supplies chest X-rays with free-text radiology reports, and MIMIC-5x200 evaluates the same five tasks.
- COVID is evaluated as a binary X-ray classification dataset with an approximately 1:1 positive-to-negative ratio.
- RSNA Pneumonia is evaluated as a balanced binary classification dataset distinguishing pneumonia from normal cases.
4.2 Baselines
MedCLIP is compared with random, ImageNet, CLIP, ConVIRT, and GLoRIA baselines using specified vision and text encoders and medical datasets.
- Random and ImageNet are ResNet-50 baselines with random initialization or ImageNet-pretrained weights, respectively.
- CLIP is a vision-text contrastive framework pretrained on 400M internet image-text pairs.
- ConVIRT applies InfoNCE to paired X-rays and reports, while GLoRIA models image subregions and words through cross-attention.
- MedCLIP uses BioClinicalBERT and Swin Transformer backbones, with ResNet-50 included for ablation.
- The evaluation includes zero-shot classification, fine-tuned classification, and dataset statistics reported in Tables 1–3.
- Pretraining uses MIMIC-CXR and CheXpert, with held-out samples, 224 × 224 images, a 512-dimensional projection head, and augmentations.
4.4 Q1. Zero-Shot Classification
MedCLIP is evaluated for zero-shot classification across four datasets and outperforms the compared baselines, including under prompt ensembling and on an unseen COVID-19 class.
- Zero-shot classification evaluates CheXpert-5x200, MIMIC-5x200, COVID, and RSNA by matching image embeddings with disease-class prompt embeddings.
- MedCLIP outperforms all other baselines by a great margin in zero-shot image classification.
- MedCLIP benefits from prompt ensembling, whereas ensembling does not consistently help ConVIRT or GLoRIA.
- Original CLIP produces predictions essentially equivalent to random guessing across the medical datasets.
- MedCLIP exceeds 0.8 ACC on COVID despite no COVID-19-positive images appearing during pretraining.
4.5 Q2. Pre-training Data Efficiency
MedCLIP is tested for pretraining data efficiency by varying sample counts and evaluating zero-shot prediction on CheXpert-5x200.
- The experiment subsamples pretraining data at 20K, 50K, and 200K before measuring zero-shot prediction on CheXpert-5x200.
- The study frames data efficiency as a key challenge for CLIP-based methods because large-scale training is computationally expensive and difficult in medicine.
- 20K pretraining samples yield superior performance to GLoRIA trained on around 200K image-text pairs.
- MedCLIP also outperforms ConVIRT, which uses 369K training samples.
- Increasing training data improves MedCLIP accuracy, with no zero-shot ACC saturation observed at 570K samples.
4.7 Q4. Image-Text Retrieval
MedCLIP is evaluated for image-text retrieval using CheXpert-5x200 images and MIMIC-CXR sentences, and achieves the best reported performance across methods.
- The retrieval dataset contains 1,000 images and 1,000 sentences, with 200 images and sentences per each of five classes.
- Retrieval performance is measured with Precision@K by checking whether retrieved reports belong to the query image’s category.
- MedCLIP achieves the best performance across all compared methods in image-text retrieval.
- MedCLIP’s precision increases at higher K values in the retrieval evaluation.
- The representation analysis compares CheXpert-5x200 image embeddings from MedCLIP and CLIP using t-SNE.
- MedCLIP produces more clustered embeddings and detects lesion-type clusters, whereas CLIP’s visualization is homogeneous.
5 Conclusion
MedCLIP decouples medical image-text contrastive learning and incorporates medical knowledge to expand usable training data and alleviate false negatives. It achieves strong data efficiency and performance across zero-shot prediction, supervised classification, and image-text retrieval.
- MedCLIP decouples medical images and texts, expanding pretraining data with a combinatorial magnitude while using medical knowledge to alleviate false negatives.Its knowledge-guided semantic matching loss addresses cases where separate-patient images and reports share semantics.
- 1% ACC improvement over the state-of-the-art baseline is achieved with around 10× fewer pretraining data.
- MedCLIP performs strongly on zero-shot prediction, supervised classification, and image-text retrieval tasks.
- MedCLIP is expected to support a medical-domain foundational model and medical diagnosis for diverse diseases with low resource requirements.
Limitations
The paper identifies two limitations: noisy semantic similarity extraction and practical constraints in prompt-based zero-shot use. Suggested remedies include noisy-data learning and prompt-learning methods, while additional pretraining data is also desired.
- Incorrect semantic tags and missed negation or uncertainty phrases can introduce noise into the extracted semantic similarity matrix.The paper suggests learning-from-noisy-data techniques as a possible remedy.
- Prompt-based inference depends on prompt quality, and more pretraining data is desired to further enhance pretraining.Prompt-learning methods are suggested to automate downstream-task application instead of manual prompt engineering.
- Despite comparable zero-shot accuracy to a finetuned counterpart, MedCLIP is described as not yet amenable to practical use.
A Analysis of Image-text retrieval results
The retrieval analysis visualizes cosine-similarity distributions for same-class retrieved texts and relates their concentration to Precision@K. As K increases, the effective similarity threshold decreases, allowing more same-class texts to enter the result.
- The observed increase in Precision@K is associated with texts beginning to appear in the retrieval result under the changing threshold.
- CheXpert-5x200 images and MIMIC-CXR texts each contain 1000 rows, with 200 images and sentences or reports per class in CheXpert-5x200.
- The analysis plots histograms of cosine similarity for top-10 retrieved same-class texts across image classes.For each image, same-class sentences or reports among the top 10 retrieved texts are collected and binned by cosine similarity.
- As K increases in Precision@K, the cosine-similarity cutoff decreases, allowing more same-class texts to appear in the result.
- Figure 5 visualizes the similarity distributions computed from MedCLIP embeddings.