Source-linked AI summary
Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?
Sedigheh Eslami, Gerard de Melo, Christoph Meinel
TL;DR
The paper asks whether CLIP’s benefits transfer to medical visual question answering despite limited and uneven medical data. It fine-tunes CLIP on PubMed image–caption pairs and evaluates it in two MedVQA architectures across two benchmarks, finding improvements over prior visual encoders and dataset-dependent encoder behavior.
Problem
The study addresses limited and uneven evidence about whether CLIP benefits MedVQA as much as general-domain VQA.
Method
PubMedCLIP is fine-tuned on diverse PubMed image–caption pairs and substituted for visual encoders in MEVF and QCR evaluated on VQA-RAD and SLAKE.
Results
PubMedCLIP improves MedVQA performance, with gains of up to 3% over previously used pre-trained visual encoders.
Takeaways & Limitations
The results indicate that language-supervised visual representation learning can improve MedVQA and that benchmark data distributions affect which visual encoder performs best.
Takeaways & Limitations
The paper identifies the need for further explainable-AI analysis and evaluation of PubMedCLIP in additional medical use cases.
Abstract
from arXiv · showhide
Contrastive Language--Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image--text pairs collected online. Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems. This work evaluates the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA). To this end, we present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles. Our experiments are conducted on two MedVQA benchmark datasets and investigate two MedVQA methods, MEVF (Mixture of Enhanced Visual Features) and QCR (Question answering via Conditional Reasoning). For each of these, we assess the merits of visual representation learning using PubMedCLIP, the original CLIP, and state-of-the-art MAML (Model-Agnostic Meta-Learning) networks pre-trained only on visual data. We open source the code for our MedVQA pipeline and pre-training PubMedCLIP. CLIP and PubMedCLIP achieve improvements in comparison to MAML's visual encoder. PubMedCLIP achieves the best results with gains in the overall accuracy of up to 3%. Individual examples illustrate the strengths of PubMedCLIP in comparison to the previously widely used MAML networks. Visual representation learning with language supervision in PubMedCLIP leads to noticeable improvements for MedVQA. Our experiments reveal distributional differences in the two MedVQA benchmark datasets that have not been imparted in previous work and cause different back-end visual encoders in PubMedCLIP to exhibit different behavior on these datasets. Moreover, we witness fundamental performance differences of VQA in general versus medical domains.
1 BACKGROUND AND SIGNIFICANCE
MedVQA requires systems to connect medical-image content with natural-language questions, but limited and unrealistic datasets complicate this goal. This work investigates whether medical-domain CLIP fine-tuning improves MedVQA and introduces PubMedCLIP for that purpose.
- MedVQA systems must interpret images, questions, and associations between them to produce answers.
- Limited and imbalanced medical training data make MedVQA particularly challenging.
- Some benchmark questions lack semantic diversity, allowing systems to treat MedVQA as image classification without interpreting the question.
- Existing MedVQA approaches commonly combine visual and question encoders with attention-based multimodal fusion and answer classification.
- The study investigates whether fine-tuned CLIP benefits medical VQA as much as general-domain VQA and introduces PubMedCLIP using diverse medical image–text data.
2 MATERIALS AND METHODS
The study fine-tunes CLIP on diverse PubMed image–caption pairs and substitutes PubMedCLIP for the visual encoders in two MedVQA architectures. It evaluates the resulting systems on VQA-RAD and SLAKE under matched experimental setups.
- 2.1 PubMedCLIP: PubMedCLIP is fine-tuned from CLIP using more than 80K ROCO medical image–text pairs from PubMed articles.
- 2.1 PubMedCLIP: ROCO spans modalities including ultrasound, X-ray, PET, mammography, MRI, and angiography across diverse body regions.
- 2.1 PubMedCLIP: PubMedCLIP encodes images and captions separately, computes cosine similarity, and averages vision and language cross-entropy losses.
- 2.2 PubMedCLIP in MedVQA: The MedVQA pipeline replaces MEVF’s pre-trained MAML visual module with PubMedCLIP while retaining the original question encodings, fusion, and objectives.
- Datasets: Experiments use VQA-RAD and the English subset of SLAKE, whose test-image overlap with training differs across datasets.
- Experimental setup: The study follows prior MEVF and QCR setups, using Adam optimization and separate training schedules and hyperparameters for the two methods.
3 RESULTS
CLIP and PubMedCLIP improve both MedVQA methods, with PubMedCLIP providing additional gains over original CLIP and dataset-dependent best back-end encoders. Improvements occur across overall, open-end, and closed-end accuracy.
- Both MEVF and QCR improve when using CLIP or PubMedCLIP as the pre-trained visual encoder.
- VQA-RAD: PubMedCLIP improves on original CLIP by up to 1% and improves MEVF and QCR by up to 6% and 3% on VQA-RAD.
- SLAKE: On SLAKE, PubMedCLIP improves MEVF by up to 3% and QCR by up to 2%.
- The best PubMedCLIP back end differs by dataset: ResNet-50 for VQA-RAD and ViT32 for SLAKE.
- The improvement trend is consistent across overall, open-end, and closed-end accuracy scores.
4 DISCUSSION
The datasets favor different visual representations: VQA-RAD’s abnormality questions benefit from localizing CNN features, whereas SLAKE’s organ questions favor holistic Vision Transformer representations. Examples further show that PubMedCLIP improves QCR’s answers, although all models still fail on some VQA-RAD cases.
- Dataset-dependent visual encoders: VQA-RAD primarily asks about abnormalities, so ResNet-50’s stronger visual localization outperforms Vision Transformer representations.SLAKE mainly asks which organ is present, requiring holistic image understanding and long-range patch dependencies that favor Vision Transformers.
- Dataset-dependent visual encoders: SLAKE’s organ-focused questions favor Vision Transformers because they require holistic understanding of image content and long-range dependencies.
- Qualitative examples: Examples from VQA-RAD and SLAKE show MEVF often predicts answers about the wrong organ, while QCR’s answers are generally relevant but not always correct.The examples contrast MEVF’s organ confusion with QCR’s more relevant image-question interpretation.
- Qualitative examples: Using QCR with PubMedCLIP produces correct answers throughout the examples shown in Figures 3 and 4.
- Failure cases: All three models fail on selected VQA-RAD examples, including questions involving a metastatic focus and the phrase “periphery of the image.”The examples suggest difficulty with question semantics and specific visual-language interpretations, despite some answers remaining relevant to the image.
- CLIP across domains: Unlike general-domain VQA, where larger CLIP backends reportedly yield bigger gains, PubMedCLIP’s ResNet-50x4 backend appears to overfit medical images and underperform ResNet-50.The paper attributes the differing encoder behavior across medical datasets mainly to their underlying question distributions.
5 CONCLUSION
The work introduces PubMedCLIP, a PubMed-trained visual encoder for MedVQA, and reports improved performance over previously used encoders. It also identifies benchmark-distribution differences and points to future analysis and applications.
- PubMedCLIP is trained on image–caption pairs from thousands of PubMed articles for medical visual question answering.
- PubMedCLIP outperforms previously used pre-trained visual encoders on two MedVQA benchmarks by up to 3%.
- The experiments reveal differences in the underlying data distributions of the two MedVQA benchmark datasets.
- Future work includes explainable-AI analysis and investigating PubMedCLIP for medical image classification and radiology report generation.