Source-linked AI summary
CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training
Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun Kyoung Hong, Woonhyunk Baek, Byungseok Roh
TL;DR
CXR-CLIP addresses the shortage of chest X-ray image-text data that limits medical vision-language pre-training. It expands image-label data with prompted text, uses multiple study images and report sections, and adds image- and text-specific contrastive losses. The model outperforms comparable state-of-the-art models, while enlarged pre-training data improves classification with a marginal retrieval trade-off.
Problem
Medical VLP lacks sufficient chest X-ray image-text datasets because many public datasets contain image-label pairs, while high-quality clinical annotation is costly and time-consuming.
Method
CXR-CLIP converts image-label pairs into prompted image-text pairs, exploits multiple images and report sections per study, and adds ICL and TCL for study-level image and text learning.
Results
CXR-CLIP outperforms comparable models trained with the same datasets, while adding prompted pre-training data improves classification performance but marginally reduces retrieval performance.
Takeaways & Limitations
Prompted image-label data and study-level contrastive learning provide a scalable way to strengthen CXR VLP across zero-shot, few-shot, classification, and retrieval evaluations.
Abstract
from arXiv · showhide
A large-scale image-text pair dataset has greatly contributed to the development of vision-language pre-training (VLP) models, which enable zero-shot or few-shot classification without costly annotation. However, in the medical domain, the scarcity of data remains a significant challenge for developing a powerful VLP model. In this paper, we tackle the lack of image-text data in chest X-ray by expanding image-label pair as image-text pair via general prompt and utilizing multiple images and multiple sections in a radiologic report. We also design two contrastive losses, named ICL and TCL, for learning study-level characteristics of medical images and reports, respectively. Our model outperforms the state-of-the-art models trained under the same conditions. Also, enlarged dataset improve the discriminative power of our pre-trained model for classification, while sacrificing marginal retrieval performance. Code is available at https://github.com/kakaobrain/cxr-clip.
1 Introduction
CXR-CLIP addresses scarce chest X-ray image-text data by converting image-label pairs into prompted text and exploiting multiple images and report sections per study. It adds study-level image and text contrastive losses and is evaluated across zero-shot and few-shot settings.
- The method combines image-text data with image-label data through class-specific prompts, making it applicable to any image-label dataset rather than depending on a rule-based labeler.
- CXR-CLIP uses Multi-View Supervision to exploit multiple images and report texts within a study, producing additional image-text pairs for learning.
- Image contrastive loss and text contrastive loss learn study-level characteristics from CXR images and reports, respectively.
- CXR-CLIP tackles scarce CXR image-text data by generating image-text pairs from image-label datasets with radiologist-designed prompts and multiple study images and texts.The approach is intended to expand VLP training data without relying on a rule-based labeler.
- The model is validated on diverse datasets under zero-shot and few-shot settings, addressing the annotation burden in CXR diagnosis.
2 Related Work
Related work improves CXR VLP efficiency through self-supervision and study-level information, while prior methods also leverage labels through rule-based labeling or prompting. CXR-CLIP combines these directions using multiple images and report sections from one study with dedicated contrastive losses.
- Prior CXR VLP methods use self-supervision for label-efficient learning, including report-word and image-subregion alignment and modality-specific pre-training.
- A CXR study can provide multiple image views plus findings and impression sections, motivating study-level matching instead of matching independently augmented examples.
- MedCLIP uses a rule-based labeler to integrate unpaired images, texts, and labels, but decoupling image-text pairs limits retrieval of the exact report for an image.
- CXR-CLIP extends prior work by prompting image-label data and using two images and two texts per study, together with ICL and TCL for efficient learning.
3 Method
CXR-CLIP expands scarce chest X-ray image-text supervision by pairing multiple study images and report sections, while adding modality-specific contrastive objectives to learn study-level representations.
- 3.1 Data Sampling: CXR-CLIP samples two images and two texts per study, using augmentation when only one example is available and distinct views when possible.Image-label data instead generates paired texts from class-specific prompts.
- 3.1 Data Sampling: Class-specific prompt templates convert image-label records into report-like text by combining sampled prompts conditioned on class names and label values.The prompts are designed to resemble clinical reports rather than randomly combining disease attributes.
- 3.2 Model Architecture: The model uses image and text encoders with projection layers that produce normalized visual and textual embeddings for contrastive learning.The image encoder experiments use ResNet-50 and Swin-Tiny, while the text encoder uses BioClinicalBERT.
- 3.3 Loss Function: MVS averages four CLIP losses across all pairings of the first and second image and text embeddings.The four pairings combine each image embedding set with each text embedding set.
- 3.3 Loss Function: ICL and TCL apply CLIP-style losses within modalities to learn study-level image and text characteristics, respectively.ICL contrasts images across studies, whereas TCL aligns findings and impression sections or prompt variants within studies.
4 Experiment
Experiments evaluate CXR-CLIP across classification and image-to-text retrieval, using multiple pre-training and external datasets. Prompted image-label data generally improves classification, while additional image-label data can reduce retrieval performance.
- 4.3 Comparison with State-of-the-arts: CXR-CLIP outperforms matched MedCLIP models on most classification metrics, and adding more pre-training datasets generally improves classification.The SwinTiny model trained on MIMIC-CXR and CheXpert outperforms MedCLIP with the same architecture and datasets; training with three datasets performs best for most metrics.
- 4.3 Comparison with State-of-the-arts: CXR-CLIP trained on MIMIC-CXR outperforms GloRIA on Open-I image-to-text retrieval, although GloRIA performs best on CheXpert.The comparison uses Open-I because GloRIA's CheXpert image-text pairs are not publicly available.
- 4.3 Comparison with State-of-the-arts: Adding CheXpert and ChestX-ray14 image-label data degrades image-to-text retrieval, possibly because original report text is diluted.Retrieval is measured by R@K, the recall of the exact report among the top K retrieved reports.
- 4.4 Ablations: Study-level sampling, CXR-specific augmentation, Multi-View Supervision, ICL, and TCL each improve components of classification or retrieval over vanilla CLIP.CXR-specific augmentation improves classification with similar retrieval performance, MVS slightly improves both tasks, and the additional losses exploit multi-view and multi-text inputs.
- 4.4 Ablations: TCL reduces recall on CheXpert5x200, possibly because radiologic-report variation is difficult to optimize and prompts are less diverse than images.This caveat appears within the ablation analysis of the additional contrastive losses.
- 4.4 Ablations: CXR-CLIP's study-based MVS and text augmentation outperform DeCLIP's augmented examples and EDA in image-to-text recall, while the full method outperforms DeCLIP.The comparison supports using multiple images and texts from one study and preserving clinical meaning during text augmentation.
5 Conclusion
The conclusion presents CXR-CLIP as a framework that enlarges image-text training data and learns study-level characteristics through additional modality-specific losses. Enlarged data improves classification, while retrieval performance can decline; ICL and TCL improve image-text retrieval.
- 5 Conclusion: CXR-CLIP enlarges training image-text pairs by prompting image-label datasets and using multiple images and report sections from each study.The framework is designed for zero-shot and few-shot classification settings and incorporates study-level information.
- 5 Conclusion: Enlarging the dataset improves classification performance but sacrifices retrieval performance.The conclusion states this trade-off across classification and retrieval tasks.
- 5 Conclusion: ICL and TCL enhance within-modality discrimination and effectively increase image-to-text retrieval performance.These losses are intended to learn CXR domain knowledge alongside image-text contrastive learning.
Supplementary Materials, CXR-CLIP: Toward
The supplementary materials compare prompt designs, classification performance, self-supervised baselines, and evaluation prompts for CXR-CLIP.
- Prompt comparisons: CXR-CLIP prompts further improved performance on CheXpert5x200 compared with both models without prompts and models using GloRIA prompts.The comparison also reports performance gains for a model not trained with prompts, supporting evaluation of the proposed prompts.
- Self-supervised comparisons: The self-supervised comparison evaluates REFERS and MRM under linear-probing and whole-backbone fine-tuning settings using ViT-base models trained on MIMIC-CXR.The supplied table description defines the compared methods, evaluation settings, backbone, and training dataset but does not provide cell values.
- Evaluation prompts: Zero-shot classification uses dataset-specific evaluation prompts: Jang et al. prompts for VinDR-CXR and SIIM, and BioVIL prompts for RSNA.These prompts are used to standardize evaluation across the listed datasets.
- Resolution and retrieval: An additional comparison trains 512-resolution ResNet models, denoted CXR-CLIP+ R50, and evaluates RSUM as recall@1, recall@5, and recall@10 summed together.The supplied passage states that MIMIC-trained models outperform BioVIL and CXR-CLIP+, but the remaining comparison text is truncated.
- Prompt construction: The supplementary materials define default positive and negative prompt templates, class-specific expressions, concatenation, random selection, and blank-text notation.The notation includes + for concatenation, [·] for random selection, and parentheses for blank text.