CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training

• The method combines image-text data with image-label data through class-specific prompts, making it applicable to any image-label dataset rather than depending on a rule-based labeler.Source: §1 · pp. 1–2
• CXR-CLIP uses Multi-View Supervision to exploit multiple images and report texts within a study, producing additional image-text pairs for learning.Source: §1 · p. 2
• Image contrastive loss and text contrastive loss learn study-level characteristics from CXR images and reports, respectively.Source: §1 · p. 2
• CXR-CLIP tackles scarce CXR image-text data by generating image-text pairs from image-label datasets with radiologist-designed prompts and multiple study images and texts.The approach is intended to expand VLP training data without relying on a rule-based labeler.Source: §1 · pp. 1–2
• The model is validated on diverse datasets under zero-shot and few-shot settings, addressing the annotation burden in CXR diagnosis.Source: §1 · pp. 1–2

• Prior CXR VLP methods use self-supervision for label-efficient learning, including report-word and image-subregion alignment and modality-specific pre-training.Source: §2 · p. 2
• A CXR study can provide multiple image views plus findings and impression sections, motivating study-level matching instead of matching independently augmented examples.Source: §2 · p. 2
• MedCLIP uses a rule-based labeler to integrate unpaired images, texts, and labels, but decoupling image-text pairs limits retrieval of the exact report for an image.Source: §2 · p. 2
• CXR-CLIP extends prior work by prompting image-label data and using two images and two texts per study, together with ICL and TCL for efficient learning.Source: §2 · p. 2

3.1 Data Sampling

• CXR-CLIP samples two images and two texts per study, using augmentation when only one example is available and distinct views when possible.Image-label data instead generates paired texts from class-specific prompts.Source: §3 · p. 2, Figure 1 · p. 3, §3.1 · p. 4
• Class-specific prompt templates convert image-label records into report-like text by combining sampled prompts conditioned on class names and label values.The prompts are designed to resemble clinical reports rather than randomly combining disease attributes.Source: §3.1 · p. 3

3.2 Model Architecture

• The model uses image and text encoders with projection layers that produce normalized visual and textual embeddings for contrastive learning.The image encoder experiments use ResNet-50 and Swin-Tiny, while the text encoder uses BioClinicalBERT.Source: §3.2 · p. 4

3.3 Loss Function

• MVS averages four CLIP losses across all pairings of the first and second image and text embeddings.The four pairings combine each image embedding set with each text embedding set.Source: §3.3 · pp. 4–5
• ICL and TCL apply CLIP-style losses within modalities to learn study-level image and text characteristics, respectively.ICL contrasts images across studies, whereas TCL aligns findings and impression sections or prompt variants within studies.Source: §3.3 · p. 5

4.3 Comparison with State-of-the-arts

• CXR-CLIP outperforms matched MedCLIP models on most classification metrics, and adding more pre-training datasets generally improves classification.The SwinTiny model trained on MIMIC-CXR and CheXpert outperforms MedCLIP with the same architecture and datasets; training with three datasets performs best for most metrics.Source: §4.3 · pp. 6–7, Table 2 · p. 7
• CXR-CLIP trained on MIMIC-CXR outperforms GloRIA on Open-I image-to-text retrieval, although GloRIA performs best on CheXpert.The comparison uses Open-I because GloRIA's CheXpert image-text pairs are not publicly available.Source: §4.1 · p. 6, §4.3 · p. 7, Table 3 · p. 7
• Adding CheXpert and ChestX-ray14 image-label data degrades image-to-text retrieval, possibly because original report text is diluted.Retrieval is measured by R@K, the recall of the exact report among the top K retrieved reports.Source: §4.3 · pp. 7–8, Table 3 · p. 7

4.4 Ablations

• Study-level sampling, CXR-specific augmentation, Multi-View Supervision, ICL, and TCL each improve components of classification or retrieval over vanilla CLIP.CXR-specific augmentation improves classification with similar retrieval performance, MVS slightly improves both tasks, and the additional losses exploit multi-view and multi-text inputs.Source: §4.4 · p. 8, Table 4 · p. 8
• TCL reduces recall on CheXpert5x200, possibly because radiologic-report variation is difficult to optimize and prompts are less diverse than images.This caveat appears within the ablation analysis of the additional contrastive losses.Source: §4.4 · p. 8
• CXR-CLIP's study-based MVS and text augmentation outperform DeCLIP's augmented examples and EDA in image-to-text recall, while the full method outperforms DeCLIP.The comparison supports using multiple images and texts from one study and preserving clinical meaning during text augmentation.Source: §4.4 · pp. 8–9, Table 4 · p. 8

5 Conclusion

• CXR-CLIP enlarges training image-text pairs by prompting image-label datasets and using multiple images and report sections from each study.The framework is designed for zero-shot and few-shot classification settings and incorporates study-level information.Source: §5 · p. 9
• Enlarging the dataset improves classification performance but sacrifices retrieval performance.The conclusion states this trade-off across classification and retrieval tasks.Source: §5 · p. 9
• ICL and TCL enhance within-modality discrimination and effectively increase image-to-text retrieval performance.These losses are intended to learn CXR domain knowledge alongside image-text contrastive learning.Source: §5 · p. 9

Prompt comparisons

• CXR-CLIP prompts further improved performance on CheXpert5x200 compared with both models without prompts and models using GloRIA prompts.The comparison also reports performance gains for a model not trained with prompts, supporting evaluation of the proposed prompts.Source: Table 1 · p. 12

Self-supervised comparisons

• The self-supervised comparison evaluates REFERS and MRM under linear-probing and whole-backbone fine-tuning settings using ViT-base models trained on MIMIC-CXR.The supplied table description defines the compared methods, evaluation settings, backbone, and training dataset but does not provide cell values.Source: Table 2 · p. 12

Evaluation prompts

• Zero-shot classification uses dataset-specific evaluation prompts: Jang et al. prompts for VinDR-CXR and SIIM, and BioVIL prompts for RSNA.These prompts are used to standardize evaluation across the listed datasets.Source: Table 3 · p. 12

Resolution and retrieval

• An additional comparison trains 512-resolution ResNet models, denoted CXR-CLIP+ R50, and evaluates RSUM as recall@1, recall@5, and recall@10 summed together.The supplied passage states that MIMIC-trained models outperform BioVIL and CXR-CLIP+, but the remaining comparison text is truncated.Source: Table 4 · p. 13

Prompt construction

• The supplementary materials define default positive and negative prompt templates, class-specific expressions, concatenation, random selection, and blank-text notation.The notation includes + for concatenation, [·] for random selection, and parentheses for blank text.Source: Table 5 · p. 13