PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP in Medical Imaging: A Survey

Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang, Dinggang Shen

arXiv:2312.07353v6cs.CV

TL;DR

Vision models trained only on visual annotations can remain poorly aligned with human cognition, motivating interest in text supervision for medical imaging. This survey synthesizes CLIP pre-training adaptations and clinical applications, finding that prompts can convey semantics across components and be modified for different patient groups.

  • Problem

    Vision models trained only on vision annotations and tasks can have unsatisfactory out-of-distribution performance associated with limited alignment to human cognition.

    Source: §1 · p. 1

  • Method

    The survey reviews CLIP fundamentals, medical adaptations to image-report pre-training, clinical applications, and limitations, organizing studies into refined pre-training and CLIP-driven applications.

    Source: §1 · pp. 2–3

  • Results

    Descriptive text prompts demonstrate CLIP’s ability to comprehend prompt semantics and convey them to other framework components.

    Source: §4.4 · p. 17

  • Takeaways & Limitations

    The survey says CLIP-driven applications can adapt to different patient groups by modifying the input prompt’s specific content.

    Source: §4.4 · p. 17

    CLIP models may exhibit gender or racial biases that affect the fairness of predictions.

    Source: §6 · p. 20

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP), a simple yet effective pre-training paradigm, successfully introduces text supervision to vision models. It has shown promising results across various tasks due to its generalizability and interpretability. The use of CLIP has recently gained increasing interest in the medical imaging domain, serving as a pre-training paradigm for image-text alignment, or a critical component in diverse clinical tasks. With the aim of facilitating a deeper understanding of this promising direction, this survey offers an in-depth exploration of the CLIP within the domain of medical imaging, regarding both refined CLIP pre-training and CLIP-driven applications. In this paper, we (1) first start with a brief introduction to the fundamentals of CLIP methodology; (2) then investigate the adaptation of CLIP pre-training in the medical imaging domain, focusing on how to optimize CLIP given characteristics of medical images and reports; (3) further explore practical utilization of CLIP pre-trained models in various tasks, including classification, dense prediction, and cross-modal tasks; and (4) finally discuss existing limitations of CLIP in the context of medical imaging, and propose forward-looking directions to address the demands of medical imaging domain. Studies featuring technical and practical value are both investigated. We expect this survey will provide researchers with a holistic understanding of the CLIP paradigm and its potential implications. The project page of this survey can also be found on https://github.com/zhaozh10/Awesome-CLIP-in-Medical-Imaging.

1. Introduction

Vision-only models remain poorly aligned with human cognition, while clinical reports offer expert knowledge without the labor of additional annotations. This survey reviews CLIP in medical imaging, organizing work on adapted pre-training and clinical applications.

1. Introduction

  • Routine clinical reports offer expert-level knowledge at little additional cost, whereas collecting boxes or segmentation masks is laborious and time-consuming.The paper links CLIP’s text supervision to interpretability demands in healthcare.Source: §1 · p. 2
  • The review includes 224 papers, with X-ray-based studies occupying the majority of selected papers.The selection excludes studies using CLIP solely as a baseline, papers outside medical imaging, and studies primarily about diffusion models.Source: §1 · p. 2
  • The survey organizes medical-imaging CLIP research into refined pre-training and CLIP-driven applications.The former adapts web image-caption pre-training to medical images and reports; the latter uses pre-trained models in clinical tasks.Source: §1 · p. 2
  • Unlike Shrestha et al. (2023) and Azad et al. (2023), this survey focuses exclusively on CLIP and extensively covers clinical applications.The authors distinguish their coverage of clinical tasks from the other surveys’ emphases on broader pre-training architectures or foundational models.Source: §1 · p. 3
  • The authors describe this as the first review of CLIP in medical imaging and aim to summarize the rapidly growing field.They report reviewing more than 200 methods.Source: §1 · pp. 2–3
  • The survey provides a multi-level taxonomy, discusses open questions, and proposes future research directions.Its stated aim is to serve readers with different needs.Source: §1 · p. 3

2. Background

CLIP aligns image and text representations through joint contrastive pre-training, supporting zero-shot recognition through text prompts. The survey also describes CLIP variants and publicly available medical image-text datasets.

2.1. Contrastive Language-Image Pre-training

  • CLIP jointly encodes image-text batches into a shared latent space and uses contrastive pre-training to align paired representations.Its vision encoder may use ResNet or ViT, while its language encoder is transformer-based.Source: §2.1 · pp. 3–4
  • Zero-shot domain identification compares medical-image features with text embeddings generated from class-name prompts.The survey illustrates this capability with classes including Chest X-ray, Mammography, Knee X-ray, and Dental X-ray, while noting further validation is needed.Source: §2.1 · p. 4

2.2. Variants of CLIP

  • GLIP aligns image regions with text phrases, while CLIPSeg and CRIS extend CLIP to prompt-guided segmentation.CLIPSeg fixes the pre-trained encoders and adds a trainable decoder; these variants could help locate subtle findings such as masses or calcifications.Source: §2.2 · pp. 4–5

2.3. Medical image-text dataset

  • Public medical image-text datasets include research-paper figure-caption collections, clinical-report datasets, and social-media-derived histology pairs.The datasets span modalities and languages; combining datasets across language communities may introduce language bias.Source: §2.3 · p. 5

3. CLIP in medical image-text pre-training

This section surveys approaches to adapting CLIP pre-training to medical images and reports, addressing domain-specific challenges through multi-scale alignment, data efficiency, and explicit knowledge integration. It also frames these approaches as ways to exploit consistency in paired or related data.

3.1. Challenges of CLIP pre-training

  • Medical images and reports require local as well as global alignment; limited paired data constrains training, while specialized clinical concepts motivate explicit knowledge to improve robustness.These challenges make directly applying standard CLIP pre-training impractical for medical image-text datasets.Source: §3.1 · p. 6

3.2. Multi-scale contrast

  • GLoRIA aligns words with semantically weighted image regions, localizing findings, but its one-way, equal-weight objective may miss image-to-text alignment and sentence importance.Its semantic-affinity maps correctly localize regions associated with findings such as pneumonia and pneumothorax.Source: §3.2 · pp. 6–7
  • LoVT adds bidirectional sentence-to-region alignment and transformer-learned local features and weights; extensions add conditional reconstruction, prototype memory, or raw-report reconstruction.Other approaches align local image features with sentences, while volumetric studies use anatomy-level contrast, including fVLM’s use of TotalSegmentator.Source: §3.2 · pp. 7–8

3.3. Data-efficient contrast

  • Correlation-driven methods soften or replace one-hot targets using report or label correlations, mine false negatives, align samples sharing diseases, or combine modality-wise MRI alignment.These strategies address semantically overlapping reports and can use unpaired reports as useful training signals.Source: §3.3 · p. 8
  • Data mining enriches alignment with report Impression sections, shuffled sentences, and uncertainty prompts; video methods exploit sequential structure and hierarchical clip-, phase-, and video-level alignment.Video-text data have been built from online lectures using acoustic speech recognition.Source: §3.3 · pp. 8–9
  • Liu et al. (2023c) reported viable CLIP pre-training using real radiology reports paired with synthetic chest radiographs.Source: §3.3 · p. 10

3.4. Explicit knowledge enhancement

  • Explicit knowledge enhancement represents report-level entities and relations at the subject level and domain concepts as knowledge graphs or prompts, encoded by graph neural networks or pre-trained BERT models.UMLS concepts are commonly extracted and linked using ScispaCy; relations from UMLS or RadGraph can capture causal, positional, and modifying information.Source: §3.4 · pp. 10–11
  • ARL uses subject-specific UMLS entity-relation graphs, while KoBo and FLAIR add domain knowledge through radiology concepts or fundus descriptions.ARL uses TransE to train a graph attention network; KoBo uses CompGCN to mitigate negative-sample noise and semantic shifting, and FLAIR uses ClinicalBERT to interpret refined fundus knowledge.Source: §3.4 · p. 11
  • MedKLIP, KAD, and MOTOR combine subject-level and domain-level knowledge, and their improved performance supports explicit knowledge enhancement in CLIP-style pre-training.MedKLIP uses the 75 most frequent entities for its domain-level graph; the methods use ClinicalBERT or SciBERT as knowledge encoders.Source: §3.4 · p. 11

3.5. Others

  • M-FLAG addresses collapse by freezing the text encoder and applying an orthogonality loss to encourage orthogonal visual representations.Collapse here means image and text information are encoded as constant feature embeddings.Source: §3.5 · p. 11
  • Other studies pair synthetic chest x-rays with real reports or align pathology images with omics tabular data, extending CLIP beyond diagnostic-report matching.The survey describes the synthetic-image work as an initial attempt and identifies CMTA and PathOmics as exploring omics-pathology alignment.Source: §3.5 · p. 11
  • Turgut et al. (2025) use CLIP-style pre-training to align electrocardiograms with cardiac MRI, aiming to make cardiovascular diagnosis more accessible through low-cost ECGs.The passage contrasts ECGs' low price and limited diagnostic capability with the greater diagnostic information and expense of cardiac MRI.Source: §3.5 · p. 11

3.6. Summary

  • The three adaptation categories share a goal of exploiting consistency: multi-scale methods align local image and text features, data-efficient methods use inter-sample correlations, and knowledge-enhanced methods incorporate expert relationships.The survey presents these categories as contributing distinct capabilities to medical imaging.Source: §3.6 · p. 11

4. CLIP-driven applications

CLIP's alignment of visual features with human language supports interpretability in clinical settings and can provide external supervision for annotation-demanding tasks. The survey includes classification studies among its CLIP-driven applications.

4. CLIP-driven applications

  • CLIP can provide external supervision for annotation-demanding tasks such as tumor segmentation.Its alignment of visual features with human language is also significant in clinical settings where interpretability matters.Source: §4 · p. 11
  • Table 3 overviews representative classification applications of CLIP.Source: Table 3 · p. 12
  • Figure 10 illustrates positive and negative prompt engineering for zero-shot Pneumothorax diagnosis, a prompt approach also applicable to other diseases.Source: Figure 10 · p. 12

4.1. Classification

CLIP supports medical image classification through zero-shot classification and context optimization. Zero-shot approaches use domain-specific models and prompts, while context optimization adapts non-domain-specific CLIP efficiently to medical imaging.

4.1. Classification

  • Medical image classification uses zero-shot classification with prompt engineering or parameter- and data-efficient context optimization.Classification generally requires a global image assessment, such as identifying a disease or judging benignity versus malignancy.Source: §4.1 · p. 12

4.1.1. Zero-shot classification

  • Zero-shot diagnosis uses positive and negative prompts to assess diseases independently, while Xplainer estimates diagnoses from radiologist-refined descriptive observations.Standard prompts can fall short because softmax treats classes as mutually exclusive; Xplainer's qualitative errors can include contradictory findings for radiologists to detect and correct.Source: §4.1.1 · pp. 12–13

4.1.2. Context optimization

  • Channel-wise conditional prompts adapt CLIP-Lung to lung nodule malignancy prediction and improve results.The learnable prompts use channel-level feature-map information, unlike CoCoOp; broader zero-shot diagnosis is constrained by limited availability of domain-specific CLIP models.Source: §4.1.2 · p. 13

4.2. Dense prediction

CLIP is used across medical dense-prediction tasks, from detection and 2D segmentation to 3D segmentation and weakly supervised localization. The reviewed methods use text representations to convey semantic information that image-only task encodings may not capture.

4.2. Dense prediction

  • Dense prediction produces outputs for pixels or pixel subsets rather than one label per image, and CLIP variants serve as auxiliary tools to inform clinical decisions.Examples of this support include target-organ and potential-lesion regions.Source: §4.2 · p. 13

4.2.3. 3D medical image segmentation

  • CLIP text embeddings replace DoDNet’s one-hot task labels, encoding organ–tumor relationships and helping detect small tumors missed by image-only segmentation models.The review reports stronger performance on organ and tumor segmentation than image-only methods.Source: Figure 13 · p. 14, §4.2.3 · p. 15

4.2.1. Detection

  • Image-only medical detectors can struggle with subtle cues; reviewed CLIP/GLIP approaches include prompt ensembles and zero-shot nuclei detection with automated prompts and iterative box self-training.AnomalyCLIP instead uses object-agnostic prompts for normality and abnormality to focus recognition on abnormal regions.Source: §4.2.1 · pp. 14–15

4.2.2. 2D medical image segmentation

  • In 2D segmentation, studies fine-tune CLIP image encoders across modalities, while others combine its pre-trained image and text encoders in vision-language models.The reviewed examples include X-rays, ultrasound, and CT/MR represented as 2D slices.Source: §4.2.2 · p. 15

4.2.3. 3D medical image segmentation

  • Continual-learning segmentation uses text prompts to represent relationships between current and previously learned tasks, helping filter and retain task-specific information.The review reports superior performance for this approach.Source: §4.2.3 · p. 15

4.2.4. Others

  • Text-prompted weak supervision combines MedCLIP label embeddings with BioBERT descriptions to improve pseudo-label quality and outperform other CAM-based methods.This addresses CAM’s tendency to focus on distinctive object regions and produce low-quality pseudo-labels near ambiguous boundaries.Source: §4.2.4 · pp. 15–16
  • TCEIP integrates instructional text embeddings such as “left,” “middle,” and “right” into keypoint regression, improving localization over image-only methods in difficult cases with missing or sparse teeth.Source: §4.2.4 · p. 16

4.3. Cross-modal tasks

The survey reviews CLIP applications that connect medical images and text through generation, visual question answering, and retrieval. Across these tasks, approaches use CLIP’s image-text representations for report or image generation, answer generation, and retrieval augmentation.

4.3. Cross-modal tasks

  • Cross-modal tasks involve interaction between image and text modalities.Source: §4.3 · p. 16

4.3.1. Generation

  • CLIP-based report generation uses semantic-aware image representations with learnable concept embeddings or reformulates reporting as zero-shot classification over prompts for findings and locations.Source: §4.3.1 · p. 16
  • Text-guided MRI synthesis uses CLIP text embeddings to condition customized sequence generation, while another approach proposes metadata-conditioned diffusion for compressed-sensing MRI.The metadata prompts can include age, gender, and MRI acquisition parameters.Source: §4.3.1 · p. 16

4.3.2. Medical visual question answering

  • Closed-ended MedVQA studies often use only CLIP’s image encoder, and their preset answer options make the task resemble classification, constraining practical utility.Source: §4.3.2 · pp. 16–17
  • Open-ended MedVQA combines CLIP image and question encoders with a language decoder to generate answers without predefined options.A mapping layer is commonly used to mitigate the domain gap between CLIP pre-training data and the MedVQA dataset.Source: §4.3.2 · p. 17

4.3.3. Image-text retrieval

  • Global-feature retrieval can miss subtle diagnostic details; a CLIP retrieval framework uses supervised content alignment for localized markers and retrieval augmentation with multi-head attention.The reviewed framework substantially outperformed existing retrieval methods on disease classification and report retrieval.Source: §4.3.3 · p. 17

4.4. Summary

The reviewed CLIP-driven applications show performance improvements and use descriptive prompts to convey semantics to other model components. The survey notes that changing prompt content can adapt applications to patient groups without retraining or fine-tuning.

4.4. Summary

  • Descriptive prompts convey semantics to other model components, and changing their content can adapt CLIP-driven applications to different patient groups without retraining or fine-tuning.The survey gives regional or age-related disease differences as motivating contexts for such adaptation.Source: §4.4 · p. 17

5. Comparative analysis

The comparison weighs CLIP against other vision-language models and evaluates pre-trained CLIP models on medical-domain identification. BiomedCLIP performs best in that evaluation, while domain-specific models reduce disease-similarity issues and CLIP offers lower resource needs and broader task flexibility than MLLMs.

Strengths of CLIP over other vision-language models

  • CLIP uses fewer computational resources than MLLMs and can support single-modality or multimodal tasks through independent encoders.MLLMs’ tightly connected image encoder and language decoder limit single-modality use and downstream feature embeddings; MLLMs offer better interactivity, which CLIP lacks.Source: §5 · p. 18, Table 6 · p. 18

Comparison between different pre-trained CLIP models

  • The domain-identification evaluation progressively adds domains and reports accuracy at each step for four pre-trained models.The compared models are BiomedCLIP, PubMedCLIP, CLIP, and PMC-CLIP.Source: §5 · p. 18
  • 98.9% accuracy for BiomedCLIP across all nine domains exceeds PubMedCLIP’s 93.11%; original CLIP reaches 86.3%.BiomedCLIP maintained 100% accuracy as domains increased from three to seven.Source: §5 · pp. 18–19, Figure 16 · p. 18

Comparison between non-domain-specific and domain-specific pre-training

  • BiomedCLIP and BioViL markedly mitigate the high inter-disease similarity seen with OpenAI CLIP; BioViL performs best for chest X-rays.The survey recommends organ-specific pre-trained CLIP models for applications requiring fine-grained disease discrimination.Source: Figure 17 · p. 19, §5 · p. 19

Comparison between different pre-trained CLIP models

  • PMC-CLIP performs poorly on domain identification, even below natural-image CLIP, indicating limited zero-shot capability.Source: §5 · p. 19

6. Discussions and future directions

The discussion identifies gaps in application alignment, evaluation, and model coverage, alongside challenges involving bias, robustness, metadata, and semantic relationships. It proposes directions including domain-specific pre-training, broader encoder evaluation, synthetic volumetric data, gaze-based alignment, and multimodal extensions.

Inconsistency between pre-training and application

  • 76.67% of surveyed CLIP-driven applications rely on OpenAI CLIP, while domain-specific models reduce inter-disease similarity, motivating organ-specific model selection.BioViL performs best for chest X-rays; the survey recommends BiomedCLIP over OpenAI CLIP even for context optimization.Source: §6 · p. 19

Incomprehensive evaluation of refined pre-training

  • Refined pre-training evaluations favor vision tasks and can overlook text encoders, although image-text alignment depends on both encoders.The survey encourages testing report generation, phrase grounding, and VQA; BioViL and BioViL-T are noted for relatively comprehensive encoder evaluation.Source: §6 · p. 19

Challenges of volumetric imaging

  • Volumetric imaging faces high compute demands, limited paired datasets, and difficult sentence-to-volume correspondence.CT and MRI often require GPUs with over 40GB VRAM; CT-RATE contains 50,000 scans. Suggested directions include synthetic data and gaze-linked report regions for multi-scale alignment.Source: §6 · p. 20

Limited scope of refined CLIP pre-training

  • Domain-specific CLIP pre-training is concentrated on chest X-rays, with limited work on mammography, knee MRI, and histology attributed mainly to scarce public datasets.Source: §6 · p. 20

Debiasing in CLIP Models

  • CLIP biases can reflect or amplify training-data and model-design biases, including demographic stereotyping and skew from disease or group imbalances.The survey notes possible consequences including misdiagnosis or unequal care and calls for medical-domain debiasing research.Source: §6 · p. 20

Enhancing adversarial robustness of CLIP

  • Adversarial perturbations can degrade CLIP performance, and PLIP has shown susceptibility to adversarial noise despite histology-specific pre-training.Source: §6 · p. 20

Exploring the potential of metadata

  • Patient metadata such as age may correlate with visual morphology; incorporating it into prompts is proposed for brain-development studies.The discussion points to infancy-related changes in brain morphology and tissue contrast and describes CLIP text embeddings as a semantically rich alternative to direct MLP encoding.Source: §6 · p. 21, Figure 18 · p. 21

Incorporation of high-order correlations; Beyond image-text alignment

  • Medical image-text pre-training could move beyond orthogonal alignment by modeling high-order inter-sample correlations and aligning additional modalities or same-modality images.The discussion says semantic overlap among medical samples may degrade performance under conventional paired alignment; other imaging modalities include X-rays, MRI, and CT.Source: §6 · p. 21

7. Conclusion

The survey reviews refined CLIP pre-training and CLIP-driven applications in medical imaging, organizes domain-specific pre-training challenges, and discusses open issues and future directions. It presents pre-trained CLIP models as potential supplementary supervision across tasks.

Conclusion

  • The survey’s first review of CLIP in medical imaging covers refined pre-training, diverse applications, a structured taxonomy of pre-training challenges, and future directions.It also compares CLIP approaches with vision-only methods and describes pre-trained CLIP as potential supplementary supervision across tasks.Source: §7 · p. 21

Related papers

Loading 2312.07353v6…