PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision

Bobby Azad, Reza Azad, Sania Eskandari, Afshin Bozorgpour, Amirhossein Kazerouni, Islem Rekik, Dorit Merhof

arXiv:2310.18689v1cs.CV

TL;DR

Task-specific deep-learning models often rely on extensive labeled data, while foundation models can be adapted to downstream tasks with limited labels. This survey organizes medical-imaging foundation models by training strategy and related dimensions, synthesizes their applications, and discusses challenges including bias, adversarial attacks, and adaptation costs.

  • Problem

    Conventional visual-recognition models rely heavily on large-scale, task-specific, crowd-labeled data, motivating alternatives that can transfer to downstream tasks with fewer labels.

    Source: §1 · p. 2

  • Method

    The survey reviews medical-imaging foundation models and classifies them by prompting type, algorithm, organ, and imaging modality, alongside their applications and limitations.

    Source: §1 · p. 3, §6 · p. 21

  • Results

    The survey synthesizes distinct strengths across model types: contrastive textually prompted models show efficiency and performance enhancements, while generative approaches report applications in radiograph diagnosis, medical question answering, and medical VQA.

    Source: §3.1.1 · p. 9, §3.1.2 · p. 9

  • Takeaways & Limitations

    Textually prompted models suit interpretation and detailed explanations, visually prompted models suit segmentation and image-specific tasks, and conversational models support collaborative decision-making.

    Source: §4 · p. 17

    Foundation-model training requires substantial resources, and efficient downstream adaptation remains challenging because it can require careful hyperparameter selection that affects generalization.

    Source: §5.5 · p. 20

Abstract

from arXiv · show

Foundation models, large-scale, pre-trained deep-learning models adapted to a wide range of downstream tasks have gained significant interest lately in various deep-learning problems undergoing a paradigm shift with the rise of these models. Trained on large-scale dataset to bridge the gap between different modalities, foundation models facilitate contextual reasoning, generalization, and prompt capabilities at test time. The predictions of these models can be adjusted for new tasks by augmenting the model input with task-specific hints called prompts without requiring extensive labeled data and retraining. Capitalizing on the advances in computer vision, medical imaging has also marked a growing interest in these models. To assist researchers in navigating this direction, this survey intends to provide a comprehensive overview of foundation models in the domain of medical imaging. Specifically, we initiate our exploration by providing an exposition of the fundamental concepts forming the basis of foundation models. Subsequently, we offer a methodical taxonomy of foundation models within the medical domain, proposing a classification system primarily structured around training strategies, while also incorporating additional facets such as application domains, imaging modalities, specific organs of interest, and the algorithms integral to these models. Furthermore, we emphasize the practical use case of some selected approaches and then discuss the opportunities, applications, and future directions of these large-scale pre-trained models, for analyzing medical images. In the same vein, we address the prevailing challenges and research pathways associated with foundational models in medical imaging. These encompass the areas of interpretability, data management, computational requirements, and the nuanced issue of contextual comprehension.

1 Introduction

Foundation models are pretrained at scale and adapted for downstream tasks, offering an alternative to task-specific models that rely on extensive labeled data. This survey reviews medical-imaging foundation models through a structured taxonomy and discusses their applications, challenges, and future directions.

1.1 Clinical Importance

  • Medical foundation models are presented as a potential response to single-task biomedical AI, since medical practice spans imaging, text, records, laboratory findings, and genomics.The survey notes that such models could support a unified biomedical AI system for complex multimodal data.Source: §1 · p. 2, §1.1 · p. 3

1 Introduction

  • The survey reviews foundation-model fundamentals, organizes medical-imaging work by applications, organs, modalities, and algorithms, and examines methods’ strengths, limitations, challenges, and future directions.It also aims to provide a roadmap for further research.Source: §1 · pp. 2–3

1.2 Relevant Surveys

  • Compared with prior reviews, this survey classifies medical foundation models by prompting type and further organizes studies by algorithm, organ, and imaging modality.It also covers background concepts, applications, and open challenges.Source: §1.2 · p. 5

1.3 Search Strategy

  • The authors searched DBLP, Google Scholar, and Arxiv Sanity Preserver using tailored queries, then filtered out false positives to retain foundation-model papers.The search spanned peer-reviewed work, workshops, non-peer-reviewed content, and preprints.Source: §1.3 · p. 5

1.4 Paper Organization.

  • The survey groups studies into textually prompted and visually prompted models, with textually prompted models further divided into four types and visual models into adaptations and generalist models.It also identifies Section 5 as the discussion of risks, open problems, and future directions.Source: §1.4 · p. 5

2 Preliminaries

The preliminaries introduce foundation models and outline how pretraining objectives, tasks, instruction alignment, and prompting shape their capabilities. They distinguish contrastive learning’s paired-representation matching from generative approaches that reconstruct or produce data.

2 Preliminaries

  • Foundation models are described as large-scale models trained with self- or semi-supervision and adapted to multiple downstream tasks; the section reviews their architectures, objectives, instruction alignment, inference, and prompting.Source: §2 · p. 6

2.1 Pre-training Objectives

  • Pretraining objectives are broadly grouped into contrastive and generative approaches for learning vision-language relationships.Source: §2.1 · p. 6

2.1.1 Contrastive Objectives

  • Contrastive objectives bring related samples together and push unrelated ones apart; image-text contrastive loss applies this to matched and mismatched image-text pairs, with image-to-text and text-to-image terms combined.The temperature parameter τ is part of the image-text loss formulation.Source: §2.1.1 · p. 6

2.1.2 Generative Objectives

  • Generative objectives learn semantic features through image or text generation, including masked-image reconstruction and masked-language reconstruction.Other listed objectives include masked multimodal modeling, image-conditioned masked language modeling, and captioning with parallel prediction.Source: §2.1.2 · pp. 6–7

2.2 Pre-training Tasks

  • Vision pretraining tasks include inpainting, auxiliary supervised discriminative tasks, and data reconstruction to support generalization across downstream vision domains.Source: §2.2 · p. 7

2.3 Instruction-Aligning

  • Instruction alignment trains language models to follow human intent through human-feedback tuning, supervised instruction tuning, or reasoning prompts that elicit intermediate steps.Source: §2.3 · p. 7

2.4 Prompt Engineering

  • Prompt engineering adds task-specific hints to tailor a pretrained model without updating its parameters; approaches include hard prompts and learnable soft prompts.Hard prompts include instructions, in-context learning, retrieval-based prompting, and chain-of-thought; soft prompts include prompt tuning and prefix-token tuning.Source: §2.4 · p. 7

3 Foundational Models for Medical Imaging

The survey organizes medical-imaging foundation methods around their distinct training strategies and extends the taxonomy with additional information for each subcategory.

3 Foundational Models for Medical Imaging

  • The taxonomy follows standard practice but distinguishes itself by providing extensive additional information for each subcategory.Source: §3 · p. 7
  • The survey’s taxonomy distinguishes foundational methods by the training strategies designed to address medical-imaging challenges.Source: §3 · p. 8
  • The section introduces foundational methods developed to tackle diverse challenges in medical-image analysis.Source: §3 · p. 8

3.1 Textually Prompted Models

Textually prompted medical models use contrastive, generative, hybrid, and conversational strategies across medical imaging and related tasks. The surveyed examples include zero-shot classification, medical image generation, and conversational image analysis.

3.1.1 Contrastive

  • Contrastive textually prompted models align medical image and text representations, supporting classification, segmentation, and retrieval with dual-encoder or fusion designs.The survey also identifies lesion detection, disease classification, and image synthesis as potential applications.Source: §3.1.1 · p. 8
  • 70.2% average median zero-shot accuracy was achieved by MI-Zero across three real-world cancer subtyping tasks.It scores WSI patches against text prompts, then pools patch scores into slide-level predictions; Figure 4 depicts this pipeline.Source: §3.1.1 · p. 8, Figure 4 · p. 8
  • BioViL-T used temporal image-text structure for state-of-the-art progression classification, phrase grounding, and report generation, while CheXzero matched radiologists on pathology classification.CheXzero also outperformed fully supervised models on certain pathologies and adapted to unannotated pathologies.Source: §3.1.1 · p. 9

3.1.2 Generative

  • Prompt-conditioned generative models use VAEs or GANs to create medical images for augmentation and related tasks, but face challenges in variability, semantic alignment, and ethical concerns.The survey also identifies anomaly detection and varied datasets for training and evaluation as potential uses.Source: §3.1.2 · p. 9
  • Clinical-BERT outperformed counterparts in radiograph diagnosis and report generation, while practitioners favored Med-PaLM 2 responses over peers’ for clinical relevance [42].Med-Flamingo improved generative medical VQA performance by up to 20% in clinician evaluations and provided comprehensive justifications [43].Source: §3.1.2 · p. 9

3.1.3 Hybrid

  • A hybrid MRI model combined bootstrapped image-text pretraining, adjustable MedQFormer queries, and LoRA fine-tuning; experiments across five Alzheimer’s datasets assessed zero-shot classification [28].The experiments covered over 30,000 image volumes and distinguished healthy individuals, people with MCI, and those with AD.Source: §3.1.3 · p. 10

3.1.4 Conversational

  • Conversational textually prompted models target dialogue about medical images, while context, ambiguous queries, and response accuracy in complex scenarios remain challenges.The survey describes these models as a means of communication and collaboration between medical professionals and models.Source: §3.1.4 · p. 10
  • Conversational examples include LLaVA-Med, which outperformed prior models on certain metrics across three biomedical VQA datasets, and XrayGPT for chest-radiograph conversations [51] [52].Visual Med-Alpaca combined language and visual capabilities for diverse biomedical tasks using plug-and-play visual modules and parameter-efficient techniques.Source: §3.1.4 · p. 10

3.2 Visually Prompted Models

Visually prompted medical-imaging models follow two trajectories: adaptations tailor segmentation models, while generalists address broader tasks and modalities, sometimes integrating patient histories and genomic data.

Adaptations

  • MedSAM was built from over a million medical image-mask pairs for segmentation across modalities, while its training data had modality imbalances and it struggled with vessel-like structures [12].The survey presents it as an adaptable, promptable model for tasks including tumor-volume measurement.Source: §3.2.1 · p. 11
  • MedLSAM combines few-shot landmark localization with SAM segmentation, matching SAM on two 3D datasets spanning 38 organs with significantly fewer annotations.It was trained on 14,012 CT scans and autonomously generates 2D bounding boxes across slices.Source: §3.2.1 · pp. 11–12
  • 3DSAM-adapter retained most of SAM’s pretrained parameters and outperformed leading medical segmentation models in three of four tasks [88].It uses a visual sampler for the prompt encoder and a lightweight mask decoder with multi-layer aggregation.Source: §3.2.1 · p. 12
  • SAM-Med2D uses point, box, and mask prompts with interactive decoder training, while SAMed uses low-rank finetuning and SAM-U adds multi-box uncertainty estimation [47].SAM-Med2D generalized on MICCAI 2023 challenge datasets; SAM-U’s uncertainty maps identify potential inaccuracies and guide manual annotation.Source: Figure 5 · p. 11, §3.2.1 · p. 12

Generalist

  • Generalist models target broader medical tasks and modalities, potentially integrating patient histories and genomic data; GMAI may generate diagnostic outputs and treatment recommendations with justifications.The survey notes concerns about output verification, social bias, privacy, and computational and environmental costs.Source: §3.2.2 · p. 13
  • Med-PaLM M combines ViT for visual tasks and a language model for natural-language tasks, and reports zero-shot tuberculosis identification from chest X-rays and radiology reports rivaling expert radiologists in human evaluations.Its components were fine-tuned on the newly assembled MultiMedBench dataset; the study notes data scarcity, benchmark limitations, and safety and equity concerns.Source: §3.2.2 · p. 13
  • BiomedGPT was trained across radiographs, digital images, and text, with strong results in MedMNIST v2 classification and SLAKE and PathVQA question answering, but weaker MedNLI inference.The survey attributes the performance gap in part to its 182 million parameters and flags instruction sensitivity and out-of-distribution data as research challenges.Source: Figure 6 · p. 12, §3.2.2 · p. 13
  • RadFM combines radiological scans with language but faces 2D-data and report-quality limitations, while RETFound uses self-supervised learning on 1.6 million unlabeled retinal images.RETFound is described as adaptable for eye-disease diagnosis and systemic-disorder prediction with reduced reliance on extensive annotation.Source: §3.2.2 · p. 13

4 Discussion

The discussion compares textually and visually prompted model directions, their strengths and limitations, and how practical needs shape their use. It also highlights hardware and dataset demands as implementation constraints.

4 Discussion

  • Contrastive models align medical images and text for zero-shot prediction with less reliance on labeled data, but may struggle with complex pathologies and depend on large text datasets and careful hyperparameter tuning.The discussion describes them as useful when labeled data is scarce or expensive.Source: §4 · pp. 14–15
  • Generative models can provide detailed explanations and support education or report generation, while conversational models enable interactive queries but still face context, ambiguity, and fine-tuning challenges.Generative models may be computationally intensive, require significant training data, and produce text that is not concise.Source: §4 · p. 15
  • Hybrid models combine generative and contrastive methods for text and 3D medical-image integration, with potential for zero-shot prediction and visual question answering but challenges in designing integration mechanisms.Source: §4 · p. 15
  • Visually prompted adaptation models target specialized segmentation, while generalist models handle broad tasks and modalities without extensive retraining but may need task-specific data and robust out-of-distribution handling.Adaptation models may require substantial annotated data and may be less efficient across diverse imaging modalities.Source: §4 · pp. 15, 17
  • The survey describes conversational examples including LLaVA-Med, trained with biomedical figure-caption data and curriculum learning, and XrayGPT, which answers open-ended chest-radiograph questions using visual and textual information.Source: §4 · p. 16
  • Textually prompted models suit interpretation and detailed explanations, visually prompted models suit segmentation, and conversational models support collaboration; the choice depends on use case, data, and adaptability needs.The survey also describes a timeline of emerging techniques, conversational models, and generalist models.Source: §4 · p. 17, Figure 8 · p. 17

4.1 Hardware Requirements and Dataset Sizes

  • Hardware and dataset needs affect deployment feasibility: large models may require high-end GPUs or accelerators, while privacy and labor-intensive annotation constrain medical data acquisition.Dataset sizes in reviewed work range from thousands to millions of medical images and reports.Source: §4.1 · p. 18

5 Open challenges and Future Direction

The paper identifies multimodal integration and interpretability as future research directions for medical-imaging foundation models. It notes both opportunities from cross-modal knowledge and the current lack of built-in explainability.

5.1 Open-source Multimodal Models

  • Multimodal integration across imaging, text, and audio may support medical discoveries and knowledge transfer, but current self-supervised methods are not universally generalizable and often need modality-specific development.The paper calls for better methods to capture inter-modality and cross-modality relationships.Source: §5.1 · p. 19

5.2 Interpretablity

  • Most current foundation models lack built-in explainability, although understanding their capabilities, reasoning, and mechanisms is important for transparency, accountability, and healthcare use.The paper points to linking AI outputs with medical knowledge as a route to more understandable models.Source: §5.2 · p. 19

5.3 Bias and Variance in Foundational Models

The survey identifies bias in data and predictions and variance across training data as concerns for medical imaging foundation models. These concerns bear on fairness and generalization across patient populations and healthcare settings.

5.3 Bias and Variance in Foundational Models

  • Bias and variance are identified as two aspects requiring ongoing scrutiny in medical imaging foundation models.Source: §5.3 · p. 19
  • Bias in training data and predictions may involve race, ethnicity, gender, or socioeconomic factors, producing disparities in diagnosis or treatment recommendations.The survey calls for mitigation to support fairness, inclusivity, and ethical deployment.Source: §5.3 · p. 19
  • Variance is sensitivity to training-data fluctuations and can impair generalization across patient populations and healthcare settings.A model may perform well on one dataset but poorly on another, hindering reliability in real-world clinical applications.Source: §5.3 · p. 19

5.4 Adversarial Attacks

The survey highlights adversarial attacks as a concern for medical foundation models, where manipulated inputs may produce harmful outputs or compromise patient privacy. It calls for defenses to support safe and reliable medical use.

5.4 Adversarial Attacks

  • Adversarial attacks deliberately manipulate model inputs and may cause erroneous, potentially harmful outputs, including misdiagnoses and incorrect or delayed treatment.Source: §5.4 · p. 19
  • Adversarial tactics may compromise patient-data privacy, spread false medical information, and undermine trust in foundation models and healthcare systems.Source: §5.4 · pp. 19–20
  • The survey calls for robust defenses against adversarial threats to protect clinical decision-making and patient-data privacy.Source: §5.4 · p. 20

5.5 Down-stream Task Adaptation

The survey describes adaptation as a challenge alongside the substantial data and computational demands of foundation models. These costs and inference-speed constraints limit practicality in some real-world settings.

5.5 Down-stream Task Adaptation

  • Foundation models offer fine-tuning and prompting, but adapting them without losing learned knowledge remains a challenge.Efficient adaptation also faces computational and memory requirements and hyperparameter choices that can affect generalization.Source: §5.5 · p. 20

5.6 Extensive Data and Computational Demands

  • Foundation-model development, training, and deployment can be costly; large-scale training is data- and compute-intensive, and inference is costly because of many parameters.Extensive labeled data can also be expensive and time-consuming to acquire, especially in specialized domains or less-resourced languages.Source: §5.6 · p. 20
  • 50 times faster run-time speed is reported for FastSAM than SAM, with comparable performance, while SAM lacks real-time processing speed.The survey links computational demands to limited practicality for real-time use and resource-constrained edge or mobile devices.Source: §5.6 · p. 20

5.7 Prompt Engineering

The survey presents prompt engineering as a way for radiologists to interact with medical imaging foundation models and elicit explanations. It also identifies gaps in benchmarks for assessing models across tasks, modalities, and clinical scenarios.

5.7 Prompt Engineering

  • Prompt engineering can support interactive conversations in which radiologists seek clarification, add context, and ask follow-up questions about medical images.Source: §5.7 · p. 20
  • Tailored prompts let radiologists adapt model interactions to case-specific nuances rather than relying only on fixed queries or predefined prompts.Source: §5.7 · p. 20
  • Prompts can elicit detailed explanations of model conclusions, which the survey connects to interpretability and transparency.Source: §5.7 · p. 20

5.8 Lack of Effective Benchmark to Monitor Progress

  • Existing benchmarks and metrics often do not comprehensively assess performance across diverse medical imaging tasks, modalities, and real-world clinical scenarios.Source: §5.8 · p. 20
  • Standardized benchmarks could support fair model evaluation and help researchers assess models’ strengths and weaknesses across datasets and tasks.Source: §5.8 · p. 20
  • A benchmark spanning rare conditions and diverse patient populations could provide a broader assessment of model capabilities across untested scenarios.Source: §5.8 · p. 21
  • Benchmarks could set clear objectives for innovation and encourage models that address clinical needs while being robust, interpretable, and adaptable.Source: §5.8 · p. 21
  • Evaluations of performance and generalization across clinical settings could help providers select suitable models and support reliable, safe AI-assisted decisions.Source: §5.8 · p. 21

5.9 Enhancing Feature Representation from Frequency Perspective

ViT self-attention can miss high-frequency details important to medical imaging tasks and has quadratic computational complexity. The survey calls for model designs that balance computational efficiency with preservation of frequency information.

5.9 Enhancing Feature Representation from Frequency Perspective

  • ViT self-attention may neglect high-frequency textures and edges, which matter for tumor detection, cancer-type identification through radiomics, and treatment response assessment.The passage also notes that self-attention may generate redundant features.Source: §5.9 · p. 21
  • Self-attention has quadratic computational complexity, motivating consideration of more efficient ViT architectures.Source: §5.9 · p. 21
  • The survey proposes incorporating CNN layers or more efficient ViT architectures to balance computational efficiency with preservation of high-frequency information.Source: §5.9 · p. 21

6 Conclusion

The survey reviews recent medical-imaging foundation models, their potential contributions to healthcare, and directions for future research. It categorizes model types, discusses their strengths, limitations, use cases, and resource requirements, and offers a roadmap for further investigation.

6 Conclusion

  • The survey reviews recent advances in medical-imaging foundation models and their potential contributions to healthcare.Source: §6 · p. 21
  • The survey categorizes models into four groups based on text prompts and visual cues, with methodological descriptions and evaluations of each type’s strengths and limitations.Source: §6 · p. 21
  • It summarizes model pros, cons, and real-world medical-imaging use cases in Table 1.Source: §6 · p. 21
  • The survey considers hardware and dataset requirements and provides configuration strategies to clarify prerequisites for future research.Source: §6 · p. 21
  • It proposes future investigation directions as a roadmap for research on foundation models for medical imaging.Source: §6 · p. 21
Loading 2310.18689v1…