PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions

Yuting He, Fuxiang Huang, Xinrui Jiang, Yuxiang Nie, Minghao Wang, Jiguang Wang, Hao Chen

arXiv:2404.03264v1cs.CYcs.AI

TL;DR

Healthcare foundation models address the mismatch between task-specific AI and diverse healthcare practices, but their development faces challenges in data, algorithms, and computing. This survey synthesizes HFM methods, datasets, applications, challenges, and future directions, finding recurring patterns in pre-training and adaptation while identifying limitations affecting deployment.

  • Problem

    Healthcare AI has many task-specific models, but diverse healthcare practices require models that can support a wide range of tasks.

  • Method

    The survey reviews HFM methods, datasets, applications, challenges, and future directions across language, vision, bioinformatics, and multimodal fields.

  • Results

    The survey identifies that most pre-training works use annotation-efficient generative, contrastive, or hybrid learning, while fine-tuning remains widely used for adaptation.

  • Takeaways & Limitations

    HFMs provide a framework for extending healthcare AI beyond specialized tasks, while guiding development toward broader intelligent healthcare services.

Abstract

from arXiv · show

Foundation model, which is pre-trained on broad data and is able to adapt to a wide range of tasks, is advancing healthcare. It promotes the development of healthcare artificial intelligence (AI) models, breaking the contradiction between limited AI models and diverse healthcare practices. Much more widespread healthcare scenarios will benefit from the development of a healthcare foundation model (HFM), improving their advanced intelligent healthcare services. Despite the impending widespread deployment of HFMs, there is currently a lack of clear understanding about how they work in the healthcare field, their current challenges, and where they are headed in the future. To answer these questions, a comprehensive and deep survey of the challenges, opportunities, and future directions of HFMs is presented in this survey. It first conducted a comprehensive overview of the HFM including the methods, data, and applications for a quick grasp of the current progress. Then, it made an in-depth exploration of the challenges present in data, algorithms, and computing infrastructures for constructing and widespread application of foundation models in healthcare. This survey also identifies emerging and promising directions in this field for future development. We believe that this survey will enhance the community's comprehension of the current progress of HFM and serve as a valuable source of guidance for future development in this field. The latest HFM papers and related resources are maintained on our website: https://github.com/YutingHe-list/Awesome-Foundation-Models-for-Advancing-Healthcare.

I. INTRODUCTION

Healthcare AI has progressed from task-specific models toward foundation models that can generalize across diverse healthcare scenarios. This survey reviews HFM methods, datasets, applications, challenges, and future directions across four sub-fields.

  • Motivation: Task-specific healthcare AI models show specialist-level performance but remain mismatched with diverse healthcare scenarios and requirements.This motivates the question of whether one model can benefit a variety of healthcare tasks.
  • Motivation: Healthcare foundation models learn general abilities from broad data and can be applied across language, vision, bioinformatics, and multimodal healthcare settings.The survey presents HFMs as a potential answer to the limitations of narrowly targeted models.
  • Challenges: HFM development depends on healthcare data, AI algorithms, and computing infrastructure, while data ethics, diversity, heterogeneity, and cost complicate generalizable model construction.High-dimensional data such as 3D CT and whole-slide images also increase computational and environmental costs.
  • Scope: The survey covers HFM progress from 2018 to 2024 across language, vision, bioinformatics, and multimodal models, alongside challenges and future directions.Its scope is motivated by the rapid growth of HFM research and the difficulty of reviewing all papers and aspects in limited space.
  • Contributions: The survey reviews 200 technical papers, 114 large-scale datasets or databases, and 16 healthcare applications, then discusses challenges and emerging directions.It organizes the review around methods, data, applications, data-algorithm-computing challenges, and future HFM development.

II. METHODS

Healthcare foundation models pre-train broad representations and adapt them to downstream tasks through fine-tuning or prompting. The survey organizes these methods across language and other healthcare foundation-model settings.

  • Overview: Healthcare foundation models learn representations from large-scale information and adapt them to a wide range of healthcare applications.The survey distinguishes generative, contrastive, and hybrid learning paradigms for pre-training.
  • Language pre-training: Healthcare language models commonly use generative pre-training, especially next-token prediction, to learn from large medical text corpora.GatorTronGPT is described as combining medical and general text in GPT-like pre-training.
  • Language pre-training: Contrastive and hybrid learning provide alternative language-model pre-training strategies for capturing medical linguistic structures and relationships.MedCPT uses query-document pairs and in-batch negatives, while BERT-like models combine masked-language and next-sentence prediction.
  • Adaptation: Language-model adaptation uses fine-tuning or prompt engineering to transfer pretrained models to specific healthcare tasks or domains.Fine-tuning adjusts model parameters, whereas prompting guides predictions or tuning through instructions.
  • Adaptation: Parameter-efficient fine-tuning methods adjust only part of a pretrained model, preserving some pretrained representations and reducing adaptation costs.LoRA injects trainable low-rank decomposition matrices into Transformer layers.

B. Vision Foundation Models for Healthcare

Vision foundation models use supervised and self-supervised pre-training to learn transferable medical-image representations, then adapt through fine-tuning, adapters, and prompts. Their applications span classification, segmentation, detection, prognosis, retrieval, and enhancement.

  • Pre-training: Vision foundation models are pre-trained on extensive labeled or unlabeled medical datasets to support numerous downstream tasks.The reviewed methods include supervised, generative, contrastive, and hybrid pre-training.
  • Pre-training: Supervised vision pre-training uses annotations to learn broad applicability for specific tasks such as medical image segmentation.STU-Net was pre-trained on 1,204 CT volumes with masks for 104 organs.
  • Pre-training: Self-supervised pre-training addresses the high cost of medical image annotation by learning universal representations from large-scale unlabeled data.It supports later development of vision models for different downstream medical-image tasks.
  • Pre-training: Generative, contrastive, and hybrid paradigms learn visual representations by reconstructing corrupted images, contrasting positive and negative samples, or combining objectives.Hybrid examples integrate masked image modeling with contrastive learning or other self-distillation components.
  • Adaptation: Vision-model adaptation includes fine-tuning, adapter tuning, and prompt engineering, with SAM-based methods transferring segmentation capabilities to medical images.Adapters train few parameters while preserving the pretrained model’s generic representations.

C. Bioinformatics Foundation Models in Healthcare

Bioinformatics foundation models use generative, contrastive, and hybrid pre-training to learn biological representations, then adapt them to downstream analyses through fine-tuning, adapters, or prompts.

  • Pre-training: Generative learning, especially masked omics modeling and next-token prediction, learns context dependence and reconstructs masked biological information.Masked omics modeling is used for expression values and gene names in high-dimensional, sparse scRNA-seq data.
  • Pre-training: Bioinformatics models also combine specialized generative tasks, contrastive learning, and hybrid objectives to capture biological structure.Examples include RNA structural alignment, DNA sequence prediction, and combinations of masked modeling with cell-type discrimination or contrastive learning.
  • Adaptation: Bioinformatics foundation models adapt pre-trained representations to function, sequence, and related downstream analyses.Adaptation methods include full-parameter fine-tuning, adapter tuning, and prompt engineering.
  • Adaptation: Fine-tuning remains the most widely used adaptation paradigm, directly adjusting model parameters for specific downstream tasks.scBERT was fine-tuned for 9 cell type annotation tasks involving unseen and user-specific scRNA-seq data.
  • Adaptation: Adapter tuning reduces computing costs by training only added layers, while prompt-based methods use pre-trained embeddings or prompts for downstream tasks.Representative adapter methods add MLP layers after the backbone; GenePT uses ChatGPT gene embeddings based on literature.

D. Multimodal Foundation Models for Healthcare

Healthcare multimodal foundation models integrate language, vision, bioinformatics, and other modalities, using generative, contrastive, or hybrid pre-training and fine-tuning, adapters, or prompts for adaptation.

  • Motivation: Multimodal foundation models are designed to understand individual modalities and their interconnections in complex healthcare scenarios.Their multimodal scope creates additional challenges because models must understand and integrate heterogeneous information.
  • Pre-training: Generative multimodal pre-training predicts or reconstructs images, text, or other data, often through masked representation modeling.MMBERT integrates image features into a BERT architecture using image-based masked language modeling.
  • Pre-training: Contrastive pre-training aligns representations across modalities, including images and text, ECG signals, pathology images, biomedical images, and molecular structures.The approach extends from visual-language learning to several healthcare data combinations.
  • Pre-training: Hybrid pre-training combines masked representation modeling with image-text matching or contrastive learning.These combinations are used to fuse advantages from different learning paradigms.
  • Adaptation: Multimodal models use fine-tuning, adapter tuning, and prompt engineering to adapt pre-trained models to healthcare domains and tasks.Adapter layers can bridge modality-specific features, while prompts guide models toward downstream tasks.
  • Adaptation: Full-model fine-tuning can require long training times and risk overfitting or domain bias, motivating parameter-efficient adaptation methods.Adapters train added components, and projection layers can convert medical visual features into text embeddings.

E. Analysis of the paradigms in HFM

The survey’s analysis links pre-training paradigms, healthcare sub-fields, and adaptation strategies, finding broad reuse of general models, predominantly annotation-efficient pre-training, and widespread fine-tuning.

  • Overview: The Sankey diagram maps flows from pre-training paradigms through healthcare sub-fields to adaptation paradigms.“Non” denotes direct adaptation of existing pre-trained models without self-performed pre-training.
  • Current progress: More than 1/3 of works directly adapted existing general-field models in language and multimodal healthcare fields, but not bioinformatics.The survey attributes this pattern to greater portability between general and healthcare language and vision data.
  • Current progress: Most language foundation models directly adapt existing general-domain language models to healthcare tasks because language has strong portability.The passage specifically describes portability from general language pre-training to healthcare applications.
  • Current progress: Most pre-training works use self-supervised generative, contrastive, or hybrid learning to reduce annotation costs while learning general representations.The rationale is that annotating large pre-training datasets is expensive.
  • Current progress: Supervised learning remains present in vision foundation-model pre-training because visual continuity makes semantic separation difficult using only self-supervision.Supervised objectives provide a more direct optimization target for learning meaningful features.
  • Current progress: More than 1/2 of works use fine-tuning for adaptation, while newer techniques such as LoRA pursue lower parameter efficiency.The survey associates fine-tuning with a stable learning process.

2) Electronic health records:

Healthcare foundation-model data spans electronic health records, dialogue, medical images, and biological sequences, with privacy constraining some clinical datasets and scale varying by modality.

  • Electronic health records: Electronic health records contain clinically valuable disease descriptions, diagnoses, scenarios, and patient outcomes, but privacy makes these datasets smaller than literature collections.MIMIC-III contains more than 122K records from forty thousand critical-care patients, while MIMIC-IV contains 299K clinical records.
  • Healthcare dialogue: Healthcare dialogue datasets capture interactions among doctors and patients or among doctors, supporting communication and information-retrieval capabilities.The survey notes many publicly available healthcare dialogue datasets.
  • Medical images: Medical-image resources include 3D CT, MRI, PET, whole-slide images, X-rays, dermoscopy, and fundus photographs for diverse clinical tasks.Examples include 1,411 CT and 1,222 MRI images in the Medical Segmentation Decathlon, 38,842 CT volumes in ULS, and 1,214 PET-CT pairs in AutoPET.
  • Medical images: Whole-slide images pose global-analysis challenges because their resolution can reach 150,000 x 85,000 pixels.The survey describes WSI datasets spanning 33 cancer types through the TCGA program.
  • Bioinformatics data: High-throughput sequencing has produced large publicly available DNA, RNA, protein, and single-cell RNA-sequencing datasets for training bioinformatics foundation models.These datasets contain more than millions of expression values or sequences.

1) Genomics and single-cell omics data:

Genomics, transcriptomics, proteomics, and single-cell datasets provide large-scale resources for modeling genetic information, gene expression, cellular functions, and protein biology. These resources span comprehensive reference databases, population cohorts, perturbation data, and multimodal biomedical collections.

  • Genomics and single-cell omics data:: GenBank contains up to 3.7 billion publicly available DNA sequences for genomic modeling.
  • Genomics and single-cell omics data:: CellxGene Corpus covers 789 cell types across 1,219 datasets and more than 72 million cell expressions.
  • Genomics and single-cell omics data:: UK Biobank provides health, genetic, lifestyle, and related information from 500,000 participants.
  • Genomics and single-cell omics data:: Ensembl and RNAcentral provide reference resources for annotated genomes, protein sequences, and non-coding RNA sequences.
  • Genomics and single-cell omics data:: LINCS L1000 measures approximately 1,000 landmark genes after cellular perturbations such as drugs, toxins, or genetic modifications.
  • Genomics and single-cell omics data:: Healthcare multimodal datasets remain concentrated in vision-language data, with limited modal diversity despite growing data accumulation.

2) Beyond visual-language data:

Healthcare foundation models extend beyond visual-language applications into protein-text, physiological-signal, and clinical multimodal data. Language and vision models support consultation, reporting, education, diagnosis, and clinical workflows, while broader databases organize cross-subfield healthcare data.

  • Beyond visual-language data:: SwissProtCLAP contains 441,000 protein-text sequence pairs spanning 327,577 genes and 13,339 organisms.
  • Beyond visual-language data:: PTB-X pairs 21,837 ECG signals with corresponding medical reports, illustrating non-visual-language multimodal data.
  • Beyond visual-language data:: Language foundation models are applied to diagnosis, medical report generation, education, and patient consultation.
  • Beyond visual-language data:: Comprehensive healthcare databases organize data across multiple subfields, while public multimodal resources include question answering and visual question answering datasets.
  • Beyond visual-language data:: Vision foundation models support segmentation, classification, detection, diagnosis, prognosis, surgery, and medical-image enhancement workflows.

1) Medical diagnosis:

Healthcare foundation models support diagnosis and related clinical tasks across vision and biological domains. Vision models assist screening, classification, prognosis, surgery, retrieval, and enhancement, while biological models analyze sequences, interactions, molecular structure, and function.

  • Medical diagnosis:: Vision foundation models enable screening, detection, classification, segmentation, and diagnosis from medical images.They can reduce workload and costs for some low-risk imaging tasks, but trustworthiness remains a challenge for high-risk uses such as tumor grading.
  • Medical diagnosis:: Vision foundation models also support prognosis by extracting disease-related features and biomarkers from medical images.
  • Medical diagnosis:: Three-dimensional vision models can segment CT and MRI objects for surgical planning and assistance without conventional task-specific retraining.
  • Medical diagnosis:: Vision foundation models additionally support medical-image retrieval, prototyping, and enhancement across clinical workflows.
  • Medical diagnosis:: Biological foundation models address sequence analysis and interaction analysis involving genes, proteins, and RNAs.
  • Medical diagnosis:: Protein structure prediction models use sequence relationships to analyze molecular structure and function, with xTrimoPGLM outperforming advanced baselines in large-scale parameters.

4) Disease research and drug response:

Healthcare foundation models address disease research, drug response, multimodal diagnosis, report generation, biomedical language interaction, and patient consultation. Their development remains shaped by the data, algorithmic, and computing foundations of healthcare AI.

  • Disease research and drug response:: Biological foundation models support drug sensitivity, drug response, disease risk, cellular perturbation, and COVID variant prediction tasks.
  • Disease research and drug response:: Multimodal foundation models fuse patient data from multiple modalities and enable zero-shot classification for open-ended disease diagnosis.
  • Disease research and drug response:: Visual-language models generate radiology reports from medical images, alleviating professional workload and enhancing output stability.
  • Disease research and drug response:: Molecule-language models enable molecule editing with text prompts without additional molecule data and annotations.
  • Disease research and drug response:: Multimodal patient chatbots combine text and visual understanding for preliminary medical consultation.
  • Disease research and drug response:: Data, algorithms, and computing infrastructure are identified as three pillars underlying opportunities and challenges for healthcare foundation models.

A. Data

Healthcare foundation models face data challenges that limit generalizable training, including ethical constraints, long-tailed diversity, heterogeneity, cost, and annotation demands. Algorithmic responsibility also remains essential because healthcare decisions affect human life.

  • Data availability: Large-scale healthcare datasets are difficult to construct because ethics, diversity, heterogeneity, and cost constrain the data needed for foundation-model generalization.The paper identifies data scarcity as the core challenge and asks how large-scale healthcare datasets can be constructed.
  • Ethics: Ethical requirements restrict healthcare-data acquisition, including scans performed solely for AI training when modalities may injure the human body.The paper uses CT imaging as an example of a modality whose acquisition can cause bodily injury.
  • Diversity: Long-tailed modality distributions make common images abundant while specialized modalities such as OCT, DSA, and PET remain scarce and expensive.This scarcity restricts the generalization of models across task-specific healthcare applications.
  • Heterogeneity: Healthcare data vary across populations, regions, and medical centers, creating space, target, and concept shifts between training and deployment.These shifts can limit adaptation to new acquisition protocols and personalized or precise medical care.
  • Cost: Specialized healthcare acquisition and annotation require expensive equipment and skilled professionals, making large-scale collection and crowdsourced labeling impractical.The paper gives CT scans as an example of costly acquisition and describes professional annotation as inefficient and expensive.
  • Algorithm responsibility: Healthcare foundation models must address responsibility because black-box behavior and dataset bias can undermine explainability, fairness, ethics, and safety.The paper highlights explainability and fairness as central responsibility concerns for healthcare deployment.

2) Reliability:

Healthcare foundation models must remain reliable, capable, adaptable, computationally feasible, and environmentally sustainable. Their future development is framed around moving from specific tasks toward general capabilities and AI–human collaboration.

  • Reliability: Healthcare reliability is threatened by hallucinations and by models falling behind evolving medical knowledge, while model editing can be costly and introduce side effects.The paper presents outdated knowledge and imperfect updating methods as continuing reliability challenges.
  • Capability: Model capacity supports knowledge representation but increases computation, memory use, cost, and carbon emissions, especially for large healthcare data such as 3D CT volumes.The paper also notes that current functionality may not meet complex, multi-department and multimodal clinical demands.
  • Adaptability: Adaptability remains limited by real-world data heterogeneity, costly downstream adaptation data, and the difficulty of scaling large models to resource-limited devices.Wearable medical devices are identified as one setting where direct deployment of very large models is impractical.
  • Computation: Training and adapting healthcare foundation models can exceed institutional budgets because of massive parameter counts, long training times, and scarce advanced GPUs.Training LLaMA with 65B parameters reportedly required about 21 days on 2048 A100 GPUs with 80GB of RAM.
  • Environment: Large-model training and deployment impose substantial environmental costs through electricity use and carbon emissions, while supporting technologies and policies still lag.The paper reports that carbon emissions from training a BERT-based model could be offset by 40 trees over 10 years.
  • Future directions: Future directions frame healthcare foundation models as moving from specific to general tasks, with AI–human collaboration identified as a practical application opportunity.The survey organizes future directions around role, implementation, application, and emphasis.

1) Improving healthcare capabilities:

The survey identifies healthcare foundation models as a basis for human-supervised, adaptable, and deployable healthcare AI. It highlights collaboration design, dynamic modeling, low-cost adaptation, real-world robustness, and device scalability as key capability directions.

  • Human-AI collaboration: Human-AI collaboration assigns tedious tasks to AI while doctors provide judgment and correct potential mistakes.This approach is intended to support more challenging healthcare tasks than conventional AI-independent or individual-expert paradigms.
  • Human-AI collaboration: Accountability mechanisms and human supervision can increase trust and provide patients with greater legal safeguards in clinical applications.Independent AI models may be unable to assume responsibility for clinical incidents.
  • Optimizing collaboration: Future collaboration methods should optimize division of labor and scenario-adapted prompting because misallocation can make collaboration worse than independent AI.Prompting methods must also accommodate settings such as surgery, where users need to remain focused.
  • Dynamic modeling: Dynamic AI models are needed to coordinate diverse modalities and clinical requirements across departments in real-world healthcare.Representation mechanisms such as Transformers, mixture-of-experts, and selective state space models are proposed for broader data distributions.
  • Task adaptation capability: Adaptation capability should reduce data and computation costs while improving flexibility and access to knowledge learned from large-scale data.Carefully designed prompts can improve target-task performance without additional training, although broader prompting methods remain needed outside language models.
  • Scalability: Scaling methods such as learngene are needed to run foundation models efficiently on computation-limited clinical devices.This is especially relevant to resource-limited clinical scenarios and expensive devices already operating in medical centers.

1) Single-domain to multi-domain:

Healthcare foundation models are moving from single-domain and single-modality settings toward broader, multi-domain and multimodal healthcare use, while raising requirements for explainability, security, and sustainability. The survey identifies these capabilities and infrastructures as important directions for wider application.

  • Single-domain to multi-domain:: HFMs must learn and generalize across populations, regions, and medical centers to address healthcare’s serious domain heterogeneity.Domain adaptation, domain generalization, and privacy-preserving federated learning are proposed, but large-scale distributed training remains challenging.
  • Single-domain to multi-domain:: HFMs require stronger multitask capabilities than conventional single-task models because real-world healthcare spans organs, diseases, and clinical objectives.
  • Single-modality to multi-modality:: Multimodal HFMs can combine complementary modality features for more precise and reliable results, but integrating additional modalities and handling missing modalities remain open problems.Current efforts focus mainly on language and vision, with cross-modality generation, self-supervised learning, and knowledge distillation explored as learning strategies.
  • Explainable HFM:: Explainability remains necessary because neural-network black boxes make HFM behavior difficult to understand, while existing theories and evidence are insufficient for their scale.Future work includes machine-learning theory for foundational models and higher-level explanatory evidence beyond heatmaps.
  • Secure HFM:: Security and reliability are prerequisites for trusted HFM use, requiring defenses against attacks, robust outputs, reliability measurement, and accountability mechanisms.Jailbreaking can expose private information, and healthcare applications require reliable outputs because they are closely related to human life.
  • Sustainable HFM:: Sustainable HFM development requires low-power training and deployment because large-scale training consumes substantial power and produces carbon emissions.
Loading 2404.03264v1…