Source-linked AI summary
Transformers in Healthcare: A Survey
Subhash Nerella, Sabyasachi Bandyopadhyay, Jiaqing Zhang, Miguel Contreras, Scott Siegel, Aysegul Bumin, Brandon Silva, Jessica Sena, Benjamin Shickel, Azra Bihorac, Kia Khezeli, Parisa Rashidi
TL;DR
Healthcare Transformer research addresses how one architecture can analyze heterogeneous medical data across clinical language, EHRs, images, signals, and biomolecular sequences. This survey synthesizes applications identified through PRISMA-guided searches and concludes that Transformers have broad potential, while interpretability and computational cost remain important barriers.
Problem
Healthcare data span heterogeneous modalities, and evidence on how Transformers address clinical, imaging, EHR, and biomolecular tasks is distributed across application areas.
Method
The paper surveys Transformer applications across healthcare modalities and synthesizes studies selected using PRISMA-guided searches.
Results
Transformers have been rapidly adopted for healthcare tasks including clinical information extraction, medical image segmentation, MRI synthesis, and biomolecular applications.
Takeaways & Limitations
Transformers can model multiple healthcare data modalities, but many current models remain task-specific and often require robust multimodal inputs.
Takeaways & Limitations
Transformer attention can appear transparent through attention weights, but those explanations are often fragmented and not robust.
Abstract
from arXiv · showhide
With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning architecture initially developed to solve general-purpose Natural Language Processing (NLP) tasks and has subsequently been adapted in many fields, including healthcare. In this survey paper, we provide an overview of how this architecture has been adopted to analyze various forms of data, including medical imaging, structured and unstructured Electronic Health Records (EHR), social media, physiological signals, and biomolecular sequences. Those models could help in clinical diagnosis, report generation, data reconstruction, and drug/protein synthesis. We identified relevant studies using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We also discuss the benefits and limitations of using transformers in healthcare and examine issues such as computational cost, model interpretability, fairness, alignment with human values, ethical implications, and environmental impact.
5 Large Language Models (LLMs)
Large language models use Transformer architectures and self-supervised pretraining to support language generation and downstream tasks. Their scale has expanded rapidly, while healthcare adoption remains constrained by data, privacy, and model-reliability challenges.
- Foundation models are large-scale AI systems trained on vast data and adapted for many downstream tasks.
- Most large language models use Transformer architectures and self-supervised pretraining on unannotated text.
- Autoregressive decoder-only models such as PaLM, GPT-3, Chinchilla, LLaMA, PaLM2, and GPT-4 generate text and answer questions.
- Model parameters and training data have expanded to trillions of tokens, with capabilities linked to data, parameters, and computation.
- LLMs can exhibit few-shot prompting, but they also generate false information, toxic language, and racial stereotypes.
- Healthcare foundation-model adoption is limited by insufficient large, diverse medical datasets, complex data, privacy regulations, and model recency.
6 Transformers in NLP
Transformers support clinical NLP by representing medical text contextually and addressing information-extraction tasks across clinical narratives and biomedical literature. Domain-specific pretraining and task-specific fine-tuning have produced gains across several benchmarks.
- Pretraining on clinical corpora improved embedding performance, and ClinicalBERT achieved higher Pearson correlation than word2vec and fastText.
- Clinical information extraction includes named entity recognition, coreference resolution, question answering, semantic similarity, relationship extraction, and ICD coding.
- Clinical concept extraction predicts concept start and end positions, commonly using BIO tags and Transformer encoders with task-specific layers.
- Transformer-based models improved clinical coreference resolution compared with recurrent models on large-document and symptom-extraction tasks.
- BioBERT achieved state-of-the-art performance on factoid, list, and yes/no BioASQ questions, while QA-BERT outperformed LSTM and BERT baselines.
7 Transformers for Structured EHR Data
Transformers model structured EHR data by learning temporal, ontological, and multimodal relationships among diagnoses, medications, procedures, and clinical notes. These representations have been applied to diagnosis prediction and outcome forecasting.
- Structured EHR data contain temporally linked diagnoses, medications, demographics, interventions, and readmissions, with codes organized in ontological trees.
- The Graph Convolution Transformer jointly learns diagnosis–medication relationships while performing diagnosis-treatment classification.
- Separate Transformer representations or contrastive learning can fuse clinical codes with notes into multimodal patient embeddings.
- Combining multimodal representations with a BERT encoder predicted mortality and length of stay better than other baselines.
- BEHRT applies masked language modeling to sequential ICD codes and age to predict future diagnoses.
8 Transformers in Computer Vision
Transformers have been adapted across medical computer-vision tasks, especially segmentation, report generation, image synthesis, and reconstruction. Hybrid and Transformer-only architectures use attention to capture global context, while computational efficiency remains a design concern.
- Medical Image Segmentation: Medical image segmentation must capture complex pixel interactions despite limited datasets, anatomical context requirements, and inter-patient variability.
- Medical Image Segmentation: Transformer self-attention models global image context irrespective of network depth, addressing limitations of convolutional receptive fields.
- CNN-Transformer Hybrids: TransUNet combines CNN downsampling, a Transformer encoder for global representations, and cascaded upsampling for full-resolution segmentation.
- CNN-Transformer Hybrids: Deformable Transformer attention over local regions reduces computational complexity, and CoTr outperformed other CNN-Transformer hybrids on the BCV dataset.
- Report Generation: RATCHET generates radiology reports with a CNN image encoder and vanilla Transformer decoder conditioned on image features and shifted text tokens.
- Image Synthesis: PTNet synthesized infant MRI scans, outperformed previous CNN approaches, and achieved a practical execution time of 30 slices per second.
- PET Reconstruction: Transformer-GAN and related models reconstruct PET images by combining CNN spatial information with Transformer long-range dependencies.
9 Transformers for Critical Care
Transformers are applied to multimodal clinical time series and surgical video-text data for outcome prediction and intra-operative instruction generation. Reviewed studies combine attention-based sequence modeling with imaging, laboratory, demographic, tabular, or textual information.
- Long-Term Adverse Outcomes: Transformer models combine chest CT scans, blood tests, and patient characteristics to predict durable immunotherapy survival gains in non-small cell lung cancer.Simple Temporal Attention was used for asynchronous clinical time series.
- Long-Term Adverse Outcomes: A modified Transformer combined sequential carcinoembryogenic antigen measurements with tabular tumor and chemotherapy features to predict colorectal cancer recurrence.The model incorporated 1D convolutions before localized self-attention and outperformed commercial diagnostic tests.
- Long-Term Adverse Outcomes: Multi-headed self-attention was applied to French national health-insurance claims data covering 65 million individuals to predict relapse after surgery.The reviewed studies used the SNIIRAM database.
- Surgical Instruction Generation: An encoder-decoder Transformer with self-critical reinforcement learning jointly modeled surgical activity and relationships between visual information and textual descriptions.The model was evaluated on the Database for AI Surgical Instruction dataset.
10 Transformers for Social Media Data in Public Health
Transformers support public-health surveillance from social-media text, including adverse drug reactions, depression, diabetes relationships, vaccine confidence, and disease hotspots. Reviewed studies report gains from domain adaptation, multitask learning, informal terminology, and Transformer-based classification.
- Applications: Transformers have been applied to social-media monitoring of adverse drug reactions, depression, vaccine confidence, and disease hotspots.The reviewed applications use social-media and Internet information for several public-health problems.
- Monitoring Adverse Drug Reactions: FARM-BERT outperformed competing models on adverse-drug-response datasets while using separate prediction heads for text classification and response labeling.The framework supports multitask learning and is described as easier and computationally faster to train for end-to-end systems.
- Monitoring Adverse Drug Reactions: A T5-based sequence-to-sequence approach reported relative F-1 improvement for adverse-drug-event classification across multiple datasets.The training strategy used proportional mixing and temperature scaling for multi-dataset learning.
- Monitoring Diabetes: Pre-trained BERTweet detected causal sentences in diabetes-related tweets, while BERT plus a Random Field Generator extracted potential cause-effect relationships.The approach was applied to large-scale Twitter data concerning diabetes.
- Categorizing Vaccine Confidence: Domain-specific BERT outperformed alternative neural models and human annotators when categorizing maternal vaccination tweets.Comparisons included default BERT, domain-specific BERT, LSTM variants, and models with or without embeddings.
- Locating Disease Hotspots: Disease-hotspot classification achieved F1 scores up to 88% for influenza and 94% for COVID-19, with informal terminology improving performance by up to 15%.Average improvement from including informal and non-standard terminology was 8%; geolocation prediction was moderate.
11 Monitoring Bio-Physical Signals
Transformers are used to model physiological and behavioral signals, including human activity, EEG, ECG, and MRI-related data. The reviewed approaches use multimodal fusion, self-supervised pretraining, cross-modal representations, and CNN-integrated Transformer networks.
- Human Activity Recognition: Multimodal self-attention fuses RGB, depth, and skeletal data, while other studies model activity from accelerometer and gyroscope sequences.These approaches address human activity recognition using wearable, smartphone, and Internet-of-Things data.
- Electroencephalograms: Contrastive self-supervised pretraining helps Transformer models represent EEG data collected with different hardware while performing different tasks.Transformers have also been used to classify visual or motor imagery from EEG signals.
- Electrocardiograms: Transformers have been used with ECG signals, alone or with other sensory information, to predict stress.Public WESAD and SWELL-KW datasets support this application.
- Electrocardiograms: A Transformer embedded within a CNN architecture has been used to classify arrhythmias from ECG-related data.
12 Transformers for Biomolecular Sequences
Transformers model genomic, proteomic, and drug sequences for disease classification, protein-property prediction, molecular interaction analysis, drug synthesis, and metabolism prediction. The survey describes sequence translation and attention mechanisms as recurring tools across these biomolecular applications.
- Overview: Transformers model genomic, proteomic, and drug sequences for relationships between biological sequences and diseases, including drug and protein synthesis.Gene sequence alignment and synthesis tasks are treated through a machine-translation perspective.
- DNA: Gene Transformer uses multi-head self-attention and preceding 1D convolutions to detect biomarkers for classifying lung cancer subtypes.The convolutions extract low- and moderate-level features before the attention layer.
- Protein: Transformers predict global protein properties such as function and localization and local residue properties such as structure and post-translational modifications.Recent advances primarily fine-tune pretrained deep models on small datasets.
- Protein: AlphaFold’s success in practically solving protein-structure prediction marked a watershed moment for deep-learning applications to protein problems.
- Drug Synthesis: Transformers convert target-driven de novo drug synthesis into neural machine translation from amino-acid sequences to binding-drug chemical formulas.The method requires neither prior drug-structure information nor 3D structural information about the protein target.
- Drug Sensitivity: An attention-based gene-expression encoder combines tumor gene profiles with drug SMILES encodings to model anticancer drug sensitivity.A contextual attention layer computes an attention distribution over drug SMILES tokens in the genetic context.
- Drug-Target Interactions: A Transformer-based molecular translation method used cross-representation attention embeddings to classify binding and non-binding molecular substructures.SMILES were translated into IUPAC nomenclatures for 83 million PubChem compounds before classification using BindingDB data.
- Drug Metabolism Prediction: The reviewed drug-metabolism network outperformed state-of-the-art models including BioTransformer.
13 Discussion
Transformers support diverse healthcare applications but face substantial deployment challenges involving interpretability, environmental impact, computational cost, fairness, privacy, and alignment. The paper discusses compression, quantization, knowledge distillation, and federated learning as approaches for addressing some of these constraints.
- Transformers have been applied to clinical reporting, medical imaging, molecular sequencing, drug interactions, protein synthesis, surgical augmentation, and biophysical signal analysis.
- Transformer attention can expose contextual information through attention weights, but fragmented attention, skip-connections, and dynamic weight computation make interpretation difficult.
- Training large AI models has substantial environmental costs, with emissions concentrated in model development and training rather than fine-tuning or adaptation.The paper cites an estimate of 626,000 lbs of carbon dioxide for training a large-scale deep learning model.
- Healthcare deployment is constrained by high computational requirements, motivating pruning, knowledge distillation, quantization, and other efficiency techniques.The paper notes that healthcare settings often require lightweight models for real-time predictions with minimal maintenance costs.
- Quantization to 2 bits produced approximately 2.3% performance degradation while achieving 13X parameter compression and 4X compression of embeddings and activations.
- Bias can arise from under-represented patient groups and from problem formulation, data collection, preprocessing, development, validation, or deployment.
- Healthcare foundation models must align with human values while avoiding hallucinations, privacy violations, unethical behavior, and harmful outcomes.
- Federated learning enables shared training across fragmented healthcare data sources by exchanging model parameters or gradients without directly exchanging patient information.
14 Conclusion
Transformer models show potential across healthcare modalities and applications, but many remain task-specific and often require robust multimodal inputs. The paper therefore presents large-scale Transformer models as a promising direction while recognizing the need for broader inputs and continued development.
- Transformer models show potential across images, clinical text, biophysical signals, and genomic data for applications ranging from diagnosis to drug discovery.
- Most current healthcare Transformer models are task-specific, while many applications require robust multimodal inputs.