PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, Jie fu

arXiv:2110.05006v4cs.CL

TL;DR

Biomedical PLM research is fragmented across disciplines and data types, with limited comprehensive comparison and standardized terminology and benchmarks. This survey synthesizes biomedical PLMs and their downstream applications, proposes a broad taxonomy, and discusses resources, limitations, and future trends. It reports task-specific gains, including improvements over prior methods on biomedical named-entity recognition and relation extraction.

  • Problem

    Biomedical PLM work spans disciplines and heterogeneous data, but existing studies lack comprehensive comparison, discussion, and standardized terminology and benchmarks.

  • Method

    The survey systematically reviews biomedical PLMs, proposes a taxonomy covering data sources and model perspectives, and examines downstream applications and resources.

  • Results

    Biomedical PLM methods outperform prior systems by 1.78–12% in exact match F1 on 11 biomedical NER datasets and by 1.6% in Chinese medical relation extraction.

  • Takeaways & Limitations

    The survey provides a cross-disciplinary reference spanning biomedical PLMs, their data resources, downstream tasks, and benchmark-oriented comparisons.

  • Takeaways & Limitations

    Biomedical language models may reflect biases such as gender and language, creating potentially more serious consequences in biomedical use; experts should calibrate them before end-user deployment.

Abstract

from arXiv · show

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.

1 INTRODUCTION

Biomedical PLMs address limited annotations, knowledge-intensive data, and diverse biomedical sequences, while this survey organizes their development, resources, models, applications, and limitations. It proposes an inclusive taxonomy spanning text, protein, DNA, and biomedical text-image data.

  • Biomedical data vary in knowledge abstraction and scale: high-abstraction resources such as knowledge bases and EHRs are usually relatively small-scale.
  • Biomedical NLP spans bioinformatics, biomedical informatics, medicine, and computer science, covering tasks such as classification, information extraction, question answering, and drug discovery.
  • PLMs suit biomedical settings because annotated data are limited, large unlabeled corpora can provide richer features and implicit knowledge, and transformers can process protein and DNA sequences.
  • Its coverage extends beyond biomedical text to protein, DNA, and biomedical text-image pairs, and includes applications such as event detection, dialogue systems, and biological sequence tasks.
  • The paper categorizes biomedical PLM methods and benchmark datasets, introduces competitions and venues, and discusses limitations and future trends.
  • The survey reviews biomedical PLMs and downstream applications, proposes a taxonomy based on perspectives including training data sources and architecture, and enumerates resources and configurations.

2 BACKGROUND: Pre-trained Language Models

Pre-trained language models learn representations from unannotated sequential data, with Transformer architectures becoming the dominant backbone because they support parallel token processing and scalable modeling. This section introduces LSTM and Transformer components, self-attention and feed-forward networks, and the shift from static to contextualized word representations.

  • Pre-trained language models learn useful representations from unannotated data such as natural-language sequences.
  • Transformer: Transformers replace recurrence with parallel token computation and attention-based interactions that depend dynamically on the input.
  • LSTM: LSTM is a recurrent architecture for sequential modeling whose cell and gates regulate information across arbitrary time intervals.
  • Transformer: A Transformer layer alternates self-attention for token interaction with a nonlinear feed-forward network for refining token representations.
  • Transformer: The feed-forward network typically uses d_in = 4 × d_model = d_out to reduce the low-rank bottleneck caused by inactive neurons after activation.
  • Pre-training for texts: Contextualized word embeddings represent a word according to its usage, addressing the limitation of static embeddings that assign one vector independently of context.

3 PLMS IN BIOMEDICAL DOMAIN

Biomedical PLMs are motivated by the scale, knowledge intensity, limited annotation, and sequential or multimodal nature of biomedical data. The survey organizes biomedical PLMs by data and model types, including textual, vision-and-language, and protein/DNA models.

  • Motivation: Biomedical PLMs can learn from sequential biomedical data such as text, EHR histories, proteins, and DNA through self-supervised pre-training.These data often lack annotations but can be modeled effectively with language-model objectives.
  • Motivation: Limited biomedical annotation motivates PLMs because richer feature extractors may reduce dependence on labeled data and support few-shot learning.Biomedical tasks commonly require domain expertise, while large unstructured corpora provide unlabeled training material.
  • Motivation: Biomedical PLMs can capture implicit domain knowledge from large-scale plain biomedical corpora without human annotations.The survey describes them as easily used soft knowledge bases for knowledge-intensive tasks.
  • Biomedical Data: Biomedical pre-training data include EHRs, scientific publications, social media, image-text pairs, and biological sequences such as proteins.The survey discusses these sources as unstructured or sequential corpora for pre-training.
  • Pre-training Strategy: The choice between training from scratch and continual training depends on biomedical corpus scale and biomedical domain specificity.The survey identifies these as the two core factors affecting the pre-training decision.
  • Taxonomy: The survey categorizes biomedical PLMs into biomedical textual models, vision-and-language models, and protein/DNA language models.It also compares existing models and discusses their differences across biomedical communities.

3.5 Beyond Text: Biomedical Vision-and-Language Models

Biomedical vision-and-language pre-training combines medical images with text, which are widely available and often paired. The survey contrasts dual-encoder and fusion-encoder designs and also covers diffusion-based medical generation and localized modeling.

  • Motivation: Medical vision-and-language pre-training is attractive because radiology images and diagnostic reports are commonly collected as pairs.The section also notes that text and imaging data are relatively easy to obtain in medicine.
  • Fusion-Encoder Models: Fusion-encoder models interact visual and linguistic elements earlier, enabling richer multimodal representations for tasks such as visual question answering.This design can make single-modal tasks more difficult because the modalities are less easily separated.
  • Other Biomedical Models: The survey also covers medical text-to-image diffusion models and protein language models that encode biochemical and structural information.Protein language models are reported to predict protein structures with high accuracy in a challenging assessment.
  • Dual-Encoder Models: Dual-encoder models encode images and text separately, supporting single-modal and cross-modal tasks but limiting complex cross-modal interaction.Their representations can serve image classification, captioning, and retrieval, while challenging multimodal reasoning remains constrained.
  • Dual-Encoder Models: Contrastive medical image-text models reduce annotation requirements for image classification and retrieval-oriented downstream tasks.ConVIRT’s vision encoder achieved comparable or better image classification using much fewer annotated data than an ImageNet-initialized counterpart.
  • Dual-Encoder Models: Localized contrastive learning aligns sentences or image regions while encouraging spatial smoothness and sensitivity for localized medical imaging tasks.LoVT is described as the first dual-encoder study targeting localized medical imaging tasks.

4 FINE-TUNING PLMS FOR BIOMEDICAL DOWNSTREAM TASKS

The survey reviews biomedical downstream-task benchmarks and finds that domain-specific PLMs generally outperform general-domain PLMs. It highlights gains from domain-specific pre-training, citation information, and task-specific fine-tuning.

  • Benchmarks: The BLUE benchmark covers five biomedical text-mining tasks, whereas BLURB adds question answering and broader biomedical reasoning-oriented coverage.BLUE excludes question answering and mixes clinical data with biomedical literature; BLURB includes NER, PICO, relation extraction, sentence similarity, document classification, and question answering.
  • Overall Performance: Biomedical PLMs significantly outperform general-domain BERT and RoBERTa across the surveyed downstream tasks.The advantage is especially pronounced for sentence similarity and question answering.
  • Overall Performance: Biomedical PLMs such as PubMedBERT and BioLinkBERT exceed BERT and RoBERTa by more than 10% on sentence similarity and question-answering tasks.This comparison is reported for the surveyed biomedical evaluations.
  • Model Comparisons: PubMedBERT, which uses domain-specific pre-training from scratch, consistently outperforms BioBERT, ClinicalBERT, and BlueBERT across all reported tasks.BioLinkBERT additionally uses PubMed citation links during pre-training and achieves state-of-the-art performance on most tasks.
  • Question Answering: BioLinkBERT outperforms PubMedBERT by 15% on PubMedQA and 4% on BioASQ for document-level question answering.These results are reported for the respective document-level datasets.

4.1 Information Extraction

Biomedical information extraction uses PLMs for named entity recognition and relation extraction across literature, EHRs, and social media. The surveyed results generally favor biomedical pre-training and task-specific adaptation, including in low-resource settings.

  • Scope: Biomedical information extraction covers named entity recognition, relation extraction, and event extraction over literature, EHRs, and biomedical social media.These tasks extract entities, concepts, relations, and events from unstructured biomedical text.
  • Relation Extraction: A domain-specific ELMo model outperforms the previous BiLSTM-CRF state of the art by 3.4% F1 on i2b2 relation extraction.The model was trained on clinical reports and relevant Wikipedia pages.
  • Named Entity Recognition: In-domain continual pre-training improves BioNER more than directly using off-the-shelf ELMo embeddings, including a 4% F1 gain on JNLPBA.The comparison indicates that domain adaptation matters for the embedding model’s benefit.
  • Named Entity Recognition: BioALBERT improves F1 by 7.47% on NCBI-disease and 10.63% on BC5CDR-disease over BioBERT.The lightweight model uses sentence-order prediction during biomedical pre-training.
  • Named Entity Recognition: GreenBioBERT covers 60% of BioBERT’s results on eight BioNER datasets while using 2% of its cloud compute cost.Its domain-adaptation method is designed to save time and memory.
  • Relation Extraction: Fine-tuning PLMs generally outperforms traditional neural methods for biomedical relation extraction, while contrastive learning further improves performance.The survey reports the strongest results for contrastive fine-tuning on DDI, PPI, and ChemProt.

4.2 Text Classification

Biomedical text-classification studies generally find that fine-tuned PLMs outperform traditional neural networks, although performance depends on dataset domain and model specialization.

  • BERT and BioBERT outperform traditional embeddings and neural networks in electronic-health-record classification.
  • Clinical domain information and entity features can outperform general or biomedical PLMs on some clinical classification datasets.
  • RoBERTa-base and BERTweet outperform Clinical BioBERT on biomedical social-media datasets, reflecting differences in captured general semantics and domain knowledge.
  • Fine-tuning PLMs generally outperforms traditional neural-network methods on biomedical text-classification datasets.
  • BioBERT outperforms BERT on COVID-19 data but performs worse than ClinicalBioBERT on clinical MIMIC-III data.

4.3 Sentence Similarity

Biomedical sentence-similarity research compares PLMs, sentence embeddings, and traditional neural methods, with results favoring different approaches across evaluations.

  • BioSentVec is a biomedical sentence-embedding model trained on over 30 million PubMed and MIMIC-III documents.
  • BioSentVec yields better performance than existing word-embedding and sentence-encoder methods on sentence similarity and text classification.The reported advantage is attributed to better sentence-level semantic information capture.
  • BioSentVec improves traditional deep-learning models by 2% F1 score.
  • XLNet achieves the best performance among BERT, XLNet, and RoBERTa for the clinical semantic-textual-similarity task.
  • BioSentVec combined with traditional neural networks outperforms directly fine-tuned BERT and BioBERT in the cited comparison.

4.4 Question Answering

Biomedical question-answering studies improve PLM performance through domain knowledge, multi-task learning, transfer learning, and broader pre-training or fine-tuning corpora, while some answer formats remain difficult.

  • NLI transfer improves BioASQ performance on Yes/No, Factoid, and List questions by 5.59%, 0.53%, and 13.58%, respectively.
  • Shared representations between question entailment and question answering, and multi-task learning with biomedical entity recognition, improve BioBERT performance.
  • Sequence tagging with a BioBERT encoder addresses questions requiring detection of multiple answer spans and achieves the BioASQ 7b and 8b list-type questions.
  • Fine-tuning SQuAD consistently improves performance across model variants, while extended BioBERT-QA pre-trained on unstructured EHR data achieves the best reported performance in its comparison.
  • Fine-tuning or pre-training with open-domain, biomedical, and clinical corpora improves biomedical and clinical question answering.
  • Domain knowledge and multi-task learning improve BioBERT and other PLMs on biomedical question answering.Examples include disease knowledge, medical entities, and auxiliary biomedical named-entity recognition tasks.

4.5 Dialogue Systems

Biomedical dialogue systems adapt PLMs through domain or dialogue pre-training and task-specific objectives, but standard lexical-overlap metrics inadequately assess response faithfulness and factual correctness.

  • The dominant adaptation strategy pre-trains PLMs on general or medical corpora and then fine-tunes them on medical dialogue datasets.
  • MT5 achieves the best performance among the compared models for intent and slot understanding on M2-MedDialog.
  • BERT-GPT has lower perplexity than Transformer and GPT while maintaining similar diversity metrics on Chinese medical dialogue.
  • In-domain dialogue pre-training and dialogue-adaptive objectives improve response relevance, informativeness, coherence, and task performance.DAPO explicitly models coherence, specificity, and informativeness and outperforms MLM and NSP objectives.
  • BLEU, METEOR, and NIST primarily measure shallow n-gram overlap and do not account for paraphrasing or terminology variation.
  • Existing PLM-based metrics can inadequately evaluate response faithfulness, and biomedical factual-consistency metrics remain less developed.
  • Direct performance comparison is difficult because the reviewed methods use different datasets.

4.6 Text Summarization

Biomedical text summarization uses extractive and abstractive PLM-based methods, often incorporating domain knowledge or architectures for long documents. Reported results show gains over baselines, while factual accuracy and readability remain concerns.

  • Methods: Biomedical summarization methods include extractive and abstractive approaches using PLM encoders and decoders.The surveyed methods use models including BioBERT, SciBERT, BART, T5, RoBERTa, and LED.
  • Abstractive summarization: PubMedBERT-based summarization outperforms BERTSum and MatchSum, while T5 with BERTScore performs better than strong supervised methods such as BERTSum.These findings come from separate biomedical summarization comparisons.
  • Abstractive summarization: Around 2% improvement in ROUGE metrics was reported for a SciBERT-based clinical abstractive model using salient medical ontological terms.The improvement was observed on the MIMIC-CXR and OpenI datasets.
  • Limitations: Summarizers can produce fluent and relevant synopses, but factual accuracy cannot be guaranteed and evidence direction aligns with references approximately 50% of the time.The evidence-direction result concerns BART-based multi-document summarization of medical studies.
  • Readability: BART trained on CNN/DM and PubMed abstracts achieves the strongest ROUGE scores, whereas PubMed-only pre-training has the lowest readability.This comparison evaluates pre-training data choices for plain language summarization.
  • Comparative results: Domain knowledge improves performance over directly fine-tuning PLMs, while long-document methods achieve the best performance on PubMed and CORD-19.The comparison reports stronger results for knowledge incorporation and for the long biomedical text method on both datasets.

4.7 Natural Language Inference

Biomedical natural language inference determines relations such as entailment, neutrality, and contradiction between sentences. PLM-based systems incorporate medical knowledge, syntax, ensembles, or architecture search to improve performance across biomedical datasets.

  • Task definition: Natural language inference, also called text entailment, infers entailment, neutral, and contradiction relations between two sentences.The sentences are named the premise and hypothesis and support downstream tasks including question answering and evidence inference.
  • Methods and results: 98% accuracy was achieved on the REQ dataset by an ensemble incorporating syntax information.This was the best performance among participants in the MEDIQA 2019 medical NLI task.
  • Knowledge infusion: Medical knowledge infusion improved BioELMo accuracy by 0.8% and improved performances of BERT, BioBERT, SciBERT, ClinicalBERT, BlueBERT, and ALBERT in all cases.The improvements were reported for knowledge-augmented biomedical NLI models.
  • Knowledge infusion: 8.27% improvement on the REQ dataset was reported when medical entity information was infused into ClinicalBERT.The Sem-KGN framework incorporated medical knowledge-base information into BERT.
  • Comparative results: ALBERT achieved the best performance on the MedNLI dataset among the listed biomedical PLMs.The compared models include BioELMo, BERT, BioBERT, SciBERT, ClinicalBERT, BlueBERT, and ALBERT.

4.8 Proteins/DNAs Prediction

Protein and DNA language models support structure prediction and genomic sequence tasks. The surveyed results include strong performance from AlphaFold 2 and DNABERT, alongside limitations related to evolutionary information and protein-family specificity.

  • Scope: The survey notes that biomedical PLM applications span a larger space than the applications listed in this section.The section includes only applications considered well investigated or promising.
  • Protein prediction: Protein structure prediction seeks to infer three-dimensional structure from amino acid sequence, but only a small fraction of protein structures is known.Structure knowledge can facilitate understanding protein function.
  • Protein prediction: AlphaFold 2 achieved much better performance than other CASP14 methods, including template-based methods, in a blind gold-standard assessment.The authors claim precise and reliable protein structure estimates, while the assessment uses newly solved undisclosed structures.
  • Protein prediction: Existing protein structure prediction methods, including AlphaFold 2, are more family-specific than protein-specific and rely on captured evolutionary information.This qualifies the scope of their predictive behavior.
  • Protein prediction: A biologically informed pre-training task that predicts profiles derived from multiple sequence alignments can improve downstream protein structure prediction.Other work uses ProtT5 attention heads without multiple sequence alignments.
  • DNA prediction: DNABERT achieved state-of-the-art performance on promoter prediction, splice-site prediction, and transcription-factor binding-site prediction, and identified functional genetic variants.Other DNA models target enhancer–promoter interactions, RNA-binding proteins, and DNA taxonomic classification.

4.9 Competitions and Venues

Biomedical competitions and shared tasks provide recurring venues for developing and comparing methods across information extraction, retrieval, question answering, summarization, and health-related social media processing. Many recent challenges use pre-trained language model-based methods.

  • Shared tasks: MEDIQA 2019 covered medical NLI, recognizing question entailment, and question answering, while MEDIQA 2021 included three medical-document summarization tasks.The summarization tasks were question, multi-answer, and radiology report summarization.
  • Competitions: BioASQ has held annual workshops and challenges since 2013 on biomedical semantic indexing, information retrieval, and question answering.Its 2019 activities included corresponding benchmark datasets.
  • Competitions: BioCreative has evaluated biological text-mining and information-extraction methods since 2004, with BioCreative VII including five tracks.Two BioCreative VII tracks concerned COVID-19 text mining and multilabel topic classification.
  • Competitions: TREC provides annually held information-retrieval workshops with biomedical tracks on clinical decision support, precision medicine, and clinical trials.TREC began in 1992 and uses large test collections.

5 DISCUSSION

The discussion identifies limitations and concerns spanning interpretability, causality, misinformation, coverage–quality trade-offs, heterogeneous data, ethics, privacy, and evaluation. It also highlights standardized benchmarks and continued biomedical PLM development as future needs.

  • Limitations and Concerns: Biomedical training corpora may contain misinformation, while limited high-quality data forces a coverage–quality trade-off.Large corpora can provide broad coverage but may contain a smaller proportion of high-quality expert knowledge.
  • Limitations and Concerns: Biomedical PLMs need interpretable predictions because incorrect biomedical decisions may be deadly.The paper emphasizes both understandable decision processes and textual reasons for predictions.
  • Limitations and Concerns: Causal identification remains important because biomedical claims often concern causal associations between entities, events, and processes.Automatic causality recognition could suggest causal connections relevant to biomedical decisions and reduce human workload.
  • Limitations and Concerns: Heterogeneous biomedical information such as tables, numbers, figures, and graphs requires deeper preprocessing and model adaptation.Most PLMs are unable to interpret tables and numbers well, and multidomain biomedical PLMs require more attention.
  • Limitations and Concerns: Biomedical PLMs raise concerns about bias, privacy, and safe deployment in settings involving life-changing decisions.The paper discusses biased training data, possible recovery of personal health information, and expert calibration before end-user use.
  • Future Trends: Biomedical PLMs are difficult to compare because applications and evaluation criteria are not yet sufficiently standardized across communities.The paper calls for open standardized and well-categorized benchmarks to support biomedical NLP development.
  • Future Trends: Future work should develop finer-grained taxonomies, define state-of-the-art approaches, and improve biomedical data collection and sharing.The discussion presents this survey as a first step toward organizing biomedical PLMs and their applications.

6 CONCLUSION

The paper systematically surveys biomedical pre-trained language models, their data sources, downstream applications, limitations, and future trends. It aims to organize recent progress and clarify how general-domain PLMs may support biomedical domains.

  • Conclusion: The paper systematically summarizes recent advances in biomedical pre-trained language models.It covers their background, motivations, existing models, biomedical data sources, and downstream applications.
  • Conclusion: The survey discusses applications of biomedical pre-trained language models across various downstream biomedical tasks.It also addresses limitations and future trends.
  • Conclusion: The paper expects general-domain pre-trained language models to help specific biomedical domains.
Loading 2110.05006v4…