PaperlayerView exact revision on arXiv

Source-linked AI summary

AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models

Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha

arXiv:2105.00827v2cs.CLcs.AIcs.LG

TL;DR

Transformer-based biomedical pretrained language models have proliferated, but existing surveys give little emphasis to these models and no comprehensive survey covers recent trends. This paper synthesizes their foundations, methods, taxonomy, applications, challenges, and open issues, including reported task-specific findings such as a Pearson correlation of 0.8320 for clinical semantic textual similarity.

  • Problem

    More than 40 transformer-based biomedical pretrained language models have emerged, yet existing biomedical surveys focus mainly on context-insensitive embeddings and general-domain transformer models.

  • Method

    The paper surveys transformer-based biomedical pretrained language models by reviewing foundational concepts, pretraining and fine-tuning methods, embedding types, model taxonomies, applications, challenges, and open issues.

  • Results

    The survey comprehensively covers research on transformer-based biomedical pretrained language models and reports task-specific findings, including a Pearson correlation score of 0.8320 for clinical semantic textual similarity.

  • Takeaways & Limitations

    The paper organizes recent transformer-based biomedical language-model research and identifies challenges and possible solutions for improving these models.

  • Takeaways & Limitations

    The survey excludes context-insensitive biomedical embeddings and treats transformer and self-supervised-learning foundations only briefly.

Abstract

from arXiv · show

Transformer-based pretrained language models (PLMs) have started a new era in modern natural language processing (NLP). These models combine the power of transformers, transfer learning, and self-supervised learning (SSL). Following the success of these models in the general domain, the biomedical research community has developed various in-domain PLMs starting from BioBERT to the latest BioELECTRA and BioALBERT models. We strongly believe there is a need for a survey paper that can provide a comprehensive survey of various transformer-based biomedical pretrained language models (BPLMs). In this survey, we start with a brief overview of foundational concepts like self-supervised learning, embedding layer and transformer encoder layers. We discuss core concepts of transformer-based PLMs like pretraining methods, pretraining tasks, fine-tuning methods, and various embedding types specific to biomedical domain. We introduce a taxonomy for transformer-based BPLMs and then discuss all the models. We discuss various challenges and present possible solutions. We conclude by highlighting some of the open issues which will drive the research community to further improve transformer-based BPLMs.

1 INTRODUCTION

Transformer-based biomedical pretrained language models emerged after general-domain PLMs and now underpin biomedical NLP, but recent trends lacked a dedicated comprehensive survey. AMMU addresses this gap by organizing foundational concepts, model developments, applications, challenges, and future directions.

  • Background: Transformers combine self-attention, transfer learning, and self-supervised learning to model long-range relationships and support pretrained language models.Self-attention can be applied in parallel and stacked to learn complex language information.
  • Background: Traditional rule-based and feature-engineered NLP systems required substantial domain expertise, motivating deep learning with dense word representations.Earlier tf-idf and one-hot representations were high-dimensional and did not encode syntactic or semantic information.
  • Biomedical PLMs: Biomedical transformer models began with BioBERT in January 2019 and expanded to models including ClinicalBERT, ClinicalXLNet, BlueBERT, and PubMedBERT.BioBERT was the first transformer-based biomedical pretrained language model identified by the survey.
  • Research gap: Existing biomedical embedding surveys emphasized context-insensitive embeddings, while general-domain PLM surveys did not comprehensively cover biomedical transformer models.This left recent transformer-based biomedical language-model trends without a dedicated survey.

2 FOUNDATIONS

Transformer-based PLMs combine embedding layers with stacked transformer encoders to build contextual token representations. Their foundations also include self-supervised pretraining, which learns reusable representations from unlabeled data using automatically generated supervision.

  • 2.1 Embedding Layer: Embedding layers sum token, positional, segment, and other embedding-type vectors to form each token’s final input representation.The representation can include more than three embedding types in biomedical models, such as code, age, and gender embeddings.
  • 2.1 Embedding Layer: The embedding equation X = I + P + S expresses the final input embeddings as the sum of three embedding matrices.X, I, P, and S have dimensions n × e, where n is sequence length and e is embedding size.
  • 2.2 Transformer Encoder: Each transformer encoder layer combines multi-head self-attention, a position-wise feed-forward network, and Add-and-Norm operations.Add-and-Norm applies residual connections and layer normalization around both attention and feed-forward sublayers to avoid vanishing and exploding gradients.
  • 2.2.2 Multi-Head Self Attention (MHSA): Self-attention represents each token as a weighted sum of all input token vectors, allowing long-distance contextual relationships to be modeled.Multi-head attention applies this process in parallel with different weight matrices, then concatenates and projects the resulting head outputs.
  • 2.2 Transformer Encoder: A transformer encoder updates token vectors by encoding global contextual information through a sequence of encoder layers.Each layer receives the preceding layer’s output, with the first layer taking the embedding output X as input.
  • 2.3 Self-Supervised Learning: Self-supervised learning pretrains models on unlabeled corpora using automatically generated pseudo-labels defined by one or more pretraining tasks.The survey distinguishes generative, contrastive, and generate-contrastive SSL categories.

3 T-BPLMS CORE CONCEPTS

Biomedical PLM pretraining methods differ in how they combine general-domain and biomedical text, how much in-domain data they require, and whether they adapt vocabulary. Task-adaptive pretraining instead uses smaller task-related unlabeled corpora to learn domain- and task-specific knowledge more economically.

  • 3.1 Pretraining Methods: Mixed-domain pretraining combines general and biomedical text through continual or simultaneous pretraining.Continual pretraining adapts a general model sequentially, whereas simultaneous pretraining uses combined corpora with upsampled in-domain text.
  • 3.1.1 Continual Pretraining (CPT): Continual pretraining initializes a general PLM and further trains it on large biomedical corpora, as in BioBERT’s adaptation to PubMed and PMC text.It is described as the standard approach used by the biomedical NLP research community.
  • 3.1.1 Simultaneous Pretraining (SPT): Simultaneous pretraining combines general and in-domain corpora while upsampling in-domain text to maintain balanced training.This approach is presented as useful when only a small amount of in-domain text is available, whereas continual pretraining may be suboptimal in that setting.
  • 3.1.2 Domain-Specific Pretraining (DSPT): Domain-specific pretraining trains a model from scratch on in-domain text, enabling an in-domain vocabulary that reduces biomedical word fragmentation.PubMedBERT is cited as an example and is reported to achieve state-of-the-art results on BLURB.
  • 3.1.3 Task Adaptive Pretraining (TAPT): Both mixed-domain and domain-specific pretraining require large text volumes, making them expensive in computational resources and time.The survey frames task-adaptive pretraining as a smaller-corpus alternative to these resource demands.
  • 3.1.3 Task Adaptive Pretraining (TAPT): Task-adaptive pretraining gathers task-related unlabeled sentences and further pretrains the model to learn domain- and task-specific knowledge.Because it uses a relatively small task-related corpus, TAPT is less expensive than other pretraining methods.

3.2 Pretraining Tasks

Pretraining tasks provide automatically generated supervision for learning language representations, with main tasks learning representations and auxiliary tasks injecting curated biomedical knowledge. The survey covers token-, span-, and sentence-level objectives, including MLM, RTD, SBO, NSP, SOP, and ontology-based tasks.

  • Task categories: Pretraining tasks automatically generate pseudo-supervised labels and may be main tasks for language representations or auxiliary tasks for curated knowledge.Auxiliary tasks can draw on sources such as ontologies.
  • Main pretraining tasks: MLM randomly masks tokens and predicts them using both left and right context.Common variants include dynamic, whole-word, whole-entity, and whole-span masking.
  • Main pretraining tasks: RTD replaces selected tokens with generator predictions and trains a discriminator to identify whether each token was replaced.The survey describes RTD as providing more training signal than MLM.
  • Main pretraining tasks: SBO predicts every token in a masked contiguous span from representations at the span boundaries.Unlike MLM, SBO uses boundary token representations rather than the final hidden vector of each masked token.
  • Sentence-level tasks: NSP classifies whether two sentences are consecutive, whereas SOP classifies whether their order has been swapped.NSP addresses topic and coherence, while SOP focuses on sentence coherence.
  • Auxiliary pretraining tasks: Auxiliary tasks such as triple classification inject UMLS relational knowledge by predicting whether concepts are connected.Other auxiliary objectives use multi-similarity and knowledge-embedding losses.

3.3 Fine-Tuning Methods

Fine-tuning adds task-specific knowledge to general or in-domain representations. The survey describes intermediate fine-tuning across domain and task combinations, alongside multi-task fine-tuning for shared learning across tasks.

  • Fine-tuning foundations: Pretraining supplies general or in-domain knowledge, while task-specific datasets and layers provide knowledge needed for a particular task.For text classification, the architecture combines a contextual encoder with a classifier.
  • Intermediate fine-tuning: Intermediate fine-tuning on large related datasets can improve performance on small target datasets.The survey organizes IFT by whether source and target tasks and domains match.
  • Intermediate fine-tuning: Same-task IFT transfers from general-domain data to in-domain data or from generic biomedical data to a more specific biomedical target.Examples include SNLI/MNLI to MedNLI and broad biomedical NER corpora to smaller target NER corpora.
  • Intermediate fine-tuning: Different-task IFT transfers knowledge across tasks within a domain or across both task and domain boundaries.Examples include medical question answering to question similarity and general MultiNLI to biomedical QA.
  • Multi-task fine-tuning: Multi-task fine-tuning shares embedding and encoder layers while assigning each task a separate task-specific layer.It can reduce over-fitting and deployment costs, especially in low-resource biomedical settings, but may not always produce the best results.

3.4 Embeddings

Transformer-based biomedical models combine main embeddings of input sequences with auxiliary embeddings that provide positional, segment, demographic, temporal, or semantic information. The survey covers character, subword, code, and multiple auxiliary embedding types.

  • Embedding categories: Main embeddings map input sequences to vectors, while auxiliary embeddings add information; token representations sum main and auxiliary vectors.The survey classifies embeddings into main and auxiliary categories.
  • Main embeddings: Text embeddings represent words as character, subword, or code-based vectors.Code embeddings can represent patient visits through disease, medication, or other clinical codes.
  • Main embeddings: Character embeddings use letters, punctuation, special characters, and numbers, with BioCharBERT applying CharCNN to produce word representations.Character embeddings are learned during pretraining.
  • Main embeddings: Subword vocabularies represent frequent words directly and rare or misspelled words through meaningful subwords.BPE merges frequent pairs, WordPiece selects pairs using a language model, and SentencePiece treats spaces as characters.
  • Subword embeddings: SentencePiece can generate vocabularies with BPE or Unigram and is used by XLNet, ALBERT, and T5.Unigram begins with a large vocabulary and iteratively trims symbols to obtain a smaller one.
  • Auxiliary embeddings: Position embeddings encode token locations, while segment embeddings distinguish tokens belonging to different input sequences.Position embeddings may be predetermined or learned during pretraining.
  • Auxiliary embeddings: Age, gender, and semantic-group embeddings add patient, demographic, temporal, or concept-group information in models such as BEHRT, BERT-EHR, and UmlsBERT.Semantic-group embeddings also support representations for rare words.

4 T-BPLMS TAXONOMY

The survey proposes a taxonomy of transformer-based biomedical pretrained language models and organizes models by biomedical data source and training setting. It covers EHR, radiology, social media, scientific literature, hybrid corpora, and language-specific models, while also discussing knowledge integration and data limitations.

  • Taxonomy contribution: The survey presents a taxonomy and brief overview of transformer-based biomedical pretrained language models.Figure 14 summarizes the taxonomy.
  • EHR-based models: EHR-based models use clinical notes, structured codes, or multimodal patient records, with examples including ClinicalBERT, BEHRT, Med-BERT, and BERT-EHR.BEHRT uses code, position, age, and segment embeddings, while Med-BERT uses MLM and length-of-stay tasks.
  • Radiology reports: Radiology models are pretrained on radiology reports using either continued pretraining or training from scratch with domain vocabularies.The survey discusses RadBERT, FS-BERT, and RAD-BERT.
  • Social media: Social-media biomedical models use health-related posts, reviews, questions, and answers as domain data.BioRedditBERT is reported to outperform several in-domain models by up to 1.8% in health-related entity normalization.
  • Scientific literature: Scientific-literature models are trained on biomedical publications, with approaches including continued pretraining and domain-specific training from scratch.PubMed and PMC provide large collections, while PubMedBERT uses in-domain vocabulary and whole-word masking; OuBioBERT uses focused and satellite PubMed corpora.
  • Hybrid corpora: Hybrid-corpus models combine general and in-domain or related-domain text when obtaining enough in-domain data is difficult.The survey identifies this issue particularly for clinical records and includes hybrid-corpus models in its taxonomy.
  • Language-specific models: Language-specific biomedical models are trained from scratch or initialized from multilingual, monolingual, or general-language models.CHMBERT is initialized from general Chinese BERT and further pretrained on 185GB of Chinese medical text from more than 100 hospitals.
  • Knowledge-enhanced models: Biomedical models can be enhanced with human-curated resources such as UMLS, whose concepts and relations supply additional domain knowledge.The survey describes synonym- and relation-based pretraining for models including UmlsBERT and CoderBERT.

5 BIOMEDICAL NLP TASKS

The survey reviews biomedical NLP tasks addressed with transformer-based models, emphasizing task-specific representations, domain adaptation, and transfer from related datasets. It covers entity extraction, semantic similarity, relation extraction, classification, question answering, and summarization.

  • Entity Extraction: Entity extraction commonly uses contextual token representations followed by softmax or CRF-based sequence labeling.Adding BiLSTM on top of BERT generally shows little difference because transformer encoders already encode contextual token information.
  • Entity Extraction: Intermediate fine-tuning on similar datasets helps models perform entity extraction when biomedical target datasets are small.Further pretraining AraBERT on a small Arabic biomedical corpus improved Arabic biomedical entity extraction over multilingual BERT and AraBERT.
  • Semantic Textual Similarity: Clinical semantic similarity benefits from general-domain intermediate fine-tuning and iterative multi-task learning across related datasets.ClinicalBERT fine-tuned on combined general and clinical STS data achieved a Pearson correlation score of 0.8320.
  • Relation Extraction: SciBERT with softmax outperformed BERT with softmax on three biomedical relation-extraction datasets.Adding attention above BioBERT also outperformed applying LSTM to final hidden states on three biomedical extraction datasets.
  • Text Classification: PubMedBERT and BioBERT achieved the best results among evaluated in-domain BERT models for Alzheimer disease clinical-note classification.Custom attention over encoder outputs was also reported to improve clinical text classification performance and interpretability.
  • Question Answering: Biomedical question answering improves through entity masking, intermediate fine-tuning, and auxiliary-task transfer.BioMedBERT outperformed BioBERT on biomedical question answering, while intermediate fine-tuning on general QA, biomedical QA, or NER transferred useful task knowledge.
  • Text Summarization: For biomedical scientific-article summarization, BERT-large outperformed other models, including BioBERT and other in-domain BERT models.One approach embedded sentences, clustered them, and extracted informative sentences from each cluster.

6 EVALUATION

The survey describes biomedical benchmarks as tools for comparing transformer-based pretrained models across multiple tasks. It identifies BLUE, BLURB, and CBLUE as biomedical benchmark suites and summarizes model performance using BLURB.

  • 6 EVALUATION: BLUE, BLURB, and CBLUE are biomedical benchmarks proposed to evaluate pretrained models across biomedical NLP tasks.The survey summarizes the performance of various transformer-based biomedical pretrained language models in Table 10 using BLURB scores.

7 CHALLENGES AND SOLUTIONS

The survey identifies computational cost, limited biomedical data and vocabulary, noisy inputs, and representation choices as challenges for transformer-based biomedical models. It discusses adaptation, multitask learning, augmentation, semi-supervised learning, and richer sequence representations as solutions.

  • Computational Cost: Masked domain pretraining and domain-specific pretraining require substantial computing resources and long training durations.Adapting general BERT to biomedical text took BioBERT around ten days using eight GPUs.
  • Computational Cost: Task-adaptive pretraining uses task-related unlabeled instances to learn domain- and task-specific knowledge with lower adaptation cost.The passage presents TAPT as a low-cost alternative to large-scale domain adaptation.
  • Small Datasets: Small biomedical target datasets limit task-specific learning during fine-tuning and can reduce performance.Intermediate fine-tuning on larger related datasets is presented as one way to address this limitation.
  • Small Datasets: Intermediate fine-tuning, multi-task fine-tuning, data augmentation, and semi-supervised learning are proposed for low-resource biomedical tasks.These approaches use related datasets, multiple tasks, transformed examples, or pseudo-labeled instances to expand useful training signals.
  • Robustness: Noisy test instances reduce model performance because models are trained and fine-tuned mostly on less noisy data.Noisy words may also split into several subtokens, affecting model learning.
  • Vocabulary: General-domain vocabularies may represent biomedical words poorly because domain terms are split into multiple subwords.Low-cost adaptation can extend the vocabulary with biomedical terms, although this increases model size.
  • Representations: Using max-pooling, attention, or hierarchical convolution over final hidden vectors can produce better sequence representations than using only the special-token vector.This issue affects text classification and sentence-pair tasks such as NLI and STS.

8 FUTURE DIRECTIONS

The survey highlights bias, privacy leakage, domain adaptation, masked-language-modeling efficiency, benchmark coverage, probing, and pretraining cost as open directions. It calls for methods that improve fairness, privacy assessment, vocabulary adaptation, robustness, and computational efficiency.

  • 8.1 Mitigating Bias: Biomedical models can inherit dataset bias involving attributes such as gender, age, ethnicity, and marital status.The survey emphasizes identifying and reducing bias so automated decisions do not favor particular groups.
  • 8.2 Privacy Issues: Clinical records contain sensitive information, and pretrained models may leak personal information from their pretraining corpora.The KART framework is proposed to assess leakage from pretrained biomedical language models.
  • 8.3 Domain Adaptation: Low-cost domain adaptation extends general vocabularies with biomedical vocabulary but increases model size.Further research is suggested to develop additional low-cost adaptation methods.
  • 8.4 Pretraining Tasks: Masked language modeling predicts only 15% of randomly selected tokens, limiting the training signal per example.The survey identifies this as a motivation for improving pretraining tasks and methods.
  • 8.5 Evaluation: Biomedical benchmarks such as BLUE, BLURB, and ChineseBLUE assess model ability and robustness across multiple tasks.Multi-task benchmarks are needed because pretrained models are expected to generalize beyond one or two specific tasks.
  • 8.6 Probing: Intrinsic probes can reveal syntactic, semantic, factual, and common-sense knowledge learned during pretraining.The survey points to biomedical probing as an area requiring further development.
  • 8.7 Efficient Architectures: Pretraining biomedical models requires large corpora and substantial computation, motivating architectures that reduce training time and data requirements.The survey suggests investigating efficient architectures such as ConvBERT and DeBERTa for biomedical models.

9 LIMITATIONS

The survey focuses comprehensively on transformer-based biomedical pretrained language models while limiting coverage of context-insensitive embeddings and foundational concepts.

  • Context-insensitive biomedical embeddings are excluded because the survey focuses on transformer-based BPLMs.Readers seeking those embeddings are directed to Kalyan and Sangeetha’s survey.
  • Transformers and self-supervised learning are covered only briefly because they are foundation concepts rather than the survey’s main focus.

10 CONCLUSION

The paper surveys recent transformer-based biomedical pretrained language models by organizing their methods, tasks, fine-tuning approaches, embeddings, taxonomy, challenges, and open issues.

  • The survey presents recent trends in transformer-based biomedical pretrained language models.
  • It explains pretraining methods, pretraining tasks, fine-tuning methods, and biomedical embedding types.
  • It introduces a taxonomy for transformer-based biomedical pretrained language models.
  • It discusses challenges, possible solutions, and open issues for improving transformer-based biomedical pretrained language models.
Loading 2105.00827v2…