PaperlayerView exact revision on arXiv

Source-linked AI summary

AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha

arXiv:2108.05542v2cs.CL

TL;DR

Transformer-based pretrained language models address the challenge of learning useful NLP representations without relying entirely on costly labeled data or training each downstream model from scratch. This survey synthesizes their self-supervised foundations, architectures, adaptation methods, benchmarks, libraries, taxonomy, and future directions. It presents the resulting review as a reference for understanding and tracking research on T-PTLMs.

  • Problem

    NLP models face costly labeled-data requirements, limited labeled data in some domains, and the need to train downstream models from scratch.

  • Method

    The paper comprehensively reviews T-PTLM pretraining, embeddings, downstream adaptation, benchmarks, libraries, and future directions, while proposing taxonomies for models and adaptation methods.

  • Results

    The survey covers recent T-PTLM research and presents a new taxonomy categorizing models by corpus, architecture, SSL type, and extensions.

  • Takeaways & Limitations

    The paper serves as a reference for learning core T-PTLM concepts and staying updated on recent developments.

  • Takeaways & Limitations

    Sub-word embeddings cannot encode fine-grained character information and are brittle to noise such as simple typos.

Abstract

from arXiv · show

Transformer-based pretrained language models (T-PTLMs) have achieved great success in almost every NLP task. The evolution of these models started with GPT and BERT. These models are built on the top of transformers, self-supervised learning and transfer learning. Transformed-based PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch. In this comprehensive survey paper, we initially give a brief overview of self-supervised learning. Next, we explain various core concepts like pretraining, pretraining methods, pretraining tasks, embeddings and downstream adaptation methods. Next, we present a new taxonomy of T-PTLMs and then give brief overview of various benchmarks including both intrinsic and extrinsic. We present a summary of various useful libraries to work with T-PTLMs. Finally, we highlight some of the future research directions which will further improve these models. We strongly believe that this comprehensive survey paper will serve as a good reference to learn the core concepts as well as to stay updated with the recent happenings in T-PTLMs.

1 INTRODUCTION

Transformer-based pretrained language models learn universal language representations from large unlabeled text and transfer them to downstream NLP tasks. The survey organizes their foundations, taxonomies, adaptation methods, evaluation resources, libraries, and future directions.

  • Motivation: T-PTLMs learn universal language representations from large volumes of unlabeled text and transfer this knowledge to downstream tasks.This transfer avoids training downstream models from scratch and reduces reliance on large labeled datasets.
  • Motivation: Transformers use self-attention to improve parallelization and model long-term contexts compared with CNNs and RNNs.The survey presents Transformers as a response to locality bias and sequential processing limitations.
  • Survey Scope: The survey reviews self-supervised learning, pretraining methods and tasks, embeddings, and downstream adaptation methods.These topics establish the core concepts needed to understand T-PTLMs.
  • Survey Contributions: The proposed taxonomy categorizes T-PTLMs by pretraining corpus, architecture, self-supervised learning type, and extensions.It provides four perspectives for organizing different pretrained models.
  • Survey Contributions: The survey also taxonomizes downstream adaptation methods and reviews intrinsic and extrinsic benchmarks, useful libraries, and future research directions.The covered libraries range from Hugging Face Transformers to Transformer-Interpret.

2 SELF-SUPERVISED LEARNING (SSL)

Self-supervised learning uses unlabeled data to learn general representations without human annotation, addressing key limitations of supervised learning. The survey distinguishes generative, contrastive, and adversarial SSL and describes multi-task pretraining through weighted losses.

  • Why Self-Supervised Learning?: Supervised learning depends heavily on expensive labeled instances, can suffer from spurious correlations, and struggles in data-starved domains.The survey specifically identifies Medical and Legal applications as label-limited settings.
  • Why Self-Supervised Learning?: SSL learns from large amounts of unlabeled data without human labels and can improve generalization ability.It is motivated by the expense, scarcity, and domain limitations of labeled data.
  • Pretraining Objectives: Multi-task SSL combines pretraining-task losses as L_SSL = λ1L_PT−1 + λ2L_PT−2 + ... + λmL_PT−m.The λ weights control the contribution of the m pretraining tasks; BERT uses MLM and NSP, whereas RoBERTa uses MLM alone.
  • Types of SSL: Generative SSL learns by decoding encoded input through autoregressive, autoencoding, or hybrid language models.Autoregressive models predict subsequent tokens, whereas autoencoding models predict masked tokens.
  • Types of SSL: Contrastive SSL learns by comparing inputs, while adversarial SSL identifies replaced, shuffled, or randomly substituted tokens.BERT’s NSP and ALBERT’s SOP are examples of contrastive SSL.

3.1 Pretraining

Pretraining transfers language knowledge learned from large unlabeled corpora to downstream NLP tasks, reducing the need to train task models from scratch. The process involves preparing data and vocabulary, designing tasks and methods, and choosing pretraining dynamics.

  • Motivation: Pretraining on large unlabeled text lets models learn common language representations before adapting to task-specific datasets.This approach provides a good initialization and can improve performance with fewer labeled instances.
  • Pretraining Steps: The five pretraining steps are preparing the corpus, generating vocabulary, designing tasks, choosing a method, and selecting pretraining dynamics.The listed workflow covers corpus construction, tokenization, learning objectives, training strategy, and optimization choices.
  • Pretraining Tasks: Pretraining tasks should capture meaningful semantics, provide substantial training signal, and resemble downstream tasks.Examples include sentence-order prediction, replaced-token detection, span-boundary prediction, and gap-sentence generation.
  • Pretraining Methods: Training from scratch with self-supervised learning is computationally expensive, motivating alternatives that combine self-supervision, knowledge distillation, or multiple corpora.The survey mentions KIPT, continual pretraining with new vocabulary, adapt-and-distill, and simultaneous pretraining as alternatives.
  • Pretraining Corpus: Pretraining corpora may come from general, social-media, or domain-specific text, whose noise, style, and vocabulary differ.The survey describes four corpus types in Figure 1, while the supplied passage names general, social-media, and specific-domain examples.
  • Pretraining Dynamics: Larger and better-curated corpora have driven model development from BERT's 16GB corpus to C4 at about 750GB and CC-100 at about 2.5TB.The survey also notes that deduplicating near-duplicate or repetitive text can improve the pretraining corpus.

3.2 Types of Pretraining Methods

The survey classifies pretraining methods by how models use general, domain-specific, task-specific, and inherited knowledge. These methods trade off computational cost, data availability, vocabulary coverage, task adaptation, and knowledge reuse.

  • Pretraining from Scratch (PTS): Pretraining from scratch initializes transformer parameters randomly and learns them using one or more self-supervised tasks over large unlabeled corpora.BERT, for example, uses masked language modeling and next sentence prediction; this approach requires substantial computational resources.
  • Continual Pretraining (CPT): Continual pretraining initializes an existing language model and further trains it on target-domain text, avoiding training all parameters from scratch.It is commonly used for social media, biomedical, legal, news, and networking domains.
  • Continual Pretraining (CPT): CPT can suffer when general-domain vocabulary lacks target-domain terms, causing domain-specific words to split into subwords and degrading downstream performance.BioBERT is cited as an example of biomedical adaptation with a general-domain vocabulary.
  • Simultaneous Pretraining (SPT): Simultaneous pretraining combines general and domain-specific text, upsampling domain-specific data to improve vocabulary coverage and balance training when in-domain text is limited.Japanese clinical BERT trained with SPT outperformed a Japanese clinical BERT trained from scratch.
  • Task Adaptive Pretraining (TAPT): Task Adaptive Pretraining uses a small amount of task-specific unlabeled text to learn fine-grained task knowledge alongside domain knowledge, and it can complement PTS or CPT.The survey reports that applying TAPT after PTS or CPT further improves model performance.
  • Knowledge Inherited Pretraining (KIPT): Knowledge Inherited Pretraining combines self-supervised learning with knowledge distillation to reuse knowledge from existing pretrained models when training a new model.Unlike conventional knowledge distillation, KIPT uses self-supervised learning on the pretraining corpus, and its student can be larger than its teacher.

3.3 Pretraining Tasks

Pretraining tasks create self-supervised signals from unlabeled text, ranging from directional and masked prediction to sentence-level, discriminative, span-based, and cross-lingual objectives.

  • Token-level objectives: CLM predicts the next word from left-to-right or right-to-left context, whereas MLM masks tokens to exploit bidirectional context.MLM avoids letting a token see itself, enabling bidirectional representation learning.
  • Token-level objectives: RTD identifies every token as replaced or not, using generator-produced corruptions to provide more training signal and avoid mask-token discrepancy.ELECTRA trains a generator with MLM and a discriminator with RTD.
  • Token-level objectives: RTS randomly substitutes 15% of tokens and matches RoBERTa’s MLM performance while requiring less training time and no separate generator.RTS is presented as a sample-efficient alternative to RTD.
  • Token-level objectives: SLM replaces tokens with random vocabulary items at probability 0.15, reducing the discrepancy between pretraining and fine-tuning stages.Like MLM, SLM predicts the corrupted tokens, but it avoids using a special mask token.
  • Cross-lingual objectives: TLM masks tokens in parallel sentence pairs so both sentences provide context, while ALM masks code-switched sentences generated by substituting translated phrases.The survey reports that ALM outperforms XLM and is a better alternative to TLM for cross-lingual pretraining.
  • Span and sentence objectives: SBO predicts masked contiguous spans from span boundaries and position embeddings, making it more challenging than standard MLM.Its prediction uses boundary tokens and position information rather than only masked-token vectors.
  • Span and sentence objectives: NSP classifies sentence pairs as consecutive or non-consecutive, while SOP classifies whether sentences are swapped to focus on sentence coherence.SOP was introduced partly because topic prediction in NSP was considered easier.

3.4 Embeddings

T-PTLMs represent text through character, sub-word, hybrid, or code embeddings, supplemented by auxiliary embeddings that encode position, sentence segments, language, entities, or medical attributes.

  • Text embeddings: Transformer-based models generally prefer character or sub-word embeddings because their inputs must be numerical vectors and sub-word units offer flexible vocabularies.The survey distinguishes character, sub-word, and combined character–sub-word representations.
  • Text embeddings: Sub-word vocabularies can be built with WordPiece, BPE, bBPE, Unigram, or SentencePiece tokenizers.Different models use different tokenizers to generate sub-word vocabularies.
  • Text embeddings: Vocabulary size balances sequence length against model size: smaller vocabularies create longer sequences, while larger vocabularies increase overall model size.Multilingual models may require larger vocabularies to accommodate more languages.
  • Text embeddings: Hybrid embeddings combine character and sub-word representations, as in CharBERT’s dual-channel CNN-based interaction module.The interaction module models relationships between the two embedding types.
  • Specialized embeddings: Domain-specific biomedical models can use medical-code embeddings, with vocabularies drawn from standard clinical ontologies.BERT-EHR, MedBERT, and BEHRT represent patient information as sequences of medical codes.
  • Auxiliary embeddings: Auxiliary embeddings add structured information: position marks token order, segments distinguish sentence pairs, and language embeddings identify input languages.Position and segment embeddings are common, while language embeddings support multilingual models.
  • Auxiliary embeddings: Specialized models also encode entity types, age and gender, or semantic groups to provide academic, temporal, demographic, or medical-concept information.OAG-BERT uses entity type embeddings, while medical models incorporate patient and ontology attributes.

4 TAXONOMY

The taxonomy organizes T-PTLMs by pretraining corpus, architecture, SSL type, and extensions, covering general, social-media, language-specific, multilingual, and domain-specific models.

  • Taxonomy perspectives: The proposed taxonomy classifies T-PTLMs from four perspectives: pretraining corpus, model architecture, type of SSL, and extensions.Figure 9 presents the taxonomy used to track model development.
  • Pretraining corpus: General-corpus models train on sources such as Books, Wikipedia, or collections assembled from multiple text sources.GPT-1 uses Books Corpus, while BERT and UniLM use English Wikipedia and Books Corpus.
  • Pretraining corpus: Social-media models address limited performance on informal, noisy text through training from scratch or continual pretraining.BERTweet was trained from scratch on 850M tweets, whereas continual pretraining was preferred when corpora were smaller.
  • Pretraining corpus: Multilingual models range from mBERT, trained on Wikipedia in 104 languages, to models using larger CC-100 and combinations of parallel and non-parallel data.The survey also identifies multilingual generative models such as mT5, mT6, and mBART.
  • Pretraining corpus: Multilingual models face a curse of multilinguality because limited capacity and low-resource representation prevent equal performance across languages.Adding languages beyond a limit can reduce model performance.
  • Pretraining corpus: The embedding barrier leaves low-resource languages underrepresented in multilingual vocabularies, causing longer tokenized sequences and more expensive training.Bangla accounts for less than 1% of the multilingual vocabulary described in the survey.
  • Pretraining corpus: Domain-specific models use continual pretraining to add knowledge from finance, legal, news, programming, dialogue, networking, academic, or biomedical corpora.Continual pretraining converges faster and is less expensive than training from scratch, but may retain unsuitable general-domain vocabulary.
  • Model architecture: Encoder-based models produce contextual representations for NLU, decoder-based models support autoregressive generation, and encoder-decoder models target sequence-to-sequence tasks.Encoder-decoder examples include MASS, T5, mT5, mT6, BART, mBART, PLBART, PEGASUS, and PALM.

4.3 SSL

The survey groups SSL in T-PTLMs into generative, contrastive, adversarial, and hybrid paradigms, with models often combining multiple pretraining objectives.

  • Generative SSL: Generative SSL learns by predicting next tokens, masked tokens, or original text reconstructed from corrupted input.CLM, MLM variants, and denoising autoencoding exemplify these scenarios.
  • Contrastive SSL: Contrastive SSL learns by comparison and is used mainly for continual pretraining to improve sentence-level semantics.CERT, Mirror-BERT, and SimCSE are cited as examples.
  • Adversarial SSL: Adversarial SSL distinguishes corrupted tokens, including replaced or shuffled tokens, and can be used from scratch or during continual pretraining.ELECTRA uses RTD, while XLM-E uses multilingual and translation replaced-token detection.
  • Hybrid SSL: Hybrid SSL combines multiple paradigms, such as BERT’s generative MLM with contrastive NSP and CLINE’s generative, contrastive, and adversarial objectives.ALBERT and InfoXLM are additional examples of models combining SSL types.

4.4 Extensions

The survey discusses extensions that reduce T-PTLM size and cost, improve domain adaptation, address embedding and tokenization limitations, and increase pretraining efficiency.

  • Compression: Model compression methods include pruning, knowledge distillation, parameter sharing, and quantization to reduce storage, inference time, or parameter count.Pruning removes components, distillation trains compact students from teachers, parameter sharing reuses weights, and quantization uses fewer bits.
  • Embedding extensions: Sub-word embeddings avoid large vocabularies and OOV problems but cannot encode fine-grained character information and are brittle to typos.Character-based models such as CharacterBERT and CharBERT are proposed to address these drawbacks.
  • Green T-PTLMs: Continual pretraining adapts general models to domains or knowledge bases but is expensive in hardware, runtime, and CO2 emissions.Green models reduce this cost by extending vocabularies or adding domain-specific components instead of fully retraining the model.
  • Sentence embeddings: Sentence-embedding extensions use supervised or self-supervised objectives because pooled BERT sentence embeddings are not effective and supervised models can struggle with limited or shifted data.Examples include Siamese fine-tuning, mutual-information maximization, contrastive learning, and denoising autoencoding.
  • Tokenization-free T-PTLMs: Tokenization-free models operate directly on character sequences, avoiding language-specific tokenizers, large vocabularies, and some noise issues.CANINE applies convolutions to shorten character sequences before transformer encoding and uses the BERT pretraining tasks.
  • Efficient T-PTLMs: DeBERTa improves pretraining efficiency through disentangled attention and an enhanced masked decoder, outperforming RoBERTa while using 78GB rather than 160GB of data.ConvBERT combines global self-attention with span-based dynamic convolution for local dependencies.

5 DOWNSTREAM ADAPTATION METHODS

The survey covers feature-based, fine-tuning, and prompt-based downstream adaptation, emphasizing intermediate, multitask, parameter-efficient, and pruning-based variants.

  • Adaptation approaches: Pretrained models support feature-based adaptation, fine-tuning, and prompt-based tuning for downstream tasks.Feature-based adaptation supplies contextual embeddings to task-specific models, while fine-tuning minimizes task-specific loss.
  • Fine-tuning: Fine-tuning adapts model weights with task-specific loss because downstream performance requires task-specific knowledge.All transformer layers can change, with larger changes generally occurring in higher layers.
  • Fine-tuning: T5 uses a unified text-to-text format across tasks, whereas BERT, RoBERTa, and ELECTRA require task-specific layers during fine-tuning.
  • Intermediate fine-tuning: Intermediate or multitask fine-tuning can reduce vanilla fine-tuning’s overfitting problems on small datasets by adding domain- or task-specific knowledge.Intermediate fine-tuning uses a labeled intermediate dataset before adaptation to the target task.
  • Intermediate fine-tuning: Task-adaptive intermediate fine-tuning improved BERT’s GLUE score by 1.4 when using a general NLI dataset, with larger improvements when target labels were scarce.The source and target tasks may be the same or related even when their domains differ.
  • Intermediate fine-tuning: Intermediate fine-tuning does not always improve transferability; high-level inference and reasoning tasks such as NLI and QA tend to be more useful intermediate tasks.A large-scale study examined 110 intermediate–target task combinations.
  • Parameter-efficient fine-tuning: Adapter fine-tuning matches fully fine-tuned performance using only 3% of task-specific parameters, while AdapterFusion can outperform full fine-tuning and single- or multitask adapters.Adapters insert small trainable modules instead of updating the whole pretrained model.

6 EVALUATION

The survey distinguishes intrinsic evaluation of pretrained knowledge from extrinsic evaluation of downstream generalization, covering factual, linguistic, multilingual, social-media, and domain benchmarks.

  • Evaluation types: Intrinsic evaluation probes syntactic, semantic, factual, or common-sense knowledge encoded during pretraining, while extrinsic evaluation measures downstream task performance.Intrinsic findings can inform the design of better pretraining tasks.
  • Intrinsic evaluation: LAMA evaluates factual and common-sense knowledge by converting relation facts or question-answer pairs into fill-in-the-blank queries.The model is judged by whether it assigns the highest probability to the ground-truth token.
  • Intrinsic evaluation: LAMA is limited by single-token entities, vocabulary dependence, English-only coverage, and easily guessed triples; XLAMA expands coverage to 53 languages and multi-token entities.XLAMA also uses relation-specific candidate sets rather than the entire vocabulary.
  • Extrinsic evaluation: Extrinsic benchmarks standardize cross-task generalization using datasets, leaderboards, and a single metric.GLUE and SuperGLUE evaluate English natural-language understanding, with SuperGLUE adding more challenging QA, WSD, and coreference tasks.
  • Extrinsic evaluation: GENIE, GEM, and GLGE target English NLG, while XGLUE and XTREME target cross-lingual evaluation across NLU and NLG settings.XTREME-R extends XTREME to ten challenging NLU tasks and 50 languages.
  • Extrinsic evaluation: TweetEval and UMSAB evaluate social-media models through tweet classification, with UMSAB covering eight languages and TweetEval covering English.Additional benchmarks target individual languages, code-switching, and specialized domains.

7 USEFUL LIBRARIES

The survey summarizes libraries for training, evaluating, adapting, and accelerating inference with transformer-based pretrained language models.

  • Training and evaluation: Transformers and Fairseq support model training and evaluation, while SimpleTransformers, HappyTransformer, and AdaptNLP simplify these workflows.
  • Inference: FastSeq, DeepSpeed, FastT5, OnnxT5, and LightSeq are presented as libraries for improving inference speed.

8 DISCUSSIONS AND FUTURE DIRECTIONS

The discussion identifies efficiency, representation, adaptation, evaluation, robustness, and responsible-use challenges that motivate future improvements to T-PTLMs. It highlights approaches addressing pretraining cost, position encoding, parameter efficiency, noise sensitivity, domain adaptation, and model risks.

  • Efficient Pretraining Tasks: Sample-efficient tasks use every token, whereas masked language modelling uses only the 15% of tokens that are masked.RTD, RTS, and STD are described as early attempts to define pretraining objectives over all tokens.
  • Efficient Models: Pretraining efficiency remains a central challenge because large T-PTLMs require extensive computation and unlabeled data.DeBERTa achieves better performance than RoBERTa using 78GB of data, while ConvBERT outperforms ELECTRA using one-quarter of its pretraining cost.
  • Better Position Encoding Mechanisms: Position encoding must address the trade-off between absolute embeddings’ ease of implementation and relative embeddings’ sequence-length robustness.CAPE is presented as combining advantages of both approaches.
  • Beyond Vanilla Fine-tuning: Parameter-efficient adaptation reduces deployment costs by updating small task-specific components instead of maintaining fully fine-tuned copies.Prefix-tuning requires 0.1% of task-specific parameters, compared with 3% for adapter-based fine-tuning.
  • Benchmarks: Existing benchmarks do not cover all evaluation scenarios across general, domain-specific, monolingual, and multilingual models.The survey therefore identifies benchmark coverage as an ongoing limitation for evaluating T-PTLM progress.
  • Robustness and Responsible Use: Future improvements also target robustness, domain vocabulary, privacy, bias, and fine-tuning stability.The discussion links brittleness to sub-word embeddings, notes vocabulary limitations in continual pretraining, and describes risks from data leakage, bias, and unstable fine-tuning.

9 CONCLUSION

The paper provides a comprehensive review of transformer-based pretrained language models and organizes their methods, tasks, embeddings, adaptation approaches, benchmarks, libraries, taxonomy, and future directions.

  • 9 CONCLUSION: The survey reviews pretraining methods, pretraining tasks, embeddings, downstream adaptation methods, benchmarks, libraries, and future research directions.It also presents a new taxonomy for categorizing T-PTLMs.
Loading 2108.05542v2…