PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-trained Models for Natural Language Processing: A Survey

Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, Xuanjing Huang

arXiv:2003.08271v4cs.CLcs.LG

TL;DR

NLP has faced limited labeled data and costly annotation, while large neural models require substantial data to avoid overfitting. This survey synthesizes PTMs through a four-perspective taxonomy, reviews their architectures, tasks, extensions, adaptation, resources, and applications, and identifies future directions. It also reports that BERT-of-Theseus is 1.94× faster while retaining more than 98% of the source model’s performance.

  • Problem

    Large NLP models need much more labeled data to prevent overfitting, but building large-scale labeled datasets is costly, especially for syntax- and semantics-related tasks.

  • Method

    The survey comprehensively reviews PTMs and organizes them by representation type, model architecture, pre-training task, and scenario-specific extensions.

  • Results

    1.94× faster while retaining more than 98% performance of the source model, BERT-of-Theseus demonstrates efficient model compression through module replacement.

  • Takeaways & Limitations

    PTM knowledge can be transferred beyond full fine-tuning through shared models, adaptation modules, feature extraction, distillation, data augmentation, and external knowledge use.

  • Takeaways & Limitations

    Fine-tuning is parameter-inefficient because each downstream task has its own fine-tuned parameters, motivating more efficient transfer methods.

Abstract

from arXiv · show

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

1 Introduction

The survey reviews pre-trained models for NLP in response to limited labeled data and the overfitting challenges of large neural networks. It organizes PTMs through a four-perspective taxonomy, reviews their development and resources, and discusses future directions.

  • 1 Introduction: Limited labeled datasets make large neural networks prone to overfitting, motivating pre-training for NLP.Most supervised NLP datasets are small, while deep models have many parameters and require large datasets to train effectively.
  • 1 Introduction: PTMs learn universal language representations from large corpora that support downstream NLP tasks without training each model from scratch.The survey describes PTMs as a way to reuse learned representations for downstream tasks.
  • 1 Introduction: The survey provides a comprehensive review covering PTM background, architectures, pre-training tasks, extensions, adaptation approaches, and applications.These topics span the survey’s treatment of how PTMs are built, extended, transferred, and used.
  • 1 Introduction: Its taxonomy categorizes NLP PTMs by representation type, model architecture, pre-training task, and scenario-specific extensions.The four perspectives provide the survey’s organizing framework for existing PTMs.
  • 1 Introduction: The survey collects open-source implementations, visualization tools, corpora, and paper lists as resources for PTM research and use.These resources are intended to support practical engagement with PTMs.
  • 1 Introduction: The survey analyzes existing PTM limitations and suggests possible future research directions.Future directions are presented as a dedicated part of the survey’s contribution.

2 Background

Language representations map text into distributed vectors, progressing from static embeddings to contextual encoders. Pre-training addresses limited labeled data by learning from large unlabeled corpora, while Transformer-based PTMs offer strong capacity but require substantial data.

  • Language Representation Learning: Distributed representations encode text with low-dimensional vectors, and contextual embeddings vary according to the surrounding text.
  • Language Representation Learning: Non-contextual embeddings remain static across contexts and face out-of-vocabulary problems, motivating character-level and subword representations.
  • Contextual Embeddings: Contextual encoders derive each token representation from the whole input sequence through a neural encoder.
  • Contextual Encoders: Sequence, convolutional, recurrent, tree- or graph-based, and self-attention models provide different mechanisms for capturing contextual information.
  • Contextual Encoders: Transformers model dependencies between every pair of words and have become the mainstream PTM architecture, but they usually require large corpora and can overfit smaller datasets.
  • Why Pre-training?: Large unlabeled corpora enable pre-training to learn universal representations, improve initialization and generalization, accelerate convergence, and reduce overfitting on small data.

3 Overview of PTMs

The survey categorizes PTMs by their contextual encoders, pre-training tasks, and purposes, providing a taxonomy for comparing existing models.

  • Overview of PTMs: PTMs differ in how they use contextual encoders, which form the backbone for representing language.
  • Overview of PTMs: The taxonomy distinguishes PTMs according to their pre-training tasks.
  • Overview of PTMs: The survey also compares PTMs by their purposes and organizes the discussion around this taxonomy.

3.1 Pre-training Tasks

The survey organizes NLP pre-training tasks into supervised, unsupervised, and self-supervised learning, then describes language modeling and reconstruction-based objectives. It also explains how task designs address context limitations and pre-training–fine-tuning mismatch.

  • Task Categories: Pre-training tasks are grouped into supervised, unsupervised, and self-supervised learning.Self-supervised learning generates labels automatically by predicting parts of the input from other parts.
  • Masked Language Modeling: Masked language modeling predicts masked tokens from the remaining tokens, while sequence-to-sequence MLM uses an encoder-decoder architecture for autoregressive reconstruction.Sequence-to-sequence MLM can benefit question answering, summarization, and machine translation.
  • Language Modeling: Language modeling estimates sequence probabilities by predicting each token from its linguistic context.The context is encoded by a neural encoder, and the prediction layer estimates the vocabulary distribution; training uses maximum likelihood estimation.
  • Language Modeling: Bidirectional language modeling combines forward and backward unidirectional models to encode context from both directions.This addresses the limitation that unidirectional language models encode only leftward context and the current token.
  • Masked Language Modeling: PLM randomly permutes prediction order without changing natural token positions to reduce the mismatch caused by [MASK] tokens absent during downstream use.The objective replaces standard MLM as a response to the pre-training–fine-tuning gap.
  • Denoising Autoencoding: Denoising autoencoders corrupt text through masking, deletion, infilling, or sentence permutation and reconstruct the original sequence.These corruption strategies vary whether the model must recover tokens, positions, span lengths, or sentence order.
  • Next Sentence Prediction: Removing the NSP loss can match or slightly improve downstream performance when training uses blocks from a single document.Subsequent work also reported unreliable NSP impact and superiority of single-sentence training without NSP in some settings.

3.2 Taxonomy of PTMs

The survey proposes a four-perspective taxonomy for clarifying relationships among NLP pre-trained models: representation type, architecture, pre-training task type, and scenario-specific extensions.

  • Representation Type: The taxonomy categorizes PTMs by representation type, distinguishing non-contextual from contextual models.The distinction depends on the representation used for downstream tasks.
  • Architectures: The architecture perspective covers LSTM, Transformer encoder, Transformer decoder, and the full encoder-decoder Transformer.Transformer decoders use masked self-attention to prevent tokens from attending to future positions.
  • Pre-Training Task Types: The pre-training-task perspective classifies models according to the types of objectives used during pre-training.These task types are discussed in the survey’s pre-training-task section.
  • Extensions: The extensions perspective covers knowledge-enriched, multilingual or language-specific, multi-model, domain-specific, and compressed PTMs.These extensions target different application scenarios and are discussed separately in the survey.
  • Overview: Figure 3 presents the taxonomy with representative PTMs, while Table 2 distinguishes representative models in more detail.Together, they provide an overview and a more detailed comparison of existing models.

3.3 Model Analysis

Model analysis examines what linguistic and world knowledge PTMs capture, using probing and induction studies across non-contextual and contextual representations. The evidence shows both learned regularities and uneven capabilities across knowledge types and tasks.

  • Knowledge Analysis: Analysis of PTMs focuses on identifying linguistic and world knowledge captured by non-contextual and contextual embeddings.This analysis is motivated by the need to understand what knowledge models store and how it can be induced.
  • Non-Contextual Embeddings: Static word embeddings capture linguistic regularities, relation-specific offsets, compositionality, and taxonomic properties but fail on some attributive properties.Examples include vector relations between countries and capitals and the distinction between “dog is an animal” and “swan is white.”
  • Model Categories: Representative models span non-contextual CBOW or Skip-Gram and GloVe, contextual ELMo, GPT, and BERT, and architectures including LSTM and Transformer variants.The taxonomy also lists models such as SpanBERT, XLNet, and RoBERTa under Transformer encoders.
  • Linguistic Knowledge: BERT performs well on simple syntactic tasks and encodes subject-verb agreement, semantic roles, and recoverable dependency and constituency structures.Its layers also solve tasks in an order resembling traditional NLP pipelines.
  • Linguistic Knowledge: BERT is less capable on semantic and fine-grained syntactic tasks than on simple syntactic tasks.The passage reports this limitation relative to simpler syntactic evaluations.
  • World Knowledge: LAMA experiments found BERT contains world knowledge competitive with traditional information extraction methods, though later work questioned the evaluation.Subsequent studies also investigated relational and commonsense knowledge in BERT.

4 Extensions of PTMs

PTM extensions incorporate external knowledge, multilingual and cross-lingual objectives, multimodal inputs, and domain-specific data or tasks. These adaptations broaden coverage while exposing trade-offs between generality, language, and domain specialization.

  • Knowledge-Enriched PTMs: External knowledge can be injected during pre-training or incorporated into existing PTMs during downstream fine-tuning.The knowledge may be linguistic, semantic, commonsense, factual, or domain-specific.
  • Knowledge-Enriched PTMs: K-Adapter trains separate adapters for different pre-training tasks to support continual infusion of multiple knowledge types.This approach addresses catastrophic forgetting concerns associated with repeatedly updating PTM parameters.
  • Knowledge-Enriched PTMs: Knowledge-graph-conditioned language models condition prediction on knowledge graphs and show potential for pre-training.Examples include KGLM and LRLM.
  • Multilingual PTMs: Multilingual PTMs learn shared representations across languages, with approaches ranging from shared-vocabulary MLM to bilingual and cross-lingual objectives.mBERT uses monolingual Wikipedia MLM, while XLM and Unicoder add cross-lingual tasks.
  • Multilingual PTMs: XLM-R uses monolingual MLM on 2.5TB of CommonCrawl data across 100 languages and achieves state-of-the-art results on multiple cross-lingual benchmarks.The reported benchmarks include XNLI, MLQA, and NER.
  • Cross-Lingual Generation: Multilingual generation models typically pre-train encoder and decoder jointly, supporting tasks such as translation and cross-lingual summarization.MASS and XNLG use sequence-to-sequence or staged objectives for multilingual generation.
  • Language-Specific PTMs: Single-language PTMs have been reported to significantly outperform multilingual results in some settings.The survey also describes language-specific models and adaptations for languages such as French, Finnish, Russian, and Chinese.
  • Multimodal PTMs: Multimodal PTMs extend pre-training to video-text, image-text, and audio-text pairs for tasks including captioning, visual question answering, and speech question answering.These systems use shared or separate encoders and extended pre-training objectives.

4.5 Model Compression

PTM compression reduces model size or inference cost through pruning, quantization, parameter sharing, distillation, module replacement, and early exit. These approaches trade structural compactness, hardware requirements, or inference efficiency against retained performance.

  • Five compression strategies are identified: pruning, quantization, parameter sharing, knowledge distillation, and module replacing.
  • Pruning removes neural-network components to reduce model size and speed inference, while quantization lowers parameter precision but often requires compatible hardware.
  • Parameter sharing reduces parameter counts, but generally does not improve inference computational efficiency; ALBERT can even require longer training and inference than BERT.
  • Knowledge distillation trains a small student to reproduce a fixed teacher's behavior through soft targets, other knowledge, or structurally different models.
  • 1.94× faster and more than 98% of source-model performance are reported for BERT-of-Theseus, which progressively replaces modules using one task-specific loss.
  • Early-exit methods train internal classifiers and design exit strategies so inputs can leave before traversing the entire Transformer.

5 Adapting PTMs to Downstream Tasks

Adapting PTMs requires choosing transfer settings, representations, and tuning strategies that fit the downstream task. The survey covers layer selection, feature extraction versus fine-tuning, intermediate or multitask transfer, adapters, and prompts.

  • 5.1 Transfer Learning: Adapting PTM knowledge to downstream tasks remains a key problem, framed as sequential transfer learning with labeled target-task data.
  • 5.1 Transfer Learning: PTM selection should match the downstream task's pre-training objective, architecture, and corpus, because these choices produce different effects across tasks.
  • 5.2 Choosing appropriate layers: Different layers encode different information, with syntactic information appearing earlier and higher-level semantic information appearing at higher layers.
  • 5.2 Choosing appropriate layers: Representations may come from embeddings, the top layer, or a learned soft combination of all layers before task-specific processing.
  • 5.2.3 To tune or not to tune?: Fine-tuning is generally more convenient than feature extraction, although feature extraction can benefit NLP tasks while requiring more complex task-specific architecture.
  • 5.3 Fine-Tuning Strategies: Other strategies include two-stage and multitask fine-tuning, gradual or sequential unfreezing, and discrete prompts that can support in-context or parameter-free task adaptation.
  • 5.3 Fine-Tuning Strategies: Adapters address fine-tuning's parameter inefficiency by keeping original PTM parameters fixed and adding small task-specific modules; PALs matched separately fine-tuned models with roughly 7 times fewer parameters.

6 Resources of PTMs

The survey points readers to online PTM resources and related surveys for further exploration. These resources include implementations, paper lists, visualization tools, and other materials.

  • Online PTM repositories provide third-party implementations, paper lists, visualization tools, and other related resources.
  • Several additional survey papers on PTMs for NLP are identified as related resources.

7 Applications

PTMs are applied across major NLP applications, including evaluation, question answering, sentiment analysis, named entity recognition, translation, summarization, and adversarial robustness. The survey also catalogs benchmark and implementation resources for these applications.

  • General Evaluation Benchmark: PTM evaluation requires large-scale benchmarks, with GLUE covering nine natural language understanding tasks and SuperGLUE introducing more challenging and diverse formats.GLUE includes classification, similarity, and ranking tasks; SuperGLUE adds formats such as coreference resolution and question answering.
  • Question Answering: Question answering applications use PTMs for extractive span prediction, multi-round generative answering, and interpretable multi-hop reasoning.BERT reframes extractive QA as start- and end-span prediction, while other systems combine PTMs with adversarial training, rationale tagging, distillation, or selection modules.
  • Sentiment Analysis: BERT surpasses previous state-of-the-art models on SST-2 through fine-tuning, while fine-grained aspect-based sentiment analysis shows less significant improvement from direct application.ABSA methods therefore adapt BERT through auxiliary sentence construction, post-training, adversarial training, pooling, and joint aspect-sentiment learning.
  • Other NLP Applications: The survey discusses PTM applications in sentiment transfer and adversarial robustness alongside repositories, paper lists, visualization tools, and other resources.Sentiment transfer masks sentiment tokens and infills them using BERT with a target sentiment embedding; adversarial research addresses vulnerabilities in deep models.
  • Other NLP Applications: PTMs support NLP applications by supplying contextual representations for named entity recognition, initializing translation models, and improving summarization systems.Translation approaches initialize encoders or encoder-decoders, while summarization methods use sentence representations, intersentence Transformers, or summary-level matching.

8 Future Directions

The survey identifies future directions involving larger and more efficient PTMs, task-oriented pre-training and compression, knowledge transfer beyond fine-tuning, and improved interpretability and reliability. These directions address computational limits, task mismatch, parameter inefficiency, opacity, and adversarial vulnerability.

  • Upper Bound of PTMs: PTMs can be improved through more training steps, larger corpora, deeper architectures, and more efficient training techniques, but larger models increase training costs.Examples include Megatron-LM with 8.3 billion parameters and Turing-NLG with 17 billion parameters.
  • Architecture of PTMs: Transformer computation is quadratic in input length, so most PTMs are limited to sequences no longer than 512 tokens under GPU memory constraints.Future work includes more efficient Transformer variants, alternative non-Transformer architectures, and neural architecture search for longer-range context.
  • Task-oriented Pre-training and Model Compression: Task-oriented pre-training should address mismatches in architecture and data distribution, while model compression targets deployment on low-capacity devices and low-latency applications.Text generation may require encoder-decoder pre-training, whereas text matching benefits from sentence-pair objectives.
  • Knowledge Transfer Beyond Fine-tuning: Fine-tuning is parameter-inefficient because each downstream task has its own fine-tuned parameters; adapter modules and other transfer methods can share a PTM across tasks.The survey also identifies feature extraction, knowledge distillation, data augmentation, and external knowledge use as alternatives.
  • Interpretability and Reliability of PTMs: PTM decision-making remains non-transparent, and attention-based interpretability is controversial; adversarial vulnerability also makes reliability an open concern.The survey calls for further work on interpretability, reliability, and adversarial defenses to guide model usage and improvement.

9 Conclusion

The survey reviews PTMs across their foundations, architectures, pre-training tasks, extensions, adaptation methods, resources, and applications. It proposes a four-perspective taxonomy and identifies future research directions.

  • Conclusion: The survey provides a comprehensive overview of PTMs and proposes a taxonomy organized by representation type, model architecture, pre-training task, and scenario-specific extensions.It also suggests possible future research directions for PTMs.
Loading 2003.08271v4…