Pre-trained Models for Natural Language Processing: A Survey

1 Introduction

• Small supervised NLP datasets cause large neural networks to overfit and generalize poorly, limiting performance gains and favoring shallow models.The survey notes machine translation as an exception to the generally small datasets.Source: §1 · p. 1
• PTMs evolved from shallow, context-free word embeddings to deep contextual encoders that downstream tasks use to represent words in context.The survey names Skip-Gram and GloVe as first-generation examples, and CoVe, ELMo, OpenAI GPT, and BERT as second-generation examples.Source: §1 · pp. 1–2
• The survey reviews PTM background, architectures, tasks, extensions, adaptation, and applications; it also collects resources and analyzes limitations to suggest future directions.Source: §1 · p. 2
• The survey proposes a taxonomy that categorizes PTMs by representation type, model architecture, pre-training task, and extensions for specific scenarios.Source: §1 · p. 2

2.1 Language Representation Learning

• Language representations are low-dimensional vectors; static embeddings cannot distinguish a word’s senses by context, whereas contextual embeddings depend on the full input text.Subword and character-level approaches, including CharCNN [18], FastText [19], and BPE [20], address out-of-vocabulary limitations of static embeddings.Source: §2.1 · p. 2

2.2.1 Sequence Models

• Sequence models capture local context: convolutional models aggregate neighboring words [2], while recurrent models such as LSTMs and GRUs [22] can struggle with long-term dependencies.Bidirectional LSTMs or GRUs collect information from both sides of a word.Source: §2.2.1 · p. 3

2.2.2 Non-Sequence Models

• Non-sequence models use tree or graph structure, but building these structures is challenging and often depends on expert knowledge or external tools; self-attention instead learns word relations over a fully connected graph.Examples include Recursive NN, TreeL-STM [7], and GCN [24]; the Transformer [10] [25] adds positional embeddings, normalization, residual connections, and feed-forward layers.Source: §2.2.2 · p. 3

2.2.3 Analysis

• Sequence models are easier to train but have difficulty capturing long-range interactions, whereas Transformers model pairwise dependencies but can overfit modest datasets and need large corpora; Transformers became mainstream for PTMs.The paper attributes Transformers’ suitability for long-range dependencies to their ability to directly model dependencies between every pair of words.Source: §2.2.3 · p. 3

2.3 Why Pre-training?

• Pre-training uses large unlabeled corpora to learn reusable language representations, improving downstream initialization, generalization, and convergence while helping prevent overfitting on small datasets.The paper also frames pre-training as a way to address the high cost of building large labeled datasets.Source: §2.3 · pp. 3–4

2.4.1 First-Generation PTMs: Pre-trained Word Embeddings

• Early NLP PTMs included pre-trained embeddings: Collobert et al. [31] transferred embeddings learned from unlabeled data, while Mikolov et al. [11] introduced CBOW and Skip-Gram and GloVe used global co-occurrence statistics.These embeddings were effective but context-independent, leaving the rest of a downstream model to be learned from scratch.Source: §2.4.1 · p. 4

2.4.2 Second-Generation PTMs: Pre-trained Contextual Encoders

• Later PTMs pre-trained contextual encoders: ELMo [14] and contextual string embeddings served as fixed features, while ULMFiT [38] fine-tuned a language model and GPT and BERT [16] helped establish fine-tuning as mainstream.ULMFiT used general-domain language-model pre-training, target-data language-model fine-tuning, and target-task fine-tuning; modern PTMs increasingly use larger corpora, deeper architectures, and new pre-training tasks.Source: §2.4.2 · pp. 4–5

• This section describes pre-training tasks and presents a taxonomy of PTMs, while referring readers to Section 2.2 for contextual-encoder architectures.Source: §3 · p. 5

3.1 Pre-training Tasks

• Pre-training tasks support universal language representations and should be challenging with substantial training data; the survey groups them as supervised, unsupervised, and self-supervised learning.Self-supervised learning generates labels automatically and predicts part of an input from other parts, as in masked language modeling.Source: §3.1 · p. 5

3.1.1 Language Modeling (LM)

• Autoregressive language modeling estimates sequence probabilities from leftward context and can be trained by maximum likelihood on a large corpus, but each token representation lacks rightward context.Bidirectional LM combines forward and backward unidirectional language models to incorporate context in both directions.Source: §3.1.1 · pp. 5–6

3.1.2 Masked Language Modeling (MLM)

• MLM predicts masked tokens from the remaining sentence to overcome unidirectional context limits, but masking can create a pre-training/fine-tuning mismatch.Seq2Seq MLM instead uses an encoder-decoder to produce masked tokens autoregressively and can benefit sequence-to-sequence tasks such as question answering, summarization, and machine translation.Source: §3.1.2 · p. 6

3.1.3 Permuted Language Modeling (PLM)

• PLM replaces MLM by predicting selected targets in a randomly permuted token order while conditioning on other tokens and targets’ natural positions.Only the last few tokens are predicted in practice because convergence is slow, and target-aware representations use special two-stream self-attention.Source: §3.1.3 · pp. 6–7

3.1.4 Denoising Autoencoder (DAE)

• Denoising autoencoders reconstruct original text from corrupted input, using token masking or deletion, text infilling, sentence permutation, or document rotation.For text infilling, sampled spans are replaced by one [MASK] token and the model predicts how many tokens are missing.Source: §3.1.4 · p. 7

3.1.5 Contrastive Learning (CTL)

• Contrastive learning compares semantically similar and dissimilar text pairs; replaced-token detection instead classifies whether tokens were substituted, with ELECTRA training a generator then a discriminator.The generator is trained with MLM, initializes the discriminator, and is frozen during its training; only the discriminator is fine-tuned downstream, and RTD avoids the [MASK] mismatch.Source: §3.1.5 · pp. 7–8
• NSP predicts whether sentence pairs are consecutive, but its necessity is disputed; SOP swaps consecutive same-document segments, and ALBERT consistently outperforms BERT downstream.NSP examples use an actual next sentence half the time and a random sentence half the time; later work found removing NSP can match or slightly improve downstream performance.Source: §3.1.5 · p. 8

3.1.6 Others

• Other auxiliary pre-training tasks target factual knowledge, cross-lingual tasks, multimodal applications, or other specific tasks.Source: §3.1.6 · p. 8

• PTMs are classified by whether their downstream-task representations are non-contextual or contextual.Source: §3.2 · p. 8
• Backbone architectures include LSTM, Transformer encoder, Transformer decoder, and the full encoder-decoder Transformer; decoder attention masks future positions.The decoder uses triangular masked self-attention to prevent tokens from attending to future (right) positions.Source: §3.2 · p. 8
• PTMs are also categorized by their pre-training task types, discussed in Section 3.1.Source: §3.2 · p. 8
• The extensions category covers knowledge-enriched, multilingual or language-specific, multi-model, domain-specific, and compressed PTMs.The survey particularly introduces these extensions in Section 4.Source: §3.2 · p. 8

3.3.1 Non-Contextual Embeddings

• Static word vectors capture linguistic regularities and syntactic and semantic relationships, but distributional representations predict taxonomic properties better than attributive ones [11].Analogy and compositionality findings include vec(“China”) − vec(“Beijing”) ≈ vec(“Japan”) − vec(“Tokyo”) and vec(“Germany”) + vec(“capital”) being close to vec(“Berlin”) [11]. Word2vec embeddings also encode entity attributes that simple supervised models can predict with reasonable accuracy.Source: §3.3.1 · pp. 8, 10

3.3.2 Contextual Embeddings

• BERT performs well on simple syntactic tasks but less well on semantic and fine-grained syntactic tasks.Studies also find that BERT handles tasks in an order resembling NLP pipelines, contains subject-verb agreement and semantic-role knowledge, and encodes syntactic structure; its linguistic features appear in separate semantic and syntactic subspaces, and it distinguishes word senses finely [123].Source: §3.3.2 · p. 10
• LAMA cloze probes found BERT contains world knowledge competitive with traditional information-extraction methods.Because query generation is simple, LAMA may measure only a lower bound on language-model knowledge [130], and subsequent work questioned its findings [132]; other studies induce relational and commonsense knowledge from BERT for downstream tasks.Source: §3.3.2 · pp. 10–11

4.1 Knowledge-Enriched PTMs

• Knowledge-enriched PTMs incorporate linguistic, semantic, commonsense, factual, or domain knowledge through pre-training objectives, independent adapters, downstream injection, or knowledge-graph-conditioned language models.Most knowledge-injection approaches update PTM parameters, while K-Adapter trains separate adapters for different pre-training tasks to support continual knowledge infusion.Source: §4.1 · p. 11

4.2.1 Multilingual PTMs

• mBERT generalizes cross-lingually from monolingual MLM over 104 languages, while XLM adds translation language modeling and XLM-R scales monolingual MLM training.Multilingual generation models jointly pre-train encoder and decoder components: MASS uses monolingual Seq2Seq MLM, XNLG uses two-stage cross-lingual objectives, and mBART uses Seq2Seq denoising across 25 languages.Source: §4.2.1 · pp. 11–12

4.2.2 Language-Specific PTMs

• Single-language PTMs significantly outperform multilingual results in recent work, with models developed for French, Finnish, Dutch, Arabic, and other language-specific settings.For Chinese, larger-granularity and multigranularity word representations have shown success; transfer learning has also adapted multilingual PTMs to Russian.Source: §4.2.2 · p. 12

4.3 Multi-Modal PTMs

• Cross-modal PTMs commonly encode visual and linguistic features from large paired corpora using visual MLM, masked visual-feature modeling, and visual-linguistic matching.Source: §4.3 · p. 12

4.3.1 Video-Text PTMs

• Video-text PTMs convert video and speech into visual and linguistic token sequences, then train a Transformer encoder for downstream tasks such as video captioning.UniViLM further proposes decoder pre-training with generation tasks used downstream.Source: §4.3.1 · p. 12

4.3.2 Image-Text PTMs

• Image-text PTMs use separate or unified Transformer encoders, with MLM and image-text matching objectives after images are represented as regions.Source: §4.3.2 · p. 12

4.3.3 Audio-Text PTMs

• SpeechBERT pre-trains a single Transformer encoder on speech and text with MLM, then fine-tunes it for end-to-end speech question answering.Source: §4.3.3 · p. 12

4.4 Domain-Specific and Task-Specific PTMs

• Domain- and task-specific PTMs address general-corpus limitations with specialty-trained models, application adaptation, and task-oriented pre-training objectives.Examples include BioBERT, SciBERT, and ClinicalBERT; adaptations target biomedical entity normalization, patent classification, progress-note classification, and keyword extraction, while objectives include sentiment LA-MLM, GSG for summarization, and Noisy Words Detection for disfluency detection.Source: §4.4 · p. 13

4.5 Model Compression

• Five PTM compression approaches are pruning, quantization, parameter sharing, knowledge distillation, and replacing modules with compact substitutes.The survey frames these approaches as ways to reduce model size and increase computation efficiency for PTMs that are difficult to deploy on online services or resource-restricted devices.Source: §4.5 · p. 13

4.5.1 Model Pruning

• Pruning removes network components such as weights, neurons, layers, channels, or attention heads to reduce model size and speed inference.Prior work explored when and how to prune, including pruning entire Transformer self-attention heads.Source: §4.5.1 · p. 13

4.5.2 Quantization

• Quantization compresses higher-precision parameters into lower precision, but often requires compatible hardware.Shen et al. and Zafrir et al. [105] focus on quantization.Source: §4.5.2 · p. 13

4.5.3 Parameter Sharing

• Parameter sharing reduces parameter count; ALBERT [63] combines cross-layer sharing with factorized embeddings, but sharing generally does not improve inference efficiency.ALBERT’s training and inference time are even longer than standard BERT’s.Source: §4.5.3 · p. 13

4.5.4 Knowledge Distillation

• Knowledge distillation trains a smaller student to reproduce a usually well-pretrained teacher’s behavior, using soft targets, other knowledge, or transfer to different structures.Distillation from soft targets commonly approximates teacher logits; examples also transfer intermediate representations and attention, while alternative student structures can include RNNs or CNNs.Source: §4.5.4 · pp. 13–14

4.5.5 Module Replacing

• BERT-of-Theseus is 1.94× faster while retaining more than 98% of its source model’s performance.Theseus Compression progressively substitutes source-model modules with modules having fewer parameters and uses one task-specific loss function rather than knowledge distillation.Source: §4.5.5 · p. 14

4.5.6 Early Exit

• Early exit reduces inference time by allowing input-dependent stopping at an off-ramp, with methods training internal classifiers and an exit strategy.Exit criteria include prediction entropy, maximum softmax score, unchanged predictions across successive layers, voting, and window-based uncertainty for token-level sequence labeling.Source: §4.5.6 · pp. 14–15

5.1 Transfer Learning

• Adapting PTMs to downstream tasks is sequential transfer learning: tasks are learned in sequence, and the target task has labeled data.Transfer learning adapts knowledge from a source task or domain to a target task or domain.Source: §5.1 · p. 15

5.2.1 Choosing appropriate pre-training task, model architecture and corpus

• For a target task, choose a PTM whose pre-training task, architecture, and corpus fit the task: for example, NSP can benefit QA and NLI, while BERT is difficult to use for language generation.Downstream data distribution should approximate that of the PTM.Source: §5.2.1 · p. 15

5.2.2 Choosing appropriate layers

• Representations differ by layer: syntactic information appears earlier and high-level semantics higher, while task models can use top-layer representations or soft mixtures of all layers.Embedding-only transfer misses higher-level information such as word sense.Source: §5.2.2 · pp. 15–16

5.2.3 To tune or not to tune?

• Fine-tuning is generally more convenient and general across downstream tasks than feature extraction, which requires more complex task-specific architectures.Both approaches can significantly benefit most NLP tasks.Source: §5.2.3 · p. 16

5.3 Fine-Tuning Strategies

• Fine-tuning can vary substantially across random seeds [193], motivating strategies such as two-stage transfer, multi-task tuning, self-ensemble and distillation, and gradual or sequential unfreezing.Related-domain further pretraining achieved state-of-the-art performance on eight text-classification datasets; related supervised intermediate tasks improved BERT, GPT, and ELMo [194].Source: §5.3 · pp. 16–17
• PAL-equipped shared BERT matched separately fine-tuned GLUE models with roughly 7 times fewer parameters, while adapters add few trainable parameters per task and keep original weights fixed.Adapters support adding new tasks without revisiting previous ones.Source: §5.3 · p. 16

5.3.1 Prompt-based Tuning

• Discrete prompts reformulate inputs to help PTMs model downstream tasks; GPT-3 in-context learning can achieve competitive performance without parameter tuning [59], and manual prompts enable BERT entity prediction without training.PET applies discrete prompts to classification and entailment, while LPAQA, AutoPrompt, and LM-BFF automate prompt generation,, [72].Source: §5.3.1 · p. 17
• Continuous prompts optimize prompt vectors rather than concrete word prompts; they can outperform concrete prompts on relational tasks, and prompt tuning becomes more competitive as model scale increases,, [205].Prefix-Tuning achieved comparable fully supervised performance and outperformed model fine-tuning in few-shot settings [74]; WARP and P-Tuning are other continuous-prompt approaches [73], [75].Source: §5.3.1 · p. 17

• Table 5 lists PTM repositories with third-party implementations, paper lists, visualization tools, and related resources, alongside other PTM surveys for NLP [212] [173].Source: §6 · p. 17

7.1 General Evaluation Benchmark

• Comparable PTM assessment requires large-scale benchmarks; GLUE covers nine NLU tasks and robustness/generalization, while SuperGLUE adds harder, more diverse formats as GLUE headroom eroded.GLUE includes single-sentence, pairwise classification, text-similarity, and ranking tasks, and uses an evaluation server because test labels are withheld.Source: §7.1 · p. 18

7.2 Question Answering

• QA spans SQuAD extractive, CoQA multi-round generative, and HotpotQA multi-hop tasks; PTM encoders support span prediction [16] [218], rationale-based generative QA [219], and SAE selection.The surveyed methods initialize QA model encoders from PTMs, while other parameters are randomly initialized.Source: §7.2 · pp. 18–19

7.3 Sentiment Analysis

• BERT surpassed prior SST-2 models [16] and transfer learning set a Japanese sentiment-analysis state of the art, while ABSA work improves on BERT’s weaker direct gains.ABSA approaches include domain and cross-domain post-training [223], adversarial training [224], intermediate-layer pooling [225], joint detection and classification, and SentiLR’s linguistic Label-Aware MLM [79]; “Mask and Infill” uses BERT for sentiment transfer.Source: §7.3 · p. 19

7.4 Named Entity Recognition

• NER uses pretrained character- and word-level embeddings and layer mixtures [14], while BERT enables first-BPE labeling, multilingual zero-shot transfer, CPU-sized inference, and biomedical/domain NER [16].Layer-wise pruning and dense connections were also used to speed ELMo inference for NER.Source: §7.4 · p. 19

7.5 Machine Translation

• NMT gains from pretrained initialization [36], BERT memory and staged fine-tuning, and joint Seq2Seq pretraining with MASS [41] or mBART [61] [50].Reported improvements span unsupervised and English-Romanian supervised MT, out-of-domain data, and sentence- and document-level translation.Source: §7.5 · pp. 19–20

7.6 Summarization

• BERTSUM uses intersentence Transformer layers for extractive summarization and two-staged fine-tuning for abstractive summarization; MATCHSUM uses Siamese-BERT semantic matching and achieves 44.41 ROUGE-1 on CNN/DailyMail using base BERT.Source: §7.6 · p. 20

7.7 Adversarial Attacks and Defenses

• PTMs are vulnerable to text attacks, and BERT-Attack outperformed prior strategies in success rate and perturbation percentage while preserving fluency and meaning [239] [240]; adversarial pretraining or fine-tuning can improve robustness and generalization [242].Text adversarial examples must mislead models while remaining imperceptible to people, grammatical, and semantically consistent.Source: §7.7 · p. 20

Upper Bound of PTMs

• PTMs may improve with more training and larger corpora, but general-purpose models are costly, motivating deeper models and more efficient training designs.Megatron-LM [243] and Turing-NLG illustrate deeper, larger models; proposed efficiency improvements include architectures, self-supervised tasks, optimizers, and training skills, with ELECTRA [56] offered as an example.Source: §8 · p. 21

Architecture of PTMs

• Transformer computation grows quadratically with input length, and GPU memory limits most PTMs to sequences of 512 tokens, motivating more efficient and alternative architectures.The survey also suggests neural architecture search (NAS) for designing deep architectures.Source: §8 · p. 21

Task-oriented Pre-training and Model Compression

• Task-oriented pre-training and model compression may address task mismatch and deployment constraints, including low-capacity devices and low-latency applications.Generation may require pre-training both encoder and decoder, whereas text matching benefits from sentence-pair tasks. Compression for NLP PTMs is just beginning, and the Transformer’s fully connected structure makes it challenging.Source: §8 · p. 21

Knowledge Transfer Beyond Fine-tuning

• Fine-tunable adaptation modules can let one shared PTM serve multiple tasks, while feature extraction, distillation, data augmentation, and external-knowledge use offer other transfer routes.Fine-tuning is parameter-inefficient because each task has its own parameters; more efficient transfer methods are expected.Source: §8 · p. 21

Interpretability and Reliability of PTMs

• PTMs’ opaque decision-making and vulnerability to adversarial attacks make interpretability and reliability important open directions.Analyses of linguistic and world knowledge provide some transparency, but attention-based interpretability remains controversial; adversarial defenses could improve robustness, and further work can guide PTM use and improvement.Source: §8 · pp. 21–22

9 Conclusion

• The survey comprehensively reviews PTMs for NLP, covering background, architectures, pre-training tasks, extensions, adaptation approaches, resources, and applications.Source: §9 · p. 22
• It proposes a new taxonomy of PTMs organized from four different perspectives.Source: §9 · p. 22
• The survey suggests several possible future research directions for PTMs.Source: §9 · p. 22