PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, Dan Roth

arXiv:2111.01243v1cs.CLcs.AIcs.LG

TL;DR

NLP research lacks a unified account of how large pre-trained language models support diverse tasks and complementary data-generation uses. This survey organizes recent work across fine-tuning, prompting, text generation, and PLM-based data generation, then synthesizes limitations and future directions. It concludes that these paradigms and combinations of PLMs have shown promise across NLP applications.

  • Problem

    The field needs a comprehensive account of how large pre-trained language models are used across NLP tasks and paradigms.

  • Method

    The paper surveys pre-train-then-fine-tune, prompt-based learning, NLP as text generation, and PLM-based data generation.

  • Results

    The surveyed applications show promise, and combining multiple PLMs yields further improvements across a range of NLP tasks.

  • Takeaways & Limitations

    The survey provides fundamental concepts and a comprehensive view of the paradigm shift toward PLM-based NLP.

  • Takeaways & Limitations

    Instruction tuning’s improvements on held-out zero-shot tasks may depend on using very large PLMs, while prompt success may reflect training-data patterns rather than meaningful instructions.

Abstract

from arXiv · show

Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.

1 Introduction

Large pre-trained language models shift NLP toward learning general representations from abundant unlabeled text and adapting them across tasks. The survey organizes this work around fine-tuning, prompting, text generation, and PLM-based data generation.

  • Motivation: PLMs learn generic language representations from abundant unlabeled text that can be shared across NLP tasks.This addresses the need to relearn representations for each task and the limitations of task-specific training data.
  • Three paradigms: Pre-train then fine-tune uses general-purpose pre-training followed by a small amount of task-specific adaptation.
  • Three paradigms: Prompt-based learning reformulates tasks to resemble pre-training or proxy objectives, often enabling few-shot approaches.
  • Three paradigms: NLP as text generation reformulates tasks as text generation to leverage knowledge encoded in generative PLMs such as GPT-2 and T5.
  • Complementary method: PLMs can automatically generate silver-labeled or auxiliary data that is added to or supports target-task training.
  • Scope: The survey covers these paradigms, generated-data methods, limitations, and future research directions across a broad range of NLP tasks.

2 Paradigm 1: Pre-Train then Fine-Tune

Pre-training and fine-tuning replace separate task-specific representation learning with a shared language-modeling objective followed by downstream adaptation. The paradigm became prominent with ELMo and ULMFiT and later benefited from Transformer-based modeling and scalable self-supervision.

  • Core paradigm: Pre-training learns representations from massive unlabeled data using a shared language-modeling task, followed by task-specific fine-tuning.
  • Challenges: The paradigm’s adoption was delayed by the difficulty of choosing suitable self-supervised tasks and the need for much larger models and corpora.
  • Core paradigm: Self-supervised pre-training derives targets from unlabeled text, allowing training to scale with the web’s near-infinite text supply.
  • Historical development: ELMo and ULMFiT helped establish pre-training and fine-tuning in NLP using bidirectional or standard LSTM language-modeling objectives.
  • Transformer foundations: Transformer multi-head self-attention captures long-range dependencies efficiently and produces increasingly expressive representations across layers.

2.2 Modern Pre-Trained Language Models

Modern PLMs fall into autoregressive, masked, and encoder-decoder classes, distinguished by their architectures and training objectives. These models support next-token prediction, bidirectional masked recovery, or flexible text-to-text generation.

  • Autoregressive Language Models: Autoregressive models predict each next token from preceding tokens using stacked Transformer decoder layers with masked self-attention.
  • Autoregressive Language Models: GPT fine-tuning achieved 45.4 on CoLA versus 35.0 for the previous state of the art.
  • Modern variants: GPT-2 and GPT-3 use generative designs for prompting or direct language generation rather than the original GPT fine-tuning approach.
  • Masked Language Models: Masked language models recover randomly masked tokens using bidirectional context, with BERT, RoBERTa, and XLM-R as representative examples.
  • Encoder-Decoder Language Models: Encoder-decoder models generate output sequences from input sequences, providing a flexible text-in/text-out formulation.
  • Encoder-Decoder Language Models: BART and T5 pre-train encoder-decoder models by reconstructing original sequences from corrupted inputs such as rotated, permuted, or masked text.

2.3 Pre-Training Corpora

Pre-training corpora differ in source, size, quality, and domain, all of which shape PLM development and downstream applicability. Corpus sizes and diversity have increased substantially, while filtering and domain matching remain important.

  • Corpus characteristics: Pre-training corpora are distinguished by both dataset size and source-data quality.
  • Scaling and quality: When dataset quality is held constant, model and dataset size typically drive primary performance gains, although genre matching can matter more in some settings.
  • Scaling and quality: Filtering heuristics improve performance on large crawled datasets compared with using unfiltered data.
  • Domain selection: Domain-specific PLMs can substantially outperform general-purpose models in their intended domains.

2.4 Fine-Tuning: Applying PLMs to NLP Tasks

Fine-tuning adapts pre-trained language models to downstream NLP tasks, ranging from frozen embeddings and simple prediction heads to customized architectures and parameter-efficient methods. These strategies trade task-specific modeling capacity against data and computational requirements.

  • Frozen PLMs: Frozen PLMs provide contextual embeddings without updating language-model weights, useful when labeled data or computational resources are insufficient.They can also reduce training complexity for complex or unsupervised tasks.
  • Fine-tuning the PLM: Standard fine-tuning updates some or all PLM layers jointly with simple prediction heads that map token or span representations to task labels.This approach suits sequence classification, sequence tagging, and span extraction tasks.
  • Customized models: Customized models add substantial task-specific architecture above a fine-tuned PLM, particularly for structure prediction, parsing, sequence tagging, and encoder-based sequence-to-sequence tasks.However, one machine-translation study found no significant improvement over frozen BERT word embeddings.
  • Efficient fine-tuning approaches: Adapter-based fine-tuning updates small per-task modules, typically fewer than 5% of PLM weights, while keeping the shared PLM frozen.Adapters support weight sharing, module reuse and combination, and prevention of catastrophic forgetting.
  • Efficient fine-tuning approaches: Selective masking and pruning can retain performance by learning task-specific sparse parameter subsets or fine-tuning only the most sensitive layers.Masking top-down is more effective than masking bottom-up in the cited results, while setting 14% of pre-trained parameters to zero can retain performance.

3 Paradigm 2: Prompt-based Learning

Prompt-based learning adds natural-language instructions or task formulations to guide pre-trained models toward desired behaviors. The survey distinguishes instruction and demonstration learning, template-based learning, and proxy-task approaches.

  • Prompt-based learning: Prompting adds natural-language text to inputs or outputs to encourage pre-trained models to perform specific tasks.In-context prompting may avoid parameter updates and reduce computational requirements.
  • Prompt-based learning: The survey discusses learning from instructions and demonstrations, template-based learning, and learning from proxy tasks.Table 2 provides example prompt designs for learning from instructions.

3.1 Learning from Instructions and Demonstrations

Instruction-based learning uses natural-language instructions and demonstrations to teach PLMs multiple tasks and improve zero- and few-shot performance, but its effectiveness depends on model scale and instruction design.

  • Instruction-based learning: Natural-language instructions can teach varied tasks in a text-to-text format, but early approaches required large amounts of labeled data.
  • Instruction-based learning: GPT-3 performs few-shot tasks through in-context instructions and input/output examples without gradient updates.The approach requires a very large language model and does not scale beyond a few examples.
  • Instruction-based learning: Prompt-based approaches include instruction learning, template-based learning, and proxy-task learning, with Figure 3 illustrating their distinct input designs.
  • Instruction-based learning: Instruction tuning on more than 60 datasets across 12 task groups improves zero- and few-shot performance on unseen tasks.Wei et al. report gains over a standard language model, while Mishra et al. report improved performance on unseen crowdsourced tasks using decomposed instructions.
  • Instruction-based learning: Instruction tuning yields significant held-out-task gains in zero-shot settings primarily with very large PLMs, such as 68B-parameter models.The cited passage identifies scaling as a limitation for broad applicability and notes weaker applicability to smaller models.
  • Instruction-based learning: Instruction learning may use detailed task descriptions containing definitions, cautions, required output properties, and positive and negative examples.

3.2 Template-based Learning

Template-based learning reformulates NLP tasks as language-modeling problems using discrete or continuous prompts, enabling label prediction, generation, and flexible prompt composition.

  • 3.2 Template-based Learning: Template-based learning converts NLP tasks into forms closer to PLM pre-training, reducing or eliminating task-specific training examples.
  • 3.2.1 Template Design: Cloze-style templates ask PLMs to predict missing words or phrases and map those predictions to task-specific labels.For textual entailment, PET maps predictions such as “yes” or “no” to entailment labels.
  • 3.2.1 Template Design: Multiple-choice templates probe commonsense knowledge by asking the PLM to select among candidate hypotheses.
  • 3.2.1 Template Design: Prefix prompts prepend task-specific vectors and suit generation or next-word prediction because they match autoregressive left-to-right modeling.
  • 3.2.1 Template Design: Demonstration learning appends a few labeled examples to templates to make prompts more informative.
  • 3.2.2 Template Construction: Prompts may be manually crafted, automatically generated in discrete or continuous spaces, or combined through ensembling, augmentation, decomposition, and composition.Continuous prefix tuning can fine-tune only 0.1% of total model parameters.
  • 3.2.2 Template Construction: Prompt quality strongly affects accuracy, with different manually designed prompts ranging from near chance to near state-of-the-art performance.
  • 3.2.2 Template Construction: Prompt ensembling combines multiple prompts by averaging, weighted averaging, or majority voting, while decomposition creates separate prompts for subproblems.

3.3 Learning from Proxy Tasks

Proxy-task learning recasts target NLP problems as supervised natural-language understanding tasks such as question answering or textual entailment, improving transfer to zero- and few-shot settings.

  • 3.3 Learning from Proxy Tasks: Proxy-task learning uses supervised NLU tasks instead of self-supervised language modeling and can improve zero- and few-shot performance with relatively small PLMs.Knowledge-rich classification benefits because prompts preserve the meaning of class labels rather than treating them as indices.
  • 3.3.1 Question Answering as Proxy Task: Question answering serves as a proxy for information extraction because answering simple questions is easier than expert annotation of complex linguistic phenomena.
  • 3.3.1 Question Answering as Proxy Task: QA prompts can jointly identify and classify extracted information by asking questions that target a particular entity, relation, or event role.For example, “Who bought something?” targets the Buyer argument role in an Exchange-Ownership event.
  • 3.3.1 Question Answering as Proxy Task: Event extraction can decompose argument identification and classification into natural questions that mutually improve each other.
  • 3.3.1 Question Answering as Proxy Task: QA models trained on QAMR achieved the best results for argument extraction, and QA reformulations improved zero-shot slot filling over direct slot-description encoding.
  • 3.3.1 Question Answering as Proxy Task: QA-based methods cover NER, coreference resolution, relation extraction, event extraction, dialogue state tracking, and zero-shot slot filling.
  • 3.3.1 Question Answering as Proxy Task: QA prompts may be hand-crafted, contextualized with text, generated neurally, asked sequentially, or combined across formulations through weighted voting.
  • 3.3.2 Textual Entailment as Proxy Task: Textual entailment supports classification by converting each candidate label into a natural-language verbalization and selecting the most probable hypothesis.The approach is motivated by entailment models’ strong few-shot learning ability and their framing as language-understanding tasks.

4 Paradigm 3: NLP as Text Generation

This section surveys reformulating NLP tasks as text generation problems, enabling generative PLMs to produce task outputs within a unified framework. It covers label-augmented texts, indices, templates, and other output-sequence designs.

  • Reformulating NLP Tasks as Text Generation: Generative PLMs reformulate non-generative NLP tasks as text generation, designing output sequences that encode desired labels for each input.The reformulated input-output pairs are typically used to fine-tune the models with maximum likelihood.
  • Advantages: A unified encoder-decoder text-to-text framework supports multi-task and transfer learning across NLP tasks of different natures.Directly generating labels also lets models use label semantics and capture dependencies among prediction steps in structure prediction.
  • Generating Label-Augmented Texts: Label-augmented generation copies the input and adds markers or labels that identify entities, spans, relations, or other task annotations.Examples include bracketed entity mentions with types and relation indicators, while post-processing can recover annotations from generated text.
  • Generating Label-Augmented Texts: Label-augmented text generation can train one model on multiple tasks and improve individual-task performance and few-shot data efficiency.The surveyed studies report these benefits for tasks including sequence labeling and entity-relation extraction.
  • Generating Word Indices: Index generation represents spans and labels through word-position indices, supporting flat, nested, and discontinuous named-entity recognition.For an input sequence x = [x_1, x_2, ..., x_n], generated tuples encode span starts, ends, and entity types.
  • Template Filling: Template filling supplies structured prompts whose missing fields are generated, and templates can include natural descriptions or label definitions.This approach is illustrated for document-level event argument extraction using conditional generative models such as BART.

5 Data Generation via PLM

This section surveys using PLMs to generate additional training data or auxiliary sequences for NLP. Generated labeled data can augment sparse datasets, while auxiliary generations can support explainability and reasoning.

  • Overview: PLM-generated data complements the paper’s three PLM-for-NLP paradigms by augmenting training data or producing auxiliary information.The two uses are presented as distinct but complementary applications of data generation.
  • Training Data Augmentation: Generated labeled examples can be combined with original data to increase diversity and alleviate limitations of sparse or hand-labeled training sets.The surveyed approaches fine-tune PLMs for task characteristics, reformulate training data, and filter generation noise.
  • Auxiliary Data: Auxiliary generations such as counterexamples, clarifying questions, contexts, and inference rules can provide insight into NLP model behavior.The section connects these generations with machine-learning explainability and the challenge of limited explanations for model decisions.
  • Information Extraction (IE): GPT-2-based synthetic event-detection data can cover domains including newswire and cybersecurity, but may contain grammatical, semantic, or annotation noise.A student-teacher framework constrains learning with anchor knowledge from original labeled data and yields significant improvement across event-detection datasets.
  • Question Answering (QA): Round-trip consistency retains generated QA training triples only when a second model reproduces the sampled answer.This filtering strategy is designed to mitigate noise in generated question-answer data.
  • Question Answering (QA): Augmented data can produce competitive results with few or no human-labeled examples, including settings that extend evidence across document paragraphs.The surveyed work applies generated data to support question answering and other downstream NLP tasks.
  • Knowledge Extraction: Prompting PLMs with partial knowledge elicits missing information, while prompting with full knowledge generates natural-language descriptions for downstream applications.A T5-based approach converts Wikidata relation triples into textual data through a multi-stage generation process.

6 Discussion

The discussion surveys how PLM paradigms can be combined and highlights unresolved questions about their mechanisms, data requirements, generalization, semantic understanding, and computational cost.

  • Mix of paradigms or PLMs: The three PLM paradigms are not mutually exclusive and can be combined, including prompts with fine-tuning and generation with prompting.A unified QA approach combines prompts, text generation, and fine-tuning with models such as T5.
  • Mix of paradigms or PLMs: Combining multiple PLMs can further improve performance across a range of NLP tasks.ACE combines ELMo, BERT, mBERT, and XLM-R, while leaving their complementarity as a future research direction.
  • Labeled data: Rigorous studies are still needed to determine labeled-data requirements and evaluate cost-benefit trade-offs across paradigms.The authors connect this gap to minimizing human labeling effort when designing NLP systems.
  • Computation: PLM use has increased computation and environmental footprint, motivating efficiency metrics alongside accuracy and lower-cost training approaches.Green AI emphasizes floating-point operations and financial cost; one proposal targets BERT pre-training in 24 hours on a low-end server.
  • Understanding versus memorization: The survey calls for separating semantic understanding from memorization in knowledge-intensive tasks such as QA and information retrieval.Reported observations include poor precision in memorized training facts and difficulty answering closed-book questions.
  • Explicit linguistic information: Integrating syntactic or semantic information during pre-training or fine-tuning has improved GLUE performance in reported studies.Examples include semantic information in RoBERTa fine-tuning and joint syntactic-semantic multi-task learning with BERT.
  • Meaningful prompts: Prompt success may reflect exploitation of training-data patterns rather than meaningful instruction following.Irrelevant templates matched meaningful ones in few-shot entailment experiments, and continuous prompts improve few-shot performance without meaningful instructions.
  • Theoretical and empirical analysis: Theoretical understanding remains preliminary, with limited evidence about cross-model and cross-language generalization.The survey identifies prompting and generation generalization across languages as especially understudied.

7 Conclusion

The paper surveys three major paradigms for applying pre-trained language models to NLP and also covers PLM-based data generation. It aims to provide a comprehensive view of this paradigm shift and identify future research directions.

  • Conclusion: The survey covers three PLM paradigms for NLP and approaches that automatically generate data to improve task performance.It presents these approaches as a comprehensive view of the field’s paradigm shift.

A PLMs for specialized domains or languages

The appendix summarizes PLMs organized by specialized domains and by languages.

  • PLMs for specialized domains or languages: Table 6 lists PLMs pre-trained on specific domains, while Table 7 lists PLMs pre-trained on different languages.The two tables organize models along domain and language dimensions.

B Pre-train then fine-tune approaches

The appendix summarizes prior work on pre-train-then-fine-tune approaches and notes related resources covering domain, language, and strategy organization.

  • Pre-train then fine-tune approaches: Table 8 organizes prior work by strategies in the pre-train-then-fine-tune paradigm.The appendix also notes that contextual embeddings are included because they are architecturally similar despite frozen PLM weights.
Loading 2111.01243v1…