PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig

arXiv:2107.13586v1cs.CLcs.AIcs.LG

TL;DR

Prompt-based learning is reshaping NLP, but its methods and challenges need systematic organization. This survey synthesizes the paradigm, unifies its concepts, and identifies important open problems for future research.

  • Problem

    Prompt design is difficult to optimize manually, and even experienced designers may fail to discover optimal prompts.

  • Method

    The paper systematically summarizes prompt-based learning, unifies its mathematical notation, and organizes methods across models, prompts, and tuning strategies.

  • Results

    The survey characterizes prompt-based learning as a promising paradigm and highlights its core challenges, commonalities, and differences across NLP research paradigms.

  • Takeaways & Limitations

    The survey provides researchers with a structured understanding of prompt-based learning and directions for scientifically meaningful advances.

  • Takeaways & Limitations

    Systematic comparisons of how different pre-trained language models benefit from prompt-based learning remain scarce.

Abstract

from arXiv · show

This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.

1 Two Sea Changes in NLP

NLP has undergone two sea changes: the field shifted from fully supervised learning to pre-train and fine-tune, and is now moving toward pre-train, prompt, and predict. This survey organizes the emerging prompting paradigm, its methods and models, and resources for learning more effectively.

  • 1 Two Sea Changes in NLP: Fully supervised learning trained task-specific models solely on input-output examples, but insufficient datasets led early NLP systems to rely heavily on feature engineering.The paradigm had long played a central role in NLP before the later shifts.
  • 1 Two Sea Changes in NLP: As of 2021, pre-train, prompt, and predict was replacing pre-train and fine-tune by reformulating downstream tasks with textual prompts to resemble original language-model training.This approach replaces objective engineering with task reformulation toward the language model’s training format.
  • 1 Two Sea Changes in NLP: The survey organizes prompting through formal definitions, pretrained language models, prompt and answer engineering, multi-prompt learning, and prompt-aware training.It also highlights systematic resources, including companion websites, updates, slides, typologies, comparisons, prompt organizations, and research timelines.

2 A Formal Description of Prompting

Prompt-based learning replaces direct supervised prediction P(y|x; θ) with language-model scoring of prompted text P(x; θ), enabling prediction through prompt construction, answer search, and answer mapping. The framework’s design choices include the pre-trained model, prompt and answer engineering, paradigm extensions, and parameter-training strategies.

  • 2.1 Supervised Learning in NLP: Traditional supervised NLP learns P(y|x; θ) from labeled input-output pairs, including classification and conditional text generation.Classification predicts labels from a fixed set, while generation produces output text such as translations.
  • 2.2 Prompting Basics: Prompt-based learning instead models text probability P(x; θ) with a language model, reducing or potentially eliminating the need for large supervised datasets.This approach addresses the difficulty of obtaining abundant supervised data for many tasks.
  • 2.2.1 Prompt Addition: A prompting function transforms input x into prompt x′ by applying a template with input [X] and answer [Z] slots, then filling [X] with x.The resulting prompt leaves an intermediate answer text z to be generated and later mapped into y.
  • 2.2.1 Prompt Addition: Prompt slots may form cloze prompts when [Z] appears inside the text or prefix prompts when the input precedes [Z], and template words may be virtual words.The supplied examples include sentiment-analysis and machine-translation templates.
  • 2.2.2 Answer Search: Answer search defines permissible answers Z, scores their filled prompts with a pre-trained language model, and selects outputs by argmax search or sampling.For classification, Z can be a small set of words representing the label set; for generation, Z can encompass the language.
  • 2.2.3 Answer Mapping: The searched answer ẑ is converted into output ŷ directly when the answer is the task output, or through a mapping when multiple answers represent one output.For example, several sentiment-bearing words can map to the same class.
  • 2.3 Design Considerations for Prompting: Prompting methods require choices about the pre-trained model, prompt engineering, answer engineering, paradigm expansion, and prompt-based training strategies.These choices affect the calculated text probability, task specification, answer representation, applicability, and whether prompt or language-model parameters are trained.

3 Pre-trained Language Models

Pre-trained language models primarily learn to predict text through standard autoregressive or denoising reconstruction objectives. Their objectives, corruption functions, and representation directionality shape which prompting tasks they support, including prefix, cloze, generation, and classification tasks.

  • Training objectives: Standard language models optimize P(x) over corpus text, typically predicting tokens autoregressively from left to right.The factorization commonly follows P(x) = P(x1) × ··· P(xn|x1 ··· xn−1).
  • Training objectives: Denoising objectives apply a noise function ˜x = f_noise(x) and train models to reconstruct the original text by optimizing P(x|˜x).Corrupted Text Reconstruction computes loss only on noised parts, whereas Full Text Reconstruction computes loss over the entire input.
  • Objective-task alignment: Training objectives influence prompting suitability: left-to-right autoregressive models favor prefix prompts, while reconstruction objectives favor cloze prompts.Standard LM and FTR objectives may better support text generation, whereas classification can be formulated with other objectives.
  • Noising functions: Reconstruction models use masking, replacement, deletion, or permutation to corrupt tokens or spans before reconstruction.Masking can encode prior knowledge, such as emphasizing entities, while deletion is usually paired with FTR loss.
  • Representation directionality: Representations may be computed left-to-right from prior words or bidirectionally from all words, with mixed and permuted strategies also possible.These strategies are generally implemented through attention masking in neural models.

4 Prompt Engineering

Prompt engineering constructs prompting functions that improve downstream task performance by choosing prompt shapes and designing templates manually or automatically. Methods span cloze and prefix prompts, discrete and continuous representations, and static or input-specific prompting strategies.

  • 4 Prompt Engineering: Prompt engineering creates a prompting function fprompt(x) intended to achieve effective downstream-task performance through prompt-shape selection and manual or automated template design.The process first considers prompt shape and then chooses between manual and automated approaches.
  • 4.1 Prompt Shape: Cloze prompts fill blanks in textual strings, whereas prefix prompts continue string prefixes; prefix prompts generally better suit generation and standard autoregressive language models.The choice depends on both the task and the model.
  • 4.2 Manual Template Engineering: Manual templates use human intuition to construct prompts, supporting tasks including knowledge probing, question answering, translation, and commonsense reasoning.Examples include LAMA cloze templates and manually crafted prefix prompts.
  • 4.3 Automated Template Learning: Manual template engineering is time-consuming and experience-dependent, and even experienced designers may fail to discover optimal prompts.These limitations motivate automated template-learning methods, especially for complicated tasks such as semantic parsing.
  • 4.3 Automated Template Learning: Prompting functions may be static, using one template across inputs, or dynamic, generating a custom template for each input; hybrid methods combine hard templates with tunable embeddings.P-tuning inserts trainable variables into embedded inputs and models prompt-token interactions with a BiLSTM.
  • 4.3.1 Discrete Prompts: Discrete prompt methods automatically search natural-language templates through mining, paraphrasing, gradient-based token search, prompt generation, or language-model scoring.MINE extracts frequent middle words or dependency paths; other methods generate candidates, optimize tokens, or select the highest-scoring filled prompt.
  • 4.3.2 Continuous Prompts: Continuous prompts optimize prompting directly in the model’s embedding space rather than restricting prompts to human-interpretable natural language.Prefix tuning prepends continuous task-specific vectors while keeping language-model parameters frozen.
  • 4.3.2 Continuous Prompts: Continuous prefix learning is more sensitive to initialization in low-data settings than discrete prompts, while special-token embedding tuning uses fewer added parameters.Lester et al. tune the embeddings of prepended special tokens without adding tunable parameters within every network layer.

5 Answer Engineering

Answer engineering searches for an answer space Z and a mapping to output Y, considering both answer shape and answer design method. Answer spaces may be manually or automatically designed, with choices ranging from tokens to sentences and from unconstrained to constrained or optimized representations.

  • Overview: Answer engineering searches for an answer space Z and a mapping to the original output Y, considering answer shape and answer design method.Unlike prompt engineering, it focuses on designing acceptable answers and their mapping to outputs.
  • Answer Shape: Answer shapes range from tokens and short spans to sentences or documents, with the appropriate granularity depending on the task.Spans are usually paired with cloze prompts, while sentences are commonly paired with prefix prompts.
  • Answer Shape: Token or span answer spaces are widely used for classification, relation extraction, and named entity recognition, whereas longer answers often support generation and multiple-choice question answering.Multiple-choice systems can compare language-model scores across multiple phrases.
  • Manual Design: Manual answer design uses unconstrained tokens, spans, or sequences with identity mappings, or constrained spaces for limited-label tasks such as classification, entity recognition, and multiple-choice question answering.Constrained designs can manually associate candidate words with topics, emotions, or other aspects of the input.
  • Automatic Design: Automatic answer search includes paraphrase expansion, prune-then-search procedures, label decomposition, and soft answer tokens optimized through gradient descent.Paraphrasing can use back-translation, while soft answer tokens learn label-specific embeddings directly rather than using embeddings learned by the language model.

6 Multi-Prompt Learning

Multi-prompt learning extends single-prompt methods by using multiple prompts to improve efficacy through ensembling, augmentation, composition, and decomposition. These approaches combine prompt predictions, provide demonstrations, divide composable tasks into sub-prompts, or separately address multiple predictions.

  • 6 Multi-Prompt Learning: Multi-prompt learning extends single-prompt learning by using multiple prompts for varied motivations and potential effectiveness gains.The survey organizes these methods into prompt ensembling, augmentation, composition, and decomposition.
  • 6.1 Prompt Ensembling: Prompt ensembling combines multiple unanswered discrete or continuous prompts at inference to exploit complementary strengths, reduce prompt-selection costs, and stabilize downstream performance.Continuous prompts are typically produced with different initializations or random seeds.
  • 6.1 Prompt Ensembling: Ensembling methods include uniform probability averaging, performance-based weighted averaging, majority voting for classification, and distillation into a final model.Weights may be pre-specified or optimized on training data; distillation trains a final model from an unlabeled dataset annotated by prompt-specific models.
  • 6.2 Prompt Augmentation: Prompt augmentation, or demonstration learning, prepends answered examples to an input prompt so the language model can infer how to produce the desired answer.For example, answered capital-city examples can precede an unanswered capital-city prompt.
  • 6.2 Prompt Augmentation: Few-shot prompt augmentation is sensitive to both which examples are selected and how they are ordered, with embedding-based selection and entropy-based permutation search used to improve performance.Reported performance can range from near state-of-the-art accuracy to near random guess depending on example choice.
  • 6.3 Prompt Composition: Prompt composition uses multiple sub-prompts for fundamental subtasks and combines them into a composite prompt for composable tasks such as relation extraction.One approach manually creates sub-prompts for entity recognition and relation classification, then composes them using logic rules.
  • 6.4 Prompt Decomposition: Prompt decomposition breaks a holistic prompt into separate sub-prompts when one sample requires multiple predictions, such as sequence labeling or named entity recognition.Each sub-prompt is answered separately to identify the multiple required predictions.

7 Training Strategies for Prompting Methods

Training strategies range from zero-shot or data-assisted prompting to full-data and few-shot learning, with parameter updates determining the trade-offs among efficiency, applicability, expressiveness, and overfitting. The survey organizes five parameter-tuning strategies according to whether LM and additional prompt parameters are updated.

  • 7.1 Training Settings: Zero-shot prompting applies a text-predicting language model directly to cloze or prefix prompts without explicit downstream-task training data.Annotated examples may nevertheless have influenced prompt construction or validation, making the zero-shot designation debatable.
  • 7.1 Training Settings: Full-data learning trains with many examples, whereas few-shot learning uses very few examples and benefits from prompts when examples cannot fully specify desired behavior.Prompting methods can be used either without task-specific training or in concert with training data.
  • 7.2.1 Promptless Fine-tuning: Promptless fine-tuning updates all or some pre-trained LM parameters on a task dataset without prompts, offering simplicity but risking overfitting or instability on small datasets.Updating all LM parameters can help the model fit larger training datasets.
  • 7.2.2 Tuning-free Prompting: Tuning-free prompting generates answers from prompts while keeping LM parameters fixed, enabling efficient zero-shot use but requiring substantial prompt engineering.In-context learning optionally augments inputs with answered prompts, which can be slow at test time when many examples are provided.
  • 7.2.3 Fixed-LM Prompt Tuning: Fixed-LM prompt tuning updates only additional prompt parameters using downstream supervision, preserving LM knowledge and often improving few-shot accuracy but excluding zero-shot use.Its representation power is limited in large-data settings, and learned prompts are usually not human-interpretable or manipulable.
  • 7.2.4 Fixed-prompt LM Tuning: Fixed-prompt LM tuning updates LM parameters while using fixed prompts to specify behavior, potentially improving learning particularly in few-shot scenarios.Discrete templates apply to training and test examples, while answer engineering and partial LM fine-tuning can reduce prompt engineering.
  • 7.2.5 Prompt+LM Tuning: Prompt+LM tuning fine-tunes prompt-relevant parameters together with some or all pre-trained-model parameters, providing the most expressive strategy but requiring all parameters and risking small-data overfitting.Adding prompts can provide bootstrapping at the start of training, and the method is likely suitable for high-data settings.

8 Applications

Section 8 organizes prompting methods by the applications to which they have been applied, including probing, classification, information extraction, reasoning, question answering, and text generation. Across these applications, prompts reformulate tasks for language models, while task-specific challenges affect template design, answer engineering, decoding, and reasoning ability.

  • Probing: Prompting methods are applied to factual and linguistic probing, retrieving knowledge or eliciting phenomena such as analogies, negation, semantic similarity, and rare-word understanding.Factual probing typically fixes model parameters and transforms inputs into manually crafted or automatically discovered cloze prompts.
  • Classification: Classification applications commonly use cloze prompts, with extensive exploration of prompt engineering, answer engineering, and fixed-prompt LM tuning in few-shot settings.Text classification often uses templates such as “the topic of this document is [Z],” while NLI commonly uses manually pre-selected answer spaces and few-shot template search.
  • Information Extraction: Information extraction requires more carefully designed prompts than classification, with relation extraction facing larger label spaces and tagging tasks requiring token-level predictions and latent label relationships.Semantic parsing has been approached through paraphrasing and grammar-constrained decoding in few-shot, in-context settings.
  • Reasoning: Prompting has been used to probe reasoning through commonsense benchmarks and mathematical tasks, although pre-trained models perform simple arithmetic yet fail when numbers become larger.Commonsense tasks include Winograd Schemas and multiple-choice sentence completion.
  • Question Answering and Text Generation: Question answering applications span extractive, multiple-choice, and free-form formats, while text generation uses prefix prompts with autoregressive language models for tasks such as summarization and machine translation.Prompting has also been applied to evaluation framed as text generation, where adding “such as” improved correlation on German-English MT evaluation.

9 Prompt-relevant Topics

The section relates prompt-based learning to neighboring methods, emphasizing prompt ensembling, contextual augmentation, query reformulation, QA formulations, controlled generation, supervised attention, and data augmentation. It highlights that prompts can combine template-generated results and deliver accuracy gains comparable to adding hundreds of training examples.

  • Ensemble Learning: Prompt ensembling extends ensemble learning by using different prompt templates to generate multiple results that are combined.Traditional ensembles exploit complementarity from varied architectures, training strategies, data ordering, or random initialization.
  • Larger-context Learning: Prompt augmentation resembles larger-context learning by adding relevant labeled samples to the input, although added context need not be labeled data in larger-context learning.Larger-context learning augments inputs with contextual information retrieved from training sets or external sources.
  • Query Reformulation: Prompt-based learning and query reformulation both reformulate inputs to elicit answers, but prompting typically treats a language model as the knowledge base and can convert classification into cloze tasks.Traditional query reformulation usually targets search engines or question-answering systems.
  • QA-based Task Formulation: Prompting resembles QA-based task formulation because textual questions specify the task, but it additionally emphasizes using knowledge in pre-trained language models.Earlier QA formulations unified multiple NLP tasks under a question-answering framework, including information extraction and text classification.
  • Data Augmentation: Prompts can achieve accuracy improvements similar to adding 100s of data points on average across classification tasks, suggesting implicit data augmentation.Data augmentation increases training data by modifying existing data, whereas prompts provide a comparable downstream accuracy improvement.

10 Challenges

The survey identifies open challenges spanning task coverage, prompt and answer design, model and tuning choices, prompt transfer and sharing, theoretical understanding, pre-training, and calibration.

  • Task Coverage: Prompt-based learning remains underexplored for information extraction and text analysis because designing prompts for these tasks is less straightforward.Future approaches may need to reformulate such tasks as classification or text generation.
  • Structured Information: Expressing trees, graphs, tables, and relational structures in prompts or answers remains a major challenge.Existing work has explored additional marks for lexical information and structured prompts based on hypertext markup language.
  • Prompt and Answer Engineering: Selecting and jointly optimizing templates, answers, class spaces, multi-token decoding, and multiple generation references remain unresolved answer-engineering problems.Many-class classification creates a difficult combinatorial optimization problem, while multi-token answers and multiple semantically equivalent references remain difficult to decode or exploit.
  • Models, Tuning, and Pre-training: Systematic comparisons of prompt-tuning strategies and pre-trained language models are still lacking, leaving their tradeoffs and relative benefits poorly understood.The survey also asks whether pre-training methods developed for pre-train-and-fine-tune models apply as-is to prompting or should be rethought.
  • Theory and Calibration: Theoretical guarantees for prompting are scarce, and calibration remains important because pre-trained language-model probabilities are typically not well calibrated.Existing theory includes soft-prompt tuning that relaxes non-degeneracy assumptions for downstream recovery, while identified calibration pitfalls include majority-label and recency biases.
  • Transferability: Prompt transferability depends on how prompts are selected: tuned few-shot prompts generalize across similarly sized models more effectively than true few-shot prompts.The distinction is between having a larger validation set for prompt selection and having only a few training samples.

11 Meta Analysis

This meta-analysis organizes prompt-based learning research chronologically and quantifies trends across years, tasks, search targets, and prompt-engineering approaches. The literature surges in 2021, centers on text classification and factual probing, favors template search over answer search, and shifts from discrete toward continuous search.

  • Timeline: The survey provides a chronological timeline of prompt-based learning papers to clarify the field’s evolution for newcomers.The timeline orders existing research works chronologically.
  • Year: 2021 saw a huge surge in prompt-based learning research, likely driven by GPT-3’s popularity in few-shot multi-task prompting.The analysis attributes this increase to the emergence of different pretrained language models and GPT-3’s influence.
  • Tasks: Most prompt-based learning works investigate text classification and factual probing, where template and answer engineering are relatively easy and experiments inexpensive.Tasks with fewer than five relevant works are grouped into “Others.”
  • Prompt vs. Answer Search: Current research focuses mainly on template search rather than answer search because generation references can serve as answers and classification labels are often easy to select.The passage identifies conditional generation and classification as reasons answer search receives less attention.
  • Discrete Search vs. Continuous Search: Prompt engineering has shifted from discrete search toward continuous search, likely because continuous methods optimize more easily and soft prompts represent more information.The analysis examines automatic template search because few works study automatic answer search.

12 Conclusion

The paper surveys statistical NLP paradigms and argues that prompt-based learning is a promising new paradigm that may substantially change how NLP is viewed.

  • 12 Conclusion: The survey aims to help researchers understand prompt-based learning more effectively and comprehensively, including its core challenges.It seeks to support more scientifically meaningful advances in the field.

A Appendix on Pre-trained LMs

This appendix provides auxiliary information about pre-trained language models to help readers understand the current state of this dynamic research area.

  • A Appendix on Pre-trained LMs: The appendix summarizes auxiliary information on pre-trained LMs and situates readers in the current landscape of this dynamic research area.It is intended to support readers’ understanding of the field.

A.1 Evolution of Pre-trained LM Parameters

The section surveys the growth of popular pre-trained language models by comparing their parameter counts, highlighting GPT3, CPM2, and PanGu-α as the largest listed models.

  • A.1 Evolution of Pre-trained LM Parameters: GPT3, CPM2, and PanGu-α are the three largest models, each exceeding 150 billion parameters.Figure 7 compares popular pre-trained language models with parameter counts ranging from 0 to 200 billion.

A.2 Auxiliary Objective … UniLM

The survey catalogs auxiliary pre-training objectives and organizes mainstream pre-trained language models by diverse dimensions, including architecture, corruption, auxiliary tasks, data parallelism, and applications. Its model taxonomy spans autoregressive, masked, and encoder–decoder families across language, multimodal, multilingual, and domain-specific settings.

  • A.2 Auxiliary Objective: The subsection lists additional auxiliary objectives for pre-training language models.Examples include Next Sentence Prediction, Sentence Order Prediction, and Capital Word Prediction.
  • A.2 Auxiliary Objective: Auxiliary objectives include multilingual and translation replaced-token detection, plus knowledge embedding for encoding entities and relations.MRTD shares generator and discriminator components across languages, while TRTD distinguishes real and masked tokens in translation pairs.
  • Pre-trained Language Model Families: The survey clusters mainstream pre-training models across diverse dimensions because their increasing number makes differences difficult to grasp clearly.The taxonomy is presented through pre-training tasks, corruption types, applications, and model characteristics.
  • Pre-trained Language Model Families: Autoregressive families such as GPT, GPT-2, GPT-3, Codex, and ELMo use left-to-right language modeling, with applications including natural language generation.The listed GPT variants are associated with NLG, while ELMo is associated with both NLU and NLG.
  • ELMo: Masked-language-model families include BERT variants, domain-specialized models, multilingual models, and multimodal models using token, span, entity, phrase, region, or frame units.Examples include BERT, SciBERT, BioBERT, FinBERT, mBERT, ERNIE variants, VLBERT, ViLBERT, BEIT, VideoBERT, and TaBERT.
  • UniLM: Encoder–decoder families such as BART, mBART, T5, mT5, M6, MASS, PEGASUS, and ByT5 support applications including natural language understanding, generation, translation, and summarization.Their listed pre-training setups use translation or corrupted-text reconstruction with token-, sentence-, span-, byte-span-, or document-level units.
  • UniLM: UniLM1 and UniLM2 combine language-modeling and sequence-modeling objectives with conditional representation learning for natural language understanding and generation.UniLM1 includes Next Sentence Prediction, whereas UniLM2 is listed without that auxiliary objective and uses token-level corruption.
Loading 2107.13586v1…