PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT Understands, Too

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, Jie Tang

arXiv:2103.10385v2cs.CLcs.LG

TL;DR

Manual discrete prompts are unstable, with small wording changes causing large performance differences. P-Tuning adds trainable continuous prompt embeddings to discrete prompts and improves performance while stabilizing adaptation across LAMA and SuperGLUE settings.

  • Problem

    Manual discrete prompts can be unstable, and existing automatic search methods do not change that instability.

  • Method

    P-Tuning concatenates trainable continuous prompt embeddings with discrete prompts and optimizes them through backpropagation.

  • Results

    P-Tuning improves performance and stabilizes training with both frozen and tuned language models under fully supervised and few-shot settings.

  • Takeaways & Limitations

    P-Tuning provides a prompt-based adaptation method that is effective across LAMA and SuperGLUE task settings.

Abstract

from arXiv · show

Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that manual discrete prompts often lead to unstable performance -- e.g., changing a single word in the prompt might result in substantial performance drop. We propose a novel method P-Tuning that employs trainable continuous prompt embeddings in concatenation with discrete prompts. Empirically, P-Tuning not only stabilizes training by minimizing the gap between various discrete prompts, but also improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is generally effective for both frozen and tuned language models, under both the fully-supervised and few-shot settings.

1 Introduction

Manual discrete prompts can be highly unstable, motivating P-Tuning, which adds trainable continuous prompt embeddings to improve stability and NLU performance.

  • Manual discrete prompts can change performance substantially when only one prompt word changes.This instability remains a practical challenge even when the language model is tuned, especially in few-shot settings.
  • P-Tuning concatenates trainable continuous prompt embeddings with discrete prompts.The continuous prompts are updated by backpropagation, and a prompt encoder models dependencies between them.
  • P-Tuning outperforms manual and searched prompts on LAMA with frozen language models and outperforms PET on SuperGLUE with finetuned models.These comparisons cover both fully supervised and few-shot settings.
  • P-Tuning reduces performance gaps between discrete prompts across a wide range of tasks and settings.The reported reduction in prompt sensitivity improves stability for language-model adaptation.

2 Method

The method reformulates prompting with continuous embeddings alongside discrete prompt tokens, enabling differentiable optimization and learned dependencies among prompt embeddings.

  • Discrete prompt search can remain unstable and may not fully leverage backpropagation gradients.P-Tuning addresses this limitation by introducing continuous prompt embeddings.
  • Discrete prompts organize inputs and labels into templates that reformulate tasks as filling masked text.For example, a capital-prediction task becomes “The capital of Britain is [MASK].”
  • P-Tuning concatenates continuous prompt embeddings with discrete prompt tokens before feeding them to the language model.The resulting inputs combine learned embeddings with the original prompt and data embeddings.
  • An embedding function maps trainable continuous prompts to model inputs, which are then updated to optimize the task loss.The method uses LSTMs or MLPs to model dependencies between continuous prompt embeddings.
  • P-Tuning can concatenate discrete and continuous prompts and applies to both frozen and finetuned language models.

3 Experiments

Experiments evaluate P-Tuning on LAMA and SuperGLUE under frozen, fully supervised, and few-shot settings, reporting improvement across the tested task settings.

  • The experiments use LAMA for knowledge probing and SuperGLUE for general NLU.SuperGLUE is evaluated in both fully supervised and few-shot learning settings.
  • P-Tuning improves overall performance across the reported task settings and stabilizes performance on LAMA and few-shot SuperGLUE.For fully supervised SuperGLUE, discrete-prompt differences are smaller and training is stable without P-Tuning.
  • LAMA uses frozen language models, whereas SuperGLUE jointly tunes language-model parameters and continuous prompts.This setup evaluates P-Tuning with both frozen and tuned language models.
  • The overall task setup and summary of results are presented in Table 2.

3.1 Knowledge Probing

On LAMA knowledge probing, P-Tuning uses continuous prompts with the frozen language model and substantially improves over manual and searched discrete prompts.

  • Setup: LAMA evaluates pretrained knowledge by predicting entities in cloze tests constructed from knowledge-base triples.
  • Setup: The evaluation uses LAMA-34k and the vocabulary-intersection subset LAMA-29k.LAMA-34k contains 34,039 testing triples across 41 Wikidata relations.
  • Setup: Continuous prompts are concatenated with original discrete prompts during prompt training.The number and positions of prompt tokens are selected using development sets.
  • Main results: From 43.3% to 50.6% on LAMA-34k, P-Tuning improves the best knowledge-probing result.On LAMA-29k, it improves the best result from 45.2% to 64.2%.
  • Main results: P-Tuning outperforms AutoPrompt and LPAQA on same-size models.The result supports the claim that discrete prompts may not be optimal.

3.2 Fully-supervised Learning

P-Tuning is evaluated on seven SuperGLUE tasks using BERT and GPT models against standard finetuning and PET. It improves fully-supervised performance across both model families, especially on lower-resource tasks.

  • Setup: P-Tuning is compared with standard classification finetuning and PET on BERT-Base, BERT-Large, GPT2-Base, and GPT-medium.The experiments use seven SuperGLUE tasks because ReCoRD does not adopt discrete prompts.
  • Main Results: P-Tuning achieves the best performance on 5/7 BERT-Base tasks and 4/7 BERT-Large tasks.Its exceptions are WiC and MultiRC.
  • Main Results: P-Tuning consistently achieves the best performance on all tasks with GPT2-Base and GPT-medium models.The paper also reports average improvements over the considered baselines.
  • Main Results: P-Tuning may provide larger gains on low-resource tasks than on high-resource tasks such as WiC and MultiRC.The authors report that gains over CLS-FT may be limited when training sets are relatively large.

3.3 Few-Shot Learning

The few-shot experiments evaluate P-Tuning on SuperGLUE with controlled validation and prompt-pattern construction. P-Tuning improves average performance over PET and Prompt Tuning, while prompt-token placement and count materially affect results.

  • Setup: Few-shot evaluation uses random data splits for model selection on a small labeled set to reduce overfitting to a large development set.The procedure follows FewNLU to address high variance from prompts, example order, and random seeds.
  • Setup: P-Tuning constructs pattern candidates by inserting different numbers of continuous prompt tokens into manual PET prompts and selecting among them with FewNLU validation.This preserves the underlying discrete prompts while varying continuous-token placement and count.
  • Main Results: P-Tuning improves average few-shot performance by more than 1 point over PET and more than 13 points over Prompt Tuning on seven ALBERT tasks.Each result is averaged over four runs with different data splits.
  • Ablation Study: LSTM, MLP, and embedding-based prompt encoders all work, while LSTM and MLP generally perform well on new tasks.The encoder ablation considers recurrent, multilayer, and direct-embedding optimization designs.
  • Ablation Study: Prompt-token placement performs better when continuous tokens do not segment complete sentences, with no special preference for edge versus middle placement.The authors recommend writing several candidates and searching over them for each task.
  • Ablation Study: The number of continuous prompt tokens strongly affects few-shot performance, but increasing it does not always improve results.The paper recommends selecting the token count through model selection because excessive tokens may be difficult to learn with limited data.
  • Comparison with Discrete Prompt Search: P-Tuning can be combined with automatically searched discrete prompts while providing further performance improvement.The comparison uses the top-three searched patterns under the stated evaluation procedure.

3.4 Stabilizing Language Model Adaptation

P-Tuning reduces sensitivity to the choice of discrete prompt patterns in few-shot SuperGLUE and LAMA. It improves weak patterns and lowers variation across patterns.

  • Stabilizing Language Model Adaptation: P-Tuning improves the performance of worst-performing few-shot patterns and achieves a smaller standard deviation across multiple patterns.Compared with PET-FT, it increases stability with respect to pattern choice.

4 Related work

P-Tuning differs from prior prompting methods by using continuous prompt embeddings alongside discrete prompts and by targeting NLU across broader adaptation settings. The paper presents this combination as improving both performance and stability.

  • Language Model Prompting: Prior prompting work uses in-context examples, cloze patterns, or automatically searched discrete prompts to adapt pretrained language models.Search methods include corpus mining, gradient-based search, and pretrained generative models.
  • Language Model Prompting: P-Tuning instead uses continuous prompt embeddings that are complementary to discrete prompts in the reported experiments.Its technical design combines hybrid continuous-discrete prompts with a prompt encoder.
  • Related Methods: Prefix-tuning adds continuous prompts at every layer for natural language generation, whereas P-Tuning targets natural language understanding.The distinction is explicitly made against this concurrent method.
  • Related Methods: Prompt-encoder ablations report that LSTM and MLP generally work well, while direct embedding optimization can substantially underperform on some tasks.The cited table specifically mentions WiC and CB as examples of instability or underperformance for EMB.
  • Novelty: The paper states that P-Tuning improves performance and stabilizes training with frozen or tuned models in both few-shot and fully-supervised settings.It identifies this combination of model-training and supervision settings as a conclusion distinguishing the approach from concurrent NLU work.

5 Conclusions

P-Tuning uses continuous prompts alongside discrete prompts to improve performance and stabilize pretrained language model adaptation. It works with both frozen and tuned models in few-shot and fully-supervised settings.

  • P-Tuning uses continuous prompts concatenated with discrete prompts for pretrained language model adaptation.
  • P-Tuning improves performance and stabilizes training across pretrained language model adaptation.
  • P-Tuning is effective with both tuned and frozen language models under few-shot and fully-supervised settings.
Loading 2103.10385v2…