PaperlayerView exact revision on arXiv

Source-linked AI summary

The Power of Scale for Parameter-Efficient Prompt Tuning

Brian Lester, Rami Al-Rfou, Noah Constant

arXiv:2104.08691v2cs.CL

TL;DR

Prompt-based adaptation reduces the need to duplicate large models, but manually designed prompts can underperform tuned models. This paper introduces learned soft prompts for frozen language models and finds that prompt tuning rivals model tuning as scale increases, while improving zero-shot domain transfer.

  • Problem

    Manual prompt design is error-prone and limited by input length, and prompt-based adaptation can lag behind tuned models in downstream task quality.

  • Method

    Prompt tuning freezes the pretrained model and trains a small set of task-specific soft tokens end-to-end using labeled data.

  • Results

    As model size increases, prompt tuning rivals model tuning on SuperGLUE and improves generalization in zero-shot domain transfer.

  • Takeaways & Limitations

    Frozen models with lightweight task-specific prompts support efficient multi-task serving and prompt ensembling while retaining strong task performance.

  • Takeaways & Limitations

    Manual GPT-3 prompts performed far below GPT-3 on comparable T5 checkpoints, possibly because of differences in pretraining data, architecture, and sequence length.

Abstract

from arXiv · show

In this work, we explore "prompt tuning", a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's "few-shot" learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant in that large models are costly to share and serve, and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed "prefix tuning" of Li and Liang (2021), and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.

1 Introduction

The paper introduces prompt tuning, which adapts a frozen language model using trainable input tokens rather than updating all model parameters. It addresses the quality and serving limitations of prompt-based adaptation, showing competitiveness with model tuning as model scale increases and additional robustness and ensembling benefits.

  • Scaling and competitiveness: Prompt tuning outperforms few-shot prompts and closes the quality gap with model tuning, becoming more competitive as language-model capacity increases.The approach uses no intermediate-layer prefixes or task-specific output layers and is competitive with model tuning at large scale.
  • Practical benefits: Reusing one frozen model across downstream tasks preserves efficient serving, while separating task prompts from generalist parameters improves resilience to domain shifts.Prompt tuning stores only small task-specific prompts and supports mixed-task inference with the original pretrained model.
  • Motivation and problem: Prompt-based adaptation avoids task-specific model copies but suffers from error-prone human prompt design, input-length limits, and lower downstream quality than model tuning.GPT-3 175B scores 71.8 on SuperGLUE versus 89.3 for fine-tuned T5-XXL, despite using 16 times more parameters.
  • Proposed method: Prompt tuning freezes the entire pretrained model and learns k task-specific tokens prepended to the input, training the soft prompt end-to-end on labeled data.The learned prompt condenses signal from a full labeled dataset while leaving the generalist model parameters fixed.
  • Additional findings: Prompt ensembling, which learns multiple prompts for one task, can boost quality, and the paper reports that prompt tuning outperforms model tuning on domain-shift problems.The paper also studies design choices and finds that quality and robustness improve with scale.

2 Prompt Tuning

Prompt tuning casts tasks as conditional text generation and learns a separate soft prompt while keeping the T5 model frozen. The section also specifies prompt construction and initialization choices, parameter-cost considerations, and experiments adapting T5 away from span-corruption behavior.

  • 2 Prompt Tuning: T5 casts classification as conditional generation, modeling Prθ(Y |X) where Y is a token sequence representing the class label.This follows T5’s text-to-text formulation rather than modeling a single class probability Pr(y|X).
  • 2 Prompt Tuning: Prompting prepends tokens P to X and maximizes Prθ(Y |[P; X]) while keeping the model parameters θ fixed.
  • 2 Prompt Tuning: Prompt tuning gives P dedicated parameters θP instead of restricting prompt representations to the frozen model’s embedding table.This replaces manual or non-differentiable prompt-token search with learned prompt representations.
  • 2 Prompt Tuning: Soft prompts use Pe ∈R^p×e, concatenate with Xe to form [Pe; Xe] ∈R^(p+n)×e, and train only the prompt parameters.
  • 2.1 Design Decisions: Prompt representations can start from random initialization, vocabulary embeddings, or valid target-token embeddings for classification tasks.Target-token initialization is intended to prime outputs toward legal classes.
  • 2.1 Design Decisions: The parameter cost is EP, where E is embedding dimension and P is prompt length, motivating a minimal prompt that still performs well.
  • 2.2 Unlearning Span Corruption: T5 is pretrained with an encoder-decoder span-corruption objective, reconstructing masked spans marked by sentinel tokens.Targets contain masked content separated by sentinels and end with a final sentinel.
  • 2.2 Unlearning Span Corruption: The study compares Span Corruption, Span Corruption + Sentinel, and LM Adaptation, with adaptation performed once to create one reusable frozen model.LM Adaptation continues self-supervised training on natural-text continuation for up to 100K steps.

3 Results

Prompt tuning becomes increasingly competitive with model tuning as T5 scales, matching multitask model tuning at 11B parameters while using over 20,000 times fewer task-specific parameters, and it substantially outperforms GPT-3 prompt design. Ablations show that larger models require less prompt conditioning and are more robust to initialization, while LM adaptation is important for reliable performance.

  • Scaling: At 11 billion parameters, prompt tuning matches the stronger multitask model-tuning baseline while using over 20,000 times fewer task-specific parameters.Prompt tuning becomes more competitive with model tuning as model scale increases.
  • Comparison with GPT-3: Prompt-tuned T5-Small matches GPT-3 XL, while prompt-tuned T5-Large beats GPT-3 175B.These comparisons correspond to models over 16 times and over 220 times larger, respectively.
  • Prompt Length: For most model sizes, increasing prompt length beyond one token is critical, but XXL remains strong with a single-token prompt.The results suggest that larger models need less conditioning signal to achieve a target behavior.
  • Prompt Initialization: Class-based initialization performs best at smaller model sizes, whereas initialization differences disappear at XXL scale.Random initialization lags behind more advanced initializations, but the gaps vanish for XXL.
  • Prompt Initialization: Class labels typically persist in learned prompts under class-label initialization, but the prompts are otherwise not interpretable.Nearest token embeddings in cosine distance generally match the initialization tokens.
  • Pre-training Objective: LM adaptation improves prompt-tuning quality and continues providing gains through 100K adaptation steps, while span-corruption pre-training causes instability across model sizes.In the non-optimal span-corruption setting, some mid-sized models fail to output legal class labels and score 0%; LM-adapted versions work reliably across all sizes.

4 Comparison to Similar Approaches

Prompt tuning is the most parameter-efficient learnable-prompt method for billion-parameter models, while simplifying prompt placement and retaining a frozen base model. Compared with related approaches, it avoids task-specific heads, input modifications, and model tuning, while adapters remain a broader parameter-efficient alternative.

  • Parameter efficiency: Prompt tuning requires less than 0.01% task-specific parameters for models over a billion parameters, making it the most parameter-efficient method among those with learnable parameters.The comparison measures task-specific parameter requirements across continuous-prompt methods.
  • Prefix tuning: Unlike prefix tuning, prompt tuning prepends a single prompt representation to the embedded input, allowing intermediate-layer task representations to update with each example.Prefix tuning learns prefixes at every transformer layer, whereas prompt tuning uses one input-level prompt and contextualizes intermediate representations by the example.
  • WARP: Unlike WARP, prompt tuning supports outputs beyond classification and requires neither input-layer changes nor a task-specific head.WARP relies on a [MASK] token and a learnable output layer that projects the mask to class logits, restricting it to a single output.
  • P-tuning: Prompt tuning simply prepends continuous prompts and keeps the original language model frozen, whereas P-tuning interleaves prompts and requires model tuning for strong SuperGLUE results.P-tuning also requires anchor tokens in the input, while prompt tuning leaves inputs untouched.
  • Adapters: Adapters provide another parameter-efficient alternative, achieving GLUE performance close to full model tuning with BERT-Large while adding 2–4% parameters.Adapters insert small bottleneck layers between frozen pretrained network layers.

5 Resilience to Domain Shift

Prompt tuning improves robustness to domain shift by freezing the language model and reducing overfitting to dataset-specific cues. It outperforms model tuning across most out-of-domain QA datasets and transfers better from QQP to MRPC, while gains are smaller in the reverse direction.

  • Motivation: Freezing the language model preserves its general language understanding and limits overfitting to lexical cues and spurious correlations.Prompt representations modulate input representations indirectly rather than modifying the core model parameters.
  • Question answering: Prompt tuning outperforms model tuning on most out-of-domain QA datasets, including a 12.5 point F1 gap on TextbookQA.Its gains are larger for greater domain shifts, while model tuning performs better on DROP, which shares Wikipedia’s domain with SQuAD.
  • Paraphrase detection: In the reverse MRPC-to-QQP direction, prompt tuning produces a small accuracy improvement but a small F1 drop.The asymmetric results support the view that model tuning is more prone to overfitting the training task at the expense of transfer across domains.

6 Prompt Ensembling

Prompt ensembling offers a storage- and inference-efficient alternative to ensembling full language models by sharing frozen parameters across multiple learned prompts. On SuperGLUE, five-prompt ensembles using one frozen T5-XXL outperform single-prompt averages and match or exceed the best individual prompt.

  • Motivation: Ensembling full models becomes impractical at scale because storing N copies and running N distinct inference passes impose substantial costs.Each T5-XXL copy requires 42 GiB of storage.
  • Method: Prompt ensembles train N task-specific prompts while sharing the pretrained model parameters, reducing storage and enabling one forward pass with batch size N.This replaces N separate model forward passes with a single batched computation.
  • Results: Five-prompt ensembles trained with one frozen T5-XXL model and majority voting outperform the single-prompt average across all SuperGLUE tasks.The experiment uses the authors’ default hyperparameters.
  • Results: The prompt ensemble matches or exceeds the best individual prompt on every SuperGLUE task.Table 3 reports that the ensemble either beats or matches the strongest individual prompt.

7 Interpretability

Soft prompts are difficult to interpret because they are learned in continuous embedding space, but nearest-neighbor analysis reveals word-like semantic structure and possible domain-specific priming. Longer prompts may also contain redundant tokens, reflecting excess capacity or limited positional localization.

  • Interpretability method: Nearest-vocabulary neighbors assessed with cosine distance provide a method for interpreting continuous soft-prompt representations.Each learned prompt token is compared with vocabulary embedding vectors from the frozen model.
  • Semantic structure: Top-5 nearest neighbors form tight semantic clusters, suggesting that learned prompt tokens acquire word-like representations.Clusters range from lexical variants such as technology-related words to semantically related terms such as entirely and completely.
  • Initialization effects: Class-label initialization often preserves labels among nearest neighbors after tuning, while other initializations distribute labels across multiple prompt tokens.This pattern was observed for class-label, Random Uniform, and Sampled Vocab initialization strategies.
  • Prompt length: Longer prompts often contain multiple tokens with identical nearest neighbors, suggesting excess capacity or difficulty localizing information by position.This observation concerns prompts of size 100 and may reflect the lack of sequential structure in prompt representations.
  • Domain priming: Science-, technology-, and engineering-related neighbors frequently appear for BoolQ prompts, whose questions include approximately 20% in the Nature/Science category.The authors suggest that prompts may prime the model to interpret inputs within a specific domain or context, such as scientific content.

8 Conclusion

Prompt tuning competitively adapts frozen pretrained language models, matching model tuning more closely as model size increases and improving zero-shot domain-transfer generalization. Frozen models also reduce storage and serving costs while enabling efficient multitask serving and prompt ensembling.

  • Task performance: On SuperGLUE, prompt tuning rivals traditional model tuning, with the performance gap vanishing as model size increases.The comparison concerns adapting frozen pretrained language models to downstream tasks.
  • Domain transfer: Prompt tuning improves generalization under zero-shot domain transfer.The conclusion links this benefit to freezing general-purpose language-understanding parameters while restricting downstream learning.
  • Efficiency: Using frozen pretrained models reduces storage and serving costs while enabling efficient multitask serving and high-performing prompt ensembling.The paper frames task-defining parameters as distinct from general language-modeling parameters, opening avenues for further research.

A Reproducibility · A.1 Experimental Settings

The experiments evaluate benchmark datasets with their specified metrics, using T5 open-source evaluation code and frozen T5 1.1 models. Prompt training uses different TPU configurations by model size, with parameter counts and convergence runtimes reported separately.

  • A.1 Experimental Settings: GLUE and SuperGLUE datasets are evaluated with each benchmark’s specified metric.The evaluation code is reused from the publicly available T5 open-source release.
  • A.1 Experimental Settings: SQuAD and MRQA are evaluated using F1, which considers partial answer spans.F1 is one of the metrics used by the SQuAD benchmark, and metric calculation again uses the T5 open-source release.
  • A.1 Experimental Settings: All models use T5 1.1 as the base frozen model.Additional details and pretrained checkpoints are available on GitHub.
  • A.1 Experimental Settings: Prompts for T5 Small and Base are trained on 4 TPU v2 chips.The hardware allocation differs for larger models.
  • A.1 Experimental Settings: Prompts for larger models are trained on 16 TPU v3 chips.This contrasts with the 4 TPU v2 chips used for T5 Small and Base prompts.
  • A.1 Experimental Settings: Parameter counts for each prompt are reported in Table 4.The passage directs readers to Table 4 for these counts.
  • A.1 Experimental Settings: Average runtimes until convergence are reported in Table 5.The passage directs readers to Table 5 for these runtimes.

A.2 Hyperparameter Search

The study used extensive manual hyperparameter search and repeated training runs for its main results, ablations, domain-shift experiments, and ensemble. Experimental graphs report means and standard deviations over three runs, while parameter-count variation is shown across prompt lengths of 1–100.

  • Hyperparameter Search: 77 hyperparameter search trials covered prompt tuning and single-task model tuning, with 40 and 37 trials respectively.Hyperparameter tuning used manual selection, with bounds provided in Table 6.
  • Hyperparameter Search: 195 runs supported the main result and ablations, using three training runs with validation evaluation for each baseline configuration and ablation setting.The study also ran 18 additional domain-shift experiments and 24 extra runs to create the ensemble.
  • Hyperparameter Search: Experimental graphs plot the mean and standard deviation computed over three runs.Some settings have sufficiently low variance that the standard-deviation line is hidden behind the plotted line.
  • Hyperparameter Search: Figure 4 reports the mean and standard deviation of each method’s parameter count as prompt length varies from 1–100.The Prefix Tuning (Train) curves appear to have no visible standard deviation because reparameterization parameters dominate the parameter count.

A.3 Datasets

The evaluation uses English GLUE, SuperGLUE, SQuAD, and out-of-domain MRQA datasets with established splits and T5-compatible preprocessing. WSC is recast as referent generation, excluding training examples with incorrect supplied referents.

  • Dataset sources and splits: All datasets are in English, using TensorFlow Datasets’ shipped training, validation, and test splits for GLUE and SuperGLUE.GLUE uses version 1.0.0 and SuperGLUE uses version 1.0.2.
  • Dataset sources and splits: SQuAD uses TensorFlow Datasets v1.1:3.0.0, while out-of-domain datasets use development splits from the MRQA shared task.SQuAD follows its provided training, validation, and test splits.
  • Preprocessing: The study follows open-source T5 preprocessing and text-to-text formatting, omitting the SuperGLUE dataset-prefix indicator.MRQA datasets use the T5 SQuAD preprocessing code.
  • Preprocessing: WSC is reformulated as free-form referent generation, so training examples with incorrect supplied referents are omitted.The model predicts the correct referent directly rather than classifying whether a supplied referent is correct.
  • Data collection: No new data was collected for this work.
Loading 2104.08691v2…