Source-linked AI summary
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, Jie Tang
TL;DR
Prompt tuning reduces task-specific costs but prior methods struggled on normal-sized models and hard sequence-labeling tasks. P-Tuning v2 adapts deep prompt tuning for NLU and reports fine-tuning-comparable performance across scales and tasks with a small trainable parameter fraction. The authors present it as a potential alternative to fine-tuning and baseline for future work.
Problem
Prior prompt-tuning methods underperformed fine-tuning on medium-sized models and hard sequence-labeling tasks, leaving universality across scales and NLU tasks unresolved.
Method
P-Tuning v2 adapts deep prompt tuning for NLU by applying continuous prompts at every pretrained-model layer and optimizing implementation details.
Results
P-Tuning v2 matches fine-tuning across model scales from 300M to 10B parameters and hard sequence-labeling tasks with 0.1% to 3% trainable parameters per task.
Takeaways & Limitations
P-Tuning v2 is presented as a potential alternative to fine-tuning and a strong baseline for future work.
Takeaways & Limitations
Prior prompt tuning performs much worse than fine-tuning on widely used medium-sized models from 100M to 1B parameters.
Abstract
from arXiv · showhide
Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning \cite{li2021prefix,qin2021learning} optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.Our code and data are released at https://github.com/THUDM/P-tuning-v2.
1 Introduction
Prompt tuning freezes the pretrained model and tunes continuous prompts, reducing task-specific costs but historically falling short on smaller models and hard sequence labeling tasks. P-Tuning v2 reports that properly optimized prompt tuning can match fine-tuning across scales and NLU tasks.
- Motivation: Fine-tuning updates all model parameters, creating substantial training memory and per-task storage costs.Gradients and optimizer states must be stored, and a model copy is kept for each task during inference.
- Prompt Tuning: Prompt tuning freezes pretrained-model parameters and updates only trainable continuous prompts.The method reduces the number of parameters trained for each task compared with fine-tuning.
- Limitations of Prior Methods: Prompt tuning historically underperformed fine-tuning below 10B parameters and on hard sequence labeling tasks.The cited hard tasks include extractive question answering and related sequence-labeling settings.
- Contribution: P-Tuning v2 applies continuous prompts at every pretrained-model layer and adds optimization and implementation details for NLU.The paper describes this as an optimized and adapted implementation of deep prompt tuning.
- Results: 0.1% to 3% trainable parameters per task accompanies performance matching fine-tuning across model scales from 300M to 10B and hard sequence-labeling tasks.Reported tasks include extractive question answering and named entity recognition.
2 Preliminaries
NLU tasks are divided into simple classification and hard sequence labeling. Continuous prompt tuning replaces discrete natural-language prompts with trainable embeddings while keeping the pretrained model frozen.
- NLU Tasks: NLU challenges are categorized as simple classification tasks and hard sequence labeling tasks.Simple tasks classify over a label space, whereas sequence labeling classifies each input token.
- Prompting: Discrete prompting appends natural-language prompt tokens and asks the language model to predict a masked token.The example concatenates an input with “It is [MASK]” and uses predicted words as labels.
- Prompt Tuning: Continuous prompt tuning appends trainable embeddings to the input sequence while freezing pretrained-language-model parameters.The sequence is represented as [e(x), h0, ..., hi, e([MASK])].
3 P-Tuning v2
Existing prompt-tuning methods lacked universality across model scales and tasks, especially for medium-sized models and hard sequence tagging. P-Tuning v2 adapts deep prompt tuning by inserting prompts across layers and tailoring implementation choices to NLU.
- Lack of Universality across Scales: Prior prompt tuning performed much worse than fine-tuning on widely used medium-sized models from 100M to 1B parameters.Earlier work found competitiveness mainly when models scaled beyond 10B parameters.
- Lack of Universality across Tasks: Prior prompt tuning was poorly verified or performed poorly on hard sequence tagging tasks compared with fine-tuning.Sequence tagging predicts a label sequence for input tokens and can be incompatible with verbalizers.
- Deep Prompt Tuning: P-Tuning v2 adapts deep prompt tuning as a universal solution across model scales and NLU tasks.Its central design change is inserting continuous prompts into multiple layers rather than only the input embedding sequence.
- Deep Prompt Tuning: P-Tuning v2 increases task-specific capacity from 0.01% to 0.1%-3% while keeping prompts parameter-efficient and giving deeper prompts more direct predictive impact.Prompts are added as prefix tokens in different layers.
- Optimization and Implementation: Optimization choices include task-dependent reparameterization, task-dependent prompt lengths, optional multi-task initialization, and a classification head instead of verbalizers.Simple classification tasks generally prefer prompts shorter than 20, while hard sequence-labeling tasks prefer lengths around 100.
4 Experiments
The experiments evaluate P-tuning v2 across model scales, NLU and sequence-labeling tasks, parameter settings, and prompt-depth choices. Across these settings, P-tuning v2 generally matches fine-tuning while using frozen backbones and few task-specific parameters.
- Across Scales: P-tuning v2 matches fine-tuning across model scales, including models ranging from about 300M to 10B parameters.The evaluation includes BERT-large, RoBERTa-large, DeBERTa-xlarge, and GLM models.
- Efficiency: 0.1% task-specific parameters are needed by P-tuning v2 compared with fine-tuning at larger model scales.The experiments derive this ratio by comparing continuous-prompt parameters with transformer parameters.
- Across Tasks: P-tuning v2 is generally comparable to fine-tuning on named entity recognition, extractive question answering, and semantic role labeling.The compared NER and SRL metrics are micro-F1 scores.
- Ablation Study: There is no significant performance difference between using a verbalizer with an LM head and using a [CLS] label with a linear head.The comparison is conducted on RoBERTa-large while keeping other hyperparameters fixed.
- Prompt Depth: Adding continuous prompts to deeper layers performs better than adding the same number of prompts to beginning layers.In RTE, prompts on layers 17–24 achieve performance very close to prompting all layers.
5 Conclusions
P-tuning v2 is a prompt-tuning method whose main contribution is the finding that prompt tuning can match fine-tuning across model scales and tasks. Its parameter efficiency makes it a potential alternative to fine-tuning and a baseline for future work.
- Conclusion: P-tuning v2 is presented as a prompt-tuning method with relatively limited technical novelty.Its contribution is primarily an empirical finding about prompt-tuning effectiveness.
- Conclusion: Prompt tuning can be comparable to fine-tuning across scales from 330M to 10B parameters and across tasks.The conclusion describes this as a novel finding about prompt tuning’s universality.
- Conclusion: P-tuning v2 can be a potential alternative to fine-tuning and a strong baseline for future work.The conclusion associates this potential with high accuracy and parameter efficiency.
A Problem Formulation on Sequence Tagging
The paper evaluates P-Tuning v2 on sequence-labeling problems including named entity recognition, extractive question answering, and semantic role labeling, using task-specific datasets and classifiers.
- Named entity recognition predicts spans corresponding to entity classes and is evaluated on CoNLL03, OntoNotes 5.0, and CoNLL04.
- Multi-task NER training combines three datasets while sharing continuous prompts and using separate linear classifiers for each dataset.
- Extractive question answering identifies answer spans in context by assigning start and end labels to tokens.
- Semantic role labeling assigns semantic-role labels to words or phrases, using the target verb to identify which predicate is being analyzed.
- Multi-task SRL pre-training combines CoNLL05, CoNLL12, and propbank-release, with a strategy similar to NER.
B More Ablation Study
The ablations show that reparameterization and prompt length are task- and dataset-dependent design choices rather than universally optimal settings.
- MLP reparameterization generally outperforms embedding in RTE and CoNLL04, while the methods are competitive in BoolQ and embedding wins in CoNLL12.
- Shorter prompts usually suit simple NLU tasks, whereas hard sequence tasks generally benefit from prompts longer than 100.
- Optimal prompt length varies across tasks, with shorter prompts preferred for simple classification and approximately 100-token prompts for hard sequence labeling.
- MLP reparameterization reaches its optimum earlier than embedding in RTE, CoNLL04, and BoolQ.