Source-linked AI summary
How Many Data Points is a Prompt Worth?
Teven Le Scao, Alexander M. Rush
TL;DR
Classification with pretrained models can use either a generic head or task-specific prompting, but the sample-efficiency benefit of prompting requires fair quantification. The paper compares these approaches across tasks and data scales while controlling fine-tuning variables, finding task-dependent gains often worth hundreds of training examples. Its scope includes human-written prompts and raises bias concerns from prompts and pretrained models.
Problem
The paper asks how much task-specific prompting improves sample efficiency over head-based classification, especially in low-data settings.
Method
The study compares prompted and head-based supervised fine-tuning across multiple tasks, data scales, prompts, and repeated runs, using average data advantage to quantify gains.
Results
Prompting provides task-dependent data-efficiency gains, averaging hundreds of equivalent data points across almost all tasks, including approximately 3500 on MNLI and up to 750 on BoolQ.
Takeaways & Limitations
Prompting's main benefit is data efficiency rather than compute efficiency, with potential relevance to low-resource language applications.
Takeaways & Limitations
The study uses human-written prompts and notes risks of introducing human and pretrained-model biases, especially in few-shot settings.
Abstract
from arXiv · showhide
When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.
1 Introduction
Prompting adapts pretrained language models for classification by injecting task-specific information through custom prompts. This paper asks how much prompting improves sample efficiency and introduces a metric to quantify that benefit.
- Prompt-based classification adapts pretrained language models through autoregressive generation or cloze-task completion instead of an explicit classifier head.
- Task-specific prompts are argued to provide extra classifier information, particularly when fine-tuning with limited data.
- The study isolates prompting effects through diverse prompts, multiple runs, and low-data fine-tuning practices.
- The average data advantage metric quantifies prompting's practical impact in equivalent training-data units.
- 3500 data points on MNLI, 280 on RTE, and up to 750 on BoolQ are attributed to prompting.
2 Related Work
Prior work uses prompting for zero-shot prediction, priming, and fine-tuning, while PET combines prompts with additional techniques. This paper narrows the comparison to prompting itself under supervised fine-tuning and human-written prompts.
- Prompting has been applied to zero-shot generation, supervised priming, and word-based fine-tuning for classification.
- PET adds pseudo-labeling, prompt ensembling, and distillation, making it difficult to attribute its results specifically to prompting.
- This study aims to isolate the specific contribution of prompting within supervised fine-tuning.
- The experiments use human-written prompts rather than automatically discovered prompts, leaving automated prompt impact as an open question.
3 Comparison: Heads vs Prompts
The comparison contrasts generic classifier heads with task-specific prompt patterns that elicit textual class outputs. Prompt fine-tuning trains the model to produce the correct verbalization from pattern-verbalizer pairs.
- Head-based models use a generic layer over pretrained representations, whereas prompt-based models use task-specific patterns to produce textual class outputs.
- A prompt consists of a pattern that converts input into a cloze task and a verbalizer that maps masked-token predictions to classes.
- In the BoolQ example, the pattern presents a question with a masked answer, and verbalizer tokens represent True or False.
- Several pattern-verbalizer pairs can serve one task, differing in pattern, verbalizer, or both.
- Prompt fine-tuning minimizes cross-entropy between the correct verbalization and probabilities assigned to verbalizer tokens.
4 Experimental Setting
Experiments compare prompted and head-based fine-tuning across English classification tasks, data scales, and repeated runs using a shared pretrained setup. Figure 1 visualizes performance differences and the matched-accuracy regions used to estimate data advantage.
- Experimental setup: All experiments use the same roberta-large checkpoint with 355M parameters.
- Experimental setup: The evaluation covers SuperGLUE and MNLI tasks spanning entailment, multiple-choice question answering, and commonsense reasoning.
- Data scales: Training data scales from 10 examples exponentially to each task's full dataset.
- Figure 1: Figure 1 compares the best prompt and head performance at each data level across six SuperGLUE tasks and four runs.
- Figure 1: The cross-hatched matched-accuracy region estimates data advantage from the area between linearly interpolated performance curves.
- Repeated runs: 1892 training runs are conducted across tasks, with every experiment repeated four times to reduce variance.
5 Results
Prompt-based fine-tuning generally outperforms head-based fine-tuning across data scales, except on WiC, while the size of the advantage varies by task. The study quantifies this difference as an equivalent number of training data points.
- Prompting provides a substantial advantage on every evaluated task except WiC, and remains better as training data increases.Both approaches improve with more data, but the gap varies across tasks.
- The average data advantage assumes the head- and prompt-based curves would eventually match asymptotically.The analysis is limited by the available data range.
- 3500 data points is the approximate average training-data advantage of prompting on MNLI.The advantage is computed by comparing the areas between interpolated head and prompt performance curves.
- Hundreds of additional data points is the typical prompting advantage across almost all tasks.The paper reports this quantity as Average advantage for every task.
6 Analysis
The analysis tests whether prompting’s benefit comes from semantic verbalizers or task patterns, and examines sensitivity to prompt choice and metric parameterization. Prompting can retain data-efficiency benefits with null verbalizers, while prompt-choice effects are usually smaller than run-to-run variance.
- Average Advantage: 3500 additional data points is the reported MNLI average prompting advantage; MultiRC’s value is a lower bound because the head baseline remains near majority-class performance.Table 1 reports advantages for MNLI and SuperGLUE tasks, with P denoting prompt and H denoting head models.
- Impact of Pattern vs Verbalizer: With more training data, models can adapt a null verbalizer while retaining the pattern’s inductive-bias benefit.A null verbalizer replaces semantic class words with random first names; on small-data CB, it removes much of prompting’s benefit.
- Impact of Different Prompts: Prompt-choice differences usually fall within run-to-run variance, except in low-data BoolQ, where one prompt has a significant few-shot advantage.MultiRC illustrates the generally inconsistent prompt differences across runs.
- Metric sensitivity: Changing the metric parameterization has no consistent effect for or against prompting.Emphasizing gains near convergence increases the advantage on CB and MNLI but decreases it on COPA and BoolQ.
7 Conclusion
The paper concludes that prompting delivers task-dependent data-efficiency gains throughout training, is generally robust to pattern choice, and can learn with an uninformative verbalizer. Its benefits remain useful on large datasets, though they are smaller in performance terms.
- Prompting consistently yields varying improvements throughout training, with benefits that are mostly robust to pattern choice.The analysis also finds that prompting can learn without an informative verbalizer.
- On large datasets, prompting remains similarly helpful in equivalent data points but is less beneficial in performance.The paper identifies data efficiency, rather than compute efficiency, as prompting’s main benefit.
8 Impact statement
The impact statement emphasizes prompting’s data-efficiency benefit while noting substantial computational cost and risks from human and pretrained-model biases.
- Impact and cost: The experiments consumed about 2,200 GPU hours, including roughly 1,800 hours for runs and 400 hours for prototyping and search.The authors estimate this would usually correspond to about 400kg of CO2, although they used a carbon-neutral cloud provider.
- Impact and cost: Prompting’s main benefit is data efficiency rather than compute efficiency.The experiments used about 2,200 GPU hours, despite individual runs usually taking under an hour.
- Risks: Prompting may introduce human biases and reproduce biases already present in the pretrained language model.The authors identify this risk as especially relevant in few-shot settings where prompting relies heavily on the pretrained model.
B Influence of the reporting method over runs
The reported prompting advantage depends on how performance across runs is summarized: accumulated maxima preserve prior peaks, whereas means can vary substantially under skewed outcomes.
- Maximum reporting: Accumulated maximum reporting retains a previously attained maximum when later performance at more data is lower.This approach reports the best performance achieved across runs while preserving earlier higher values.
- Mean reporting: Mean reporting can vary significantly because run outcomes are heavily left-skewed or even bimodal, with poor-performance outliers.The appendix contrasts mean reporting with maximum-based summaries.
- Prompting advantage: On average across performance levels, an MNLI prompt model matches an MNLI head model trained with 3500 additional data points.Table 2 reports this comparison using accmax reporting.
C Curves on all tasks
The figures compare prompted and head-based classification performance across training-data scales and summarize prompt performance across tasks.
- Figure 4: Figure 4 compares the best prompt and head performance at each training-data level across four runs for seven SuperGLUE tasks and MNLI.The curves extend to the full dataset.
- Figure 4: Figure 4 uses accuracy differences and matched-accuracy regions to estimate prompting’s data advantage.Cross-hatching marks the lowest- and highest-accuracy matched regions of the curves.
- Figure 5: Figure 5 reports median performance across runs for each prompt on every task.