True Few-Shot Learning with Language Models

Ethan Perez; Douwe Kiela; Kyunghyun Cho

Source-linked AI summary

True Few-Shot Learning with Language Models

Ethan Perez, Douwe Kiela, Kyunghyun Cho

arXiv:2105.11447v1May 24, 2021cs.CLcs.LGstat.ML

TL;DR

Prior few-shot LM studies often tune prompts and hyperparameters with many held-out examples, leaving performance unclear when such data are unavailable. This paper evaluates true few-shot model selection using cross-validation and minimum description length, finding marginal gains over random selection but substantially worse performance than held-out selection. The findings indicate that model selection is a major difficulty for practical few-shot learning.

Problem
Few-shot LM performance depends on choices such as prompts and hyperparameters, while prior work often selects them using large validation sets unavailable in true few-shot learning.
Method
The paper evaluates cross-validation and minimum description length for selecting prompts and hyperparameters using only a small training set from a single distribution.
Results
True few-shot selection marginally outperforms random selection but greatly underperforms held-out-example selection, with prompt selection 5-7% absolute below the best held-out prompt on average.
Takeaways & Limitations
The results suggest that prior work significantly overestimated LM true few-shot ability because effective model selection is difficult without many labeled examples.
Takeaways & Limitations
The study is not exhaustive, and other selection criteria or unexplored heuristics might enable effective true few-shot model selection.

Abstract

from arXiv · show

Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.

1 Introduction

Few-shot LM performance depends heavily on prompts, example order, decoding, hyperparameters, and learning algorithms, making model selection crucial. This paper questions whether prior methods remain effective when large validation sets are unavailable.

Few-shot learning lets LMs learn tasks from small numbers of examples, avoiding the expense and poor scalability of collecting labeled data.Labeled data collection is expensive, often requires experts, and scales poorly across tasks.
LM few-shot performance is highly sensitive to prompts, example order, decoding strategy, hyperparameters, and learning algorithms.These dependencies make effective model selection important for obtaining good few-shot performance.
Prior work often uses large train or held-out sets to select prompts and hyperparameters, while methods claiming no validation set omit how other algorithmic choices were designed.The paper argues that sophisticated algorithms likely used validation examples even when this was not described.
True few-shot model selection marginally outperforms random selection but greatly underperforms held-out-example selection.The result motivates evaluating few-shot methods without large validation sets.
Both cross-validation and minimum description length provide only limited improvements over random selection and perform much worse than held-out-example selection.The study evaluates these two standard criteria for selecting prompts and hyperparameters.
True few-shot hyperparameter selection reduces performance by 2-10% across 8 tasks for ADAPET and often selects models worse than random choices.Similar findings persist when varying examples, computation, and selection conservativeness.

2 Can We Do Model Selection in Few-Shot Learning?

The paper distinguishes few-shot settings by their available data and evaluates whether model selection is possible with only a small training set from one distribution. It focuses on cross-validation and minimum description length, while emphasizing both expected performance and selection variance.

Few-shot learning has multi-distribution, tuned, and true settings that assume access to different kinds and amounts of data.The paper introduces this distinction to prevent comparisons between methods operating under different data assumptions.
Multi-distribution few-shot learning evaluates algorithms using many small train-validation sets drawn from distinct distributions.Each task distribution is sampled from a distribution over related distributions.
Tuned few-shot learning uses a large validation set from one distribution to choose prompts, stopping criteria, and other hyperparameters.The paper argues that relying on many labeled validation examples undermines the practical meaning of few-shot learning.
True few-shot learning selects an algorithm using only a small training set drawn from a single distribution, typically by approximating expected loss.Cross-validation is given as one way to approximate the algorithm’s expected loss.
The study evaluates cross-validation and minimum description length as model-selection criteria for tuned methods in the true few-shot setting.Cross-validation forms train-validation splits from labeled examples, while MDL uses online coding across progressively larger training folds.
Cross-validation and MDL aim to estimate generalization, with MDL using label compression and online coding to evaluate performance as data increases.Better compression is treated as implying better generalization.
Reliable selection requires low expected loss and low variance across datasets and random factors.The paper links low variance to reliably choosing algorithms that generalize, which matters when worst-case performance is important.
The study also tests variance-aware selection and notes that alternative criteria or heuristics might achieve effective true few-shot model selection.Variance-aware selection uses a conservative estimate of cross-validation performance based on a desired probability.

3 True Few-Shot Prompt Selection

The paper evaluates prompt and model selection when only a few labeled examples are available, finding that CV and MDL struggle to identify reliably effective prompts across models, data regimes, computation levels, and tasks.

3.1 Experimental setup: Few-shot prompting supplies examples as input to a pretrained LM, without parameter updates, while prompt wording and other design choices strongly affect performance.The setup uses manually written and paraphrased candidate prompts, with negative log-likelihood for CV and MDL evaluation.
3.2 How well does prompt selection do in true few-shot learning?: CV and MDL improve marginally over random prompt selection but remain 5-7% below held-out-performance-selected prompts on average.They close only 20-40% of the gap between average and best prompts, across model sizes spanning three orders of magnitude.
3.3 How reliably does prompt selection improve over the average prompt?: CV/MDL-chosen prompts improve over average prompts only about 56% and 55% of the time, respectively, for most model sizes.Selection has a long-tailed outcome distribution, including an approximately 27% chance of a roughly 13% accuracy drop.
3.4 Can we increase the likelihood of improved performance from prompt selection?: Variance-minimizing conservative CV produces a performance-gain distribution similar to ordinary CV and does not reliably improve over random selection.CV outperforms the average prompt 50% of the time, compared with 51% for α = 2.
3.5 Does prompt selection improve with more labeled examples?: Increasing the number of labeled examples up to 40 does not consistently improve prompt-selection performance or accuracy at identifying the best prompt.This pattern holds across model sizes, with CV/MDL still struggling even in higher-data regimes.
3.6 Does prompt selection improve with more computation?: Additional computation does not notably improve selection after roughly one MDL forward pass or N CV forward passes.The plateau appears across shot counts and models below 6.7B parameters, suggesting available examples—not computation—constrain selection.
3.7 To what extent are chosen prompts specific to the model?: Prompt-transfer behavior differs by criterion: test-selected prompts generalize reasonably across similarly sized models, whereas CV transfer is weaker and MDL is not strongly model-specific.Thus, selection criteria differ not only in accuracy but also in how prompts transfer between model sizes.
3.8 Is prompt selection challenging on other tasks?: Across RTE, CB, and WiC, CV/MDL prompts generally trail the best prompts and often perform worse than average, with especially poor outcomes on CB.For GPT-3 175B on CB, CV has a 20% chance of producing a 15% accuracy drop; improvement frequencies vary widely by task and model size.

4 True Few-Shot Hyperparameter Selection

The paper evaluates whether cross-validation and minimum description length can select ADAPET hyperparameters in the true few-shot setting. Across SuperGLUE, these criteria perform no better than random selection and below validation-selected choices.

Experimental setup: CV/MDL choose ADAPET’s early-stopping checkpoint and masked-LM rate using few-shot validation examples.The evaluated masking fractions are 0.075, 0.10, 0.105, and 0.15; checkpoints are selected from 250, 500, 750, and 1000 updates.
Results: Across all SuperGLUE tasks, CV/MDL-selected hyperparameters perform similarly to or worse than average randomly chosen hyperparameters.The comparison covers four 32-shot training sets and reports mean and standard deviation.
Results: CV/MDL-selected hyperparameters perform several points worse than the best validation-selected hyperparameters.On MultiRC, CV/MDL selects hyperparameters with performance similar to the worst hyperparameters.
Results: True few-shot hyperparameter selection lowers ADAPET’s average SuperGLUE performance below earlier PET and iPET methods.The result highlights the difference between true few-shot selection and selection using validation examples.
Results: Choosing better-than-average hyperparameters may require several thousand examples, making model selection difficult in very low-data regimes.The paper reports this as a preliminary analysis and concludes that model selection is challenging beyond prompt selection.

5 Conclusion and Future Work

The conclusion argues that true few-shot learning makes basic algorithm decisions difficult and recommends explicit assumptions and strict evaluation protocols. It further identifies model selection without labeled data as an even harder open problem.

Future work: True few-shot learning may require additional assumptions, such as meta-learning, transfer learning, data augmentation, or unlabeled data.The paper considers these directions more promising than true few-shot learning given the difficulty of model selection.
Recommendations: Few-shot studies should report all prompts and hyperparameters considered, along with the criteria used to select them.This recommendation makes model-selection choices auditable.
Recommendations: Validation examples should count toward the examples used by a few-shot algorithm, including examples used for prompts, objectives, decoding, or architecture decisions.The protocol treats every example used to decide an aspect of learning as validation data.
Recommendations: Researchers should submit a finalized model directly for testing and report the total number of test evaluations conducted.Validation should be reserved for post-test ablations rather than algorithm decisions.
Recommendations: Future benchmarks may need to keep large test sets private while releasing only a few labeled examples to enforce strict protocols.The proposed benchmark design aims to prevent researchers from using hidden test data for model-selection decisions.
Future work: Model selection in a true zero-shot setting is presented as an open question and is expected to be harder than in true few-shot learning.The paper notes that zero-shot prompt tuning can otherwise rely on human supervision or large labeled training sets.

6 Limitations and Broader Impact

The paper tests alternative criteria and broader implications for true few-shot learning, while acknowledging that its study is not exhaustive. It also highlights risks for deployment and aims to redirect future work toward practical few-shot settings.

Limitations and broader impact: The study is not exhaustive: other selection criteria or unexplored heuristics might still enable effective true few-shot model selection.The authors note that negative findings could discourage work on low-data applications, while aiming to redirect research toward more practical settings.
Limitations and broader impact: The findings may discourage deployment of language models in useful applications and could help adversaries target susceptible models.The paper specifically mentions question-answering, hate speech detection, translation, and commercial dialogue systems.
Alternative criteria: Other generalization criteria are evaluated to determine whether poor prompt selection is specific to CV and MDL.The evaluation examines prompt accuracy and how often each criterion selects the highest-accuracy prompt.
Bayesian CV: Bayesian CV estimates held-out-fold loss after marginalizing over models using a posterior distribution.The setup assumes a uniform prior over random factors and independence between random factors and inputs.
Interpolating CV and MDL: The interpolated criterion weights held-out-fold losses across increasing training sizes, with MDL and CV as limiting cases.MDL uses uniform weights over train sizes, whereas CV weights only the largest train size.
Alternative loss: The study also evaluates joint input-label log-probability as an alternative loss for CV and MDL.This loss adds input NLL to the label NLL and is computed over all tokens in prompt-formatted input-label pairs.

A.4 Results

Across model sizes, alternative criteria choose prompts with accuracy close to randomly selected prompts and are similarly inaccurate at selecting the highest-accuracy prompt. The figure also examines ADAPET accuracy as examples increase and shows variation across 16 hyperparameter settings.

Prompt selection: All evaluated criteria select prompts with average LAMA-UHN accuracy close to randomly selected prompts across model sizes.Figure 8 compares criteria-selected prompts with worst, average, and best prompts across five training-set runs.
Prompt selection: All criteria are similarly inaccurate at selecting the highest-accuracy prompt.Figure 8 reports mean and standard error across five runs with different training sets.
Hyperparameter selection: Figure 10 plots ADAPET accuracy from cross-validation-chosen hyperparameters as the number of examples increases.The shaded region spans accuracies from 16 hyperparameter settings trained on the same training set.

B Additional Results with MDL

Additional MDL results reproduce the main difficulty observed with CV: selection has high variance, does not consistently improve with more examples, and remains challenging near the context limit.

Variance: MDL-chosen prompts have high test-accuracy variance relative to average prompts, especially for larger models.The same pattern was previously observed for CV, suggesting the variance is not specific to CV.
Effect of examples: Increasing examples does not consistently improve MDL’s gain over random selection relative to the best prompt.Some model sizes may improve, but standard errors are high.
Effect of examples: Increasing examples does not consistently improve MDL’s accuracy at selecting the best prompt.Overall MDL remains challenging even as the number of examples approaches the GPT context limit.

C How many examples do you need for effective model selection?

The study tests how many examples are needed for cross-validation to select better-than-average ADAPET hyperparameters. The required number varies by task and can remain large even with thousands of examples.

Experimental setup: The analysis varies ADAPET’s training-example count while keeping the experimental setup and hyperparameter range from Section 4.ADAPET permits an arbitrary number of examples, unlike GPT-based models constrained by input length.
Task-dependent requirements: For WiC and MultiRC, cross-validation requires >2-3k examples to select better-than-average hyperparameters.Figure 10 compares CV-selected hyperparameters with worst, average, and best settings.
Task-dependent requirements: For BoolQ, cross-validation performs similarly to average hyperparameters even with up to 9k examples.The authors suggest retraining variance may contribute to this result.

D.1 LAMA

The appendix describes prompt selection and prompt-based classification procedures, alongside the SuperGLUE tasks used for evaluation. Prompts are filtered or selected for compatibility with left-to-right language models.

Prompts Used: LAMA prompts are drawn from a larger collection, with up to 90 prompts available per relation and a smaller subset used for evaluation.The full LPAQA prompt list is available externally, but the experiments use only a subset.
Prompts Used: For RTE, CB, and WiC, the appendix lists prompts and varies textual label names used to obtain output-class logits.For RTE, “True” and “False” logits represent entailment and non-entailment, and class probabilities are computed with a softmax.
Prompts Used: Prompts ending with a blanked target answer are filtered out because they are difficult to use with left-to-right language models such as GPT.Mined prompts are selected by Wikipedia frequency, while paraphrased prompts are selected up to a fixed limit.
Datasets: SuperGLUE evaluation includes Boolean question answering, causal-choice prediction, textual entailment, word-sense disambiguation, coreference resolution, and multi-sentence reading comprehension.The listed tasks include BoolQ, COPA, RTE, CB, WiC, WSC, and MultiRC.

D.3 Dataset and model licenses

The appendix records licenses and computational resources for the datasets and models, and describes how MDL handles the first training-data fold when finetuning cannot begin without examples.

Dataset and model licenses: The SuperGLUE and LAMA licenses permit research use, while the datasets contain no private personally identifiable information but may include offensive content.GPT-2 and DistilGPT-2 use modified MIT licensing, while GPT-3 access is licensed through the OpenAI API.
Method: MDL assigns a uniform distribution over labels on the first fold because finetuning requires initial training data.This corresponds to compressing the first fold using a uniform distribution rather than a learning algorithm.
Computational resources: GPT-3 experiments cost $2826.73 in total, while GPT-2 and DistilGPT-2 inference used one AMD MI50 GPU for at most 8 hours.ADAPET training used the same GPU for up to 12 hours.