Multitask Prompted Training Enables Zero-Shot Task Generalization

• The authors hypothesize that explicit multitask supervision in pretraining plays a large role in zero-shot generalization and test whether supervised multitask training can induce it without massive scale.Source: §1 · p. 2
• The approach templates natural-language tasks into prompted datasets, trains a T5 variant on a subset, and evaluates it on tasks and prompts absent from training.A public-contributor interface supported collecting multiple prompts per dataset.Source: §1 · p. 2
• Our model matched or exceeded GPT-3 (Brown et al., 2020) on 9 of 11 held-out datasets despite being about 16× smaller.Source: §1 · p. 3
• It improved over a large baseline language model on 13 of 14 BIG-bench tasks.Source: §1 · p. 3
• More prompts per dataset consistently improved median performance and reduced variability on held-out tasks; prompts from more datasets generally improved the median but did not consistently reduce variability.Source: §1 · p. 3

2 RELATED WORK

• Explicit multitask learning mixes tasks in supervised training; prior work found NLP gains, used shared formats, and explored generalization to new datasets with pretrained models.These lines of work are associated with Caruana (1997), Collobert and Weston (2008), Hashimoto et al. (2016), McCann et al. (2018), Vu et al. (2020), and Ye et al. (2021).Source: §2 · p. 3
• Single-prompt approaches recast datasets as QA or NLI, but typically do not generalize to new prompts or tasks outside their fixed formats.Text-to-text models such as T5 (Raffel et al., 2020) make natural-language prompts useful for multitask learning; examples include Khashabi et al. (2020a), Zhong et al. (2021), and Wang et al. (2021).Source: §2 · p. 3
• Unlike Mishra et al. (2021), who trained on 61 narrowly defined tasks from 9 datasets with labeled examples, this work studies zero-shot generalization across 62 datasets and 12 traditional NLP tasks.The paper also distinguishes its work from concurrent research by Wei et al. (2021) on prompt diversity, model scale, and held-out-task scheme.Source: §2 · p. 3
• Although prompts are hypothesized to act as task instructions, challenges to the role of semantic meaningfulness lead the authors to remain agnostic about why prompts support generalization.They claim only that prompts are a natural multitask-training format that empirically supports generalization to held-out tasks.Source: §2 · p. 3

• Zero-shot generalization is evaluated by training on a subset of NLP tasks and testing on datasets from held-out tasks, where each task groups datasets measuring a general ability.Because task boundaries are imperfect, the taxonomy prioritizes task format over required skill, following conventions in prior work (Khashabi et al., 2020b; Vu et al., 2020; Ye et al., 2021).Source: §3 · p. 3
• The task taxonomy treats task grouping as an imperfect heuristic because datasets assigned to broad skills such as commonsense vary substantially in their content and difficulty.Examples range from innate knowledge and grade-school science to DIY instructions, US cultural norms, and graduate-level theorems.Source: §3 · p. 3
• The dataset collection contains 12 tasks and 62 datasets with publicly contributed prompts, excluding non-English data and tasks requiring special domain knowledge such as biomedicine.Experiments use datasets from the Hugging Face datasets library (Lhoest et al., 2021).Source: §3 · p. 4
• All datasets from four task types—natural language inference, coreference resolution, sentence completion, and word sense disambiguation—are held out for zero-shot evaluation.Sentence completion is included because it may be too similar to NLI; the main model excludes datasets used in Brown et al. (2020) evaluations, and the authors check for pretraining-data leakage.Source: §3 · p. 4
• BIG-bench evaluation uses a language-oriented subset with preliminary maintainer results and English text compatible with the T5 tokenizer; all selected tasks are held out from training.The benchmark is a community-driven collection of difficult tasks for assessing large language models.Source: §3 · p. 4

4 A UNIFIED PROMPT FORMAT

• Each dataset is rendered in natural language using input and target templates that combine data fields with text and metadata, enabling zero-shot experimentation.Metadata can specify label choices and evaluation metrics; each example can be materialized with multiple templates.Source: §4 · pp. 4–5
• 36 contributors from 24 institutions in 8 countries created diverse prompts that were grammatical, understandable, and expressed without explicit counting or numerical indexing.Prompt writers were encouraged to use formal or creative wording and varied data orderings.Source: §4 · p. 5
• Prompts that permuted the original task were included in training mixtures for diversity but excluded from evaluation because they diverged from original dataset metrics and baselines.Source: §4 · p. 5
• P3 contains 2073 prompts for 177 English datasets, averaging 11.7 prompts per dataset; experimental prompts come from P3 except BIG-bench prompts supplied by its maintainers.The collection excludes datasets with potentially harmful content or non-natural language such as programming languages.Source: §4 · p. 5

Model

• The model encodes input text and autoregressively generates target text using standard maximum-likelihood training, rather than generating the input.Unlike decoder-only language models such as GPT-3, the model is not trained to generate its input.Source: §5 · pp. 5–6
• All models start from T5, pretrained on 1T C4 tokens, then use T5+LM, adapted on 100B additional C4 tokens with a standard language-modeling objective (Raffel et al., 2020; Lester et al., 2021).This adapts T5’s masked-token generation objective toward the natural text generation format of prompted datasets.Source: §5 · p. 6

Training

• T0, T0+, and T0++ use 11B-parameter T5+LM initialization, with training mixtures expanding to include GPT-3 evaluation datasets and SuperGLUE (Wang et al., 2019a).T0++ excludes RTE and CB from SuperGLUE, leaving NLI and BIG-bench tasks held out; T0 (3B) uses the T0 mixture with a 3B-parameter initialization.Source: §5 · p. 6
• The selected checkpoint maximizes validation performance on training datasets, without using examples from held-out tasks (Perez et al., 2021).This checkpoint-selection procedure preserves the true zero-shot setting.Source: §5 · p. 6
• Training examples are combined and shuffled, sampling datasets in proportion to size while capping large datasets at 500,000 examples divided by their number of templates.The cap addresses dataset sizes that otherwise vary by two orders of magnitude.Source: §5 · p. 6
• Training truncates inputs and targets to 1024 and 256 tokens, respectively, and uses packing, batches of 1024 sequences, Adafactor, a 1e-3 learning rate, and 0.1 dropout.Packing combines examples to reach the maximum sequence length.Source: §5 · p. 6

Evaluation

• Evaluation covers 11 datasets across four held-out traditional NLP tasks and 14 novel BIG-bench tasks, using accuracy and, unless specified, validation splits.The traditional tasks are natural language inference, coreference, word sense disambiguation, and sentence completion.Source: §5 · p. 6
• For choice tasks, rank classification selects the option with the highest target log-likelihood without length normalization; prompt results use median performance and interquartile range, not validation-based prompt selection (Brown et al., 2020; Perez et al., 2021).Reporting across prompts measures robustness to prompt wording and avoids validation-based selection that can leak evaluation information.Source: §5 · p. 6

6.1 GENERALIZATION TO HELD-OUT TASKS

• T0 matches or exceeds GPT-3 on 9 of 11 held-out datasets and outperforms it on every NLI dataset, while also gaining over the identical T5+LM baseline on all compared datasets.The comparison uses T0’s median and interquartile range across prompts, whereas GPT-3 reports a single prompt per dataset.Source: Figure 4 · p. 7, §6.1 · p. 7
• On BIG-bench, at least one T0 variant outperforms all diagnostic baselines on every task except StrategyQA, and performance usually improves as training datasets increase.BIG-bench tasks are zero-shot for all reported models; T0, T0+, and T0++ differ in their numbers of training datasets.Source: §6.1 · p. 7, Figure 5 · p. 8

6.2 PROMPT ROBUSTNESS

• With dataset count fixed, increasing prompts from one to an average of 5.7 raised median performance on 8/11 held-out datasets and narrowed spread on 7/11; using all prompts improved those counts to 9/11 and 8/11.One prompt per dataset already substantially improved held-out performance over the non-prompted baseline, though its spread did not consistently improve; all-prompt T0 averages 8.03 prompts per dataset and includes non-original-task prompts.Source: §6.2 · p. 8, Figure 6 · p. 9
• Increasing training datasets from 39 to 49 to 55 raised median performance on all five held-out datasets at each step, but reduced spread for only 1/5 and then 2/5 datasets.Thus, increasing dataset count improved performance but did not consistently improve robustness to prompt wording.Source: §6.2 · p. 8
• On RTE, nine of GPT-3’s ten evaluated prompts yielded roughly random-guessing performance, with median accuracy = 52.96% and interquartile range = 1.28%, suggesting T0 may be more robust to prompt wording.The ten prompts were the same ones used to evaluate T0; the prompt identical to Brown et al. (2020, p. 59) scored 58.8%, versus 63.5% reported there.Source: §6.2 · pp. 8–9

• T0 beats FLAN on CB and RTE, matches it on Story Cloze and COPA, and trails on Winogrande and ANLI despite 11B versus FLAN’s 137B parameters.T0++ outperforms FLAN on CB, RTE, and COPA and matches its performance on Winogrande and ANLI.Source: §7 · p. 9
• Removing instructions raises T0’s HellaSwag median from 33.65% to 57.93%, matching FLAN, while FLAN’s instruction-free Winogrande prompt yields no substantial change (accuracy = 62.15%).Source: §7 · p. 9
• Unlike Wei et al. (2021)’s comparable-size 8B ablation, multitask prompted training improves held-out performance in models as small as 3B, potentially reflecting masked-language-model pretraining and more diverse prompts.The authors identify the encoder-decoder model’s pretraining objective and qualitatively more diverse prompts as possible explanations for the discrepancy.Source: §7 · pp. 9–10
• Adding datasets consistently raises median zero-shot performance across T0, T0+, and T0++, but does not always narrow held-out-task interquartile ranges.Source: Figure 7 · p. 10

8 CONCLUSION

• Multitask prompted training enabled strong zero-shot generalization and offered an effective alternative to unsupervised pretraining, with T0 often outperforming models many times its size.Source: §8 · p. 10
• Ablations demonstrated the importance of including many diverse prompts and examined the impact of increasing the number of datasets in each task.Source: §8 · p. 10
• The authors released all models trained in the paper, their prompt collection, and a prompt annotation tool to support future work on zero-shot generalization.Source: §8 · p. 10

A CONTRIBUTIONS AND PROJECT STRUCTURE

• BigScience was a year-long open-research initiative studying large models and datasets outside large technology companies, involving 600 researchers from 50 countries and more than 250 institutions [4].Source: A CONTRIBUTIONS AND PROJECT STRUCTURE · p. 18
• The work was led by joint first authors: Sanh and Bach led prompt collection, Webson led dataset selection and results analysis, and Raffel trained the models and built the main evaluation system.Rush helped develop the prompt templating language and tool and co-managed paper writing.Source: A CONTRIBUTIONS AND PROJECT STRUCTURE · p. 18
• Authorship included contributors who supplied at least three accepted prompted datasets, and authors were ordered based on code contributions.Source: A CONTRIBUTIONS AND PROJECT STRUCTURE · p. 18
• The paper highlights contributors’ work on evaluation and writing, prompting-tool development, model and dataset release, and contamination analysis.The named contributors include Lintang Sutawika, Urmish Thakker, Mike Tian-Jian Jiang, Shanya Sharma, Arnaud Stiegler, Manan Dey, M Saiful Bari, and Teven Le Scao.Source: A CONTRIBUTIONS AND PROJECT STRUCTURE · p. 18

B.1 ENVIRONMENTAL COSTS

• T0’s single run is listed at 0.9 tCO2e, versus 46.7 tCO2e for T5-11B’s single run; all experiments in the paper are listed at 8.9 tCO2e.The paper notes that T0 is based on the largest T5 variant, whose emissions were estimated at 46.7 tCO2e.Source: §B.1 · pp. 18–19, Table 1 · p. 19

B.2 RISKS IN DEVELOPING AND RELEASING LARGE LANGUAGE MODELS

• The authors report strong zero-shot capabilities from multitask prompt training, but performance remains significantly behind traditional fine-tuning on the given task.They describe the work as an empirical exploration and say further research is needed.Source: §B.2 · p. 19
• The work uses public Hugging Face datasets (Lhoest et al., 2021) and T5+LM (Lester et al., 2021), and releases its prompts, trained models, and materialized prompted data.The authors argue that withholding the resources would chiefly affect less-funded researchers and individuals.Source: §B.2 · p. 19

B.3 BIAS AND FAIRNESS

• To avoid training on potentially harmful content, the authors excluded datasets such as Sentiment140 and Hate Speech 18 (de Gibert et al., 2018).The excluded examples came from Twitter content and a white supremacist forum, respectively.Source: §B.3 · pp. 19–20
• Despite excluding potentially harmful training data, T0++ remained bias-prone, producing conspiracist answers and gender-stereotyped completions such as “Architect” for a man and “Nanny” for a woman.The examples include answering yes to “Is the earth flat?” and “Do vaccines cause autism?”Source: §B.3 · p. 20, Table 2
• The bias evaluation tests recognition with WinoGender and CrowS-Pairs classification accuracy, and reproduction with WinoBias coreference accuracy across pro- and anti-stereotype subsets.The authors evaluate 5–10 prompts for each recognition dataset and six prompts for WinoBias.Source: §B.3 · p. 20

C ANNOTATION SYSTEM - PROMPTSOURCE

• PromptSource’s lightweight, locally run Streamlit interface uses helicopter, sourcing, and prompted-dataset views to help users select datasets, write templates, and verify generated prompts.The prompted-dataset view displays each original example beside its generated prompt, with substituted text highlighted.Source: C ANNOTATION SYSTEM - PROMPTSOURCE · pp. 20–21
• PromptSource uses Jinja as a middle ground between expressive code and rigid rules, with conditionals, controlled choice, and explicit valid-completion lists.Conditional templates can skip inapplicable examples, while choice functions can vary prompts reproducibly or generate combinations; valid completions support scoring candidate answers.Source: C ANNOTATION SYSTEM - PROMPTSOURCE · pp. 21–22

D.1 CATEGORIZING DATASETS INTO TASKS

• The taxonomy includes sentiment and topic classification, paraphrase identification, NLI, word sense disambiguation, coreference resolution, summarization, and structure-to-text generation.For QA, CrossFit and UnifiedQA group by format, while Brown et al. (2020) group by content.Source: §D.1 · p. 22
• The discussion contrasts format-based QA groupings in CrossFit and UnifiedQA with Brown et al. (2020)'s content-based categories, including reading comprehension, commonsense, and closed-book QA.Source: §D.1 · p. 22
• Content-based QA categories seem more meaningful than format-based ones, and closed-book QA is a relatively clear distinct task assessing world-knowledge memorization.The authors note that commonsense boundaries are blurrier and dataset definitions vary, often involving physical cognition and US-centric cultural norms.Source: §D.1 · p. 22
• Brown et al. (2020, p. 17) define commonsense as physical or scientific reasoning distinct from other QA types, but the authors question whether scientific reasoning is commonsense.They find most ARC questions difficult to answer, and to a lesser degree OpenBookQA questions, despite ARC's grade-school science origin.Source: §D.1 · p. 22
• Because NLI and coreference datasets often require commonsense, the authors find it difficult to define commonsense as a standalone task category.Source: §D.1 · p. 22
• The authors therefore default to categorizing QA by format, placing ARC in multiple-choice QA because closed-book QA requires answers without provided options.Source: §D.1 · p. 22

D.2 HOW UNSEEN ARE THE HELD-OUT TASKS?

• Because QA is broadly defined, held-out datasets may contain entailment or coreference examples; ReCoRD, held out as SuperGLUE, exclusively asks for referent identification.The authors say inspecting every dataset and slicing out such examples is impractical.Source: §D.2 · p. 22
• Paraphrase identification was retained because it tests bidirectional, not NLI’s unidirectional, entailment; QQP-before-RTE training has also been reported to hurt versus entailment-only training (Pruksachatkun et al., 2020).Manual review of ANLI and RTE found almost no entailment examples that were also valid paraphrases.Source: §D.2 · pp. 22–23
• Because NLI’s boundary with pragmatic inference, coreference, and continuation-style prompts is blurry, the authors categorically held out sentence completion.Source: §D.2 · p. 23
• BIG-bench evaluations target diverse, difficult, novel skills, so their datasets are considered unlikely to overlap substantially with T0’s training tasks.Source: §D.2 · p. 23

D.3 LAMBADA

• LAMBADA requires open-ended next-word prediction, unlike choice-based sentence completion, although Brown et al. (2020) classify it in that task group.Brown et al. (2020) used a cloze-style prompt and a nonstandard dataset format; they estimated 57% of LAMBADA test examples appeared in GPT-3’s training set.Source: §D.3 · p. 23
• On standard unprompted LAMBADA, T5+LM scored 6.2% and T0 18.7%, versus 72.5% for comparably sized GPT-3-13B; cloze prompting raised T0 to 27.8%.On the GPT-3 variant, T0 reached 40.5% and T5+LM 10.7%; the authors omitted LAMBADA because of the format discrepancy and its dissimilarity to other sentence-completion tasks.Source: §D.3 · p. 23

D.4 TABLE OF ALL DATASETS

• Table 5 lists all training and evaluation datasets using their Hugging Face dataset identifiers, with the subset name appearing after the slash.Full citations are included in Appendix G.Source: Table 5 · p. 24
• Hotpot QA is recast as closed-book QA because its input length is too long.Source: Table 5 · p. 24

E CONTAMINATION ANALYSIS OF PRETRAINING CORPUS ON TEST TASKS

• The authors searched for long common substrings between prompted zero-shot test inputs and C4 documents, the model’s pretraining corpus.They indexed C4 with the suffix-array method of Lee et al. (2021), queried exact matches for groups of 16 tokens—over-counting length-32 overlaps—and manually inspected flagged examples.Source: E CONTAMINATION ANALYSIS OF… · p. 25
• HellaSwag had 912/10000 matches (9.12%) and RTE hypotheses 156/3000 (5.2%); ANLI hypotheses had 6/1000, while other datasets were reported free of contamination.The authors note that overlap could help on RTE when factual knowledge is useful, and could be problematic for HellaSwag because the correct continuation may occur on the same internet page as the input.Source: E CONTAMINATION ANALYSIS OF… · p. 25
• Across CB, HellaSwag, Lambada, Story Cloze, WiC, Winogrande, and WSC, matches were 1/250, 912/10000, 15/5153, 3/1871, 20/1400, 0/1767, and 4/146, respectively.Source: E CONTAMINATION ANALYSIS OF… · p. 25
• ANLI and RTE premise matches were 337/1000 and 329/3000, respectively; hypothesis matches were 6/1000 and 156/3000, respectively.The analysis separates premises from hypotheses because premises tend to come from the internet, whereas a novel paired hypothesis may limit the usefulness of memorization.Source: E CONTAMINATION ANALYSIS OF… · p. 25

F FULL RESULTS

• Figure 8 compares T0 3B with T0 11B, while Table 6 covers T5+LM and T0 variants across all tasks and Table 7 covers T0 variants on selected BIG-bench tasks.Table 6 marks results that are not zero-shot in grey.Source: Figure 8 · p. 26, Table 6 · p. 26, Table 7 · p. 27

G LIST OF ALL PROMPTS

• The appendix enumerates prompts used to train and evaluate T0 models, organized by task category and canonical Hugging Face dataset name, with examples, templates, references, and notes on prompts differing from original task descriptions.Some entries also include answer-choice templates; the prompt collection's most up-to-date version is identified at the project repository.Source: G LIST OF ALL PROMPTS · pp. 28–29

1.1.1 CROWS PAIRS

• CrowS-Pairs, from Nangia et al. (2020), is identified as an evaluation dataset.The supplied data example includes more- and less-stereotypical sentence fields, annotations, and anonymized annotator IDs.Source: §1.1.1 · p. 31

Data Example

• The CrowS-Pairs example lists paired `sent more` and `sent less` fields, five annotation entries, and five anonymized annotator IDs.The sentence text is truncated in the supplied passages.Source: Data Example · p. 31

Prompts

• CrowS-Pairs templates frame the paired sentences as stereotype or anti-stereotype judgments, sentence rewrites, or bias-category choices.One template asks whether sentence A illustrates a stereotype; another asks for a minimally modified, less stereotypical sentence.Source: Prompts · pp. 31–33

1.1.2 SUPER GLUE AXG

• Given a premise, one supplied prompt asks whether to assume the hypothesis is true and explicitly requests a “Yes or no?” answer, framing the premise–hypothesis judgment as binary.Source: Prompts · p. 34

1.2.1 SUPER GLUE WSC.FIXED

• SuperGLUE WSC.fixed, from Levesque et al. (2012), is identified as an evaluation dataset, and its supplied prompts ask whether a pronoun refers to a named span.The examples include yes/no and true/false formulations.Source: §1.2.1 · p. 37, Prompts · pp. 37–38

1.2.2 WINOGRANDE WINOGRANDE XL

• Winogrande XL, from Sakaguchi et al. (2019), is identified as an evaluation dataset, with prompts asking which of two options fills a blank or resolves a reference.The supplied example shows a sentence field, and prompt outputs include the selected option.Source: §1.2.2 · p. 41, Data Example · p. 41, Prompts · p. 41

Data Example

• The CB example pairs the hypothesis “the language was peeled down” with the premise “It was a complex language,” followed by “Not written down but ha...”.Source: Data Example · p. 43

Prompts

• CB prompts express inference judgments in varied forms, using answer sets such as yes/no/maybe and true/false/inconclusive.Examples include asking whether a hypothesis follows from a premise or is guaranteed true.Source: Prompts · pp. 43–46

1.3.2 SUPER GLUE RTE

• The RTE dataset is attributed to Dagan et al. (2005) and is identified as used in evaluation.Source: §1.3.2 · p. 48

Prompts

• RTE prompts ask whether a hypothesis follows from a premise, recasting the question across yes/no and true/false formulations.Source: Prompts · pp. 48–50

1.3.3 ANLI

• ANLI is attributed to Nie et al. (2020) and is identified as used in evaluation; its example includes a trolleybus-route hypothesis and multiple prompt formulations.The shown prompt variants ask whether the premise implies the hypothesis, including yes/no/maybe and true/false/neither answer sets.Source: §1.3.3 · p. 51, Data Example · pp. 51–53

Data Example

• Inference prompts rephrase premise–hypothesis questions and vary answer labels, including yes/maybe/no and always/sometimes/never.Examples ask whether a hypothesis follows from a premise or is always, sometimes, or never true.Source: Data Example · pp. 54–56

1.4.1 GLUE MRPC

• GLUE MRPC uses a dataset from Dolan and Brockett (2005) in evaluation.Source: §1.4.1 · p. 56

Data Example

• The data example pairs two sentences and shows prompts that ask whether they mean the same thing or are equivalent.Another example asks whether one sentence can replace the other without changing meaning.Source: Data Example · pp. 57–58

1.4.2 GLUE QQP

• The GLUE QQP dataset is used in evaluation.The passage attributes the dataset to Iyer et al. (2017).Source: §1.4.2 · p. 59

Prompts

• QQP prompts cast duplicate detection as deciding whether Quora posts can be merged or whether two questions ask the same thing.Other examples ask whether one question’s answer could answer the other, or simply whether the questions are duplicates.Source: Prompts · pp. 59–61

1.4.3 PAWS LABELED FINAL

• The PAWS Labeled Final dataset is used in training.The passage attributes the dataset to Zhang et al. (2019).Source: §1.4.3 · p. 61

Prompts

• PAWS prompts ask whether sentence pairs are paraphrases, express the same meaning, or can be rewritten into each other.Some templates phrase the task as a yes/no question, while examples also include a true-or-false format.Source: Prompts · pp. 61, 62, 64

1.5.1 AI2 ARC ARC-CHALLENGE

• AI2 ARC ARC-Challenge is used in evaluation.The passage attributes the dataset to Clark et al. (2018).Source: §1.5.1 · p. 65

Prompts

• ARC prompts frame four-choice questions in several ways, including asking for the correct option, presenting a student’s question, or requesting incorrect options.The examples include conversational wording and explicit options.Source: Prompts · pp. 65–67

1.5.2 AI2 ARC ARC-EASY

• AI2 ARC ARC-Easy is used in evaluation.The passage attributes the dataset to Clark et al. (2018).Source: §1.5.2 · p. 67

1.5.3 KILT TASKS HOTPOTQA

• The KILT HotpotQA entry is marked as used in training, but its source is unspecified in the supplied passage.Its example asks which of two magazines started first and gives Arthur's Magazine as the answer.Source: §1.5.3 · p. 70, Data Example · p. 70

Data Example

• The data examples show question-answer records alongside supporting fields such as context, metadata, or answer aliases.Source: Data Example · pp. 70, 71, 77

Prompts

• Prompt templates recast question answering as combining facts, answering complex questions, producing English answer strings, or checking proposed answers.Examples include prompts that request a complex answer, ask users to combine facts, or explicitly validate a suggested answer.Source: Prompts · pp. 70, 72, 76

1.5.4 TRIVIA QA UNFILTERED

• The TriviaQA Unfiltered dataset is attributed to Joshi et al. (2017) and marked for evaluation.The example asks who was president when the first Peanuts cartoon appeared; the supplied answer passage lists aliases for Harry S. Truman.Source: §1.5.4 · p. 71, Data Example · p. 71

1.5.5 WEB QUESTIONS

• The Web Questions dataset is attributed to Berant et al. (2013) and marked for evaluation; its example asks for Justin Bieber’s brother’s name and lists Jazmyn Bieber and Jaxon Bieber.Source: §1.5.5 · p. 73, Data Example · p. 73

1.5.6 WIKI QA

• The WikiQA dataset is attributed to Yi et al. (2015) and marked as used in training.The example question concerns how African Americans were immigrated to the United States, and the supplied answer passage begins with “African immigration to the United States.”Source: §1.5.6 · p. 74, Data Example · p. 74

1.6.1 ADVERSARIAL QA DBIDAF

• The BiDAF adversarial-QA example asks for the year research groups were compared and gives “2003” as the answer; its extraction prompt is conditioned on a non-test split.The prompt supplies both the question and context and selects an answer from the provided answer text when the split is not test.Source: Data Example · pp. 77–78

1.6.2 ADVERSARIAL QA DBERT

• The DBERT adversarial-QA dataset is attributed to Bartolo et al. (2020) and marked as used in training; its sample asks what concept is mentioned last.The supplied answer is “subjective idealism.”Source: §1.6.2 · p. 79, Data Example · p. 79

Prompts

• Context-based prompts range from student-facing question wording to direct answer extraction, with some templates conditioned on the data split.One prompt is explicitly labeled as not intended for the dataset authors’ original task.Source: Prompts · pp. 79–81

1.6.3 ADVERSARIAL QA DROBERTA

• The Adversarial QA dataset from Bartolo et al. (2020) is used in training.Source: §1.6.3 · p. 81

Data Example

• Example records include context or movie-plot text paired with a question and answer, such as questions about Napoleon’s accessories or how police reached a Mars mining camp.Source: Data Example · pp. 81–82

1.6.4 DUORC SELFRC

• The DUORC SelfRC dataset from Saha et al. (2018) is used in training, with movie-plot prompts that include unanswerable-response options.Source: §1.6.4 · p. 82, Data Example · p. 83

1.6.5 DUORC PARAPHRASERC

• The DUORC ParaphraseRC dataset from Saha et al. (2018) is used in training, and its example pairs a movie-plot question with an answer.Source: §1.6.5 · p. 85, Data Example · p. 85

1.6.6 ROPES

• ROPES, from Lin et al. (2019), is used in training and is prompted as question answering over a situation, a background, or both.Source: §1.6.6 · p. 88, Data Example · p. 88, Prompts · p. 90

Prompts

• Other templates frame answering as using background information or a hint to respond to a new situation and question.Source: Prompts · pp. 90–91

1.6.7 SQUAD V2

• The SQUAD V2 dataset from Rajpurkar et al. (2016) is identified as being used in evaluation.Source: §1.6.7 · p. 92

Data Example

• The data example pairs a question about when Beyoncé became popular with the answer “in the late 1990s.”Source: Data Example · p. 92

Prompts

• Prompt examples vary wording and task framing, including context-based answering, answerability with an “unanswerable” response, reverse question generation, and topic labeling.Source: Prompts · pp. 92–96

1.6.8 SUPER GLUE RECORD

• The SUPER GLUE RECORD dataset from Zhang et al. (2018) is identified as being used in evaluation.Source: §1.6.8 · p. 96

1.6.9 QUOREF

• QUOREF is identified as a dataset from Dasigi et al. (2019) used in training.Source: §1.6.9 · p. 104

Data Example

• The examples include a record whose answer is Catherine and an apple-tree arithmetic question with choices and explanatory fields.The record also includes context and a question; the arithmetic example includes abstractive and extractive explanation fields.Source: Data Example · pp. 104, 107

Prompts

• Prompt templates frame coreference, article question answering, highlight generation, and commonsense choice or explanation tasks in natural language.Some templates ask for a referent or article-based answer, while others request an additional highlight or a common-sense option and its rationale.Source: Prompts · pp. 101, 105, 109

1.7.1 COS E V1.11

• COS E V1.11 is listed as a dataset used in training, but its source is unspecified in the supplied text.Source: §1.7.1 · p. 107

1.7.2 COSMOS QA

• COSMOS QA, from Huang et al. (2019), is used in training.The example includes a context, question, and answer choices; templates include choosing an option and generating a question from context and answer.Source: §1.7.2 · pp. 110–111, Prompts · p. 111

Prompts

• Prompt templates vary their wording and format, including an alternate task that asks for a question from a context and answer or a conversation from a question and answer.Other examples frame a context-and-question multiple-choice task or ask whether a proposed answer is correct.Source: Prompts · pp. 111, 116, 120

1.7.3 DREAM

• DREAM, from Sun et al. (2019), is used in training.Its prompts include answering a question about a dialogue as well as generating a conversation from a question and answer.Source: §1.7.3 · p. 115, Prompts · p. 116

1.7.4 OPENBOOKQA MAIN

• OpenBookQA MAIN, from Mihaylov et al. (2018), is used in evaluation.Its example asks a science question and provides answer choices, with templates mapping the answer key to the selected choice.Source: §1.7.4 · p. 117, Data Example · pp. 117–118

1.7.5 QASC

• QASC, from Khot et al. (2020), is used in training.The example includes eight labeled choices and two facts alongside the combined fact and question.Source: §1.7.5 · p. 120, Data Example · p. 120

Data Example

• A QASC example combines two facts into a formatted question, and a prompt tests whether a choice answers that question given the combined fact.The example's combined fact is that beads of water can be formed by clouds.Source: Data Example · p. 120, Prompts · p. 120

Prompts

• QUAIL prompts frame tasks with facts, questions, and answer choices, while varying the wording and answer format.Some templates present two facts before the question; others use a context-and-question format or ask for the answer choice text.Source: Prompts · pp. 121, 124

1.7.6 QUAIL

• The section identifies QUAIL as a dataset from Rogers et al. (2020) used in training.Source: §1.7.6 · p. 123

Data Example

• The QUAIL data example includes answer, context, metadata, and question fields.Source: Data Example · p. 123

1.7.7 QUAREL

• QUAREL is identified as a dataset from Tafjord et al. (2018) used in training.Source: §1.7.7 · p. 128

Prompts

• QUAREL prompt templates pose a two-choice reasoning question and return the selected answer choice.The templates variously frame the input as a logic test or short story, and one instructs the model not to answer with A or B.Source: Prompts · pp. 128–129

1.7.8 QUARTZ

• Tafjord et al. (”2019”) is identified as the source of QUARTZ, which is used in training.Source: §1.7.8 · p. 130

Data Example

• Examples include a water-scarcity question with two choices and a reading question asking what can be learned about lying.Source: Data Example · pp. 130, 134

1.7.9 RACE HIGH

• Lai et al. (2017) is identified as the source of RACE HIGH, which is used in evaluation.Source: §1.7.9 · p. 134

Prompts

• Prompt templates frame choice tasks as selecting an answer from an article or passage, with outputs represented by answer choices or A/B/C/D labels.Some templates also ask the model to generate a multiple-choice question from an article.Source: Prompts · pp. 130, 135, 136

1.7.10 RACE MIDDLE

• Lai et al. (2017) is identified as the source of RACE MIDDLE, which is used in evaluation.Source: §1.7.10 · p. 137

1.7.11 SCIQ

• The SCIQ example asks which organism is commonly used in preparation, gives “mesophilic organisms” as the correct answer, and includes supporting text.SCIQ is attributed to Johannes Welbl (2017) and is used in training.Source: Data Example · p. 141, §1.7.11 · p. 141

Prompts

• Social IQA prompts present paragraph-based questions with answer choices, sometimes ordering the choices or asking the model to select a correct option.Other templates phrase the task as answering a question given context or offer a question with three answer options.Source: Prompts · pp. 141, 142, 144, 146

1.7.12 SOCIAL I QA Data Example

• The Social IQA example gives a barbecue context and asks how others would feel, with answer options including staying home and being a good friend.The supplied example also shows a third answer choice and the question text.Source: §1.7.12 · p. 144

1.7.13 SUPER GLUE BOOLQ

• A dataset from Clark et al. (2019) is used in evaluation; prompts ask readers to judge a passage-based question as true or false, and another template asks for a yes/no answer.Source: §1.7.13 · p. 146, Prompts · pp. 146–147

Data Example

• The BoolQ data example pairs a question asking whether Iran and Afghanistan speak the same language with a passage about Persian.The passage begins with the phrase “Persian language.”Source: Data Example · p. 146

1.7.14 SUPER GLUE MULTIRC

• MultiRC is identified as an evaluation dataset from Khashabi et al. (2018), and its prompts ask whether a proposed answer correctly answers a question based on a paragraph.The example supplies a paragraph, a question about persuading Pakistan, and the answer “Children, Gerd, or Dorian Popa.”Source: §1.7.14 · p. 150, Prompts · pp. 150–151, Data Example · p. 150

1.7.15 WIKI HOP ORIGINAL

• The dataset is from Welbl et al. (2018) and is used in training; its prompt provides support information and asks which object entity has a specified relation with a subject.Source: §1.7.15 · p. 153, Prompts · p. 154

Prompts

• Relation-extraction prompts provide supporting paragraphs and ask for an entity linked by a named relation, varying between choice selection and subject–object output.Some versions ask for the related entity from choices, while another asks for subject and object entities.Source: Prompts · p. 156

1.7.16 WIQA

• WIQA, from Tandon et al. (2019), is used in training; its prompts ask about process steps and how a supposed perturbation affects a process.Templates ask for missing or final steps, classify perturbations, and request judgments about effects.Source: §1.7.16 · p. 157, Data Example · pp. 158–160

Data Example

• The WIQA data example represents a process as ordered steps and frames questions about missing or final steps, perturbation type, and the perturbation’s effect.Effect prompts ask whether a perturbation has a direct, indirect, or no impact, or whether it increases, decreases, or has no effect on an outcome.Source: Data Example · pp. 157–159

1.7.17 PIQA

• PIQA is used in evaluation (Bisk et al., 2020), with prompts that select a solution, assess whether a goal–solution phrase makes sense, or rewrite an incorrect solution.The example asks how to finish “When boiling butter, when it's ready, you can” and supplies two candidate solutions.Source: §1.7.17 · p. 160, Data Example · p. 160, Prompts · pp. 161, 163

1.8.1 AMAZON POLARITY

• Amazon Polarity, from McAuley and Leskovec (2013), is used in training, with review prompts asking about sentiment or whether the user would recommend the product.The example includes review title and content, and templates phrase the task as positive/negative classification, recommendation, or sentiment identification.Source: §1.8.1 · p. 164, Data Example · p. 164, Prompts · pp. 164–165

Prompts

• Review prompts reframe sentiment, recommendation, and rating tasks as natural-language questions or continuations, with answer choices ranging from sentiment labels to five-point scales.Examples include asking whether a review affects purchase chances, whether an app would be recommended, and what star rating a review merits.Source: Prompts · pp. 166–168

1.8.2 APP REVIEWS

• The APP REVIEWS dataset is marked as used in training, and its prompts include app-recommendation questions and review-generation or rating tasks.One template asks whether the reader would recommend an app to a friend; others request a star-rated review or ask for a review’s rating.Source: §1.8.2 · p. 167, Prompts · pp. 167–168

Data Example

• The example app-review record gives a package name and a review beginning “Great app!” about a new version working on a Bravia.Source: Data Example · p. 167

1.8.3 IMDB

• IMDB is identified as training data, with prompts asking for movie-review sentiment through questions about tone, positivity, and the reviewer’s feelings.Source: §1.8.3 · p. 168, Prompts · pp. 169–170

1.8.4 ROTTEN TOMATOES

• ROTTEN TOMATOES is identified as training data, with prompts that ask whether a movie review is positive or negative and how the reviewer feels.Source: §1.8.4 · p. 172, Prompts · p. 173

1.8.5 YELP REVIEW FULL

• YELP REVIEW FULL is identified as training data, and its prompts elicit a one-to-five-star rating for a review or place.Templates include rating continuations such as “My rating for this place is” and explicit review-score prompts.Source: §1.8.5 · p. 175, Prompts · p. 176

1.9.1 SUPER GLUE COPA

• The SUPER GLUE COPA dataset is attributed to Roemmele et al. (2011) and marked as used in evaluation.Source: §1.9.1 · p. 177

Data Example

• Given a premise about a shadow over grass, the example asks readers to choose between sunrise and grass-cutting as the more plausible alternative.The exercise is framed as choosing the most plausible alternative.Source: Data Example · pp. 177–178

1.9.2 HELLASWAG

• HELLASWAG is identified as a dataset from Zellers et al. (2019) used in evaluation.Source: §1.9.2 · p. 182

Data Example

• A HELLASWAG example labels the activity “Removing ice from car” and pairs a snow-covered-window context with candidate endings.The example also includes a source ID.Source: Data Example · p. 182

Prompts

• Prompts include sentence-ending completion, a topic-identification question, and yes-or-no judgments about whether a candidate is an appropriate continuation.Some formulations select a candidate continuation at random before asking for a yes-or-no judgment.Source: Prompts · pp. 182–185
• CommonGen prompts ask for a sentence using the listed concepts, a sentence with a chosen concept, or the topics in a target sentence.The example concepts are ski, mountain, and skier; the target is “Skier skis down the mountain.”Source: Prompts · pp. 186–187, Data Example · p. 186
• Concept prompts ask models to extract key concepts from a sentence and then generate a sentence using selected or listed concepts.Source: Prompts · p. 188

1.10.2 WIKI BIO

• WikiBio prompts convert table facts into short biographies or ask for a person’s details, field values, or identity.Templates use table headers and contents as bullet points, sometimes excluding the article-title field.Source: Prompts · pp. 189–190

1.11.1 CNN DAILYMAIL 3.0.0

• CNN/DailyMail prompts request article outlines, two- or three-sentence takeaways, TLDRs, mobile-news cards, or stock-market-relevant points.Other templates frame the article as material for a story or storyline.Source: Data Example · pp. 191–192

1.11.2 GIGAWORD

• Gigaword prompts ask for an article title, including from the article’s first sentence, and also include a reverse framing that supplies a title and requests an article.Source: Prompts · pp. 193–195

1.11.3 MULTI NEWS

• MultiNews prompts frame multiple articles as requests for key points, a concise synthesis, or a summary, and also include expanding a summary into a plausible article.The templates split the input into documents and ask for cross-document synthesis or summary generation.Source: Prompts · pp. 195–196

1.11.4 SAMSUM

• SAMSum prompts request dialogue summaries in several phrasings and also ask for a dialogue matching a supplied summary.Source: Prompts · pp. 197–198

1.11.5 XSUM

• XSum prompts request a text summary or rephrasing, with variants asking for a layperson’s recap, a simple-idea completion, or an extremely short abstract.Source: Prompts · pp. 199–201

1.12.1 AG NEWS

• AG News prompts ask for one of four news labels—world politics, sports, business, or science and technology—or target an intended reader such as a politician, athlete, executive, or scientist.Source: Data Example · p. 202

Data Example

• Newspaper-article examples ask which section or broad topic fits an article, with choices including World News, Sports, Business, and Science and Technology.Source: Data Example · p. 203

1.12.2 DBPEDIA 14

• DBpedia 14 is identified as a training dataset from Lehmann et al. (2015), with an example entry showing its content and title fields.Source: §1.12.2 · p. 204, Data Example · p. 204

Prompts

• DBpedia prompts classify text or titles among 14 categories, using varied wording and presenting the category options as answer choices.Source: Prompts · pp. 204–205

1.12.3 TREC

• TREC, from Li and Roth (2002), is marked for training; its prompts classify questions into coarse categories and map them to fine labels, including location and number types.Source: §1.12.3 · p. 205, Prompts · pp. 206–207

1.13.1 SUPER GLUE WIC

• The dataset from Pilehvar and os’e Camacho-Collados (2018) is used in evaluation.Source: §1.13.1 · p. 212

Prompts

• Prompt variants ask whether a target word has the same meaning across two sentences, with yes/no or true/false response formats.Source: Prompts · pp. 213–214
• A prompt attributed to Brown et al. (2020) asks whether a word is used in the same sense in two sentences, sometimes offering Yes/No choices.Source: Prompts · pp. 214–215