PaperlayerView exact revision on arXiv

Source-linked AI summary

Multitask Prompted Training Enables Zero-Shot Task Generalization

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, Alexander M. Rush

arXiv:2110.08207v3cs.LGcs.CL

TL;DR

The paper asks whether explicit multitask learning can induce zero-shot generalization without relying on massive pretraining scale. It trains a prompted multitask model and finds strong held-out-task performance, matching or exceeding GPT-3 on 9 of 11 datasets.

  • Problem

    It remains unclear whether zero-shot generalization arises from implicit multitask learning in pretraining or can be induced through explicit supervised multitask training.

  • Method

    The authors convert diverse supervised datasets into natural-language prompts with multiple prompt variants, then train a pretrained encoder-decoder model on the resulting multitask mixture.

  • Results

    9 out of 11 held-out datasets matched or exceeded all GPT-3 models, while multitask prompted training also improved performance on most BIG-bench tasks.

  • Takeaways & Limitations

    Multitask prompted training provides an effective alternative to unsupervised language-model pretraining for strong zero-shot generalization, including in smaller models.

  • Takeaways & Limitations

    The contribution of prompt diversity relative to other model differences remains unresolved and is left for future investigation.

Abstract

from arXiv · show

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.

1 INTRODUCTION

The paper tests whether explicit supervised multitask training with natural-language prompts can induce zero-shot generalization to held-out tasks and robustness to prompt wording. The resulting model matches or exceeds GPT-3 on 9 of 11 held-out datasets and improves over a large baseline on 13 of 14 BIG-bench tasks, while more prompts improve performance consistency.

  • Motivation and approach: The study targets explicit supervised multitask training across many natural-language-prompted tasks to improve generalization without requiring massive model scale.The approach converts a large set of natural-language tasks into prompted form using templating.
  • Experimental questions: The experiments ask whether multitask prompted training improves held-out-task generalization and whether broader prompt coverage improves robustness to wording.These are the paper’s two central experimental questions.
  • Zero-shot generalization: 9 out of 11 held-out datasets: the model matches or exceeds GPT-3 despite being about 16× smaller.This result is presented as evidence that multitask training enables zero-shot task generalization.
  • Zero-shot generalization: 13 out of 14 BIG-bench tasks: the model improves over a large baseline language model.The comparison uses tasks from the BIG-bench benchmark.
  • Prompt robustness: Training on more prompts per dataset consistently improves median performance and decreases performance variability on held-out tasks.The finding addresses robustness to prompt wording.

2 RELATED WORK

The related work distinguishes implicit from explicit multitask learning and presents natural-language prompting as a shared format for supervised multitask training. Prior studies broaden prompting to generic NLP tasks, while this work scales prompted training across 62 datasets and 12 traditionally defined tasks and remains agnostic about why prompts work.

  • Multitask learning: Explicit multitask learning mixes multiple tasks into one supervised training process, requiring a shared output format because tasks produce different outputs.Multitask learning has long improved NLP performance, and prior work has used several shared formats.
  • Prompting: Natural-language prompting reformats NLP tasks as natural-language responses to natural-language inputs, making it useful for text-to-text multitask learning.T5-style text-to-text pretrained models help enable this use of prompts; prior work reformatted 20 question-answering datasets into one prompt.
  • Prompted multitask training: This work differs from prior prompted multitask training by measuring generalization across 62 datasets and 12 traditionally defined NLP tasks, rather than 61 narrowly defined tasks from 9 datasets.Mishra et al. adapted narrowly defined tasks from crowdsourcing instructions, whereas this work uses tasks traditionally defined in the NLP literature.
  • Prompt interpretation: Although prompts are commonly hypothesized to function as task instructions that support held-out-task generalization, the role of semantic meaningfulness remains contested, so this work stays agnostic.The semantic meaningfulness of prompts has been challenged by prior studies.

3 MEASURING GENERALIZATION TO HELD-OUT TASKS

The paper measures zero-shot generalization by training on some NLP tasks and evaluating on entirely held-out tasks, while acknowledging that task categories are fuzzy. Its evaluation covers 12 tasks and 62 datasets, holds out four tasks, and includes a language-oriented BIG-bench subset.

  • Evaluation setup: Zero-shot generalization is evaluated by training on a subset of tasks and testing on a held-out group of tasks.A task denotes a general NLP ability assessed by multiple specific datasets.
  • Task taxonomy: Task categorization is inherently fuzzy because datasets grouped under one ability, such as commonsense, can require substantially different knowledge.Commonsense datasets range from innate knowledge and grade-school science to DIY instructions, cultural norms, and graduate-level theorems.
  • Task taxonomy: The taxonomy prioritizes task format over inferred skill, reflecting conventions in prior literature because grouping by task is an imperfect heuristic.This organizational choice follows conventions from Khashabi et al. (2020b), Vu et al. (2020), and Ye et al. (2021).
  • Dataset construction: 12 tasks and 62 datasets comprise the training and evaluation mixtures after excluding non-English datasets and those requiring special domain knowledge.All experiments use datasets from the Hugging Face datasets library.
  • Held-out evaluation: Four tasks are fully held out for zero-shot evaluation: natural language inference, coreference resolution, sentence completion, and word sense disambiguation.All constituent datasets of these tasks are excluded from training; NLI is selected partly because humans commonly perform it without explicit training.
  • BIG-bench evaluation: The study also evaluates a language-oriented BIG-bench subset containing English-language text that is in-vocabulary for the T5 tokenizer.The selected tasks have preliminary results prepared by BIG-bench maintainers and exclude emojis or other special characters.

4 A UNIFIED PROMPT FORMAT

The paper introduces a unified natural-language prompt format and tooling that convert diverse datasets into prompted input-target examples for zero-shot experimentation. The resulting Public Pool of Prompts provides broad prompt diversity across English datasets while excluding harmful or non-natural-language content.

  • Prompt format: The unified format represents each prompt as input and target templates plus metadata that map raw examples into natural-language sequences.The templates can render dataset fields and prompt-specific metadata such as label choices.
  • Prompt collection: The prompting interface supported interactive authoring and community contributions from 36 contributors affiliated with 24 institutions in 8 countries.Contributors were encouraged to vary prompt formats because prompt effectiveness remains unresolved and the model was intended to be robust to them.
  • Evaluation scope: Most prompts preserve the original task, while permuted-task prompts increase training diversity but are excluded from evaluation because they change the original metrics and baselines.The paper gives document generation from a summary as an example of a permitted non-original-task prompt.
  • Prompt collection: 2073 prompts for 177 datasets, averaging 11.7 prompts per dataset, comprise the Public Pool of Prompts.The collection covers English datasets while excluding potentially harmful content and non-natural language such as programming languages.

5 EXPERIMENTAL SETUP

The experiments fine-tune T5-based encoder-decoder models on prompted multitask mixtures and evaluate their true zero-shot generalization on held-out traditional NLP and BIG-bench tasks. Model variants differ in training mixtures and scale, while checkpoint selection avoids using held-out-task examples.

  • Model: The models fine-tune pretrained T5 encoder-decoder architectures to autoregressively generate target text from prompted input, unlike decoder-only models trained to generate the input.T5 was pretrained with a masked language modeling-style objective on 1T tokens from C4, so the experiments use the LM-adapted T5+LM variant.
  • Training: T0 uses the Section 3 multitask mixture, T0+ adds GPT-3’s evaluation datasets, and T0++ additionally adds SuperGLUE except RTE and CB.With these additions, NLI and BIG-bench tasks remain the only held-out tasks for T0++.
  • Training: The primary T0 variants are initialized from 11B-parameter T5+LM, while T0 (3B) uses the 3B-parameter version with the original T0 training mixture.The smaller model is trained to study scaling and support researchers with fewer resources.
  • Evaluation: Checkpoint selection uses the highest validation score on training datasets, preserving true zero-shot evaluation because held-out-task examples are excluded.For each dataset, results use median performance across prompts and report the interquartile range to assess robustness to prompt wording.
  • Evaluation: Evaluation covers 11 datasets across four held-out traditional NLP tasks and 14 novel BIG-bench tasks, reporting validation-split accuracy unless otherwise specified.The four traditional tasks are natural language inference, coreference, word sense disambiguation, and sentence completion.

6 RESULTS

Multitask prompted training improves zero-shot generalization to held-out tasks, with T0 matching or exceeding GPT-3 on most evaluated datasets. Increasing prompt and dataset diversity generally improves performance and may make T0 more robust to prompt wording.

  • Held-out task generalization: T0 achieves significant gains over the identical T5+LM baseline on all four held-out datasets.The comparison isolates multitask prompted training from language-modeling training because the models use identical prompts and architecture.
  • Held-out task generalization: T0 matches or exceeds all GPT-3 models on 9 out of 11 held-out datasets.T0 reports median and interquartile-range performance across all prompts, while GPT-3 reports a single prompt per dataset.
  • BIG-bench generalization: T0, T0+, and T0++ are evaluated zero-shot on BIG-bench tasks covering novel skills absent from their training tasks.Examples include ordering objects, solving logic-grid puzzles, and distinguishing true statements from common misconceptions.
  • Prompt diversity: Adding more training prompts consistently raises median performance and generally lowers interquartile range on held-out tasks.The main T0 model uses p = 8.03 prompts per dataset, including prompts that do not map onto the dataset’s original task.
  • Prompt robustness: On RTE, GPT-3’s other 9 prompts produce median accuracy = 52.96% and interquartile range = 1.28%, suggesting lower robustness to wording than T0.The Brown et al. reported prompt scores an accuracy of 58.8%, below the reported 63.5%.

7 DISCUSSION

The discussion situates T0 alongside FLAN and examines why prompted multitask training helps some held-out tasks but not others. It attributes differing findings partly to encoder-decoder pretraining and greater prompt diversity, while noting that adding datasets raises median performance without consistently narrowing variability.

  • Comparison with FLAN: T0 and FLAN use broadly similar prompted multitask methods, but T0 trains one model with multiple held-out tasks whereas FLAN trains separate decoder-only models with one held-out task each.The distinction reflects different evaluation setups for generalization across diverse tasks.
  • Comparison with FLAN: Both T0 and FLAN underperform GPT-3 on Winogrande and HellaSwag, but removing instructions improves HellaSwag performance.The reevaluation follows the conjecture that instructions are largely redundant for tasks formatted as incomplete-sentence completion.
  • Scaling the multitask mixture: Adding more datasets consistently increases median zero-shot performance, although it does not always reduce the interquartile range for held-out tasks.This comparison covers T0, T0+, and T0++ with varying numbers of datasets.

8 CONCLUSION · A CONTRIBUTIONS AND PROJECT STRUCTURE · B BROADER IMPACTS

The paper shows that multitask prompted training enables strong zero-shot generalization, often outperforming much larger models, while emphasizing prompt and dataset diversity. It also documents the BigScience project’s open, international collaboration and contributor-based authorship structure.

  • 8 CONCLUSION: Multitask prompted training enables strong zero-shot generalization in language models and offers an effective alternative to unsupervised language-model pretraining.The approach often enables T0 to outperform models many times its size.
  • 8 CONCLUSION: Many diverse prompts are important for performance, and increasing the number of datasets per task affects zero-shot generalization.These findings come from ablation studies.
  • A CONTRIBUTIONS AND PROJECT STRUCTURE: The research was conducted within BigScience, a year-long open-research initiative studying large models and datasets outside large technology companies.The project’s stated goal was to study language models in a public environment.
  • A CONTRIBUTIONS AND PROJECT STRUCTURE: BigScience included 600 researchers from 50 countries and more than 250 institutions.The project was initiated by Thomas Wolf at Hugging Face.
  • A CONTRIBUTIONS AND PROJECT STRUCTURE: The joint first-authors led the project, with Victor Sanh overseeing prompt engineering and evaluation, Albert Webson selecting datasets and leading analysis, and Colin Raffel directing research and model training.Their listed responsibilities also included prompt collection, prompt materialization, ablation design, writing, and evaluation infrastructure.
  • A CONTRIBUTIONS AND PROJECT STRUCTURE: The work is co-authored by working-group contributors who supplied at least 3 accepted prompted datasets, with authors sorted by code contributions.The paper explicitly highlights additional contributors who supported evaluation and writing.

B.1 ENVIRONMENTAL COSTS … C ANNOTATION SYSTEM - PROMPTSOURCE

The paper reports environmental costs and limitations of multitask prompt training, while emphasizing openness and documenting bias risks. It also introduces PromptSource, a lightweight annotation system using structured templates and flexible prompt-generation features.

  • B.1 ENVIRONMENTAL COSTS: All experiments in this paper v3-512 270 europe-west4-a 410 8.9, illustrating that training hardware energy use can impose substantial environmental costs.The paper attributes these costs to energy required to power training hardware.
  • B.1 ENVIRONMENTAL COSTS: T5-11B (single run) v3-1024 528 Taiwan 540 46.7, whereas T0 (single run) v3-512 27 europe-west4-a 410 0.9.Table 1 presents these carbon-emissions records for the reported training runs.
  • B.2 RISKS IN DEVELOPING AND RELEASING LARGE LANGUAGE MODELS: The work empirically explores multitask prompt training by converting datasets into prompted examples, fine-tuning pretrained models, and observing strong zero-shot capabilities across multiple tasks.Its zero-shot performance remains significantly behind models fine-tuned on the given task through traditional transfer learning.
  • B.2 RISKS IN DEVELOPING AND RELEASING LARGE LANGUAGE MODELS: The authors use publicly available datasets and a publicly available T5+LM model, favor openness, and release prompts, trained models, and materialized prompted data for reproducibility.They identify financial constraints as the main barrier to replication by dozens of organizations worldwide.
  • B.3 BIAS AND FAIRNESS: The training mixture excluded potentially harmful datasets, including Sentiment140 and Hate Speech 18, but the resulting models are not bias-free.The exclusions were intended to avoid training the model to generate harmful outputs.
  • B.3 BIAS AND FAIRNESS: T0++ produced problematic predictions including affirming flat-Earth and vaccine-autism claims and assigning stereotyped occupations in sentence completion.These examples are presented as evidence that the model can generate conspiracist or biased answers.
  • C ANNOTATION SYSTEM - PROMPTSOURCE: PromptSource provides a downloadable Streamlit interface that lets users inspect datasets, write templates, verify them, and upload results to a central repository.Pilot testing led to helicopter and sourcing views for prioritizing datasets and browsing examples while authoring prompts.
  • C ANNOTATION SYSTEM - PROMPTSOURCE: PromptSource uses Jinja templates as a middle ground between pure code and rigid structure, with placeholders, conditionals, seeded choice functions, and support for generating varied prompts.Templates separate conditioning text from desired completions with the ||| separator, while inapplicable examples can be skipped.

D DATASETS · D.1 CATEGORIZING DATASETS INTO TASKS

The paper organizes datasets into task categories based primarily on task content, while treating question answering as a difficult case because its varied forms lack a commonly accepted subdivision. The authors ultimately categorize ambiguous QA datasets by format when content-based distinctions, especially commonsense, prove unclear.

  • D.1 CATEGORIZING DATASETS INTO TASKS: The taxonomy covers sentiment analysis, topic classification, paraphrase identification, natural language inference, word sense disambiguation, coreference resolution, summarization, and structure-to-text generation.
  • D.1 CATEGORIZING DATASETS INTO TASKS: Question answering is difficult to subdivide because many datasets share the broad label “question answering” without a commonly accepted classification scheme.
  • D.1 CATEGORIZING DATASETS INTO TASKS: The authors argue that categorizing by content is preferable to categorizing by format, since history and physics exams represent different tasks regardless of their response formats.
  • D.1 CATEGORIZING DATASETS INTO TASKS: Closed-book QA is treated as a distinct task because it largely evaluates memorization of world knowledge.
  • D.1 CATEGORIZING DATASETS INTO TASKS: The paper questions whether scientific reasoning should count as commonsense, noting that ARC evaluates scientific knowledge beyond commonsense.
  • D.1 CATEGORIZING DATASETS INTO TASKS: Because NLI and coreference datasets often require commonsense knowledge, the authors avoid commonsense as a standalone category and classify QA by format.
  • D.1 CATEGORIZING DATASETS INTO TASKS: ARC is categorized as multiple-choice QA because other closed-book QA datasets require generating answers without provided answer options.

D.2 HOW UNSEEN ARE THE HELD-OUT TASKS? · D.3 LAMBADA

The paper examines whether held-out tasks are genuinely distinct from training tasks, noting overlap risks in broad task categories while arguing that some distinctions remain meaningful. For LAMBADA, T0 performs poorly in its original open-ended format but improves with GPT-3-style cloze prompting.

  • D.2 HOW UNSEEN ARE THE HELD-OUT TASKS?: Broad QA categories may conceal overlap with entailment or coreference, making strict held-out-task claims impractical to verify exhaustively.ReCoRD, held out as part of SuperGLUE, exclusively asks questions amounting to referent identification.
  • D.2 HOW UNSEEN ARE THE HELD-OUT TASKS?: Paraphrase identification is treated as distinct from NLI because it tests bidirectional rather than unidirectional entailment, with manual review finding almost no valid paraphrase overlap.The passage contrasts paraphrase identification with NLI and reports manual review of ANLI and RTE.
  • D.2 HOW UNSEEN ARE THE HELD-OUT TASKS?: Training on QQP before RTE has been reported to hurt performance relative to training on the entailment task alone, supporting a distinction between paraphrase and NLI.This finding is cited as additional evidence relevant to the task-boundary discussion.
  • D.2 HOW UNSEEN ARE THE HELD-OUT TASKS?: Sentence completion is another potentially overlapping category because pragmatic continuation, coreference, and NLI-style inference may not be cleanly separated.SWAG was proposed as commonsense inference to supplement NLI, but the passage notes that deductive and natural pragmatic inference are not clearly distinguished in many NLI datasets.
  • D.2 HOW UNSEEN ARE THE HELD-OUT TASKS?: BIG-bench evaluation datasets were designed to test diverse, difficult, and novel skills, so they are unlikely to overlap heavily with T0’s training tasks.This supports treating BIG-bench tasks as relatively novel held-out evaluations.
  • D.3 LAMBADA: LAMBADA differs from other sentence-completion tasks by requiring open-ended next-word prediction rather than selection among a few continuations.Brown et al. (2020) classify LAMBADA under sentence completion, but the paper identifies its open-ended form as a significant exception.
  • D.3 LAMBADA: 6.2% accuracy was achieved by T5+LM on unprompted LAMBADA, versus 72.5% for the comparably-sized GPT-3-13B variant and 18.7% for T0.These results show weak performance in the standard unprompted next-word-prediction format.
  • D.3 LAMBADA: 27.8% accuracy was achieved by T0 after switching to the GPT-3-style cloze prompt, improving over its 18.7% result in the original format.The passage attributes the improvement to evaluating LAMBADA with the same cloze-style prompted form used by GPT-3.

D.4 TABLE OF ALL DATASETS

Table 5 catalogs all training and evaluation datasets, using Hugging Face dataset identifiers and documenting a closed-book reformulation of Hotpot QA. Full dataset citations appear in Appendix G.

  • Table 5 lists all training and evaluation datasets.
  • Dataset names use Hugging Face identifiers, with the portion after / indicating the subset name.
  • Hotpot QA is recast as closed-book question answering because of its long input length.
  • Full dataset citations are provided in Appendix G.

E CONTAMINATION ANALYSIS OF PRETRAINING CORPUS ON TEST TASKS

The analysis tests whether zero-shot results are inflated by memorization of test-task text in C4, the model’s pretraining corpus. It uses substring matching and finds negligible ANLI-hypothesis overlap but 5.2% overlap for RTE hypotheses, mostly factual sentences.

  • Purpose: Zero-shot estimates may be confounded when C4 contains test-task text, because memorization can improve performance without generalization.The analysis searches for long common substrings between prompted zero-shot inputs and C4 documents.
  • Method: 16-token sentence groups enable exact-match queries against a suffix-array index of C4, over-counting length-32 token overlaps and flagging matched examples.The procedure uses the suffix-array method from Lee et al. (2021) to count substring occurrences efficiently.
  • NLI analysis: Premises and hypotheses are separated for NLI datasets because internet-sourced premises match frequently, while novel hypotheses may not aid memorization.This distinction is applied to both ANLI and RTE.
  • Match rates: 912/10000 HellaSwag examples matched C4, compared with 20/1400 for WiC, 15/5153 for Lambada, 4/146 for WSC, 3/1871 for Story Cloze, 1/250 for CB, and 0/1767 for Winogrande.These are the reported match counts across the listed zero-shot test tasks.
  • NLI analysis: 5.2% of RTE hypotheses were contained in pretraining, whereas ANLI hypotheses had negligible overlap; the RTE matches were largely short factual sentences.ANLI and RTE premises showed high match proportions, at 337/1000 and 329/3000, respectively, while hypotheses were 6/1000 and 156/3000.

F FULL RESULTS … Data Example

The appendix reports full results for T0 model variants and documents the prompt resources used across evaluation datasets. It also provides dataset examples and varied prompt templates, including prompts that depart from original task descriptions.

  • F FULL RESULTS: Full-results tables compare T5+LM with all T0 model variants across tasks, including zero-shot and non-zero-shot results.The appendix also includes a comparison of T0 3B against T0 11B and results on a subset of BIG-bench tasks.
  • G LIST OF ALL PROMPTS: The prompt appendix enumerates prompts used to train and evaluate T0 models and directs readers to Promptsource for the latest versions.Datasets are organized by task category and canonical Hugging Face dataset name.
  • Data Example: For each dataset, the appendix gives a contextual data example followed by prompt templates, bibliographic references, input and target templates, and sometimes answer-choice templates.It also marks prompts that do not correspond to the dataset authors’ original task description.
  • 1 PROMPTS: The Crows Pairs example presents paired sentences labeled “sent more” and “sent less,” with annotations and anonymous annotator identifiers.Its prompts ask whether a sentence illustrates a stereotype or anti-stereotype, classify bias types, or minimally modify stereotypical wording.
  • Prompts: The SuperGLUE AXG example pairs a premise with a hypothesis and uses multiple yes-or-no entailment prompt formulations from several cited works.Templates include questions about assuming, inferring, or establishing whether the hypothesis follows from the premise.
  • Data Example: Coreference examples for WSC.FIXED and Winogrande present text or sentences with ambiguous pronouns or blanks and offer alternative referents as answers.Their templates recast the task as yes-or-no, true-or-false, or explicit option-selection questions.
  • Prompts: Natural-language inference examples for SuperGLUE CB, RTE, and ANLI use premise–hypothesis pairs with prompts offering three-way labels such as yes, no, or maybe.The appendix includes alternative label sets such as true, false, or inconclusive, alongside dataset-specific examples and prompt variants.

Data Example … Prompts

The paper illustrates how datasets are represented as structured examples and converted into diverse natural-language prompts. Across paraphrase, question answering, multiple-choice, and reasoning tasks, prompts vary wording while preserving the underlying input-output objective.

  • Data Example: Paraphrase examples pair two sentences or questions and ask whether they express the same meaning, are equivalent, or can be rewritten into one another.The examples include sentence pairs and Quora questions, with answer-choice variables supplying the target label.
  • Prompts: Across tasks, prompt templates use conditional logic, variable substitution, answer-choice mappings, and randomized wording to preserve task structure while changing surface form.The passages show Jinja-style conditionals, fields such as question, context, and answer, and sequences from which alternative phrasings are selected.
  • Prompts: Prompt variants reframe paraphrase classification through equivalence, replacement, duplication, and answerability formulations.Examples include asking whether two Quora posts can be merged, whether they are duplicates, or whether one answer serves both questions.
  • Data Example: Multiple-choice examples expose a question alongside four labeled options and identify the correct answer through a label-indexed answer choice.The ARC examples include options labeled A, B, C, and D, with templates selecting the answer corresponding to answerKey.
  • Prompts: Multiple-choice prompts vary between direct selection, student-question scenarios, hesitation between four choices, and requests to copy incorrect options.These templates preserve the same four-option decision while changing the conversational framing and output format.
  • Data Example: Question-answering examples combine questions with contexts, passages, plots, backgrounds, or situations, sometimes including explicit answers and metadata.The supplied examples span HotpotQA, TriviaQA, adversarial QA, movie-plot QA, ROPES, and SQuAD-style inputs.
  • Prompts: Question-answering prompts ask models to answer directly, extract answers from text, generate questions from answers or passages, and handle unanswerable cases.Templates use outputs such as selected answer aliases, extracted spans, generated questions, and explicit responses including “Not answerable” or “Answer not in context.”
  • Data Example: Some examples separate background knowledge from a new situation and ask questions requiring the supplied information to be applied to that situation.The ROPES examples include a background, a situation, a question, and an answer, while prompts explicitly connect the background to the new situation.

Data Example … Prompts

The paper illustrates how diverse datasets are converted into prompted tasks by pairing structured examples with natural-language templates. Prompts vary across continuation, generation, summarization, classification, and question-answering formulations while preserving answer-choice or target-label mechanisms.

  • Data Example: Data Example: A cause-or-effect task presents a premise and two alternatives, such as a body casting a shadow over grass, then asks for the most plausible option.The alternatives are “The sun was rising.” and “The grass was cut.”
  • Prompts: Prompts: The same cause-or-effect instance is rephrased through varied questions about likely causes, effects, continuations, and best options.Examples include selecting a more likely cause or effect, choosing a plausible continuation, and asking what could happen next.
  • Data Example: Data Example: A HELLASWAG-style instance combines an activity label, preceding context, candidate endings, and a source identifier.The example concerns removing ice from a car and includes context beginning with a man writing over snow on a windshield.
  • Prompts: Prompts: Continuation tasks ask models to complete descriptions or sentences, identify topics, or judge whether a candidate ending is appropriate.Templates include four answer choices, randomized candidate continuations, and Yes-or-No judgments.
  • Data Example: Data Example: COMMON GEN represents concepts such as ski, mountain, and skier alongside the target sentence “Skier skis down the mountain.”The example pairs a concept list with a generated sentence target.
  • Prompts: Prompts: Concept-to-text templates ask for sentences using all concepts or a selected concept, while reverse templates extract topics from a target sentence.Several templates explicitly restrict generation to the supplied concept list.
  • Data Example: Data Example: Other examples cover table-to-biography generation, article and dialogue summarization, article titling, and multi-document synthesis.The supplied instances include biographical tables, news articles, dialogues, article summaries, and collections of documents.
  • Prompts: Prompts: Classification templates recast labels as audience, newspaper-section, category, or question-type choices, using explicit answer-choice lists and label mappings.Examples span AG NEWS, DBPEDIA 14, and TREC-style prompts with alternatives such as World politics, Sports, Business, and Science and technology.
Loading 2110.08207v3…