Finetuned Language Models Are Zero-Shot Learners

• Large language models such as GPT-3 perform few-shot learning well but are less successful zero-shot, including on reading comprehension, question answering, and natural language inference.Without few-shot examples, prompts may be less similar to pretraining formats, making them harder for models to handle.Source: §1 · p. 2
• FLAN is a 137B-parameter pretrained model instruction-tuned on a mixture of more than 60 NLP datasets expressed through natural-language instructions.Instruction tuning uses finetuning supervision to improve responses to inference-time text interactions.Source: §1 · p. 2
• To test zero-shot transfer to unseen task types, the authors hold out each task-type cluster for evaluation and instruction-tune FLAN on the remaining clusters.For natural language inference, for example, the tuning tasks include commonsense reasoning, translation, and sentiment analysis, but no natural language inference tasks.Source: §1 · p. 2
• 20 of 25 datasets: FLAN outperforms zero-shot 175B GPT-3, and it beats few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.FLAN also substantially improves zero-shot performance over its 137B-parameter base model.Source: §1 · p. 2
• Ablations show that increasing the number of task clusters in instruction tuning improves unseen-task performance, while its benefits emerge only at sufficient model scale.Source: §1 · p. 2

2.1 TASKS & TEMPLATES

• The instruction-tuning mixture combines 62 publicly available text datasets covering language understanding and generation, organized into twelve task clusters.Figure 3 depicts the datasets and clusters.Source: §2.1 · p. 3, Figure 3 · p. 3
• Each dataset receives ten natural-language instruction templates, including up to three that turn the task around; examples use a randomly selected template.For example, sentiment classification can be reframed as generating a movie review.Source: §2.1 · p. 3

2.2 EVALUATION SPLITS

• Zero-shot evaluation holds out every dataset in a task cluster, and trains a separate model for each held-out cluster.A dataset counts as unseen only if none of its task clusters appeared in instruction tuning.Source: §2.2 · p. 3

2.3 CLASSIFICATION WITH OPTIONS

• For generation tasks, FLAN responds in free text without further modifications.Its decoder-only language model architecture naturally produces free-text outputs.Source: §2.3 · p. 4
• For classification, FLAN appends OPTIONS and the task’s output classes so the model knows which choices are desired.This addresses the possibility that probability mass is divided among alternative ways of expressing an answer.Source: §2.3 · p. 4

2.4 TRAINING DETAILS

• LaMDA-PT is a dense left-to-right, decoder-only transformer language model with 137B parameters, pretrained on web documents, code, dialog data, and Wikipedia tokenized into 2.49T BPE tokens using SentencePiece with a 32k vocabulary.Around 10% of the pretraining data was non-English; LaMDA-PT had language-model pretraining only.Source: §2.4 · p. 4
• FLAN training caps each dataset at 30k examples and uses examples-proportional mixing with a maximum mixing rate of 3k.Models are finetuned for 30k gradient steps, and evaluation uses the final checkpoint. Training takes around 60 hours on a TPUv3 with 128 cores.Source: §2.4 · p. 4

Overall results

• On the best dev-selected template, zero-shot FLAN beat zero-shot GPT-3 on 20 of 25 datasets and zero-shot GLaM on 13 of 19 available datasets.FLAN also surpassed few-shot GPT-3 on 10 datasets and one-shot GLaM on 11 of 19 available datasets.Source: §3 · p. 4
• Instruction tuning helped most on naturally verbalized tasks, while it was less effective when language-modeling tasks made instructions largely redundant.The latter included commonsense reasoning and coreference resolution tasks formatted as sentence or paragraph completions.Source: §3 · pp. 5–6

Natural language inference (NLI)

• On five NLI datasets, FLAN outperformed all baselines by a large margin when examples were phrased as natural questions.The phrasing asked whether the premise meant that the hypothesis was true.Source: §3 · p. 5, Figure 5 · p. 5

Reading comprehension

• For reading comprehension, FLAN outperformed baselines on MultiRC and OBQA and GPT-3 by a large margin on BoolQ, where LaMDA-PT already performed well.Figure 5 summarizes results for reading comprehension alongside NLI, closed-book QA, and translation.Source: §3 · p. 5, Figure 5 · p. 5

Closed-book QA

• For closed-book QA, FLAN beat GPT-3 on all four datasets and outperformed GLaM on ARC-e and ARC-c, but trailed it slightly on NQ and TQA.These tasks ask models to answer world-knowledge questions without access to information containing the answer.Source: §3 · p. 5, Figure 5 · p. 5

Translation

• On translation, FLAN outperformed zero-shot GPT-3 in all six evaluations but underperformed few-shot GPT-3 in most cases.It was strong translating into English and weaker translating from English into other languages; the authors note its English sentencepiece tokenizer and predominantly English pretraining data as possible reasons.Source: Figure 5 · p. 5, §3 · p. 6

Additional tasks

• Across seven commonsense reasoning and coreference tasks, FLAN beat LaMDA-PT on only three, indicating little benefit when instructions duplicated the pretraining objective.On sentiment analysis, paraphrase detection, struct-to-text, and other datasets without GPT-3 results, FLAN generally beat zero-shot LaMDA-PT and matched or exceeded few-shot LaMDA-PT.Source: §3 · p. 6

4.1 NUMBER OF INSTRUCTION TUNING CLUSTERS

• Average performance across three held-out clusters improved as more clusters and tasks were added, except for sentiment, and showed no apparent saturation.The ablation found minimal added value from the sentiment-analysis cluster but could not determine which tuning cluster contributed most to each evaluation cluster.Source: §4.1 · p. 6, Figure 6 · p. 6

4.2 SCALING LAWS

• Instruction tuning improved held-out performance for models around 100B parameters but hurt it for models of 8B parameters or smaller.The authors suggest that learning the instruction-tuning tasks may consume small models’ capacity, while larger models retain capacity to learn instruction following and generalize.Source: §4.2 · p. 7, Figure 7 · p. 7

4.3 ROLE OF INSTRUCTIONS

• Both instruction-free finetuning setups performed substantially worse than FLAN, indicating that natural-language instructions are crucial for zero-shot performance on unseen tasks.The two setups used inputs and outputs only, or prepended each input with the task and dataset name.Source: §4.3 · p. 7, Figure 8 · p. 7

4.4 INSTRUCTIONS WITH FEW-SHOT EXEMPLARS

• Few-shot exemplars improved performance over zero-shot FLAN across all task clusters, especially on tasks with large or complex output spaces, and reduced template sensitivity.The authors suggest exemplars may help the model understand output formats; standard deviation among templates was lower for few-shot FLAN in every cluster.Source: §4.4 · p. 8, Figure 9 · p. 8

4.5 INSTRUCTION TUNING FACILITATES PROMPT TUNING

• Prompt tuning with FLAN outperformed LaMDA-PT in all tested scenarios, with more than 10% improvement in many cases, especially in the 32-example low-resource setting.The evaluation covered both fully supervised training and a low-resource setting with 32 training examples.Source: §4.5 · p. 8, Figure 10 · p. 8

5 RELATED WORK

• Related areas include zero-shot learning, prompting, multi-task learning, and language models for NLP applications (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020; Efrat & Levy, 2020; Aghajanyan et al., 2021; Li & Liang, 2021).The paper directs readers to Appendix D for extended discussion and focuses here on two narrower subareas most closely related to its work.Source: §5 · p. 8
• Unlike QA-based task formulations that mainly target multi-task learning, this work emphasizes zero-shot learning and pretrained-model knowledge, and exceeds Chai et al. (2020) and Zhong et al. (2021) in model scale and task scope.QA-based formulations cast NLP tasks as questions and answers over a context (Kumar et al., 2016; McCann et al., 2018); Liu et al. (2021) noted that they generally were not motivated by using existing pretrained-LM knowledge.Source: §5 · p. 9
• Prior instruction-following studies report few-shot gains on unseen tasks with 140M BART and zero-shot gains with 11B T5, while InstructGPT uses finetuning and reinforcement learning to improve human-rater preference (Mishra et al., 2021; Ye et al., 2021; Sanh et al., 2021; Ouyang et al., 2022).Mishra et al. (2021) use few-shot exemplars; Ye et al. (2021) place less emphasis on instructions. The paper notes these results suggest task-collection finetuning can improve unseen-task few-shot performance even at smaller scale.Source: §5 · p. 9

6 DISCUSSION

• FLAN improves over an untuned model and surpasses zero-shot GPT-3 on most evaluated tasks; unseen-task gains rise with task-cluster count and require sufficient model scale.Instruction tuning can also be combined with few-shot prompting and prompt tuning.Source: §6 · p. 9
• Instruction tuning shows labeled data can help large language models perform many unseen tasks, making task-specific training complementary to general language modeling and motivating generalist-model research.Source: §6 · p. 9
• Limitations include subjective task clustering, typically single-sentence instructions, possible pretraining overlap without post-hoc evidence of substantial impact, and costly serving at 137B scale.Future work includes more task clusters, cross-lingual experiments, FLAN-generated training data for classifiers, and finetuning for bias and fairness (Solaiman & Dennison, 2021).Source: §6 · p. 9

7 CONCLUSIONS

• Instruction tuning is presented as a simple method for improving large language models’ ability to perform zero-shot tasks based purely on instructions.Source: §7 · p. 9
• FLAN compares favorably with GPT-3, signaling the potential for large language models to follow instructions.Source: §7 · p. 9
• The authors hope the work will spur research on instruction-based NLP, zero-shot learning, and using labeled data to improve large language models.Source: §7 · p. 9

• Finetuning on labeled datasets with undesirable biases can propagate those biases into zero-shot downstream applications.Source: ETHICAL CONSIDERATIONS · p. 10
• Instruction-tuned models may require less data and expertise to use, lowering access barriers and potentially increasing both benefits and associated risks.Source: ETHICAL CONSIDERATIONS · p. 10

ENVIRONMENTAL CONSIDERATIONS

• 451 MWh and 26 tCO2e were the pretrained models’ energy cost and carbon footprint, while FLAN’s additional tuning used less than 2% as many gradient steps as pretraining.The paper estimates that the additional energy cost of instruction tuning was comparatively smaller.Source: ENVIRONMENTAL CONSIDERATIONS · p. 10

A ADDITIONAL RESULTS

• Table 1 presents translation and struct-to-text results, and Table 2 presents results for eight NLU task clusters.Table 2 uses accuracy or exact match except for DROP, MultiRC, and SQuAD v1 and v2, which use F1.Source: A ADDITIONAL RESULTS · p. 22, Table 1 · p. 22, Table 2 · p. 23
• Summarization evaluation is deferred because the mean input of most summarization datasets exceeds FLAN’s 1024-token input length.Source: A ADDITIONAL RESULTS · p. 22

PARAPHRASE

• The paraphrase results excerpt lists MRPC, QQP, and PAWS Wiki, but does not identify the columns associated with their reported values.Source: PARAPHRASE · p. 23

COREFERENCE

• Table 2 reports results for eight NLU task clusters; it omits few-shot FLAN results for WSC273 and reports Trivia QA exact match on both a Wikipedia dev subset and the full TFDS dev set.Source: Table 2 · p. 23

B.1 DATASETS PER TASK CLUSTER & TEMPLATES PER DATASET

• Using more datasets per cluster improved held-out performance by almost 10% on average, while template effects were negligible and disappeared with four datasets per cluster.The held-out clusters were NLI, commonsense reasoning, and closed-book QA.Source: §B.1 · p. 24, Figure 11 · p. 24

B.2 ROLE OF INSTRUCTIONS DURING FINETUNING; B.3 FURTHER ANALYSIS: INSTRUCTION TUNING FACILITATES PROMPT TUNING

• The analysis points to per-cluster ablation results in Table 3 and per-dataset results in Table 4, and suggests exploring summarization, question answering, or supervised-dataset finetuning.The passage says the per-dataset tasks are all classification.Source: §B.3 · p. 24, §B.2 · p. 24

C DATA CONTAMINATION ANALYSIS

• The analysis removes examples with roughly 13-gram overlaps with pretraining data, then compares clean-subset performance with performance on the original dataset.The n-gram size varies by dataset and follows Brown et al. (2020); a lower clean-subset score would suggest contamination inflated results.Source: C DATA CONTAMINATION ANALYSIS · p. 25
• DROP and SQuADv2 had almost total overlap, but overlaps rarely covered both question and answer, limiting answer-specific memorization.For DROP, 99.6% of overlapping n-grams were in contexts and none occurred in both question and answer; for SQuADv2, the figures were 97.2% and 5 of 11,153 examples, respectively.Source: C DATA CONTAMINATION ANALYSIS · pp. 25–26
• Across datasets, clean-subset evaluation showed no correlation with overlap and no general tendency to perform worse; small clean subsets had higher variance.Figure 12 summarizes the comparison; the paper notes many datasets had substantial overlap with pretraining data.Source: C DATA CONTAMINATION ANALYSIS · p. 25, Figure 12 · p. 26
• ANLI R1 and R2 had almost complete contamination, but most overlaps were in example contexts, which the authors consider unlikely to affect performance on unseen hypotheses.The paper reports 97.3% of R1 and 98.2% of R2 overlaps in contexts, and suggests newer Wikipedia data may explain the overlap.Source: C DATA CONTAMINATION ANALYSIS · p. 26
• Only ReCoRD and PIQA had clean-subset performance more than 1% below overall performance, so the authors flag these results as potentially contamination-sensitive.Both are language-modeling tasks; PIQA had goal-and-solution overlaps in 93 of 1,838 examples, while ReCoRD had query overlaps in 2,320 of 10,000 training examples.Source: C DATA CONTAMINATION ANALYSIS · p. 26
• The authors caution that conservative n-gram matching can produce false positives, clean subsets may differ in distribution, and contamination detection lacks established best practices.They also note that FLAN’s pretraining corpus is almost five times larger than GPT-3’s, potentially increasing false positives.Source: C DATA CONTAMINATION ANALYSIS · p. 26

D.1 LANGUAGE MODELS AND MULTI-TASK LEARNING; D.2 ZERO-SHOT LEARNING AND META-LEARNING

• Instruction tuning frames multitask learning around generalizing zero-shot to unseen tasks, while this paper uses zero-shot to mean generalization to unseen tasks.The related-work discussion contrasts this motivation with prior multitask learning focused on training-task performance or new domains, and distinguishes zero-shot learning from meta-learning’s typical use of a few examples for adaptation.Source: §D.1 · p. 28, §D.2 · p. 28

D.3 PROMPTING; E FREQUENTLY ASKED QUESTIONS

• FLAN uses natural-language instructions rather than prompts that mimic pretraining text or simply tag a dataset, aiming to make large models accessible to non-technical users.The paper presents prompt tuning as orthogonal to instruction tuning, which can additionally improve performance; FLAN prompts are formulated as requests to perform tasks.Source: §D.3 · p. 28, E FREQUENTLY ASKED QUESTIONS · p. 30

D.4 FINETUNING LARGE LANGUAGE MODELS; D.5 MULTI-TASK QUESTION ANSWERING; D.6 INSTRUCTIONS-BASED NLP

• Unlike prior work focused on downstream-task finetuning, domain transfer, or QA formulations, this work studies zero-shot performance on unseen tasks across a broad task range.The related-work discussion contrasts the scope with prior large-model finetuning, QA-based task formulations, and instructions-based studies using few-shot learning; it notes that comparisons with Sanh et al. (2021) and Min et al. (2021) are difficult because model sizes, types, data, task mixtures, and instructions differ.Source: §D.4 · p. 29, §D.5 · p. 29, §D.6 · p. 29

E FREQUENTLY ASKED QUESTIONS

• FLAN-style instruction prompts performed near zero on most generation tasks with untuned LaMDA-PT, so the paper used established GPT-3 prompts for its LaMDA-PT baselines.For example, LaMDA-PT continued a translation request as ordinary text rather than translating it.Source: E FREQUENTLY ASKED QUESTIONS · p. 30

What are some limitations/failure cases of FLAN?; Can FLAN be used when large amounts of training data are available?

• FLAN has failure cases, a 1024-token context limit, and mostly English training data; the paper leaves possible positive transfer among seen tasks for future work.The reported failures include returning the wrong word and translating a question instead of answering it; prompt tuning on the FLAN checkpoint provides promising evidence of positive transfer in a supervised setting.Source: What are some limitations/failure… · p. 30, Can FLAN be used when large amounts of… · p. 30

Are the ten unique templates per dataset or per task cluster?

• The ten templates were assigned per dataset, and untuned LaMDA-PT did not always improve with scale on zero-shot reading comprehension and sentiment tasks.The paper says this artifact is consistent with reported GPT-3 declines on BoolQ and DROP and findings for SST-2; it notes that cross-validation instead of a single split might have smoothed the effect.Source: Are the ten unique templates per… · pp. 30–31

F QUALITATIVE EXAMPLES; CHANGES FROM V4 TO V5; CHANGES TO V4 FROM V3; CHANGES TO V3 FROM V2

• Qualitative examples show FLAN changing sentiment answers appropriately when questions are flipped, answering questions in another language when instructed, and performing zero-shot rule-based manipulations.Source: Figure 13 · p. 31, Figure 14 · p. 31, Figure 15 · p. 32

CHANGES TO V2 FROM V1

• Zero-shot FLAN beats zero-shot GPT-3 on TriviaQA, bringing the reported total to 20 of 25 tasks.The v2 revision added an evaluation on GPT-3’s Wikipedia dev set of 7,993 examples because the v1 result used the 11,313-example TFDS dev set; the original result remains in Table 2 without a GPT-3 comparison.Source: CHANGES TO V2 FROM V1 · p. 35

G.1 NATURAL LANGUAGE INFERENCE

• Natural language inference covers relations between two sentences and uses ANLI, CB, MNLI, QNLI, SNLI, WNLI, and RTE.ANLI includes three rounds, with training examples capped at 30,000 in rounds R2 and R3.Source: G TASKS AND DATASETS · p. 36, Table 6 · p. 38

G.2 READING COMPREHENSION

• Reading-comprehension datasets include BoolQ, DROP, MultiRC, OBQA, SQuADv1, and SQuADv2.The appendix describes questions answered from a passage; OBQA uses four-way multiple-choice questions based on facts.Source: G TASKS AND DATASETS · p. 36, Table 11 · p. 41

G.3 COMMONSENSE REASONING

• Commonsense reasoning uses COPA, HellaSwag, PiQA, and StoryCloze to assess causal inference, plausible sentence completion, physical reasoning, and story endings.COPA selects a cause or effect from two choices, while PiQA selects a solution to a goal from two choices.Source: G TASKS AND DATASETS · p. 36, Table 12 · p. 42, Table 14 · p. 43

G.4 CLOSED-BOOK QA

• Closed-book QA includes ARC, NQ, and TriviaQA, covering science questions, open-ended questions answerable from Wikipedia, and trivia question-answer pairs.The listed task cluster asks models to answer world-knowledge questions without access to information containing the answer.Source: G TASKS AND DATASETS · p. 36, Table 17 · p. 44

G.5 COREFERENCE RESOLUTION

• Coreference resolution tests identifying expressions of the same entity in text; the datasets used are DPR, Winogrande, and WSC273, which requires sentence completion that depends on understanding the entities.WSC273 is test-set only.Source: G TASKS AND DATASETS · pp. 36–37, Table 20 · p. 45

G TASKS AND DATASETS

• The appendix includes commonsense reading, structured-to-text, summarization, and miscellaneous clusters, and caps each dataset’s training set at 30,000 examples.When TFDS provides a labeled test set, it is used; otherwise, the validation set serves as test data and training data is split into train and dev sets.Source: G TASKS AND DATASETS · p. 37

G.6 READING COMPREHENSION WITH COMMONSENSE

• Reading comprehension with commonsense combines both abilities, with CosmosQA and ReCoRD as its datasets.ReCoRD uses a cloze-style question with a masked entity.Source: G TASKS AND DATASETS · p. 37, Table 21 · p. 46

G.7 TRANSLATION (7 LANGUAGES)

• Translation covers eight English-to-language directions: French, German, Turkish, Czech, Finnish, Romanian, Russian, and Spanish. En–Fr uses WMT’14; En–De, En–Tr, En–Cs, En–Fi, En–Ro, and En–Ru use WMT’16; En–Es uses Paracrawl.Source: G TASKS AND DATASETS · p. 37