PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Abilities of Large Language Models

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus

arXiv:2206.07682v2cs.CL

TL;DR

Scaling laws do not fully explain downstream task behavior when some abilities are absent in smaller models but present in larger ones. This paper defines and surveys these emergent abilities across language models, tasks, and prompting settings, finding that meaningful performance can appear only at certain computational scales.

  • Problem

    Scaling laws predict broad improvements, but some downstream task abilities do not continuously improve with scale and remain difficult to predict from smaller models.

  • Method

    The paper defines emergence by scale and surveys prior examples across few-shot prompting, augmented prompting, language models, task types, and experimental settings.

  • Results

    Emergent abilities span diverse language models, task types, and experimental scenarios, with meaningful performance observed only at certain computational scales.

  • Takeaways & Limitations

    Emergence makes understanding how abilities arise and whether further scaling enables additional abilities important future research directions for NLP.

  • Takeaways & Limitations

    The paper’s analyses do not completely explain why emergent abilities occur or how to predict them.

Abstract

from arXiv · show

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

1 Introduction

Scaling language models generally improves downstream performance and sample efficiency predictably, but this paper examines emergent abilities: capabilities absent in smaller models yet present in larger ones. It analyzes emergence through training compute and model-parameter scale, emphasizing that such abilities cannot be predicted by extrapolating smaller-model improvements.

  • Scaling language models can improve performance and sample efficiency across downstream NLP tasks, often in ways predicted by scaling laws.
  • The paper focuses on the unpredictable phenomenon of emergent abilities in large language models.
  • Emergent abilities are absent in smaller-scale models but present in large-scale models, so smaller-model performance improvements cannot simply predict them.
  • The paper studies emergence with respect to model scale measured by training compute and the number of model parameters.

2 Emergent Abilities Definition

The paper defines emergent abilities as capabilities absent in smaller models but present in larger ones, whose sudden improvement cannot be predicted by extrapolating smaller-model scaling laws. It analyzes emergence primarily through training computation and model size while emphasizing that observed thresholds depend on correlated training factors and are not immutable.

  • An ability is emergent when it is absent in smaller models but present in larger models.
  • Emergent abilities show near-random performance until a critical scale threshold, followed by substantially above-random performance—a qualitative phase transition.
  • The analysis plots performance against training compute measured in FLOPs and additionally examines model parameters because greater compute generally corresponds to more parameters.
  • Training dataset size is not used as the primary capability axis because many model families hold training examples fixed across model sizes, and no single proxy captures all aspects of scale.
  • The scale at which emergence is observed depends on training-data quality, data availability, compute, and parameter count rather than being an immutable property of the ability.

3 Few-Shot Prompted Tasks

Few-shot prompting exhibits emergent abilities: models can remain at random performance until a scale threshold, after which performance rises well above random. Across BIG-Bench and other benchmarks, these thresholds vary by task and model family, with some abilities appearing only at the largest tested scales.

  • Few-Shot Prompting: Few-shot prompting supplies input-output examples in a model’s context and requires no further training or gradient updates.The prompting setup uses a preamble of examples before the task request.
  • Emergent Abilities: Emergent few-shot abilities remain near random until a scale threshold, then increase sharply to well above random across eight tasks and five model families.This pattern defines emergence in the few-shot prompting setting.
  • BIG-Bench: 2 · 10^22 training FLOPs for GPT-3 and 10^23 training FLOPs for LaMDA mark above-random jumps on BIG-Bench arithmetic, with similar-scale emergence on transliteration, scrambled-word recovery, and Persian question-answering.The corresponding model sizes are 13B parameters for GPT-3 and 68B parameters for LaMDA.
  • TruthfulQA: 5 · 10^23 training FLOPs (280B parameters) enables Gopher to exceed random performance by more than 20% on TruthfulQA, while GPT-3 remains at random at its largest scale.TruthfulQA is adversarially curated against GPT-3 models and measures truthful question answering.
  • Multi-task Language Understanding: On MMLU, models at ∼10^22 training FLOPs (∼10B parameters) or smaller perform no better than guessing, while 3–5 · 10^23 training FLOPs (70B–280B parameters) substantially surpass random.MMLU aggregates 57 tests spanning topics including math, history, and law.
  • Word in Context: WiC remains at or below random for GPT-3 and Chinchilla at ∼5 · 10^23 FLOPs, but above-random performance emerges for PaLM at 2.5·10^24 FLOPs (540B parameters).The PaLM threshold is much larger than the largest GPT-3 and Chinchilla scales described.

4 Augmented Prompting Strategies

Specialized prompting and finetuning strategies can exhibit emergent abilities, improving performance only after models reach sufficient scale. Examples include chain-of-thought prompting for multi-step reasoning and scratchpads for program execution.

  • Overview: Augmented prompting and finetuning techniques are considered emergent when they do not improve, or harm, baseline performance until models reach a large-enough scale.The section frames these strategies as alternatives to few-shot prompting for augmenting language-model abilities.
  • Multi-step reasoning: 1023 training FLOPs (∼100B parameters) is the scale at which chain-of-thought prompting surpasses standard prompting without intermediate steps.Chain-of-thought prompting guides models to produce intermediate reasoning steps before the final answer.
  • Multi-step reasoning: A similar emergence in performance gain occurs when few-shot prompting is augmented with explanations that follow the final answer.This provides another example of an augmented prompting strategy whose benefit appears with sufficient model scale.
  • Instruction following: Instruction finetuning enables models to respond appropriately to instructions describing unseen tasks, without few-shot exemplars.The approach finetunes models on a mixture of tasks phrased as instructions.
  • Program execution: ∼9 · 1019 training FLOPs (40M parameters) is the threshold at which scratchpads begin helping on 8-digit addition.Scratchpads are intermediate outputs predicted during finetuning to enable successful execution of multi-step computations.

5 Discussion

The discussion emphasizes that emergent abilities are difficult to predict from smaller models and that scaling has important limitations, risks, and evaluation dependencies. It outlines future directions including more data, improved prompting, and investigation of frontier tasks.

  • Emergence and implications: Emergent few-shot abilities cannot be predicted by extrapolating smaller-model performance, raising whether further scaling could produce additional capabilities.These tasks are not explicitly included in pre-training, and their full scope is likely unknown.
  • Emergence and implications: 540B parameters led to a significant performance jump without the major architectural changes suggested by prior work.The cited result associates the improvement with scaling compute and parameters rather than significant architectural changes.
  • Limitations and risks: Evaluation metrics can disguise gradual improvements as emergence by requiring exact final answers for long-sequence, multi-step, or arithmetic tasks.Exact string match and final-answer scoring provide no credit for partially correct solutions.
  • Limitations and risks: Scaling can face hardware bottlenecks, leave some abilities unobserved, fail on far-distribution tasks, or produce abilities that later plateau.The discussion presents these as limitations of relying only on training compute, model parameters, and dataset size.
  • Limitations and risks: Scaling can increase undesirable behaviors, including bias in ambiguous contexts, toxic responses, memorization, and imitation of human falsehoods.The cited benchmarks and studies report these risks across bias, toxicity, memorization, and truthfulness evaluations.
  • Future directions: Future work includes further scaling, larger training datasets, improved prompting, and research on frontier tasks that remain near or below random performance.These directions address computational and hardware constraints, model capabilities under fixed size, elicitation methods, and currently unperformed tasks.

6 Conclusions … A.1.1 Generative tasks

The paper reviews emergent abilities that appear at certain computational scales across models, tasks, and scenarios, and finds that scaling can improve small models in ways standard downstream metrics miss. Its analysis of six BIG-Bench tasks shows these abilities remain difficult to predict, motivating further study of their emergence, risks, and potential expansion through scaling.

  • 6 Conclusions: Emergent abilities show meaningful performance only at certain computational scales and span language models, task types, and experimental scenarios.The paper identifies their emergence through scaling as a recently discovered phenomenon and a major direction for future NLP research.
  • Broader Impact Statement: The paper surveys existing literature rather than proposing new methods or models, and emphasizes that emergent abilities are unpredictable and include emergent risks.These phenomena are presented as warranting careful study and raising important questions for the field.
  • A.1 Cross-entropy loss analysis: The analysis compares six few-shot prompted BIG-Bench tasks considered emergent for LaMDA, using EM or BLEU for generative tasks and accuracy for classification tasks.Three tasks are generative, while the other three are classification tasks.
  • A.1 Cross-entropy loss analysis: For small models (≤1022 FLOPs / ≤27B params), EM/BLEU/acc performance is close to random, so the analysis also examines cross-entropy loss.Cross-entropy can capture probability improvements even when downstream predictions remain wrong.
  • A.1 Cross-entropy loss analysis: Outcome 2 predicts that cross-entropy improves while EM/BLEU/acc remains random, indicating that small models improve in ways not reflected by the downstream metric.At a critical model scale, these improvements may enable the downstream metric to rise above random as an emergent ability.
  • A.1 Cross-entropy loss analysis: All six BIG-Bench tasks fall under Outcome 2, but the analysis provides no straightforward indicators for predicting their emergent behaviors.Thus, downstream metrics fail to capture some improvements without making emergence predictable.

A.1.2 Classification tasks

Across three emergent BIG-Bench classification tasks, cross-entropy loss can improve while accuracy remains near random. In multiple-choice tasks, correct and incorrect response log-probabilities initially decrease together before separating at a scale where performance rises substantially.

  • Classification tasks: Cross-entropy loss consistently improves for higher-compute models even when classification error rates remain close to random.The passages state that accuracy-based performance can mask consistent improvements in the likelihood of target sequences.
  • Multiple-choice analysis: In three emergent multiple-choice tasks, cross-entropy loss decreases for both correct and incorrect responses.Both log-probabilities can decrease together even though probabilities across available responses are normalized.
  • Multiple-choice analysis: At a certain scale, average log-probabilities for correct and incorrect responses deviate, coinciding with a substantial increase in task performance.Before this deviation, both response types can have very small log-probabilities.

A.2 Different metrics for generative tasks

Emergent behavior on three generative BIG-Bench tasks persists across all evaluation metrics, including metrics that award partial credit rather than requiring exact string matches.

  • A.2 Different metrics for generative tasks: Emergent behavior appears across all evaluation metrics for all three generative BIG-Bench tasks.The evaluated metrics include BLEU, ROUGE, and BLEURT, which award partial credit for non-exact answers.

A.3 BIG-Bench task analysis

The analysis manually classified 210 BIG-Bench tasks as emergent or not and examined emergence across task keywords and model scales. It found no clear trends identifying which task types were most emergent, while several categories showed substantial flat scaling.

  • Method: The authors manually classified all 210 BIG-Bench tasks as emergent or not using the paper’s near-random-until-larger-model definition.BIG-Bench contains over 200 tasks, each associated with author-provided keywords.
  • Emergence by keyword: Figure 8 stratified emergent tasks by whether they first emerged with LaMDA 137B or GPT-3 175B, or only with PaLM models.Non-emergent tasks were either smoothly increasing or flat, while 40 remaining tasks were not included in the plot.
  • Emergence by keyword: The most emergent keywords depended on comparing absolute counts versus percentages because keyword frequencies varied substantially and most had fewer than twenty tasks.Absolute counts can primarily reflect common keywords rather than emergence propensity.
  • Emergence by keyword: Reasoning-related tasks were not consistently more emergent, with only two of the top five tasks involving reasoning; arithmetic and mathematics also had relatively low emergent-task percentages.The passage reports no particularly high fraction for logical reasoning or causal reasoning, despite arithmetic being an early emergence example.
  • Flat scaling: Visual reasoning had the largest fraction of flat scaling tasks at 8/13, while non-language, repeated interaction, context length, computer code, and multi-step categories also had many flat tasks.These categories target weaknesses of large language models, and visual reasoning is outside their primary language design.

B Further MMLU analysis · C All Model Details · D Scaling with Parameter Count

The appendices examine how MMLU emergence varies across categories, document the models’ parameter and compute ranges, and illustrate emergence as a function of model scale. They also cover emergent prompting methods and task-general models surpassing task-specific state of the art.

  • B Further MMLU analysis: MMLU comprises 57 topics across four mutually exclusive categories, analyzed against training FLOPs, model parameters, and WikiText103 perplexity.The categories are Humanities, STEM, Social Science, and other.
  • B Further MMLU analysis: Social Science and Humanities show the largest performance jumps from the second-largest to largest model, while STEM shows the smallest.Across all three x-axes, the four categories have similar plot shapes.
  • B Further MMLU analysis: For both Chinchilla and Gopher, Social Science has the highest emergence and STEM the least; 7B models perform around random at 25% across categories.The largest models are 70B Chinchilla and 280B Gopher, compared with 7B second-largest models.
  • C All Model Details: The model collection spans from LaMDA at 2.1M parameters to PaLM at 540B parameters and 2.5E+24 training FLOPs.PaLM’s training budget is roughly 8x GPT-3’s computational budget.
  • D Scaling with Parameter Count: Figures 11–13 examine emergent abilities with number of model parameters as the x-axis.The appendix separately presents scaling with parameter count.
  • D Scaling with Parameter Count: Few-shot prompting can remain near random until a scale threshold, after which performance increases significantly above random.Models with more parameters typically use more training compute, motivating an analogous training-FLOPs view.
  • D Scaling with Parameter Count: Specialized prompting or finetuning methods can have no positive effect until a certain model scale, while task-general models can surpass task-specific state-of-the-art performance.These examples include results from LaMDA, Anthropic, and task-general models evaluated on selected benchmarks.

E BIG-Bench Task Classification … E.5 Other

The appendix classifies BIG-Bench tasks by scaling behavior, including smoothly increasing, emergent, flat, and other patterns. It also identifies exclusions, potential future emergence, evaluation limitations, and miscellaneous cases.

  • E BIG-Bench Task Classification: Each task appears in one category, and tasks initially emergent with GPT-3 or LaMDA are excluded from the PaLM emergence category.These annotations support Figure 8 in Appendix A.3.
  • E.4 Flat (no model better than random): Appendix E.4 lists tasks where no model performs better than random, making them potential candidates for future emergence if later models exceed random performance.These tasks exhibit flat scaling curves.
  • E.1 Smoothly increasing: The smoothly increasing category contains tasks whose performance improves with scale across diverse reasoning, language, knowledge, and understanding benchmarks.Examples include abstract narrative understanding, chess state tracking, date understanding, and unit conversion.
  • E.2 Emergent with GPT-3 or LaMDA: Tasks emergent with GPT-3 or LaMDA include analytic entailment, logical deduction, physical intuition, social IQA, StrategyQA, and word sorting.The category spans reasoning, knowledge, social understanding, and language tasks.
  • E.3 Emergent wih PaLM: Tasks emergent with PaLM include analogical similarity, auto debugging, causal judgment, elementary math QA, physics questions, temporal sequences, and fact verification.The listed tasks cover coding, reasoning, mathematics, science, and language understanding.
  • E.5 Other: Other tasks are better than random but not correlated with scale, including boolean expressions, dynamic counting, truthful QA, and social support.The category also includes bias, geometry, theorem-identification, and scientific-communication tasks.
  • E.5 Other: Model performance worsens with scale on BBQ Lite, bias from probabilities, diverse social bias, movie recommendation, and UnQover.These tasks form a distinct E.5 pattern rather than a conventional scaling improvement.
  • E.5 Other: E.5 also records arithmetic as emergent at 1B, while other entries have insufficient examples, incomplete evaluations, or unclear BLEURT behavior.Insufficient-example tasks include known unknowns, suicide risk, and what is the Tao; incomplete evaluations include convinceme, long context integration, and medical questions Russian.

F PaLM 62B is emergent but GPT-3 and LaMDA are not

PaLM 62B exhibits emergent abilities on many BIG-Bench tasks, whereas GPT-3 175B and LaMDA 137B do not. This shows that scale is not the only factor in emergence, since PaLM 62B has fewer parameters and less training FLOPs.

  • F PaLM 62B is emergent but GPT-3 and LaMDA are not: PaLM 62B’s emergence indicates that scale is not the only factor determining emergence.The passage explicitly connects this conclusion to PaLM 62B’s emergence despite its smaller parameter count and lower training FLOPs than the comparison models.
  • F PaLM 62B is emergent but GPT-3 and LaMDA are not: PaLM 62B shows emergence on many BIG-Bench tasks for which GPT-3 175B and LaMDA 137B do not.The comparison holds despite PaLM 62B having fewer model parameters and less training FLOPs.
  • F PaLM 62B is emergent but GPT-3 and LaMDA are not: The tasks include anachronisms, ascii word recognition, conceptual combinations, cryptonite, disambiguation qa, emoji movie, and goal step wikihow.They also include gre reading comprehension, linguistics puzzles, logic grid puzzle, metaphor boolean, metaphor understanding, odd one out, and parsinlu qa.
Loading 2206.07682v2…