PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Emergent Abilities of Language Models from the Loss Perspective

Zhengxiao Du, Aohan Zeng, Yuxiao Dong, Jie Tang

arXiv:2403.15796v3cs.CLcs.AIcs.LG

TL;DR

The paper addresses whether emergent abilities are genuinely tied to model scale and whether discontinuous metrics create their appearance. It studies downstream performance through pre-training loss across varied models and tasks, finding shared loss-linked behavior and thresholded emergence that persists with continuous metrics. These results motivate defining emergence by lower versus higher pre-training loss.

  • Problem

    Emergent abilities are disputed because smaller models can perform well on claimed emergent tasks and discontinuous metrics may produce their appearance.

    Source: §1 · pp. 1–2

  • Method

    The study pre-trains more than 30 Transformer models with varied model and data sizes under fixed corpus, tokenization, and architecture, then evaluates 12 diverse datasets and continuous metrics.

    Source: §1 · p. 2, §3.2 · p. 8

  • Results

    Across varied models and tasks, pre-training loss predicts downstream performance, while certain abilities improve beyond random guessing only below a specific threshold, including under continuous metrics.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Emergent abilities are redefined as abilities present at lower but absent at higher pre-training loss, highlighting tipping points that simple extrapolation may miss.

    Source: §1 · p. 2, §6 · p. 9

    The study does not cover fundamentally different architectures or optimizers, and pre-training losses from different corpora are not directly comparable.

    Source: §7 · p. 10

Abstract

from arXiv · show

Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities on certain tasks -- regardless of the continuity of metrics -- when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.

1 Introduction

The paper reframes emergent abilities around pre-training loss rather than model size or compute, addressing challenges to size-based emergence and discontinuous evaluation. Across varied models and tasks, it finds loss-linked performance patterns and thresholded improvements that motivate a new definition of emergence.

Motivation and approach

  • Smaller language models can outperform larger ones on claimed emergent tasks, while discontinuous evaluation metrics may also create the appearance of emergence.These challenges motivate studying loss rather than size and testing emergence with continuous metrics.Source: §1 · p. 1
  • Pre-training loss is proposed as a more direct lens for downstream scaling than model size or training compute, whose relationship to task performance remains less studied.The study uses more than 30 models with varied model and data sizes while fixing the corpus, tokenization, and architecture, and evaluates 12 diverse datasets.Source: §1 · pp. 1–2

Main findings

  • Models with the same pre-training loss show predictable downstream performance despite differing model and data sizes, across diverse tasks, languages, prompting types, and answer forms.The relationship is further examined using open LLaMA [55] and Pythia [3] models.Source: §1 · p. 2
  • Certain tasks improve beyond random guessing only after pre-training loss falls below a specific threshold, while other task performance improves from the outset.The threshold is reported as shared across these emergent tasks.Source: §1 · p. 2
  • The thresholded improvement persists under both discontinuous and continuous metrics, challenging the explanation that emergent abilities arise only from metric choice.Continuous evaluation uses CorrectChoiceProb and Brier Score [5] on intermediate MMLU and C-Eval checkpoints.Source: §1 · p. 2

Redefinition and implications

  • The paper defines an ability as emergent when it is absent at higher pre-training loss but present at lower loss, capturing tipping points that simple extrapolation may miss.Because loss depends on model size, data size, and compute through scaling laws [22] [28], the definition can also account for size- or compute-based emergence.Source: §1 · p. 2

2 Does Pre-training Loss Predict Task Performance?

The study evaluates whether pre-training loss predicts downstream performance across varied model sizes, token counts, languages, tasks, and prompting settings. Across controlled experiments and external model series, models with similar loss generally show similar performance, while several tasks exhibit thresholded emergence.

2.1 Pre-training Setting

  • The study uses a fixed English-Chinese corpus mixture, LLaMA-like architecture, and intermediate checkpoints to relate training loss to task performance.The pre-training corpus contains English and Chinese tokens in a 4:1 ratio, and checkpoints are evaluated approximately every 43B tokens.Source: §2.1 · p. 3, §2.3 · pp. 3–4

2.2 Evaluation Tasks

  • The evaluation covers 12 English and Chinese datasets spanning closed-book QA, commonsense NLI, reading comprehension, coreference, examinations, and math word problems.Prompting includes few-shot, zero-shot, and few-shot chain-of-thought, with multiple answer forms and metrics.Source: §2.2 · p. 3

2.3 Pre-training Loss vs. Performance

  • Performance generally improves as training loss decreases, and different model sizes follow the same performance-versus-loss trend across downstream tasks.Correlation is strong on tasks such as TriviaQA, HellaSwag, RACE, and WinoGrande, while it is weaker on MMLU, C-Eval, GSM8K, and GSM8K-Chinese.Source: §2.3 · pp. 4–5
  • MMLU, C-Eval, GSM8K, and GSM8K-Chinese remain near random performance until pre-training loss falls to about 2.2, after which performance increases.These tasks therefore differ from tasks whose performance improves smoothly as loss decreases from the beginning.Source: §2.3 · p. 4, §2.4 · p. 5

2.4 Training Token Count vs. Performance

  • Models with the same pre-training loss generally achieve the same downstream performance across model sizes and training-token counts on 12 tasks.The experiments vary model sizes from 300M to 32B and training tokens from 33B to 500B while keeping the corpus, tokenization, and architecture fixed.Source: §2 · p. 3, §2.3 · p. 4, §2.4 · p. 5

2.5 LLaMA’s Loss vs. Performance

  • LLaMA models of 7B, 13B, 33B, and 65B parameters mostly follow the same upward loss-performance trend, with an early LLaMA-65B exception above loss 1.8.The authors note that the outliers comprise the initial 10% of training tokens and may reflect exponential smoothing in the source plots.Source: §2.5 · p. 6

3 Analysis of Different Tasks and Metrics

Tasks divide into smoothly improving abilities and thresholded abilities that remain at random-guessing performance until pre-training loss falls sufficiently. Continuous metrics preserve the same tipping-point pattern, so discontinuous evaluation does not fully explain the emergence.

3.1 Performance Trends of Different Tasks

  • TriviaQA, HellaSwag, RACE, WinoGrande, NLPCC-KBQA, ClozeT, CLUEWSC, and C3 improve smoothly as pre-training loss decreases from the outset.Their correlation coefficients are close to -1, whereas the thresholded-task group has weaker correlations.Source: §3.1 · pp. 6–7
  • For MMLU, accuracy remains around 25% when pre-training loss exceeds 2.2, matching random guessing, then increases as loss decreases below 2.2.C-Eval, GSM8K, and GSM8K-Chinese show a similar pattern, with performance-improvement thresholds around 2.2 despite differing languages, tasks, prompting types, and answer forms.Source: §3.1 · p. 7
  • RACE and MMLU share an in-context multiple-choice format but have different curves, suggesting task difficulty as a possible explanation for their distinct performance trends.The paper contrasts easier reading-comprehension or commonsense tasks with broader-knowledge examinations and math word problems.Source: §3.1 · p. 7
  • A model-size threshold can correspond to the loss tipping point, after which models exceed random-chance performance on thresholded tasks.With fixed training tokens, the scaling-law relationship between model size and pre-training loss provides this correspondence.Source: §3.1 · pp. 7–8

3.2 Influence of Different Metrics

  • Accuracy, CorrectChoiceProb, and BrierScore all show performance changes below a loss threshold, while BrierScore’s above-threshold decrease does not necessarily indicate task improvement.For uniform four-option answers, a context-free predictor achieves BrierScore 0.75, so lower BrierScore before the threshold can still correspond to random-level performance.Source: §3.2 · p. 8
  • Continuous metrics do not eliminate the observed tipping point, supporting emergent abilities as a loss-threshold phenomenon rather than solely a discontinuous-metric artifact.This conclusion is drawn from MMLU and C-Eval evaluations using correct-answer probability and Brier Score alongside accuracy.Source: §3.2 · p. 8

4 Defining Emergent Abilities from the Loss Perspective

The paper defines an emergent ability by pre-training loss: it is absent in higher-loss models but present in lower-loss models. A thresholded normalized-performance formulation connects this definition to scaling relations in model size.

4 Defining Emergent Abilities from the Loss Perspective

  • An ability is emergent when it is absent in models with higher pre-training loss but present in models with lower pre-training loss.This definition follows observations that some tasks improve from random-guessing performance only after loss crosses a threshold.Source: §4 · p. 8
  • Normalized performance is modeled as a decreasing function of loss with threshold η, remaining at normalized random-guess performance below the threshold.The supplied formulation specifies f(L) as monotonically decreasing in L and assigns normalized random-guess performance of 0.Source: §4 · p. 8
  • With fixed training-token count, the loss–model-size relation follows a power law plus an irreducible-loss constant.The scaling relation uses model size N, irreducible loss L∞, and coefficient α_N.Source: §4 · pp. 8–9
  • Normalized performance is zero below model size N0 · (η − L∞)^−1/α_N and improves once model size exceeds that threshold.Increasing model size lowers pre-training loss in this regime, producing improved normalized performance.Source: §4 · p. 9

5 Related Work

Related work studies loss and downstream performance through transfer learning, prompting, or metric design, but often uses narrower model, task, corpus, or regime settings. This paper positions its contribution as evidence for loss-based emergence even with continuous metrics.

5 Related Work

  • Transfer-learning studies attribute differences at equal pre-training loss to architectural or model-shape inductive bias, often in fine-tuning settings.The cited theory is limited to saturation regimes, whereas contemporary large language models are generally under-trained.Source: §5 · p. 9
  • Prompted-performance studies report mixed relationships between perplexity and in-context learning, with evidence limited by model, benchmark, or corpus coverage.The paper contrasts OPT and subset-of-BIG-Bench evidence with findings that low perplexity does not always imply high in-context performance when the corpus changes.Source: §5 · p. 9
  • Prior challenges attribute emergent abilities to sufficient data in smaller models or to discontinuous evaluation metrics, while this paper argues for emergence from the pre-training-loss perspective with continuous metrics.The related-work discussion links these competing explanations to the paper’s stated contribution.Source: §5 · p. 9

6 Conclusion

The paper concludes that pre-training loss better represents scaling effects than model size or training compute for the studied tasks. Emergent improvements appear below a loss threshold even under continuous evaluation, identifying critical training junctures for future study.

6 Conclusion

  • Pre-training loss is presented as a better scaling metric than model size or training compute, with emergent performance increasing below a threshold even for continuous metrics.This conclusion summarizes the reported empirical pattern across the paper.Source: §6 · p. 9
  • The loss-based definition characterizes critical points in training trajectories where emergent abilities manifest and motivates studying the shifts occurring at those junctures.The conclusion connects these junctures to future investigation of capability development.Source: §6 · p. 9

7 Limitation

The study’s scope is limited by its focus on particular architectures, optimizers, tokenizers, and pre-training corpora. It also cautions that larger-scale training may not produce further tipping points and that instruction tuning changes zero-shot performance.

7 Limitation

  • The analysis does not cover fundamentally different architectures or optimizers, although the relationship holds for the studied models and LLaMA.The examined models use Transformer architectures and AdamW; routed, non-Transformer, and alternative-optimizer settings remain outside the study.Source: §7 · p. 10
  • Pre-training losses from models trained with different tokenizers or corpora are not directly comparable.The paper suggests normalized perplexity on a public validation set as one possible solution.Source: §7 · p. 10
  • The paper does not recommend expanding model and data sizes because larger scales are not guaranteed to reveal new tipping points.Instruction tuning can also improve zero-shot performance on unseen tasks, including MMLU and GSM8K.Source: §7 · p. 10

A.1 Pre-training Corpus

The experiments use a mixed English–Chinese corpus, BPE tokenization, and fixed training settings, while evaluation covers selected dataset splits. These choices define the corpus and evaluation conditions for the study.

A.1 Pre-training Corpus

  • Table 3 reports the ratio of different sources in the English corpus.The table provides the corpus-source breakdown for the English portion.Source: Table 3 · p. 19
  • The pre-training corpus mixes English and Chinese documents at a 4:1 token ratio, drawing from webpages, Wikipedia, books, and papers.Wikipedia and Books are repeated for multiple epochs, whereas 93.4% of documents are never repeated.Source: §A.1 · p. 19
  • The corpus uses SentencePiece byte pair encoding with a 65k vocabulary.This tokenization choice is part of the fixed pre-training setup.Source: §A.1 · p. 19

A.2 Hyperparameters

  • Training uses sequence length 2048 and AdamW with β1 = 0.9 and β2 = 0.95.These hyperparameters are reported for the pre-training configurations.Source: §A.2 · p. 19

B Evaluation Settings

  • Evaluation follows specified English and Chinese dataset splits, combining train and validation data for the small CLUEWSC validation set.GSM8K-Chinese is machine-translated from GSM8K and human-proofread.Source: B Evaluation Settings · p. 19

C Are Emergent Abilities of Language Models a Mirage?

The paper argues that apparent emergent abilities are not solely artifacts of discontinuous metrics, while smaller-model trajectories reproduce the reported performance–loss trend. Its analysis examines metric baselines and complete checkpoint curves.

C Are Emergent Abilities of Language Models a Mirage?

  • The paper disputes the claim that emergent abilities disappear under continuous metrics, arguing that correct and incorrect option perplexities diverge only near the end of training.This divergence is presented as evidence that emergent abilities remain observable beyond metric discontinuity concerns.Source: C Are Emergent Abilities of Language… · p. 20
  • The authors argue that correct-option perplexity and Brier Score can improve without reflecting learned question–answer relationships.For four-option questions, a Brier Score of 0.75 is treated as the random-guess baseline, and decreases above it are not necessarily real task improvement.Source: C Are Emergent Abilities of Language… · pp. 19–21
  • Only the largest model surpasses random-guess baselines on the evaluated BIG-Bench tasks, supporting emergent abilities under Brier Score analysis.The random-guess baselines are 0.25, 0.25, 0.1875, and 0.16 for tasks with 2, 2, 4, and 5 options.Source: C Are Emergent Abilities of Language… · p. 21

D Complete Performance-vs-Loss Curves of Smaller Models

E Loss vs Compute as an Indicator of Performance

Across compute comparisons, Pythia checkpoints, and BIG-Bench tasks, the paper evaluates whether pre-training loss better aligns with downstream performance and reveals tipping points.

E Loss vs Compute as an Indicator of Performance

F Pythia’s Loss vs. Performance

  • Pythia models fall on the same performance–loss curve across six downstream tasks, supporting the predictiveness of pre-training loss.The analysis uses SciQ, LAMBADA, WinoGrande, ARC-Easy, ARC-Challenge, and PIQA.Source: F Pythia’s Loss vs. Performance · p. 22, Figure 7 · p. 23
  • Pythia and LLaMA cannot analyze emergent abilities because Pythia does not exceed random chance on MMLU and GSM8K, while LLaMA lacks intermediate checkpoints.Consequently, their available evidence supports loss–performance predictiveness but not emergent-ability analysis.Source: F Pythia’s Loss vs. Performance · p. 22

G Loss vs. Performance on BIG-bench

H Compute Resources

The paper reports compute requirements for its models and connects loss-based scaling to estimating training costs. It also cautions that extrapolating prior performance trends to larger models may waste compute when pursuing new emergent abilities.

H Compute Resources

I Broader Impact

  • Pre-training loss can help estimate the compute required to reach a target performance and the cost of training a large model.Source: I Broader Impact · p. 23
  • Expanding model and data sizes to pursue new emergent abilities may waste compute because prior performance trends may not apply to larger models.Source: I Broader Impact · p. 23

Related papers

Loading 2403.15796v3…