PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Downstream Task Performance of Large Language Models

Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, Sanmi Koyejo

arXiv:2402.04177v3cs.CLcs.LGstat.ML

TL;DR

Scaling-law research has mainly examined pretraining loss, leaving downstream performance after finetuning less understood. This paper studies machine-translation transfer by varying pretraining data and finetuning size, finding that alignment shapes whether downstream scores scale monotonically and predictably. It also identifies conditions where more pretraining may not improve translation quality.

  • Problem

    Scaling-law research has primarily studied upstream loss, while whether scaling laws predict downstream task performance after pretraining and finetuning remains largely unanswered.

    Source: §1 · p. 1

  • Method

    The authors vary multilingual pretraining data type and size and finetuning dataset size, then measure translation scores and downstream cross-entropy in machine-translation tasks.

    Source: §1 · p. 2

  • Results

    With sufficient alignment, translation scores and downstream cross-entropy improve monotonically with more pretraining, and a log-law predicts translation scores; with low alignment and small finetuning sets, scores may fluctuate or worsen while cross-entropy improves monotonically.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Assessing pretraining data for translation should use downstream translation scores, since cross-entropy may improve even when translation scores do not.

    Source: §1 · p. 2

    The alignment measure is a controlled translation-specific score based on balanced language overlap, and the authors note that a more linguistic approach could better explain alignment.

    Source: §3.2 · p. 4, §6 · p. 11

Abstract

from arXiv · show

Scaling laws provide important insights that can guide the design of large language models (LLMs). Existing work has primarily focused on studying scaling laws for pretraining (upstream) loss. However, in transfer learning settings, in which LLMs are pretrained on an unsupervised dataset and then finetuned on a downstream task, we often also care about the downstream performance. In this work, we study the scaling behavior in a transfer learning setting, where LLMs are finetuned for machine translation tasks. Specifically, we investigate how the choice of the pretraining data and its size affect downstream performance (translation quality) as judged by: downstream cross-entropy and translation quality metrics such as BLEU and COMET scores. Our experiments indicate that the size of the finetuning dataset and the distribution alignment between the pretraining and downstream data significantly influence the scaling behavior. With sufficient alignment, both downstream cross-entropy and translation quality scores improve monotonically with more pretraining data. In such cases, we show that it is possible to predict the downstream translation quality metrics with good accuracy using a log-law. However, there are cases where moderate misalignment causes the downstream translation scores to fluctuate or get worse with more pretraining, whereas downstream cross-entropy monotonically improves. By analyzing these, we provide new practical insights for choosing appropriate pretraining data.

1 INTRODUCTION

The paper studies whether scaling laws can predict downstream performance after LLM pretraining and finetuning, focusing on machine translation. It finds that alignment and finetuning data size shape scaling, with translation scores predictable by a log-law in favorable cases but potentially non-monotonic under misalignment.

1 INTRODUCTION

  • The study addresses a largely unanswered question: whether scaling laws can predict downstream task performance after unsupervised pretraining and task-specific finetuning.Prior scaling-law work primarily examined upstream perplexity or cross-entropy.Source: §1 · p. 1
  • The scaling relationship depends on both pretraining–task alignment and finetuning dataset size, so translation quality may gain little from more pretraining when the finetuning set is already large.The paper argues that pretraining data for translation should be assessed using downstream translation scores, not cross-entropy alone.Source: §1 · p. 2
  • The experiments vary multilingual pretraining data type and size, then finetune for machine translation and evaluate BLEU, ROUGE, COMET, and downstream cross-entropy.The authors describe downstream task performance as task-related metrics such as translation scores, distinct from next-token prediction metrics.Source: §1 · pp. 1–2
  • For aligned settings, the paper proposes a log scaling law to predict BLEU, ROUGE, and COMET, and a power law for downstream cross-entropy as pretraining data scales.The authors also report that the log-law generalizes beyond translation to SuperGLUE tasks.Source: §1 · p. 2
  • With sufficient pretraining–task alignment, translation scores and downstream cross-entropy improve monotonically as pretraining data grows; with low alignment and small finetuning sets, scores can fluctuate or worsen.In the latter setting, downstream cross-entropy still improves monotonically according to a power law.Source: §1 · p. 2

2 RELATED WORK

Prior scaling-law research largely studied upstream loss, while transfer-learning work considered finetuning-data scaling. This paper instead examines downstream performance as pretraining data scales, alongside related work on pretraining value and transferability metrics.

2 RELATED WORK

  • Most scaling-law research focused on upstream cross-entropy or perplexity as pretraining data, model size, or training duration increased [Kaplan et al., 2020; Hoffmann et al., 2022].Other work addressed translation models, theoretical foundations, and vision models.Source: §2 · p. 3
  • Hernandez et al. (2021) studied cross-entropy as the finetuning dataset grows, whereas this paper studies downstream metrics as pretraining data scales, an out-of-distribution setting.The related work identifies Sun et al. (2017) as the only known prior scaling-law study of downstream performance versus pretraining size, using smaller vision models.Source: §2 · p. 3
  • Transferability metrics rank source models or pretraining datasets for a target task, but are not designed to predict performance as pretraining data size scales.The paper notes that pretraining may fail to improve or may hurt downstream performance, including in its machine-translation experiments.Source: §2 · p. 3
  • Prior work in vision found that pretraining can have no effect or hurt downstream performance; this paper reports similar cases in machine translation when data are misaligned or finetuning sets are large.Related machine-translation studies also support the importance of aligned pretraining data.Source: §2 · p. 3

3 SCALING LAWS FOR TRANSFER LEARNING

This section proposes scaling laws for translation quality and downstream cross-entropy, defines a translation-specific alignment score, and describes when the laws can guide pretraining-data valuation.

3.1 A SCALING LAW FOR TRANSLATION QUALITY METRICS

  • Translation scores scale closer to a log-law than a power-law, with fitted coefficients depending on alignment and finetuning size; the law also applies to several other task types.The additional tasks listed are question answering, reasoning, reading comprehension, and textual entailment.Source: §3.1 · pp. 3–4

3.2 TRANSLATION ALIGNMENT SCORE

  • The Translation Alignment Score measures balanced language overlap between pretraining data and a translation task, using the proportions of source and destination languages in the mixture.For en-to-fr, a 50% en and 50% fr mixture scores 1, while 100% en scores 0.7.Source: §3.2 · p. 4

3.3 IS CROSS-ENTROPY LOSS ALWAYS A GOOD METRIC?

  • Downstream cross-entropy follows a power-law as pretraining data grows, but BLEU, ROUGE, or COMET may worsen with less-aligned data, making cross-entropy alone a misleading proxy.Figure 1 compares the metrics for alignment scores A = 1 and A = 0.7 on en-to-fr translation.Source: §3.3 · p. 4, Figure 1 · p. 5

3.4 WHEN DO SCALING LAWS FALL SHORT IN TRANSFER LEARNING?

  • With insufficient alignment, translation scores may be non-monotonic even as downstream cross-entropy decreases; aligned, high-performing models show monotonic score gains captured by the proposed law.The law’s fit can serve as a first check of a pretraining dataset’s value for translation.Source: §3.4 · p. 5

3.5 A GUIDE FOR PRETRAINING DATA VALUATION

  • The proposed valuation guide tracks translation scores at checkpoints, fits the scaling law, and uses fit or non-monotonic behavior to assess whether further pretraining may be worthwhile.If the law cannot be fit, it recommends comparing the best finetuned checkpoint with a model trained directly on the downstream task.Source: §3.5 · p. 5

4 EXPERIMENTAL SETUP

The experiments use pretrained encoder-decoder models, multilingual pretraining data, and translation finetuning to study scaling across data sizes and alignment conditions.

EXPERIMENTAL PROCEDURE

  • Models are pretrained without more than one pass over examples, then selected checkpoints are finetuned to collect pretraining-token, translation-score, and downstream-loss pairs for scaling-law analysis.The plots use log-log axes; BLEU is reported in the section, with similar COMET observations in the appendix.Source: §4 · p. 6

MODELS AND DATASETS

  • The main experiments use a 3-billion-parameter T5 encoder-decoder and English, German, French, and Romanian MC4 data, with language-pair mixtures and WMT translation tasks.Additional results cover a 770-million-parameter T5 model and SuperGLUE tasks including question answering, reasoning, reading comprehension, and textual entailment.Source: §4 · p. 6

SCALING-LAW FITTING

  • Scaling-law coefficients are fit with Huber loss and L-BFGS; the first four, least-pretraining data points fit the laws, while remaining points evaluate prediction accuracy.The method selects the best fit from a grid of initializations; additional points can make coefficient optimization more robust to outliers.Source: §4 · pp. 6–7

PLOT CONVENTIONS

  • In the aligned A = 1 plots, markers show experimental results, fitted curves vary by finetuning-set size, and horizontal lines give directly trained non-pretrained baselines.The finetuning dataset sizes increase from dotted to dashed to solid curves.Source: §4 · p. 7

5 RESULTS AND ANALYSIS

With well-aligned pretraining data, downstream translation quality and cross-entropy improve as pretraining data grows, and scaling laws fit these trends. Misalignment can break BLEU scaling even when cross-entropy continues to scale, so cross-entropy alone may not reflect translation quality or data alignment.

5 RESULTS AND ANALYSIS

  • With aligned pretraining data, BLEU and downstream cross-entropy improve monotonically as pretraining data grows; these scaling laws also capture COMET trends.The reported fits work across multiple finetuning-data sizes, but pretraining contributes less when the finetuning dataset is large enough.Source: §5 · pp. 7, 8, 10
  • With less-aligned pretraining, BLEU scaling can break for en-fr even as cross-entropy follows its scaling law; en-de BLEU scaling still fits across the tested datasets.For en-fr, the break appears for some pretraining datasets, including 100% English, German, or Romanian; French-only pretraining does not show the same break.Source: §5 · pp. 8–9
  • For both en-de and en-fr, the authors hypothesize that German or French pretraining helps more than English pretraining because translation generates text in the target language.They leave this explanation for future investigation.Source: §5 · p. 9
  • Downstream cross-entropy alone cannot reliably indicate pretraining alignment or BLEU: the metrics correlate for en-de but can vary independently for en-fr.The en-fr results include cases where BLEU and cross-entropy both increase, contrary to the exponential relation reported by Gordon et al. (2021).Source: §5 · pp. 9–10, Figure 5 · p. 10
  • In the experiments, BLEU’s non-monotonic behavior comes from n-gram precision, not the brevity penalty, which equals 1.BLEU combines precision across n-grams 1 through 4 using a geometric mean.Source: §5 · p. 10

5.1 OTHER TASKS

  • The proposed log scaling law also captures metric scaling as pretraining data grows on SuperGLUE question answering, reasoning, reading comprehension, and textual entailment tasks.Source: §5.1 · p. 10

6 DISCUSSION AND CONCLUSION

The study finds that downstream scaling in machine translation depends on pretraining–task alignment and finetuning-data size, not just pretraining-data growth. Translation scores can be predicted with a log-law under sufficient alignment, but may fluctuate even as downstream cross-entropy improves monotonically.

6 DISCUSSION AND CONCLUSION

  • With sufficient pretraining–task alignment, translation quality scores can be predicted accurately by a log scaling law, while less alignment can produce unpredictable fluctuations.Downstream cross-entropy improves monotonically even in cases where translation scores fluctuate; sufficiently large finetuning datasets leave pretraining with little to no value.Source: §6 · p. 10
  • The scaling behavior depends significantly on pretraining–downstream alignment and finetuning dataset size.The paper argues that downstream performance metrics should be studied rather than decisions being made solely from upstream or downstream cross-entropy.Source: §6 · p. 10
  • When the finetuning dataset is sufficiently large, pretraining has little to no value for downstream performance.This is one of the paper’s reported conditions affecting the practical value of additional pretraining.Source: §6 · p. 10
  • Translation-score scaling laws fit empirical data well and predict scores at scale when alignment is sufficient, but some scores do not scale monotonically.The authors note that a more linguistic approach to translation alignment could provide better understanding.Source: §6 · p. 11
  • The experiments use publicly available datasets and models, with versions, training procedures, and pretraining and finetuning hyperparameters specified.The cited reproducibility statement locates dataset and model versions in Section 4 and Appendix A.Source: §6 · p. 11

A ADDITIONAL EXPERIMENTAL DETAILS

The appendix reports training time for T5-3B experiments and notes that further compute specifications were withheld for anonymity.

A ADDITIONAL EXPERIMENTAL DETAILS

  • Pretraining T5-3B for 1M steps takes 15–20 hours, and finetuning takes 5–7 hours on an 8x8 TPU.The authors say further compute specifications cannot be provided for anonymity and will be added upon acceptance.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 16
  • The reported T5-3B timings cover 1M pretraining steps and the subsequent finetuning stage.Pretraining takes 15–20 hours and finetuning takes 5–7 hours.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 16
  • The experiments run on an 8x8 TPU, but additional compute specifications are withheld for anonymity.The authors state they will add those details upon acceptance.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 16

A.1 MODEL ARCHITECTURES

The appendix points to architecture tables for T5-3B and T5-770M and describes how the scaling-law coefficients are optimized using Huber loss and L-BFGS.

A.1 MODEL ARCHITECTURES

A.2 OPTIMIZING THE SCALING LAW COEFFICIENTS

  • Huber loss is used to reduce overfitting to outliers when optimizing scaling-law coefficients.The appendix says Huber loss suppresses the effect of outlier data points in the optimization.Source: §A.2 · p. 16
  • The Huber-loss parameter is δ = 0.1 for the BLEU score law and δ = 1e −3 for the downstream cross-entropy law.Source: §A.2 · p. 16
  • L-BFGS optimizes the COMET/BLEU score law and downstream cross-entropy law against their respective observed values.For the score law, Dpi is the pretraining dataset size and fi is the COMET/BLEU score; for cross-entropy, Dpi is the pretraining dataset size and Li is downstream cross-entropy.Source: §A.2 · pp. 16–17

B SUPERGLUE EXPERIMENTS

The SuperGLUE appendix reports that the translation-score scaling law also captures scaling on several task types, using T5-3B pretrained on en-MC4.

B SUPERGLUE EXPERIMENTS

  • The SuperGLUE experiments use T5-3B pretrained on en-MC4 and finetune for 300 steps with batch size 128 and sequence length 512.A constant learning rate is selected from {0.001, 0.005, 0.01, 0.05, 0.1, 0.5}.Source: B SUPERGLUE EXPERIMENTS · p. 17
  • The same scaling law used for translation scores captures scaling for BoolQ and MultiRC question answering, COPA reasoning, ReCoRD reading comprehension, and RTE textual entailment.Source: B SUPERGLUE EXPERIMENTS · p. 17

C ADDITIONAL EXPERIMENTAL RESULTS

C.1 RESULTS WITH COMET SCORES

With sufficient pretraining–downstream alignment, the proposed law describes COMET scaling; with insufficient alignment, COMET scores fluctuate and can worsen as pretraining grows.

C.1 RESULTS WITH COMET SCORES

  • With sufficient alignment, the proposed law describes COMET scaling; with insufficient alignment, COMET scores fluctuate and sometimes worsen as pretraining increases.The comparison covers balanced pretraining data and experiments using 100% en-MC4.Source: §C.1 · p. 17

C.2 RESULTS ON T5-770M

The T5-770M experiments generally follow the T5-3B trends: the proposed laws fit aligned data, while en-MC4 pretraining for en-fr translation shows a break consistent with misalignment.

C.2 RESULTS ON T5-770M

  • For T5-770M, the proposed laws describe downstream behavior well when pretraining and downstream data are aligned.The appendix reports trends similar to those for T5-3B.Source: §C.2 · p. 17
  • With 100% en-MC4 pretraining, en-fr translation shows a scaling-law break, suggesting the same misalignment observed for T5-3B.Source: §C.2 · p. 17

C.3 OPTIMIZED COEFFICIENTS AND PREDICTION ERRORS OF THE SCALING LAWS

This section reports fitted scaling-law coefficients and prediction errors, alongside plots of metric behavior across pretraining-data sizes and task settings.

C.3 OPTIMIZED COEFFICIENTS AND PREDICTION ERRORS OF THE SCALING LAWS

  • Tables 3–6 report fitted coefficients for the BLEU and downstream cross-entropy laws, together with prediction errors.Source: §C.3 · p. 17, Table 3 · p. 19, Table 5 · p. 21
  • Figure 6 plots SuperGLUE scores against pretraining dataset size; markers show experiments and black horizontal curves show direct-training baselines.Source: Figure 6 · p. 18, §C.3 · p. 18
  • Figure 7 plots COMET scaling-law fits for balanced and 100% en-MC4 pretraining, with experimental markers and direct-training baselines.Source: Figure 7 · p. 19
  • Huber loss uses δ = 0.1 for BLEU laws and δ = 10−5 for downstream cross-entropy laws.Source: §C.3 · pp. 19, 21, 22, Figure 2 · p. 20
  • The WMT plots pair BLEU curves with cross-entropy validation loss across en-de and en-fr tasks and multiple finetuning dataset sizes.Source: §C.3 · pp. 20–21
Loading 2402.04177v3…