PaperlayerView exact revision on arXiv

Source-linked AI summary

Establishing Task Scaling Laws via Compute-Efficient Model Ladders

Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A. Smith, Dirk Groeneveld, Pang Wei Koh, Jesse Dodge, Hannaneh Hajishirzi

arXiv:2412.04403v2cs.CLcs.AI

TL;DR

Predicting individual downstream-task performance efficiently remains an open problem, despite its relevance to LM pretraining experimentation. The paper uses small ladder models and a two-step loss-to-accuracy approach to predict overtrained target models. Predictions are within 2 absolute points on four ranked-classification tasks, while task predictability varies with checkpoint variance.

  • Problem

    Efficient prediction of individual downstream-task performance remains open; earlier approaches predict average error across tasks or one task and use at least 5% of target-model compute.

    Source: §1 · p. 2

  • Method

    Small ladder models fit a two-step approach that predicts task-specific loss from model and data size, then predicts ranked-classification accuracy from that loss.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    Task-specific loss yields accuracy predictions within 2 absolute points on MMLU, HellaSwag, PIQA, and Social IQa for both targets; higher-error tasks also show higher checkpoint variance.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    The fixed ladder models cost 1% of target-model compute and support recommendations for choosing design choices for tasks and developing downstream scaling laws for new models.

    Source: §1 · p. 2, §8 · p. 10

    Multiple-choice accuracy is harder to extrapolate from smaller ladder models, and validation on 70B-parameter-and-beyond models remains future work.

    Source: §8 · p. 10

Abstract

from arXiv · show

We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks.

1 Introduction

Scaling laws can make LM pretraining experimentation more resource-efficient, but standard language-modeling loss does not answer how a model will perform on individual downstream tasks. This work predicts ranked-classification accuracy for individual tasks using a two-step approach and a small compute budget.

1 Introduction

  • Predicting individual downstream-task performance efficiently remains open: prior work predicts average error across tasks or one task, using at least 5% of target-model compute and compute-optimal-model testbeds.Downstream tasks also serve as development benchmarks for training recipes and data mixtures.Source: §1 · p. 2
  • A two-step method uses model size and training data size to predict an intermediate loss, then predicts task accuracy from that loss.It targets multiple-choice problems in ranked-classification format and evaluates 7B and 13B models.Source: §1 · p. 2
  • With task-specific loss as the intermediate feature, predictions are within 2 absolute accuracy points on four tasks, using ladder models that cost 1% of combined target-model compute.Across both targets and all tasks, average absolute error is 4 points.Source: §1 · p. 2

2 Setup

The study evaluates predictions for two overtrained OLMo 2 target models using fixed-size ladder models that span model and data scales. It measures task loss and ranked-classification accuracy across selected OLMES tasks.

2 Setup

  • The targets are OLMo 2 models trained from scratch on the same data mixture: 7B-4T and 13B-5T, evaluated before stage 2 annealing.The 7B target has 3.95T total training tokens after an additional 50B-token linear-decay run to account for incomplete stage 1 training.Source: §2 · p. 2, §2.1 · p. 3

2.1 Ladder models

  • Sixteen fixed-size ladder models span four sizes from 190M to 1.3B parameters and four training-data multiples, from 1xC to 10xC.Ladder models share the target models’ architecture and data mix; their compute is less than 1% of the combined target-model compute.Source: §2.1 · pp. 2–3, Table 1 · p. 3

2.2 Intermediate features and accuracy

  • Task loss is the correct answer sequence’s negative log-likelihood per byte, while ranked classification selects the answer with the lowest task loss.The study focuses on ranked classification rather than multiple-choice answer-code prediction.Source: §2.2 · pp. 3–4

2.3 Task selection

  • The evaluation uses eight selected OLMES tasks in the 5-shot setting, excluding BoolQ and Winogrande because their task loss and accuracy are noisier.The test set is used where possible, with validation data otherwise.Source: §2.3 · p. 4

3 Method

The method predicts task performance in two stages: first model- and data-scale-dependent loss, then accuracy from that loss. It fits task-specific functions using ladder-model observations and chains the predictions for target models.

3 Method

  • The method fits separate task-specific functions: a power form for loss versus model and data scale, and a sigmoid for accuracy versus intermediate loss.Step 1 uses final ladder checkpoints; step 2 also uses intermediate checkpoints, with moving averages and early-checkpoint exclusion to smooth noisy measurements.Source: §3 · p. 5, §3.1 · p. 5, §3.2 · p. 6

3.1 Step 1: Use (N, D) to predict the intermediate feature

  • For step 1, a power function models task-specific loss from model size and training tokens; the fit has 0.2%–1.2% average relative error and target loss prediction is within 3% on four tasks.The four tasks are MMLU, HellaSwag, PIQA, and OpenBookQA; across tasks, average relative prediction error is 5.2% for 7B-4T and 6.6% for 13B-5T.Source: §3.1 · pp. 5–6

3.2 Step 2: Use intermediate feature to predict accuracy

  • For step 2, a sigmoid maps intermediate loss to accuracy, reflecting random accuracy at high loss and accuracy that approaches 100% at low loss.The fit uses loss and accuracy from final and intermediate ladder checkpoints; average relative fitting error ranges from 0.4% to 2.6%, with average relative prediction errors of 3.7% for 7B-4T and 3.5% for 13B-5T.Source: §3.2 · pp. 6–7

3.3 Chaining the two steps

  • Chaining the two stages gives average absolute errors of 3.8 points for 7B-4T and 4.2 points for 13B-5T; predicting accuracy directly from model and data size has higher average errors, above 30 points.The chained predictions use task loss as the intermediate feature.Source: §3.3 · p. 7

4 Results

Task loss as the intermediate feature predicts ranked-classification accuracy within 2 absolute points on four tasks. The results also report weaker predictions on ARC-C and ARC-E and compare alternatives to task loss.

4 Results

  • Within 2 absolute accuracy points on MMLU, HellaSwag, PIQA, and Social IQa; ARC-C and ARC-E show overestimated task loss followed by underestimated accuracy.The paper reports that these latter tasks also have higher checkpoint variance.Source: §4 · p. 7
  • TaskCE as the intermediate feature yields higher overall prediction errors on several tasks, including MMLU and HellaSwag.Full TaskCE results are referenced in §C.2.Source: §4 · p. 7
  • C4-en validation LM loss predicts accuracy well for several tasks in this setup, with possible domain overlap offered as one explanation.The paper discusses pros and cons of LM loss in §6.Source: §4 · p. 7

5 Analysis: Task predictability with the ladder

Checkpoint variance provides a signal about task predictability: higher loss variance is associated with higher accuracy-prediction error, while task difficulty, noisy targets, and random-chance accuracy can also challenge ladder-based predictions.

5 Analysis: Task predictability with the ladder

  • High variance in intermediate loss weakens function fitting, high variance in task accuracy spreads prediction targets, and difficult tasks may leave smaller ladders at random-chance accuracy.These are three anticipated sources of prediction failure.Source: §5 · pp. 7–8
  • OpenBookQA metrics appear noisier than MMLU, and higher intermediate-checkpoint noise is associated with higher prediction error.Figure 5 illustrates SD10 for these two tasks; Figure 6 shows results across all tasks.Source: Figure 5 · p. 8, §5 · p. 8
  • For 1B-10xC, relative SDn compares variability between tasks as the standard deviation of the final n training checkpoints divided by their mean, multiplied by 100%.Task-loss and task-accuracy noise is measured using intermediate checkpoints of the largest ladder model, 1B-1xC.Source: §5 · p. 8
  • Loss SD10 correlates with accuracy-prediction error for both targets: r = 0.821, p = 0.004 for 7B-4T and r = 0.855, p = 0.002 for 13B-5T.Tasks with above-average SD10 tend to have higher prediction errors; SD10 can be reported alongside predictions to flag benchmarks that may be harder to predict.Source: Table 4 · p. 8, §5 · p. 9

6 Discussion and Recommendations

The discussion recommends checking variance and feature-to-accuracy mappings when choosing task scaling laws, and reports that the same ladder also estimates performance for a larger target model.

6 Discussion and Recommendations

  • For new tasks, assess variance on the largest ladder model affordable, since high target variance generally corresponds to lower predictability.High variance in an intermediate feature may be mitigated by trying alternative features.Source: §6 · p. 9
  • A useful intermediate feature must be predictable and map well to accuracy; noisy mappings can compound errors across the two prediction steps.ARC-C and OpenBookQA have high variance in both task loss and accuracy, while HellaSwag and PIQA show the least variance and are easiest to predict across design choices.Source: §6 · p. 9
  • OpenBookQA's 500 instances accompany high prediction error for most design choices, while MMLU and HellaSwag have sample sizes five times larger than the other tasks.The discussion recommends validating a single design choice on tasks with many instances.Source: §6 · p. 9
  • Using the same ladder models for 32B-6T yields higher absolute errors but the same trend: lower-variance tasks have relatively lower prediction errors.The ladder uses 0.45% of the target model compute in this evaluation.Source: §6 · p. 9

7 Related Work

Prior scaling-law work addressed language-modeling loss or downstream performance at aggregate or task-specific levels, while this paper focuses on predicting individual downstream-task performance using fixed ladders.

7 Related Work

8 Conclusion and Future Work

The paper develops fixed model ladders and task scaling laws for overtrained language models, then examines variance and design choices for prediction. In a preliminary multiple-choice case study, fitting later training checkpoints yields low end-to-end MMLU accuracy error, but requires substantial target-model compute.

8 Conclusion and Future Work

  • The paper predicts individual downstream-task performance for 7B-4T and 13B-5T using fixed ladder models that cost 1% of target compute.It also analyzes task variance and offers recommendations for design choices.Source: §8 · p. 10
  • Future work includes improving predictability, extending the method to MC tasks, and validating it on models with 70B parameters and beyond.The authors note that MC accuracy is harder to extrapolate from smaller ladder models.Source: §8 · p. 10

A Fitted Parameters

B.1 Additional variance analysis results

  • The reported variance results connect checkpoint noise across tasks with the difficulty of predicting target-model performance.Figure 6 shows intermediate checkpoints across tasks; Table 4 relates checkpoint noise to target prediction difficulty.Source: §B.1 · p. 14

B.2 Predicting task accuracy under the MC format

  • Multiple-choice accuracy in smaller models can remain random until a size threshold, producing training curves that do not follow a sigmoidal shape.The rapid-growth phase occurs around 70k steps for 7B-4T and 20k steps for 13B-5T, across tasks.Source: §B.2 · pp. 14–15
  • Chaining the two prediction steps gives MMLU accuracy absolute errors of 0.3 points for 7B-4T and 0.4 points for 13B-5T in the multiple-choice case study.The step-2 mapping is fitted using checkpoints from the third phase of each model's accuracy curve.Source: §B.2 · p. 15
  • MC fitting uses early target-model checkpoints, consuming about 50% of 7B-4T training compute and 25% of 13B-5T compute.For 7B-4T, the fitting data extend to 450k of 928k steps; for 13B-5T, to 150k of 596k steps.Source: §B.2 · p. 15

B.3 Compute requirements for predicting each task

Prediction error generally falls as ladder model size or total compute increases, while extending training duration has a less pronounced effect. With a fixed ladder budget, increasing model size is more likely to improve predictions than training models longer.

B.3 Compute requirements for predicting each task

  • Some tasks have high noise between adjacent checkpoints, and a sigmoid cannot fit the full task-accuracy versus task-loss data.The sigmoid is fitted only on selected intermediate-checkpoint ranges.Source: Figure 6 · p. 16, Figure 7 · p. 17
  • Prediction error trends downward for most tasks as the largest model included in the ladder increases toward the target model.The comparison progressively includes models up to 760M parameters.Source: §B.3 · p. 16, Figure 9 · p. 18
  • With 3.2% of target compute, the full ladder gets 1.3% error on MMLU; at 0.1%, error rises to ∼12%.Prediction error generally declines as ladder FLOPs increase, though ARC-C, ARC-E, and OBQA show greater variation.Source: §B.3 · p. 16, Figure 8 · p. 17
  • Including models trained to higher Chinchilla multipliers yields only a slight downward error trend, noticeable on MMLU, and less than varying model size.The 7B-4T target is trained to approximately 28xC.Source: §B.3 · p. 17, Figure 10 · p. 18
  • Given a fixed ladder budget, increasing model size is more likely to improve prediction error than training for longer.The section recommends exploring both dimensions to reduce overall ladder cost.Source: §B.3 · pp. 16–17

C.1 Compute-FLOPs C instead of (N, D) for task loss prediction

The section tests compute-FLOPs as a direct input for task-loss prediction in the overtrained regime. This proxy generally fits and predicts worse than using model size and training data size, so the authors exclude it from their main method.

C.1 Compute-FLOPs C instead of (N, D) for task loss prediction

  • The tested alternative predicts task loss directly from compute-FLOPs C using a fitted power function.A, α, and E are the parameters to fit.Source: §C.1 · p. 18
  • Compute-FLOPs produce higher relative fitting error on all tasks and generally worse target-model task-loss predictions than using (N, D).ARC-Challenge and ARC-Easy are exceptions to the general target-model prediction pattern.Source: §C.1 · p. 18, Figure 11 · p. 19
  • Compute-FLOPs cannot distinguish compute-optimal from overtrained models with equal FLOPs but different losses, so the main method does not use this input.The authors note that the compute-based power function has fewer free parameters than the function using (N, D).Source: §C.1 · p. 18

C.2 Task cross-entropy as the intermediate feature

The section compares TaskCE and general language-modeling loss as intermediate features, then evaluates a single-step alternative. TaskCE can better predict ranked-classification accuracy, while the single-step approach is less robust than the two-step method.

C.2 Task cross-entropy as the intermediate feature

  • 2.75% vs. 3.6% average step-2 prediction error across 7B and 13B models favors TaskCE over correct-answer loss for ranked-classification accuracy.The TaskCE fit is particularly good on ARC-Challenge and OpenBookQA, but substantially worse on HellaSwag, where observations remain in the log-sigmoid’s linear regime.Source: §C.2 · p. 20
  • TaskCE fitting errors are more systematically skewed than the comparison fit, and its narrow value range makes accuracy predictions sensitive to small changes in TaskCE.The passage notes substantial TaskCE errors for MMLU and ARC-Challenge of 4.2%–4.8%, and that less than 0.05 nats can matter on PIQA.Source: §C.2 · p. 20

C.3 General language modeling loss as the intermediate feature

  • Using C4 loss instead of task loss yields higher chained prediction error on five tasks, but lower error on ARC-Challenge and ARC-Easy.The fitted step-1 function underestimates C4 loss by 2.0% for 7B-4T and 3.8% for 13B-5T.Source: §C.3 · pp. 21–22

C.5 Combining step 1 and 2 into a single step

  • The single-step fit is harder to estimate because it has more free parameters, is not convex, and cannot use intermediate-checkpoint data used in step 2.It removes the a priori definition of a specific intermediate feature while preserving the function’s representational power.Source: §C.5 · p. 23
  • The single-step function has higher prediction error on four of eight tasks and is less robust than the two-step approach.On CommonsenseQA, the fitted function does not vary with model size, indicating degenerate fitting.Source: §C.5 · p. 23

D Scaling law predictions for 32B-6T

The ladder is tested on the larger OLMo 2 32B-6T model. Prediction errors are higher for high-variance tasks, while several lower-variance tasks remain predictable within three accuracy points.

D Scaling law predictions for 32B-6T

  • Within 3 absolute-error points, the ladder predicts HellaSwag, PIQA, and SocialIQA accuracy for 32B-6T; high-variance ARC-C and ARC-E show higher errors.Overall prediction error is higher for this larger target, but the pattern of higher errors on high-variance tasks resembles that for the smaller targets.Source: Figure 18 · p. 24, D Scaling law predictions for 32B-6T · p. 25
  • 0.45% of 32B-6T compute is the cost of the ladder models used for this larger-model prediction.The target model was trained to 6T tokens.Source: D Scaling law predictions for 32B-6T · p. 24

Related papers

Loading 2412.04403v2…