Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments

• The study evaluates finetuning-time scaling laws across 9 language understanding tasks, using models ranging from 10K to roughly 100M parameters.Source: §1 · p. 2
• Scaling laws emerge only in some finetuning tasks, with SQuAD showing a stronger fit than MRPC.Figure 1 reports R2 = 0.998 for SQuAD 1.1 and R2 = 0.763 for MRPC.Source: Figure 1 · p. 1, §1 · p. 2
• Some tasks require a minimum model size; on MRPC, only models with at least four hidden layers performed significantly better than chance.Source: §1 · p. 2
• Careful hyperparameter tuning is crucial for exposing precise finetuning scaling laws and can substantially change larger-scale predictions.Without tuning, Figure 1 reports a lower goodness-of-fit of R2 = 0.83.Source: Figure 1 · p. 1, §1 · p. 2
• When small-scale fits are strong, scaling laws can predict larger-model performance and support model selection, including the MLM-versus-PMI case study.The reported criterion is high goodness-of-fit, R2 > 0.95, at smaller scales.Source: §1 · p. 2
• Scaling-law analysis offers potential development benefits but requires hyperparameter control and convergence checks whose overhead may offset computational savings.Source: §1 · pp. 1–2

2.1 Experimental setup

• The experiments use encoder-only transformer models scaled from as few as 10K parameters to approximately 85M by increasing layers while controlling aspect ratio.Source: §2.1 · pp. 2–3
• The study compares MLM and PMI pretraining objectives to test whether small-scale experiments can identify the better objective for downstream tasks.Source: §2.1 · p. 3
• Finetuning covers nine language understanding datasets, including SQuAD, MNLI, QNLI, SST-2, RACE, CoLA, SQuAD 2.0, MRPC, and BoolQ.Source: §2.1 · p. 3

2.2 Evaluation

• Goodness-of-fit is estimated by fitting a power law through least-squares regression in log-log space and reporting R2 across scales and runs.Source: §2.2 · p. 3
• A hierarchical bootstrap with B = 1000 resamples estimates confidence intervals for the fitted slope and predicted points.Source: §2.2 · p. 3
• Predictive power is measured with Mean-Relative-Error between predicted and true performance at larger scales.Source: §2.2 · pp. 3–4

3 Pretraining

• Pretraining evaluation loss follows power laws for both MLM and PMI across both aspect ratios, with R2 exceeding 0.99 in every case.Source: §3 · p. 4
• The AR 64 slope is slightly better than AR 32 for both objectives, but overlapping confidence intervals prevent confident prediction of a performance difference at larger scales.Source: §3 · p. 4
• The largest 85M-parameter MLM model performs slightly worse than its power-law prediction, possibly indicating under-training.Source: §3 · p. 4

3.1 Debugging convergence with scaling laws

• Scaling-law fits can reveal convergence failures when early-stopped models have losses above the value predicted from other scales.Increasing patience produced further loss reduction and convergence at the predicted value.Source: §3.1 · p. 4
• Convergence is sensitive to early-stopping settings such as patience and minimum decrease, so defining convergence precisely matters for reproducible scaling-law research.Source: §3.1 · p. 4

4.1 Downstream tasks are not born equal

• Reliable scaling laws appear only on some tasks: SQuAD 1.1, MNLI, and QNLI have R2 > 0.96, whereas BoolQ is not monotonic with parameter count.Tasks with fewer than 10K training examples generally have R2 < 0.85.Source: §4.1 · pp. 4–5
• SQuAD 2.0’s fit improves from R2 = 0.797 to 0.915 when evaluated with the SQuAD 1.1 metric on answerable questions, implicating metric and task formulation differences.The authors conjecture that detecting non-answerable questions and using Best-F1 diverge from the language-modeling objective.Source: §4.1 · p. 5
• R2 improves substantially for SQuAD 2.0, RACE, and MRPC when the smallest model is excluded, indicating a critical minimum model size for some tasks.On MRPC, only models with at least four hidden layers performed significantly better than chance.Source: §4.1 · p. 6

4.2 Hyperparameter tuning

• Per-scale hyperparameter tuning produces cleaner power-law fits and prevents smaller models’ suboptimal settings from steepening slopes and overshooting large-scale predictions.The paper emphasizes that hyperparameters optimized for one scale cannot be assumed optimal at another.Source: §4.2 · p. 6

4.3 Predictive power

• Models with 1–6 layers predict 7–8-layer performance within MRE ≤2.5% on most tasks; SQuAD 1.1 reaches a 0.6% relative F1 difference.Low R2 can coexist with low MRE when scores vary little or remain near a constant baseline, as in SST-2 and BoolQ.Source: §4.3 · pp. 6–7
• For high-fit tasks, 1–8-layer fits predict BERT-Base performance within less than 3% difference, sometimes nearly 0.1%; excluding single-layer models improves prediction for critical-size tasks.The largest small model has 14x fewer parameters than BERT-Base.Source: §4.3 · p. 7

4.4 Case study: MLM vs. PMI objectives

• In the MLM-versus-PMI case study, high-R2 fits correctly predict the sign of the performance gap for model selection, with especially accurate gaps on SQuAD 1.1, QNLI, and MNLI.The paper leaves extrapolation over multiple orders of magnitude for future work.Source: §4.4 · p. 7
• Even after hyperparameter tuning and finetuning across scales and tasks, the study observes a 2.5x FLOP reduction versus training the larger model, while smaller models require more epochs and convergence monitoring.The authors estimate 5x savings in the parameter-count comparison and expect runtime savings to be greater because models can be trained in parallel.Source: §4.4 · pp. 7–8

Scaling laws in transformers

• Transformer scaling-law research spans language modeling, vision, multimodal learning, translation, and theoretical explanations.The cited literature connects scaling behavior to model shape, data–compute trade-offs, intrinsic dimension, kernel spectra, and nearest-neighbor analogies.Source: §5 · p. 8

Harnessing scaling laws for model design

• Small-scale extrapolation has been used to estimate larger-model generalization and optimize training data, but prior work mainly focused on pretraining.This paper extends that line by testing predictive power on downstream language tasks.Source: §5 · p. 8
• Neural Architecture Search also extrapolates architectures from small experiments, yet its resulting designs do not always scale well.Source: §5 · p. 8

• Scaling laws are effective predictive and diagnostic tools when the fit is good, with R2 ≥0.95 serving as a useful indicator.Source: §6 · p. 8
• Independent hyperparameter tuning at every scale is crucial for model selection and for revealing scaling laws.Source: §6 · p. 8
• Pretraining models to convergence is important for observing downstream scaling behavior in transformer models.Source: §6 · p. 8

• Scaling laws emerge for some downstream tasks, potentially depending on proximity to the pretraining objective and the amount of fine-tuning data.Source: §7 · p. 8
• Scaling laws can help debug convergence in large-model training and predict larger-model behavior when they emerge at small scale.Source: §7 · p. 9

• The study lacks empirical evidence for significantly larger models and does not evaluate decoder-based models.Source: §8 · p. 9
• Its nine downstream tasks use English-only datasets and exclude generation capabilities.Source: §8 · p. 9
• The rule for judging prediction reliability does not explain why scaling laws fail on all downstream tasks.Source: §8 · p. 9

A.1 Experimental details

• Models were trained with Transformers, Datasets, PyTorch, and tracked using Comet.ML across standardized pretraining and finetuning datasets.MNLI used 25% of its training samples, and no mid-training was performed.Source: §A.1 · p. 15
• Five finetuning seeds were used per model-task pair, whereas pretraining used one seed because it was more expensive.Source: §A.1 · p. 15

A.2 Hyperparameter tuning

• Finetuning experiments varied learning rate and total epochs after fixing the scheduler, dropout, warm-up, and batch size based on preliminary comparisons.The selected configuration used a CONSTANT scheduler, 10% dropout and warm-up, and batch size 64.Source: §A.2 · p. 15
• The experimental model configurations were documented with parameter counts, including BERT-Base as the largest reference configuration.Source: Table 6 · p. 15
• Hierarchical bootstrap estimates were generated by sampling scales and then points within each sampled scale before fitting power-law lines.Source: §A.2 · p. 15
• Pretraining used batch size 256, learning rate 10^-4, a 500K-step horizon, linear scheduling, 0.1 dropout, and 10K warm-up steps.These settings produced comparable converged results to tuned hyperparameters, although convergence rates differed.Source: §A.2 · p. 15

A.3 Evaluation

• Power-law goodness-of-fit is computed by fitting a least-squares regression in log-log space and evaluating its R2 over all scale-seed observations.Source: §A.3 · pp. 15–16
• Hierarchical bootstrapping produces more conservative confidence intervals by incorporating variance across pretraining scales as well as finetuning seeds.The resulting slope intervals support uncertainty estimates for larger-scale predictions.Source: §A.3 · p. 16

A.4 Model convergence

• Early stopping can falsely suggest convergence when learning-rate decay or stopping thresholds suppress visible improvement before optimization is complete.The BERT-Base loss curve continued improving after long plateaus, indicating under-training and contributing to prediction error.Source: §A.4 · pp. 16–17, Figure 6 · p. 17

A.5 PMI vs. MLM results

• Differences between reported BERT-Base results and prior PMI results were attributed to fewer update steps and the absence of Book Corpus pretraining data.Prior work reported 81.4 Best-F1 on SQuAD 2.0 and 70.1 accuracy on RACE after 1M steps, versus the shorter training used here.Source: §A.5 · p. 17

A.6 Fitting Evaluation Loss

• Evaluation-loss scaling fits can be poor for classification because high-confidence errors inflate log-loss even while the target metric improves.Fitting evaluation loss instead of evaluation accuracy caused an average drop of over 10% in R2.Source: §A.6 · pp. 17–18, Figure 7 · p. 18, Table 7 · p. 18

A.7 SQuAD 2.0

• Using the SQuAD 1.1 objective for SQuAD 2.0 models produced a cleaner scaling law and significantly increased R2, implicating metric choice in the poorer original fit.Source: §A.7 · p. 18, Figure 8 · p. 18