Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families

1 Introduction

• Training and data-processing differences make one scaling law unreliable across families, while family-specific laws require resource-intensive training of multiple model sizes.The paper positions Sloth as an alternative to both approaches.Source: §1 · p. 1
• Sloth fits scaling laws to public benchmark data, using cross-benchmark information to improve prediction and interpretability with fewer parameters.Its skill-based formulation assumes benchmarks reflect low-dimensional capabilities such as reasoning and instruction following.Source: §1 · pp. 1–2
• Sloth models benchmark performance through low-dimensional skills shaped by training tokens and model size, with family-specific compute efficiency estimated from one model per family.The paper identifies accurate prediction across 12 benchmarks and interpretable insights into scaling behavior as empirical contributions.Source: §1 · p. 2

1.1 Related work

• Unlike prior scaling-law work centered on test loss and models trained under particular pretraining settings, Sloth fits existing benchmark data across model families.Prior work also studied latent LLM skills, but the cited discussion notes that some approaches omitted training-set size or family information.Source: §1.1 · pp. 2–3

2.1 Problem Statement

• The objective is to estimate a large family-i model’s score on benchmark j from smaller same-family models, with size s measured in parameters and t in training tokens.Y_ij(s,t) denotes the score, and the target is its central tendency, such as the mean or median.Source: §2.1 · p. 3

2.2 Previous approaches to scaling laws for benchmarks

• Earlier benchmark scaling laws often assumed scores depend on size and tokens only through training FLOPs, c(s,t)=6st.Some formulations also constrained whether model-family parameters could vary.Source: §2.2 · p. 3
• Prior approaches were either restrictive in their compute, family, or activation assumptions, or difficult to estimate when coefficients varied by family.The cited Ruan et al. discussion describes fitting family- and benchmark-specific coefficients with only 3–5 models per family.Source: §2.2 · p. 3
• The paper aims to predict a larger model before training it while using family information, supporting decisions about investing resources in training.It compares adaptations of Owen [2024] and Ruan et al. [2024] as baselines despite their different prediction settings.Source: §2.2 · pp. 3–4

3.1 Model architecture

• Sloth models benchmark correlations through shared low-dimensional skills, rather than modeling each benchmark score directly, to reduce fitted parameters and overfitting.A benchmark-loading matrix represents overlapping or distinct skills measured by each benchmark.Source: §3.1 · p. 4
• Skill levels use a translog production function, with log size, log tokens, and their interaction as input features.The interaction allows size and token effects on skills to depend on one another.Source: §3.1 · p. 4
• Family-specific skill intercepts capture efficiency in converting compute to skills, while skill slopes are shared across families and may depend on size and tokens beyond total FLOPs.The benchmark link can be learned as a monotonic neural network or fixed to a sigmoid.Source: §3.1 · p. 4

3.2 Model fitting

• Model fitting minimizes Huber loss with gradient descent and Adam, while constraining benchmark link functions to be increasing.The optimization is nonconvex, so multiple initializations can be used to seek a better fit.Source: §3.2 · p. 5

3.3 Interpretability and practical considerations post model fitting

• Sloth’s parameters are identifiable up to transformations, so the authors fit without constraints and then rotate the skills into an interpretable configuration.The theoretical identification result is established for the model parameters up to those transformations.Source: §3.3 · p. 5

4.1 Data

• The expanded dataset covers 30 model families: 28 appear on Open LLM Leaderboard v1, 17 on v2, and 15 on both.The dataset also includes instruction-tuned versions of the base models considered.Source: §4.1 · p. 5

4.2 Comparing scaling laws in terms of prediction errors

• Across LLM families, Sloth’s average within-family MAE is similar to or lower than the “Size and Tokens” variant, including recent families LLaMa 3, Qwen 2, and Yi 1.5.The evaluation uses leave-one-out cross-validation by model family; results are qualitatively similar when the two smallest test-family models are observed at training time and when MAPE replaces MAE.Source: §4.2 · pp. 5–7, Figure 1 · p. 6
• With a shared intercept, Sloth is a strong alternative to FLOPs on some benchmarks and sometimes achieves errors similar to fuller Sloth versions.This shows family-specific intercepts are not always needed for good predictions on the selected benchmarks.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Interpreting the latent skills

• Reasoning loads on math, science, logic, and multistep benchmarks, and is more affected by model size than tokens; instruction tuning can increase or decrease it.The Reasoning interpretation is reported as robust across different values of d.Source: Figure 4 · p. 7, §4.3 · p. 8
• Knowledge responds strongly to model size and tokens, while Instruction Following depends on both and rises with instruction tuning.The Instruction Following interpretation is mixed with other skills at d = 2 and does not hold at d = 4.Source: Figure 4 · p. 7, §4.3 · pp. 8–9

4.4 Predicting LLM performance on downstream tasks

• Sloth skills plus task-level regression predict held-out LLaMa 3 70B code-completion and instruction-model emotional-intelligence performance with decent accuracy.Reasoning is the most important skill for coding prediction, while emotional intelligence uses a mixture of Reasoning and Knowledge.Source: §4.4 · p. 9, Figure 6 · p. 9

4.5 Predicting performance behavior with scaled inference compute

• Coupling Sloth with Item Response Theory accurately predicts MATH test-time scaling behavior for LLaMa 3 Instruct, Gemma, and Pythia models.The predictions extend to hypothetical models before committing resources to training them.Source: §4.5 · pp. 9–10, Figure 7 · p. 10

4.6 Compute-optimal scaling of skills

• Under a FLOPs budget and observed-data bounds, the analysis finds that the optimal model-size and token allocation does not depend on family.Table 1 reports instruction-following results; results for the other two skills are in Appendix B.Source: §4.6 · p. 10

• Sloth accurately predicts LLM performance across multiple benchmarks while providing insights into relationships between computational resources and model capabilities.Source: §5 · p. 10
• Benchmark correlations and interpretable, low-dimensional latent skills let Sloth estimate performance across tasks with minimal data from individual model families.Source: §5 · p. 10
• Prediction usually depends on seeing at least one model from the target family, and flexible neural link functions lack extrapolation guarantees for very different target models.Source: §5 · p. 10

A Identifiability of model parameters and interpretability

• The identifiability proof requires fixing an invertible σ and treating the benchmark lower-asymptote parameters γj as known.For MMLU, the paper gives 25% as the lower asymptote, based on the chance score for multiple-choice alternatives.Source: A Identifiability of model parameters… · p. 14
• Under Assumption A.1, equivalent parameterizations have the same intercept and differ by paired invertible transformations of Λ and B; the transformation is orthogonal when Ψ = Id.The assumptions include standardized skills, rank(Λ) = d, and rank(X) = p; Ψ = Id corresponds to uncorrelated skills.Source: A Identifiability of model parameters… · p. 14

A.1 Interpretability

• For interpretability, the authors first transform fitted skills to identity covariance, then apply Geomin rotation to make the loadings interpretable while updating B to preserve outputs.They finally center skill columns and translate the intercept to keep the model invariant.Source: §A.1 · pp. 15–16

B Compute-optimal scaling

• Under a fixed computation budget 6st = c, compute-optimal scaling selects the model configuration that maximizes a specified skill for a given family.The skill is modeled as αik + βk0 log(s) + βk1 log(t) + βk2 log(s) log(t).Source: B Compute-optimal scaling · p. 16
• The optimization reduces to a one-variable function gik(u), whose maximizer is its vertex or an interval endpoint when βk2 > 0.Here u = log(s), v = log(t), and the fixed-budget constraint gives u + v = l.Source: B Compute-optimal scaling · p. 16
• The solution is restricted to ranges of model size and token counts observed in training data, with bounds set from training-data quantiles.Table 2 reports optimal parameter and token allocations across skills for various FLOPs budgets.Source: B Compute-optimal scaling · p. 16, Table 2 · p. 16

C Connections with factor analysis

• Factor analysis models correlated observed variables through fewer latent factors; in Sloth, skills are functions of observable covariates and family-specific intercepts fitted from data.The loadings Λ represent how the latent skills account for observed benchmark variability.Source: C Connections with factor analysis · pp. 16–17

D Motivating the interaction term in Sloth

• For MMLU-PRO and BBH, within-family score slopes vary with token counts in the shown plots, motivating an interaction between log(s) and log(t), though the pattern is not universal.The passage contrasts Pythia's small training dataset and nearly zero hypothetical slopes with Qwen2's more data and higher slopes; Figures 8 and 9 show inputs versus scores.Source: D Motivating the interaction term in… · p. 17, Figure 8 · p. 17, Figure 9 · p. 17

E PCA approach formulation

• The PCA comparison reduces benchmark scores to the first d principal components, predicts them from logFLOPs using linear regressions, then maps predictions back to benchmark coordinates.The regression intercept or both intercept and slope may be family-dependent.Source: E PCA approach formulation · p. 18

F Sloth is parameter-efficient: parameter count analysis

• The parameter-efficiency comparison is attributed to Sloth’s assumed latent-skill structure.The passage specifies the example setting as three skills and twelve benchmarks.Source: F Sloth is parameter-efficient:… · p. 18
• For d = 3 and J = 12, Sloth uses 69 + 3f parameters versus 36 + 12f for the FLOPs baseline and 50 + 12f for Size and Tokens; for moderately large f, it uses fewer parameters than these top-performing baselines.Source: F Sloth is parameter-efficient:… · p. 18
• For f ≥ 4, Sloth uses fewer parameters than either baseline.This threshold follows from the parameter-count comparison for d = 3 and J = 12.Source: F Sloth is parameter-efficient:… · p. 18

G Models in our dataset

• Table 3 distinguishes Family from OriginalFamily, marks test-set membership with TestFamily, and records benchmark availability by model.Its benchmark columns cover Open LLM Leaderboards v1 and v2 and HumanEval.Source: G Models in our dataset · p. 19
• EQ data is available only for the models listed in the dataset description.The passage names the available models, including Gemma, Llama, Qwen, and Yi variants.Source: G Models in our dataset · p. 19

H Extra performance prediction results

• The appendix presents full versions of main-text figures and additional results.Source: H Extra performance prediction results · p. 22

H.1 Mean Absolute Percentage Error (MAPE) plot

• Sloth produces the best predictions on both Open LLM Leaderboards in the MAPE comparison, qualitatively matching the MAE result.The MAPE results are reported in percentage points.Source: Figure 10 · p. 22

H.2.1 Average prediction loss across models

• The average within-family MAE across LLM families is plotted for methods with shared intercepts and for one law fitted across both leaderboards.Source: Figure 11 · p. 23, Figure 12 · p. 24

H.2.2 Family-specific prediction losses

• Family-specific average MAE is plotted across benchmarks separately for Open LLM Leaderboard v1, v2, and their combination.Source: Figure 13 · p. 25, Figure 14 · p. 26, Figure 15 · p. 27

H.3.1 Average prediction loss across models

• The figures plot average within-family MAE across LLM families for different methods, including results using the intersection of both leaderboards.Source: Figure 16 · p. 28, Figure 17 · p. 29

H.3.2 Family-specific prediction losses

• Family-specific average MAE across benchmarks is plotted for Open LLM Leaderboard v1, v2, and their combination.Source: Figure 18 · p. 30, Figure 19 · p. 31, Figure 20 · p. 32

I.1 Results for d = 2

• For d = 2, benchmark loadings are assigned skill names, and the figures show skill correlations, instruction-tuning gains, and parameter-count/training-token level curves.The instruction-tuning figure reports a large positive impact on instruction following and a negative impact on mathematical reasoning.Source: Figure 23 · p. 33, Figure 21 · p. 33, Figure 22 · p. 33

I.2 Results for d = 3

• For d = 3, the results figure presents correlations among the estimated skills.Source: Figure 25 · p. 34

I.3 Results for d = 4

• The figures report estimated benchmark skill loadings and names, skill correlations, instruction-tuning gains across families on three latent skills, and level curves for latent abilities by parameter count and training tokens.Instruction tuning has a large positive impact on instruction following and a negative impact on mathematical reasoning.Source: Figure 26 · p. 34, Figure 27 · p. 35, Figure 28 · p. 35
• Figure 30 describes predicting LLaMa 3 70B (instruct) performance on emotional intelligence from Sloth-predicted skills, after excluding its base and instruct models from Sloth training and fitting a skills-performance regression.Source: Figure 30 · p. 36

J Extra downstream task plots

• One figure presents predictions of agentic capabilities for Llama-2-70B-chat.Source: Figure 31 · p. 36
• A separate figure presents loadings for downstream prediction tasks.Source: Figure 32 · p. 36

K Insights from the different link functions

• The comparison uses LLaMA-3 and Yi-1.5 as test families with constant training-set sizes within each family, studies the two Open LLM Leaderboards separately, and uses model size on the x-axis.For LLaMA-3, only one model is included in the training set, and no family-specific PCA+FLOPs slope is trained.Source: K Insights from the different link… · p. 36
• Training the link function better predicts saturation; shared-intercept FLOPs usually predicts poorly, while PCA+FLOPs often underestimates larger models.The examples include Yi-1.5 performance on ARC and HellaSwag for saturation, and TruthfulQA, GSM8k, and MMLU for underestimation.Source: Figure 33 · p. 37, Figure 34 · p. 37, K Insights from the different link… · p. 37

L Comparing against Ruan et al. [2024] in their observational scaling law setting

• Out-of-sample prediction errors are similar; Sloth fits coding better in-sample, Ruan et al. [2024] fits emotional intelligence better, and their skill loadings lack one-to-one correspondence.The loading comparison identifies some similarities, including instruction following, but no one-to-one correspondence between skills.Source: L Comparing against Ruan et al. [2024]… · p. 38, Figure 37 · p. 39, Figure 38 · p. 39