Distillation Scaling Laws

• Distillation is motivated by the need to match capable small overtrained models while reducing training and lifetime inference costs.Compute-optimal models become increasingly expensive to serve, while overtrained models require many trillions of tokens; prior distillation results across several LM families are mixed.Source: §1 · pp. 1–2
• The proposed distillation scaling law predicts student cross-entropy from student size, teacher size, teacher training tokens, and distillation tokens.The study evaluates transformer teachers and students from 143M to 12.6B parameters trained on a few billion to 512B tokens.Source: §1 · p. 2
• Teacher size and teacher training tokens affect student cross-entropy only through the resulting teacher cross-entropy LT.Source: §1 · p. 2
• The teacher-loss effect follows a power law with two regimes determined by relative learning capacity, explaining the capacity gap in which a stronger teacher can produce a worse student.The capacity gap reflects differences in hypothesis space and optimization ability, not merely relative model size.Source: §1 · p. 2
• Distillation is more efficient than supervised learning when student compute or tokens remain below a student-size-dependent threshold and the teacher already exists or has uses beyond one distillation.With sufficient data or compute, distillation cannot achieve lower cross-entropies than supervised learning.Source: §1 · p. 2

• Cross-entropy L is empirically and theoretically predictable as a power law of model parameters N and training data D.Source: §2 · p. 2
• Compute-optimal models maximize loss variation under a fixed experiment budget but maintain a constant token-to-parameter ratio M ≡ D/N, eliminating one degree of freedom.Source: §2 · p. 3
• Reliable scaling-law coefficients require varying data for fixed model families and varying model size and data jointly under compute constraints.Hoffmann et al. (2022) combines fixed-model, varied-data runs with IsoFLOP profiles for fitting.Source: §2 · p. 3
• Distillation scaling aims to predict the student cross-entropy LS and identify the lowest achievable loss for a given student size and compute budget.The prediction is intended to reveal the value of increasing compute for distillation.Source: §2 · p. 3

Notation

• Sequences are represented with slice and context notation, while X* denotes sequences of arbitrary positive length.Source: §3 · p. 3

Language modeling

• Language modeling predicts tokens from a vocabulary using a parameterized next-token classifier conditioned on preceding context.Source: §3 · p. 3
• Autoregressive language models factor sequence likelihood into conditional next-token probabilities and train by maximizing observed-data likelihood through the NTP loss.Source: §3 · p. 3

Distillation

• Distillation replaces the one-hot target with the teacher’s predicted next-token distribution to train the student’s predictive distribution.Source: §3 · p. 3
• The distillation objective is equivalent to minimizing teacher–student KLD, and combining losses yields a temperature-scaled total token-level student loss.The temperature satisfies τ > 0.Source: §3 · p. 3

4. Distillation Scaling Laws

• The section proceeds from experimental protocols and coefficient identification to specifying, fitting, and extrapolating the distillation scaling law.Its stated objective is to estimate student cross-entropy as a function of student parameters and tokens and teacher parameters and training tokens.Source: §4 · p. 3, §4.2 · p. 4

4.1. Experimental Setup

• The experimental setup uses 143M–12.6B-parameter Transformer models on English C4, with teachers smaller or larger than students and pure distillation at λ = 1 and τ = 1.The setup uses sequence length 4096, RoPE, RMSNorm, separate teacher and student data splits, and µP for learning-rate transfer across model sizes.Source: §4.1 · p. 4

4.2. Distillation Scaling Law Experiments

• The scaling-law experiments vary student and teacher compute, sizes, and token budgets across three protocols to identify how NS, DS, NT, and DT affect student cross-entropy.The study trains six fixed-M teachers, distills across student IsoFLOP profiles, varies teachers for fixed students, and separately explores ten teachers with five student sizes.Source: §4.2 · pp. 4–5
• Student cross-entropy can worsen as teacher capability increases, producing a capacity gap in which stronger teachers eventually reduce student performance.The student finds increasingly capable teachers more challenging to model, limiting its ability to benefit from further teacher improvements.Source: §4.2 · p. 5, Figure 4 · p. 5

4.3. Distillation Scaling Law Functional Form

• The proposed functional form summarizes teacher size and training tokens through teacher cross-entropy and uses a broken power law to model the capacity-gap transition.The transition depends on the ratio of teacher to student algorithmic learning capacities, LT/eLS = d1.Source: §4.3 · pp. 5–6

4.4. Distillation Scaling Law Parametric Fit

• ≲1% relative prediction error was achieved by the supervised and distillation scaling laws, including extrapolation from weaker to stronger models.For fixed model size, infinite-data distillation is also consistent with supervised learning on infinite data.Source: §4.4 · p. 6

5.1. Fixed Tokens or Compute (Best Case)

• With a fixed token or compute budget, distillation helps at modest budgets, but supervised learning eventually outperforms it as student resources increase.In the large-data regime, supervised learning can reach the model-size-limited optimum, while distillation is limited unless the teacher has the optimal loss; this differs from the patient-teacher result of Beyer et al. (2022) because of the supervised baselines.Source: §5.1 · p. 7, Figure 6 · p. 7

5.2. Fixed Tokens or Compute (Teacher Inference)

• For fixed data, the optimal teacher cross-entropy decreases as a power law with student size until matching the student optimum, after which its decrease sharpens.Teacher cross-entropy, rather than teacher parameter count itself, determines student cross-entropy; this generalizes Zhang et al. (2023a)'s linear teacher-to-student scaling as a compute-optimal special case.Source: §5.2 · pp. 7–8, Figure 7 · p. 7

5.3. Compute Optimal Distillation

• When teacher training is included, supervised learning achieves lower student cross-entropy than distillation for a single target model, whereas distillation can be computationally preferable for model families or server-model use.The supervised advantage persists across the compute-optimal comparison, while the transition favoring supervised learning occurs at higher compute for larger students.Source: §5.3 · p. 8, Figure 8 · p. 8
• Compute-optimal distillation allocates student and teacher tokens by power laws, with student tokens growing faster and teacher size eventually plateauing because large-teacher inference is expensive.The application accounts for teacher pretraining and logit-inference costs using an FLOP expression that includes attention contributions, rather than relying on the 2N approximation.Source: §5 · p. 6, Table 2 · p. 6, §5.3 · p. 8

• The distillation scaling law estimates model performance from compute budgets and their allocation between the student and teacher.The law also determines compute-optimal distillation scenarios and helps practitioners select suitable teachers.Source: §6 · p. 9
• Distillation is more efficient than supervised learning only below a student-size-dependent compute threshold and when the teacher already exists or has broader applications.These conditions mitigate the cost of training a teacher for a single student.Source: §6 · p. 9
• The largest controlled empirical study of distillation reported systematic ablations and a roadmap toward smaller, more powerful models with lower inference costs, reduced carbon footprints, and feasible test-time scaling.Source: §6 · p. 9

• Distillation should or should not be performed depending on compute efficiency relative to supervised learning, framing it as an alternative to overtraining for capable language models.Source: Impact Statement · p. 9
• Compute-optimal distillation recipes can produce powerful models with lower inference costs, reducing the largest component of language-model carbon footprints.Lower inference costs reduce the largest component of the total carbon footprint from training to inference.Source: Impact Statement · p. 9
• Combining distillation and established scaling laws expands the known compute-optimal model space while reducing compute, hardware, and climate costs for a target capability.Source: Impact Statement · p. 9
• The distillation scaling law reduces experimentation by showing that teacher cross-entropy primarily drives student cross-entropy, removing teacher size and token count as search dimensions.Source: Impact Statement · p. 9
• Distillation can transfer bias from teacher pretraining data to students, even when students are trained on unbiased data, while smaller powerful models can facilitate malicious misinformation generation.Small powerful models also democratize research on capable models and their safety aspects.Source: Impact Statement · p. 10

A. Limitations

• The study is limited to English language modeling on repeated C4 data, constraining domain generality and downstream evaluation, especially for reasoning.The authors also caution that repeated-data effects established for supervised learning may not transfer to distillation.Source: A. Limitations · p. 23
• The conclusions may not generalize beyond full-distribution transformer distillation, matching architectures, same-distribution data, or the fitted scaling-law coefficients.The authors call for studies of sequence-level distillation, architectural disparities, teacher-student domain differences, and a first-principles derivation.Source: A. Limitations · p. 23

B.1. Knowledge Distillation

• Prior work establishes knowledge distillation, its fidelity and calibration tradeoffs, patient teachers, bias-variance effects, self-distillation, and weak-to-strong transfer; this study isolates distillation without repeating teacher-training data.The scaling law agrees that fixed-teacher-cross-entropy distillation does not improve the supervised data-scaling law.Source: §B.1 · p. 24

B.2. Neural Scaling Laws

• Unlike transfer-learning and neural-scaling studies, this work determines and verifies a distillation scaling law and identifies regions where distillation or supervised pretraining is preferable.The analysis connects to prior scaling-law research while focusing on training a new student from a teacher.Source: §B.2 · p. 24

B.3. The Knowledge Distillation Capacity Gap

• The capacity gap creates an optimal teacher size because teacher size, training tokens, and loss jointly determine whether the student can learn effectively.The scaling law shows a U-shaped trend; temperature and learning-rate analyses find an optimal temperature of one, while calibration alone cannot explain the gap.Source: §B.3 · p. 25

C. Teacher Student Capacity Gaps

• The capacity-gap analysis combines kernel regression and synthetic MLP mapping to study fundamental limitations when teacher and student capacities differ.These settings provide theoretical and practical perspectives on capacity mismatch in distillation.Source: C. Teacher Student Capacity Gaps · p. 26

C.1.1. SETUP

• In the setup, teacher and student occupy finite Hilbert spaces with capacities m and n, while teacher error decreases monotonically as teacher capacity m increases.The analysis assumes infinite training data and studies the optimization entirely in function space.Source: §C.1.1 · pp. 26–27

C.1.3. U-SHAPE IN THE STUDENT ERROR

• Student error decreases as teacher capacity m grows for m < n, but increases for m ≥ n, producing a U-shaped curve from the teacher–student capacity gap.The increase occurs because the teacher learns basis functions unreachable by the student, reducing emphasis on functions the student can learn.Source: §C.1.3 · pp. 27–28

C.2.2. EXPERIMENTAL FINDINGS

• In synthetic MLP experiments, student performance improves with teacher size until an optimum, then worsens, in both cross-entropy and accuracy.The same capacity-gap effect appears because the student cannot learn an overly large teacher completely, whereas a sufficiently large student can match the teacher.Source: §C.2.2 · p. 29
• Teacher completeness can harm a lower-capacity student even when teacher size is fixed, because increasing teacher training data eventually worsens student performance.This supports the interpretation that the teacher’s learned problem representation, not only its parameter count, drives the capacity-gap effect.Source: §C.2.2 · p. 30

D.1. Experimental differences resolving the apparent contradiction with patient teachers

• Patient teaching improves distilled performance with longer training, but becomes compute-suboptimal when supervised learning is more efficient or equally efficient at a given compute level.If the teacher is optimally chosen for the student, large-token distillation can match supervised learning; otherwise supervised learning outperforms it.Source: §D.1 · p. 31

D.2. Fixed tokens or compute (best case)

• Distillation can outperform supervised learning with enough teacher training tokens or compute, but supervised learning wins once student compute or tokens are sufficiently large.Adequate teacher size and training are required; otherwise the student may perform worse than with direct supervised learning.Source: §D.2 · p. 31

D.3. Fixed size or compute (teacher inference)

• For fixed student size, increasing student tokens slightly lowers optimal teacher cross-entropy before saturation, with the effect especially visible for sufficiently large students.At NS = 30B, the optimal teacher loss initially decreases, then saturates and declines slowly as student tokens increase.Source: §D.3 · p. 33
• With teacher-inference costs, small compute budgets favor a large weak teacher, whereas sufficient compute makes a stronger teacher optimal across student sizes.A larger teacher can reduce available distillation tokens, so the preferred teacher depends on the tradeoff between teacher quality and token count.Source: §D.3 · p. 33

D.4.1. SETUP

• The scaling-law applications select model sizes and token allocations by minimizing distillation loss subject to a total FLOPs budget while optionally charging teacher inference and pretraining.The setup uses constrained numerical optimization and reports compute-optimal student and teacher cross-entropies for the defined scenarios.Source: §D.4.1 · p. 35, §D.4.2 · p. 35

D.4.2. CROSS-ENTROPY

• At low compute, distillation outperforms supervised learning, but both produce the same cross-entropy at sufficiently large compute; the crossover occurs later for larger students.Larger students benefit from distillation over larger compute budgets because distillation accelerates progress toward supervised-learning saturation.Source: §D.4.2 · pp. 35–36, Figure 17 · p. 36

D.4.3. DISTILLATION (BEST CASE)

• In the best-case setting, all compute goes to student training because teacher pretraining and logit-production costs are ignored, making this an experimental upper bound.This applies when a teacher already exists, serves another purpose, or produces logits reusable across many students.Source: §D.4.3 · pp. 37–38

D.4.5. DISTILLATION (TEACHER PRETRAINING)

• When teacher training is paid but logits are reusable, the optimal teacher is supervised-compute-optimal, and increasing compute shifts relative allocation from teacher training toward student training.Teacher size and tokens eventually plateau while student tokens continue increasing, because improving the student requires more compute than strengthening the learning signal.Source: §D.4.5 · pp. 40–41

D.4.6. DISTILLATION (TEACHER PRETRAINING + INFERENCE)

• For one student with teacher training and inference costs, optimal teachers transition from compute-optimal at low budgets to overtrained at high budgets.For small students, allocation increasingly favors student training and logit production; for large students, it initially favors teacher training before undergoing the same transition.Source: §D.4.6 · pp. 41–43

D.4.7. OPTIMAL TEACHER TRAINING AND STUDENT DISTILLATION TOKENS

• Student tokens increase polynomially with compute across scenarios, with best-case distillation matching supervised allocation and other methods differing mainly by scenario-dependent offsets.Teacher-pretraining distillation approaches the same allocation asymptotically at large compute.Source: §D.4.7 · pp. 43–44

D.4.8. OPTIMAL TEACHER SIZE

• Optimal teacher tokens and sizes interpolate between compute-allocation scenarios: at low compute, pretraining-based strategies coincide, while at high compute they approach inference-only or best-case solutions.As student-token budgets grow, teacher-pretraining costs become less important, producing the same interpolation mechanism for teacher size.Source: §D.4.7 · p. 44, §D.4.8 · p. 44

D.5. Compute and data efficiency gains for distillation compared to supervised learning

• When teacher training is discounted, distillation is often more compute- and data-efficient than supervised learning, but supervised learning is more efficient near the strongest attainable student performance.This suggests a hybrid strategy that begins with distillation and transitions to supervised learning at a token or cross-entropy threshold.Source: §D.5 · pp. 45–46, Figure 33 · p. 45, Figure 34 · p. 46
• If teacher pretraining is included in production cost, supervised learning is always more compute- and data-efficient than distillation across student sizes.The teacher-pretraining and teacher-pretraining-plus-inference scenarios require more compute than supervised learning.Source: Figure 33 · p. 45, §D.5 · p. 46, Figure 34 · p. 46

E. Additional Results

• The additional evaluations cover downstream tasks, teacher configurations, calibration, KLD, limiting behavior, and weak-to-strong generalization; GSM8K and MMLU remain near-random for most models.The study does not target competitive downstream scores on those benchmarks.Source: E. Additional Results · p. 46, §E.1 · p. 47

E.1. Downstream evaluations

• Downstream performance tracks validation cross-entropy similarly for supervised and distilled models, plausibly because KL-matching students behave like low-variance expectations of biased teachers.The authors caution that alternative distillation strategies may change this relationship.Source: §E.1 · p. 47

E.2. Teachers used in distillation

• The teacher dataset comprises 74 runs across fixed-ratio, fixed-token, and IsoFLOP profiles, while the fitted scaling law achieves at most about 1% relative prediction error, including weak-to-strong extrapolation.The teacher profiles include eleven fixed-M models, six 512B-token models, and four IsoFLOP profiles.Source: §E.2 · p. 48, Figure 36 · p. 48

E.3. Fixed-M teacher/fixed-M students and the capacity gap

• Increasing teacher quality can eventually hurt students because teacher–student KLD rises with teacher size, while weaker teachers can enable larger students to outperform them at sufficient compute.Stronger students require stronger teacher signals to beat supervised baselines; weaker students benefit more broadly from distillation.Source: §E.3 · pp. 48–49, §E.4 · p. 49, Figure 38 · p. 49

E.6. Distillation with infinite data

• Under infinite distillation data, optimally chosen teachers produce student losses consistent with supervised-learning lower bounds for model sizes from 100M to 100B.Other teachers yield higher student cross-entropies because they are too weak or create a capacity gap.Source: §E.6 · pp. 50–51, Figure 40 · p. 51

E.7. Weak-to-strong generalization

• Weak-to-strong generalization occurs only with finite distillation data; with enough tokens, a larger student eventually matches the teacher across the data manifold.The student can contain the teacher’s function and eventually reproduce it, but this does not explain student outperformance.Source: §E.7 · p. 51, Figure 41 · p. 51

E.8.1. TEACHERS

• Teacher models have ECE between 0.4% and 0.6%, with confidence estimates closely matching empirical accuracies across seven teacher sizes.The setup uses 21 uniform probability bins and evaluates calibration using each prediction’s maximum probability.Source: §E.8.1 · p. 52, Figure 42 · p. 52, §E.8 · p. 52

E.8.2. 198M STUDENTS TRAINED ON 20N TOKENS

• Students trained on the full teacher distribution are calibrated to ground-truth data, whereas top-1 distillation produces overconfidence because top-1 labels omit plausible outcomes.Against teacher top-1 labels, the relationship reverses: top-1-trained students are calibrated, while full-distribution students are underconfident.Source: §E.8.2 · pp. 53–54, Figure 43 · p. 53
• Distributional calibration reveals that full-distribution students underestimate low-confidence outcomes as teachers grow, while top-1 students shift from overconfidence toward underconfidence.These mismatches indicate difficulty learning rare or low-probability events and align with increasing KLD and capacity gaps.Source: §E.8.2 · pp. 54–55, Figure 45 · p. 55

E.8.3. 198M STUDENTS TRAINED ON 128B TOKENS

• Increasing distillation tokens improves calibration when training and evaluation use an unbiased or matching metric, but worsens it when the objectives mismatch.Full-distribution training improves data and distributional calibration, whereas top-1 training improves teacher-top-1 calibration but can degrade other measures.Source: §E.8.3 · pp. 56–57, Figure 47 · p. 56

F. Scaling coefficients

• The scaling-coefficient analysis follows established procedures while fitting modified supervised and distillation laws independently in log space without parameter tying.The supervised fit uses 73 samples, while the distillation fit uses 697 samples and optimizes their respective parameter sets.Source: F. Scaling coefficients · p. 58, §F.1 · p. 58, §F.2 · p. 58

F.1. Supervised scaling law coefficient estimation

• Supervised coefficients are fit with Huber δ = 10−4 on 73 samples, whereas distillation coefficients fit 697 samples using parameters for student size, tokens, teacher loss, and validation loss.Both optimizations use L-BFGS-B initialization grids; the supervised L ≥2.2 subset has 48 samples and the distillation LS ≥2.3 subset has 551.Source: §F.1 · p. 58, §F.2 · p. 59

F.3. Scaling law coefficients parameteric fit

• Table 6 reports fitted scaling-law coefficients with 90% bootstrap confidence intervals, with supervised a and b consistent with Hoffmann et al. (2022).The intervals use 4096 bootstrap resamples.Source: Table 6 · p. 59, §F.3 · p. 59
• The fitted irreducible error is lower than Hoffmann et al. (2022), possibly because the experiments use µP (Yang & Hu, 2021; Yang & Littwin, 2023; Yang et al., 2022; Wortsman et al., 2023; Yang et al., 2023).The paper presents the µP explanation as a suspicion rather than a confirmed mechanism.Source: §F.3 · p. 59

G. Distilling language models in practice

• Pure distillation, unit temperature, and learning rate η = 0.01 under µP provide robust student performance across model scales.These settings correspond to λ = 1, τ = 1, and the stated learning rate.Source: G. Distilling language models in… · p. 59
• Top-k and Top-p distribution truncation degrade performance unless combined with ground-truth next-token prediction.The truncation results are analyzed in Appendix G.4.Source: G. Distilling language models in… · p. 59
• Forward KL divergence distillation, D_KL(ˆp_T || ˆq_S), consistently outperforms reverse KL.This comparison is verified in Appendix G.5.Source: G. Distilling language models in… · p. 59
• The practical section restates the components of the student’s token-level loss and refers readers to Section 2 for discussion of each term.This provides the loss formulation’s location rather than an additional empirical finding.Source: G. Distilling language models in… · p. 60

G.1. Mixing coefficient (λ) sensitivity analysis

• λ > 0.5 keeps student cross-entropy mostly flat, making pure distillation (λ = 1) a robust default except when teachers are particularly weak.Lower λ values are preferred when the supervised signal is more informative; optimal λ∗ varies across teacher-student combinations.Source: §G.1 · pp. 60–61

G.2. Temperature (τ) sensitivity analysis

• τ = 1 achieves optimal performance across model scales, whereas τ > 3 or τ < 1 reduces effectiveness by weakening or concentrating teacher probability relationships.Higher temperatures particularly harm smaller students, while τ = 1 preserves the teacher’s original log-probability structure.Source: §G.2 · p. 61, Figure 52 · p. 61

G.3. Learning rate (η) sensitivity analysis, verification of µP for distillation

• η = 0.01 is optimal across tested student and teacher sizes, with cross-entropy changing by less than 0.1 nats over one order of magnitude.This supports reusing the supervised reference learning rate in distillation and validates µP’s scale-invariant training dynamics in these settings.Source: §G.3 · p. 62, Figure 53 · p. 62

G.4. Distribution truncation methods: Top-k and Top-p sensitivity

• 0.01 nats: Top-k with k = 128 and λ = 0.7 matches full distillation, whereas unmitigated Top-k and Top-p increase validation loss by 0.11 and 0.13 nats, respectively.Mixing truncated teacher predictions with ground-truth next-token supervision enables efficient post-hoc distillation while preserving performance.Source: §G.4 · p. 63, Figure 54 · p. 63

G.5. Forward and reverse KL divergence

• 0.28 nats: forward KL achieves lower data cross-entropy than reverse KL while avoiding the latter’s student-entropy computation, motivating standard forward KL.The comparison studies distillation from a 1.82B teacher to a 546M student; reverse KL is slightly more expensive with vocabulary size V.Source: §G.5 · pp. 63–64, Table 7 · p. 64

H. Parameters and Floating Operation Estimation

• Attention’s quadratic, context-dependent cost must remain in FLOPs estimates for capable small models, unlike approximations that ignore this term.Because the architectures have fixed aspect ratios, the authors use a modified approximation and fit scaling laws using non-embedding parameters.Source: H. Parameters and Floating Operation… · p. 64

H.1. Alternative approximation for FLOPs per token as a function of N

• Unlike common approximations, the method retains context-dependent and embedding-related contributions that can cause systematic errors for small, capable models.The standard expression C = 6ND may omit terms that matter when models have large context windows; Kaplan et al. (2020) and Hoffmann et al. (2022) use differing conventions.Source: §H.1 · pp. 64–65
• The approximation reduces to the familiar C_forward = 2N when N is large or context and vocabulary sizes are small, while backward FLOPs are twice forward FLOPs.The backward-FLOPs convention follows Blondel & Roulet (2024).Source: §H.1 · p. 65
• Fixed-aspect-ratio models provide simpler compute expressions, tighter scaling-law fits, improved work-based intuition, and more predictable large-scale hyperparameters.The authors recommend that the scaling-law community consider adopting fixed aspect ratios.Source: §H.1 · p. 65

H.2. Model parameters

• The parameter-counting framework compares existing formulas and accommodates normalization, gated linear attention, and grouped-query attention choices in the target architecture.The comparison includes Kaplan et al. (2020), Hoffmann et al. (2022), Narayanan et al. (2021), Ainslie et al. (2023), and Shazeer (2020).Source: §H.2 · p. 66
• The non-embedding parameter approximation drops subleading terms and supports estimating forward FLOPs per token from model size.Tables 9 and 10 compare the parameter counts with existing expressions and provide simplified notation for analysis.Source: §H.2 · p. 66, Table 9 · p. 66, Table 10 · p. 66

H.3. FLOPs per token

• The FLOPs accounting adopts causal-attention computation because it reflects efficient best practice, unlike bidirectional counts that include masked attention operations.The discrepancy is identified relative to Kaplan et al. (2020), Hoffmann et al. (2022), and Narayanan et al. (2021), with efficient causal computation aligned with Dao et al. (2022) and Dao (2024).Source: §H.3 · p. 67
• The proposed forward FLOPs-per-token approximation retains context and vocabulary terms alongside the non-embedding computation, enabling estimation from model size.The approximation drops subleading terms and is used to estimate forward FLOPs per token from model size.Source: §H.3 · p. 67

I. Model architecture

• All models use fixed-ratio Transformer architectures with MHA, RMSNorm pre-normalization, RoPE, 4096-token sequences, gated activations, and µP (simple) with a 1e−2 learning rate.Models also use decoupled weight decay of 10−4 and fixed dmodel = 128 and ρffn = 8/3.Source: §I · p. 68
• Training applies global-gradient-norm rescaling to 1.0, cosine decay with 2000-step warmup, a final learning rate of one thousandth of peak, and Z-loss of 10−4.The Z-loss improves stability by slightly reducing norm growth late in training.Source: §I · pp. 68–69
• Experiments use English-only C4, splitting its roughly 180B tokens into distinct 90B-token teacher and student pools for distillation.Except for the largest models, Chinchilla-optimal models do not repeat data; repetition beyond 90B tokens is permitted and has limited reported loss impact up to four repeats.Source: §I · p. 69

J. Contributions

• The authors jointly designed experiments, discussed results, and led the paper’s research direction, framing, question identification, prioritization, and writing.Most writing was done by Dan Busbridge, Jason Ramprapam, and Amitis Shidani.Source: J. Contributions · p. 69
• Scaling-law contributions include fixed-aspect-ratio models, FLOP counting, implementation, dataset preparation, IsoFLOP design, teacher training, distillation, and extended 512B-token training.These contributions were made primarily by Dan Busbridge, Amitis Shidani, and Floris Weers, with dataset preparation and long-duration training by Floris Weers.Source: J. Contributions · p. 69
• The scaling-law analysis combines fitting code, JAX-compatible generalized implementations, numerical compute-optimal minimization, and the collaboratively developed functional form in Equation 8.The original fitting code was based on Besiroglu et al. (2024).Source: J. Contributions · p. 69
• Additional contributions cover calibration, CDF and top-k metrics, downstream evaluations, kernel-regression and MLP capacity-gap demonstrations, and extensive practical distillation sensitivity analyses.The practical analyses examine mixing coefficients, temperature, learning rate, truncation, combined mixing and truncation, and reverse KL divergence.Source: J. Contributions · p. 69