PaperlayerView exact revision on arXiv

Source-linked AI summary

Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Nicholas Roberts, Ahmed Ahmed, Suhas Kotha, Percy Liang, Yu Wang, Alex Renda, Nitya Mani

arXiv:2610.31867v1cs.LGcs.CL

TL;DR

Synthetic rewrites are common in pretraining, but their compute cost and ability to support models beyond the rewriter remain open questions. The paper fits a cost-aware rewriting scaling law in compute- and data-limited settings, finding that models trained on rewrites can outperform their rewriters and that optimal rewrite fractions are predictable at large budgets.

  • Problem

    Synthetic rewrites are costly to generate, and it is unclear whether a model trained on them can exceed its rewriter’s capabilities.

    Source: §1 · p. 2

  • Method

    A rewriting scaling law predicts loss from rewrite fraction and other training quantities, accounting for rewrite-generation compute to estimate cost-adjusted optimal fractions.

    Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 4

  • Results

    10 out of 16 general web text sources favor the rewrite-trained model over its rewriter at roughly one-tenth the rewriter’s training compute; data-limited mixtures also outperform their rewriter.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The results show that optimal rewriting choices can be predicted with scaling laws, and scaling need not break down beyond the rewriter’s capabilities.

    Source: §1 · p. 3

    The reported optimal fractions use FineWeb-Edu and only four rewrite styles and one external rewriter per setting, limiting conclusions across seed sets, prompts, and rewriter families.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit this scaling law in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under $10\%$ at $3\times10^{18}$ FLOPs to $32\%$ at $10^{22}$ FLOPs. Our scaling forecast extrapolates $33\times$ in FLOPs to train a 22B-parameter model, whose final loss is within $1.1\%$ of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. In the data-limited setting, we assume we only have a fixed seed set of natural data and that our rewriter is a model of the same size as the target, pretrained on that seed set. Again, we find the optimal rewrite fraction increases with the compute budget. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter's capabilities.

1 INTRODUCTION

The paper studies how to choose synthetic rewrites for pretraining when natural data is scarce and rewriting consumes compute, including whether rewrite-trained models can surpass their rewriters. It proposes a cost-aware rewriting scaling law and evaluates it in compute-limited and data-limited settings.

1 INTRODUCTION

  • Natural-data availability has lagged training-compute growth, while rewrites cost compute to generate and may be limited by the rewriter’s capabilities.These pressures motivate asking how much compute to spend rewriting and whether rewrite-trained models can exceed their rewriters.Source: §1 · pp. 1–2
  • The rewriting scaling law predicts loss from rewrite fraction and accounts for the compute available after paying to generate rewrites.The paper tests whether it predicts optimal fractions and loss, and whether rewrites help models surpass their rewriters.Source: §1 · p. 2
  • The experiments distinguish a compute-limited setting with plentiful natural data and a higher-compute external rewriter from a data-limited setting with bounded data and a bootstrapped rewriter.The latter rewriter is trained on a subset of available data and for fewer FLOPs than the candidate target run.Source: §1 · p. 2
  • At large budgets, predicted rewrite fractions achieve loss within 1.1% of prediction, while a model with 32% rewritten data beats its rewriter on 10 out of 16 general web text sources at roughly one-tenth its training compute.In the data-limited setting, models trained on mixtures containing rewrites also consistently outperform the rewriter.Source: §1 · pp. 2–3
  • Optimal rewrite fractions increase with scale and vary by task, including math, code, question-answering, and general web text.The results support using scaling laws to predict methodological choices, while showing scaling need not fail beyond the rewriter’s capabilities.Source: §1 · p. 3

2 METHODOLOGY

The methodology models held-out loss for training data that mixes natural tokens and rewrites, then uses the fitted law to choose a rewrite fraction under a compute budget. The budget includes both target-model training and rewrite-generation costs.

2.1 A REWRITING SCALING LAW

  • The corpus contains D tokens drawn from a seed set of T natural tokens, with natural data potentially repeated and rewrites not repeated; R is the rewrite-token fraction.The target model has N parameters, and the law predicts its held-out loss from these quantities.Source: §2.1 · p. 3
  • The loss law combines parameter- and dataset-size terms with a rewrite-fraction-dependent irreducible-loss term and a repetition penalty for epoching natural data.The repetition penalty is nonzero only when natural tokens are repeated, so it applies in the data-limited setting.Source: §2.1 · p. 3

2.2 PREDICTING OPTIMAL REWRITE FRACTIONS AT A COMPUTE BUDGET

  • For a fixed compute budget, the method chooses the rewrite fraction that minimizes fitted loss while charging both target training and rewrite generation.The compute approximation assumes equal source and rewrite lengths and does not amortize generation costs across target-model runs.Source: §2.2 · pp. 3–4
  • The compute approximation charges 6ND FLOPs for target training and 4NrDR for rewriting, assuming rewrite length matches source length and prompt overhead is small.The 4NrDR charge comprises source prefill and rewrite decoding; the total-cost constraint is used in optimization.Source: §2.2 · p. 4

3 EXPERIMENTS

The experiments test a rewrite scaling law in compute-limited and data-limited settings, accounting for rewrite-generation compute and varying data availability and rewriter strength. Across both settings, the law predicts useful rewrite fractions, and rewrites can help targets outperform their rewriters.

3.1 EXPERIMENTAL SETUP

  • The compute-limited experiments use abundant natural data and an external rewriter trained with more compute than any target run; data-limited experiments use a fixed seed set and a bootstrapped rewriter.Runs use Qwen3 dense models, and mixed corpora interleave natural and rewritten documents in proportion to R.Source: §3 · p. 4, §3.1 · p. 4

3.2.1 COMPUTE-LIMITED EXPERIMENTAL RESULTS

  • The compute-optimal rewrite fraction rises with FLOP budget on every evaluation, but its growth varies by task.It reaches R = 1 within the tested ladder for Math and Code; the fitted asymptotes are 0.77 for QA and 0.32 for Paloma.Source: §3.2.1 · pp. 5–6, Figure 2 · p. 5
  • At 10^22 FLOPs, the compute-limited scaling forecast’s error for the R = 0.32 run is −1.1%.The held-out run was forecast from a fit using budgets through 3 × 10^20 FLOPs.Source: Figure 3 · p. 6, §3.2.1 · p. 6
  • At 10^22 FLOPs, the predicted-optimum R = 0.32 run beats SmolLM2-1.7B-Instruct on 10 of 16 Paloma sources.Its macro-average is 0.983 versus the rewriter’s 0.972 BPB.Source: §3.2.1 · p. 6, Figure 3 · p. 6, Table 1 · p. 7

3.3.1 DATA-LIMITED EXPERIMENTAL RESULTS

  • For the 3B target, the best observed rewrite fraction rises from 0.13 at W = 3 to 0.71 at W = 24.Across target sizes, the best observed fraction also increases with epoch budget and is predictable at held-out budgets.Source: §3.3.1 · p. 8
  • For held-out data-limited budgets W = 21 and W = 24, the selected fraction’s measured loss is within 0.74% of the forecast at every target size.For the 3B target, the selected fraction is within 1.14% of the best observed fraction at W = 24.Source: Figure 4 · p. 7, §3.3.1 · p. 8
  • At every target size, training at the scaling-law-selected rewrite fraction achieves lower Paloma BPB than natural-only training.For epoch budgets W > 3, the natural-only runs overfit.Source: §3.3.1 · p. 8, Figure 5 · p. 8
  • On Math and Code, mixtures with rewrites beat the best natural-only run at every budget; on QA, they do so only beyond W = 6.The result is for the 3B target; across sizes, the rewrite fraction that performs best depends on task.Source: §3.3.1 · p. 8, Figure 6 · p. 8

4 RELATED WORK

Related work covers synthetic rewriting as a pretraining ingredient, its relationship to distillation, and scaling-law studies of synthetic data. The paper distinguishes its compute-aware treatment from prior mixture analyses that treat generation as free.

Data-constrained pretraining

  • Prior work on finite-corpus pretraining models diminishing returns from repeated epochs and studies fixed-data regimes with unlimited compute.Lovelace et al. (2026) use a repetition penalty that activates beyond the first epoch.Source: §4 · p. 9

Synthetic rewriting

  • Synthetic rewrites are used in several high-quality pretraining corpora, and prior work reports benefits persisting with scale.Rewrites typically restate a pretraining document, often in a target style.Source: §4 · p. 9

Rewriting as distillation

  • Prior work finds rewrite benefits can persist with scale, while rewriter size and benchmark performance do not straightforwardly predict rewrite value.Reported findings include gains from a 1.8B rewriter close to a 7B rewriter and gains from a model synthesizing its own training documents.Source: §4 · p. 9

Scaling laws for synthetic pretraining data

  • Prior scaling-law studies of synthetic pretraining data do not account for the compute spent generating rewrites, so their optimal mixtures answer how much to rewrite when rewrites are free.The studies include rephrased and textbook-style data, token-effectiveness modeling, and synthetic math-data scaling.Source: §4 · p. 9

5 CONCLUSION AND LIMITATIONS

The conclusion reports that the rewriting scaling law predicts cost-adjusted rewrite choices and that rewrites can help even when the rewriter is less capable than the target. The authors bound these findings by seed-set, rewrite-style, rewriter, and compute-accounting choices.

5 CONCLUSION AND LIMITATIONS

  • At 33× compute, the small-budget fit identifies a rewrite fraction minimizing loss among trained fractions; the 32%-rewrite model beats its rewriter on 10/16 web domains.In the data-limited setting, held-out optimal fractions and losses are also predictable, and a positive rewrite share is additive even with a less-capable rewriter.Source: §5 · p. 9
  • Every reported optimal rewrite fraction uses FineWeb-Edu; rewrites also help on DCLM, but by less.The authors say this suggests a strong natural seed set benefits less from rewriting.Source: §5 · p. 9
  • The experiments cover only four FinePhrase styles and one external rewriter per setting, limiting conclusions about prompt distributions and rewriter families.Source: §5 · p. 10
  • Compute accounting charges 4Nr FLOPs per rewritten token, assumes decoding costs the same as prefill and rewrites match source length, and excludes rewriter pretraining.Source: §5 · p. 10

APPENDIX ROADMAP

  • The appendix roadmap points to setup and evaluation details, data-limited experiments, rewrite analyses, compute-limited experiments, and DCLM diagnostics.Source: APPENDIX ROADMAP · p. 16

A DATA SOURCES

The data sources section describes FineWeb-Edu as the primary natural corpus, with different sampling choices for data- and compute-limited experiments, and identifies the rewrite sources and DCLM control.

A DATA SOURCES

  • FineWeb-Edu supplies the primary natural text; data-limited runs use a fixed 5B-token seed set, while compute-limited runs sample fresh documents at each token budget.Documents dated 2023 or later are dropped to limit LM-generated text.Source: A DATA SOURCES · p. 16
  • DCLM replaces FineWeb-Edu in the source-quality control.Source: A DATA SOURCES · p. 16
  • Compute-limited rewrites come from FinePhrase’s approximately 338M FineWeb-Edu rewrites in FAQ, math, table, and tutorial styles; data-limited rewrites are generated from the seed set by size-matched bootstrapped rewriters.Source: A DATA SOURCES · p. 16

B MODEL ARCHITECTURE

The appendix specifies a shared dense Qwen3 architecture and describes how model configurations, optimization settings, and compute-budget candidates are selected and scaled.

B MODEL ARCHITECTURE

  • Runs use dense Qwen3, the Llama 3 tokenizer, 4096-token sequences, and an MLP intermediate dimension of 4d; the layer rule gives 11 layers at d=1024 and 21 at d=2048.Source: B MODEL ARCHITECTURE · p. 16

C HYPER-PARAMETER TUNING

  • The Completed(d)P recipe scales a reference configuration to each run’s batch size and token count.Source: C HYPER-PARAMETER TUNING · p. 16
  • A 156M-parameter FineWeb-Edu reference model anchors tuning, with hyperparameters selected by Gaussian-process search on Paloma validation macro cross-entropy.The reference uses batch size 64 and 2.5 × 10^9 tokens; runs use Adam and Hyperball with linear warm-up and decay.Source: C HYPER-PARAMETER TUNING · p. 17
  • Runs target 216 steps, choose a power-of-two batch size, and halve it until learning-rate and β2 constraints are met, within batch sizes 8–8192.Source: C HYPER-PARAMETER TUNING · p. 17
  • Candidate architectures are enumerated by hidden dimension for each budget and assigned D=C/(3Ffwd) tokens, with nominal budgets and compute accounting defined using C=6ND.Source: C HYPER-PARAMETER TUNING · p. 17

D EVALUATION DETAILS

Evaluation uses bits per byte for language-model fit and task suites, with lower values better; the appendix details how Paloma and capability metrics are aggregated and reports matched-compute target pairs for Figure 7.

D EVALUATION DETAILS

  • BPB is reference-continuation negative log-likelihood in bits divided by UTF-8 byte length, and lower is better.Source: D EVALUATION DETAILS · p. 17
  • Paloma macro BPB uniformly averages 16 sources, with bytes pooled within each source; SmolLM2 is evaluated at a forced 4096-token context for the rewriter comparison.Source: D EVALUATION DETAILS · p. 17
  • Question answering averages ARC, HellaSwag, and MMLU uniformly, using gold-answer BPB for scaling surfaces and gold-answer rank-first fraction as a secondary metric.Source: D EVALUATION DETAILS · pp. 17–18
  • Figure 7 pairs 1B and 3B targets at the nearest grid budgets, with paired FLOP ratios of 0.97, 0.97, and 1.09.Source: Figure 7 · p. 18
  • The math suite uniformly averages nine gold-answer BPB tasks; code BPB uniformly averages HumanEval, MBPP, and multilingual MBPP-v2fix after averaging its 17 language variants.The code metric is reference-code BPB, not execution accuracy.Source: D EVALUATION DETAILS · p. 18

E DATA LIMITED BASE REWRITERS

The 1B target benefits from higher-capacity rewrites, while data-limited rewriter checkpoints are selected by an LM-judge comparison against SmolLM2-1.7B rewrites.

E DATA LIMITED BASE REWRITERS

  • At W = 12, the 550M rewriter trails the 3B rewriter by 0.011 BPB at R = 0.13 and 0.069 BPB at R = 1.Figure 8 compares 1B target loss using 550M, 1B, and 3B rewriters and shows the 1B–3B target gap closed with 3B rewrites.Source: E DATA LIMITED BASE REWRITERS · p. 18, Figure 8 · p. 19
  • At nearest total-compute budgets on the grid, the 3B target has lower BPB than the 1B target in 17 of 18 pairs, by 0.031 BPB at the smallest and 0.088 at the largest budget pair.Source: E DATA LIMITED BASE REWRITERS · p. 19

F REWRITER INSTRUCTION TUNING

  • The rewriters receive SFT on 4,000 GLM-5.2 rewrite examples from 1,000 seed documents, using four FinePhrase prompts.The examples total 7M tokens; checkpoint selection uses an LM judge.Source: F REWRITER INSTRUCTION TUNING · p. 19

F.1 CHECKPOINT SELECTION WITH AN LM JUDGE

  • GLM-5.2 and MiniMax-M2.7 judge rewrite pairs, and averaged Elo-derived win rates against SmolLM2-1.7B set each checkpoint’s score.The judge compares rewrites of the same document and prompt, with each pair presented in both orders.Source: §F.1 · p. 19
  • No checkpoint reaches a 50% win rate against SmolLM2-1.7B; the seed-set checkpoint is highest-scoring at 550M, 837M, and 3B.At 1B, it scores 0.7 points below the highest-scoring checkpoint, whose confidence interval overlaps its own.Source: §F.1 · p. 20

G ALIGNMENT

The alignment control compares SFT+DPO rewrites with SFT-only rewrites for the 1B rewriter.

G ALIGNMENT

  • Across 15 runs with R > 0 and W ∈{3, 6, 9}, SFT+DPO lowers Paloma macro BPB in all 15 by 0.046 BPB on average.Programming-languages and RedPajama sources account for 80% of the reduction, while BPB increases on 8 of 16 Paloma sources.Source: G ALIGNMENT · p. 20

H DATA-LIMITED EXPERIMENTAL RESULTS

The data-limited experiments evaluate rewrite mixtures across target sizes, epoch budgets, and domains, including a best-checkpoint analysis. Mixtures combine natural seed-set documents with rewrites, whose generation and reuse are controlled across training.

H DATA-LIMITED EXPERIMENTAL RESULTS

  • The full grid covers four target sizes, eight epoch budgets, and six rewrite fractions, with compute accounting for rewrite generation.Each size and budget has six rewrite-fraction runs; generated rewrite tokens are used once, while the repetition penalty applies only to the natural seed set.Source: H DATA-LIMITED EXPERIMENTAL RESULTS · p. 20, Table 3 · p. 22

H.1 DOMAIN LOSS ACROSS TARGET SIZES

  • On Math and Code, the best observed rewrite fraction is interior at every target size from W = 3, while QA often favors R = 0 at smaller budgets.For QA, R = 0 is best at W ≤6 for 3B, at W ≤9 for 1B and 837M, and at every budget but one for 550M.Source: §H.1 · p. 22

H.2 BEST-CHECKPOINT EVALUATION

  • Best-checkpoint evaluation selects an earlier checkpoint only when its Paloma validation loss is at least 0.01 nats below the final checkpoint, and changes the best Paloma rewrite fraction in 4 of 32 size–budget settings.Paloma alone determines checkpoint selection; QA, Math, and Code are evaluated at the selected checkpoint.Source: §H.2 · p. 22
  • For the 3B target at W = 24, best-checkpoint selection lowers natural-only Paloma BPB from 1.512 to 1.241, below pure rewriting at 1.264.The lowest Paloma BPB is at R = 0.71; QA and Code minima are at R = 0.47, while Math ties at R = 0.13 and R = 0.47 after rounding.Source: §H.2 · p. 22

I MIXTURE CONSTRUCTION

  • Each mixture combines filtered FineWeb-Edu documents and FinePhrase rewrites pooled across four styles, sampling rewrite tokens by token count.Natural and rewritten components receive weights 1 − R and R; deterministic block assignment keeps each block-aligned stream prefix at rewrite fraction R, up to rounding.Source: I MIXTURE CONSTRUCTION · pp. 22–23

J GENERATING REWRITES

  • Same-size data-limited rewriters are pretrained on the seed set, instruction-tuned on GLM-5.2 rewrites, and used to rewrite the seed set at temperature 1.0.The synthetic pool is generated before target training and remains fixed; synthetic tokens are seen once, while natural tokens may repeat.Source: J GENERATING REWRITES · p. 23

K REWRITE-STYLE AND LINE-CONTENT ABLATIONS

Style and line-content ablations compare rewrite-mixture variants, while the scaling-law details describe mixture-dependent loss floors and assume rewrites change the floor rather than model capacity or data scaling.

K REWRITE-STYLE AND LINE-CONTENT ABLATIONS

  • Line classification measures surface form, not faithfulness to the source; table rewrites are 59% table lines and tutorial rewrites have the largest code share at 0.80%.The analysis samples 6,000 documents per FinePhrase variant.Source: K REWRITE-STYLE AND LINE-CONTENT… · p. 23, Table 4 · p. 24
  • Style-only effects vary: FAQ is higher than FAQ style-out at all fractions, Math is higher at 7 of 10, while Table and tutorial are lower in most comparisons.Mean differences are +0.0226 BPB for FAQ, +0.0032 for Math, −0.0039 for Table, and −0.0038 for tutorial.Source: K REWRITE-STYLE AND LINE-CONTENT… · pp. 24–25
  • Removing any rewrite line type raises mean BPB by 0.0120 to 0.0255, with the largest increase from removing prose.These comparisons use the nine fractions below R = 0.9 and exclude the full-mixture run at R = 0.9.Source: K REWRITE-STYLE AND LINE-CONTENT… · p. 25

L REWRITING SCALING LAW DETAILS

  • The mixture loss floor is convex in rewrite fraction; it increases with rewrites on natural-text evaluation, falls toward zero on rewrite evaluation, and can have an interior minimum on other distributions.The floor is the irreducible cross-entropy on the evaluation distribution for a model with unlimited capacity and data trained on the mixture.Source: L REWRITING SCALING LAW DETAILS · pp. 25–26
  • The floor-optimal rewrite fraction is unique under positive λn and λs, and is interior when E′(0) < 0 < E′(1).Equation (10) represents the cost of removing natural text and the benefit of adding rewrites.Source: L REWRITING SCALING LAW DETAILS · p. 26
  • The central modeling assumption is that rewrites shift the loss floor, not how loss scales with model size N and token count D.For (1 − R)D > T, rewrites also dilute the natural-data repetition penalty and reduce natural epochs; rewrite generation costs 4Nr FLOPs per token, while rewriter-training cost is amortized across experiments.Source: L REWRITING SCALING LAW DETAILS · p. 26

L.1 COMPUTE-LIMITED SETTING

The compute- and data-limited fits use rewriting scaling laws that account for rewrite-generation compute and repeated natural data. Across the analyses, the fitted optima and forecasts support rewrite fractions that vary with compute budget, while several fit assumptions and extrapolations remain important qualifications.

L.1 COMPUTE-LIMITED SETTING

  • In the compute-limited setting, each rewritten token incurs generation and training compute, so the optimal rewrite fraction increases with budget as rewriting overhead becomes relatively cheaper.With rewrites, the optimum trains a slightly larger model on fewer tokens than the natural-only allocation at the same training compute.Source: §L.1 · p. 27

N.1.2 E(R) DERIVATION

  • The data-limited loss law combines a rewrite-dependent floor with a training-data term and a repetition penalty for repeated natural tokens, which rewrites dilute.The floor is convex in R; its minimizer can be interior when the evaluation distribution matches neither natural text nor rewrites.Source: §L.2 · p. 27, §N.1.2 · p. 28, §N.1.5 · p. 34

M DEPENDENCE OF EFFICIENT-FRONTIER ALLOCATION ON R

  • Per-R token optima lie within 0.85–1.28× of the shared D*(C), and the shared model-size choice costs at most 0.0015 BPB on each sweep's parabola.Offsets improve RMSE from 0.00831 to 0.00762 BPB, but the largest shift in D*(R)/D*(0) is 1.196 at R = 0.8.Source: M DEPENDENCE OF EFFICIENT-FRONTIER… · p. 27, §N.1 · p. 28

N.1.1 DO REWRITES STILL HELP IN THE INFINITE-COMPUTE LIMIT?

  • Across nine budgets, 82 IsoFLOP-optimal observations collapse onto one rewrite-gain curve with RMSE 0.0044 BPB; the fitted floor shift is −0.0825 BPB.The estimated floor-optimal fraction is 0.328, and a natural-only model at 10^22 training FLOPs needs 5.01× as many tokens to match the mixture floor.Source: §N.1.1 · p. 28, Figure 17 · p. 31, Figure 18 · p. 31

L.2 DATA-LIMITED SETTING

  • The data-limited fit has U-shaped Paloma BPB curves, with the fitted optimal rewrite fraction increasing with epoch budget in every target–rewriter pair.The best-mixture asymptote is no higher than the natural-only asymptote for every pair, but the fits do not isolate a target-size effect because rewriter quality changes with target size.Source: §L.2 · p. 27, §N.1.4 · p. 32, Figure 19 · p. 32

N.1.4 FITTING L(N, D, R, T)

  • At held-out W = 21 and W = 24, the data-limited forecast's measured loss at the best observed R is within 0.74% of its forecast.The best observed R is 0.71 at every target–rewriter pair and both budgets.Source: §N.1.4 · p. 32, Table 6 · p. 33
  • At 10^22 FLOPs, the compute-limited forecast has mean error −1.0%, with every measured loss 0.9–1.4% below its forecast.The forecast extrapolates from a fit ending at 3 × 10^20 FLOPs.Source: §N.1.4 · p. 33, Table 8 · p. 34

N.1.5 REPETITION-PENALTY FORM ABLATION

  • Removing the additive repetition penalty raises RMSE by 1.8–3.4×, while removing its dilution factor makes the fit degenerate; the repetition-count exponent is not identified by this grid.The paper retains the linear penalty as the most parsimonious form that reproduces the U-curves.Source: §N.1.5 · pp. 34–35

N.2 DOMAIN-SPECIFIC COMPUTE OPTIMA

Evaluation-specific fits yield different optimal rewrite fractions and model allocations, with break-even compute varying substantially by domain. A QA accuracy fit also favors a large rewrite fraction, though its estimated optimum depends on the evaluation criterion.

N.2 DOMAIN-SPECIFIC COMPUTE OPTIMA

  • At 10^22 FLOPs, the QA-optimal run measures 0.696 BPB versus SmolLM2-Instruct’s 0.709; fitted Math and Code break-even points are ≈6 × 10^22 and ≈4 × 10^23 FLOPs.The Math and Code break-even points are beyond the largest trained budget.Source: §N.2 · p. 36
  • Floor-optimal rewrite fractions reach 1.0 for every Math and Code task and 0.77 for QA; at 10^22 FLOPs, QA’s optimal model has 1.5× Code’s parameters.C4, RefinedWeb, and Wikipedia remain at a floor-optimal rewrite fraction of 0 throughout the evaluated compute range.Source: §N.2 · p. 38, Figure 26 · p. 40

N.3 EVALUATIONS OF THE 1022-FLOP RUNS

  • TheoremQA’s fitted optimum switches from R=0 to R=1 because its floor is nearly linear in R; its fit excludes the two budgets with the fewest model widths.MMLU, Minerva, and multilingual MBPP average 57, 7, and 17 subtasks, respectively.Source: §N.3 · p. 39

N.4.1 DEFINITION AND RELATION TO TRAINING COMPUTE

  • Break-even is the smallest total compute, including rewrite generation, where the fitted compute-optimal frontier reaches SmolLM2-Instruct’s score; SmolLM2’s own training compute is excluded from the crossing.SmolLM2-Instruct’s training compute is 1.1286 × 10^23 FLOPs.Source: §N.4.1 · p. 39

N.4.2 BREAKEVEN N, D, AND R PER DOMAIN

  • Paloma, QA, and Math cross below SmolLM2’s training compute, whereas Code crosses above it.The training-compute column is 6N*D*; the difference from total compute is rewrite-generation cost.Source: §N.4.2 · p. 40

N.5.1 PARAMETRIC ACCURACY MODEL

  • QA accuracy is fit across 607 observations by applying a logistic link to the BPB surface, with a soft-L1 objective at scale 0.02.The fit refits every parameter and is initialized from the QA-BPB surface.Source: §N.5.1 · p. 40

N.5.2 ACCURACY FITS

  • The accuracy fit has R2=0.98795 and RMSE 0.00453, but weakly identifies asymptotes; its 10^22-FLOP rewrite optimum is 0.638 versus 0.749 under QA-BPB.Both criteria favor a large rewrite fraction, but the estimated optimum and crossing depend on the criterion.Source: §N.5.2 · pp. 40–41

N.6.1 ALTERNATIVE SURFACE SPECIFICATIONS

  • M1’s rewrite-dependent floor shifts agree with M2 and M3 within ≈0.005 BPB, while M1 achieves 99.3% of M3’s R2 using 15% of its parameters.Nested F tests reject M1 against M2 and M3; the authors use M1’s shared center.Source: §N.6.1 · p. 42

N.7 OPTIMAL REWRITE FRACTION ACROSS SOURCE QUALITY

This section examines optimal rewrite fractions across source quality, reporting that the estimated floor-optimal fraction is nonzero but smaller on DCLM than on FineWeb-Edu, and that DCLM has lower Paloma BPB in matched natural-only comparisons. The other supplied passages are illustrative or tutorial material and provide no additional findings about rewrite fractions or source quality.

N.7 OPTIMAL REWRITE FRACTION ACROSS SOURCE QUALITY

  • DCLM’s fitted natural-only data frontier scales as D*(C) ∝ C^0.55091, with 0.000989 BPB RMSE, compared with α = 0.20 and D* ∝ C^0.46 on FineWeb-Edu.The natural-only ladder includes 81 completed R = 0 runs across eight budgets from 3 × 10^18 to 10^22 FLOPs; its single 10^22-FLOP run is only an anchor in the fit.Source: §N.7 · p. 43
  • 0.078 is the fitted floor-optimal rewrite fraction on DCLM, where R = 0.1 minimizes Paloma BPB at both tested budgets and yields a 0.00577-BPB gain over R = 0.The gain over natural-only training is smaller than the best rewrite-fraction gains on FineWeb-Edu at the same budgets.Source: §N.7 · pp. 42, 44, Figure 27 · p. 44
  • DCLM has lower Paloma BPB than FineWeb-Edu in all 77 matched natural-only pairs, while the pairs differ somewhat in model size, token count, and tuned reference point.The DCLM models have 1.02–1.16× the parameters and 0.91–0.99× the tokens of their matched FineWeb-Edu runs.Source: §N.7 · pp. 42, 44

O QUALITATIVE REWRITE EXAMPLES

  • The selected qualitative examples span FAQ, table, tutorial, and math styles and rewriters from 550M to 3B parameters, illustrating errors rather than measuring fidelity.Reported errors include arithmetic mistakes, invented details, and a contradiction of the source; the loss results in Sections 3.2 and 3.3 do not depend on aggregate fidelity.Source: O QUALITATIVE REWRITE EXAMPLES · p. 47

O.0.4 1B: FAQ

  • Bioelectric therapy is described as using electrical currents to interrupt pain messages, with pacemakers and pregnancy listed as reasons to avoid it.The passages also describe it as a treatment for some chronic and acute pain conditions.Source: §O.0.4 · pp. 52–53

O.0.5 3B: TABLE

  • Pharyngitis and tonsillitis are commonly viral or bacterial, with sore throat as the main symptom; care may include antibiotics for bacterial infection or home symptom relief.The listed home measures include acetaminophen or ibuprofen, cool liquids, saltwater gargling, and throat-numbing spray.Source: §O.0.5 · p. 53

O.0.6 3B: TUTORIAL

  • The Hoover Institution’s stated mission includes recalling efforts to preserve peace and safeguarding the American way of life, while promoting individual freedom and private enterprise.The passage says its work includes collecting knowledge, generating ideas, and disseminating both.Source: §O.0.6 · pp. 54–55

O.0.7 1B: MATH

  • ZomeTools is presented as a geometry education system whose components incorporate the Fibonacci sequence and Golden Proportion and whose connectors provide 62 buildable directions.The passage describes its molded ABS plastic parts as tools for hands-on exploration of math, geometry, and science.Source: §O.0.7 · p. 57
  • The 1B math passages include a geometry description and fragments of a ZomeTools word problem, but no stated rewrite-scaling result.The problem fragments do not provide a coherent answer.Source: §O.0.7 · p. 57

O.0.8 3B: MATH

  • The 3B math solution says the publisher must order 720 copies of the teacher’s guide to cover the map costs.The preceding problem describes the second map as costing $24 per copy.Source: §O.0.8 · pp. 58–59

O.0.9 1B: TUTORIAL

  • The tutorial describes the Hoover Institution’s principles as individual, economic, and political freedom, private enterprise, and representative government.It says the Institution seeks to safeguard peace, improve the human condition, and limit government intrusion into individual lives.Source: §O.0.9 · p. 59

O.0.10 3B: FAQ

The FAQ describes bioelectric therapy as a non-invasive, drug-free option for some pain conditions. Electrical currents interrupt pain messages and stimulate endorphin production, but the treatment is not suitable for everyone.

Foundations

  • The FAQ lists chronic and acute pain uses including back and muscle pain, headaches and migraines, blood-flow disorders, TMJ syndrome, diabetic neuropathy, and skin pain or ulcers linked to poor circulation or scleroderma.Source: §O.0.10 · pp. 61–62
  • Electrical currents block pain messages before they reach the brain, while nerve stimulation prompts the body to produce pain-relieving endorphins.The FAQ describes the treatment as non-invasive and drug-free.Source: §O.0.10 · p. 62

How Bioelectric Therapy Works

  • Treatment may be received regularly as an outpatient or, in some cases, within a rehabilitation program.Source: §O.0.10 · p. 62

Preparation and Limitations

  • Bioelectric therapy is not recommended for people with a pacemaker, pregnancy, thrombosis, or a bacterial infection.Source: §O.0.10 · pp. 61, 63
  • The FAQ advises against using the therapy for chronic pain, back pain, muscle pain, or headaches and migraines that do not respond to other treatments.Source: §O.0.10 · p. 63

Related papers

Loading 2610.31867v1…