The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

1 Introduction

• SPT retains general pretraining knowledge more effectively during finetuning and improves downstream tasks; on domains far from web text, a 1B SPT model outperforms a 3B model trained without domain data during pretraining.Source: §1 · pp. 2–3
• Across MusicPile, ChemPile, and ProofPile, δ=1–5% SPT mixtures outperform δ=0 at all token scales and improve post-finetuning performance.The comparison uses 300M-token domain datasets.Source: Figure 2 · p. 2, §1 · p. 2
• Although a 1B SPT model costs more to train than finetuning a 3B model, its smaller serving cost can yield savings after about 1 trillion inference tokens.The paper frames this as an amortized compute and money break-even.Source: Figure 3 · p. 3, §1 · p. 3
• SPT reduces pretraining tokens needed to reach a given domain loss by up to 1.75×; at 200B tokens, it improves MATH accuracy by up to 6 percentage points and MusicTheoryBench by up to 4 percentage points over finetuning-only.Source: §1 · p. 3
• SPT helps most when the target domain is underrepresented; dataset size and pretraining duration also affect whether early or later domain-data introduction and larger or smaller mixtures are preferable.Source: §1 · p. 3
• Overfitting scaling laws model pretraining loss and the growing train-test gap, allowing practitioners to estimate an optimal domain fraction from pilot runs instead of exhaustive training sweeps.Source: §1 · p. 3

2 Specialized Pretraining Drives Domain Specific Capabilities

• The paper studies whether introducing domain data during pretraining improves the model obtained after finetuning, comparing SPT with standard pretraining followed by finetuning across three domains.Standard pretraining uses general web data; SPT includes a small domain-data fraction.Source: §2 · pp. 3–4

2.1 Notation and Experimental Setup

• SPT mixes a small fraction of domain data into general-data pretraining, then finetunes on the domain dataset; repeated domain examples are evaluated against a held-out test split.The experiments use three roughly 300M-token domain corpora and 200B-token pretraining runs; finetuning uses early stopping based on test loss.Source: §2.1 · p. 4, §2.2 · p. 5

2.2 Experimental Results

• SPT improves domain performance, convergence speed, and parameter efficiency across all three domains, with no observed trade-off among these axes.It reaches NPT’s best performance with 1.75× fewer pretraining tokens on MusicPile, 1.56× fewer on ProofPile, and 1.40× fewer on ChemPile; the 1B SPT model closes 133% of the 1B-to-3B gap on ProofPile.Source: Figure 4 · p. 4, §2.2 · pp. 5–6

2.3 SPT Learns More and Forgets Less

• SPT outperforms NPT on downstream tasks across all benchmarks and most pretraining budgets, while retaining more general knowledge after finetuning.For ChemPile, SPT models have lower domain and general loss after finetuning; the combined lower losses correlate with improved downstream performance.Source: Figure 5 · p. 5, §2.3 · p. 6

2.4 Specialized Pretraining Reduces Overfitting During Finetuning

• SPT reduces finetuning overfitting: mixed domain and general data regularize repeated domain exposure during pretraining, whereas domain-only finetuning overfits rapidly.Across domains, test loss begins rising after roughly five finetuning epochs; SPT models enter finetuning with lower domain loss and need less adaptation.Source: §2.4 · pp. 6–7, Figure 6 · p. 6

2.5 Key Takeaways

• SPT models retain general and domain performance together, because their lower starting domain loss requires less aggressive finetuning and entails less forgetting.The paper presents this dual benefit as central to SPT’s practical appeal.Source: §2.5 · p. 7

3.1.1 A controlled study of distributional overlap

• As Japanese monolingual data falls from 10% to 0.1%, SPT’s relative gain rises from approximately 2% to 5%, indicating larger gains with greater distributional misalignment.The gain plateaus at approximately 5%.Source: Figure 7 · p. 8, §3.1.1 · p. 8

3.1.2 Correlation Analysis Across Domains

• In the controlled Japanese sweep, all five overlap metrics correlate strongly with Rgain (|r| > 0.85), but only post-finetuning domain loss correctly ranks MusicPile, ProofPile, and ChemPile.Their Rgain values are 2.0%, 1.5%, and 0.8%, respectively.Source: §3.1.2 · p. 8

3.2 Domain Dataset Size

• At 40B tokens, SPT yields Rgain ≥0.5% across 3M-, 30M-, and 300M-token datasets; at longer horizons, gains persist for 300M tokens but can turn negative for smaller datasets.For 30M tokens, SCPT outperforms full SPT; for 3M tokens, SCPT at a 1% mixture still overfits.Source: §3.2 · pp. 8–9

3.3 Model Size

• On MusicPile-300M, the SPT–NPT test-loss gap widens with model size, with the 3B model showing the largest post-finetuning loss reduction under SPT.The comparison uses NPT with δ = 0% and SPT with δ = 2%.Source: Figure 8 · p. 9, §3.3 · p. 9

3.4 Pretraining compute budget

• At under 30B tokens, δ = 10% achieves the lowest test loss; by 200B tokens, δ = 2–5% performs best, while an appropriate SPT mixture outperforms NPT across budgets.Large mixtures begin to overfit as training progresses, shifting the optimal mixture toward smaller fractions.Source: Figure 9 · p. 10, §3.4 · p. 10

4 Does Specialized Pretraining Help Under Replay as Well?

• The comparison tests whether replaying general data during finetuning can match early domain exposure, using MusicPile test loss across finetuning trajectories.The replay rates are 0%, 10%, and 20%, with learning rates tuned separately for each setting.Source: §4 · p. 10
• Across 0%, 10%, and 20% replay, SPT→FT consistently achieves lower MusicPile test loss than NPT→FT; 10% replay helps NPT but does not close the gap.The figure shows the SPT→CPT versus NPT→CPT trajectories and reports the same persistent gap across replay rates.Source: §4 · p. 10, Figure 10 · p. 11
• The authors hypothesize that diffuse domain exposure during pretraining and general-data replay during finetuning have qualitatively different effects, but the precise mechanism remains open.Source: §4 · p. 11

5 Predicting Overfitting with Scaling Laws

• Predicting loss across specialized pretraining and finetuning is challenging because standard power laws do not anticipate overfitting and post-finetuning loss entangles both training stages.The approach models SPT train and test losses first, then separately models the test-loss difference before and after finetuning.Source: §5 · p. 11

5.1 Overfitting during Specialized Pretraining

• The model sums a decreasing training-loss power law and an increasing train–test-gap power law, with coefficients modeled as functions of mixture fraction δ.The gap exponent remains below 1 for mixture percentages up to 10%, so overfitting grows sublinearly over extended pretraining.Source: Figure 11 · p. 12, §5.1 · pp. 12–13

5.2 Predicting Test Loss After Finetuning

• The change in domain test loss after finetuning follows a power law in pretraining tokens, allowing the finetuning contribution to be modeled separately from pretraining loss.With early stopping, the difference is positive because the best finetuned loss is no greater than the pretrained loss.Source: §5.2 · p. 13

5.3 Forecasting Examples

• Using pilot runs with δ ∈ [0, 5], the scaling laws predict that 10% SPT begins overfitting around 70B tokens and that 2% SPT surpasses 5% SPT around 280B tokens.The forecasts track the observed 10% loss curve across 200B tokens and extrapolate post-finetuning loss beyond 120B tokens.Source: Figure 13 · p. 14, §5.3 · p. 14

6 Discussion

• A 1B SPT model costs more to train than finetuning a 3B model, but its smaller size lowers serving costs and can ultimately save compute and money.The discussion also links fewer deployed parameters to lower energy use and carbon footprint.Source: §6 · p. 14
• The appropriate strategy depends on domain-data quantity, distance from web text, serving constraints, model size, and pretraining budget.Overfitting scaling laws are intended to help navigate the tradeoff with a small number of pilot runs rather than exhaustive search.Source: §6 · pp. 14–15
• The paper rejects both “fine-tuning is always enough” and “always pretrain your model,” framing real systems as occupying a continuum of strategies.The continuum is described in terms of domain data, distance from web text, and serving constraints.Source: §6 · p. 14
• Across three domains, mixing 1–5% domain data into pretraining outperformed pretraining followed by finetuning, with gains persisting under replay-based continued pretraining.The discussion concludes that when domain data enters the pipeline has a lasting impact on performance.Source: §6 · p. 15

Contributions

• Christina Baek led the project and conducted the core experiments; Pratyush Maini provided direction and contributed to experimental design and analysis.Source: Contributions · p. 15
• David Schwab, Ricardo Monti, Aditi Raghunathan, Zico Kolter, Bogdan Gaza, Ari Morcos, and Matthew Leavitt provided guidance and draft feedback, while the Datology team supported discussions and experimental infrastructure.Source: Contributions · p. 15

A.1 Catastrophic Forgetting and Replay in Domain Adaptation

• Replay mixes previously seen general data into later training; prior work also reports that it can mitigate forgetting and improve target-task performance.The related-work passage notes that replay may help more when less target data is present in the pretraining mix.Source: §A.1 · p. 20
• SPT’s gains persist across replay settings, supporting the claim that replay during continued pretraining is not a substitute for early domain exposure.Source: §A.1 · p. 20

A.2 Scaling Laws for Data Mixing and Repeated Data

• The overfitting scaling laws model training loss and train-test gap with opposing power-law exponents, capturing non-monotonic test loss under heavy repetition and enabling extrapolation across mixture fractions and training horizons.The paper presents this as addressing a regime not covered by prior scaling-law formulations.Source: §A.2 · p. 20

A.3 The synergy between pretraining and post-training

• The related-work discussion identifies data-introduction timing as a common thread across safety, multilingual, and post-training research, and describes SPT as applying it through interleaved domain tokens.The passage characterizes interleaving as a regularization effect analogous to on-policy sampling.Source: §A.3 · p. 21

A.4 Specialized Pretraining in Deployed Systems

• Industry examples span custom pretraining and post-training; the paper frames the choice around domain-data scale, distance from web distributions, and deployment requirements.The stated goal is to characterize when additional pretraining is warranted and when finetuning suffices.Source: §A.4 · p. 21

B.1 Pretraining Hyperparameter Configurations

• OLMo-1B pretraining matches documented settings where possible; context length is 2048, with cosine learning rate from 4e-4 to 4e-5 over 2T tokens, cut off at 200B.Source: §B.1 · p. 22

B.2 Finetuning Hyperparameter Configurations

• Finetuning ablates learning rate and warmup while keeping other settings fixed; batch size is 512, weight tying is off, and the scheduler is constant with warmup.Source: §B.2 · p. 22

C Construction of Domain Datasets

• MusicPile is subsampled from music-specific sources and includes substantial musical-composition testing in ABC notation alongside general music knowledge.Source: C Construction of Domain Datasets · p. 22
• ChemPile is subsampled from its reasoning, instruction, and education domains.Source: C Construction of Domain Datasets · p. 22
• ProofPile is subsampled from the full ProofPile dataset.Source: C Construction of Domain Datasets · p. 22

D.1 Justification for Agap and Ctrain

• The learned coefficients roughly follow a Gamma kernel for Aδ and a log-linear function for Cδ, with final scaling-law fits jointly optimizing A(δ) and C(δ).Separate coefficients were first fit to power-law curves for each SPT run; Figures 15–17 show the final fits.Source: §D.1 · p. 23

D.2 Modeling Difference in Test Loss Post-Finetuning Across PT Tokens

• Figure 18 depicts ΔTest following a power-law relationship with pretraining steps.Source: Figure 18 · p. 27

E.1 Metrics Description

• Five metrics capture different aspects of domain-to-corpus difference: JSD over n-grams, embedding-based MAUVE, and classifier-based C2ST.JSD uses unigram, bigram, and trigram distributions; MAUVE compares GPT-2 Large embeddings, while C2ST uses sentence embeddings and classification AUC.Source: §E.1 · p. 28

E.2 Experimental Setup

• The Japanese overlap sweep varies Japanese web-text share from 0% to 10% against a held-out English-Japanese translation corpus, while cross-domain analysis compares three specialized domains with Dolma.The domains are chemistry literature, ABC-format symbolic music notation, and formal mathematical proofs.Source: §E.2 · pp. 28–29

Japanese Overlap Sweep

• In the Japanese overlap sweep, all five metrics correlate strongly with Rgain (Pearson |r| > 0.85), with higher divergence or separability and lower MAUVE similarity associated with higher gain.Figure 20 plots each metric against Rgain.Source: Figure 20 · p. 29, §E.3 · p. 29

Cross-Domain Analysis

• C2ST’s reversed cross-domain pattern may reflect sentence embeddings capturing surface similarity: MusicPile has the lowest AUC (0.988) yet the highest Rgain.ABC music notation may resemble web text in embedding space because both use ASCII characters and structured patterns.Source: §E.3 · pp. 29–30
• Across ChemPile, MusicPile, and ProofPile, JSD correlations with Rgain are weak (r = 0.32 to 0.57), MAUVE is weakly negative (r = −0.23), and C2ST AUC reverses to r = −0.98.Figure 21 compares these cross-domain metric correlations.Source: §E.3 · p. 29, Figure 21 · p. 30

• Up to 5% domain-data mixture has a marginal impact on Dolma pretraining loss, while 10% mixture produces notable degradation.Source: F General Pretraining Loss during SPT · p. 30