Overtrained Language Models Are Harder to Fine-Tune

Central claim

• Catastrophic overtraining challenges the assumption that scaling pre-training data is an unambiguous improvement after instruction tuning or other forms of post-training.The paper frames the phenomenon as a general limitation of the widely adopted strategy of scaling language-model pre-training.Source: §1 · pp. 1–2

Real-world evidence

• 3% worse on AlpacaEval and 2% worse on ARC, OLMo-1B trained on 3T tokens performs below its 2.3T-token checkpoint after post-training.The phenomenon also appears after instruction tuning and multimodal tuning across multiple models and tasks.Source: §1 · pp. 1–2, Figure 2 · p. 2

Controlled experiments

• Extended pre-training increases progressive sensitivity, making equally large parameter modifications cause greater forgetting of capabilities acquired during pre-training.Catastrophic overtraining occurs when this additional forgetting outweighs pre-training improvements; limiting update magnitude can mitigate degradation but restrict adaptation.Source: §1 · p. 2

Theoretical analysis

• A linear transfer-learning analysis shows that incremental feature learning produces progressive sensitivity and inevitable catastrophic overtraining.Fine-tuning regularization can delay its onset, but at the cost of downstream performance.Source: §1 · p. 2

2.1. Experimental setup

• The study evaluates intermediate checkpoints from OLMo-1B, OLMo-2-7B, and LLM360-Amber-7B using Anthropic-HH and TULU instruction tuning plus LLaVA visual instruction tuning.Models are selected by best in-distribution task performance after learning-rate tuning and evaluated on ten OOD benchmarks spanning reasoning, QA, commonsense, and knowledge extraction.Source: §2 · p. 2, §2.1 · p. 3

2.2. Results

• Up to 3% lower AlpacaEval response rates follow instruction tuning at 3T versus 2.3T pre-training tokens, with OOD performance falling to the level of 1.5T-token models.The degradation appears on reasoning and question-answering benchmarks including ARC-Easy, ARC-Challenge, HellaSwag, and PIQA.Source: §2.2 · p. 3
• Extended pre-training consistently improves base models, but these gains do not reliably transfer after post-training and can instead produce active performance losses.The comparison spans models trained to different pre-training budgets, as shown for OLMo-1B in Figure 2.Source: §2.2 · p. 3
• Multimodal fine-tuning improves VLM scores with extended pre-training but causes greater forgetting and larger OOD drops, severely hurting PIQA performance in some cases.Thus, additional pre-training can actively reduce post-training performance even when the target multimodal score continues improving.Source: §2.2 · p. 3

3.1. Catastrophic overtraining in the real-world

• OLMo-1B exhibits catastrophic overtraining beyond 2.5T tokens for instruction and multimodal fine-tuning, while OLMo-7B shows none through 3T tokens.Instruction tuning degrades both in-domain and out-of-domain performance; multimodal fine-tuning degrades some out-of-domain tasks but not VLM score at this threshold.Source: §3.1 · p. 4

3.2. Catastrophic overtraining in a controlled setup

• In controlled experiments, pre-training models from 4B to 128B tokens and fine-tuning across QA, sentiment, math, and code reveals how modifications can distort retained pre-training knowledge.The study uses C4 perplexity to track retained pre-training knowledge and held-out in-domain perplexity to measure fine-tuning performance, favoring smoother metrics over accuracy.Source: §3.2 · p. 4

3.3. Warmup: Gaussian perturbations

• For fixed Gaussian perturbation magnitude, sensitivity increases monotonically with pre-training tokens while base-model perplexity improves, yielding a U-shaped perturbed-perplexity curve.Larger perturbations accelerate the inflection point, whereas smaller perturbations may delay or prevent observable catastrophic overtraining.Source: §3.3 · p. 5

3.4.1. FINE-TUNING WITH FIXED LEARNING RATE

• For fixed learning rates, fine-tuning sensitivity also increases monotonically with pre-training tokens, and sufficiently large rates produce U-shaped C4 perplexity after fine-tuning.Larger learning rates cause greater degradation and lower-token inflection points, while ID perplexity can improve at the cost of C4 degradation.Source: §3.4.1 · p. 6, Figure 5 · p. 6

3.4.2. BALANCING FINE-TUNING GAINS WITH DEGRADATION

• After tuning learning rates for in-domain performance, extended pre-training can still degrade C4 or ID perplexity, depending on the task.C4 perplexity degrades for GSM8k, Starcoder-Python, MR, and RTE, while ID perplexity degrades for RTE and TREC.Source: Figure 6 · p. 7, §3.4.2 · p. 7
• The optimal learning-rate trajectory determines whether extended pre-training degrades both ID and OOD performance, only OOD performance, or neither.A constant rate degrades both metrics; a slowly decreasing rate can preserve ID gains while harming OOD performance; a quickly decreasing rate can improve both.Source: §3.4.2 · p. 7, Figure 7 · p. 8
• Using a non-optimal smaller learning rate can delay catastrophic overtraining, but it reduces in-domain fine-tuning performance.The section argues that explicit regularization limiting parameter updates may similarly mitigate or delay degradation.Source: §3.4.2 · p. 7
• Progressive sensitivity under Gaussian noise and structured fine-tuning eventually outweighs extended pre-training gains, producing catastrophic overtraining across downstream evaluations.For fixed perturbation magnitude or fine-tuning learning rate, degradation increases with pre-training tokens; optimal learning-rate evolution determines whether in-domain or out-of-domain performance declines.Source: §3.3 · p. 5, §3.4.2 · pp. 7–8

4.1. Pre-training setting

• The theory models pre-training as gradient-flow learning of a full-rank linear map’s singular-value features, which are acquired incrementally over ordered timesteps.Catastrophic overtraining is studied through the inflection point, the earliest timestep at which a post-training modification worsens pre-training loss.Source: §4.1 · pp. 8–9

4.2. Gaussian perturbation setting

• Gaussian parameter perturbations inevitably produce catastrophic overtraining: beyond an inflection point r, perturbed pre-training loss increases monotonically with further pre-training.The inflection point is the smallest n satisfying 2dγ2 > σpre_n, and smaller or faster-decaying features induce smaller inflection points.Source: §4.2 · p. 9

4.3. Fine-tuning

• Fine-tuning causes progressive sensitivity: under (α, 1)-misalignment with α > 1, pre-training loss change is nonnegative and increases with learned pre-training features.This establishes that elongating pre-training systematically increases degradation during fine-tuning in the analyzed setting.Source: §4.3 · p. 9
• Regularization can delay the inflection point for sufficiently distinct downstream tasks, but limiting parameter updates can reduce downstream fine-tuning performance.Without regularization, catastrophic overtraining with respect to pre-training loss is inevitable in the stated setting.Source: §4 · p. 8, §4.3 · p. 10

No regularization

• Without regularization, for sufficiently misaligned pre-training and downstream tasks, pre-training loss after fine-tuning monotonically increases after a threshold number of steps.This establishes catastrophic overtraining when λ = 0, for α > α0 and n ≥ r.Source: §1 · p. 10

Regularization

• Regularization can delay pre-training degradation but simultaneously limits downstream adaptation and can harm downstream performance.The inflection point and unregularized fine-tuning loss increase monotonically with regularization strength λ.Source: §1 · p. 10

Setting and mechanism

• Progressive sensitivity and catastrophic overtraining arise in linear networks that learn task-related features incrementally.The analysis considers post-training modifications including Gaussian perturbations and fine-tuning on a canonical task.Source: §1 · p. 10

Main results

• Elongating pre-training inevitably results in progressive sensitivity and catastrophic overtraining, including under fine-tuning (Theorems 4.4, 4.6 and 4.7).The results characterize the inflection point under Gaussian perturbations and fine-tuning on a canonical task.Source: §1 · p. 10

Loss of plasticity

• Unlike prior small-scale or reinforcement-learning studies, this work shows that generic-web overtraining can degrade fine-tuning and increase sensitivity to pretraining-loss degradation.Prior work mainly examined loss of plasticity in small models, small datasets, synthetic settings, or reinforcement learning.Source: §5 · p. 10

Catastrophic forgetting

• Overtraining can make catastrophic forgetting more severe, extending documented sequential-task forgetting to this setting.Existing mitigation strategies include regularization, generative replay, and memory buffers.Source: §5 · p. 10

Relationship between pre-training loss and downstream performance

• The work relates pretraining-loss degradation to downstream-loss degradation, while distinguishing its web-scale, rarely converged regime from studies finding improved transfer after convergence.Other work also studies how architecture affects downstream generalization at matched pretraining loss.Source: §5 · pp. 10–11

Scaling laws for optimal pre-training

• Unlike related scaling-law studies, this work argues that fewer pretraining tokens can improve fine-tuned downstream performance, including degradation from overtraining.Prior research identifies compute-optimal token budgets, extends scaling laws across settings, and reports limits or U-shaped trends in downstream prediction.Source: §5 · p. 11
• Overtraining beyond compute-optimal budgets is increasingly used for capable small models, even as pretraining loss can continue decreasing to 10,000 tokens per parameter.Sardana et al. (2024) report continued loss reduction, while Gadre et al. (2024) propose scaling laws for this regime.Source: §5 · p. 11

Transfer learning theory

• The theoretical analysis builds on deep-linear transfer-learning theory used to study knowledge storage, transfer, feature learning, and fine-tuning-induced out-of-distribution degradation.The cited literature examines how pretrained features benefit downstream tasks and how fine-tuning can reduce out-of-distribution performance.Source: §5 · p. 11

6. Discussion

• Catastrophic overtraining arises as longer pre-training increases parameter sensitivity, with degradation especially severe under task misalignment and potentially unavoidable despite regularization.The discussion identifies broader perturbation sensitivity as the underlying mechanism and notes that misaligned pre-training and fine-tuning can make degradation inevitable.Source: §6 · p. 11
• The study leaves the roles of optimizer, pre-training distribution, and training objective unresolved, while regularization mitigates catastrophic overtraining only sometimes.Potential alternatives include data replay, LP-FT, and WiseFT, but their effectiveness in this context remains open.Source: §6 · p. 11
• Catastrophic overtraining may extend beyond fine-tuning to model editing and unlearning, while parameter-efficient deployment and increasingly costly inference-time reasoning may amplify its effects.The authors call for scaling decisions that account for the entire training pipeline rather than pre-training alone.Source: §6 · p. 11

A.1. Formal Definitions and Assumptions

• The formal analysis models a two-layer linear network θ = W1W2 trained on a matrix pre-training task, then fine-tuned on a shared-singular-vector task with a different spectrum.Pre-training follows population-loss gradient flow from small equal identity initializations, while fine-tuning uses finite-batch gradient descent from the pre-trained iterate.Source: §A.1 · p. 17

A.2. Formal Statement and Proof of Theorem 4.4

• The appendix defines Gaussian weight perturbations by independently adding isotropic noise matrices to both factors before evaluating perturbed pre-training loss.The perturbations take the form (W1 + α)(W2 + β), with independent entries distributed as N(0, γ2).Source: §A.2 · p. 18
• Under sufficiently small initialization, Gaussian-perturbed pre-training loss is monotonically increasing after a finite pre-training stage.The theorem establishes that for some s < d, the perturbed loss increases for all n > s, specifically with ˜Lpre(θn(0)) − ˜Lpre(θn−1(0)) > 0.Source: §A.2 · p. 18
• The proof compares real and ideal pre-training initializations, bounding their perturbed-loss difference through Frobenius-norm factorization and the small-initialization assumption.The argument uses Gaussian isotropy, independence, zero mean, Lemma A.5, and Lemma A.6 to transfer monotonicity from the ideal to the real initialization.Source: §A.2 · pp. 19–20

A.3. Dynamic Analysis of Finetuning Process

• The analysis introduces ideal-initialization and infinite-batch auxiliary dynamics, then shows real finetuning closely tracks the infinite-batch ideal and approximately converges to the minimum.The infinite-batch dynamic uses the same ideal initialization, while the finite-batch dynamic follows the original update structure.Source: §A.3 · p. 21

A.3.1. ANALYTICAL EXPRESSION FOR THE IDEAL INITIALIZATION DYNAMIC WITH INFINITE BATCH SIZE

• The infinite-batch ideal dynamic admits an analytical expression, with its transformed weight matrices remaining well bounded by Γ under bounded learning rate and regularization.The analysis also studies convergence of transformed diagonal entries under a sufficiently large finite number of steps.Source: §A.3.1 · pp. 21–22

A.3.2. CORRESPONDENCE BETWEEN IDEAL INITIALIZATION DYNAMIC WITH INFINITE BATCH SIZE AND FINITE BATCH SIZE

• For sufficiently large batches, the finite-batch ideal dynamic remains close to the infinite-batch dynamic with probability 1−δ and an error controlled by batch size.The correspondence is established through concentration and stability bounds across the finetuning steps.Source: §A.3.2 · pp. 22–24

A.3.3. ERROR INCURS BY DIFFERENT INITIALIZATION

• The real-initialization finetuning process stays close to the ideal-initialization process, with the error depending on the pretraining initialization scale.This comparison holds across all k≤K under the stated event and conditioning requirements.Source: §A.3.3 · p. 24

A.3.4. COMBING TWO APPROXIMATIONS

• Assumption A.21 requires finite regularization force, a bounded finetuning learning rate, a finite number of steps K, and a large enough batch size.Source: §A.3.4 · p. 25

A.4. Formal Statement and Proof of Theorem 4.6

• Theorem A.24 shows with probability 1−δ that Δpre(n) is nonnegative and does not decrease with n.Here Δpre(n) = Lpre(θn(K))−Lpre(θn(0)).Source: §A.4 · p. 26
• The proof of Theorem A.24 combines the bounds from Lemmas A.22 and A.23 to obtain a positive lower bound for Δpre(n) for 1≤n≤d.Defining Δ0=0 completes the monotonicity argument.Source: §A.4 · p. 26

A.5. Formal Statement and Proof of Theorem 4.7

• Under Assumption A.21, when λ = 0 and Apre and Aft are (4, r)-misaligned, Lpre(θn(K)) − Lpre(θn−1(K)) > 0 with probability 1 −δ for n ≤r.This is the stated theorem condition and conclusion.Source: §A.5 · p. 27

A.6. Technical Lemmas

• The technical lemmas establish that f(x; η, λ, σ, σ0) −σ∗ has the same sign as x −σ∗ within the specified interval.The proof uses the auxiliary function g and bounds on x in [min{σ, σ0}, max{σ, σ0}].Source: §A.6 · p. 28

B.1. Pre-trained models.

• The experiments use released checkpoints from OLMo-1B, OLMo-2-7B, and LLM360-Amber.The checkpoints are selected from the models’ HuggingFace pages and listed in Table 1.Source: §B.1 · p. 29, Table 1 · p. 29

B.2. Fine-tuning setup.

• Fine-tuning evaluates instruction tuning and multimodal tuning as two common post-training paradigms.Instruction tuning uses Anthropic-HH and a compute-efficient 200k-example subset of TULU 1.0; LLaVA provides the multimodal framework.Source: §B.2 · p. 29
• Anthropic-HH contains 180k instructions and responses, while the TULU 1.0 SFT mixture contains 490k and contributes a randomly selected 200k subset.Anthropic-HH is adapted for standard instruction tuning by treating each chosen response as the target.Source: §B.2 · p. 29
• LLaVA training first fine-tunes a vision-language adapter and then fine-tunes the entire model to follow instructions with images.The setup uses standard SFT hyperparameters for instruction tuning and LLaVA defaults for unspecified hyperparameters, as reported in Table 2.Source: §B.2 · p. 29

B.3. Evaluations

• Evaluations distinguish downstream tasks representative of fine-tuning goals from generalist tasks measuring broader language understanding and inference.Source: §B.3 · p. 29
• AlpacaEval measures instruction-response win-rate against a same-architecture reference model fine-tuned on the same dataset, with LLaMA-3-70B-Instruct as evaluator.The reference uses seed 0 and learning rate 10−5; its score is 50% by definition, and scores are not comparable across graphs.Source: §B.3 · pp. 29–30
• AlpacaEval contains 20k instructions and is designed to evaluate the quality of a model’s response to an instruction.Source: §B.3 · p. 29
• The downstream LLaVA metric, “VLM score,” averages MME, GQA, AI2D, POPE, and TextVQA.Source: §B.3 · p. 30
• Generalist evaluation reports 5-shot performance across ten benchmarks covering reasoning, commonsense, natural language inference, and sentence completion.Source: §B.3 · p. 30

C. Experimental Details from Section 3: Controlled Experiments

• This subsection provides additional experimental details for the controlled experiments presented in Section 3.Source: C. Experimental Details from Section… · p. 30

C.1. Pre-training and fine-tuning setup.

• All controlled experiments use the OLMo codebase and μP parameterization.The implementation choices follow Groeneveld et al. (2024b) and Yang et al. (2022).Source: §C.1 · p. 30
• The controlled pre-training study trains OLMo-15M, OLMo-30M, and OLMo-90M models for 4B–128B tokens on OLMo’s pre-tokenized C4 high-quality web data.Experiments use 8×A100 GPUs, and all models use cosine learning-rate decay to zero.Source: §C.1 · p. 30
• Pre-trained models are fine-tuned on nine classification datasets and two language-modeling tasks, including GSM8k treated as math-style text modeling.The downstream suite includes SUBJ, BoolQ, MR, CR, RTE, TREC, English Tweet sentiment, SIQA, GSM8k, and Starcoder-Python; Starcoder-Python uses 5k examples.Source: §C.1 · pp. 30–31
• Evaluation tunes on validation splits but reports test-set scores, using class likelihood perplexity for classification, standard language-modeling loss for generative tasks, and held-out C4 perplexity for generalist capability.Figure 5 visualizations restrict plotted learning rates to those whose maximum pre-training perplexity remains below 6 across token budgets.Source: §C.1 · p. 31
• Fine-tuning distance is approximately proportional to learning rate and independent of pre-training token budget for OLMo-30M.This distance is measured by the L2 norm between pre-trained and fine-tuned parameters in Figure 8.Source: §C.1 · p. 31, Figure 8 · p. 32

C.2. Gaussian perturbations.

• Gaussian perturbations sample mean-zero random values for each parameter and assess degradation in pre-training perplexity.Perturbation magnitudes are scaled approximately with each layer’s parameter magnitude while normalized to the layer’s initialization magnitude, keeping magnitude comparable across token budgets.Source: §C.2 · p. 32

• Progressive sharpening describes increasing Hessian spectral norms during fixed-learning-rate training, with full-batch gradient descent approaching 2/η; other Hessian norms can behave similarly.Source: D. Connection Between Progressive… · p. 32
• For quadratic losses, Gaussian-perturbation sensitivity is exactly linked to the Hessian and remains closely related to progressive sharpening under a second-order approximation.The relationship applies generally when the loss is well-approximated by its Taylor expansion, such as when λ is small.Source: D. Connection Between Progressive… · pp. 32–33
• Large perturbations produce losses substantially above Hessian-based quadratic predictions for both Gaussian and fine-tuning perturbations, so progressive sharpness alone cannot explain sensitivity.For Gaussian perturbations, the approximation breaks down at large λ; the same occurs when interpolating toward an AG News fine-tuned model.Source: D. Connection Between Progressive… · p. 33
• Progressive sensitivity generalizes progressive sharpness by describing a global increase in perturbation sensitivity beyond what increasing Hessian eigenvalues and quadratic approximations predict.Source: D. Connection Between Progressive… · p. 33

E. Omitted Figures from Section 2: Large Model Experiments

• The large-model appendix provides omitted extended-experiment figures spanning Alpaca, Anthropic-HH, TULU, and VLM across OLMo-1B, OLMo-2-7B, and LLM360-7B.Table 5 organizes the figure references by dataset, model, and learning-rate tuning variant.Source: E. Omitted Figures from Section 2:… · p. 34, Table 5 · p. 34
• The Anthropic-HH figures evaluate OLMo-1B, OLMo-2-7B, and LLM360-7B across eight datasets as pre-training tokens increase, using both tuned and all-learning-rate views.AlpacaEval is the main downstream evaluation, while the remaining datasets measure out-of-distribution generalist performance; base models are included in the all-learning-rate views.Source: Figure 11 · p. 35, Figure 12 · p. 36, Figure 13 · p. 37
• The TULU figures apply the same eight-dataset evaluation framework to OLMo-1B, OLMo-2-7B, and LLM360-7B, with tuned-learning-rate and all-learning-rate variants.The all-learning-rate figures also compare against the base model, and AlpacaEval is the main evaluation of interest.Source: Figure 17 · p. 41, Figure 18 · p. 42, Figure 19 · p. 43
• The VLM figures evaluate the three models over increasing pre-training tokens on VLM Score and seven out-of-distribution datasets, using tuned and all-learning-rate analyses.VLM Score is the main evaluation, while the all-learning-rate views compare against the base model.Source: Figure 23 · p. 47, Figure 24 · p. 48, Figure 25 · p. 49

F. Omitted Figures from Section 3: Controlled Experiments

• The controlled-experiment appendix supplies omitted figures from Section 3 to extend the paper’s controlled analyses.These figures are presented as supplements to the main controlled experiments rather than as a separate experimental setup.Source: F. Omitted Figures from Section 3:… · p. 53

F.1. Sensitivity

• In the controlled setup, Figure 29 plots fine-tuned-model sensitivity at a fixed learning rate as a function of pre-training tokens for OLMo-30M.Sensitivity is measured by the perplexity difference between the fine-tuned model and the base model.Source: Figure 29 · p. 53, §F.1 · p. 53
• Sensitivity progressively increases throughout training across all datasets, so corresponding figures for the other controlled settings are omitted because they are sufficiently similar.Figure 29 supplements Figure 6 from the main paper and illustrates the same progressive trend in the fixed-learning-rate setting.Source: §F.1 · p. 53, Figure 29 · p. 53

F.2. Extended fine-tuning experiments.

• F.2 extends controlled fine-tuning experiments by ablating batch size, learning-rate scheduler, and model size, with Table 6 indexing the associated figures.The extended experiments report results through figures covering these experimental settings.Source: §F.2 · p. 53, Table 6 · p. 53
• For OLMo-30M, batch-size experiments compare batch sizes 256 and 32 using pre-training and fine-tuning perplexity across pre-training budgets.The figures also include learning-rate-tuned variants and the optimal learning rate for each budget.Source: Figure 30 · p. 54, Figure 31 · p. 54, Figure 32 · p. 55
• The OLMo-30M scheduler experiments compare constant learning-rate schedules, with and without warmup, against the configuration using a Cosine scheduler.Each setting reports pre-training and fine-tuning perplexity, plus tuned-learning-rate variants and corresponding optimal learning rates.Source: Figure 40 · p. 59, Figure 41 · p. 59, Figure 42 · p. 60
• The extended figures evaluate OLMo-15M and OLMo-30M across pre-training budgets, showing pre-training and fine-tuning perplexity with fixed hyperparameters and pre-training perplexity with a fine-tuning-optimal learning rate.Source: Figure 50 · p. 64, Figure 51 · p. 64, Figure 52 · p. 65
• The model-size extensions provide analogous fixed- and tuned-learning-rate analyses for OLMo-15M, OLMo-30M, and OLMo-90M, extending the main-paper fine-tuning figures.The corresponding figures explicitly identify their relationships to Figures 5 and 6 from the main paper.Source: Figure 50 · p. 64, Figure 52 · p. 65, Figure 55 · p. 66

F.3. Extended Gaussian perturbations experiments.

• F.3 tests Gaussian perturbations on OLMo-15M, OLMo-90M, OLMo-1B, OLMo-2-7B, and LLM360-Amber (7B) across pre-training token budgets.Each parameter receives Gaussian noise scaled by λ, and Figure 65 reports the resulting C4 web data perplexity; further setup details are in Appendix C.Source: §F.3 · p. 72, Figure 65 · p. 72