Overtrained Language Models Are Harder to Fine-Tune

Jacob Mitchell Springer; Sachin Goyal; Kaiyue Wen; Tanishq Kumar; Xiang Yue; Sadhika Malladi; Graham Neubig; Aditi Raghunathan

Source-linked AI summary

Overtrained Language Models Are Harder to Fine-Tune

Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, Aditi Raghunathan

arXiv:2503.19206v2Mar 24, 2025cs.CLcs.AI

TL;DR

The paper asks whether ever-larger pre-training budgets reliably improve post-trained language models. Through experiments and theoretical analysis, it identifies catastrophic overtraining, finding that 3T-token OLMo-1B performs 3% worse on AlpacaEval and 2% worse on ARC than a 2.3T-token counterpart. The results attribute this degradation to progressive parameter sensitivity and leave several pre-training factors for future study.

Problem
The paper examines the gap between improving pre-training performance and reliably improving downstream models after post-training.
Method
The authors combine evaluations across models and post-training settings with controlled sensitivity experiments and a linear transfer-learning analysis.
Results
3T-token OLMo-1B performs 3% worse on AlpacaEval and 2% worse on ARC than a 2.3T-token checkpoint after post-training, with degradation appearing across models and tasks.
Takeaways & Limitations
Pre-training design should consider downstream adaptability because longer pre-training can increase sensitivity to later parameter modifications and degrade post-trained performance.
Takeaways & Limitations
The study leaves the effects of optimizer choice, pre-training distribution, and training objective on overtraining severity for future work.

Abstract

from arXiv · show

Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon catastrophic overtraining. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model.

1. Introduction

The paper challenges the assumption that scaling pre-training is uniformly beneficial, showing that longer pre-training can harm post-training performance through increasing parameter sensitivity. It combines empirical evidence, controlled experiments, and theory to characterize catastrophic overtraining.

1. Introduction: Longer pre-training can harm final performance after instruction tuning or other post-training, a phenomenon termed catastrophic overtraining.The paper frames this as a challenge to the assumption that scaling pre-training data is an unambiguous win.
1. Introduction: 3T-token OLMo-1B performs 3% worse on AlpacaEval and 2% worse on ARC than a 2.3T-token checkpoint after post-training.The comparison is made between an OLMo-1B model trained on 3T tokens and an intermediate checkpoint trained on 2.3T tokens.
1. Introduction: Controlled experiments identify progressive sensitivity: longer-pretrained models forget more when subjected to parameter modifications of equal magnitude.The authors connect this increased forgetting to the degradation observed after post-training.
1. Introduction: A linear transfer-learning analysis shows that incremental feature learning produces progressive sensitivity and inevitable catastrophic overtraining.Regularization can delay its onset, but at the cost of downstream performance.
1. Introduction: The phenomenon appears across existing models and tasks, including instruction tuning and multimodal fine-tuning.The authors describe catastrophic overtraining as prevalent rather than isolated to one model or task.

2. Extended pre-training can hurt post-training

The study evaluates extended pre-training across instruction-tuning and multimodal fine-tuning setups. Although base-model performance consistently improves, post-trained performance can decline, including severe degradation on some out-of-distribution tasks.

2. Extended pre-training can hurt post-training: The study evaluates instruction tuning and multimodal fine-tuning across OLMo-1B, OLMo-2-7B, and LLM360-Amber-7B checkpoints.Instruction tuning uses Anthropic-HH and TULU, while multimodal fine-tuning uses the LLaVA framework.
2. Extended pre-training can hurt post-training: Models are evaluated on both in-distribution fine-tuning tasks and out-of-distribution benchmarks spanning reasoning, question answering, commonsense, and knowledge extraction.For each checkpoint, the learning rate is tuned and the model with the best in-distribution performance is selected.
2. Extended pre-training can hurt post-training: Extended pre-training consistently improves base-model performance on all evaluated downstream tasks.This improvement is represented by the dashed curve in Figure 2.
2. Extended pre-training can hurt post-training: 3T-token pre-training yields up to 3% lower AlpacaEval response rate than 2.3T-token pre-training after instruction tuning.The 3T-token models also underperform on reasoning and question-answering benchmarks, reaching the level of models trained on 1.5T tokens.
2. Extended pre-training can hurt post-training: In multimodal fine-tuning, extended pre-training improves VLM score but increases forgetting and can actively hurt performance on OOD benchmarks such as PIQA.The gap between base and fine-tuned models also grows on several OOD benchmarks.
2. Extended pre-training can hurt post-training: Across additional models and fine-tuning setups, pre-training gains do not always translate into post-training gains.The authors report settings in which extended pre-training actively hurts post-training performance.

3. Catastrophic overtraining

Catastrophic overtraining occurs when extended pre-training improves the base model but progressively increases its sensitivity to later modifications, eventually degrading post-training or pre-training performance. The phenomenon appears across perturbation, fine-tuning, instruction-tuning, and multimodal settings, with outcomes depending partly on learning-rate evolution.

Extended pre-training beyond a token-budget threshold can hurt final performance after subsequent model modifications, a phenomenon termed catastrophic overtraining.
The paper’s scope excludes further modifications such as reinforcement learning and pruning, which are left for future work.
Instruction-tuned OLMo-1B models exhibit catastrophic overtraining beyond 2.5T tokens, degrading both in-domain and out-of-domain task performance.
Multimodal fine-tuning also degrades beyond 2.5T tokens on some out-of-domain tasks, while in-domain VLM performance shows no degradation at that threshold.
Under matched setups, OLMo-7B shows no catastrophic overtraining up to 3T pre-training tokens.
For fixed perturbation magnitude or learning rate, progressive sensitivity increases degradation as pre-training tokens grow, eventually outweighing base-model improvements.
Larger perturbations or learning rates accelerate the inflection point and intensify degradation, producing U-shaped post-modification perplexity trends.
After tuning for in-domain performance, degradation can remain in either in-domain or out-of-domain metrics, depending on how the optimal learning rate changes with pre-training.

4. A theoretical perspective of overtraining

The paper analyzes catastrophic overtraining in two-layer linear networks, where incremental feature learning increases sensitivity to post-training modifications. It characterizes when fine-tuning or perturbations degrade pre-training performance and how regularization changes this trade-off.

4.3. Fine-tuning: Regularization limits parameter deviation and can delay catastrophic overtraining, but it can also reduce downstream adaptation and fine-tuning performance.The trade-off follows because regularization simultaneously constrains degradation of pre-training performance and adaptation to the downstream task.
4.1. Pre-training setting: The theoretical setting uses two-layer linear regression whose pre-training model incrementally learns the singular-value features of a full-rank linear map.The network is trained with gradient flow on Gaussian inputs, allowing the evolution of learned features to be tracked over time.
4.1. Pre-training setting: The inflection point is the earliest pre-training stage at which a post-training modification makes pre-training loss increase rather than continue improving.The analysis studies this point for Gaussian parameter perturbations and fine-tuning on canonical downstream tasks.
4.2. Gaussian perturbations: For Gaussian perturbations, newly learned features create additional directions in which perturbations degrade loss, and degradation increases monotonically after the inflection point.The inflection point is characterized by the smallest n satisfying 2dγ2 > σpre_n; smaller or faster-decaying features produce an earlier inflection point.
4.3. Fine-tuning: For a sufficiently misaligned downstream task, fine-tuning produces progressive sensitivity: the pre-training loss change increases monotonically with the number of learned pre-training features.The analysis tracks the diagonal feature coordinates after fine-tuning and defines progressive sensitivity through the monotonicity of ∆pre(tn).
4.3. Fine-tuning: Without regularization, sufficiently distinct fine-tuning tasks cause pre-training performance to degrade as pre-training is extended.The theoretical results establish catastrophic overtraining under task misalignment and characterize it through the degradation rate relative to pre-training improvement.

1. Catastrophic overtraining is inevitable without regularization.

The theoretical results show that catastrophic overtraining is inevitable for sufficiently distinct downstream tasks without regularization. Regularization can postpone degradation, but the protection comes at a cost to downstream performance.

1. Catastrophic overtraining is inevitable without regularization.: For sufficiently misaligned tasks and λ = 0, pre-training loss after fine-tuning increases monotonically once the learned-feature index reaches r.This establishes catastrophic overtraining under the stated task-misalignment condition.
1. Catastrophic overtraining is inevitable without regularization.: Regularization delays the inflection point, while the corresponding unregularized fine-tuning loss increases with the regularization strength.The result formalizes a trade-off between preserving pre-training performance and adapting to the downstream task.
1. Catastrophic overtraining is inevitable without regularization.: The degradation rate is identified by comparing the increase in fine-tuning-induced pre-training loss with the rate of improvement from additional pre-training.Catastrophic overtraining begins when degradation outpaces pre-training improvement.
1. Catastrophic overtraining is inevitable without regularization.: The linear-network analysis concludes that extending pre-training inevitably produces progressive sensitivity and catastrophic overtraining, with regularization delaying onset at possible downstream cost.These conclusions cover both Gaussian perturbations and fine-tuning modifications.

5. Related Work

The paper distinguishes catastrophic overtraining from related loss-of-plasticity and catastrophic-forgetting work while situating it within research on pre-training loss, scaling laws, and transfer-learning theory.

Loss of plasticity: Prior loss-of-plasticity research mainly studies small models with small datasets or reinforcement learning, whereas this work examines large-scale autoregressive language modeling.The paper also emphasizes sensitivity of pre-training loss to degradation, which prior literature largely overlooked.
Loss of plasticity: Unlike settings where pre-training is already harmful to downstream fine-tuning, this work reports degradation after overtraining on generic web data despite pre-training being expected to help.The paper presents this as a distinction from prior loss-of-plasticity studies.
Catastrophic forgetting: Catastrophic forgetting concerns sequential-task models forgetting prior tasks, and prior work includes regularization-based mitigation strategies.The paper relates its analysis to this literature without equating the phenomena.
Relationship between pre-training loss and downstream performance: Prior studies of pre-training loss and downstream performance have reported improved transfer beyond convergence, while this paper studies web-scale pre-training that rarely converges in practice.The paper states that these settings do not contradict its findings.
Scaling laws for optimal pre-training: Related scaling-law work studies downstream translation or overtraining regimes but does not observe the degradation reported here.Other studies also show continued pre-training-loss decreases at very high token-per-parameter ratios.
Transfer learning theory: The theoretical analysis builds on deep linear-network transfer-learning frameworks used to study feature learning, knowledge storage, and downstream transfer.The paper also distinguishes work on out-of-distribution degradation from its focus on catastrophic overtraining.

6. Discussion

Longer pre-training can reduce downstream adaptability because models become more sensitive to parameter perturbations. The analysis also identifies scope conditions and open directions for mitigating or characterizing this degradation.

Longer pre-training makes models more sensitive to parameter perturbations, degrading post-trained model adaptability.The paper links this sensitivity to worse outcomes after fine-tuning, especially when pre-training and fine-tuning tasks are misaligned.
The paper leaves the effects of optimizer, pre-training distribution, and training objective unresolved, while regularization only sometimes mitigates catastrophic overtraining.It also identifies data replay, LP-FT, and WiseFT as possible future strategies and notes broader applicability to parameter perturbations.
The theoretical setup models a two-layer linear network trained on a pre-training task before fine-tuning on a related task.The model uses θ = W1W2, with pre-training and fine-tuning tasks represented by matrices sharing singular vectors.
The fine-tuning analysis compares real and ideal initializations through batch-based gradient updates and tracks their final fine-tuned loss.The derivation defines L_ft(n) as the loss after K fine-tuning iterations and bounds differences between initialization dynamics.
Gaussian perturbation loss increases monotonically with extended pre-training under sufficiently small initialization.The result establishes increasing sensitivity to isotropic Gaussian weight perturbations as the pre-training trajectory progresses.

A.3. Dynamic Analysis of Finetuning Process

The appendix analyzes fine-tuning through three coupled dynamics: ideal infinite-batch, ideal finite-batch, and real initialization. It establishes progressively tighter approximation results under regularity and batch-size assumptions.

The analysis introduces ideal-initialization dynamics and an infinite-batch counterpart to compare fine-tuning trajectories.The infinite-batch process replaces empirical covariance with the identity matrix.
The infinite-batch ideal dynamic has an analytical expression, with bounded well-conditioned weights under learning-rate and regularization assumptions.The auxiliary weights remain bounded by Γ throughout the analyzed process.
Finite-batch ideal dynamics remain close to infinite-batch dynamics with an error bound controlled by batch size and concentration events.The result holds with probability 1 − δ under a sufficiently large batch-size assumption.
Real-initialization dynamics remain close to ideal-initialization dynamics, with error depending on pre-training initialization scale.This connects the practical trajectory to the idealized one through bounds on the weight and parameter differences.

A.5. Formal Statement and Proof of Theorem 4.7

This appendix formalizes how task misalignment and regularization affect fine-tuning outcomes. It also records technical lemmas and the experimental setup used to evaluate the theory.

A.5. Formal Statement and Proof of Theorem 4.7: With no regularization, (4, r)-misalignment implies increasing post-fine-tuning pre-training loss through rank r.The result holds with probability 1 − δ under the stated assumptions.
A.5. Formal Statement and Proof of Theorem 4.7: The inflection point r_λ is the smallest stage after which pre-training loss increases monotonically.This definition identifies where catastrophic overtraining begins under regularization strength λ.
A.5. Formal Statement and Proof of Theorem 4.7: Stronger regularization yields an inflection point no later than weaker regularization, while the theorem also compares unregularized fine-tuning loss.For λ1 > λ2 > 0, the result gives r_λ1 ≤ r_λ2 with probability 1 − δ.
A.6. Technical Lemmas: The technical lemmas establish properties of the scalar update function governing the idealized dynamics.The function remains within the interval bounded by σ and σ0, and its update direction preserves the sign relative to the fixed point.
B. Experimental Details from Section 2: Large Model Experiments: The appendix states that the omitted experimental details are provided for replication.These details cover model checkpoints, fine-tuning configurations, and evaluation procedures.
B.2. Fine-tuning setup.: The experiments use checkpoints from OLMo-1B, OLMo-2-7B, and LLM360-Amber, with instruction-tuning and multimodal-tuning setups.Instruction tuning uses Anthropic-HH and TULU, while multimodal tuning uses LLaVA.

B.3. Evaluations

The evaluation section defines downstream and generalist assessments for instruction-tuned and multimodal models. It also documents the evaluation-specific hyperparameter tables and experimental details.

Downstream evaluations measure performance on tasks representative of the fine-tuning objective, while generalist evaluations assess broader language understanding and inference.The generalist suite covers reasoning, commonsense, natural language inference, and sentence completion.
AlpacaEval measures instruction-following quality as win-rate against a reference model.The reference model matches the evaluated model’s architecture and fine-tuning dataset, so scores are not comparable across different graphs.
Table 2 records the hyperparameters used for instruction tuning and LLaVA, including defaults for unspecified settings.The section presents these configurations as part of the evaluation methodology.
The VLM score averages five vision-language benchmarks: MME, GQA, AI2D, POPE, and TextVQA.This score evaluates downstream performance of LLaVA models.

C.1. Pre-training and fine-tuning setup.

The controlled experiments pre-train OLMo models across several token budgets, then fine-tune them on classification and language-modeling tasks while evaluating both downstream and generalist capability. Learning-rate choices and perturbation size are explicitly examined to relate fine-tuning changes to pre-training sensitivity.

Pre-training: Controlled experiments use OLMo-15M, OLMo-30M, and OLMo-90M models trained on C4 for 4B–128B tokens.All models use muP parameterization and cosine learning-rate annealing to zero.
Fine-tuning: Models are fine-tuned on ten classification and language-modeling datasets, including SUBJ, BoolQ, RTE, SIQA, and GSM8k.GSM8k is treated as a language-modeling task for learning math-style text.
Evaluation: Evaluation reports heldout test performance and C4 perplexity as a measure of generalist capability.Validation sets tune models, while separate heldout test sets provide reported scores.
Experimental conditions: Learning rates plotted in the main comparison are restricted to those whose maximum pre-training perplexity remains below 6 across token budgets.This filtering keeps models within a range that retains pre-training capability.
Experimental conditions: The pre-trained-to-fine-tuned parameter distance is approximately proportional to learning rate and independent of pre-training amount.This supports using learning rate as an approximate proxy for fixed perturbation size.
Perturbation analysis: Gaussian perturbations scale parameter noise to account for differences in parameter magnitude across layers.The experiments evaluate degradation of pre-training perplexity under these perturbations.

D. Connection Between Progressive Sensitivity and Sharpness

The paper connects progressive sensitivity to Hessian-based sharpness only in regimes where the loss is well approximated quadratically. Experiments with larger perturbations show that progressive sharpness alone does not explain the broader sensitivity observed after longer training.

Quadratic connection: For a quadratic loss, Gaussian perturbation sensitivity is exactly related to the Hessian through the expected second-order loss change.The derivation uses a unit Gaussian perturbation and linearity of expectation.
Quadratic connection: When the loss is well approximated by a second-order Taylor expansion, including for small perturbations, progressive sensitivity and progressive sharpening are closely related.This connection is therefore local rather than universally valid.
Empirical comparison: For large Gaussian perturbations, the empirical loss of OLMo-30B models trained on 32B, 64B, and 128B tokens is not fully explained by the Hessian approximation.The approximation is accurate for small λ but diverges at larger perturbation strengths.
Empirical comparison: Interpolating between the base and AG News-fine-tuned OLMo-30B models likewise shows that the loss is not well approximated by the quadratic model.Here λ = 0 denotes the base model and λ = 1 the fine-tuned model.
Conceptual distinction: Progressive sharpening is the increase of the Hessian spectral norm during training, whereas progressive sensitivity concerns broader loss sensitivity to parameter perturbations.The latter can exceed what the quadratic approximation predicts.

E. Omitted Figures from Section 2: Large Model Experiments

The omitted large-model experiments organize post-training evaluations across four datasets and three model families, using both tuned and all-learning-rate views. The figures track downstream performance on AlpacaEval alongside out-of-distribution generalist performance.

Figure organization: The appendix provides omitted large-model figures from Section 2 and organizes them by dataset, model, and learning-rate tuning variant.Table 5 maps figures to Alpaca, Anthropic-HH, TULU, and VLM evaluations.
Anthropic-HH: Figures 11–16 evaluate OLMo-1B, OLMo-2-7B, and LLM360-7B after post-training on Anthropic-HH.Each model has views with tuned learning rates and across all learning rates.
Evaluation axes: Across these figures, AlpacaEval is treated as the main downstream metric for instruction-style evaluations, while the remaining datasets measure generalist performance.The VLM experiments use VLM Score as the corresponding main evaluation.
TULU: Figures 17–22 evaluate the same three model families after post-training on TULU.The paired figures again distinguish tuned-learning-rate and all-learning-rate evaluations.
VLM: Figures 23–25 evaluate OLMo-1B and OLMo-2-7B after VLM post-training.The VLM Score is the main downstream evaluation, while other datasets represent out-of-distribution generalist performance.

F.2. Extended fine-tuning experiments.

The extended fine-tuning experiments vary batch size, learning-rate scheduler, and model size while tracking post-fine-tuning pre-training and fine-tuning perplexity. Tuned-learning-rate figures additionally report the learning rate selected for best fine-tuning performance.

Experimental scope: The extended experiments ablate batch size, learning-rate scheduler, and model size, with Table 6 mapping settings to figures.The appendix includes fixed-hyperparameter and fine-tuning-optimal-learning-rate variants.
Measured quantities: Untuned and tuned figures separately show pre-training perplexity, fine-tuning perplexity, and the optimal learning rate as functions of pre-training budget.The tuned figures use the learning rate selected for best fine-tuning performance.
OLMo-15M: Figures 50–54 extend the analysis to OLMo-15M using the Table 3 configuration and fine-tuning-optimal learning rates.These figures are presented as analogues of Figures 5 and 6 from the main paper.