Source-linked AI summary
Scaling Laws for Transfer
Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish
TL;DR
The paper asks how transfer changes scaling when large models are fine-tuned on limited data from a different distribution. It measures transfer as equivalent downstream data using transformer experiments across pre-training and fine-tuning curricula. Effective transferred data follows a low-data power law, pre-training multiplies fine-tuning data, and fine-tuned models retain scaling improvements where from-scratch models plateau.
Problem
The paper asks how pre-training affects scaling and data requirements when models are fine-tuned on a different distribution in an unsupervised, low-data setting.
Method
The authors train transformers of varied sizes under from-scratch and pre-training-plus-fine-tuning curricula, defining transfer as equivalent same-size from-scratch downstream data.
Results
Effective transferred data is well described by a power law; pre-training multiplies fine-tuning data, and fine-tuned models continue improving with size when from-scratch models become data-limited.
Takeaways & Limitations
Transfer can be measured in intuitive data units and used to predict performance, compute, and data needs for scaled-up fine-tuned models.
Takeaways & Limitations
Transfer was measured only for Python fine-tuning and unsupervised distribution pairs, so generalization to broader distribution pairs, supervised learning, or reinforcement learning is unclear.
Abstract
from arXiv · showhide
We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data "transferred" from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size. We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.
1 Introduction
The paper studies transfer in low-data language-model fine-tuning by measuring pre-training in equivalent units of downstream data. It finds predictable power-law scaling with model size and fine-tuning data, while pre-training improves data and compute efficiency.
- Transfer is studied because neural networks often require more direct task experience than humans, making sample efficiency an important way to characterize data.
- The experiments compare transformers trained from scratch on Python, pre-trained on text then fine-tuned on Python, and pre-trained on text plus other programming languages before Python fine-tuning.
- Effective transferred data is the additional Python data a same-size from-scratch model would need to match a pre-trained model’s Python loss, and it follows a low-data power law.
- β = 0.38 was identical for text pre-training and text-plus-code pre-training, suggesting this exponent depends on architecture and target-distribution generalization.
- Smaller α indicates closer directed distribution proximity; for text-to-Python transfer, a 10x larger model is approximately worth a 100x larger fine-tuning dataset.
- In data-limited settings, from-scratch models plateau as parameters increase, whereas fine-tuned models continue improving with a reduced slope.
- Fine-tuning is more compute-efficient than training from scratch in the low-data regime, especially when using an existing pre-trained model whose pre-training cost is excluded.
2 Methods
The study trains transformer language models across sizes and data curricula, then evaluates their Python-code performance after convergence. It uses large text and Python corpora with standardized optimization and held-out evaluation.
- Models are trained from scratch on Python, or pre-trained on language before fine-tuning on Python, using comparable optimization settings and convergence-based early stopping.
- The text pre-training corpus contains 24 billion characters, while the Python corpus contains 22 billion characters from public GitHub repositories, with 3% held out for evaluation.
3 Results
The results characterize when pre-training helps or harms fine-tuning. Pre-training improves data and compute efficiency in low-data settings, but can reduce effective data and converged performance in high-data settings for small models.
- 3.1 Ossification – can pre-training harm performance?: Pre-training can ossify model weights, reducing their ability to adapt to the fine-tuning distribution in the high data regime.
- 3.1 Ossification – can pre-training harm performance?: The smallest from-scratch models outperform fine-tuned models on large datasets, showing that pre-training can reduce performance in the high data regime.
- 3.1 Ossification – can pre-training harm performance?: D/D(N) approaching 1.0 marks a regime where pre-training reduces effective data, and small fine-tuned models fail to reach from-scratch performance even with 10x or 100x D(N).
- 3.2 Fine-tuning is usually compute efficient (ignoring pre-training compute): With approximately 30x more data, the fine-tuning compute-efficient frontier becomes similar to the from-scratch frontier, while fine-tuning remains easier to place on it.
- 3.2 Fine-tuning is usually compute efficient (ignoring pre-training compute): Once data matches or exceeds the amount needed for from-scratch training, fine-tuning has substantially worse converged performance and is less compute efficient.
- 3.2 Fine-tuning is usually compute efficient (ignoring pre-training compute): When training to convergence, pre-trained models are more compute efficient than from-scratch models on small datasets, and fine-tuning more readily reaches the compute frontier.
4 Related Work
Related work situates the paper within scaling laws, transfer learning, pre-training, few-shot learning, and sim-to-real transfer. Prior studies motivate analyzing how transfer and data efficiency behave across modalities and tasks.
- The paper focuses on how transfer scales with compute, data, and parameters, building on prior work about power laws and neural-network scaling trends.
- Transfer and meta-learning have attracted substantial attention across modalities, with recent literature reviews providing broader context.
- Pre-training on image datasets has improved performance and data efficiency, while CLIP demonstrated transfer from captioned images to image classification.
- Few-shot learning research includes probabilistic program induction, designing models for fine-tunability, and transfer in large-model benchmarks such as SuperGLUE.
- Sim-to-real transfer for robotic Rubik’s-cube solving provides a related setting where fine-tuning data is much more expensive than pre-training data.
5 Limitations
The study identifies methodological and scope limitations, including untuned training choices, Python-only unsupervised transfer, transformer-only evaluation, and imperfect modeling of zero-shot and from-scratch performance.
- Experimental setup: Models were not tuned specifically for fine-tuning or code, limiting how directly the results reflect optimized transfer performance.The authors also report that learning-rate scans were not comprehensive.
- Experimental setup: Small-dataset results may be confounded because training ended before the learning-rate warmup completed.
- Scope: Transfer was measured only for Python and only between distributions in an unsupervised setting, leaving broader distribution pairs and supervised or reinforcement-learning setups untested.
- Modeling: The analysis lacks a good closed-form model for from-scratch results, so overall fine-tuned performance is defined through relative performance instead.
- Scope: The study measured performance only on transformer architectures.
- Modeling: The zero-shot case requires approximating fine-tuning with one character, and cheaper measurement strategies were not explored.
6 Discussion
The discussion interprets transfer scaling laws, considers their practical use, and qualifies the evidence through distribution choice and observed failure modes. It also outlines directions for extending the analysis.
- Discussion: Pre-training can effectively multiply the fine-tuning dataset in the low-data regime, although the multiplier decreases as the fine-tuning dataset grows.
- Discussion: For text pre-training and Python fine-tuning, one character of fine-tuning corresponds to approximately 3.7e8 characters of Python data from scratch, while 300 characters multiply effective data by 2.8.
- Discussion: Text-plus-code pre-training yields DE = 4.1e9 characters, with 300 characters of examples worth a factor of 1.7x.
- Potential applications of these scaling laws: A proposed practical experiment compares 1% and 10% fine-tuning subsets, then varies model size to estimate the performance cost of collecting less data.
- Discussion: Effective-data scaling may provide a better-behaved way to compare architectures and algorithms when performance is measured by loss rather than downstream accuracy.
- How similar are Python and English?: The analysis treats English–Python transfer as representative of distant distributions, while noting closer examples such as English–French and Wikipedia–arXiv.
- Discussion: Small fine-tuned models did not reach from-scratch performance even with 10x or 100x D(N), suggesting that substantial pre-training may become impractical under such conditions.
- Future work: Future work includes measuring transfer across more distributions, broader targets, other learning settings, and architectures, plus predicting ideal pre-training ratios.
7 Conclusion
The conclusion presents transfer as measurable and predictably scalable, with effective data as an intuitive unit for estimating performance, compute, and data needs in fine-tuning.
- Conclusion: Transfer is measurable across language models spanning a wide range of sizes and scales predictably.
- Conclusion: Measuring transfer in units of data provides an intuitive framework for understanding data as an ML ingredient and the generality of AI systems.
- Conclusion: The approach can measure transfer cheaply in the low-data regime.
- Conclusion: The scaling laws help predict performance, compute, and data needs for scaled-up fine-tuned models.
Contributions
The contributions section lists project roles: leadership, analysis, writing, engineering, dataset generation, and oversight were distributed among the named contributors.
- Contributions: Danny Hernandez led the project and performed most of the experiments, analysis, and writing.
- Contributions: Jared Kaplan contributed directly to the analysis and advised the project from its inception.
- Contributions: Tom Henighan maintained the underlying code base and paired on engineering challenges.
- Contributions: Sam McCandlish oversaw the work, helped formulate its direction, generated the dataset, supported engineering, and contributed to the analysis.
A Data Regime
The paper defines low-data behavior using D(N), then fits effective transfer data across model and fine-tuning dataset sizes. The resulting analyses examine fit structure, learning-curve parallelism, and epoch behavior.
- A Data Regime: D(N) is the amount of data needed to reach 99% of the performance obtainable with infinite python data for a given model size.The low-data regime is defined as DF ≤ 10% of D(N).
- A Data Regime: D(N) is estimated from curve intersections, using 99% performance for from-scratch models and 95% for fine-tuned models.The thresholds were selected partly because fine-tuning intersections appeared noisier.
- A Data Regime: The analysis relates total effective data DE to transferred data DT and fine-tuning data DF, with the relationship illustrated in Figure 1.The supplementary equation defines DE as the sum of DT and DF.
- A Data Regime: The global fitting procedure fits dataset-specific fits, using approximately common logit-fit exponents and equal weighting across fine-tuning dataset sizes.The resulting fit is used to generate the global fit in Figure 2.
- A Data Regime: Fine-tuning learning curves are qualitatively very parallel as model scale increases, although quantitative performance predictions were too inaccurate to use.The largest model visibly overfits near the end of training.
D Figure 3, including medium data regime
The transfer fit breaks down in the medium-data regime, while several explanations point to interpolation instability and insufficient tuning. Attempts to construct a unified from-scratch scaling foundation were also unsuccessful.
- D Figure 3, including medium data regime: The fit given by equation 3.2 breaks down for transfer in these distributions in the medium-data regime.The cited boundary is D(N) > .10.
- D Figure 3, including medium data regime: High-data effective-data estimates are poorly conditioned because performance curves lie nearly on top of one another.Small performance variations can therefore substantially change the interpolated effective-data estimate.
- D Figure 3, including medium data regime: The authors also believe tuning matters more in the high-data regime, where they performed relatively little tuning.
- D Figure 3, including medium data regime: Global power-law fits to from-scratch python runs were too poor to support a unified scaling law.The authors note that more careful tuning or a modified functional form might produce a better fit.
- D Figure 3, including medium data regime: Zero-shot performance for larger models was similar to training from scratch on python data equal to about .3% of the text pre-training dataset.The authors added a known 0.3% amount of python during pre-training to investigate this transfer.
G Analysis of best epoch
Training duration depends on fine-tuning dataset size: large datasets require roughly from-scratch training durations, while small datasets reach their useful endpoint sooner but introduce schedule-related confounds.
- G Analysis of best epoch: Large fine-tuning datasets require a similar training duration to from-scratch training, typically 1-10 epochs for code.Epoch counts generally decrease with model size and increased data.
- G Analysis of best epoch: Parametrizing results in terms of D(N) makes the training-duration patterns appear cleaner.
- G Analysis of best epoch: The smallest dataset, containing 100,000 tokens, required 2-5x fewer epochs than training from scratch.Early stopping is practical when overfitting becomes obvious.