Source-linked AI summary
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler
TL;DR
The paper asks whether upstream Transformer scaling findings transfer to downstream fine-tuning, an important question because training is financially and environmentally costly. It studies scaling across pretraining and fine-tuning, finding that model shape and compute region matter, and proposes more efficient DeepNarrow configurations while releasing over 100 checkpoints.
Problem
The paper addresses whether scaling findings from upstream pretraining transfer to downstream applications, where training decisions carry financial and environmental costs.
Method
The authors run pretraining and fine-tuning experiments across Transformer configurations and investigate scaling strategies over model shape and compute regions.
Results
50% fewer parameters and 40% faster training achieve quality on par or better than canonical model sizes with the DeepNarrow strategy.
Takeaways & Limitations
Model shape and compute region should be considered in downstream scaling decisions, and the paper releases over 100 checkpoints for further analysis.
Takeaways & Limitations
Depth scaling is limited by non-parallelizable sequential computation, making comparisons with width scaling more difficult for larger models.
Abstract
from arXiv · showhide
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.
1 INTRODUCTION
The paper studies how Transformer scaling choices transfer from pretraining to downstream fine-tuning under costly compute constraints. It finds that downstream quality depends on model shape, scaling strategies vary by compute region, and DeepNarrow models can match canonical quality more efficiently.
- Motivation: Training Transformers has financial and environmental costs, motivating scaling decisions that improve transfer-learning efficiency under fixed compute budgets.The paper targets practical pretraining and downstream fine-tuning rather than upstream loss alone.
- Research questions: Scaling-law findings remain unresolved about whether to scale depth or width, whether small-model strategies transfer upward, and whether pretraining results generalize downstream.The paper frames these as open questions for practical Transformer development.
- Key findings: Downstream performance strongly depends on model shape, contrary to the upstream conclusion that model size dominates and shape matters weakly.The authors report that upstream pretraining findings do not necessarily transfer to downstream applications.
- Key findings: Pretraining perplexity can be a misleading indicator of downstream quality, so scaling laws differ substantially between upstream training and downstream fine-tuning.This makes model selection based only on upstream perplexity challenging.
- Scaling strategy: 50% fewer parameters and 40% faster training achieve model quality on par or better than canonical model sizes through the DeepNarrow strategy.The strategy preferentially increases depth and is reported as applicable across model sizes.
- Resources: The paper releases more than 100 pretrained checkpoints, including intermediate checkpoints, to support further research on Transformer scaling and fine-tuning.The release includes T5 configurations using improved scaling protocols.
2 RELATED WORK
Prior work established broad neural scaling-law analyses and many Transformer-efficiency techniques, but practical guidance for scaling Transformers remains limited. This paper situates its study around compute costs and scaling choices beyond parameter count alone.
- Background: Transformers are widely used across language, vision, reinforcement learning, and computational biology, increasing interest in understanding their empirical scaling laws.Prior scaling studies examine axes such as model size, compute, and dataset size.
- Scaling-law literature: Kaplan et al. studied upstream autoregressive language-modeling loss and argued that model size matters strongly while architectural shape matters weakly.The paper uses this prior result as a comparison point for downstream transfer.
- Scaling-law literature: Existing work also studies dataset transfer and scaling across vision, video, and multimodal generative-model domains.These studies extend scaling-law analysis beyond a single language-modeling setting.
- Efficiency motivation: Training and scaling Transformers carry financial and environmental costs, motivating practical recommendations that account for accelerator time and compute efficiency.The paper frames efficient scaling as a practical research problem alongside performance improvement.
- Efficiency methods: Transformer efficiency has been approached through distillation, compression, parameter sharing, efficient attention, architecture search, and alternative architectures.These methods address efficiency from multiple architectural and training perspectives.
3 METHODS
The study evaluates T5-style encoder-decoder Transformers across standardized model sizes, pretraining, and 17 downstream tasks. It varies explicit architectural scaling operators and reports downstream results primarily using compute and SuperGLUE accuracy.
- Architecture: The experiments use a T5-style Transformer encoder-decoder with relative attention and a Seq2Seq framework suitable for both encoder-like and decoder-like tasks.Experiments use Mesh Tensorflow and the default T5 library.
- Model configurations: Eight model sizes are defined: tiny, mini, small, base, large, XL, XXL, and XXXL.The canonical T5 sizes are retained, while tiny and mini extend coverage toward lower compute regions.
- Pretraining: Models are pretrained on C4 with span-based masked language modeling for 219 steps, using 16 TPU-v3 chips or 64 chips for larger models.The training duration follows the setup used for most experiments in the original T5 work.
- Downstream tasks: Downstream evaluation covers 17 GLUE, SuperGLUE, and SQuAD tasks, with aggregate GLUE/SuperGLUE scores used for key results.The task mixture includes both high-resource and low-resource natural-language-understanding tasks.
- Scaling operators: Scaling operators independently vary encoder or decoder layers, key-value size, hidden size, number of heads, or feed-forward dimension.The paper uses short codes such as EL, DL, KV, DM, NH, and FF to identify these operations.
- Evaluation convention: FLOPs are the main compute metric except in Figure 1, while downstream charts use SuperGLUE accuracy as the y-axis.Parameter counts include embedding parameters, and other metrics and tasks appear in supplementary material.
- Parallelism: Larger models use model parallelism settings of 2 for large, 8 for XL, and 32 for XXL models, alongside default data parallelism.These settings follow the T5 convention described in the experimental setup.
4 ANALYSIS AND RESULTS
The analysis finds that downstream Transformer performance depends on model shape, compute region, and scaling strategy rather than parameter count alone. DeepNarrow configurations improve compute–quality trade-offs, although depth has parallelization limits.
- 4.1 MODEL SHAPE MATTERS: Downstream performance depends strongly on model shape, whereas upstream pre-training performance is largely predicted by model size.Fine-tuning weakens the strong pre-training correlation with model size, making shape important for transfer.
- 4.1 MODEL SHAPE MATTERS: Pre-training perplexity can mislead model selection because improved upstream quality does not guarantee better downstream transfer.The paper gives a counter-example in which a model with better validation perplexity performs substantially worse after transfer.
- 4.2 SCALING BEHAVIOUR AT DIFFERENT COMPUTE REGIONS IS DIFFERENT: Downstream scaling behavior and Pareto frontiers differ across small, base, and large compute regions, limiting the transfer of small-scale scaling decisions to larger models.The paper cautions that improvements found at smaller scale may not generalize effectively to higher compute regions.
- 4.3 NOT ALL SCALING STRATEGIES AND MODEL SHAPES ARE CREATED EQUAL: Depth scaling has the strongest effect on the compute–performance boundary, while head-count scaling has little influence; width-based strategies are generally less Pareto-efficient.Examples include NL outperforming increased model dimension and several FF configurations being dominated by deeper alternatives.
- 4.4 SCALING RECOMMENDATIONS: The DeepNarrow strategy preferentially increases depth and produces more efficient alternatives to canonical T5 sizes.A Small 16L model matches Base quality while using 50% fewer parameters and 40% less training time; other deeper models outperform larger canonical models on downstream tasks.
- 4.4 SCALING RECOMMENDATIONS: Depth scaling is constrained by sequential, non-parallelizable computation, unlike width scaling, which can parallelize across many devices.The paper nevertheless reports improved Pareto efficiency within experiments using up to 64 workers and model parallelism of 32.
- 4.5 TRANSFERABILITY OF RESULTS TO VISION TRANSFORMERS (VIT): DeepNarrow scaling also improves ViT Pareto efficiency: ViT-SL=24 achieves higher few-shot accuracy with 15% fewer parameters and 11% fewer FLOPs than ViT-B.The deeper model is somewhat slower because depth adds sequential operations, but remains considered a compelling alternative.
5 CONCLUSION
The paper finds that Transformer scaling behaves differently across upstream and downstream settings, with model shape and compute region affecting practical efficiency. Its improved scaling protocol achieves comparable downstream quality with substantially lower parameter and training costs.
- Scaling laws differ between upstream pretraining and downstream fine-tuning, making model shape important for downstream performance.
- Scaling a small model can behave differently from scaling a large model, so scaling decisions may not transfer across compute regions.
- 50% fewer parameters and 40% faster training achieve similar downstream fine-tuning quality with the improved scaling protocol.
APPENDIX
The appendix presents performance plots for upstream pretraining and multiple downstream tasks across model sizes. These plots compare performance against parameters, FLOPs, and throughput for the configurations studied.
- Performance is plotted against number of parameters, FLOPs, and throughput for upstream and downstream tasks.
- Small-model results are reported separately for upstream and different downstream tasks.
- Base-model results are reported separately for upstream and different downstream tasks.
- Large-model results are reported separately for upstream and different downstream tasks.