Source-linked AI summary
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah Smith
TL;DR
Fine-tuning pretrained language models is brittle because random seeds can produce substantially different downstream results. The paper repeatedly fine-tunes BERT on four GLUE datasets, isolates weight initialization and data order, and finds substantial gains, comparable variance contributions, and practical early-stopping opportunities.
Problem
Fine-tuning performance can vary substantially across training episodes despite fixed hyperparameter values, making random-seed effects an important source of uncertainty.
Method
The authors fine-tune BERT hundreds of times on four GLUE datasets while varying random seeds controlling weight initialization and training data order.
Results
Substantial performance gains occur over previous BERT results, while weight initialization and data order contribute comparably to performance variance across trials.
Takeaways & Limitations
Multiple trials, early stopping, and rigorous reporting of random-seed effects can improve fine-tuning evaluation and resource use.
Takeaways & Limitations
The study defers investigating why some weight initializations and data orders are consistently better to future work.
Abstract
from arXiv · showhide
Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. This process, however, is often brittle: even with the same hyperparameter values, distinct random seeds can lead to substantially different results. To better understand this phenomenon, we experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials. Further, we examine two factors influenced by the choice of random seed: weight initialization and training data order. We find that both contribute comparably to the variance of out-of-sample performance, and that some weight initializations perform well across all tasks explored. On small datasets, we observe that many fine-tuning trials diverge part of the way through training, and we offer best practices for practitioners to stop training less promising runs early. We publicly release all of our experimental data, including training and validation scores for 2,100 trials, to encourage further analysis of training dynamics during fine-tuning.
1. Introduction
Fine-tuning BERT can vary substantially across trials despite fixed hyperparameters, and changing only random seeds produces large performance differences. The paper analyzes sources of this variance, trial-count effects, early stopping, and released training data.
- Motivation: Fine-tuning performance can vary substantially across training episodes even when hyperparameter values are fixed.The paper investigates this brittleness in pretrained contextual embedding models.
- Trial budgets: Even after hundreds of fine-tuning trials, the best-found validation performance has not fully converged.The paper examines how the best-found model changes as the number of trials increases.
- Performance gains: Substantial improvements over previous BERT results arise on four GLUE tasks when only random seeds are changed.On RTE and CoLA, the improvement over previous results with the same model is 7% absolute.
- Sources of variance: Weight initialization and training data order contribute comparably to performance variance across fine-tuning trials.Some weight initializations also perform well across multiple tasks, while some data orders and initializations are consistently better within a dataset.
- Early stopping: A simple early stopping algorithm reduces the computational resources needed to reach a given validation performance.The method distinguishes worse-performing models from better ones early in training and provides recommendations for different computational budgets.
- Data release: The authors publicly release data from 2,100 fine-tuning episodes, including training losses and validation performance during training.The release is intended to support further analysis of fine-tuning dynamics.
2. Methodology
The study fine-tunes pretrained BERT on four GLUE binary-classification datasets while varying only weight initialization and training-data order seeds. It uses a shared classification-layer architecture and repeated experiments under fixed hyperparameters.
- Experimental setup: The experiments fine-tune pretrained BERT on four GLUE datasets, using identical hyperparameters while changing weight-initialization and data-order seeds.The four datasets are MRPC, RTE, CoLA, and SST.
- Model architecture: All tasks are binary classification, so each model appends the same two-output classification layer to pretrained BERT.
- Datasets: MRPC measures the average of F1 and accuracy, while RTE predicts whether a premise entails its hypothesis.
- Datasets: CoLA classifies sentences as grammatical or ungrammatical using Matthews correlation, whereas SST uses binary positive-versus-negative sentiment labels from movie reviews.
- Training configuration: BERTlarge, uncased is fine-tuned for three epochs across 340 million parameters, with a randomly initialized final layer containing 2048 parameters.
- Experimental scale: MRPC, RTE, and CoLA each receive 625 experiments, while SST receives 225 experiments using combinations of distinct weight-initialization and data-order seeds.
3. The large impact of random seeds
Fine-tuning outcomes vary substantially across random seeds, especially on smaller datasets, and selecting the best among more trials improves expected validation performance. More frequent validation evaluation further raises expected best-model performance.
- Seed sensitivity: On smaller datasets, some runs diverge toward majority-label performance, producing substantial variation and making the best of multiple trials much stronger than a typical single run.
- Performance gains: The best-found models substantially outperform previous results using the same BERT model and optimization procedure, with 7% absolute gains on RTE and CoLA.
- Trial budgets: Expected validation performance continues increasing with the number of trials through the largest tested budget, suggesting that hundreds of trials have not fully converged.
- Trial budgets: Increasing the number of trained models raises the expected validation performance of the best model, while smaller datasets show especially large variance at low budgets.
- Validation evaluation: Evaluating validation performance frequently during training yields higher expected best-model values and smaller variances than evaluating only at epoch ends or after training.
- Validation evaluation: The authors recommend frequent validation evaluation because evaluation is relatively cheap and can run in parallel with training.
4. Weight initialization and data order
Weight initialization and training-data order both contribute comparably to fine-tuning variance, while their effects differ by dataset size and some weight initializations generalize across tasks. Small datasets also show frequent divergence under unfavorable seed combinations.
- Decoupling randomness: Each experiment combines one weight-initialization seed with one data-order seed; Figure 2 encodes these combinations as rows and columns sorted by average performance.
- Divergence: On MRPC, RTE, and CoLA, a fraction of models diverge near majority-baseline performance, helping explain their large validation variance.
- Decoupling randomness: Weight-initialization and data-order seeds contribute comparable amounts to validation-performance variance, each slightly below the overall within-task standard deviation.
- Seed quality: Worst weight initializations and data orders concentrate more densely in low-performance regions than the best seeds, with divergence especially visible on MRPC and RTE.
- Seed quality: The best and worst weight initializations and data orders have significantly different expected performances for every dataset, according to ANOVA tests with p < 0.05.
- Globally good initializations: WI seed 12 ranks first on CoLA and RTE, second on MRPC, and third on SST, indicating that some weight initializations perform consistently well across tasks.
- Open questions: The study defers analysis of why certain initializations and data orders are good, releasing 2,100 experiments to support future investigation.
5. Early stopping
Early performance often identifies weaker fine-tuning trials, enabling early stopping to allocate computation toward more promising random seeds. The proposed strategy starts many trials, stops some partway through training, and fully trains the remaining promising subset.
- Early stopping: Early stopping achieved better performance with the same computational resources by terminating the least promising trials before full training.The comparison baseline fully trained multiple experiments without early stopping.
- Early discovery of failed experiments: High early–late validation rank correlation on MRPC, RTE, and CoLA suggests that weaker trials can often be stopped before completing the first epoch.The correlation is between model rankings at different evaluation points; SST is the exception noted in the passage.
- Early stopping: The early-stopping algorithm trains t trials, evaluates them after fraction f of training, and fully trains only the p most promising trials.Its total cost is (tf + p(1−f))s steps, where s is the full-training step count.
- Start many, stop early, continue some: Across datasets and budgets, the best configurations started substantially more trials than they fully trained and often stopped weak trials after 20–30% of training.The passage reports that the fully trained trial count was often around x/2, with less-than-one-epoch stopping favored on three datasets.
- Limitations: The approach does not distinguish between data order and weight initialization, although the authors defer more specialized criteria to future work.The authors note that initial results suggest this distinction could support more sophisticated early-stopping rules.
6. Related work
The related work situates random seed choices as overlooked optimization factors alongside hyperparameter tuning, initialization schemes, and data-order methods. The paper focuses on evaluating sampled weight initializations and data orders rather than proposing new initialization distributions or active-learning criteria.
- Hyperparameter optimization: Prior hyperparameter optimization commonly tunes learning rate, layer width, and regularization strength, whereas this work examines weight initialization and data order.The paper treats these two choices as overlooked hyperparameters that can still provide room for optimization.
- Weight initialization: Initialization research has proposed Xavier, Kaiming, and orthogonal schemes to preserve gradients or prevent exploding and vanishing gradients.This paper instead studies how different samples from an initialization distribution behave.
- Efficient search: Early-stopping configurations can be optimized under a fixed computational budget by varying how many trials start, stop, and continue fully.The paper reports that different parameter assignments can yield distinct expected performance despite identical budgets.
- Data order: Active-learning research selects data orders using criteria such as uncertainty or mini-batch diversity, while this paper provides tools for evaluating seeds in stochastic training.The tools can also compare different active-learning algorithms, according to the passage.
7. Conclusion
The paper finds substantial random-seed variance and performance gains across four GLUE datasets, then proposes early stopping with practical recommendations to reduce the computational cost of multiple trials. It releases thousands of fine-tuning episodes for further analysis.
- Conclusion: Experiments on four GLUE datasets show significant variance across random-seed trials and substantial performance gains on all tasks.
- Conclusion: A simple early-stopping strategy with practical recommendations is included to alleviate the computational costs of running multiple fine-tuning trials.
- Conclusion: The authors publicly release experimental data containing thousands of fine-tuning episodes.
A. Appendix
The appendix compares the variability attributable to weight initialization and data order with overall validation-performance variability. Both factors show comparable variance magnitudes across the examined datasets.
- Variance decomposition: Comparable inter-seed variance magnitudes arise from weight initialization and data order when aggregating final validation performance.Figure 8 compares distributions of standard deviations under fixed seeds for each factor and shows the overall standard deviation for each dataset.
B. ANOVA assumptions
The ANOVA analysis relies on independence, comparable group variances, and approximate normality, with robustness depending on sample structure and size.
- ANOVA assumes independent samples, roughly equal variances across groups, and normally distributed data.
- The experiments satisfy independence because each data-order and weight-initialization sample is independently and identically distributed.
- Equal group sizes make ANOVA generally robust to somewhat differing variances in these experiments.
- Larger sample sizes improve robustness to non-normality; the largest-dataset experiments are near-normal, while smaller-dataset performance is less normal but more extensively sampled.
C. Pearson Correlation
Pearson correlation measures how similarly model performance behaves across two training evaluation points, complementing the rank-correlation analysis shown elsewhere.
- Each plot point reports Pearson correlation between performance at iterations i and j across trials.
- High correlation indicates that model performance is similar at the two compared evaluation points.
- Figure 9 uses Pearson correlation, whereas Figure 5 uses rank correlation on the same data.
- The axes in Figure 9 represent epochs, and symmetric coordinates compare best-found validation performance after different evaluation points.