Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models

1 Introduction

• Seed sensitivity in pretrained transformers complicates experimental interpretation and benchmarking, while its effect in LLM fine-tuning remains underaddressed.The introduction notes that a review of 85 ACL Anthology papers found potential misuse of seeds in over 50%, including 24 papers using a single fixed seed.Source: §1 · p. 1
• 60% accuracy for each of seeds 42 and 52 coincides with only 20% overlapping individual predictions, illustrating why aggregate performance can miss prediction-level differences.The passage uses this contrast to motivate measuring micro-level consistency alongside macro-level performance.Source: §1 · p. 1, Figure 1 · p. 1
• Individual-prediction instability matters especially in applications such as medical diagnosis and autonomous driving, where predictions are sensitive to specific test points.The paper argues that identifying unstable examples can guide targeted improvements to model design and evaluation.Source: §1 · p. 2
• The study analyzes seed-induced variability in pretrained LLMs on GLUE and SuperGLUE at both macro and micro levels, introducing a metric for individual-prediction stability.Its experiments find significant variability in both standard and consistency metrics.Source: §1 · p. 2

2 Macro Metric: Variance

• Macro impact is measured as the variance of a standard evaluation metric across multiple random seeds.The metric may be accuracy for classification or MAE for regression.Source: §2 · p. 2
• The variance calculation takes the performance values ζ_1 through ζ_S from S fine-tuned seeds and compares them with their mean.The mean is ζ̄, the average of the seed-specific values.Source: §2 · p. 2
• A smaller VAR indicates less variation in macro performance across seeds.Source: §2 · p. 2

3 Micro Metric: Consistency

• Consistency is an LLM's ability to produce consistent predictions for the same input across models fine-tuned with different hyperparameter settings.Correct-consistency further specifies consistent correct predictions in that setting.Source: §3 · p. 2
• For two models evaluated on N data points, π_A,B scores the alignment between their predictions, with higher consistency indicating less micro-level variation.The scoring function can be binary or probabilistic, depending on the NLP task.Source: §3 · p. 2
• Correct-consistency uses π_A,B,r to quantify prediction alignment with ground truth.The ground truth for each data point is denoted r_i.Source: §3 · p. 2
• The study uses consistency for classification tasks and evaluates seed effects on the same pretrained LLM, rather than comparing arbitrary model architectures or settings.The paper notes that consistency metrics can generally compare models across architectures, hyperparameters, or training settings, but its study focuses on the same pretrained LLM.Source: §3 · p. 2

4.1 Benchmarks and pretrained models

• The benchmark suite includes six GLUE tasks and six SuperGLUE tasks; all use accuracy except CoLA, which uses MCC.STSB, WSC, and MNLI are omitted; RTEG denotes GLUE RTE and RTES denotes SuperGLUE RTE.Source: §4.1 · p. 3
• The study tests RoBERTa-large and Llama3.2-3B, using LoRA fine-tuning for the latter to examine findings across model scales.The passage reports approximately 350M trainable parameters for RoBERTa-large and approximately 3.21B for Llama3.2-3B.Source: §4.1 · p. 3

4.2 Settings

• Each task is fine-tuned with ten randomly chosen seeds, and CON and CCON are averaged across the 45 unique seed pairs.The setup follows previously reported best settings unless otherwise specified and uses the PyTorch GLUE script.Source: §4.2 · p. 3
• RoBERTa-large settings follow Liu's configurations and reference SOTA scores; for Llama3.2-3B, the study uses its own settings because reference configurations or reports were unavailable.The paper points to appendix tables for the detailed data and learning settings.Source: §4.2 · p. 3

5.1 Macro impact

• RoBERTa-large shows high macro VAR on RTES (16.67), COPA (16.83), and MultiRC (13.30), but low VAR on QQP (0.06), QNLI (0.38), SST2 (0.55), and MRPC (0.89).The paper notes that SuperGLUE tasks generally show higher VAR than GLUE tasks.Source: §5.1 · p. 3
• Llama3.2-3B with LoRA has significantly lower VAR than RoBERTa-large across most tasks; the authors suggest that updating about 2.3 million parameters constrains seed-induced variance.They also propose decoder-only architecture as a possible contributor, while noting that deeper analysis is future work.Source: §5.1 · pp. 3–4

5.2 Micro impact

• For RoBERTa-large, CON is high on SST2 (96.83), QNLI (95.64), and QQP (95.57), but lower on COPA (64.88) and RTES (70.05); large CON–CCON gaps can indicate consistent but inaccurate predictions.WiC and CoLA have CON–CCON differences of 21.61 and 11.19, respectively; WiC also shows low VAR alongside low CON and CCON.Source: §5.2 · p. 4
• Unlike macro VAR, CON and CCON show no corresponding lower-variability trend for Llama3.2-3B with LoRA compared with RoBERTa-large.The authors conclude that LoRA has limited effect on micro consistency.Source: §5.2 · p. 4

ance and consistency in general?

• Training size has a weak negative correlation with VAR (-0.3918) and weak or moderate positive correlations with CON (0.4257) and CCON (0.4259), without a consistent effect across tasks.MultiRC has high VAR despite a relatively large dataset; Figure 2 plots these correlations for RoBERTa-large, with tasks ordered by training size.Source: ance and consistency in general? · p. 4, Figure 2 · p. 4

5.3 Discussion

• Seed-induced variability is substantial at both macro and micro levels, and larger training sets are not a guaranteed remedy, especially for complex tasks.The authors report that training-size effects depend on data quality, task complexity, and label noise.Source: ance and consistency in general? · p. 4
• Snapshot ensembles address consistency but are computationally prohibitive for LLMs because they require training numerous models.The authors identify dynamically weighting individual-example losses during gradient propagation as a future direction for reducing macro and micro variance without significant computational overhead.Source: ance and consistency in general? · pp. 4–5

• The study finds seed-induced variability at macro and micro levels and calls for accounting for seed sensitivity in benchmarking to support more reliable and reproducible results.Source: §6 · p. 5

• Because the study focuses on classification tasks in GLUE and SuperGLUE, its findings may not generalize to other NLP task types.Source: §7 · p. 5

• For two LLMs A and B and dataset D = d1, · · · , dN, the appendix identifies N data points and refers to Table 2 for the task-specific metric ζ and scoring function π(·) in Equations 2 and 3.Source: §A.1 · p. 6

A.2 Data Description

• MultiRC is split at the question–answer pair level, while COPA becomes two premise–question–choice sequences with labels remapped from 1/2 to 0/1.The datasets use predefined train, dev, and test files; the dev set also serves as test because test labels are unavailable.Source: §A.2 · p. 6

A.3 Hyperparameter Settings

• Unless stated otherwise, the experiments use default Hugging Face hyperparameter values.Detailed configurations are provided in Tables 5 and 6.Source: §A.3 · p. 6

A.4 Replicated SOTA Scores

• Exact SOTA scores could not be reproduced because of differences in fine-tuning scripts and missing settings, though replicated GLUE and SuperGLUE accuracies are reported as comparable.WSC and MNLI were omitted because their representations or model settings could not be replicated; STSB was omitted because it is a regression task.Source: §A.4 · pp. 6–7

A.5 Additional Results

• Tables 7 and 8 show significant macro-level variance across many tasks, with similar VAR patterns in precision, recall, and F1.The additional results report average values and standard deviations across accuracy, precision, recall, F1, CON, and CCON.Source: §A.5 · p. 7

Is there a universally superior random seed?

• Across ten RoBERTa-large seeds, Figure 3 shows no discernible pattern supporting a universally superior seed.The heatmap displays normalized accuracy across tasks and seeds, with darker colors indicating better accuracy.Source: Is there a universally superior random… · p. 7, Figure 3 · p. 7

ance and consistency in general?

• Figure 4 shows training-size correlations with VAR, CON, and CCON for Llama3.2-3B, with tasks ordered by increasing training size.The reported patterns are consistent with those discussed in Section 5.3.Source: ance and consistency in general? · p. 7, Figure 4 · p. 7