The MultiBERTs: BERT Reproductions for Robustness Analysis

1 INTRODUCTION

• The MultiBERTs release provides 25 BERT-Base, Uncased checkpoints to study robustness to parameter initialization and training-example order.The shared release preserves low-cost experiments and apples-to-apples comparisons while supporting broader conclusions about the BERT pre-training procedure.Source: §1 · p. 2
• The Multi-Bootstrap estimates uncertainty in experimental results across multiple pre-training seeds and is accompanied by recommendations and a software library.Source: §1 · p. 2
• The paper illustrates the approach by investigating counterfactual data augmentation’s impact on gender bias in BERT-based coreference resolution.Additional examples document challenges in reproducing the original BERT checkpoint.Source: §1 · p. 2
• The release includes 140 intermediate checkpoints from five runs to support studies of learning dynamics.Source: §1 · p. 2
• The work complements concurrent releases by focusing on BERT, using more pre-training seeds, and presenting a method for robust performance conclusions.Source: §1 · p. 2

2.1 TRAINING

• The released models use BERT-Base, Uncased with 12 layers and 768-dimensional embeddings, trained on BooksCorpus and English Wikipedia.The original BERT dataset was unavailable, so training used a newer version collected with the same methodology; the authors also held sequence length at 512 from the start and used two million steps.Source: §2.1 · pp. 2–3, §2.2 · p. 3

2.2 PERFORMANCE BENCHMARKS

• On most GLUE and SQuAD tasks, original BERT falls within the MultiBERTs score range, but it beats all reproductions on QQP and trails them on SQuAD.Source: Figure 1 · p. 3, Figure 2 · p. 4, §2.2 · p. 4

2.1 TRAINING

• The release contains 25 models trained for two million steps each, plus 28 additional checkpoints from each of five runs.In total, the release contains 165 checkpoints.Source: §2.1 · p. 3
• 1728 TPU-hours were estimated for each pre-training run, with total experiment emissions estimated at about 12.0 metric tons CO2e before offsets or clean energy.Source: §2.1 · p. 3

2.2 PERFORMANCE BENCHMARKS

• GLUE results use development sets with five fine-tuning runs per model and task, while SQuAD v1.1 and v2.0 use two-epoch runs.GLUE uses three epochs, batch size 32, and a learning-rate sweep; SQuAD uses batch size 48 and learning rate 5e-5.Source: §2.2 · pp. 3–4
• The authors also analyze instance-level agreement among models to further illustrate variability from pre-training and fine-tuning.Source: §2.2 · p. 4

3.1 INTERPRETING STATISTICAL RESULTS

• The target is expected performance averaged over seeds, with uncertainty reflecting both the finite number of seeds and the finite evaluation set.The method estimates the distribution of the error in that average to construct confidence intervals and test thresholds.Source: §3.1 · p. 5
• For paired studies, intervention and baseline models share pre-training checkpoints, allowing the method to estimate correlation between their errors and uncertainty in the performance difference.The paired bootstrap uses the same resampled seeds and test examples for both models.Source: §3.1 · p. 5, §3.2 · p. 7
• The authors recommend testing whether the performance difference exceeds a predefined threshold, typically zero, while their theorem requires performance to be linear in the test distribution.They conjecture the method likely generalizes to metrics such as AUC, BLEU, and expected calibration error.Source: §3.1 · p. 6, §3.2 · p. 6

3.2 MULTI-BOOTSTRAP PROCEDURE

• The Multi-Bootstrap resamples pre-training seeds and test instances to approximate the uncertainty distribution of average performance across seeds.For nested randomness, fine-tuning results are averaged within each bootstrap sample.Source: §3.1 · p. 5, §3.2 · pp. 6–7

3.1 INTERPRETING STATISTICAL RESULTS

• For unpaired studies, the procedure samples seeds independently for each model and assumes zero correlation between estimation errors, yielding a conservative error estimate absent negative correlation.The authors describe the zero-correlation assumption as relatively safe because different models’ random seeds are unlikely to induce negative performance correlation.Source: §3.1 · p. 6, §3.2 · p. 7

3.2 MULTI-BOOTSTRAP PROCEDURE

• With a single fixed-baseline estimate, report where it falls within the MultiBERTs confidence interval and interpret the comparison conservatively because baseline variance is unknown.The method assesses uncertainty from MultiBERTs’ seeds and test examples, not from the fixed baseline.Source: §3.1 · p. 5, §3.2 · p. 7

4 APPLICATION: GENDER BIAS IN COREFERENCE SYSTEMS

• CDA augments pre-training data with counterfactual sentences that randomly swap English binary-gendered pronouns, aiming to weaken associations between pronouns and words such as occupations.CDA-incr continues each baseline checkpoint for 50K steps; CDA-full trains from scratch on the same dataset.Source: §4 · p. 8
• The evaluation uses a frozen BERT encoder with a coreference classifier trained on OntoNotes and tested on Winogender, with five classifier runs per pre-training run.The bias score is the Pearson correlation between per-occupation bias scores and U.S. occupational gender statistics.Source: §4 · p. 8

4.1 PAIRED ANALYSIS: CDA-INCR VS. BASE

• Across pre-training seeds, bias correlation varied from r = 0.1 to 0.7 for base models and 0.0 to 0.4 for CDA-incr, while accuracy was 62.6±1.2% for base models with only a small, nonsignificant change under CDA-incr.The paired analysis found apparent improvement for most seeds; the figure shows score distributions across five coreference-model training runs per seed.Source: §4.1 · p. 8, Figure 3 · p. 8
• CDA-incr significantly reduced Winogender bias correlation by bδ = −0.162 with p = 0.001.The paired analysis bootstrapped over the sample of 60 occupations.Source: §4.1 · p. 9, Table 1 · p. 9

4.2 UNPAIRED ANALYSIS: CDA-FULL VS. CDA-INCR

• CDA-full did not significantly improve on CDA-incr for bias correlation (δ = -0.064, p = 0.132), and overall accuracy did not change appreciably (0.622 vs. 0.623, p = 0.416).The comparison used independently trained checkpoints and an unpaired Multi-Bootstrap.Source: §4.2 · p. 9
• Sampling only seeds or only examples produced lower p-values (0.005 and 0.053), suggesting that omitting either source of finite-sample variation can yield overconfident conclusions.The ablation compared seed-only and example-only sampling with the full analysis.Source: §4.2 · p. 9

• The authors argue that progress on language-model pre-training requires distinguishing properties of specific model artifacts from properties of their generating procedures.Source: §5 · p. 9
• MultiBERTs provides 25 checkpoints, while the Multi-Bootstrap estimates uncertainty in model comparisons across multiple training seeds.Source: §5 · p. 9
• The resources were demonstrated by quantifying an intervention's effect on a type of gender bias in BERT-based coreference systems.Source: §5 · p. 9

• The function takes predictions, labels, a metric function, and a requested number of bootstrap samples, and returns an array of sampled performance values.Predictions are a two-dimensional array for different seeds, and labels are a one-dimensional array.Source: A MINIMAL IMPLEMENTATION OF THE… · p. 16
• Each bootstrap sample draws test examples and pre-training seeds, computes the metric for each seed, and averages the results over seeds.Source: A MINIMAL IMPLEMENTATION OF THE… · p. 16
• The complete implementation supports paired and unpaired analyses and multiple fine-tuning runs per pre-training seed.The repository version is optimized and supports the experiment designs described in Section 3.Source: A MINIMAL IMPLEMENTATION OF THE… · p. 16

B PROOF OF THEOREM 1

• Because losses share a seed across observations and an observation across seeds, the proof must handle a non-iid dependence structure.It uses a three-step argument, including coupling the bootstrap to independent multiplier-bootstrap samples and comparing their limiting distributions.Source: B PROOF OF THEOREM 1 · p. 17
• The proof centers the loss at zero without loss of generality by adding and subtracting its mean in the bootstrap expression.The argument says this centering preserves the result for a nonzero mean.Source: B PROOF OF THEOREM 1 · pp. 17–18
• Consistency is established, but an OP(1/n) convergence rate is difficult to show because the proof is not well-adapted to Edgeworth expansions.The authors suggest OP(1/√n) may be possible, while identifying convergence-rate bounds for the strong law on separately exchangeable arrays as the main difficulty.Source: B PROOF OF THEOREM 1 · p. 18

B.1 PROOF OF LEMMA 1

• The interaction term √n(Dn − D)(Mn − M)ℓ is oP(1) under the theorem’s assumptions.This is the stated result of Lemma 1.Source: B PROOF OF THEOREM 1 · p. 18

B PROOF OF THEOREM 1

• The scaled estimator and Multi-Bootstrap error both converge in distribution to N(0, σ²), with σ² determined by the two sampling components.The proof combines linearization, bootstrap coupling, a central limit theorem, and the delta method.Source: B PROOF OF THEOREM 1 · p. 19

B.2 PROOF OF LEMMA 2

• The proof uses the law of total variance and bounds the resulting variance by C/n to control the relevant remainder.It begins from a zero-mean conditional expectation and applies Chebyshev’s inequality.Source: §B.2 · pp. 20–21
• The first term of the variance decomposition is o(1/n) almost surely under the separately exchangeable-array strong law.The proof invokes the mean-zero assumption for this step.Source: §B.2 · pp. 21–22
• The two empirical terms converge almost surely to DMℓ² and (DMℓ)², respectively, supplying the stated variance-limit argument.The proof identifies the relevant empirical variance through the law of total variance.Source: §B.2 · p. 22

B.3 PROOF OF LEMMA 3

• The Lemma 3 argument applies Chebyshev’s inequality and the law of total variance, then uses the separately exchangeable-array strong law to show a term converges almost surely to zero.The stated finite second-moment condition supports the strong-law step.Source: §B.3 · pp. 22–23

B.4 PROOF OF LEMMA 4

• For Lemma 4, the proof establishes convergence first along a subsequence and then shows the full sequence is Cauchy.It uses kn = αn and monotonicity to bridge the subsequence argument to the full sequence.Source: §B.4 · p. 24

B.5 PROOF OF LEMMA 5

• The Lemma 5 proof uses Markov’s inequality, conditional independence, and the first Borel-Cantelli lemma in its convergence argument.The passage states a finite 1.5-moment condition in the bound.Source: §B.5 · pp. 25–26

C INSTANCE-LEVEL AGREEMENT OF MULTIBERTS ON GLUE

• 88% to 82%: agreement on anti-stereotypical HANS examples drops across pre-training seeds, and accuracy varies by nearly 20% across seeds.The figure groups accuracy distributions by pre-training seed, with five fine-tuning runs per checkpoint.Source: C INSTANCE-LEVEL AGREEMENT OF… · p. 27, Figure 4 · p. 27
• Agreement on GLUE test examples is over 90% except for RTE, but is 1–2% lower on average across different rather than same pre-training seeds.The table caption defines agreement as prediction accuracy for classification tasks or Pearson correlation for STS-B, and distinguishes same- and different-pretraining-seed pairs.Source: C INSTANCE-LEVEL AGREEMENT OF… · p. 27, Table 3 · p. 27

D CROSS-SEED VARIATION

• 0.45±0.11 per-seed average Winogender bias correlation versus 0.45±0.05 after seed-ignored resampling; the authors attribute most variance to pretraining checkpoints.The comparison uses sets of five coreference runs.Source: D CROSS-SEED VARIATION · p. 28
• Seed 1 differs significantly from seed 0 on Winogender bias correlation (δ = 0.203, p = 0.009), after training 25 additional randomly initialized coreference models.Table 4 identifies the unpaired Multi-Bootstrap comparison as seed 0 versus seed 1.Source: D CROSS-SEED VARIATION · p. 28, Table 4 · p. 28

E CASE STUDY: MULTIBERTS VS. ORIGINAL BERT

• The original BERT checkpoint is an outlier: 1M-step reproductions matched SQuAD but lagged on GLUE, while 2M steps matched GLUE and exceeded original SQuAD performance.The authors report difficulty matching all tasks simultaneously with one hyperparameter set.Source: E CASE STUDY: MULTIBERTS VS. ORIGINAL… · p. 29

E.1 HOW MANY STEPS TO PRETRAIN?

• Comparing BERT pre-training at default 1M steps with training to 2M, MNLI performs better at 2M (δ = 0.007, p < 0.001), while RTE (p = 0.14) and MRPC (p = 0.56) show no conclusive difference.Source: §E.1 · p. 29
• The bootstrap procedure’s individual-sample distributions of L for intervention f′ and baseline f overlap significantly, but the paired samples are highly correlated and their individual differences (L′−L) are nearly always positive.Source: §E.1 · p. 29

E.2 DOES THE MULTIBERTS PROCEDURE OUTPERFORM ORIGINAL BERT ON SQUAD?

• Because original BERT has only a single performance estimate, the comparison uses the unpaired Multi-Bootstrap procedure.The authors describe the baseline as a point estimate.Source: §E.2 · p. 29
• The unpaired procedure estimates variance across both MultiBERTs seeds and evaluation examples despite the single baseline estimate.The authors recommend a confidence interval around the baseline estimate in this setting.Source: §E.2 · p. 29
• 1.9% to 2.9% (95% confidence range; p < 0.001) for 2M-step MultiBERTs outperforming original BERT on SQuAD 2.0.The test evaluates the null hypothesis of no difference.Source: §E.2 · pp. 29–30