Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench

1 Introduction

• BAT validates a benchmark by comparing its model scores with established benchmarks using statistical agreement metrics.Creators may use BAT to validate benchmark properties, while consumers may use it to choose benchmarks.Source: §1 · pp. 1–2
• The paper identifies a lack of common BAT methodology and says overlooked methodological choices can undermine the validity of conclusions.Source: §1 · p. 2
• The authors analyze more than 50 common benchmarks spanning over 200 models to examine the impact of methodological choices in BAT.Source: §1 · p. 2
• Different model selections can change agreement conclusions: two benchmarks may agree highly across broad model ranges but less among top-ranked models.The paper focuses on reference benchmarks, model selection, and correlation metrics and their interpretation.Source: §1 · p. 2
• The authors propose BAT best practices and implement them in BenchBench, a Python package that supplies a framework and benchmark data.The paper also introduces the BenchBench-Leaderboard, which ranks benchmarks by agreement with user-selected reference benchmarks.Source: §1 · p. 2

2 Setup

• Over 40 benchmarks are included, with results cutoff at January 2024.The listed examples include AGI Eval, Alpaca, MMLU, Chatbot-Arena, MTBench, and Big Bench Hard.Source: §2 · pp. 2–3
• Agreement is calculated across two benchmarks using Kendall correlation over model ranks or Pearson correlation over model scores.Source: §2 · p. 3
• At least five intersecting models are required for reliable BAT in this work; benchmarks with fewer cannot be reliably used.Intersecting models are those appearing in both benchmarks.Source: §2 · p. 3

3 BAT Methodological Decisions: An Analysis

• BAT involves choices about reference benchmarks, model selection, metrics, and agreement thresholds, but benchmark creators often make arbitrary choices without consistent justification.The analysis examines how these choices can hinder the validity of BAT conclusions.Source: §3 · p. 3

3.1 The Choice of Reference Benchmark Matters

• Alpaca V2 agreement ranges from 0.57 with MT-bench to 0.82 with LMSys Arena, despite both references measuring similar abilities.Different seemingly appropriate references can produce widely varying scores, calling single-reference conclusions into question.Source: §3.1 · p. 3, Figure 3 · p. 3

The Number of Compared Models Matters

• Kendall-tau correlation between LMSysArena and MT-Bench ranges from approximately 0.65 to 0.99 depending on the number of models chosen.With small model subsets, BAT standard deviation can approach 0.25.Source: The Number of Compared Models Matters · p. 4, Figure 5 · p. 5
• Agreement for adjacent-ranked model sets is lower than for random samples, with the difference growing as the subset gets smaller.Models of similar strength can yield lower agreement than sets spanning varying model qualities.Source: The Number of Compared Models Matters · p. 4, Figure 4 · p. 4
• Reporting agreement at multiple granularities helps consumers assess whether benchmarks distinguish the model-quality range they care about.Consumers may seek either the strongest models or differences between stronger and weaker models.Source: The Number of Compared Models Matters · p. 4

3.3 The Choice of Correlation Metric (and Threshold) Matters

• Agreement scores depend on the correlation metric: Kendall-tau and Pearson have a strong linear relationship (r2 = 0.85) but differ by approximately 0.2.The score offset makes applying the same threshold across metrics potentially flawed.Source: §3.3 · p. 5, Figure 6 · p. 5

• More than 30% lower standard deviation results when an aggregate reference benchmark replaces an arbitrary reference.The aggregate combines model win-rates from benchmarks appropriate to the BAT goal.Source: §4 · p. 5
• Compare a target benchmark’s agreement with the distribution of other benchmarks’ scores against the same reference, using a Z-score and the −1σ threshold.Scores above −1σ count as agreement; those below do not, and the distribution updates as benchmarks are added.Source: §4 · pp. 5–6
• Use at least 10 randomly selected models, preferably more, spanning the available model spectrum.This sampling approach decreases BAT variance by more than 30%; larger samples raise computational costs.Source: §4 · p. 6
• Report agreement at multiple model granularities, such as 5, 10, and 20 contiguous models, to show distinctions a single resolution can miss.The recommendations include averaging across groups when more models are sampled.Source: §4 · p. 6
• BenchBench implements the recommended workflow to support reproducible BAT with complex statistical tools and benchmark data.The package addresses the difficulty of applying the guidelines in practice.Source: §4 · p. 6
• ∼67% lower standard deviation is achieved when the methodological recommendations are combined, yielding more robust BAT results.The recommendations also affect variance individually.Source: §4 · p. 6

• BenchBench implements the BAT guidelines and holds results from multiple benchmarks, standardizing practice and supporting reproducible analysis.Users can supply results for recommended models and receive a BAT report.Source: §5 · p. 6
• Users can upload benchmark results to the BenchBench database, enriching the reference distribution for future BAT efforts.Applying the best practices through BenchBench does not require additional computational resources.Source: §5 · pp. 6–7
• The BenchBench-leaderboard ranks submitted benchmarks by agreement with a desired group of reference benchmarks.The depicted default uses the aggregate of holistic references and randomly sampled subsets of 20 models; rankings may change as that reference set grows.Source: §5 · p. 7, Figure 7 · p. 7

• Liu et al. (2021) used high agreement across QA datasets to argue that more QA datasets were unnecessary, while Sun et al. (2023) reported that compositionality benchmarks did not agree.Sun et al. used Kendall-Tau and a 0.7 threshold.Source: §6 · p. 7
• Feng et al. (2024), Lei et al. (2023), and Viswanathan et al. (2023) used BAT to assess instruction-sampling or synthetic benchmarks against existing benchmarks.Chang et al. (2023) used agreement to assess whether two proposed benchmarks captured the same phenomenon.Source: §6 · pp. 7–8
• Other cited uses include testing agreement across prompts within a benchmark, validating a benchmark with six models, and comparing efficient HELM versions with the full benchmark.Mizrahi et al. (2023), Li et al. (2024b), and Perlitz et al. (2023) are cited for these examples, respectively.Source: §6 · p. 8
• Yuan et al. (2024) and Waldis et al. (2024) used low BAT scores against established benchmarks to show divergent validity.Source: §6 · p. 8

7 Discussion and Conclusions

• The authors identify choices of models, reference benchmarks, and thresholding schemes as factors that can significantly alter BAT conclusions, and offer best practices, BenchBench, and a leaderboard.The leaderboard quantifies a benchmark’s agreement with an aggregate of existing benchmarks.Source: §7 · p. 8
• The paper focuses on BAT methodology, leaving when BAT should be used and how its conclusions should be interpreted as open questions.These questions are distinguished from the methodological issues addressed in the study.Source: §7 · p. 8
• High agreement does not establish that two benchmarks measure the same qualities, though users may choose a more convenient benchmark when agreement is high.The discussion also distinguishes correlations in model abilities from correlations in the qualities measured by benchmarks.Source: §7 · p. 8
• The study accepts benchmark scores as presented and does not incorporate reliability, which can bound agreement when a benchmark cannot distinguish its top models.The authors call for future BAT methods to incorporate reliability measures.Source: §7 · p. 8

How do we use BAT to retire benchmarks?

• BAT may help identify benchmarks that are no longer viable when the mixture of abilities expected of models shifts over time.The discussion presents this as an additional possible reason for benchmark retirement beyond saturation.Source: How do we use BAT to retire benchmarks? · p. 8

7 Discussion and Conclusions

• The conclusion presents best practices, BenchBench, and the BenchBench-leaderboard as contributions to more standardized benchmark evaluations.The leaderboard quantifies agreement with an aggregate of existing benchmarks.Source: How do we use BAT to retire benchmarks? · p. 8, §7 · p. 8

8 Limitations

• Low agreement may reflect either a benchmark measuring something other than intended or unreliable, unstable rankings.When rankings are unreliable, the same benchmark may not agree with itself under small changes.Source: §8 · p. 8
• For unstable rankings, the authors say more examples or configurations are usually needed.The discussion cites Choshen et al. (2024) and Bandel et al. (2024) for these remedies.Source: §8 · p. 9
• BAT may be unnecessary when stronger sources of authority exist, and unique signals may show difference without validating their distinct purpose.The authors give expert-crafted benchmarks as one possible alternative source of authority.Source: §8 · p. 9
• BAT needs a reference benchmark, ideally multiple benchmarks measuring diverse aspects of the same construct, but results can be sensitive to reference choice.Source: §8 · p. 9

9.1 Benchmarks used

• The analysis includes benchmarks spanning cognition, instruction following, coding, knowledge, conversation, emotional intelligence, reasoning, and text understanding.Examples named include AGI Eval, Alpaca, HumanEval, MMLU, Chatbot-Arena, EQ-Bench, MAGI, and Big Bench Hard.Source: §9.1 · p. 11

9.2 Model Tier

• Bottom-tier models show Kendall correlations just below 0.5, middle-tier models below 0.2, and top-tier models around 0.3.Figure 8 plots correlation by model subset size, with error bars showing SEM across benchmark pairs.Source: Figure 8 · p. 11, §9.2 · p. 11

9.1 Benchmarks used

• HELM was excluded because few models overlapped with the other benchmarks.Source: §9.1 · p. 11

9.2 Model Tier

• The authors suggest low reliability may help explain the tier differences; bottom-ranked models have greater score standard deviations, but they caution against strong conclusions about excluding older models or switching benchmarks.They note that at most, old models may be left out of BAT, while other effects seem more pressing.Source: §9.2 · p. 12

9.3 Benchmark used for visualizations

• The visualization benchmark list includes AGI Eval, Alpaca, HuggingFace OpenLLM Leaderboard, MMLU, Chatbot-Arena, MTBench, Big Bench Hard, ARC, HellaSwag, TruthfulQA, Winogrande, and EQ-Bench.The listed benchmarks have permissive licenses allowing academic use.Source: §9.3 · p. 12