PaperlayerView exact revision on arXiv

Source-linked AI summary

The Leaderboard Illusion

Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker

arXiv:2504.20879v2cs.AIcs.CLcs.LGstat.ME

TL;DR

Chatbot Arena’s reliance on a single leaderboard creates opportunities for provider overfitting and unequal evaluation access. This paper audits Arena data and testing practices, finding that additional Arena training data can more than double win rates from 23.5% to 49.9% on ArenaHard.

  • Problem

    Reliance on a single leaderboard risks provider overfitting to leaderboard performance rather than advancing technology meaningfully.

    Source: §1 · p. 3

  • Method

    The study audits 2M battles across 243 models and 42 providers, combining Arena data sources with analyses of private testing, licensing, and Arena-data training mixtures.

    Source: §1 · p. 3, §2 · p. 8, §4.2 · p. 20

  • Results

    Increasing Arena training data from 0% to 70% more than doubled win rates from 23.5% to 49.9% on ArenaHard.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The findings support transparent limits on private testing and score disclosure, plus fairer model sampling and removal criteria for reliable rankings.

    Source: §9 · p. 31

    The study lacks Chatbot Arena’s raw data, limiting investigation of adversarial voting and other patterns in the complete dataset.

    Source: §7 · p. 28

Abstract

from arXiv · show

Measuring progress is fundamental to the advancement of any scientific field. As benchmarks play an increasingly central role, they also grow more susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard for ranking the most capable AI systems. Yet, in this work we identify systematic issues that have resulted in a distorted playing field. We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired. We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results. At an extreme, we identify 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release. We also establish that proprietary closed models are sampled at higher rates (number of battles) and have fewer models removed from the arena than open-weight and open-source alternatives. Both these policies lead to large data access asymmetries over time. Providers like Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the arena, respectively. In contrast, a combined 83 open-weight models have only received an estimated 29.7% of the total data. We show that access to Chatbot Arena data yields substantial benefits; even limited additional data can result in relative performance gains of up to 112% on the arena distribution, based on our conservative estimates. Together, these dynamics result in overfitting to Arena-specific dynamics rather than general model quality. The Arena builds on the substantial efforts of both the organizers and an open community that maintains this valuable evaluation platform. We offer actionable recommendations to reform the Chatbot Arena's evaluation framework and promote fairer, more transparent benchmarking for the field

1 Introduction

This introduction argues that Chatbot Arena’s dominance as an LLM benchmark has enabled selective private testing, unequal data access, and deprecation practices that distort rankings and encourage Arena-specific overfitting. It presents empirical findings and five reforms aimed at restoring fairness, transparency, and scientific trust.

Introduction

  • Chatbot Arena rankings are distorted by private variant selection, unequal sampling, and asymmetric model deprecation, creating conditions for overfitting to Arena dynamics rather than general model quality.The study audits 2M battles involving 42 providers and 243 models from January 2024 through April 2025.Source: §1 · pp. 3–4
  • 27 Meta variants were privately tested before Llama 4, while providers could retract weaker submissions and publicly disclose only favorable results.The authors report that select providers, including Meta, Google, OpenAI, and Amazon, benefited from extensive private testing and that real-world experiments and simulations show best-of-N selection systematically improves Arena ratings.Source: §1 · p. 4
  • Google and OpenAI received an estimated 19.2% and 20.4% of Arena test prompts, whereas 41 fully open-source models collectively received 8.9%.The introduction attributes these asymmetries to unequal sampling rates and deprecation policies affecting proprietary, open-weight, and open-source providers differently.Source: §1 · pp. 4–5
  • Increasing Arena training data from 0% to 70% more than doubled win-rates on ArenaHard, from 23.5% to 49.9%, indicating substantial performance gains from leaderboard access.The authors characterize this estimate as conservative because some providers may also access disproportionate private API data.Source: §1 · p. 5
  • Deprecation undermines ranking reliability: 205 of 243 public models were silently deprecated, versus 47 officially listed, and 64% of silently deprecated models were open-weight or fully open-source.The authors argue that these removals can violate assumptions of the Bradley-Terry model underlying Arena scoring and disproportionately disadvantage open models.Source: §1 · p. 5

Critical Recommendations to Restore Trust to the Chatbot Arena

  • The proposed reforms prohibit score retraction, limit private variants, equalize removals, implement fair sampling, and publish a comprehensive deprecation list.The recommendations include permanently publishing all submitted results, transparently limiting concurrent private testing, applying removals equally across model groups, reducing sampling bias, and disclosing removed models.Source: §1 · p. 7

2 Overview of Methodology

The methodology tests whether Chatbot Arena rankings reliably reflect generative AI capability by examining private testing, data access, overfitting, and model deprecation. It analyzes 2M battles spanning 243 models and 42 providers, using license-based model groupings and the Arena Score’s Bradley–Terry foundation.

2 Overview of Methodology

  • The study evaluates Arena reliability through four questions covering private testing and selective disclosure, data-access asymmetries, overfitting risk, and model deprecation.These questions correspond to Sections 3, 4.1, 4.2, and 5, respectively.Source: §2 · p. 8
  • The analysis combines 2M battles covering 243 models across 42 providers and groups models as proprietary, open-weights, or open source by reported Chatbot Arena licenses.The license groupings support comparisons of who benefits from the Arena as a free, open community resource; cited examples include Aya (Dang et al., 2024b; Üstün et al., 2024), Meta, and Gemma (Team et al., 2025).Source: §2 · p. 8

2.1 Introduction to the Arena Score

  • Chatbot Arena ranks models with an Arena Score, a normalized Bradley–Terry model that estimates skill from human pairwise comparisons.Bradley–Terry accommodates ties and missing comparisons, produces calibrated estimates and confidence intervals under its model conditions, and is presented as more statistically grounded than Elo (Boubdir et al., 2024).Source: §2.1 · pp. 8–9
  • Arena Score reliability depends on unbiased representative comparisons, transitive rankings, and a fully interconnected comparison network.Biased comparisons or fragmented matchup graphs can compromise the consistency and reliability of estimated model strengths.Source: §2.1 · p. 9

3 Results: Impact of Private Testing and Selective Retraction on Arena Scores

Private testing and selective retraction let providers submit multiple variants, retain the highest-scoring result, and systematically inflate Arena rankings. The results show that this advantage affects preferred providers, simulated rankings, and real-world scores even when variants are identical.

3.1 Preferred Providers Frequently Use Private Testing

  • Meta, Google, and Amazon were the main private-testing beneficiaries, with Meta testing 27 and Google 10 private models during January–March 2025.Meta’s private testing peaked before Llama 4, and no academic labs were observed using private testing during the period.Source: §3.1 · pp. 10–11, Figure 6 · p. 11

3.2 Simulated Experiments on Private Testing and Retraction

  • Best-of-N submissions violate unbiased sampling because providers retain the maximum observed score across multiple variants, making reported ratings extreme values rather than unbiased skill estimates.The expected best-variant estimate exceeds that of each individual variant when N ≥2 and observed performance has nonzero variance.Source: §3.2 · p. 12
  • Testing more variants increases the best discovered Arena Score, enabling weaker model families with private testing to outrank stronger families restricted to one public submission.Simulations report an approximately 50-point increase after testing 20 variants, while selective best-of-N reporting violates unbiased Bradley-Terry sampling and inflates ratings.Source: §3.2 · pp. 12–14, Figure 7 · p. 12

3.3 Real-world Chatbot Arena Experiment

  • Real-world tests raised Arena scores from 1052 to 1069 for identical Aya-Vision-8B checkpoints and from 1059 to 1097 for different Aya-Vision-32B variants.The identical-checkpoint comparison provides a conservative lower bound, while strategically selected variants produce a larger gap.Source: Figure 9 · p. 14, §3.3 · p. 15

4 Results: Impact of Data Access Asymmetries on Arena Scores

The results show that Chatbot Arena data access is highly asymmetric, benefiting a small set of commercial providers, while sustained access can produce large Arena-specific gains that do not generalize broadly. These effects arise from unequal variant testing, sampling, model deprecation, and prompt redundancy or distribution shifts.

4.1 Disparity in access to Chatbot Arena Data

  • 62.8% of Arena data went to OpenAI, Google, Meta, and Anthropic, 68 times the combined share of leading academic and nonprofit laboratories.These disparities reflect multiple private variants, provider-dependent sampling, and silent deprecation policies that appear inconsistent with publicly stated Arena policies.Source: §4 · p. 15, §4.1 · p. 17
  • Three tested variants increased collected prompts from 5.9% to 19.4%, while Google and OpenAI reached 34% maximum daily sampling versus Reka’s 3.3%.The study also identified 205 silently deprecated models, compared with 47 officially deprecated models; API-hosted providers can access all test prompts, while third-party-hosted providers may receive only 20%.Source: §4.1 · pp. 16–17

4.2 Risk of Potential Overfitting

  • 49.9% ArenaHard win-rate for 70_arena versus 23.5% for 0_arena produced a 112.3% relative gain, while MMLU declined from 66.5% to 65.9%.The 30_arena variant achieved a 42.7% win-rate and an 81.7% relative gain; ArenaHard improvements therefore diverged from broader benchmark performance.Source: Figure 10 · p. 17, §4.2 · pp. 20–21
  • Prompt distributions shifted substantially over time, while 20.14% of prompts were removed by deduplication on average and 7.3% reappeared exactly the following month.High-similarity cross-month duplication reached 9%, indicating that recent Arena data can help predict subsequent test prompts.Source: §4.2 · pp. 18–19

5 Results: Impact of Model Deprecation on Arena Scores

Model deprecation destabilizes Chatbot Arena rankings by breaking transitivity under changing task distributions and fragmenting the comparison graph. Deprecation disproportionately affects open-weight and open-source models, further compromising ranking reliability and fairness.

5.1 Transitivity Under Changing Evaluation Conditions

  • Deprecation under changing task distributions produces unreliable rankings because outdated battle histories no longer represent current model performance.In simulation, deprecating Model D between task-distribution phases caused rankings to diverge from true relative performance, lowering Models A and D while raising Models B and C.Source: §5 · p. 21, §5.1 · p. 24, Figure 14 · p. 24

5.2 Sparse Battle History Risks

  • Disconnected comparison graphs caused by model removals violate Bradley-Terry reliability and produce inaccurate skill estimates and rankings.Dense graphs aligned closely with true skill rankings, whereas sparse or disconnected graphs did not; preserving connectivity requires distributed comparisons across active models.Source: §5 · p. 21, Figure 15 · p. 25, §5.2 · p. 26

5 Results: Impact of Model Deprecation on Arena Scores

  • 87.8% of open-weight models and 89% of open-source models were deprecated, compared with 80% of proprietary models; 47 were publicly listed and 205 were silently deprecated.Silent deprecation involved reducing active sampling rates to near zero.Source: §5 · p. 21, Figure 13 · p. 22
  • The Bradley-Terry model requires stable evaluation conditions and a fully interconnected comparison network for transitive rankings to remain reliable.Changing prompt categories invalidate historical comparisons, while missing links prevent reliable inference between model groups.Source: §5 · pp. 22–23

6 Recommendations and Guidelines for Improving Leaderboards

The section proposes reforms to prevent score manipulation, constrain private testing and model retirement, improve sampling fairness, and make leaderboard operations auditable. Together, these recommendations target overfitting, provider-type bias, unstable rankings, and opaque deprecations.

Private testing and score disclosure

  • Chatbot Arena should prohibit post-submission score retraction, permanently display all tested variants, and require providers to disclose historical private testing volumes.Retraction enables selective disclosure that can obscure genuine progress and encourage overfitting.Source: §6 · p. 26
  • Chatbot Arena should cap private variants per provider for each launch and disclose the same limit to proprietary, open-weight, open-source providers, and the community.The cap should apply at the provider level because API hosting makes model-level auditing impractical, while preserving limited iteration.Source: §6 · p. 26

Model deprecation

  • Retirement rules should use clear, auditable criteria and retire the bottom 30th percentile separately within proprietary, open-weight, and open-source categories after rankings converge.Stratified pruning should be based on availability and performance to preserve balance, visibility of strong underrepresented models, and comparison-graph connectivity.Source: §6 · pp. 26–27

Sampling fairness

  • Chatbot Arena should adopt and periodically report its active sampling strategy, which prioritizes under-evaluated, high-variance model pairs to reduce ranking uncertainty.The authors report no evidence that this formally described rule is deployed in the current leaderboard.Source: §6 · p. 27

Transparency and oversight

  • Chatbot Arena should publish rolling records of all tested models, deprecations, and pairwise sampling rates, including models no longer regularly sampled.The recommendation addresses silent deprecations: FastChat marks 47 models as deprecated, while four times that number were reportedly deprecated without warning.Source: §6 · pp. 27–28

7 Limitations

The study’s conclusions are limited by incomplete Arena data, a short observation window, conservative training-data estimates, and approximate provider attribution for anonymous models.

Data access

  • Incomplete access to Chatbot Arena’s raw data prevents examining adversarial voting and excludes private battles from released datasets.Available proprietary API data covers only a subset of battles, while prior work identifies adversarial voting as a reliability concern.Source: §7 · p. 28

Temporal coverage

  • The January–March 2025 scraped-data snapshot may underestimate private-variant counts for providers with fewer model launches during that period.The period coincided with Meta’s Llama 4 launch, making Meta the provider with the most private variants in this analysis.Source: §7 · p. 28

Training scale

  • The overfitting estimates are conservative because experiments used only a fraction of the data potentially available to proprietary providers.Proprietary models may be trained on 5 to 10 times more data than used in the experiments, potentially increasing overfitting risk.Source: §7 · p. 28

Model attribution

  • Provider attribution for anonymous models relies on self-identification and may contain misattributions because model responses can be inconsistent or ambiguous.The authors provide codenames and estimated identities in Appendix E.4 to support validation.Source: §7 · p. 28

8 Related Work

Prior work shows that benchmarks shape research incentives but can suffer from overfitting, inconsistent standards, poor data quality, limited reproducibility, and weak real-world validity. Human-voting benchmarks improve evaluation of nuanced qualities and adaptability, yet remain vulnerable to manipulation and other reliability concerns.

8.1 Meta-studies on the Rigor of Benchmarking in AI

  • Benchmarking shapes machine-learning priorities, but benchmark assumptions, dependencies, and design choices can substantially affect outcomes and raise unresolved reliability challenges (Koch & Peterson, 2024; Bartz-Beielstein et al., 2020).Creating meaningful and reliable benchmarks requires addressing these design and validity challenges.Source: §8.1 · p. 29
  • Static leaderboards face contamination and implicit overfitting, while dynamic benchmarks reduce these risks but can still favor providers with disproportionate data access through repeated or best-of-N submissions (Deng et al., 2024; Golchin & Surdeanu, 2024; Yang et al., 2023).Such practices may encourage deliberate optimization for Chatbot Arena performance rather than general capability.Source: §8.1 · p. 29
  • Inconsistent metrics and task definitions hinder comparisons, while benchmark saturation, poor data quality, limited reproducibility, and weak real-world validity can make high scores misleading (Ethayarajh & Jurafsky, 2020; Ruder, 2021; Vendrow et al., 2025; Digan et al., 2021).Models may perform well on benchmarks yet underperform in practical applications as evaluations saturate.Source: §8.1 · pp. 29–30

8.2 Human Voting-based Benchmarks

  • Human voting evaluates nuanced qualities such as coherence, harmlessness, and readability, supports large-scale real-world feedback and alignment, and offers lower contamination risk than static benchmarks (Van Der Lee et al., 2019; Christiano et al., 2017; Chiang et al., 2024).Platforms commonly use Elo-like or Bradley–Terry-style rankings, while human preferences support RLHF and can mitigate some LLM-as-a-judge biases.Source: §8.2 · p. 30
  • Human-voting leaderboards remain vulnerable to de-anonymization, targeted attacks, adversarial voting, and provider-informed manipulation despite security safeguards (Huang et al., 2025; Zhao et al., 2024; Min et al., 2025).This paper does not investigate adversarial voting, but notes that testing notifications and disclosed model aliases may facilitate manipulation.Source: §8.2 · p. 30

9 Conclusion … C Unbiased Sampling: Why Selecting the Maximum Introduces Bias?

The paper argues that Chatbot Arena’s provider practices and sampling policies undermine fair, reliable rankings, while outlining reforms to restore trust. It contextualizes Arena’s human-preference and Bradley–Terry evaluation framework and formalizes why reporting the best of multiple noisy estimates creates selection bias.

9 Conclusion

  • Chatbot Arena’s preferential policies and provider coordination have jeopardized scientific integrity and reliable rankings.The paper acknowledges the organizers’ substantial work while arguing that the leaderboard’s growing importance enabled systematic problems.Source: §9 · pp. 30–31
  • The proposed reforms prohibit selective score disclosure, limit private variants equally across providers, require transparent model-removal criteria, and promote uncertainty-aware sampling.The recommendations aim to reduce ranking distortions and prevent leaderboard benefits from concentrating among a handful of providers.Source: §9 · p. 31

A Chatbot Arena Background

  • LMSYS launched Chatbot Arena in 2023, and the platform later evolved into LMArena, a community evaluation system with millions of participants and over 3 million votes.LMSYS originated as a multi-university collaboration and became a nonprofit corporation in September 2024.Source: A Chatbot Arena Background · p. 42
  • Users compare model outputs in prompt-based battles, and their preferences determine ratings through methods such as Online Elo and Bradley–Terry.Arena rankings also incorporate confidence intervals when score differences are uncertain (Chiang et al., 2024).Source: A Chatbot Arena Background · pp. 42–43, B Bradley-Terry Rating Model · p. 43

B Bradley-Terry Rating Model

  • The Bradley–Terry model represents pairwise outcomes with a design matrix and estimates latent model strengths by minimizing binary cross-entropy via maximum likelihood.Estimated coefficients can then be transformed into Elo-like ratings.Source: B Bradley-Terry Rating Model · p. 43

C Unbiased Sampling: Why Selecting the Maximum Introduces Bias?

  • For N ≥2 non-degenerate noisy skill estimates, the expected maximum is strictly greater than the common expected estimate, formalizing selection bias from reporting only the best result.If the estimate has zero variance and is constant, the maximum equals the common expectation.Source: C Unbiased Sampling: Why Selecting the… · p. 44
  • The maximum-selection result applies directly to leaderboard settings where providers submit multiple variants and report only the top-performing score.The mechanism is that statistical fluctuations cause the maximum to overestimate typical performance; the phenomenon is studied in order-statistics theory (Arnold et al. (1992); David & Nagaraja (2003)).Source: C Unbiased Sampling: Why Selecting the… · p. 44

D Data sources … E.3 Encountered Public Models in Scraping

The study combines large-scale Arena battle, prompt, leaderboard, and scraped samples to analyze distribution drift, duplication, provider access, deprecation, sampling, and private testing. Scraping de-anonymized dozens of private models across providers while also cataloging public leaderboard models.

D Data sources

  • 2M battles covering 243 models across 42 providers form the study’s core real-world data sources.These sources include historical battles, API prompts, leaderboard statistics, and randomly scraped battles.Source: D Data sources · p. 44

D.1 Public and Private Battles

  • The historical-battles dataset combines 1.8M Arena battles from April 2023–January 2025 with public and provider-shared proprietary data.Public data largely lacks prompts or completions, whereas provider-shared battles contain complete conversations and are 46% multilingual.Source: D Data sources · p. 44, §D.1 · p. 46

D Data sources

  • The API-prompts dataset contains 197,217 analyzed single-turn conversations, with 62% labeled Aya and 38% Command, for prompt-duplication analysis.The data came from 567,319 Cohere API entries collected between November 2024 and April 2025, after excluding null and multi-turn records.Source: D Data sources · p. 45
  • Leaderboard snapshots from January 9, 2024 to April 23, 2025 yield 14.3K records for 243 models, enabling analyses of provider access and model deprecation.Models are categorized as proprietary, open-weight, or open-source, and the period reflects use of the latest Bradley-Terry model after December 2023.Source: D Data sources · p. 45

E Our Scraping Methodology of LMArena Statistics

  • The scraping methodology collected 5.8K random battles and about 500 Vision-leaderboard samples during January–March 2025 while using de-anonymizing and tie-oriented prompts to avoid score updates.The scraped samples were used to estimate sampling rates, identify private variants, and track model ownership.Source: D Data sources · pp. 45–46, E Our Scraping Methodology of LMArena… · p. 47

E.1 De-anonymizing Model Identities

  • Using responses to identity prompts while crawling battles, the procedure identified 64 private models corresponding to 10 providers and captured 14 additional private models that could not be de-anonymized.Source: §E.1 · p. 47

E.2 Encountered Private Models in Scraping

  • Meta had 43 identified private models—27 on the Overall leaderboard and 16 additional models on Vision—while the private-model inventory also included models from Nvidia, Alibaba, and StepFun.Table 2 reports provider-level private models identified across the Overall and Vision leaderboards.Source: Table 2 · p. 48, §E.2 · p. 49

E.3 Encountered Public Models in Scraping

  • Google and OpenAI had the most public models active during scraping, with 15 and 9 models, respectively, alongside public models from other listed providers.The public-model inventory covered the Overall and Vision leaderboards during January–March 2025.Source: Table 3 · p. 50, §E.3 · p. 51

E.4 Assignment of Private Variants to Providers · E.5 Sampling Rates · F License Categories

This section examines how private variants are assigned to providers, reports provider-level sampling rates, and defines license categories for leaderboard models.

E.4 Assignment of Private Variants to Providers

  • The assignment table maps captured private-model aliases to providers based on responses revealing their identities, while noting that some models may have appeared in more battles than the sample captured.Identity recovery was limited by scraping errors and by battles that occurred before the de-anonymizing prompt was introduced; raspberry disclosed Amazon in only three of 37 responses.Source: Table 4 · p. 52
  • The cited variants identified themselves as Meta’s Llama models: kronus with 73 disclosures, polus with 57, frost with 47, and anonymous-engine-2 with 12.Source: §E.4 · pp. 52–53
  • Captured disclosures also attributed private variants to Google, OpenAI, Cohere, and Amazon, including Google-trained models such as gemini-test and enigma and OpenAI’s ChatGPT variants.The passages also include Cohere’s Command and Aya Vision identifiers and Amazon-built systems, but these assignments are supported by fewer or lower-salience disclosures.Source: §E.4 · pp. 55–57
  • Private variants were assigned to providers based on model responses identifying organizations including Amazon, Alibaba, xAI, Reka, StepFun, NVIDIA, OpenAI, and Google.Examples include variants claiming to be Amazon Titan, Qwen, Grok, Yasa, Llama Nemotron, ChatGPT, or Google-trained models.Source: §E.4 · pp. 58–59

E.5 Sampling Rates

  • Google and OpenAI reached a maximum daily sampling rate of 34%, whereas Reka registered the lowest rate at 3.3%.The estimate uses scraped-random-sample data collected from January to March 2025 and includes only days with at least 100 Chatbot Arena samples.Source: Table 5 · p. 60

F License Categories

  • Leaderboard models were grouped into three license categories: Proprietary, Open-Weights, and Open-Source.The categorization was derived from licenses released in LMArena’s leaderboard statistics and used for Figures 2 and 3 and related reporting.Source: F License Categories · p. 65, Table 6 · p. 65

G Data Access Estimation for Different Providers

Figure 4 estimates substantial data-access disparities: proprietary providers collectively access far more Chatbot Arena data than the broader research community, giving large-industry labs a competitive advantage.

G Data Access Estimation for Different Providers

  • Figure 4 estimates that proprietary providers collectively access considerably more Chatbot Arena data than the broader research community, which receives only a fraction.Chatbot Arena collected around 3M user votes, corresponding to approximately 6M model API calls; Figure 4 represents 5M calls because it shows only selected providers.Source: G Data Access Estimation for Different… · p. 65

H Analysis of Prompt Repetitions in Arena Data … J A. Fisher Information for a Single BT Match

The appendix shows that repeated Arena prompts enable leaderboard-specific overfitting, while private best-of-N testing inflates reported scores through selection of the strongest observed variant. This inflation is larger and persistent for genuinely heterogeneous checkpoints, whereas equal-skill BT matches provide a Fisher-information basis for quantifying Arena uncertainty.

H Analysis of Prompt Repetitions in Arena Data

  • High within-month duplication rates under exact-match and embedding-similarity metrics, together with substantial cross-month duplication, reveal recurring Arena prompts that models can learn.Figure 16 reports cross-month and within-month duplication using exact string matching and text-embedding similarity.Source: H Analysis of Prompt Repetitions in… · pp. 65–66, Figure 16 · p. 66

I.1 Background

  • Arena Score is a linear re-parameterization of Bradley-Terry log-odds, with one Arena Score point equal to ln 10/400 ≈0.00576 on the log-odds scale.The appendix uses this conversion to connect BT statistical uncertainty with Arena-score units.Source: §I.1 · p. 66

I Simulation for Expected Lift from Private Testing

  • Testing N private variants and publicly retaining only the highest-scoring one creates extreme-value selection bias because the retained estimate is conditioned on being the maximum of noisy measurements.The simulation reports a lift of 50 when 20 non-identical private variants are tested, with non-identical variants treated as the more realistic scenario.Source: I Simulation for Expected Lift from… · p. 66, §I.1 · p. 67

Asymptotics

  • For identical checkpoints, selection bias eventually disappears as n →∞ because Arena-score noise scales as 1/√n.Under identical true scores, the best-of-N uplift is driven only by measurement noise; the appendix gives 2 ln 50 = 17.7 Arena Score as a numerical illustration.Source: Asymptotics · p. 67, §I.2 · p. 67

I.3 Heterogeneous Variants (σtrue > 0)

  • Heterogeneous checkpoints differ in true skill because of initialization, training seeds, data curation, or hyperparameters, so observed Arena scores combine intrinsic performance with sampling noise.The model assigns each checkpoint a distinct true skill, with σtrue measuring the spread across variants.Source: §I.3 · pp. 67–68

I.3.1 Extreme-value uplift

  • 56 Arena Score points of bias arise at N = 50 and n = 3 000 when heterogeneous variants have σtrue = 20 Arena Score, and this selection bias does not vanish as n →∞.True skill differences make leaderboard inflation substantially larger than in the identical-variant case because increasing n removes sampling noise but not variation in checkpoint quality.Source: §I.3.1 · pp. 68–69

J A. Fisher Information for a Single BT Match

  • Each equal-skill Bradley-Terry match contributes Fisher information I = 0.25 under the βi = βj assumption, corresponding to the point of maximum Bernoulli uncertainty.The equal-strength approximation is justified as both analytically convenient and relevant to many similarly rated Chatbot Arena matchups (Casella & Berger, 2002; Hunter, 2004a).Source: J A. Fisher Information for a Single… · p. 69

K Silent Model Deprecation: Additional Details · L Transitivity Under Changing Evaluation Conditions: Additional Details

The additional analyses show that silent deprecation is widespread, disproportionately affects open models, and can reduce their sampling as private variants receive preferential exposure. A simulation appendix specifies task-dependent pairwise win probabilities for two changing evaluation conditions.

K Silent Model Deprecation: Additional Details

  • Between March 3 and April 23, 2025, 205 of 243 public models participated in an average of 10 or fewer battles, based on leaderboard-stats.Figure 18 shows active, officially deprecated, and silently deprecated models by provider, based on model activity during this period.Source: K Silent Model Deprecation: Additional… · p. 69, Figure 18 · p. 70
  • Higher sampling weights for top-10 and new models, combined with limited daily votes, can silently reduce public-model sampling as private variants increase.Private variants also receive high sampling weights because they are new models.Source: K Silent Model Deprecation: Additional… · p. 70
  • Providers including Google, OpenAI, Anthropic, Amazon, Meta, and DeepSeek AI have 3–10 actively sampled public models because top-10 models receive higher sampling weights.The passage links provider-level activity to Chatbot Arena’s sampling policy.Source: K Silent Model Deprecation: Additional… · p. 70
  • 86.6% of open-weight and 87.8% of open-source Arena models are silently deprecated, compared with 30% of officially deprecated proprietary models and 2.4% of officially deprecated open-weight models.Silent deprecations therefore disproportionately affect open-weight and open-source models.Source: K Silent Model Deprecation: Additional… · p. 70, Figure 19 · p. 71

L Transitivity Under Changing Evaluation Conditions: Additional Details

  • The changing-condition simulation assigns task-specific win probabilities to every competing model pair.These probabilities define the simulated battles under alternative task distributions.Source: L Transitivity Under Changing… · p. 71
  • Task 1 simulation probabilities include a 0.1 tie rate for A vs B.Table 7 reports the win-rates used for Task 1.Source: Table 7 · p. 71
  • Task 2 simulation probabilities include 0.2 tie rates for both A vs B and A vs C.Table 8 reports the win-rates used for Task 2.Source: Table 8 · p. 71

M Overfitting Experiments: Additional Evaluations

Additional evaluations show that training on Arena Battles improves Arena-specific scores but has little to no effect on the non-Arena MMLU benchmark, where models perform similarly.

MMLU Evaluation

Related papers

Loading 2504.20879v2…