PaperlayerView exact revision on arXiv

Source-linked AI summary

Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, Marzieh Fadaee

arXiv:2311.17295v1cs.CLcs.AI

TL;DR

Elo is increasingly used to evaluate LLMs, but its suitability for models with static capabilities has not been comprehensively examined. The paper studies reliability and transitivity through theoretical and empirical analyses, finding rating sensitivities and cases where transitivity is not assured.

  • Problem

    Elo’s compatibility with LLMs, which have static capabilities and operate in a time-agnostic context, had not been comprehensively examined.

    Source: §1 · p. 2

  • Method

    The paper uses an axiomatic framework and theoretical and empirical analyses, including human-feedback data, to examine Elo’s reliability and transitivity.

    Source: §1 · p. 3

  • Results

    Elo ratings are sensitive to comparison order and hyperparameters, and transitivity may be unreliable without comprehensive human feedback for all unique model pairs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors offer guidelines for robust Elo use, noting that deviations may lead to inaccurate rankings, particularly for closely matched models.

    Source: §1 · p. 3

    The study leaves tie outcomes and multi-category Bernoulli synthetic data for future work.

    Source: §9 · pp. 13–14

Abstract

from arXiv · show

In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" paired comparisons. However, while popular, the system's suitability for assessing entities with constant skill levels, such as LLMs, remains relatively unexplored. We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity. We conduct extensive evaluation of Elo behaviour, illustrating that individual Elo computations exhibit volatility and delving into the impact of varying the Elo rating system's hyperparameters. We show that these axioms are not always satisfied raising questions about the reliability of current comparative evaluations of LLMs. If the current use of Elo scores is intended to substitute the costly head-to-head comparison of LLMs, it is crucial to ensure the ranking is as robust as possible. Guided by the axioms, our findings offer concrete guidelines for enhancing the reliability of LLM evaluation methods, suggesting a need for reassessment of existing comparative approaches.

1 Introduction

Elo is increasingly used to rank LLMs from paired comparisons, but its compatibility with LLMs’ static capabilities has not been comprehensively examined. This study tests reliability and transitivity, finding that ratings can depend on comparison order and hyperparameters and that transitivity is not guaranteed.

1 Introduction

  • Although Elo is used for LLM evaluation, its compatibility with models that have static capabilities and operate in a time-agnostic context had not been comprehensively examined.The authors note that match ordering can affect final ratings and rankings in this setting.Source: §1 · p. 2
  • The study applies theoretical and empirical analysis grounded in human feedback to examine reliability and transitivity and provide guidance for Elo use in LLM evaluation.The paper also warns that inaccurate rankings can affect research directions and real-world NLP applications.Source: §1 · p. 3
  • Elo ratings are sensitive to comparison order and hyperparameters, and transitivity may be unreliable without human feedback for every unique model pair.Sensitivity is more pronounced for models with similar performance; the paper offers guidelines for hyperparameter selection and matchmaking.Source: §1 · p. 3

2 Elo Algorithm Explained

The section describes Elo ratings for models, including expected scores based on current ratings and a post-match update controlled by a K-factor.

2 Elo Algorithm Explained

  • Each model in the set M receives an initial numerical rating R_i.The algorithm is presented for the setting of LLM evaluation.Source: §2 · p. 3

2.1 Expected Score Computation

  • Expected scores E_A and E_B are computed for a paired match-up from models A and B’s ratings; a 400-point advantage corresponds to 10:1 odds.For equal ratings, each model’s expected score is 0.5.Source: §2.1 · pp. 3–4

2.2 Rating Update Mechanism

  • After each match, ratings are adjusted using the observed score, with the K-factor controlling how quickly ratings change.The actual score S_A is either 0 or 1.Source: §2.2 · p. 4

3 Desirable Properties of Elo

The section motivates transitivity and reliability as properties needed for Elo-based model rankings. It examines whether rankings remain logically consistent and stable across comparison orders and K-factor choices.

3 Desirable Properties of Elo

  • Robust ratings and transitive rankings matter because they allow a new model’s position to be inferred with limited pairwise annotations.The section identifies transitivity and reliability as desirable evaluation-framework axioms.Source: §3 · p. 4

3.1 Axiom 1: Transitivity

  • Elo assumes transitivity, but synthetic scenarios can violate it, potentially affecting model rankings and relative performance assessments.Transitivity means that if A beats B and B beats C, A is expected to beat C.Source: §3.1 · p. 4

3.2 Axiom 2: Reliability

  • Reliability is examined through sensitivity to match ordering and the K-factor, with Figure 2 tracking average Elo and SEM as permutations increase across win probabilities.The section notes that independent, parallel LLM matches can amplify ordering effects and that conventional chess K-factor values may not transfer directly to LLM evaluation.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

4 Synthetic Human Feedback

The paper uses controlled synthetic feedback to examine Elo properties, modeling paired model preferences with Bernoulli trials and extending the setup to multiple models and comparisons.

4 Synthetic Human Feedback

  • Synthetic Bernoulli feedback lets the authors control scenarios when costly human evaluation makes it difficult to study Elo behavior; they test transitivity and sensitivity to order and K-factor.The paper says it later extends evaluation to real-world human feedback.Source: §4 · pp. 5–6

4.1 The Bernoulli Analogy

  • The setup draws an analogy between paired human preferences and Bernoulli experiments, each yielding a win or loss.This analogy frames pairwise LLM evaluation as a sequence of outcomes.Source: §4.1 · p. 6, §4.1.2 · p. 6

4.1.1 Preliminaries

  • A Bernoulli trial has two possible outcomes, success and failure, represented by X = 1 and X = 0 with probabilities p and 1 − p.The success probability is bounded by 0 ≤ p ≤ 1.Source: §4.1.1 · p. 6

4.1.2 Mapping to Human Feedback

  • Synthetic feedback samples an outcome using P(Awin): success favors model A, while failure favors model B.The setup treats repeated comparisons between two models as independent, identically distributed trials.Source: §4.1.2 · p. 6

4.1.3 Extending to Multiple Players

  • For a finite set of models, the setup considers unique model pairs and runs multiple Bernoulli experiments for each pair.These experiments discern which model performs better over a sequence of trials.Source: §4.1.3 · p. 6

4.2 Synthetic Data Generation

  • For N independent comparisons, the number of preferences for one model follows a binomial distribution, with P(k; N, p) denoting the probability of k preferences.The binomial coefficient counts ways to choose k successes from N trials.Source: §4.2 · p. 7

5 How Robust Are Elo Scores?

The section tests whether Elo rankings for LLMs are robust to match order and hyperparameters, and whether they preserve transitivity. It finds that stability depends on win-rate differences, permutation count, and K-factor.

5 How Robust Are Elo Scores?

  • The stress tests examine ordering sensitivity, K-factor sensitivity, and transitivity to develop empirically grounded guidance for Elo use.Source: §5 · p. 7

5.1.1 Experimental Setup

  • The ordering experiment used 1,000 fixed match outcomes, reshuffled them into permutations, reset ratings to 1400, and averaged ratings across sequences.It tested win probabilities of 0.51, 0.55, and 0.6, with Nperms ranging from 1 to 10,000.Source: §5.1.1 · pp. 7–8

5.1.2 Key Findings

  • For P(Awin)≈0.5, Elo ratings were unstable in a single sequence, but averaged ratings stabilized beyond 100 permutations and reflected the preset win probabilities.At P(Awin)=0.55, Model A’s average rating consistently exceeded Model B’s across permutations; stability was high for P(Awin)≥0.6.Source: §5.1.2 · p. 8

5.2.1 Experimental Setup

  • The K-factor experiment compared values 1, 8, 16, 32, and 64 using 1,000 games and Nperms={1, 100}.It summarized differences between the models’ average final Elo scores to assess stability and expected ranking.Source: §5.2.1 · p. 8

5.3.2 Key Findings

  • With Nperms=100, K=1 still produced ranking discrepancies in scenarios R and B, while K=16 yielded more consistent rankings.The authors suggest K=1 may update too conservatively to capture transitive relations quickly.Source: Table 1 · p. 9, §5.3.2 · p. 10

5.2.2 Key Findings

  • At Nperms=1, rankings became unstable near 0.5 win probability, especially at higher K-factors; using at least 100 permutations with higher K-factors improved stability.With multiple permutations, higher K-factors also sped differentiation between models, with more reliable convergence at higher win probabilities.Source: §5.2.2 · p. 9

5.3.1 Experimental Setup

  • The transitivity tests simulated A-versus-B and B-versus-C outcomes in four scenarios, varying win probabilities from near-even to 0.75, then reordered the matches.The scenarios included strong wins in both pairs, a near-even result in either pair, and two near-even results.Source: §5.3.1 · pp. 9–10

6 Validation on Real-World Human Feedback

The section tests whether patterns from synthetic experiments hold with real human annotations and examines Elo under practical LLM-evaluation conditions. It finds that stability varies with win-rate disparity and rating settings, and that transitivity is not always maintained.

6 Validation on Real-World Human Feedback

  • Figure 5 shows three models’ Elo scores across Nperms={1, 100} and K-factor={1, 8, 16, 32}; intersecting score surfaces indicate ranking sensitivity to these settings.Source: Figure 5 · p. 11

6.1 Experimental Setup

  • The evaluation used 400 prompts from four datasets and human comparisons of Dolly and Flan models, testing Nperms={1, 100} and K-factors from 1 to 36.The evaluated pairings included one decisive Flan comparison and one near-even Dolly comparison.Source: §6.1 · pp. 11–12, Table 2 · p. 11

6.2 Key Findings

  • Ratings were consistent across K-factors and Nperms for the decisive Flan pairing, while the closely matched Dolly pairing was more volatile at Nperms=1 and stabilized at Nperms=100 with relatively small K-factors.Source: §6.2 · p. 12
  • Real-world transitivity was not universal, and rankings of similar-performing models depended on K-factor and Nperms; the authors advise caution without comprehensive paired comparisons.Source: §6.2 · p. 12

7 Empirical Guidelines for Robust Elo-based Evaluation

The guidelines recommend averaging Elo ratings across many permutations and tailoring the K-factor to the matchup. They also caution that Elo transitivity is not assured, particularly for models with similar performance.

7 Empirical Guidelines for Robust Elo-based Evaluation

  • Use numerous permutations, ideally Nperm ≥100, to improve rating stability over single or fewer permutations.The authors describe these as empirically derived recommendations for Elo-based LLM evaluation.Source: §7 · p. 12
  • Use a smaller K-factor for closely matched win rates, but a higher K-factor to speed convergence when performance disparity is clear.This matchup-dependent guidance contrasts with the one-size-fits-all K-factor frequently used in traditional Elo.Source: §7 · pp. 12–13
  • Elo does not consistently satisfy transitivity, especially when models have similar performance levels.Thus, A beating B and B beating C does not ensure Elo ranks A above C.Source: §7 · p. 13
  • The guidelines are intended to improve the robustness and interpretability of Elo-based LLM evaluations and the reliability of conclusions drawn from human judgments.Source: §7 · p. 13

8 Related Work

Prior work has proposed more complex Elo variants and studied transitivity limits in games, while broader machine-learning research examines factors affecting reproducibility. This paper focuses on Elo reliability in LLM evaluation.

8 Related Work

  • Glicko and TrueSkill add statistical complexity to Elo to address limitations involving more than two players or teams and outcomes beyond wins and losses.The cited variants are Glicko (Glickman, 1995; 1999; 2012) and TrueSkill™.Source: §8 · p. 13
  • Prior work reports Elo transitivity limitations in non-transitive cyclic games, including rock-paper-scissors and StarCraft ii.Source: §8 · p. 13
  • Machine-learning reproducibility research has examined hyperparameters, random seeds, model design, data parallelism, hardware, and test-set construction.The paper positions its evaluation of Elo hyperparameters and experimental settings as complementary to this work.Source: §8 · p. 13

9 Conclusion

The paper presents an axiomatic study of Elo reliability for evaluating LLMs with human feedback and offers guidelines for applying the system. It identifies tie outcomes and multi-category synthetic data as directions for further study.

9 Conclusion

  • The study applies an axiomatic framework to Elo reliability for LLM evaluation using human feedback, identifies factors affecting rating robustness, and offers application guidelines.Source: §9 · p. 13
  • Future work could add tie outcomes and multi-category Bernoulli synthetic data to examine Elo convergence in the fast-evolving language-model landscape.Source: §9 · pp. 13–14

A Extension to Multiple Outcomes

B Impact of Ordering on Elo Ratings: Skewed Win Rates

The appendix summarizes match-sequence effects for win probabilities Prob(A beats B) ≥0.65. Figure 6 presents observed win rates and Elo ratings across win probabilities and permutation settings.

B Impact of Ordering on Elo Ratings: Skewed Win Rates

  • The appendix summarizes the impact of match sequences on Elo ratings for Prob(A beats B) ≥0.65.Source: B Impact of Ordering on Elo Ratings:… · p. 21
  • Figure 6 compares Model A and Model B at win probabilities 0.9, 0.8, 0.7, and 0.65, showing observed win rates, single-permutation Elo, and mean and SEM across 100 permutations.Source: Figure 6 · p. 21

C Experiment Flan-t5-xxl vs. Dolly-v2-12b Results

In the Flan-t5-xxl vs. Dolly-v2-12b experiment, recorded win rates were 0.79 vs. 0.21.

C Experiment Flan-t5-xxl vs. Dolly-v2-12b Results

Loading 2311.17295v1…