PaperlayerView exact revision on arXiv

Source-linked AI summary

Drawing Conclusions from Draws: Rethinking Preference Semantics in Arena-Style LLM Evaluation

Raphael Tang, Crystina Zhang, Wenyan Li, Carmen Lai, Pontus Stenetorp, Yao Lu

arXiv:2510.02306v1cs.CL

TL;DR

Arena-style ratings treat draws as model parity, but this paper questions whether draws instead reflect query properties. Across three real-world datasets and four rating systems, it tests ignoring draw updates and finds improved prediction accuracy, alongside associations between draws and very easy or highly objective queries.

  • Problem

    Arena-style rating systems treat draws as equal ability and equalize ratings, but the paper questions whether draws indicate model parity rather than query properties.

    Source: §1 · p. 1

  • Method

    The paper evaluates established rating systems with draw updates ignored and analyzes how query difficulty, subjectivity, and model ratings relate to draw probability.

    Source: §1 · p. 1

  • Results

    Ignoring draw updates improved outcome prediction accuracy by 1–3% across three real-world datasets, while draws were associated with very easy and highly objective queries (risk ratios 1.37 and 1.35).

    Source: §1 · p. 1, §5 · p. 4

  • Takeaways & Limitations

    Future rating systems should reconsider draw semantics and explicitly account for query properties in rating updates.

    Source: §5 · p. 4

Abstract

from arXiv · show

In arena-style evaluation of large language models (LLMs), two LLMs respond to a user query, and the user chooses the winning response or deems the "battle" a draw, resulting in an adjustment to the ratings of both models. The prevailing approach for modeling these rating dynamics is to view battles as two-player game matches, as in chess, and apply the Elo rating system and its derivatives. In this paper, we critically examine this paradigm. Specifically, we question whether a draw genuinely means that the two models are equal and hence whether their ratings should be equalized. Instead, we conjecture that draws are more indicative of query difficulty: if the query is too easy, then both models are more likely to succeed equally. On three real-world arena datasets, we show that ignoring rating updates for draws yields a 1-3% relative increase in battle outcome prediction accuracy (which includes draws) for all four rating systems studied. Further analyses suggest that draws occur more for queries rated as very easy and those as highly objective, with risk ratios of 1.37 and 1.35, respectively. We recommend future rating systems to reconsider existing draw semantics and to account for query properties in rating updates.

1 Introduction

Arena-style ratings commonly treat draws as equal ability and equalize model ratings, but this paper questions that interpretation. It tests whether draws reflect query properties and reports that ignoring draw updates improves prediction accuracy.

1 Introduction

  • The paper questions the game-match assumption that a draw indicates equal model ability and should equalize ratings.Existing approaches commonly apply Elo and its derivatives, treating draws as equal ability.Source: §1 · p. 1
  • The authors hypothesize that draws relate more strongly to query difficulty and subjectivity than to model-ability parity.They propose that easy queries let both models succeed equally and objective queries make matching answers more likely.Source: §1 · p. 1
  • Ignoring draw updates increased battle prediction accuracy by a relative 1–3% across four rating systems and three real-world datasets.The improvement was present in 11 of the 12 dataset–rating system combinations.Source: §1 · p. 1
  • Queries rated 0 out of 5 for difficulty and subjectivity were associated with a 35–37% increased relative risk of observing a draw, whereas rating proximity had no substantial connection to draw probability.Source: §1 · p. 1
  • The authors recommend reconsidering draw semantics and accounting for query properties in future rating updates.They report that difficulty and subjectivity better predict draw likelihood than rating closeness.Source: §1 · p. 1

2 Arena-Style Evaluation

Arena-style evaluation elicits judgments on paired LLM responses, then updates the models’ ratings from each outcome. The section formalizes this process and describes four rating systems through their update rules.

2.1 Preliminaries

  • Arena-style evaluation elicits user judgments on paired LLM responses, then adjusts the two models’ ratings based on the outcome.Wins transfer points from the losing model to the winner; draws narrow the rating gap.Source: §2.1 · p. 2
  • Battles form a time-ordered sequence of model pairs and relative judgments; each update rule maps their ratings and outcome to new ratings while leaving other models unchanged.Source: §2.1 · p. 2

2.2 Rating Systems

  • Chatbot Arena’s online Bradley–Terry model updates ratings for wins and losses, while a draw applies simultaneous win and loss updates that reduce the rating gap.Source: §2.2 · p. 2
  • TrueSkill treats ratings as Gaussian performance distributions and computes posterior ratings with Bayesian message passing over a factor graph.Source: §2.2 · p. 2
  • Elo uses expected win probabilities and a learning-rate update based on the observed outcome.Source: §2.2 · p. 2
  • Glicko-2 tracks rating deviation and scales updates with uncertainty and volatility; τ is a constant for volatility change.Source: §2.2 · p. 2

3 Experiments

The experiments test whether omitting draw updates improves arena rating predictions and whether draws track query properties or model-rating closeness. Results favor omitting draw updates and associate draws more with very easy, highly objective queries than with rating closeness.

3 Experiments

  • The study uses 106K LMArena, 24K SearchArena, and 30K VisionArena battles; roughly 30–40% of each dataset are draws.The four systems are evaluated with chronological prequential battle prediction accuracy, using a draw margin ε for outcome decisions.Source: §3 · p. 3

3.1 Draw Update Ablation Study

  • Ignoring draw updates improves overall accuracy by a relative 0.5–3.0% on average across four rating systems; gains are significant in 18 of 23 cases.Median overall and win–loss accuracy improvements are 1.2% and 0.7%, respectively; random omission of an equivalent number of updates has net-zero change.Source: §3.1 · p. 3
  • For the draw-update ablation, the first 5% of each dataset calibrates ε, while the remaining 95% validates predictions.The best ε from the draw-update condition is reused for the no-draw-update condition to avoid biasing the comparison.Source: §3.1 · p. 3

3.2 Draw Semantics Study

  • Draws are more likely at difficulty and subjectivity ratings of 0 (RRs 1.37 and 1.35), while model-rating closeness is largely uninformative.Rating-difference RRs stay close to 1.0 until the 90–100th percentile, where they range from 0.89–0.96.Source: §3.2 · p. 4, Figure 1 · p. 4, Figure 2 · p. 4

4 Related Work

Prior work examines arena evaluation's origins and judging pitfalls, as well as limitations of Elo-based ratings. This paper positions its analysis as complementary by questioning draw semantics.

4 Related Work

  • Research on arena evaluation has examined pairwise-judging pitfalls including position bias, test contamination, and misalignment with real-world utility.The discussion traces ordinal response comparisons to InstructGPT and situates arena evaluation alongside broad-coverage benchmarks such as BIG-Bench.Source: §4 · p. 4
  • Other studies identify Elo reliability, transitivity, and stylistic-fluency concerns; this work complements them by questioning draw semantics.The cited work proposes am-ELO and MERS to address distinct rating concerns.Source: §4 · p. 4

5 Conclusions

Across three real-world datasets, the paper finds that ignoring draw updates improves prediction accuracy and that draws are associated with query properties. It recommends reconsidering draw semantics and accounting for query properties in future rating systems.

5 Conclusions

  • Across three real-world datasets, ignoring draw updates improves outcome prediction accuracy by 1–3%, while draws are disproportionately associated with very easy and highly objective queries.The respective risk ratios are 1.37 and 1.35; the paper recommends reconsidering draw semantics and accounting for query properties in future rating systems.Source: §5 · p. 4

A Further Ablation

Varying the draw threshold reveals a trade-off between draw and win–loss prediction accuracy, with ignoring draw updates performing better.

A Further Ablation

  • Ignoring draw updates achieves higher AUC and is Pareto-better than including all updates across draw-threshold trade-off curves.The curves vary the draw threshold ε and compare draw with win–loss prediction accuracy.Source: A Further Ablation · p. 6

B Example User Interface

Figure 4 presents an example interface for arena-style evaluation, where users choose which response is better or declare a draw.

Related papers

Loading 2510.02306v1…