Source-linked AI summary
Challenges in Trustworthy Human Evaluation of Chatbots
Wenting Zhao, Alexander M. Rush, Tanya Goyal
TL;DR
Reliable human evaluation of free-form outputs remains challenging, while open community platforms collect preference judgments at scale. The paper examines apathetic, adversarial, and arbitrary votes on Chatbot Arena and finds that small fractions of poor-quality votes can affect rankings, with 10% apathetic votes shifting two of three models by five places.
Problem
Open platforms are trusted sources of LLM rankings, yet minimal quality controls raise the question of how to ensure reliable judgments without losing user scale.
Source: §1 · p. 1Method
Using Chatbot Arena as a case study, the paper analyzes apathetic, adversarial, and arbitrary sources of poor-quality preference votes.
Source: §1 · p. 1, §3.3 · p. 5Results
10% of apathetic votes can shift the leaderboard rankings of 2 of 3 models by 5 places, while small apathetic or adversarial fractions can also have non-trivial impact.
Source: §1 · p. 2, §3.1 · p. 3Takeaways & Limitations
Open platforms face a challenge of balancing quality controls with incentives and user experience that sustain participation.
Source: §4 · p. 5The analysis focuses on Chatbot Arena; extending it to similar platforms could yield insights specific to vision-language model evaluation.
Source: Limitations · p. 6
Abstract
from arXiv · showhide
Open community-driven platforms like Chatbot Arena that collect user preference data from site visitors have gained a reputation as one of the most trustworthy publicly available benchmarks for LLM performance. While now standard, it is tricky to implement effective guardrails to collect high-quality annotations from humans. In this paper, we demonstrate that three sources of bad annotations, both malicious and otherwise, can corrupt the reliability of open leaderboard rankings. In particular, we show that only 10\% of poor quality votes by apathetic (site visitors not appropriately incentivized to give correct votes) or adversarial (bad actors seeking to inflate the ranking of a target model) annotators can change the rankings of models by up to 5 places on the leaderboard. Finally, we discuss open challenges in ensuring high-quality human annotations.
1 Introduction
Open community-driven platforms collect large-scale human preferences and are widely trusted, but minimal quality controls leave their rankings vulnerable. The paper examines three sources of poor-quality votes and their effects on ranking reliability.
1 Introduction
- The paper asks whether an open community-driven platform can ensure reliable rankings without sacrificing user scale.The authors note that such rankings also serve as ground truth for auditing automatic evaluators.Source: §1 · p. 1
- The analysis considers apathetic random voting, adversarial votes intended to inflate a target model, and arbitrary preferences on open-ended subjective queries.The first two sources are examined for ranking impact; the third concerns the inherent subjectivity of some judgments.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Small fractions of apathetic or adversarial votes can materially affect target-model rankings, and such votes are difficult to detect post hoc.Even carefully recruited annotators show low agreement on subjective queries, limiting inter-annotator filtering.Source: §1 · p. 2
- The paper discusses open challenges in annotation quality and calls for research on quality controls for open platforms.It also argues that these platforms remain valuable for research on automatic evaluators and reward models.Source: §1 · p. 2
2 Background
This section describes Chatbot Arena’s preference-collection and ranking pipeline, its quality controls, and the publicly released dataset used for experiments.
2 Background
- The analysis uses Chatbot Arena as a case study, while noting that its insights may apply to similar community-driven preference platforms.The passage introduces the subsequent description of the preference pipeline and quality controls.Source: §2 · p. 2
- Users choose between two model outputs, with model identities revealed only after preference submission; each logged record includes the query, outputs, model pair, and label.Labels include either model choice or a tie.Source: §2 · p. 2
- Pairwise preferences estimate model win probabilities, which feed a Bradley-Terry model to produce scores and the final ranking.Models are sorted by their estimated scores.Source: §2 · p. 2
- Arena filters include bot detection, vote limits, prompt deduplication, and other checks, but focus more on bots than vote quality; the analysis assumes minimal backend quality filtering.The described checks include malicious-user detection and automated low-quality-data filtering.Source: §2 · p. 2
- The experiments use a public dataset of 55k preference annotations covering response pairs from 64 models.The platform uses adaptive sampling that favors uncertain model pairs and newly introduced models, though exact details are not public.Source: §2 · p. 2
- The table summarizes leaderboard changes across arbitrary-vote percentages and reports that 10% poor-quality annotations can shift 2 of 3 systems by 5 places.The caption describes ranking gains and losses for three test models; it does not provide the individual table-cell values here.Source: Table 1 · p. 3
3 Case Studies: Sources of Poor Quality Votes and Their Impact
The case studies examine apathetic, adversarial, and arbitrary preference votes and their effects on open-platform rankings. They assess how these votes can be detected or interpreted, using Chatbot Arena as the primary case study.
3 Case Studies: Sources of Poor Quality Votes and Their Impact
- The case studies consider three sources of poor-quality votes: random or low-quality apathetic votes, targeted adversarial votes, and votes on difficult-to-distinguish responses.The authors note that available filtering strategies focus more on bots than on differences in vote quality.Source: §3 · p. 3, §3.1 · p. 3
3.1 Apathetic Voting
- 10% apathetic votes can change the ranks of 2 of 3 selected models by 5 places.The models were selected arbitrarily, and their frequency in the dataset affects susceptibility to ranking changes.Source: §3.1 · p. 3
Intrinsic Evaluation of Detector Algorithm
- The target-attribution detector uses target-model logits to estimate whether an output came from that model, enabling targeted votes when it appears.On the arena dataset, it achieved 91.13% TPR and 88.46% TNR for Llama-2-7b-chat; TPR was positively correlated with token count.Source: §3.2 · pp. 3–4, Intrinsic Evaluation of Detector… · p. 4
3.2 Adversarial Voting
- 10% adversarial votes can shift all three tested models’ leaderboard ranks by more than 4 places.The experiment assumes adversaries upvote their target model; the reported attack is the simplest version, and the authors say downvoting competitors could increase the changes.Source: Table 2 · p. 4, Intrinsic Evaluation of Detector… · p. 4
Intrinsic Evaluation of Detector Algorithm
- A proof-of-concept submitted 100 queries and successfully cast preference votes after bypassing bot detection; the researchers used tie votes to avoid dataset contamination.The authors report that model up-sampling can be exploited by adversaries seeking target-model upvotes.Source: Intrinsic Evaluation of Detector… · p. 4
3.3 Arbitrary Voting
- Even well-intentioned annotators with clear guidelines showed low agreement when evaluating subjective-query responses, regardless of model-pair performance differences.The study recruited four undergraduate CS students and used ratings across five dimensions; its task differed from Chatbot Arena’s pairwise preferences.Source: §3.3 · p. 5
- Low agreement makes it difficult to distinguish apathetic votes from legitimate differences in judgment, limiting inter-annotator agreement as a filtering strategy.The authors argue arbitrary votes can still signal relative model performance, but become problematic when most queries fail to distinguish models; informative examples could be up-weighted.Source: §3.1 · p. 3, §3.3 · p. 5
4 Conclusion & Future Directions
The conclusion argues that open community-driven leaderboards can be corrupted by apathetic or adversarial votes, making it important to balance quality controls with a usable experience. It highlights richer feedback, stronger guardrails, and dataset access as directions for addressing annotation quality.
4 Conclusion & Future Directions
- Reliable leaderboards require balancing necessary quality controls with incentives and a user experience that sustains platform use.The authors identify intentional and unintentional corruption as concerns for platforms treated as ground-truth rankings.Source: §4 · p. 5
- Rationales and fine-grained feedback may encourage more critical voting or abstention and help filter low-quality annotations.The paper also notes that fine-grained preferences and added evaluation context are recommended, but incorporating these into open platforms should not inordinately increase annotation burden (Gehrmann et al., 2023; Krishna et al., 2023; Goyal et al., 2022b; Malaviya et al., 2024).Source: §4 · pp. 5–6
- Potential guardrails include reputation systems, CAPTCHA, machine-learning anomaly detection, and annotator behavior-trace methods.These are presented as approaches for the community to consider (Adler and de Alfaro, 2007; Von Ahn et al., 2003, 2008; Kumar et al., 2014; Wu et al., 2016; Goyal et al., 2018).Source: §4 · p. 6
- Public access to collected platform data could spur research on annotation problems and clarify which queries distinguish models and where model families are limited.The authors frame dataset release as a resource for examining these questions.Source: §4 · p. 6
Limitations
The paper analyzes Chatbot Arena only, while noting similar lax annotation guardrails on other platforms and potential insights from extending the analysis.
Limitations
- The analysis is limited to one open community-driven platform: Chatbot Arena.The authors identify this as the scope of the paper.Source: Limitations · p. 6
- The authors note that WildVision Bench has similarly lax guardrails around annotation quality.This is offered as an example of a comparable platform beyond the study's scope.Source: Limitations · p. 6
- Extending the analysis to similar platforms could yield insights specific to vision-language model evaluation.The authors state this as a potential benefit of broader platform coverage.Source: Limitations · p. 6
A Model Attribution Algorithm
The appendix outlines the model attribution algorithm used in the adversarial attack, including a binary decision about whether an output is likely from the target model.
A Model Attribution Algorithm
- The algorithm is presented as the model-attribution component used to carry out the adversarial attack.The appendix outlines the algorithm's operation.Source: A Model Attribution Algorithm · p. 7
- Its stated inputs include a target model, input sequence, output sequence, probability threshold, and decision threshold.The output sequence is denoted y=(y1, y2, ..., yN).Source: A Model Attribution Algorithm · p. 8
- The algorithm's output distinguishes whether an output is likely or unlikely to come from the target model.The listed steps append 1 or 0 to the results, respectively.Source: A Model Attribution Algorithm · p. 8