Source-linked AI summary
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica
TL;DR
Existing benchmarks are increasingly saturated, while manually curating challenging, human-aligned evaluations is costly and static datasets require frequent updating. The paper introduces BenchBuilder to automatically curate and evaluate challenging prompts from crowdsourced data, producing Arena-Hard-Auto with stronger separation and 98.6% agreement with Chatbot Arena rankings at $20 per evaluation.
Problem
Static benchmarks struggle to differentiate advanced models, while manual curation and human evaluation are costly and difficult to update continuously.
Method
BenchBuilder uses LLMs to curate, filter, and validate crowdsourced prompts, while LLM judges automate model evaluation and confidence-aware metrics assess benchmark quality.
Results
Arena-Hard-Auto improves model separability and alignment with human preferences, achieving 98.6% agreement with Chatbot Arena rankings at $20 per evaluation.
Takeaways & Limitations
BenchBuilder provides a scalable framework for continuously creating challenging benchmarks from crowdsourced data, and Arena-Hard-Auto supports automated model evaluation.
Takeaways & Limitations
Arena-Hard-Auto lacks multi-turn and non-English evaluation, and the seven quality definitions may skew prompts toward technical domains.
Abstract
from arXiv · showhide
The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new, challenging benchmarks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time-consuming. To address this, we introduce BenchBuilder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality, we propose new metrics to measure a benchmark's alignment with human preferences and ability to separate models. We release Arena-Hard-Auto, a benchmark consisting 500 challenging prompts curated by BenchBuilder. Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings, all at a cost of $20. Our work sets a new framework for the scalable curation of automated benchmarks from extensive data.
1 INTRODUCTION
Existing benchmarks are becoming saturated, while manually curated challenging sets and live human evaluation are costly and difficult to update. BenchBuilder addresses this gap by automatically curating and evaluating challenging benchmarks from crowdsourced prompts.
- Static benchmarks are quickly saturating and struggle to differentiate state-of-the-art models.
- Manual curation produces challenging benchmarks but is expensive, difficult to scale, and vulnerable to leakage and overfitting.GPQA cost over $120,000 to curate 500 questions.
- Crowdsourced platforms provide fresh, open-ended queries, but human evaluation is costly and prompt quality varies.
- BenchBuilder automatically curates, filters, and validates prompts using seven quality indicators for continuous benchmark creation.
- Arena-Hard-Auto uses LLM judges and achieves stronger model separability, tighter confidence intervals, and 98.6% correlation with Chatbot Arena rankings.
- The authors release BenchBuilder and the Arena-Hard-Auto benchmark as open-source resources.
2 RELATED WORKS
Prior work spans static ground-truth benchmarks, human evaluation, LLM-based judging, and continuously updated benchmarks. However, existing live benchmarks do not focus on automatic curation for open-ended evaluation.
- Most established LLM benchmarks are static and ground-truth-based, covering domains such as math, science, coding, and reasoning.
- Human evaluation uses domain experts or crowd raters to assess response quality in open-ended interactions.
- Static benchmarks face test-set leakage, motivating benchmarks that are updated more frequently.
- Live benchmarks and community evaluations support continuously evolving assessment, but do not provide automatic curation pipelines for open-ended benchmarks.
3 HOW DO YOU MEASURE BENCHMARKS?
Meaningful benchmark comparisons require both confident model separation and agreement with human preferences. The paper introduces confidence-aware metrics to assess these properties and the magnitude of pairwise performance differences.
- Benchmarks should separate models with high confidence and agree with human preferences.
- Overly simple prompts and judgment variance make it difficult to distinguish genuine capability differences from noisy observations.
- Pearson and Spearman correlations may miss model separability, ranking instability, and the magnitude of pairwise performance differences.
- The paper develops Separability with Confidence, Agreement with Confidence, and Pair Rank Brier Score.
- Separability with Confidence uses bootstrapping to measure the percentage of model pairs whose benchmark-score confidence intervals do not overlap.
- Agreement with Confidence assigns agreement scores to model pairs based on whether two benchmarks confidently separate them and agree on their ordering.
- Pair Rank Brier Score rewards confidence in correct pairwise rankings and penalizes confidence when rankings are incorrect.
- Together, the metrics balance clear model differentiation with alignment to human preferences.
4 THE BENCHBUILDER PIPELINE AND ARENA-HARD-AUTO DATASET
BenchBuilder automatically scores, clusters, filters, and samples crowdsourced prompts to construct challenging, diverse benchmarks such as Arena-Hard-Auto and Wild-Hard-Auto.
- BenchBuilder Pipeline: BenchBuilder assigns each prompt a quality score based on seven qualities, including specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application.An LLM-based annotator scores how many qualities each prompt exhibits.
- BenchBuilder Pipeline: Prompts are encoded, dimensionally reduced, clustered with BERTopic, UMAP, and HDBSCAN, then topics are summarized and named using an LLM.The pipeline creates distinct topic clusters before filtering and sampling.
- BenchBuilder Pipeline: High-average-quality clusters are retained and sampled evenly, producing mostly well-defined, technical problem-solving queries while excluding trivial or poorly defined prompts.Clusters containing prompts such as “hi” are discarded.
- Arena-Hard-Auto Dataset: Arena-Hard-Auto starts from 200,000 Chatbot Arena prompts, filters duplicates, multi-turn conversations, and non-English content, and clusters them into 4,000 topics.The final benchmark construction samples two prompts from each of 250 randomly selected high-quality clusters.
- Benchmark Construction: Arena-Hard-Auto contains 500 prompts, while Wild-Hard-Auto samples two prompts from each of the 125 highest-quality clusters found in 150,000 WildChat queries.Wild-Hard-Auto improves separability and agreement over a random WildChat baseline.
- Validation and Cost: GPT-4-Turbo agrees with majority labels from GPT-4o, Claude-3-Opus, and Gemini-1.5-Pro on 85.6% of 200 sampled queries.Llama-3-70B-Instruct reduced annotation cost from approximately $500 to around $45 while yielding similar downstream improvement.
5 EVALUATION WITH LLM-AS-A-JUDGE
The evaluation framework uses LLM judges to compare model outputs against a strong baseline and derive preference-based rankings with confidence intervals. It evaluates benchmarks by their agreement with Chatbot Arena preferences and their ability to separate models confidently.
- LLM-as-a-Judge: LLM-as-a-Judge provides a scalable alternative to costly expert evaluation for challenging queries requiring domain knowledge and problem-solving.The framework is used to approximate human preferences.
- Benchmark Evaluation: Table 1 evaluates the top 20 models shared by Chatbot Arena and AlpacaEval, using English Chatbot Arena human-preference rankings as the agreement reference.Separability and agreement are calculated for each benchmark.
- Cross-Dataset Validation: Wild-Hard-Auto has significantly higher separability and agreement with human preference rankings than 250 randomly selected WildChat prompts.This comparison uses GPT-4-Turbo as the judge and supports BenchBuilder’s robustness across datasets.
- Pairwise Evaluation: A judge compares each model’s output with a strong baseline and rates the preference between them on a 5-point Likert scale.The scoring penalizes large losses more heavily, helping distinguish model performance.
- Ranking Procedure: Aggregated pairwise comparisons are processed with the Bradley-Terry model and bootstrapping to produce model win-rates, confidence intervals, and rankings.The procedure results in 1,000 judgments per model evaluation.
6 EXPERIMENTAL RESULTS
Experiments show that BenchBuilder produces benchmarks with strong model separation and human-preference alignment across datasets and evaluation settings. Arena-Hard-Auto remains highly aligned after style control, while ensemble judging reduces self-bias and improves evaluation quality.
- 6.2 COMPARING SEPARABILITY AND ALIGNMENT ACROSS BENCHMARKS: 87.4% separability and 90.8% agreement with Chatbot Arena were achieved by Arena-Hard-Auto at a $20 cost.The benchmark uses 500 prompts and achieves alignment comparable to rankings based on over 1 million human preferences.
- 6.2 COMPARING SEPARABILITY AND ALIGNMENT ACROSS BENCHMARKS: MT-Bench reached 89.9% Spearman correlation but only 22.6% confidence agreement with Chatbot Arena, exposing weak ranking granularity.The authors use this gap to motivate separability-sensitive evaluation beyond correlation alone.
- 6.3 COMPARING TO A SIMILAR DISTRIBUTION OF HUMAN PREFERENCE: 98.6% confidence agreement and 96.7% Spearman correlation were achieved against Chatbot Arena’s English Hard Prompt leaderboard.This leaderboard is based on a more challenging subset of human-evaluated battles, matching Arena-Hard-Auto’s challenging-query distribution.
- 6 EXPERIMENTAL RESULTS: Wild-Hard-Auto showed significantly higher separability and human-preference agreement than a random WildChat baseline.BenchBuilder identified 185 high-quality clusters from 150,000 WildChat examples and sampled 250 prompts from the 125 highest-quality clusters.
- 6 EXPERIMENTAL RESULTS: Arena-Hard-Auto significantly outperformed two random Chatbot Arena prompt baselines, while Llama-3-70B prompt annotation produced similar quality gains.These results support robustness to both prompt sampling and the choice of prompt-quality annotator.
- 6.5 MITIGATING STYLISTIC BIASES IN LLM-BASED EVALUATION: Style-controlled Arena-Hard-Auto achieved 98.6% agreement and correlation with style-controlled human rankings.Style control also prevented response length or markdown usage from gaming scores and reduced score correlation with answer length.
- 6.6 MITIGATING SELF-BIASES IN LLM-BASED EVALUATION: The GPT-4-Turbo and Gemini-1.5-Pro ensemble achieved higher separability and human-ranking alignment while reducing self-bias.GPT-based judging slightly favored OpenAI models and ranked Claude models lower than human preference.
7 LIMITATIONS
The pipeline may retain biases because its seven prompt-quality criteria do not cover every attribute and may favor technical prompts. Arena-Hard-Auto also does not yet evaluate multi-turn or non-English interactions.
- 7 LIMITATIONS: The seven prompt-quality criteria may skew the pipeline toward technical-domain prompts and omit other relevant attributes.The authors identify this as a potential bias despite drawing data from diverse distributions.
- 7 LIMITATIONS: Arena-Hard-Auto currently lacks multi-turn and non-English evaluation because suitable data and author language proficiency are limited.Future work targets multi-turn and multilingual data curation and broader prompt-quality definitions.
8 CONCLUSIONS
The paper concludes that BenchBuilder transforms crowdsourced data into challenging, evolving benchmarks with minimal human effort. Arena-Hard-Auto improves separability and human-preference alignment over existing benchmarks at low evaluation cost.
- 8 CONCLUSIONS: BenchBuilder transforms crowdsourced data into high-quality benchmarks using seven key prompt qualities.The pipeline is intended to support challenging and evolving evaluation for advanced language models.
- 8 CONCLUSIONS: 98.6% agreement with Chatbot Arena rankings was achieved by Arena-Hard-Auto at only $20 per evaluation.The authors report improved separability and alignment with human preferences over existing benchmarks.
A.1 PAIR RANK BRIER SCORE
The Pair Rank Brier Score evaluates how accurately and confidently a benchmark predicts pairwise model rankings. It uses bootstrap score distributions to estimate pairwise probabilities against ground-truth outcomes.
- A.1 PAIR RANK BRIER SCORE: Bootstrapping estimates the distribution of benchmark performance scores by sampling benchmark data with replacement.The resulting distributions support confidence intervals and rank-based performance probabilities.
- A.1 PAIR RANK BRIER SCORE: The benchmark predicts the probability that model π_i performs lower than model π_j on the ground-truth evaluation.This probability is estimated from bootstrapped score distributions, using a normal approximation when appropriate or empirical distributions otherwise.
- A.1 PAIR RANK BRIER SCORE: The ground-truth outcome O_πi≺πj equals 1 when π_i performs worse than π_j on the ground-truth metric.The Brier loss compares each pairwise probability prediction with this binary outcome across N model pairs.
A.2 STYLE CONTROL IN MODEL EVALUATION
The paper extends Bradley-Terry evaluation with response-style features to separate model strength from stylistic effects. Response length is the most influential style factor, and style control reduces verbosity-related score correlations.
- Style-controlled Bradley-Terry regression: The enhanced Bradley-Terry framework incorporates style features into pairwise model comparisons.It represents model strengths with β coefficients and style-feature effects with γ coefficients.
- Feature normalization: Feature differences are normalized to account for relative rather than purely absolute differences between responses.The paper motivates this choice with token-length comparisons such as 500 versus 520 and 20 versus 40 tokens.
- Style-controlled Bradley-Terry regression: Response length is the most influential style factor, while markdown-related features have secondary effects.The analysis normalizes style coefficients to support meaningful comparisons.
- Evaluation effects: Style control effectively reduces correlations between model scores and response verbosity.The comparison includes ordinary judging, a verbose policy, and Arena-Hard-Auto with versus without style control.
B EXAMPLES
The examples range from low-quality greetings and factual questions to technically demanding physics and computer-vision prompts. Additional examples illustrate the prompt-quality criteria and the judge’s comparison procedure.
- Prompt clusters: Prompt clusters span greetings, US-president queries, physics problem solving, and OpenCV image processing.Their mean scores increase from 2.7 for greetings to 5.5 for OpenCV image processing.
- Prompt clusters: The physics example requires calculating an airplane’s final velocity and net force from mass, acceleration, initial speed, and distance.It is labeled with specificity, domain knowledge, complexity, problem solving, technical accuracy, and real-world qualities.
- Prompt clusters: The OpenCV example asks for face detection in every video frame using PyTorch and displaying the count in the edited video.It is marked as satisfying all listed qualities.
- Prompt-quality evaluation: Prompt quality is assessed using seven criteria, including specificity, domain knowledge, complexity, and creativity, with satisfied criteria reported as a Python array.An LLM annotator analyzes each prompt and determines which criteria it satisfies.
- Assistant comparison: The LLM judge first generates an answer, then compares assistants’ responses against that answer for correctness, helpfulness, relevance, and concision.The judge ultimately reports a labeled verdict such as A>>B, A>B, A=B, B>A, or B>>A.