PaperlayerView exact revision on arXiv

Source-linked AI summary

WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, Yejin Choi

arXiv:2406.04770v2cs.CLcs.AI

TL;DR

Evaluating LLMs on open-ended, real-world queries remains difficult because existing benchmarks do not adequately reflect the diversity and challenge of user tasks. WildBench addresses this gap with a curated real-user benchmark, checklist-based LLM judging, and pairwise and individual metrics. Its metrics strongly correlate with human rankings, with WB-Reward reaching 0.98 Pearson correlation and WB-Score 0.95 on top-ranking models.

  • Problem

    Existing benchmarks insufficiently capture the varied and unexpected nature of real-world user inquiries and provide limited evidence for holistic evaluation.

    Source: §1 · pp. 1–2

  • Method

    WildBench curates 1,024 challenging real-user tasks and evaluates responses with task-specific checklists, WB-Reward pairwise comparisons, and WB-Score individual scoring.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 6

  • Results

    0.98 Pearson correlation for WB-Reward and 0.95 for WB-Score with human-voted Chatbot Arena Elo ratings on top-ranking models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    WildBench provides a realistic, dynamic, and interpretable framework for evaluating model capabilities across challenging real-user tasks.

    Source: §6 · p. 10

    WildBench cannot exactly match Chatbot Arena’s user and task distribution because the underlying users and data are not fully comparable or public.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) using challenging, real-world user queries. WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. For automated evaluation with WildBench, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses task-specific checklists to evaluate model outputs systematically and provides structured explanations that justify the scores and comparisons, resulting in more reliable and interpretable automatic judgments. WB-Reward employs fine-grained pairwise comparisons between model responses, generating five potential outcomes: much better, slightly better, slightly worse, much worse, or a tie. Unlike previous evaluations that employed a single baseline model, we selected three baseline models at varying performance levels to ensure a comprehensive pairwise evaluation. Additionally, we propose a simple method to mitigate length bias, by converting outcomes of ``slightly better/worse'' to ``tie'' if the winner response exceeds the loser one by more than $K$ characters. WB-Score evaluates the quality of model outputs individually, making it a fast and cost-efficient evaluation metric. WildBench results demonstrate a strong correlation with the human-voted Elo ratings from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates, as well as the 0.87 for regular win rates.

1 INTRODUCTION

WildBench addresses the difficulty of evaluating LLMs on diverse, open-ended real-world queries by providing a challenging benchmark and structured automatic judging. Its metrics and evaluation designs achieve strong agreement with human rankings.

Motivation

  • Existing benchmarks inadequately represent real-world user tasks because they vary in breadth, diversity, and skill coverage.AlpacaEval emphasizes information-seeking tasks, while ArenaHard concentrates heavily on coding and debugging.Source: §1 · pp. 1–2

Benchmark

  • WildBench V2 contains 1,024 curated tasks from real user-chatbot dialogs, preserving natural task distributions while excluding tasks judged easy by all models.Finalized tasks also undergo manual review.Source: §1 · p. 2
  • WildBench tasks are substantially harder than AlpacaEval tasks because they involve complex constraints, depth, realism, and higher-order reasoning.Examples include constrained coding, multi-constraint creative writing, and software-system design.Source: §1 · p. 2, Figure 1 · p. 2

Evaluation

  • WB-Reward compares model responses pairwise using five outcomes, while three baselines provide a more comprehensive assessment across performance levels.The three-baseline design replaces the single-baseline approach used by most prior works.Source: §1 · p. 2
  • WB-Score evaluates generations individually for faster, more cost-effective assessment, while a length-penalty method converts slight wins or losses to ties when the winner is significantly longer.Both metrics use automated evaluation designs intended to address practical judging challenges.Source: §1 · pp. 2–3

Results

  • 0.98 Pearson correlation for WB-Reward and 0.95 for WB-Score with human-voted Chatbot Arena Elo ratings on top-ranking models.These correlations exceed the reported results for ArenaHard and AlpacaEval2.0.Source: §1 · p. 3

2 WILDBENCH DATA CURATION

WILDBENCH curates challenging, diverse tasks from real user-chatbot conversations and updates the benchmark to reflect evolving user interactions. Its task distribution is more balanced and its queries support longer-context evaluation than several existing benchmarks.

Mining Challenging Tasks from WildChat

  • WILDBENCH retains 1,024 challenging and diverse tasks from WildChat’s one million human-chatbot conversations after filtering, difficulty annotation, and human quality control.Tasks rated easy by all annotating models were excluded before sampling and review.Source: §2.1 · pp. 3–4

Benchmark Statistics

  • Only ArenaHard and WILDBENCH among the compared benchmarks use real-user queries, while WILDBENCH aligns its distribution with real users’ task categories rather than overemphasizing coding and debugging.WILDBENCH is designed to be updated regularly, with V1 and V2 released on different WildChat data iterations.Source: §2.2 · p. 4, §2.1 · p. 4

Long-Context Tasks

  • WILDBENCH includes conversations of up to four turns, with over 20% containing more than two turns, and has longer queries than comparison benchmarks.These properties support evaluation of long-context problem-solving abilities.Source: §2.2 · p. 4

Task Categories

  • WILDBENCH’s task-category distribution is more balanced than AlpacaEval’s and ArenaHard’s, which concentrate over 50% of tasks in Information seeking and Coding & Debugging, respectively.The benchmark categorizes tasks into 12 categories for fine-grained capability analysis.Source: §2.2 · p. 4

3 AUTOMATIC EVALUATION WITH WILDBENCH

WILDBENCH uses task-specific checklists and structured LLM judging to make automatic evaluation more consistent and interpretable. It combines pairwise and individual metrics, with multiple baselines and an adjustable length penalty to address evaluation cost and length bias.

Instance-Specific Checklists

  • Task-specific checklists contain 5–10 interpretable, verifiable questions and are finalized from GPT-4-Turbo and Claude-3-Opus outputs before manual review.LLM judges use the checklists to evaluate responses against criteria tailored to each test query.Source: §3.1 · p. 5

Pairwise Evaluation

  • WB-Reward compares two model responses with a GPT-4-Turbo judge using structured checklists and step-by-step analysis before producing a final judgment.The comparison outcomes receive rewards from +1 for much better to −1 for much worse, with intermediate values for slight wins or losses and 0 for ties.Source: §3.2 · pp. 5–6
  • WB-Reward (Mix) averages rewards against GPT-4-Turbo-0429, Claude-3-Haiku, and Llama-2-70B-chat across 1,024 examples to reduce single-baseline noise and bias.Using multiple baselines provides a more robust performance evaluation on WILDBENCH.Source: §3.2 · p. 6

Length-Bias Mitigation

  • The length penalty converts slight wins or losses into ties when the winning response exceeds the losing response by more than K characters.K is customizable, while K = ∞ disables the penalty.Source: §3.2 · p. 6

Individual Evaluation

  • WB-Score evaluates each response individually on a 1–10 scale using checklist-based strengths and weaknesses, making evaluation quicker and more cost-effective than pairwise comparison.Scores are averaged after rescaling each score as S′ = (S − 5) × 2.Source: §3.3 · p. 6

4 RESULTS & ANALYSIS

WILDBENCH analyzes model performance through leaderboard results, task-category breakdowns, length-bias checks, and correlations with human judgments. Its three-baseline design groups models into performance tiers, while checklist-based scoring and length penalties support robust evaluation.

Leaderboard analysis

  • WILDBENCH’s leaderboard supports task-wise performance comparisons and side-by-side model-output inspection.These features are intended to expose model strengths and weaknesses across tasks.Source: §4 · pp. 6–7
  • Three baselines group tested models into three performance tiers relative to GPT-4-Turbo, Claude 3 Haiku, and Llama-2-70B-chat.Tier 1 exceeds Claude 3 Haiku; Tier 2 falls between Claude 3 Haiku and Llama-2-70B-chat; Tier 3 is below Llama-2-70B-chat.Source: §4 · p. 7
  • GPT-4-Turbo-0409 and Claude 3 Opus perform well across all five task categories, whereas Llama-3-8B-Inst and Yi-1.5-34B-chat are weaker on coding and mathematics.The six-model analysis uses WB-Score across five categories merged from twelve categories.Source: §4.1 · p. 7
  • Llama-3-8B-Inst-SimPO generally trails Yi-34B-chat and Llama-3-70B-Inst but performs comparably to Llama-3-70B-Inst on information-seeking and creative tasks.This differs from AlpacaEval-2.0, where the 8B model’s length-controlled win rate exceeds the 70B model’s.Source: §4.1 · p. 7
  • Llama-3-8B-Inst-SimPO consistently improves on Llama-3-8B-Inst across all task categories and outperforms some larger models.Its radar-plot profile retains a similar shape to the base 8B model.Source: §4.1 · p. 8

Ablation studies and discussions

  • Checklist use raises WB-Score’s Pearson correlation from 0.905 without checklists to 0.925 with checklists.The ablation compares prompts with and without the checklist components.Source: §4.3 · p. 9
  • K = 500 provides the highest correlation with human judgments among the tested length-penalty settings.The experiment evaluated K values of 100, 200, 500, 1000, and infinity.Source: §4.3 · p. 9

5 RELATED WORKS

Related benchmarks evaluate LLMs using close-ended questions, expert- or crowd-curated open-ended tasks, and real-world user data. WILDBENCH belongs to the in-the-wild evaluation line while emphasizing automated assessment and broader task coverage.

Close-ended benchmarks

  • Close-ended benchmarks such as MMLU, CMMLU, KMMLU, MMLU-Pro, and GPQA primarily use multiple-choice questions.These benchmarks cover multiple subject areas and, in some variants, more challenging or language-specific evaluation settings.Source: §5 · p. 9

Expert-curated and crowdsourced data

  • Expert-curated and crowdsourced benchmarks include MT-Bench’s manually created category examples and AlpacaEval’s author-written instructions.The cited AlpacaEval examples primarily involve simple tasks such as rewriting.Source: §5 · p. 9

In-the-wild data

  • In-the-wild benchmarks source prompts from real-world use cases, including ChatbotArena, WildVision, and ArenaHard.ChatbotArena relies on extensive human feedback, whereas ArenaHard selects user queries for automatic evaluation.Source: §5 · p. 10

Evaluation methods

  • Open-ended evaluation commonly uses either individual-output grading or pairwise comparison because no single valid ground truth exists.LLM judges are used to reduce the cost and time of human evaluation.Source: §5 · p. 10

Data leakage prevention

  • WILDBENCH keeps a subset of WildChat private to reduce contamination risk while releasing a public validation set and construction details.The private subset is reserved as expert-curated evaluation data.Source: §5 · p. 10

Other dimensions for evaluation

  • LLM evaluation research also addresses safety, fairness, logical reasoning, agentic planning, and hallucination detection beyond capability evaluation.These dimensions are identified as important complementary evaluation targets.Source: §5 · p. 10

6 CONCLUSION AND FUTURE DIRECTIONS

WILDBENCH evaluates LLMs on real-world user queries using interpretable judging and length-bias mitigation. Its primary metrics correlate strongly with human judgments, while broad experiments examine 40 proprietary and public models.

  • WILDBENCH evaluates LLMs with real-world user queries and a CoT-like LLM-as-judge method designed to improve interpretability and reduce ambiguity.The framework also incorporates a length penalty to mitigate length bias in LLM-as-judge evaluations.Source: §6 · p. 10
  • WB-Reward and WB-Score show very strong correlations with human judgments, surpassing existing evaluations.Source: §6 · p. 10
  • 40 proprietary and public LLMs are evaluated through extensive experiments and analyses on WILDBENCH.The benchmark provides detailed score breakdowns across task categories to reveal model strengths and weaknesses.Source: §6 · p. 10

A TASK CATEGORIES

The appendix organizes WILDBENCH tasks into categories and consolidated groups for fine-grained capability analysis. It also points readers to the task-turn distribution, review materials, and supplementary resources.

Task categories

  • 12 task categories support fine-grained analysis of LLM capabilities.The categories include information seeking, reasoning, planning, editing, and coding and debugging, among others.Source: A TASK CATEGORIES · p. 14

Consolidated groups

  • Five major groups consolidate related categories: Info Seeking, Math & Data, Reasoning & Planning, and Creative Tasks, with remaining types grouped under Creative Tasks.The consolidation combines information and advice seeking, math and data analysis, and reasoning and planning categories.Source: A TASK CATEGORIES · p. 14

Dataset distribution

  • Figure 8 presents the distribution of the number of turns in WILDBENCH.Source: Figure 8 · p. 14

Resources

  • The supplementary zip file contains evaluation scripts, the leaderboard, and the data, while the links are provided for double-blind review.Source: A TASK CATEGORIES · p. 14

B MORE INFORMATION ON WILDBENCH DATA

WILDBENCH’s data documentation, metadata, and public subset are available online, and the dataset is released under AI2’s ImpACT license. The authors also commit to long-term availability and continuous updates.

D PROMPT TEMPLATE FOR PAIRWISE EVALUATION METRIC WB-REWARD

The WB-Reward pairwise prompt supplies the conversation and two model responses, then uses a checklist and structured analysis to guide the judge. The judge summarizes comparative reasons and selects one of five outcomes.

Prompt structure

  • The pairwise prompt contains a high-level instruction, the task, two model outputs, a checklist, and rules for step-by-step evaluation.Its three sections cover the evaluation setup, checklist and rules, and the required judging process.Source: D PROMPT TEMPLATE FOR PAIRWISE… · pp. 14–15

Evaluation inputs

Checklist-guided analysis

Assessment choices

  • Five choices encode pairwise quality differences: A++ much better, A+ slightly better, A=B equal, B+ slightly better, and B++ much better.The labels express the direction and magnitude of the comparison between Response A and Response B.Source: D PROMPT TEMPLATE FOR PAIRWISE… · p. 15

Output format

E PROMPT TEMPLATE FOR INDIVIDUAL EVALUATION METRIC WB-SCORE

The individual WB-Score evaluation prompt asks an expert LLM judge to assess an AI response using the user query, conversation history, and a checklist. It requires structured reasoning, strengths and weaknesses, and a 1–10 rating in JSON format.

F FULL WILDBENCH LEADERBOARD

WildBench provides leaderboards dated June 5, 2024 and September 1, 2024. The updated leaderboard uses WB-Elo, which merges WB-Reward and WB-Score into pairwise comparisons and updates Elo ratings on top of existing LMSYS Elo.

Related papers

Loading 2406.04770v2…