PaperlayerView exact revision on arXiv

Source-linked AI summary

WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, Yejin Choi

arXiv:2406.04770v2cs.CLcs.AI

TL;DR

Evaluating LLMs on open-ended, real-world queries remains difficult because existing benchmarks do not adequately reflect the diversity and challenge of user tasks. WildBench addresses this gap with a curated real-user benchmark, checklist-based LLM judging, and pairwise and individual metrics. Its metrics strongly correlate with human rankings, with WB-Reward reaching 0.98 Pearson correlation and WB-Score 0.95 on top-ranking models.

  • Problem

    Existing benchmarks insufficiently capture the varied and unexpected nature of real-world user inquiries and provide limited evidence for holistic evaluation.

  • Method

    WildBench curates 1,024 challenging real-user tasks and evaluates responses with task-specific checklists, WB-Reward pairwise comparisons, and WB-Score individual scoring.

  • Results

    0.98 Pearson correlation for WB-Reward and 0.95 for WB-Score with human-voted Chatbot Arena Elo ratings on top-ranking models.

  • Takeaways & Limitations

    WildBench provides a realistic, dynamic, and interpretable framework for evaluating model capabilities across challenging real-user tasks.

  • Takeaways & Limitations

    WildBench cannot exactly match Chatbot Arena’s user and task distribution because the underlying users and data are not fully comparable or public.

Abstract

from arXiv · show

We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) using challenging, real-world user queries. WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. For automated evaluation with WildBench, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses task-specific checklists to evaluate model outputs systematically and provides structured explanations that justify the scores and comparisons, resulting in more reliable and interpretable automatic judgments. WB-Reward employs fine-grained pairwise comparisons between model responses, generating five potential outcomes: much better, slightly better, slightly worse, much worse, or a tie. Unlike previous evaluations that employed a single baseline model, we selected three baseline models at varying performance levels to ensure a comprehensive pairwise evaluation. Additionally, we propose a simple method to mitigate length bias, by converting outcomes of ``slightly better/worse'' to ``tie'' if the winner response exceeds the loser one by more than $K$ characters. WB-Score evaluates the quality of model outputs individually, making it a fast and cost-efficient evaluation metric. WildBench results demonstrate a strong correlation with the human-voted Elo ratings from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates, as well as the 0.87 for regular win rates.

1 INTRODUCTION

WildBench addresses the difficulty of evaluating LLMs on diverse, open-ended real-world queries by providing a challenging benchmark and structured automatic judging. Its metrics and evaluation designs achieve strong agreement with human rankings.

  • Motivation: Existing benchmarks inadequately represent real-world user tasks because they vary in breadth, diversity, and skill coverage.AlpacaEval emphasizes information-seeking tasks, while ArenaHard concentrates heavily on coding and debugging.
  • Benchmark: WildBench V2 contains 1,024 curated tasks from real user-chatbot dialogs, preserving natural task distributions while excluding tasks judged easy by all models.Finalized tasks also undergo manual review.
  • Benchmark: WildBench tasks are substantially harder than AlpacaEval tasks because they involve complex constraints, depth, realism, and higher-order reasoning.Examples include constrained coding, multi-constraint creative writing, and software-system design.
  • Evaluation: WB-Reward compares model responses pairwise using five outcomes, while three baselines provide a more comprehensive assessment across performance levels.The three-baseline design replaces the single-baseline approach used by most prior works.
  • Evaluation: WB-Score evaluates generations individually for faster, more cost-effective assessment, while a length-penalty method converts slight wins or losses to ties when the winner is significantly longer.Both metrics use automated evaluation designs intended to address practical judging challenges.
  • Results: 0.98 Pearson correlation for WB-Reward and 0.95 for WB-Score with human-voted Chatbot Arena Elo ratings on top-ranking models.These correlations exceed the reported results for ArenaHard and AlpacaEval2.0.

2 WILDBENCH DATA CURATION

WILDBENCH curates challenging, diverse tasks from real user-chatbot conversations and updates the benchmark to reflect evolving user interactions. Its task distribution is more balanced and its queries support longer-context evaluation than several existing benchmarks.

  • Mining Challenging Tasks from WildChat: WILDBENCH retains 1,024 challenging and diverse tasks from WildChat’s one million human-chatbot conversations after filtering, difficulty annotation, and human quality control.Tasks rated easy by all annotating models were excluded before sampling and review.
  • Benchmark Statistics: Only ArenaHard and WILDBENCH among the compared benchmarks use real-user queries, while WILDBENCH aligns its distribution with real users’ task categories rather than overemphasizing coding and debugging.WILDBENCH is designed to be updated regularly, with V1 and V2 released on different WildChat data iterations.
  • Long-Context Tasks: WILDBENCH includes conversations of up to four turns, with over 20% containing more than two turns, and has longer queries than comparison benchmarks.These properties support evaluation of long-context problem-solving abilities.
  • Task Categories: WILDBENCH’s task-category distribution is more balanced than AlpacaEval’s and ArenaHard’s, which concentrate over 50% of tasks in Information seeking and Coding & Debugging, respectively.The benchmark categorizes tasks into 12 categories for fine-grained capability analysis.

3 AUTOMATIC EVALUATION WITH WILDBENCH

WILDBENCH uses task-specific checklists and structured LLM judging to make automatic evaluation more consistent and interpretable. It combines pairwise and individual metrics, with multiple baselines and an adjustable length penalty to address evaluation cost and length bias.

  • Instance-Specific Checklists: Task-specific checklists contain 5–10 interpretable, verifiable questions and are finalized from GPT-4-Turbo and Claude-3-Opus outputs before manual review.LLM judges use the checklists to evaluate responses against criteria tailored to each test query.
  • Pairwise Evaluation: WB-Reward compares two model responses with a GPT-4-Turbo judge using structured checklists and step-by-step analysis before producing a final judgment.The comparison outcomes receive rewards from +1 for much better to −1 for much worse, with intermediate values for slight wins or losses and 0 for ties.
  • Pairwise Evaluation: WB-Reward (Mix) averages rewards against GPT-4-Turbo-0429, Claude-3-Haiku, and Llama-2-70B-chat across 1,024 examples to reduce single-baseline noise and bias.Using multiple baselines provides a more robust performance evaluation on WILDBENCH.
  • Length-Bias Mitigation: The length penalty converts slight wins or losses into ties when the winning response exceeds the losing response by more than K characters.K is customizable, while K = ∞ disables the penalty.
  • Individual Evaluation: WB-Score evaluates each response individually on a 1–10 scale using checklist-based strengths and weaknesses, making evaluation quicker and more cost-effective than pairwise comparison.Scores are averaged after rescaling each score as S′ = (S − 5) × 2.

4 RESULTS & ANALYSIS

WILDBENCH analyzes model performance through leaderboard results, task-category breakdowns, length-bias checks, and correlations with human judgments. Its three-baseline design groups models into performance tiers, while checklist-based scoring and length penalties support robust evaluation.

  • Leaderboard analysis: WILDBENCH’s leaderboard supports task-wise performance comparisons and side-by-side model-output inspection.These features are intended to expose model strengths and weaknesses across tasks.
  • Leaderboard analysis: Three baselines group tested models into three performance tiers relative to GPT-4-Turbo, Claude 3 Haiku, and Llama-2-70B-chat.Tier 1 exceeds Claude 3 Haiku; Tier 2 falls between Claude 3 Haiku and Llama-2-70B-chat; Tier 3 is below Llama-2-70B-chat.
  • Leaderboard analysis: GPT-4-Turbo-0409 and Claude 3 Opus perform well across all five task categories, whereas Llama-3-8B-Inst and Yi-1.5-34B-chat are weaker on coding and mathematics.The six-model analysis uses WB-Score across five categories merged from twelve categories.
  • Leaderboard analysis: Llama-3-8B-Inst-SimPO generally trails Yi-34B-chat and Llama-3-70B-Inst but performs comparably to Llama-3-70B-Inst on information-seeking and creative tasks.This differs from AlpacaEval-2.0, where the 8B model’s length-controlled win rate exceeds the 70B model’s.
  • Leaderboard analysis: Llama-3-8B-Inst-SimPO consistently improves on Llama-3-8B-Inst across all task categories and outperforms some larger models.Its radar-plot profile retains a similar shape to the base 8B model.
  • Ablation studies and discussions: Checklist use raises WB-Score’s Pearson correlation from 0.905 without checklists to 0.925 with checklists.The ablation compares prompts with and without the checklist components.
  • Ablation studies and discussions: K = 500 provides the highest correlation with human judgments among the tested length-penalty settings.The experiment evaluated K values of 100, 200, 500, 1000, and infinity.

5 RELATED WORKS

Related benchmarks evaluate LLMs using close-ended questions, expert- or crowd-curated open-ended tasks, and real-world user data. WILDBENCH belongs to the in-the-wild evaluation line while emphasizing automated assessment and broader task coverage.

  • Close-ended benchmarks: Close-ended benchmarks such as MMLU, CMMLU, KMMLU, MMLU-Pro, and GPQA primarily use multiple-choice questions.These benchmarks cover multiple subject areas and, in some variants, more challenging or language-specific evaluation settings.
  • Expert-curated and crowdsourced data: Expert-curated and crowdsourced benchmarks include MT-Bench’s manually created category examples and AlpacaEval’s author-written instructions.The cited AlpacaEval examples primarily involve simple tasks such as rewriting.
  • In-the-wild data: In-the-wild benchmarks source prompts from real-world use cases, including ChatbotArena, WildVision, and ArenaHard.ChatbotArena relies on extensive human feedback, whereas ArenaHard selects user queries for automatic evaluation.
  • Evaluation methods: Open-ended evaluation commonly uses either individual-output grading or pairwise comparison because no single valid ground truth exists.LLM judges are used to reduce the cost and time of human evaluation.
  • Data leakage prevention: WILDBENCH keeps a subset of WildChat private to reduce contamination risk while releasing a public validation set and construction details.The private subset is reserved as expert-curated evaluation data.
  • Other dimensions for evaluation: LLM evaluation research also addresses safety, fairness, logical reasoning, agentic planning, and hallucination detection beyond capability evaluation.These dimensions are identified as important complementary evaluation targets.

6 CONCLUSION AND FUTURE DIRECTIONS

WILDBENCH evaluates LLMs on real-world user queries using interpretable judging and length-bias mitigation. Its primary metrics correlate strongly with human judgments, while broad experiments examine 40 proprietary and public models.

  • WILDBENCH evaluates LLMs with real-world user queries and a CoT-like LLM-as-judge method designed to improve interpretability and reduce ambiguity.The framework also incorporates a length penalty to mitigate length bias in LLM-as-judge evaluations.
  • WB-Reward and WB-Score show very strong correlations with human judgments, surpassing existing evaluations.
  • 40 proprietary and public LLMs are evaluated through extensive experiments and analyses on WILDBENCH.The benchmark provides detailed score breakdowns across task categories to reveal model strengths and weaknesses.

A TASK CATEGORIES

The appendix organizes WILDBENCH tasks into categories and consolidated groups for fine-grained capability analysis. It also points readers to the task-turn distribution, review materials, and supplementary resources.

  • Task categories: 12 task categories support fine-grained analysis of LLM capabilities.The categories include information seeking, reasoning, planning, editing, and coding and debugging, among others.
  • Consolidated groups: Five major groups consolidate related categories: Info Seeking, Math & Data, Reasoning & Planning, and Creative Tasks, with remaining types grouped under Creative Tasks.The consolidation combines information and advice seeking, math and data analysis, and reasoning and planning categories.
  • Dataset distribution: Figure 8 presents the distribution of the number of turns in WILDBENCH.
  • Resources: The supplementary zip file contains evaluation scripts, the leaderboard, and the data, while the links are provided for double-blind review.

B MORE INFORMATION ON WILDBENCH DATA

WILDBENCH’s data documentation, metadata, and public subset are available online, and the dataset is released under AI2’s ImpACT license. The authors also commit to long-term availability and continuous updates.

  • The dataset documentation, metadata, and public WILDBENCH subset are available through the Hugging Face dataset repository.
  • WILDBENCH is released under AI2’s ImpACT license as a low-risk artifact, with the authors accepting responsibility for rights violations.
  • The authors intend to maintain the dataset, ensure long-term availability, and continuously update it.

D PROMPT TEMPLATE FOR PAIRWISE EVALUATION METRIC WB-REWARD

The WB-Reward pairwise prompt supplies the conversation and two model responses, then uses a checklist and structured analysis to guide the judge. The judge summarizes comparative reasons and selects one of five outcomes.

  • Prompt structure: The pairwise prompt contains a high-level instruction, the task, two model outputs, a checklist, and rules for step-by-step evaluation.Its three sections cover the evaluation setup, checklist and rules, and the required judging process.
  • Evaluation inputs: The judge evaluates two AI responses after reading the user query and conversation history carefully.
  • Checklist-guided analysis: The checklist guides evaluation, while the judge first writes analyses of both responses before providing the assessment.
  • Assessment choices: Five choices encode pairwise quality differences: A++ much better, A+ slightly better, A=B equal, B+ slightly better, and B++ much better.The labels express the direction and magnitude of the comparison between Response A and Response B.
  • Output format: The output summarizes reasons for equality, A’s superiority, and B’s superiority before recording the final choice.
  • Output format: The evaluation result is returned in JSON with analyses for both responses, three comparative-reason fields, and the selected choice.

E PROMPT TEMPLATE FOR INDIVIDUAL EVALUATION METRIC WB-SCORE

The individual WB-Score evaluation prompt asks an expert LLM judge to assess an AI response using the user query, conversation history, and a checklist. It requires structured reasoning, strengths and weaknesses, and a 1–10 rating in JSON format.

  • The evaluation template is divided into instructions and response context, checklist rules, and a step-by-step judging process.
  • The prompt supplies the user query, conversation history, and AI-generated response for evaluation.
  • The judge must analyze the response and checklist before providing an assessment according to the specified criteria.
  • Scores range from 1 to 10, from very poor or nonsensical responses to perfect and helpful responses.Intermediate ratings distinguish poor, fair, and good-enough responses based on meaningful usefulness and issues such as factual errors, hallucinations, or missing information.
  • The output records strengths, weaknesses, and the final score in a prescribed JSON format.

F FULL WILDBENCH LEADERBOARD

WildBench provides leaderboards dated June 5, 2024 and September 1, 2024. The updated leaderboard uses WB-Elo, which merges WB-Reward and WB-Score into pairwise comparisons and updates Elo ratings on top of existing LMSYS Elo.

  • WB-Elo merges WB-Reward and WB-Score into pairwise comparisons and performs Elo updates on existing LMSYS Elo ratings.The paper describes this metric as providing faster and more stable leaderboard updates.
  • The full WildBench leaderboard is reported as of June 5, 2024.
  • An updated WildBench leaderboard is reported as of September 1, 2024.
Loading 2406.04770v2…