PaperlayerView exact revision on arXiv

Source-linked AI summary

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica

arXiv:2306.05685v4cs.CLcs.AI

TL;DR

Existing benchmarks inadequately capture human preferences for open-ended, multi-turn chatbot interactions. This paper systematically studies LLM-as-a-judge using MT-bench and Chatbot Arena, finding that strong LLMs achieve over 80% agreement with human evaluations, matching human-human agreement.

  • Problem

    Existing benchmarks focus on confined, closed-ended tasks and inadequately assess open-ended instruction following and alignment with human preferences.

  • Method

    The paper systematically studies LLM-as-a-judge with MT-bench and Chatbot Arena, using multi-turn questions and crowdsourced chatbot comparisons with human ratings.

  • Results

    Over 80% agreement with human evaluations matches the level of agreement among human experts.

  • Takeaways & Limitations

    LLM-as-a-judge is a scalable alternative to traditional human evaluation for quickly assessing human preferences.

  • Takeaways & Limitations

    LLM judges have limited reliability when grading math and reasoning questions, including cases where they misjudge problems they can solve.

Abstract

from arXiv · show

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

1 Introduction

Existing benchmarks often fail to capture human preferences in open-ended, multi-turn interactions, motivating preference-based evaluation for chat assistants. The paper introduces MT-bench and Chatbot Arena and studies LLM-as-a-judge as a scalable alternative to costly human evaluation.

  • Introduction: Conventional benchmarks can favor base models despite human preference for aligned chat models, because they underassess open-ended instruction following and multi-turn dialogue.LLaMA-13B can perform competitively on conventional benchmarks, while its open-ended answers are often less preferred than Vicuna-13B’s.
  • Introduction: MT-bench evaluates multi-turn conversational and instruction-following ability with open-ended questions designed to distinguish reasoning and math capabilities.Human ratings are the primary evaluation metric for MT-bench and Chatbot Arena.
  • Introduction: Chatbot Arena provides crowdsourced human preference data through a platform for comparing chat assistants in the wild.Together with MT-bench, it forms one of the paper’s two preference-based benchmarks.
  • Introduction: GPT-4 judges matched human evaluations at an agreement rate exceeding 80%, equaling human-human agreement after examining and mitigating several judge biases.The studied limitations include position, verbosity, self-enhancement, and limited reasoning ability; the evaluation used 3K controlled expert votes and 3K crowdsourced human votes.
  • Introduction: The paper advocates a hybrid framework combining capability-based benchmarks with preference-based benchmarks and LLM-as-a-judge.This framework is intended to evaluate core capabilities and human alignment swiftly and automatically.

2 MT-Bench and Chatbot Arena

Existing benchmarks mainly assess short, closed-ended answers, leaving human preferences in open-ended, multi-turn interactions undermeasured. The section introduces MT-bench and Chatbot Arena to evaluate these preferences and distinguish state-of-the-art models.

  • Motivation: Existing language-model benchmarks primarily use closed-ended questions with short, automatically validated responses, making broader chatbot evaluation challenging.These assistants span diverse tasks, including writing, chatting, and coding.
  • Motivation: Human preferences directly measure chatbot utility in open-ended, multi-turn human-AI interactions, motivating two benchmarks designed to assess preferences and distinguish models.
  • MT-bench: 80 high-quality multi-turn questions comprise MT-bench, which tests conversation and instruction-following across challenging common-use-case prompts.The benchmark covers writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, and humanities/social-science knowledge.
  • MT-bench: 8 prompt categories guide MT-bench construction, with 10 manually designed multi-turn questions per category.
  • Chatbot Arena: Around 30K votes were collected in one month on Chatbot Arena, where users anonymously compare two models answering the same question and vote before identities are disclosed.Because questions are not predefined, the platform gathers a wide range of user interactions.

3 LLM as a Judge

LLM-as-a-judge offers a scalable, explainable alternative to costly human evaluation for open-ended chat responses, but exhibits position, verbosity, self-enhancement, and reasoning-related limitations. The section introduces judging variants and mitigation methods, including few-shot, chain-of-thought, and reference-guided judging.

  • Benefits: LLM-as-a-judge reduces human involvement for scalable, fast benchmarks while providing explanations alongside scores.These properties make judge outputs interpretable as well as easier to use for rapid evaluation iterations.
  • Judging methods: LLM judges evaluate chat responses through pairwise comparison, single-answer grading, or reference-guided grading, which can be used independently or combined.Pairwise comparison determines which of two answers is better or whether they tie; single-answer grading assigns an absolute score; reference-guided grading supplies a solution when applicable.
  • Biases and limitations: Position bias can make judgments flip when the same answers change order, with most tested judges favoring the first position.In one example, GPT-4 initially preferred GPT-3.5, then favored Vicuna after the responses were swapped; the evaluation used highly similar answers, which can also challenge humans.
  • Biases and limitations: Verbosity bias favors longer responses even when shorter answers are clearer, higher-quality, or more accurate.The repetitive-list attack made 23 MT-bench answers unnecessarily verbose by duplicating rephrased list content without adding information.
  • Biases and limitations: GPT-4 and Claude-v1 showed self-enhancement effects of 10% and 25% higher self-win rates, respectively, while GPT-3.5 did not favor itself.The study notes that limited data and small differences prevent a definitive determination of self-enhancement across models.
  • Mitigations and prompt design: Few-shot examples increased GPT-4’s position-bias consistency from 65.0% to 77.5%, while chain-of-thought and reference-guided judging target weak math and reasoning evaluation.Chain-of-thought prompting asks the judge to answer independently before grading; multi-turn MT-bench prompts must present two questions and four responses, complicating design.

4 Agreement Evaluation

Agreement evaluation on MT-bench and Chatbot Arena shows that GPT-4 judgments closely match human preferences, reaching 85% agreement with experts on MT-bench and similar non-tie agreement ratios in Arena. Agreement improves when model-pair performance differences are larger, while LLM-judge win-rate curves closely track human curves.

  • MT-bench: MT-bench collected around 3K votes from 58 expert-level human labelers evaluating answers from six models alongside LLM judges.Each human evaluated at least 20 random multi-turn questions, while LLM judges evaluated all model pairs.
  • MT-bench: 85% agreement under setup S2 between GPT-4 and human experts exceeded the 81% agreement among humans.Setup S2 excludes tied votes; GPT-4 used both pairwise comparison and single-answer grading in the evaluation.
  • Chatbot Arena: In Chatbot Arena, GPT-4 and other LLM judges achieved similar non-tie agreement ratios with humans, while GPT-4 produced many more non-tied votes.The authors interpret this as GPT-4 being more affirmative and less affected by position bias when making judgments.
  • Judge agreement: GPT-4 single-answer grading matched pairwise GPT-4 and human preferences well, offering a more scalable approach despite sometimes producing more ties.The results suggest GPT-4 maintains a relatively stable internal rubric across grading formats.
  • Breakdown analysis: GPT-4-human agreement increased from 70% to nearly 100% as model-pair performance disparities grew.The breakdown included only non-tied votes and used win-rate differences as the measure of performance disparity.
  • Win-rate analysis: LLM-judge win-rate curves closely matched human curves, while MT-bench’s second turn showed stronger human preference for Claude and GPT-3.5 than its first turn.The multi-turn benchmark better differentiated some advanced model abilities; category analysis also found GPT-4 significantly better overall and Vicuna-13B weaker in reasoning, math, and coding.

5 Human Preference Benchmark and Standardized Benchmark

MT-bench and Chatbot Arena complement standardized LLM benchmarks by evaluating different aspects of model performance, so comprehensive evaluation should use both. For LLaMA variants, MT-bench uses scalable GPT-4 single-answer grading alongside MMLU and Truthful QA (MC1).

  • Benchmark complementarity: MT-bench and Chatbot Arena complement standardized LLM benchmarks because they focus on different aspects of model performance.The recommended approach is to evaluate models comprehensively with both benchmark types.
  • Benchmark evaluation: Several LLaMA variants are evaluated on MMLU, Truthful QA (MC1), and MT-bench using GPT-4 judgment.Training details are provided in Appendix E.
  • MT-bench grading: GPT-4 single-answer grading is used for MT-bench because of its scalability and simplicity, assigning each turn a score from 0 to 10.The reported average score is 160 = 80×2 turns.

6 Discussion

The discussion identifies limitations in the study’s helpfulness-focused evaluation, outlines data-release details and societal impacts, and proposes directions for broader, more aligned, and more capable evaluation systems.

  • Limitations: The study largely neglects safety and combines accuracy, relevance, and creativity into one helpfulness metric.The authors suggest evaluating safety-related metrics through modified prompts and separating helpfulness dimensions in future work.
  • Data collection and release: Appendix C documents user instructions, the data-collection interface, participant information, and released-data contents.
  • Societal impacts: The methods may improve chatbot quality and user experiences, but their biases require attention; advanced assistants may cause job displacement while creating new opportunities.The released dataset also supports research on human preferences and model behavior.
  • Future directions: Future directions include large-scale benchmarking across broader categories, open-source judges aligned with human preferences, and stronger math and reasoning in open models.

7 Conclusion … C Data Collection

The paper establishes LLM-as-a-judge as a scalable evaluation framework aligned with human preferences, documents judge prompt designs and failure modes, and describes data collection for MT-bench and Chatbot Arena.

  • 7 Conclusion: Over 80% agreement: strong LLM judges match human experts and support an LLM-based chatbot evaluation framework.The agreement rate is reported as comparable to agreement among human experts.
  • A Prompt templates: Default pairwise and single-answer prompts instruct judges to assess response quality, instruction following, helpfulness, relevance, accuracy, depth, creativity, and detail.The paper lists prompt templates and refers readers to its GitHub repository for full details.
  • A Prompt templates: Chain-of-thought and reference-guided prompts ask judges to solve questions independently, compare answers, and identify mistakes.Additional templates cover math and reasoning questions and reference-guided pairwise comparison.
  • A Prompt templates: Multi-turn prompts extend pairwise comparison and reference-guided grading to conversations.The listed templates include multi-turn pairwise comparison and reference-guided multi-turn single-answer grading.
  • B Case Study: Position bias can reverse GPT-4’s verdict when assistants A and B swap positions, with similar patterns observed for Claude and GPT-3.5.This case study examines whether the judge’s preference depends on answer order.
  • B Case Study: GPT-3.5 and Claude-v1 favor longer repetitive answers, whereas GPT-4 detects the repetitive-list attack.The two answers are otherwise identical except for two rephrased items.
  • B Case Study: GPT-4 shows limited grading ability on math and reasoning, making arithmetic mistakes, accepting two wrong answers, and copying an erroneous answer despite chain-of-thought prompting.The examples indicate that judgments can be influenced by supplied answers even when GPT-4 can solve the problems separately.
  • B Case Study: A prompt that splits questions into two prompts can make GPT-4 mislocate assistants’ previous responses and produce an incorrect judgment.The example concerns following up on the second example from a prior response.

C.1 MT-bench human evaluation · C.2 Chatbot Arena · D Additional Experimental Results

The paper collects controlled MT-bench human judgments through a two-turn comparison interface and evaluates Chatbot Arena as a consent-based, free crowdsourcing platform. It also reports additional experimental results without further detail in the supplied passages.

  • C.1 MT-bench human evaluation: MT-bench presents two anonymous assistants’ answers, asks humans to compare each turn, and permits skipping up to five uncertain questions.Reference solutions may be shown for math and reasoning questions.
  • C.1 MT-bench human evaluation: When a human vote differs from GPT-4’s, the interface displays GPT-4’s judgment and asks whether it is reasonable.Users choose among three response buttons.
  • C.1 MT-bench human evaluation: Participants are consented through an application form, paid $20 for judging 20 questions, and are mostly graduate students from more than ten universities.The stated compensation corresponds to an hourly rate of around $35.
  • C.2 Chatbot Arena: Chatbot Arena requires users to accept terms of use that obtain consent and permit release of conversation data.The instructions appear at the top of the interface.
  • C.2 Chatbot Arena: Chatbot Arena is a free website where users receive no payment and may participate without registration.The platform is open to any user.
  • D Additional Experimental Results: The paper presents additional experimental results in Section D.No specific results are described in the supplied passage.

D.1 Position bias · D.2 Few-shot judge

The paper finds that position bias varies with prompts, question categories, and model-pair difficulty, while few-shot examples improve judge consistency without improving GPT-4’s agreement with humans on MT-bench.

  • D.1 Position bias: Different prompts affect models differently; the “score” prompt can increase GPT-3.5 consistency.The “score” prompt requests two absolute scores, while the “short” prompt removes several default instructions, including explicit position-bias avoidance.
  • D.1 Position bias: Position bias is evaluated through consistency, first-answer preference, and wrong output formats across models and prompts.Consistency measures agreement after swapping answer order; “Biased toward first” measures preference for the first answer, and “Error” captures incorrect output formats.
  • D.1 Position bias: Position bias is more noticeable on open writing and STEM/humanity knowledge questions than on math and coding questions.On math and coding, judges are more confident even though their judgments can often be wrong.
  • D.1 Position bias: With GPT-4 and the default prompt, position bias is more noticeable for model pairs with close performance.The comparison uses three different model pairs.
  • D.2 Few-shot judge: Few-shot examples significantly improve consistency for all three examined LLM judges.The improvement is reported in Table 12’s consistency results for position bias.
  • D.2 Few-shot judge: Few-shot examples almost eliminate GPT-4’s position bias but shift GPT-3.5’s bias from the first position to the second.The direction of GPT-3.5’s position preference changes rather than disappearing.
  • D.2 Few-shot judge: Few-shot GPT-4 pairwise comparison performs similarly to zero-shot GPT-4 in agreement with humans on MT-bench.The paper measures agreement between few-shot GPT-4 pairwise judgments and human preferences.

D.3 Agreement Evaluation · D.4 Category-wise scores with single-answer grading

D.3 defines judge agreement probabilistically and introduces human-majority to compare GPT-4 with aggregated human votes. D.4 evaluates six MT-bench models using single-answer grading and reports category-wise scores.

  • D.3 Agreement Evaluation: Agreement is the probability that randomly selected, non-identical judges of two types agree on a randomly selected question.The definition applies to comparisons such as GPT-4 versus Claude or GPT-4 versus humans.
  • D.3 Agreement Evaluation: Human agreement can underestimate GPT-4–human agreement when individual human votes are inconsistent.The passage illustrates this with three humans voting “A”, “A”, and “B”.
  • D.3 Agreement Evaluation: Human-majority aggregates the majority human vote for each question, enabling GPT-4 agreement to be measured against the consensus label.The agreement between human-majority and human serves as the upper bound for GPT-4–human agreement.
  • D.3 Agreement Evaluation: Table 13 reports additional MT-bench agreement results including expert labelers, authors, GPT-4 pairwise comparison, GPT-4 single-answer grading, Claude, and human-majority.The table distinguishes pairwise and single-answer GPT-4 judges and describes converting single-answer grading into pairwise results.
  • D.4 Category-wise scores with single-answer grading: Six models are evaluated on MT-bench with single-answer grading, and their category-wise scores are plotted in Figure 20.Figure 20 is titled “Category-wise scores of 6 models on MT-bench.”
  • D.3 Agreement Evaluation: Table 13 evaluates agreement under two setups, with S1 including non-tie, tie, and position-bias-related inconsistent votes.The supplied passage identifies S1 but truncates the description of the other setup.

E Training Details of Vicuna Models · F Exploring Vicuna as a judge · F.1 Zero-Shot Vicuna

Vicuna is trained by fine-tuning LLaMA on cleaned ShareGPT conversations, while the paper explores Vicuna-13B as a cheaper LLM judge and finds substantial zero-shot limitations. The training uses multiple dataset variants and standardized model settings, whereas zero-shot Vicuna-13B shows low consistency and high error rates.

  • E Training Details of Vicuna Models: 125K conversations remain after converting ShareGPT HTML to markdown and filtering inappropriate or low-quality samples.Lengthy conversations are divided into segments fitting the model’s maximum context length.
  • E Training Details of Vicuna Models: Vicuna is created by fine-tuning a LLaMA base model on user-shared ChatGPT conversations collected through ShareGPT’s public APIs.ShareGPT is a website where users share their ChatGPT conversations.
  • E Training Details of Vicuna Models: 3K sequences form the small “Selected” dataset, chosen from GPT-4-generated conversations with at least 3 turns through clustering and centroid selection.The cleaned data also yields “All” and “Single” datasets, with “Single” containing only each conversation’s first turn.
  • F Exploring Vicuna as a judge: GPT-4 is costly at scale, motivating exploration of Vicuna-13B as a more cost-friendly proxy because it is open-sourced and has strong language understanding.The paper primarily evaluates close-sourced models such as GPT-4 as proxies for human evaluations.
  • F.1 Zero-Shot Vicuna: 11.2% to 16.2% consistency and 22.5% to 78.8% error rates show that zero-shot Vicuna-13B performs poorly across prompt templates.Its weaker instruction-following often produces generic comparisons such as “Answer A is better than answer B” without following the intended evaluation procedure.

F.2 Arena Fine-tuned Vicuna

Fine-tuning Vicuna-13B on 22K Chatbot Arena human votes substantially improves position consistency and yields strong agreement with human labels, approaching GPT-4 performance.

  • Training: 22K single-turn Chatbot Arena votes spanning 15 supported chatbot models were used to fine-tune Vicuna-13B.The training data exposed the model to a wider range of chatbot outputs.
  • Position bias results: 65.0% position consistency after fine-tuning, up from 16.2% for zero-shot Vicuna-13B.The classification formulation produced a 0% error rate because every output was recognizable.
  • Agreement results: 56.8% agreement across all three labels and 85.5% excluding tie predictions and labels, versus GPT-4’s 66% and 87%.These results substantially exceed random guesses of 33% and 50%, respectively.
  • Agreement results: Fine-tuned Vicuna-13B shows strong potential as a cheaper open-source replacement for expensive closed-source LLMs.The paper notes that a concurrent paper reached a similar conclusion.
Loading 2306.05685v4…