Source-linked AI summary
JudgeBench: A Benchmark for Evaluating LLM-based Judges
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica
TL;DR
Existing evaluations often rely on crowdsourced human preferences, which can be unreliable for complex tasks requiring factual and logical correctness. The paper introduces a hierarchical framework and JudgeBench, finding that strong judges such as GPT-4o still struggle with challenging response pairs.
Problem
Crowdsourced human preferences can be unreliable for complex evaluations requiring domain-specific knowledge and reasoning, making judge reliability difficult to assess objectively.
Method
The paper proposes a hierarchical evaluation framework and a pipeline that converts labeled datasets into 350 challenging response pairs across knowledge, reasoning, mathematics, and coding.
Results
JudgeBench challenges current judges substantially: GPT-4o reaches only 50% accuracy with a vanilla prompt and 56% with Arena-Hard, while o3-mini reaches 80.86% at high reasoning.
Takeaways & Limitations
JudgeBench provides a platform for assessing whether LLM-based judges can distinguish factually and logically correct outputs as AI systems advance.
Takeaways & Limitations
Response-pair generation using multiple LLMs can produce incorrect answers that are too easy to identify, expose stylistic confounds, and trigger self-enhancement bias.
Abstract
from arXiv · showhide
LLM-based judges have emerged as a scalable alternative to human evaluation and are increasingly used to assess, compare, and improve models. However, the reliability of LLM-based judges themselves is rarely scrutinized. As LLMs become more advanced, their responses grow more sophisticated, requiring stronger judges to evaluate them. Existing benchmarks primarily focus on a judge's alignment with human preferences, but often fail to account for more challenging tasks where crowdsourced human preference is a poor indicator of factual and logical correctness. To address this, we propose a novel evaluation framework to objectively evaluate LLM-based judges. Based on this framework, we propose JudgeBench, a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding. JudgeBench leverages a novel pipeline for converting existing difficult datasets into challenging response pairs with preference labels reflecting objective correctness. Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks, with many strong models (e.g., GPT-4o) performing just slightly better than random guessing. Overall, JudgeBench offers a reliable platform for assessing increasingly advanced LLM-based judges. Data and code are available at https://github.com/ScalerLab/JudgeBench.
1 INTRODUCTION
JudgeBench addresses the limited reliability of human-preference evaluations for complex factual and logical judgments by introducing a hierarchy that prioritizes instruction following, correctness, and then style. It operationalizes this framework through a benchmark and pipeline for constructing objectively labeled, challenging response pairs.
- Motivation: Existing evaluations often rely on crowdsourced human preferences, which become unreliable as task difficulty and required reasoning increase.Annotators may favor plausible or longer responses over factually correct ones.
- Benchmark: JudgeBench focuses on evaluating factual and logical correctness in complex responses that are difficult for crowdsourced annotators to assess reliably.This distinguishes it from prior work focused on instruction following or stylistic preferences.
- Evaluation framework: The proposed hierarchy prioritizes instruction following, factual and logical correctness, and stylistic preference in that order.Style should influence judgments only after instruction following and correctness are satisfied.
- Benchmark construction: The pipeline transforms datasets with ground-truth labels into judge-oriented response pairs containing one objectively correct response and one subtly incorrect response.The benchmark spans general knowledge, reasoning, mathematics, and coding, with 350 response pairs.
- Contributions: The paper provides a principled evaluation framework, a dataset-construction pipeline, and a benchmark intended to test whether judges distinguish factually correct responses.The authors report that JudgeBench is substantially more challenging than prior benchmarks.
2 RELATED WORK
Prior work evaluates judges through prompting, fine-tuning, multi-agent pipelines, and reward-model verification, but benchmarks largely emphasize agreement with human preferences. JudgeBench extends this landscape toward objective correctness evaluation.
- LLM-based judges: Prompted judges use carefully designed prompts without additional training, while fine-tuned judges learn from preference data or distilled judgments.Fine-tuned judges can perform well on benchmarks but may struggle to generalize.
- LLM-based judges: Multi-agent judges combine several LLM outputs to produce evaluations, potentially improving robustness at significantly higher inference cost.
- Reward models and verifiers: Reward models are distinct from LLM-based judges and are primarily trained on preference data for reinforcement learning from human feedback.They can also act as verifiers that classify solution correctness or select best-of-N responses.
- JudgeBench: JudgeBench targets factual and logical correctness in challenging response pairs, complementing prior work centered on human-preference alignment.
- Existing benchmarks: Existing judge benchmarks such as LLMEval, MTBench, and FairEval mainly assess agreement between judge outputs and human evaluations.RewardBench evaluates reward models across domains including safety, chat, and reasoning.
3 JUDGEBENCH
JudgeBench constructs difficult, objectively labeled response pairs by sampling and filtering outputs from a strong model on challenging datasets. The approach improves difficulty and stylistic consistency but introduces potential bias toward the response-generating model.
- Pipeline: The pipeline samples k responses from a strong model, grades correctness, and retains questions with at least one correct and one incorrect response.These responses form pairs with objective ground-truth labels.
- Pipeline rationale: Using one model to generate all candidates keeps response styles consistent, reducing superficial stylistic cues and self-enhancement bias.
- Limitations: The single-generator design may make the dataset disproportionately difficult for the base model, because different models may struggle on different questions.The paper examines this bias through an ablation study.
- Dataset design: JudgeBench can transform existing datasets with ground-truth labels and verification mechanisms into response-pair datasets, provided the source problems are sufficiently difficult.
- Data filtering: Additional LLM verification filters out responses where automated checkers and semantic correctness judgments disagree because of formatting errors.The pipeline also applies randomized filtering to balance selected subsets.
- Dataset composition: The resulting dataset contains 350 questions: 154 Knowledge, 98 Reasoning, 56 Mathematics, and 42 Coding.
4 EVALUATION
JudgeBench evaluates prompted, fine-tuned, multi-agent, and reward-model judges on objectively correct versus incorrect responses, revealing substantial difficulty across challenging tasks. Results indicate that reasoning capability, fine-tuning, and training data affect performance, while judge verification ability closely tracks problem-solving ability.
- Evaluation setup: JudgeBench evaluates prompted, fine-tuned, multi-agent, and reward-model judges on challenging response pairs.The experiments include multiple judge categories and compare reward models separately.
- LLM-based judges: GPT-4o reaches only 50% accuracy with the vanilla prompt and 56% with Arena-Hard, remaining near random performance.The advanced prompt improves accuracy slightly, but overall performance remains low.
- LLM-based judges: 57.43% overall accuracy makes Skywork the strongest fine-tuned judge, while other fine-tuned judges perform significantly below random.Against base Llama-3.1 models, fine-tuning improves accuracy by over 12% for 8B and 5% for 70B models.
- Underlying models: OpenAI o3-mini performs best overall at 80.86%, 76.57%, and 70.57% accuracy for high, medium, and low reasoning levels.Larger models generally outperform smaller counterparts, and Claude-3.5-Sonnet reaches 64.29% among general-purpose models.
- Reward models: Reward-model accuracies range from approximately 59% to 64%, with training-dataset quality appearing more influential than model size alone.Skywork’s Llama-3.1-8B reward model reaches 62.29% versus 40.86% for its base model.
- Ablation studies: The judge’s ability to verify solution pairs closely mirrors its ability to solve the underlying problem, with coding harder to evaluate than mathematics.Solvers outperform judges in coding, whereas judges significantly outperform solvers in math.
CONCLUSION
The paper introduces a hierarchical evaluation framework and JudgeBench to assess whether LLM-based judges distinguish factually and logically correct outputs.
- JudgeBench evaluates LLM-based judges’ ability to distinguish factually and logically correct outputs.
A.1 DETAILS OF THE JUDGES
The evaluation follows official judge implementations, using reproducible greedy decoding and provider-specific serving arrangements.
- Judges follow their official implementations, with modifications made only when necessary.
- Greedy decoding at temperature=0 is used across judges to ensure reproducibility.
- Proprietary models are accessed through official APIs, while most open-weight models are served locally in half-precision.
A.1.1 PROMPTED JUDGES
The prompted judges differ in whether they require explanations, generate reference answers, support ties, and expose generation controls.
- The Vanilla judge outputs only a label identifying the better response and does not allow ties.
- Arena-Hard Judge generates a reference answer before comparing candidates and evaluates five ordered preference options.
- Google Vertex Judge is evaluated pairwise with Vertex AI’s predefined question-answering quality metric.
- Google Vertex Judge offers little to no control over generation parameters such as temperature.
A.1.2 FINE-TUNED JUDGES
The fine-tuned judges are trained from different base models and preference or critique datasets, with varied scoring and inference procedures.
- PandaLM is based on LLaMA-7B or LLaMA-70B and fine-tuned on crowdsourced human preference data, but only its 7B variant is evaluated.
- Prometheus 2 is fine-tuned from Mistral 7B and Mixtral 8x7B using feedback and preference collections whose resulting weights are merged.
- Prometheus 2 uses the official “factual validity” criterion and does not support ties.
- JudgeLM is fine-tuned from Vicuna using instruction-tuning datasets augmented with candidate responses and GPT-4 judgments, assigning integer scores to candidates.
- Auto-J is fine-tuned from Llama-2-13b-chat on preference datasets augmented with GPT-4 judgments and supports ties.
- Skywork Critics provides 8B and 70B generative judges based on Llama-3.1 models and fine-tuned on proprietary and open-source critic datasets.
A.1.3 MULTI-AGENT JUDGES
ChatEval is a multi-agent judge that assigns roles to two GPT-4o-powered agents, has them discuss sequentially, and aggregates their independent scores.
- ChatEval uses two GPT-4o-powered agents acting as the general public and a critic.The agents discuss in round-robin fashion for at most four turns.
- After discussion, each agent independently scores every candidate response from 1 to 10.
- The final decision averages the scores assigned by both agents.
A.2 ADDITIONAL ANALYSIS OF FINE-TUNED JUDGES
The analysis finds substantial weaknesses among fine-tuned judges, including below-chance performance, invalid or tied outputs, and inconsistent decisions across trials, while some fine-tuned judges outperform prompted counterparts.
- Performance and failure modes: Many fine-tuned judges score below the 50% random-guessing baseline.The analysis attributes this pattern to truncated responses, ties and invalid decisions, and inconsistent judgments.
- Ties and invalid decisions: PandaLM selected the tie option in 479 of 700 judgments.
- Ties and invalid decisions: Prometheus2-bgb-8x7b produced an unextractable invalid judgment in 215 of 700 judgments.Examples included “10/10”, “Neither A nor B”, and “3”.
- Inconsistent judgments: JudgeLM-7B, JudgeLM-13B, and Prometheus2-7b were inconsistent across games on 59.71%, 54.57%, and 52.29% of pairs, respectively.
- Fine-tuned versus prompted judges: Skywork-Llama-3.1B-8B and Skywork-LLaMA-3.1B-70B outperformed corresponding arena-hard judges by 12.57 and 5.14, respectively.The comparison uses the same base model for each fine-tuned and prompted judge pair.
A.3 DATASET FILTERING
JudgeBench constructs filtered response pairs from difficult datasets by checking answer correctness, retaining questions with both correct and incorrect generations, and using standardized prompts and validation procedures.
- Dataset construction: MMLU-Pro questions were sampled across 14 disciplines, yielding 154 knowledge pairs after filtering and selecting 11 pairs per discipline.Only questions containing both correct and incorrect responses were retained for the GPT-4o and Claude-3.5-Sonnet generations.
- Dataset size: The GPT-4o JudgeBench split contains 350 instances, comparable in size to several prior benchmarks.FairEval contains 80 unique questions, LLMEval-2 480, MT-Bench 80, and LLMBar 419.
- Controlling length bias: Correct and incorrect responses averaged 562.29 and 561.16 tokens, respectively, reducing concern about length bias.Each pair samples two responses from the same model, producing similar response lengths.
- Correctness checking: Correctness was checked by parsing final answers against ground truth, using regex and dataset-specific post-processing procedures.For MMLU-Pro, questions were adapted to the LiveBench format and requested repeated final-letter outputs.
- Correctness-checking prompt: The evaluation prompt asks GPT-4o-mini to return a JSON boolean indicating whether an LLM response selects the correct option.The prompt supplies the question, LLM response, and ground-truth answer.
- Correctness checking: The correctness-checking methods agreed on 97.7% of GPT-4o responses and 99.5% of Claude-3.5-Sonnet responses across 7,000 responses each.