PaperlayerView exact revision on arXiv

Source-linked AI summary

No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding

Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, Chris Tanner

arXiv:2503.05061v3cs.CL

TL;DR

The paper asks whether LLM-as-a-Judge can reliably assess correctness in high-stakes domains where correctness matters more than style. It constructs expert-grounded benchmarks and judgments, then finds that judge agreement depends on answerability and reference correctness, with accurate references substantially improving reliability. The results support human verification of references and continued human oversight.

  • Problem

    It remains unclear how accurately LLM-as-a-Judge methods assess response correctness in high-stakes business and finance settings.

  • Method

    The paper creates BFF-Bench and VERDICTS from expert-written references, expert correctness judgments, and 1,200 model responses across BFF-Bench and corrected MT-Bench.

  • Results

    LLM Judge agreement with human experts is strongly related to the judge’s ability to answer the underlying question and the correctness of the provided reference.

  • Takeaways & Limitations

    Reliable LLM-as-a-Judge use requires that the judge can answer the question or that it receives a correct reference.

  • Takeaways & Limitations

    Pairwise grading excludes pairs with two correct or two incorrect responses, so its results should be interpreted only relatively among judge models.

Abstract

from arXiv · show

Reliable evaluation of large language models (LLMs) is critical as their deployment rapidly expands, particularly in high-stakes domains such as business and finance. The LLM-as-a-Judge framework, which uses prompted LLMs to evaluate response quality, is appealing due to its scalability, low cost, and strong correlations with human stylistic preferences. However, it remains unclear how accurately these methods can assess response quality in domains where correctness matters more than style. To address this gap, we introduce the Business and Finance Fundamentals Benchmark (BFF-Bench), a dataset of 160 challenging questions and long-form responses authored by financial professionals. These experts subsequently evaluated the correctness of 1,200 responses generated by a diverse set of LLMs on both BFF-Bench and a challenging subset of MT-Bench. With this expert-annotated dataset of judgments (VERDICTS), we analyze the agreement between a suite of automated grading methods and human experts. While we observe that LLM Judges are more reliable than other grading methods, our findings reveal a clear pattern in LLM Judge performance: when not provided with a correct reference, judges show high agreement with human experts only on questions the judges were able to correctly answer themselves. We demonstrate that providing the judges with expert-written references largely mitigates this issue, highlighting the limits of using LLM-as-a-Judge without any form of human verification.

1 Introduction

LLM-as-a-Judge is scalable for evaluating open-ended responses, but correctness-focused evaluation remains uncertain in high-stakes business and finance settings. The paper introduces expert-grounded benchmarks and finds that judge agreement depends strongly on the judge’s own answerability and reference correctness.

  • Motivation: LLM-as-a-Judge evaluates candidate responses through single or pairwise quality judgments, commonly validated against human preferences.Human preferences can vary across multiple axes, making it difficult to identify why one response is preferred.
  • Motivation: In business and finance, correctness outweighs other attributes, and judging correctness can require complex reasoning and domain knowledge.The paper therefore asks whether LLMs can reliably judge correctness and quality in these high-stakes domains.
  • Contributions: BFF-Bench contains 80 conversations spanning 160 challenging questions with long-form reference answers created and verified by domain experts.The benchmark focuses on correctness in business and finance.
  • Contributions: VERDICTS contains expert judgments of correctness for 1,200 LLM responses from BFF-Bench and MT-Bench, supporting evaluation of single and pairwise grading.The dataset is designed to assess the reliability of automated grading methods against expert judgments.
  • Key finding: LLM Judge agreement with human experts is strongly related to whether the judge can answer the underlying benchmark question and whether its reference is correct.Figure 1 compares agreement across answerable and unanswerable questions and across reference conditions.

2 Related Work

Prior work studies biases and correctness evaluation in LLM-as-a-Judge, while newer approaches incorporate expert-verified references and domain-specific reasoning. The cited examples motivate benchmarks that distinguish stylistic preference from correctness in difficult technical settings.

  • Limitations and mitigation: LLM Judges can be biased by response length, order, and tone, motivating randomization, jury models, rubrics, prompt optimization, and related mitigation strategies.These approaches aim to improve agreement with human annotators.
  • Correctness evaluation: Prior studies report high human agreement for correctness judgments on short-answer datasets, while other work finds standard pipelines may ignore correctness-related evidence.The literature therefore leaves open how reliably judges assess correctness in more demanding settings.
  • Expert-informed evaluation: Several datasets incorporate verified references, and domain-specific studies compare LLM ratings with judgments from professionals.These efforts connect LLM evaluation with expert knowledge and annotated reasoning errors.

3 Datasets

The paper builds expert-grounded datasets for challenging correctness judgments in business, finance, mathematics, and reasoning. It combines newly authored questions, corrected references, diverse model responses, and expert annotations, while filtering ambiguous cases.

  • Dataset construction: BFF-Bench and VERDICTS combine expert-authored questions and reference solutions with human expert labels of model response correctness.The construction minimizes synthetic data and uses domain experts for question creation and grading.
  • BFF-Bench: BFF-Bench contains 80 two-turn conversational questions with gold answers designed to be challenging, unambiguous, and dependent on domain knowledge or quantitative reasoning.Financial professionals created and iteratively verified the question-response pairs.
  • (C)MT-Bench: The corrected MT-Bench subset includes 40 math and reasoning questions whose references were manually rewritten and verified after 15 (37.5%) original references were found incorrect or inconsistent.The corrected references contain both the correct answer and a coherent reasoning chain.
  • Model responses: Responses from six models across model families and sizes produced 1,200 question-response pairs for expert annotation.The models ranged from Gemma 2 2b to GPT-4o and were evaluated at temperature 0.7.
  • Expert annotation: For BFF-Bench, financial professionals graded responses against human-written references, marking a response incorrect if any part was wrong.Annotators could use external resources except LLM-based chatbots.
  • VERDICTS: The dataset statistics table reports benchmark and response-grading counts after filtering examples for annotator consensus.Its Single and Pairwise columns distinguish the two grading-task formats.

4 Experimental Setup

The experiments compare single and pairwise correctness judgments against human annotations using LLM Judges and automated baselines, with several reference types and Cohen’s κ evaluation. The study also tests robustness across prompts and controls for positional bias.

  • Judgment Tasks: The evaluation measures agreement with human annotators for single grading and pairwise grading, where responses are classified individually or compared for correctness.Each pair contains one expert-labeled correct response and one incorrect response.
  • Limitations: Pairwise results are interpretable only relatively because the dataset excludes pairs containing two correct or two incorrect responses.This design isolates the distinction between correct and incorrect responses but simplifies real-world pairwise evaluation.
  • Evaluation: Cohen’s κ is the primary evaluation metric, with F1 reported as an alternative.The reported judge comparisons cover both single and pairwise tasks.
  • Implementation: The judges use chain-of-thought and majority voting across five temperature-0.7 inferences, while pairwise references are order-reversed to address positional bias.The judge models are drawn from the candidate models, except Gemma 2 2B, which is candidate-only.
  • Grading References: The study evaluates no-reference, judge-generated, and human-written references, alongside wrong and randomly selected reference baselines.Wrong references preserve style while introducing inaccurate information or reasoning; random references come from other questions.
  • Baselines: Automated baselines include lexical-overlap metrics, embedding similarity models, and reward models, with pairwise choices based on their quality scores.Embedding scores use cosine similarity to human-written references, while reward models select the response with the higher predicted human preference.

5 Results

Human and automated evaluations show that BFF-Bench is challenging and that reference quality strongly affects LLM-Judge agreement. Human references make even the weakest judge outperform the strongest baseline, whereas no-reference judging can approach heuristic performance.

  • Candidate Correctness: 68.1% correctness was achieved by the best model on BFF-Bench, which was more challenging than (C)MT-Bench.Human experts rated candidate-model correctness on both benchmarks, with performance generally correlated across them.
  • Reference Effects: With a human reference, even the worst LLM Judge outperforms the best baseline method on single and pairwise judgment.The comparison includes lexical, embedding, and reward-model baselines.
  • Reference Effects: Without a provided reference, some LLM Judges and reward models perform on par with heuristics.This contrasts with the stronger performance observed when human references are supplied.
  • Judgment Tasks: Pairwise grading shows higher agreement with human annotators than single grading, partly because each pair contains a clear correct-versus-incorrect distinction.The two setups should not be interpreted as directly establishing that one is universally better.
  • Baseline Limitation: The embedding-model baseline could not include some of the five highest-performing MTEB models because their weights were unavailable.This limits the completeness of that baseline comparison.

6 Analysis

Human-written references produce the strongest agreement with expert judgments, while reference-free judging is reliable mainly when the judge can answer the underlying question. Reference correctness also shapes error patterns, including self-preference and failures caused by subtly wrong references.

  • Human-written references yield the highest agreement with human annotators across single and pairwise grading.This can allow a smaller judge with a human reference to outperform a larger judge without one.
  • Without a human-written reference, judges achieve high agreement mainly on questions they themselves answered correctly.For GPT-4o on questions it could not answer, pairwise Cohen’s κ fell from 0.86 with its own response as reference to 0.16.
  • This relationship is robust across both datasets, other models, and LLM juries.
  • Judges show higher false-positive rates when grading their own responses, especially with self-generated or absent references.Human references reduce both false-positive and false-negative rates and narrow the self-versus-other gap.
  • Wrong references produce low false-positive and high false-negative rates, whereas random references do not show the same effect.This indicates that judges are sensitive to reference correctness, and slightly incorrect references can be worse than unrelated or absent references.
  • Judges sometimes overrule incorrect references, but many failures still arise from reasoning errors about whether the response and reference agree.
  • Reference correctness, rather than human authorship alone, explains why reliable references improve agreement.The comparison includes human-written gold, verified GPT-4o, wrong, and random references.

7 Conclusion

The paper releases benchmarks and expert judgments for testing correctness-focused LLM evaluation, then concludes that trustworthy judging requires either judge competence on the question or a correct reference. Human verification remains necessary despite the scalability of automated judging.

  • Reliable LLM-as-a-Judge agreement requires either the judge’s ability to answer the underlying question or a correct reference.
  • Judgment performance changes substantially with the provided reference: accurate references improve agreement, while slightly incorrect ones can be worse than no reference.
  • Trustworthy LLM-as-a-Judge pipelines still require human oversight, including at minimum verification of reference responses.The paper notes that stylistic evaluation may scale consistently, but correctness judgments remain limited.
  • Verified synthetic references may improve agreement and scalability but can increase self-preference bias, leaving their broader pitfalls for future work.
  • The annotation process used domain experts to create, verify, and grade responses, with iterative review until 80 two-turn questions had verified answers.Annotators classified model responses as Correct, Incorrect, or Not sure using verified references as guides.

A.5 F1 Results

This section defines the agreement metric and reports baseline comparisons, prompt robustness, reference examples, and implementation details for F1-based results. Reward models generally outperform embedding models, which outperform lexical-overlap methods.

  • Results are compared across modified MT-Bench prompts and newly written prompts, with broadly similar outcomes indicating robustness to the underlying prompt.
  • An MT-Bench example highlights an incorrect reference in which the triangle’s correct area is 3 rather than −3.
  • Cohen’s κ measures agreement between judges and the human consensus while accounting for chance agreement.κ ranges from -1 for total disagreement through 0 for chance agreement to 1 for total agreement.
  • The reported confidence intervals use bootstrap estimates converted to symmetric intervals by selecting the bound producing the widest interval.
  • Reward models generally outperform embedding models, which in turn outperform lexical-overlap methods on the baseline comparisons.
  • The evaluation includes five reward models and five embedding models, while lexical metrics use default ROUGE, BLEU, and METEOR settings.
  • The reference-based classifiers select the response with greater similarity to the human-written reference, using either reward scores or embedding cosine similarity.

F Judge Overruling

GPT-4o sometimes identified that a reference answer was wrong but still marked the response correct. The example concerns fundamental misdefinitions of annuities and IRAs.

  • F Judge Overruling: GPT-4o noted inconsistencies between the reference and response but overruled the reference and marked the response correct.This behavior was observed multiple times only with GPT-4o.
  • F Judge Overruling: The reference incorrectly described annuities as investment accounts, while the assistant described annuities as income-oriented financial products.The assistant also identified fixed, variable, and indexed annuities, along with tax deferral and early-withdrawal penalties.
  • F Judge Overruling: The reference incorrectly labeled IRAs as insurance products and misstated their contribution limits.The assistant instead described IRAs as tax-advantaged savings accounts with distinct tax implications.
  • F Judge Overruling: The assistant’s comparison of annuities and IRAs was more accurate than the reference answer’s comparison.The comparison covered investment control, flexibility, tax treatment, and income stream.

G LLM Judge Ablations

The ablations found no sensitivity of GPT-4o Judge performance to temperature or chain-of-thought, suggesting robustness across these two hyperparameters.

  • G LLM Judge Ablations: GPT-4o Judge performance was not sensitive to temperature or chain-of-thought.The paper reports no significant performance shifts under either ablation.

H Jury Experiment

The majority-vote jury does not remove the core limitation of LLM Judges: performance remains poor when the judges collectively answer incorrectly and improves with human references. Across both datasets, reference provision also reduces the correctness-conditioned agreement gap.

  • H Jury Experiment: A five-judge majority jury does not escape the core limitation identified for individual LLM Judges.The jury aggregates five judges and conditions analysis on whether at least three were correct.
  • H Jury Experiment: Performance on collectively incorrect questions remains poor, reaching near 0 in the Single case, and is recovered only with a human reference.The majority vote reduces the number of questions on which the jury is collectively incorrect but does not solve those remaining cases.
  • H Jury Experiment: Across both datasets, no-reference settings show a large difference between agreement when the judge answered correctly and incorrectly.The analysis uses κCorrect, κIncorrect, and ∆κ, where ∆κ is their difference.
  • H Jury Experiment: Providing a reference increases overall agreement and decreases ∆κ across both datasets.This indicates that the effect is not specific to either source dataset.

J Expanded Models

Expanded-model analyses corroborate the main behavior using GPT-4o with human-verified references as a proxy label, while reasoning effort narrows but does not eliminate the no-reference gap.

  • J Expanded Models: The expanded-model results are corroboration rather than equivalent to the main experiment because GPT-4o with human references supplies a proxy correctness label.The authors verified high agreement between this judge-reference combination and human annotators in Figure 3.
  • J Expanded Models: Higher reasoning effort can narrow the no-reference gap for some models, such as GPT-5.4 at high effort.The authors relate this pattern to judges becoming better able to answer the question themselves.
  • J Expanded Models: The frontier-model analysis reports the same overall behavior as the main experiment: agreement is high when judges answer correctly, while human references provide a large lift when they answer incorrectly.These findings are reported in the Table 12 recreation using a proxy correctness label.
  • J Expanded Models: The no-reference gap does not close entirely, and a human reference still yields the most reliable judgments.This preserves the paper’s central result across reasoning levels.
Loading 2503.05061v3…