PaperlayerView exact revision on arXiv

Source-linked AI summary

Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges

Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin

arXiv:2504.15205v1cs.CLcs.AIcs.IR

TL;DR

Whether an LLM judge can replace humans for support evaluation remains unknown, despite support’s importance to assessing RAG quality. This study compares GPT-4o and human judgments across TREC 2024 RAG submissions; exact matches were 56% from scratch and 72% with post-editing.

  • Problem

    Whether an LLM judge can replace a human judge for support evaluation remains unknown, although support is crucial for assessing RAG-system quality.

    Source: §1 · p. 1

  • Method

    The study compares human and LLM judges for support assessment across 45 TREC 2024 RAG submissions and 36 topics.

    Source: §6 · p. 9

  • Results

    56% of judgments matched GPT-4o perfectly from scratch, rising to 72% with post-editing; disagreements mainly involved partial support, and independent human and LLAMA-3.1 judges agreed more with GPT-4o.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The findings provide evidence that LLM judges may be a reliable alternative for support evaluation.

    Source: §1 · p. 2, §6 · p. 10

    The study evaluates support only, which is one aspect of RAG evaluation.

    Source: §1 · p. 2

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) enables large language models (LLMs) to generate answers with citations from source documents containing "ground truth", thereby reducing system hallucinations. A crucial factor in RAG evaluation is "support", whether the information in the cited documents supports the answer. To this end, we conducted a large-scale comparative study of 45 participant submissions on 36 topics to the TREC 2024 RAG Track, comparing an automatic LLM judge (GPT-4o) against human judges for support assessment. We considered two conditions: (1) fully manual assessments from scratch and (2) manual assessments with post-editing of LLM predictions. Our results indicate that for 56% of the manual from-scratch assessments, human and GPT-4o predictions match perfectly (on a three-level scale), increasing to 72% in the manual with post-editing condition. Furthermore, by carefully analyzing the disagreements in an unbiased study, we found that an independent human judge correlates better with GPT-4o than a human judge, suggesting that LLM judges can be a reliable alternative for support assessment. To conclude, we provide a qualitative analysis of human and GPT-4o errors to help guide future iterations of support assessment.

1 Introduction

The introduction asks whether an LLM judge can replace humans for support evaluation and compares GPT-4o with human judgments. Exact matches were 56% for manual assessments from scratch and 72% with post-editing; an independent judge agreed more with GPT-4o than with the original human judge.

  • Support evaluation asks whether cited documents support answer information, a key measure of RAG-system quality; prior RAG work relies on automatic judges.Source: §1 · p. 1
  • The study compared GPT-4o with manual judgments from scratch and with humans post-editing GPT-4o predictions.It examined 45 submissions on 36 information-based queries.Source: §1 · pp. 1–2
  • The study focuses exclusively on support, one aspect of RAG evaluation, while other track analyses address relevance and nugget evaluation.Source: §1 · p. 2
  • 56% of manual-from-scratch judgments matched GPT-4o perfectly, rising to 72% with human post-editing.The study reports these results for its two manual assessment conditions.Source: §1 · p. 2
  • An independent human judge agreed more with GPT-4o than with the original human judge in a 537-pair disagreement study (Cohen’s κ of 0.27 vs. 0.07).The independent judge reassessed randomly sampled pairs from both assessment conditions.Source: §1 · p. 2

2 Background and Related Work

The background defines sentence-level support assessment and situates the study among prior automatic-judge work. Its large human annotation study provides a basis for comparing human and GPT-4o judges across RAG systems and varied query types.

  • Table 2 presents Taylor Swift examples of GPT-4o and human support judgments, highlighting the passage fragment supporting the answer sentence.Source: Table 2 · p. 3
  • Support is evaluated for answer-sentence and cited-document pairs, with a human or LLM judge assigning a scalar support value.Answers are segmented into sentences, each associated with document citations.Source: §2 · pp. 2–3
  • The study contributes 11K human assessments across multiple RAG systems and 36 topics with non-factoid, decompositional, and multi-perspective queries.The study design provides a backdrop for comparing human and GPT-4o judges.Source: §2 · p. 3
  • Prior RAG support work used NLI [10], prompted LLMs [9], and fine-tuned custom LLMs [26]; support has also been studied for summarization and natural-language explanations.Source: §2 · p. 3

3 Track Description & Assessment Methodology

The TREC 2024 RAG Track supplied queries and candidate passages for systems to generate sentence-segmented answers with citations. Support was judged using GPT-4o and human assessments, with sparse annotations and weighted precision and recall to evaluate citation support.

3.1 TREC 2024 RAG Track

  • The track’s generation task required systems to answer queries using candidate passages and associate citations with each answer sentence.Topics came from a fresh Bing Search-log scrape, and evaluation covered a budget-selected subset of 36 topics.Source: §3.1 · p. 4

3.2 Support Assessment

  • Support assessment used a three-level scale, with human judges working from scratch or post-editing GPT-4o labels.GPT-4o assessed each answer sentence with its cited passage; sentences without citations were automatically labeled no support.Source: §3.2 · pp. 4–5

3.3 Computational Cost & Evaluation Tradeoffs

  • To balance annotation breadth against cost, both judges assessed only the first cited passage for each answer sentence.This sparse approach covered more topics but evaluated fewer cited passages per sentence than dense annotation.Source: §3.3 · p. 5

3.4 Support Evaluation Metrics

  • Weighted precision measures support among citations, while weighted recall measures support among answer sentences; both use full, partial, and no-support weights.Precision penalizes overcitation and recall penalizes undercitation; because only the first citation was judged, the scores are identical when every sentence has at least one citation.Source: §3.4 · pp. 5–6

4 Experimental Results

Human and GPT-4o support scores were strongly correlated at the run level, though humans tended to assign lower support. Exact label agreement increased with post-editing, from 56% to 72.1%.

4 Experimental Results

  • Assessments covered 36 topics: 22 in the manual from-scratch condition and 14 with post-editing, with up to 45 participant submissions per topic.The evaluated annotations numbered 6,742 from scratch and 4,165 with post-editing.Source: §4 · p. 6

4.1 Weighted Precision and Recall

  • Above 0.79 Kendall’s τ, run-level weighted precision and recall scores correlated between GPT-4o and human judgments, while humans generally scored runs lower.The figure also shows variation across topics and individual participant scores.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 Confusion Matrices

  • 72.1% of post-edited judgments matched exactly, compared with 56% from scratch; post-editing reduced disagreements over partial support.In the post-editing condition, the full-support versus no-support reversal rose to 6.3% from 5.9% from scratch.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

5 Annotator Disagreements

An independent human judge agreed more with GPT-4o than with the NIST human judge on sampled disagreements. The analysis also surfaces distinct label tendencies and error patterns among judges.

Disagreement study

  • The disagreement study re-assessed 537 randomly sampled answer-sentence and first-cited-passage pairs across both assessment conditions.It included an independent human judge and LLAMA-3.1 405B using GPT-4o’s prompt.Source: §5 · p. 8

Results

  • On sampled disagreements, the independent human judge matched GPT-4o more often than the NIST human judge in both conditions, while LLAMA-3.1 405B aligned more with GPT-4o than with humans.For the independent judge, full-match rates were 51% versus 31% with the NIST human in from-scratch cases, and 52% versus 37% with post-editing; LLAMA-3.1 405B’s κ with GPT-4o was 0.60 and 0.46.Source: §5 · p. 8
  • In the disagreement sample, both LLMs labeled about 49–51% of pairs partial support; the human judge labeled 47% no support, while the independent judge labeled 50% full support.The authors leave the reasons for these differing label distributions for future work.Source: §5 · p. 8

Qualitative analysis

  • GPT-4o errors included confusing similar phrases, overlooking sentence endings and favoring full support, or assigning partial support when the passage supported none of the answer.The examples came from the Taylor Swift topic; Table 4 highlights the passage fragments supporting answer sentences.Source: §5 · p. 9, Table 4 · p. 9
  • Human judges sometimes missed support stated within passages and labeled answers no support, or labeled answers full support without passage evidence.The authors suggest the latter may reflect reliance on memory or topic understanding rather than the passage text.Source: §5 · p. 9

6 Conclusion

Across TREC 2024 RAG Track support judgments, GPT-4o matched human assessments more often when annotators post-edited its predictions, while disagreements clustered around partial support. In disagreements, an independent human and a second LLM agreed more with GPT-4o than with the original human judgments.

  • The comparative study covered 45 TREC 2024 RAG submissions across 36 topics, with humans and LLMs judging support.Source: §6 · p. 9
  • Perfect human–GPT-4o matches rose from 56% in manual-from-scratch assessments to 72% with manual post-editing.Source: §6 · p. 10
  • Human–LLM disagreements mainly involved sentence–passage pairs judged as partially supported.Source: §6 · p. 10
  • In disagreements, an independent human judge and LLAMA-3.1 agreed more with GPT-4o than with the original human judge.Source: §6 · p. 10
Loading 2504.15205v1…