PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking LLM-based Relevance Judgment Methods

Negar Arabzadeh, Charles L. A. Clarke

arXiv:2504.12558v1cs.IRcs.CL

TL;DR

Human relevance judgments are costly and can be incomplete, while LLM assessment research has focused mainly on graded judgments rather than systematic comparisons across methods. This paper compares five LLM-based approaches across four datasets using human-label alignment and system-ranking agreement; pairwise preferences align best with human labels, while UMBRELA labels have the best or second-best ranking agreement. The conclusions are scoped to the approaches and test collections studied.

  • Problem

    LLM relevance assessment has been studied mainly through graded judgments, leaving alternative methods and their comparative alignment with human judgments underexplored.

    Source: §1 · pp. 1–2

  • Method

    The study compares five LLM-based judgment methods across TREC Deep Learning 2019–2021 and ANTIQUE, measuring human-label alignment and system-ranking agreement.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Pairwise preference labels consistently align best with human labels, while UMBRELA labels consistently show the best or second-best agreement with human system rankings.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors’ comparison methodology can be applied to other assessment approaches and test collections.

    Source: §6 · p. 10

    LLM judgments can be influenced by query-term inclusion or explicit relevance claims, and may be more lenient than human judgments.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly deployed in both academic and industry settings to automate the evaluation of information seeking systems, particularly by generating graded relevance judgments. Previous work on LLM-based relevance assessment has primarily focused on replicating graded human relevance judgments through various prompting strategies. However, there has been limited exploration of alternative assessment methods or comprehensive comparative studies. In this paper, we systematically compare multiple LLM-based relevance assessment methods, including binary relevance judgments, graded relevance assessments, pairwise preference-based methods, and two nugget-based evaluation methods~--~document-agnostic and document-dependent. In addition to a traditional comparison based on system rankings using Kendall correlations, we also examine how well LLM judgments align with human preferences, as inferred from relevance grades. We conduct extensive experiments on datasets from three TREC Deep Learning tracks 2019, 2020 and 2021 as well as the ANTIQUE dataset, which focuses on non-factoid open-domain question answering. As part of our data release, we include relevance judgments generated by both an open-source (Llama3.2b) and a commercial (gpt-4o) model. Our goal is to \textit{reproduce} various LLM-based relevance judgment methods to provide a comprehensive comparison. All code, data, and resources are publicly available in our GitHub Repository at https://github.com/Narabzad/llm-relevance-judgement-comparison.

1 Introduction

LLMs offer a scalable alternative to human relevance labeling, but alternative assessment methods and their effects on evaluation have not been comprehensively compared. This paper compares five LLM-based methods using human-label alignment and system-ranking agreement across four datasets, and releases the generated judgments and experimental resources.

1 Introduction

  • Human relevance labeling is resource-intensive and can suffer from incomplete labels, annotator inconsistency, and high costs.LLM-based assessment is presented as a scalable, cost-effective alternative, though its consistency, alignment, and transparency remain concerns.Source: §1 · p. 1
  • The study compares five LLM-based relevance assessment methods and evaluates their alignment with human labels and agreement with system rankings.The comparison is intended to put different methods on a level playing field.Source: §1 · p. 2
  • The comparison covers binary and graded judgments, pairwise preferences, and document-agnostic and document-dependent nugget-based methods.The paper aims to compare methods beyond the graded assessments that have dominated prior work.Source: §1 · p. 2
  • Alignment tests whether LLM judgments preserve human-inferred document ordering, while system-ranking agreement compares rankings produced from LLM and human labels.The alignment comparison uses relative ordering rather than absolute grades, allowing methods that express relevance differently to be compared.Source: §1 · p. 2
  • Experiments use TREC Deep Learning collections from 2019–2021 and ANTIQUE, an open-domain question-answering dataset derived from Yahoo Answers.TREC collections use MS MARCO v1 and v2; these collections were selected in part to enable comparison with prior studies.Source: §1 · p. 2
  • The released resource includes judgments from five methods, GPT-4o and Llama 3.2B, and four datasets, with code and experimental results.The paper does not present or analyze the Llama results in its main text, but includes them in the repository.Source: §1 · p. 2

2 Background

Prior work reports high rank correlations between human and LLM-graded judgments, but also calibration and label-agreement differences. This paper addresses the lack of unified cross-method comparisons with a shared set of judgments and two comparison methodologies.

2 Background

  • Automated evaluation can scale benchmarking, but human judgment remains important for subjective quality and alignment with human labels.The background also identifies interpretability of system comparisons and evaluation of RAG systems as relevant settings for LLM-based relevance assessment.Source: §2 · p. 3
  • Earlier LLM relevance studies report Kendall’s τ correlations above 0.85 in many human-versus-LLM graded-label comparisons, alongside a tendency for LLMs to assign higher grades.Reported agreement on labels can nevertheless be modest: studies found binary κ values as low as 0.07 and none of seven groups exceeded 0.45 against human labels.Source: §2 · p. 3
  • System-ranking comparisons commonly use Kendall’s τ; prior work reported τ=0.8956 between two TREC-6 label sets, with corresponding κ=0.52.The background notes τ=0.9 as a possible minimum benchmark for replacing human labels.Source: §2 · p. 3
  • Prior studies had not systematically compared different LLM-based relevance assessment methods, partly because comprehensive judgments and a standardized comparison methodology were lacking.The paper addresses this with judgments from five methods and two LLMs across four datasets, plus methods for comparing alignment with human labels and agreement with system rankings.Source: §2 · p. 3

3 Relevance Assessment Methods

The section compares binary, graded, pairwise, and nugget-based relevance assessments, using prior tools and prompts where possible to support replicability and reduce prompt-engineering bias.

3.1 Binary Relevance Assessment

  • Binary assessment maps each query-document pair to relevant or non-relevant; the study reproduces a TREC-assessor prompt while noting that binary labels can oversimplify relevance.The reproduced prompt previously showed high agreement with human judgments on REC-DL 2021 and TREC-8.Source: §3.1 · p. 4

3.2 Graded Relevance Assessment

  • Graded assessment assigns query-document pairs relevance levels; this study adopts UMBRELA2 prompts in a zero-shot setting.Graded labels support finer-grained measures such as nDCG, but can be subjective and require detailed assessor guidelines and training.Source: §3.2 · p. 4

3.3 Nugget-based Relevance Assessment

  • Nugget-based methods decompose a query’s information need into components, assessing how far and from which perspective a document fulfills it.The study compares query-only Exam nuggets with AutoNuggetizer nuggets generated from relevant documents.Source: §3.3 · p. 4

3.3.1 Document Agnostic (Exam).

  • Exam generates 10 concise nuggets from the query alone, then assesses each document against them with binary or graded judgments.Exam-Graded Max aggregates per-nugget grades by selecting the maximum score.Source: §3.3.1 · p. 4

3.3.2 Document Dependent (AutoNuggetizer).

  • AutoNuggetizer iteratively derives nuggets from relevant documents, merges similar concepts, caps the set at 30, and ranks nuggets as vital or okay.It assesses document support for each nugget and aggregates scores using functions that cover all, vital-only, or weighted nuggets, with stricter variants.Source: §3.3.2 · p. 5

3.4 Pairwise Relevance Judgments

  • Pairwise assessment asks which of two documents better answers a query; swapped-order judgments that disagree become ties, and wins are counted for scores.Sampling reduces the number of judgments from O(n^2) to O(n), specifically fewer than P·n with default P=7.Source: §3.4 · p. 5

4 Methodology

The methodology evaluates LLM judgments both by their ordering of human-labeled relevance categories and by their agreement with system rankings. It uses compatibility to compare system rankings across methods with different score scales.

4.1 Alignment with Human Labels

  • Human-label alignment compares documents from different relevance categories and scores whether each method agrees, disagrees, or ties with the human-imputed ordering.The comparisons are Best vs. Acceptable, Acceptable vs. UnAcceptable, and Best vs. UnAcceptable; documents in the same category are excluded.Source: §4.1 · p. 6

4.2 Agreement with System Rankings

  • Compatibility supports comparisons across relevance methods with different score scales because it uses relative ordering rather than absolute scores or recalibration.The section notes that methods such as Exam and AutoNuggetizer can produce different output ranges depending on aggregation strategy.Source: §4.2 · p. 6
  • Compatibility compares a system ranking with the closest ideal permutation of relevance levels, allowing documents tied within a level to vary in order.The ideal ranking set is constructed from each method’s relevance levels and aligned to the actual ranking’s ordering within those levels.Source: §4.2 · p. 6
  • Compatibility uses the maximum Rank Biased Overlap (RBO) across ideal permutations, with p = 0.9 to weight agreement at higher ranks.Larger p values place greater emphasis on lower-ranked documents.Source: §4.2 · p. 6

5 Experiments and Results

Across four datasets, the experiments compare human-label alignment with agreement in system rankings. Pairwise preferences lead on human alignment, while binary and UMBRELA judgments tend to agree most with human-based system rankings; the two objectives do not always coincide.

5.1 Datasets

  • The experiments use TREC DL 2019, 2020, and 2021 passage-ranking datasets and ANTIQUE, a non-factoid question-answering benchmark.System-ranking comparisons are limited to TREC DL because ANTIQUE has too few available runs for that analysis.Source: §5.1 · p. 7, §5.3 · p. 8

5.3 Agreement with System Rankings

  • Binary and UMBRELA achieve the highest agreement with human-based system rankings; Binary leads on DL-20 and DL-21, while UMBRELA leads on DL-19.Pairwise preferences are runner-up on DL-19 and competitive on the other two datasets; nugget-based methods are less consistent across datasets.Source: Table 3 · p. 7, §5.3 · p. 8

5.2 Alignment with Human Labels

  • ANTIQUE has more ties across most methods, while nugget-based methods show variable alignment and struggle on ANTIQUE.The authors suggest that ANTIQUE’s non-factoid answers can be valid from multiple perspectives, and that broader, more diverse nuggets may contribute to nugget methods’ performance drop.Source: §5.2 · p. 7
  • 91% average agreement for Preferences on Best vs. UnAcceptable comparisons and 71% on Best vs. Acceptable comparisons across datasets.Preferences achieves the highest agreement rates across all datasets; agreement generally falls as relevance distinctions become finer.Source: §5.2 · p. 7, Figure 1 · p. 8

5.4 Alignment vs. Agreement

  • Pairwise Preferences lead human alignment, whereas Binary and UMBRELA show stronger system-ranking agreement, so the preferred method depends on the evaluation objective.Nugget-based alignment and system-ranking agreement vary with dataset and aggregation strategy.Source: §5.4 · pp. 9–10

6 Concluding Remarks

Pairwise preference judgments align best with human labels, while UMBRELA labels provide the best or second-best agreement with system rankings. The authors propose exploring human audits of UMBRELA labels that conflict with LLM preference judgments.

6 Concluding Remarks

  • The comparison evaluates LLM relevance judgments by both pairwise agreement with human labels and Kendall’s τ agreement with system rankings, using compatibility as a unified scoring function.Pairwise human-label agreement tests whether judgments preserve the relative order of relevance, independently of absolute score scale.Source: §6 · p. 10
  • Pairwise preference judgments consistently provide the best alignment with human labels, while UMBRELA labels provide the best or second-best agreement with system rankings.The authors propose exploring human audits of UMBRELA labels that are inconsistent with LLM-generated preferences.Source: §6 · p. 10
  • The authors report open-source Llama judgments in their repository and plan to extend the study to additional assessment approaches.They also note that the methodology can be applied to other assessment approaches and test collections.Source: §6 · p. 10
Loading 2504.12558v1…