PaperlayerView exact revision on arXiv

Source-linked AI summary

Perspectives on Large Language Models for Relevance Judgment

Guglielmo Faggioli, Laura Dietz, Charles Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, Henning Wachsmuth

arXiv:2304.09161v2cs.IRcs.CY

TL;DR

Human relevance judgments are costly, and it remains unclear whether LLM judgments can reliably support retrieval evaluation. This perspective paper maps human–machine judgment strategies and reports a pilot comparison with trained assessors; its findings show promise but do not support fully automated annotation at the time of writing.

  • Problem

    Human relevance judgments are time-intensive and costly, while the effectiveness of LLM-based judgments for retrieval evaluation remains unclear.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    The paper discusses LLM support strategies, proposes a human–machine collaboration spectrum, and reports a pilot comparison with trained assessors.

    Source: §3 · p. 3, §8 · p. 9

  • Results

    The pilot results show promise in mimicking human relevance assessments, but the authors identify reasons against fully automated annotation at the time of writing.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The paper presents human–machine collaboration as a possible way to employ LLMs in evaluation, while concluding that full automation is not yet warranted.

    Source: §8 · p. 9

    The authors note that there is currently no proof that LLM evaluations have any relationship to reality.

    Source: §7.2 · p. 21

Abstract

from arXiv · show

When asked, large language models (LLMs) like ChatGPT claim that they can assist with relevance judgments but it is not clear whether automated judgments can reliably be used in evaluations of retrieval systems. In this perspectives paper, we discuss possible ways for LLMs to support relevance judgments along with concerns and issues that arise. We devise a human--machine collaboration spectrum that allows to categorize different relevance judgment strategies, based on how much humans rely on machines. For the extreme point of "fully automated judgments", we further include a pilot experiment on whether LLM-based relevance judgments correlate with judgments from trained human assessors. We conclude the paper by providing opposing perspectives for and against the use of~LLMs for automatic relevance judgments, and a compromise perspective, informed by our analyses of the literature, our preliminary experimental evidence, and our experience as IR researchers.

1 INTRODUCTION · 2 RELATED WORK

The paper examines how LLMs might support or automate text relevance judgments, proposing a human–machine collaboration spectrum and a balanced discussion informed by literature and a pilot comparison with human assessors. It situates that proposal among earlier efforts to scale or automate judgments and the mixed evidence on LLM agreement with humans.

1 INTRODUCTION

  • More than 86,000 pooled documents from 50 TREC-8 topics and 129 systems required more than 700 assessor hours and about USD 15,000.Under the Cranfield paradigm, test collections include a corpus, topics, and document relevance judgments, traditionally supplied by human assessors.Source: §1 · p. 1
  • The authors propose a spectrum from manual to fully automated judgments and examine LLM assistance, risks, open questions, a pilot human-agreement experiment, and perspectives for, against, and between automation.Their discussion focuses on text-based test collections and argues that LLMs can already help humans in relevance assessment, even if fully automatic judgments may not yet be ready.Source: §1 · p. 2

2 RELATED WORK

  • Related work is organized along the collaboration spectrum, spanning conventional support tools, LLM summaries, divided human–LLM tasks, and human selection or verification of machine judgments.The spectrum also includes humans accepting or rejecting an LLM-generated judgment and fully automatic judgments.Source: §2 · p. 2, §2.3 · p. 3

2.1 Human Judgment

  • Crowdsourcing can scale human judgments but may reduce reliability; it can be viable in accessible domains with quality controls, while LLMs alone outperform crowdworkers on several text annotation tasks.Prior work examines crowdsourced reliability [14], workforce and requester interactions, and quality-management strategies [63]; viability under suitable conditions is reported in.Source: §2.1 · p. 2

2.2 Human Verification and AI Assistance

  • Human-assisted approaches reduce judgment effort while leaving humans in control of defining relevance or checking machine assessments.Options include metrics for incomplete judgments, transferring labels to similar passages, active learning, identifying documents for low-bias pools, and EXAM answerability questions.Source: §2.2 · p. 2

2.3 Fully Automated Test Collections

  • Wikimarks and related distant-supervision methods create queries and relevance labels from semi-structured, human-authored content without human judgments.Wikimarks uses Wikipedia titles and headings as queries and treats the following passage as relevant; other approaches exploit anchor text, article metadata, directory categories, Freebase glosses, or infoboxes.Source: §2.3 · p. 2
  • Prior LLM-based relevance studies report both high rank correlation with human judgments and poor correlation, while automatically derived system leaderboards were comparable to manual ones.The passage also notes that preliminary NLP annotation results had high variance in quality [92].Source: §2.3 · p. 3

3 SPECTRUM OF HUMAN–MACHINE COLLABORATION

The paper proposes a four-level human–machine collaboration spectrum for relevance judgments, from human-only judging through AI assistance and human verification to full automation. It asks where this division of labor can produce ideal judgments at the best cost, with tasks assigned according to human and machine competence.

Spectrum overview

  • The spectrum organizes relevance-judgment strategies by how labor is divided between humans and LLMs, ranging from human-only judgment to complete machine replacement, with assistance at varying degrees in between.The paper frames finding the ideal point on this spectrum at the best cost as a central research question, informed by competence partitioning.Source: §3 · pp. 3–4

Human Judgment

  • Human judgment represents the status quo: assessors decide relevance without LLM influence, using interface features such as scan-term highlighting, judged-document filtering, or similarity ordering.The paper describes humans as the only reliable judges in this approach.Source: §3 · p. 3

AI Assistance

  • AI assistance could help assessors judge documents more efficiently through LLM-generated summaries or nugget-based approaches that identify relevant information in documents.The paper asks which subtasks, such as summarization or keyphrase extraction, LLMs can take over while supporting reliable judgments.Source: §3 · p. 3

Human Verification

  • In human verification, an LLM proposes a judgment and rationale for each document, while humans make the ultimate decision, particularly in challenging or low-confidence cases.Two LLMs could also propose judgments for a human to choose between, but LLM bias may go unrecognized by assessors; the paper raises questions about which tasks still require human input and prompt engineering [91].Source: §3 · p. 3

Fully Automated

  • Fully automated systems could replace human assessors if they judge relevance reliably, potentially matching or surpassing human judgment for a specific corpus or domain.The paper also asks whether LLMs could generate complete test collections, including documents or passages, queries or conversations, and relevance judgments.Source: §3 · pp. 3–4

4 OPEN ISSUES AND OPPORTUNITIES

LLMs could make relevance judgments cheaper and enable broader evaluation, but their quality, trustworthiness, and ability to assess diverse notions of relevance remain uncertain. The section also considers how reliable LLM judgments might support more realistic evaluations and whether machine judgments could eventually exceed human ones.

4.1 LLM Judgment Cost and Quality

  • LLMs could make judgments cheaper, but their quality and substitutability across preference, binary, graded, and explained tasks remain uncertain and need validation.The authors propose LLM-specific quality estimation and assurance; fine-tuning on human assessments or active learning could help align judgments with humans [89].Source: §4.1 · p. 4, Table 2 · p. 4

4.2 Human Verification

  • Multiple LLM assessors may produce correlated judgments because they share training data; training on different data or user types could diversify or personalize their perspectives [88].Personalized models might make judgments reflecting particular user groups’ perspectives on relevance.Source: §4.2 · p. 4
  • LLMs’ ability to judge factuality is unclear, raising questions about assessing judgment reliability and verifying whether underlying training sources are accurately represented.The example of a document claiming lemons cure cancer shows why topical matching alone may not justify relevance; the authors also stress distinguishing human- from machine-generated sources in correctness-critical contexts such as journalism.Source: §4.2 · p. 4
  • Intrinsic bias may skew relevance judgments, while hallucinations motivate research into reasoned judgments that make the basis for an assessment more explicit.The authors call for evaluating how bias affects evaluations using LLM-supported or LLM-based judgments.Source: §4.2 · pp. 4–5
  • It remains open how to translate detailed guidelines into prompts and whether LLMs can assess relevance beyond topicality, including style and cultural context; human intervention may remain important.The authors note that relevance can depend on aspects not readily discernible to LLMs.Source: §4.2 · p. 5

4.3 Fully Automated

  • Using an LLM judge to evaluate an LLM ranker raises circularity concerns, but rankers may be smaller while judges can use extra information or combine complex models in an ensemble.The authors expect ranking models to favor lower cost and latency, potentially through knowledge distillation, while judging latency may permit more complex assessors.Source: §4.3 · p. 5
  • Reliable LLM judgments with little human verification could relax standard Cranfield assumptions and scale evaluation of changing information needs, such as conversational search sessions.Collaborative human–machine judgment could support more comprehensive, realistic notions of relevance.Source: §4.3 · p. 5
  • LLMs might eventually provide more numerous and consistent judgments than time-limited humans, but human-annotated gold standards cannot reveal when machine judgment quality surpasses human quality.The paper notes that machines have surpassed humans in some NLP tasks, but a human-based measurement ceiling complicates detecting such gains.Source: §4.3 · p. 5

5 PRELIMINARY ASSESSMENT

This preliminary pilot compares two LLMs with human assessors across two collections and finds that agreement varies by collection and relevance grade.

5.1 Methodology

  • The pilot compared GPT-3.5 and YouChat with human assessors using binary and graded judgments on TREC-8 and TREC-DL 2021, with two tailored prompts.The experiments were conducted in January and February 2023.Source: §5 · p. 5, §5.1 · p. 5
  • TREC-8 has long documents and detailed information needs, whereas TREC-DL 2021 has short passages and question queries; its judgments postdate GPT-3.5’s training-data cutoff.TREC-DL 2021 was also released before GPT-3.5, and was neither used to train it nor included in a participating system.Source: §5.1 · p. 5
  • Researchers sampled 1,000 topic–document pairs per collection, but YouChat scalability restricted some experiments to 100 random samples per relevance grade; prompts were intentionally left unoptimized.Prompt engineering was left for future work.Source: §5.1 · p. 6

5.2 Results

  • On TREC-8, GPT-3.5 agreed on 90% of human-labeled non-relevant documents but 47% of relevant ones, while YouChat’s agreement fell from 74% to 33%.Source: Table 3 · p. 6, §5.2 · p. 6
  • On TREC-DL 2021, YouChat agreed on 96 of 100 grade-3 pairs but only 42 of 100 non-relevant pairs, while GPT-3.5 struggled with grades 1 and 2.Source: Table 4 · p. 6, §5.2 · p. 6
  • The authors hypothesize that LLMs miss subtle distinctions humans use for borderline binary judgments and difficult grade-1-or-0 decisions on the graded scale.Source: §5.2 · p. 6

6 RE-JUDGING TREC 2021 DEEP LEARNING

Section 6 re-evaluates TREC-DL 2021 passage-ranking submissions with GPT-3.5 judgments designed to follow the track’s graded methodology; the top run remains unchanged, but agreement is fair and evaluation measures distinguish fewer run pairs.

6.1 Methodology

  • They re-judged the TREC-DL 2021 pool of 10,828 human judgments using temperature-zero GPT-3.5, following the track’s four-level relevance scale.The original pool covered 53 questions and used a combination of assessment methods, including active learning.Source: §6.1 · p. 6, §6 · p. 6
  • The few-shot prompt used examples for each relevance level, with two examples for non-relevant judgments, and substituted “relevant” for the track’s “related” label.Each LLM judgment cost about USD 0.01; total spending was USD 111.90, including duplicate requests.Source: §6.1 · p. 7

6.2 Results

  • Binarized LLM judgments yielded Cohen’s κ = 0.26, indicating fair agreement.For comparison, Cormack et al. report κ = 0.52, or moderate agreement, between two types of human judgments in a similar experiment.Source: §6.2 · p. 7
  • The official top-ranked run remained top under LLM judgments, and Figure 3 presents system-ranking correlations for MAP and NDCG@10.Voorhees reported Kendall’s τ = .90 for MAP in a similar comparison between two types of human judgments.Source: §6.2 · p. 7, Figure 3 · p. 7
  • With GPT-3.5 judgments, 65% of run pairs were distinguished under MAP and 69% under NDCG@10, versus 72% and 74% with human judgments.Sensitivity is the share of run pairs distinguished by paired t-tests at p < 0.05; the authors note that they do not correct for multiple comparisons.Source: §6.2 · p. 7

7 PERSPECTIVES FOR THE FUTURE

The paper weighs LLMs’ potential to support relevance judgments against risks to credible, human-centered evaluation. It recommends further study of human–machine collaboration and treating fully automated judgment as only one option.

7.1 In Favor of Using LLMs for Judgments

  • LLMs may provide explanations, scalable and consistent judgments, and support multilingual, multimodal, and lower-cost collection building as complements to human assessors.The authors suggest explanations may help non-expert assessors, while human review can control quality and provide feedback.Source: §7.1 · p. 8

7.2 Against Using LLMs for Judgments

  • Because relevance is subjective and human-centered, LLM judgments need verification for systematic differences, while using the same model to judge and retrieve risks circular, inflated evaluations.The authors note that human disagreement is usually not consequential absent systematic causes, and that computational expense or extra information do not necessarily resolve the circularity concern.Source: §7.2 · p. 8
  • LLM-generated search responses often lack attributable, inspectable sources and user probing, so reliable attribution and systematic interrogation remain necessary.The paper also says prompt-engineering approaches and their effects are insufficiently studied and comparatively opaque.Source: §7.2 · pp. 8–9
  • Generative AI raises socio-technical concerns about misinformation, intellectual property, bias, and uncontrolled model changes, leaving the authors divided on deployment ethics.They question whether assessors can know if unseen model information biases judgments or whether a provider changes the model without notice or control.Source: §7.2 · p. 9

7.3 A Compromise: Double-checking LLMs and Human–Machine Collaboration

  • A pilot found reasonable correlation and similar leaderboards for trained human and fully automated LLM judgments; the authors propose shared-pool checks, disclosure, and limited early-stage uses.They recommend reporting prompts, hyperparameters, and judgment paradigm, and suggest automatic judgments for prototypes, novel tasks, and large-scale training.Source: §7.3 · p. 9
  • The authors see AI assistance as a credible but underexplored alternative to fully automated evaluation, and call for research on human verification of LLM rationales.LLM rationales may help assessors identify connections requiring world knowledge, but humans must judge whether those rationales are true and reliable or hallucinated.Source: §7.3 · p. 9

8 CONCLUSION … 1 INTRODUCTION

The paper finds that LLMs show promise for relevance judgments but argues that current limitations prevent fully automated use, while proposing ways to support human assessors. It combines a human–machine collaboration spectrum and a pilot comparison with perspectives for, against, and between automatic judgments.

8 CONCLUSION

  • LLMs show promise in mimicking human relevance assessments, but several reasons still prevent their use as fully automated annotation tools.The conclusion nevertheless sees a spectrum of ways to use LLMs in support of human assessors.Source: §8 · p. 9

ABSTRACT

  • The paper presents perspectives for and against automatic LLM relevance judgments, plus a compromise informed by literature analysis, preliminary experimental evidence, and IR research experience.The authors hope to prompt constructive community discussion rather than a stalemate over using LLMs for evaluation.Source: ABSTRACT · p. 13

CCS CONCEPTS

  • The paper’s listed subject classification is information systems and relevance assessment.Source: CCS CONCEPTS · p. 13

KEYWORDS

  • The paper’s listed keywords are large language models, relevance judgments, human–machine collaboration, and automatic test collections.Source: KEYWORDS · p. 13

1 INTRODUCTION

  • Cranfield-style relevance judgments rely on human assessors, making the process time-intensive and costly.The paper frames its LLM comparison around text collections and the established methodology for assessing textual-document relevance.Source: §1 · pp. 13–14
  • The proposed collaboration spectrum ranges from manual to fully LLM-automated judgments, with human involvement and decision-making varying across strategies.The paper argues that LLMs can already assist human assessors to varying extents, even if fully automatic judgments may not yet be ready.Source: §1 · p. 14
  • The paper pilots whether LLM-generated judgments agree with trained human judgments and focuses its comparison on text-based test collections.It also analyzes LLM-specific caveats.Source: §1 · p. 14

2 RELATED WORK

Because creating relevance judgments manually is labor-intensive, prior work explores tools and human–machine strategies to improve assessment quality, consistency, and efficiency. The related work spans human judgment, machine assistance with human verification, and fully automated test collections.

2.1 Human Judgment

  • Assessment tools support human annotators with features such as text highlighting, pre-annotations, and external data integration, among 13 features identified by Neves and Seva.The examples of external data sources include ontologies and thesauri.Source: §2.1 · p. 14
  • Crowdsourcing can scale relevance labeling as an alternative to trained assessors, provided quality controls are used and the domain is accessible to non-experts.Controls include label aggregation, task design, worker selection, and extra machine-generated information to improve efficiency; crowdsourcing also raises reliability, cost and quality management, and worker–requester interaction concerns.Source: §2.1 · p. 14

2.2 Human Verification and AI Assistance

  • Human-verification approaches vary: active learning selects documents for human assessment, AutoTar-style selection can find otherwise-missed relevant documents, and AP adjustments address incomplete-judgment bias.In this setting, humans relinquish control over document selection or machine-derived assessments but retain control over defining relevance; Passage-ROUGE propagates labels to similar unjudged passages, with BertScore as an alternative similarity approach to ROUGE.Source: §2.2 · p. 14
  • EXAM uses human-designed questions answerable from relevant text and automated QA over retrieved documents, while LLM assessments may support QPP, a task performed without human-made relevance judgments.Related question-based labeling paradigms have also been used in other tasks, and LLM effectiveness for QPP is supported by prior work.Source: §2.2 · pp. 14–15

2.3 Fully Automated Test Collections

  • Fully automated test collections derive queries from Wikipedia titles and headings with passages as relevant, or infer author relevance from article anchors, metadata, categories, glosses, and infoboxes [7, 12, 10, 25, 44, 51].Wikimarks has also been applied to aspect-based summarization and text segmentation. Prior TREC CAR leaderboard correlations were very high, motivating a similar LLM study in Section 5.Source: §2.3 · p. 15

3 SPECTRUM OF HUMAN–MACHINE COLLABORATION

The section proposes a four-level human–machine collaboration spectrum, from manual human judgments to fully automated judgments, with intermediate forms of AI assistance and human verification. It identifies competence partitioning as a key question and currently places balanced human–machine task allocation within AI assistance.

3 SPECTRUM OF HUMAN–MACHINE COLLABORATION

  • The spectrum ranges from manual human judgment through AI assistance and human verification to fully automated relevance judgments.It organizes possible strategies by how much humans rely on machines.Source: §3 · p. 15

Human Judgment

  • Human judgment leaves relevance decisions to assessors, who may use interface aids such as highlighting or document ordering without automatic training or feedback.The authors describe this as the status quo, with humans remaining the only reliable judges.Source: §3 · p. 15

AI Assistance

  • AI assistance can support assessors with document summaries or use trained LLMs to count relevant information nuggets in retrieved results.The authors ask how AI tools can make judgment more efficient while maintaining reliability.Source: §3 · p. 15

Human Verification

  • In human verification, an LLM proposes a judgment with a rationale, or multiple LLMs offer judgments for a human to compare, while the human retains the final decision.Human intervention may be reserved for difficult, low-confidence, or disagreement cases; hidden LLM bias may go uncorrected if assessors do not see the relevant information.Source: §3 · pp. 15–16

Fully Automated

  • Fully automated judgments would replace human assessors if LLMs can reliably produce high-quality judgments for a given corpus or domain.The authors also raise the possibility of using generative LLMs to create test collections, including corpora, queries, abstracts, and conversations.Source: §3 · p. 16

3 SPECTRUM OF HUMAN–MACHINE COLLABORATION

  • Balanced competence partitioning assigns each task to whichever—human or machine—is better at it, but the proposed spectrum currently places it within AI assistance.The authors identify where ideal judgments can be obtained at the best cost as a central question.Source: §3 · p. 16

4 OPEN ISSUES, FORESEEABLE RISKS, AND OPPORTUNITIES

The section identifies unresolved risks in LLM relevance judgments, including uncertain quality, factuality, bias, and how to convey task-specific relevance criteria. It also considers how human–machine judgment could support richer evaluation paradigms while raising questions about whether LLMs can be evaluated against human judgments.

4.1 AI Assistance

  • LLM judgments may increase annotation volume while reducing quality, so task-specific quality assurance and learning from human decisions are needed.The extent of any quality deterioration is unclear, and quality improvement is also possible; proposed learning could begin with suggestions and improve through annotator decisions.Source: §4.1 · p. 16

4.2 Manual Verification

  • Multiple similar LLM assessors may give correlated answers of unknown correctness, while personalized models could broaden judgments but remain computationally unavailable.Personalized models trained on different user data could auto-annotate from subjective viewpoints and expand the judgment pool.Source: §4.2 · pp. 16–17
  • Fully automated judgments must assess factuality and source reliability, while raising questions about automated fact-checking and distinguishing human- from LLM-generated data.Topically relevant documents can still be false or unreliable, such as documents claiming lemons cure cancer.Source: §4.2 · p. 17
  • LLM biases may carry into relevance judgments, so models and resulting evaluations need assessment for undue bias.For example, a model could favor scientific language over informal language; Bender et al. highlight limitations associated with LLMs, and identify intrinsic bias.Source: §4.2 · p. 17
  • It remains unclear how to translate comprehensive judgment guidelines for LLMs or assess relevance beyond topicality, including style, truthfulness, and community-specific needs.The authors argue that human intervention is needed to collect and judge additional facts and document aspects not yet easily discernible to LLMs.Source: §4.2 · p. 17

4.3 Fully Automated

  • Fully automated LLM evaluation risks circularity, but smaller ranking models, judges with extra information, or ensembles are proposed as possible ways forward.The judging model could be more cost-effective to run and have access to relevant facts or questions unavailable to the evaluated system.Source: §4.3 · p. 17
  • Reliable LLM judgments could relax Cranfield assumptions and enable evaluation of evolving information needs through connected conversational paths.TREC CAsT models information needs as multi-turn conversations in which users learn about a domain.Source: §4.3 · p. 17
  • If LLMs surpass human assessors, human-annotated gold standards may hide that improvement and limit evaluation.The paper notes that humans err and have limited time, while LLMs may provide more self-consistent judgments at scale; related NLP examples include scientific abstract classification and sentiment detection.Source: §4.3 · p. 17

5 PRELIMINARY ASSESSMENT

The preliminary assessment compares human and LLM relevance judgments across two TREC collections and finds that agreement patterns vary by collection and relevance grade.

5.1 Methodology

  • The preliminary, non-exhaustive comparison tested GPT-3.5 and YouChat on TREC-8 and TREC-DL 2021, using binary and graded judgments with corpus-specific prompts.The study sampled 1,000 topic–document pairs per collection for GPT-3.5 and limited YouChat to 100 samples per relevance grade; prompts were kept simple rather than optimized.Source: §5 · p. 18, §5.1 · p. 18

5.2 Results

  • On TREC-8, GPT-3.5 agreed on 90% of human non-relevant judgments but 50% of relevant ones; YouChat’s agreement was 74% and 33%, respectively.Source: §5.2 · p. 18, Table 3 · p. 19
  • On TREC-DL 2021, YouChat agreed on 96/100 human grade-3 pairs but only 42/100 grade-0 pairs, with agreement increasing with relevance grade.Source: Table 4 · p. 19, §5.2 · p. 19
  • The authors hypothesize that humans better detect subtle relevance distinctions, while LLMs correlate better with coarse-grained grades; they suggest centering grades on borderline relevance.They propose a scale from -3 to 3 around 0 (“borderline relevant”).Source: §5.2 · p. 19

6 RE-JUDGING TREC 2021 DEEP LEARNING

The paper re-judges the TREC 2021 Deep Learning passage-ranking pool with GPT-3.5 using graded relevance labels. Agreement with human judgments is fair on the binary scale, the top-ranked run is unchanged, and LLM-based measures distinguish fewer systems.

6.1 Methodology

  • The experiment re-judged the TREC-DL 2021 pool of 10,828 four-level judgments, drawn from 63 runs and 53 queries, with GPT-3.5 text-davinci-003 at temperature 0.The original task covered 200 test queries, and the experiment accessed the model through OpenAI’s API in February 2023.Source: §6.1 · p. 19
  • The prompt, inspired by Ferraretto et al., used few-shot examples: one each for Perfectly relevant, Highly relevant, and Related, two for Irrelevant, and used Relevant instead of Related.Judgments cost around USD 1 cent each; total spending was USD 111.90, including duplicate requests due to failures and other issues. Clarke et al. reported USD 25 cents per human label for a similar-scope task with a single-page prompt and no assessor training.Source: §6.1 · p. 19

6.2 Results

  • Voorhees reports Kendall’s τ=.90 for MAP in a similar human-judgment comparison, while the official top run remained top under GPT-3.5 judgments.Figure 3 compares run performance under official human judgments and unofficial LLM judgments for MAP and NDCG@10.Source: §6.2 · pp. 19–20, Figure 3 · p. 20
  • Human judgments distinguished 72% of systems under MAP (74% under NDCG@10), versus 65% (69%) under GPT-3.5 judgments.Sensitivity was computed from paired t-tests across all pairs of runs, counting pairs with p<0.05 as distinguished; the authors did not correct for multiple comparisons.Source: §6.2 · p. 20
  • Binary judgments yielded Cohen’s κ=0.26 (“fair”), below the human-human κ=0.52 (“moderate”) reported by Cormack et al..Table 5 summarizes the confusion matrices for both the four-level and binary scales; the binary mapping follows the TREC-DL 2021 convention.Source: §6.2 · p. 19, Table 5 · p. 20

7 PERSPECTIVES FOR THE FUTURE

The paper presents arguments for and against automatic LLM relevance judgments, plus a compromise informed by literature, pilot evidence, and IR researchers’ experience. It treats full automation as one extreme and human–machine collaboration as a potential path forward.

7.1 In Favor of Using LLMs for Judgments

  • LLMs can support relevance assessment with explanations, scalability, consistency, multilingual and multimodal capabilities, and lower-cost test collections, complementing human assessors.Explanations may help crowdworkers, while human review can check LLM labels and provide feedback; the authors envision reducing annotation costs and time for high-quality collections.Source: §7.1 · pp. 20–21

7.2 Against Using LLMs for Judgments

  • Because relevance is subjective and user-grounded, the community must test for systematic differences between LLM and human judgments before trusting models as replacements.The authors note that model opacity makes systematic bias difficult to define, and that using the same model for retrieval and judgments could inflate its evaluation while disadvantaging other approaches.Source: §7.2 · p. 21
  • LLM-based search also raises concerns about unattributed, hard-to-explore responses and opaque prompting, alongside social risks, bias, and uncontrolled model changes.The authors argue that reliable source attribution and systematic interrogation are needed for many information needs; they also report disagreement among themselves about deploying generative models before ethical concerns are addressed.Source: §7.2 · p. 21

7.3 A Compromise: Double-checking LLMs and Human–Machine Collaboration

  • The pilot found reasonable correlation and similar leaderboards between a fully automated LLM and highly trained human assessors, supporting further study and double-checking on a shared judgment pool.Compare label and system-ranking correlations before sharing LLM judgments as an alternative test collection, and disclose the judgment paradigm, prompts, hyperparameters, and reproducibility details; automated judgments may also support prototypes, novel tasks, and large-scale training.Source: §7.3 · pp. 21–22
  • Rather than treating full automation as the goal, the authors favor AI assistance as a credible but underexplored approach, with humans verifying LLM suggestions and rationales.Humans can assess whether a rationale is true and reliable, including whether an apparent connection reflects hallucination; further research should examine how LLM rationales affect assessors.Source: §7.3 · p. 22

8 CONCLUSION

The paper examines LLMs for automated relevance judgments, reports promising evidence that they can mimic human assessments, and presents perspectives on their role in IR evaluation and future research.

8 CONCLUSION

  • The experiments show promise in LLMs’ ability to mimic human relevance assessments.The conclusion presents the results as promising, not definitive.Source: §8 · p. 22
  • The paper reviews earlier efforts to automate and scale up relevance judgment and offers views on whether the IR community should employ LLMs in evaluation.Source: §8 · p. 22
  • The authors call for further research on LLMs for relevance judgment and describe the paper as a starting point.Source: §8 · p. 22

Related papers

Loading 2304.09161v2…