PaperlayerView exact revision on arXiv

Source-linked AI summary

UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor

Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, Jimmy Lin

arXiv:2406.06519v1cs.IR

TL;DR

Human relevance judgments are costly, and prior LLM assessment work lacked reusable software artifacts. UMBRELA is an open-source GPT-4o toolkit that reproduces that work; across TREC Deep Learning Tracks from 2019–2023, its judgments correlate highly with retrieval-system rankings based on human judgments.

  • Problem

    Human relevance judgments are costly and labor-intensive, while Thomas et al.’s LLM assessment study did not provide reusable software artifacts.

    Source: §1 · p. 1

  • Method

    UMBRELA reproduces Thomas et al.’s LLM relevance-assessment approach as an open-source toolkit, using zero-shot DNA prompting with GPT-4o.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    Across TREC Deep Learning Tracks from 2019–2023, GPT-4o assessments verified Thomas et al.’s findings and correlated highly with rankings from human judgments.

    Source: §1 · p. 1, §4 · p. 3

  • Takeaways & Limitations

    UMBRELA is shared as an extensible resource for automatic relevance assessment and is slated for use in the TREC 2024 RAG Track.

    Source: §1 · p. 1

    The study treats human assessor judgments as “gold,” although discrepancies may reflect inaccurate assessments or incomplete information about user intent.

    Source: §3.1 · p. 2, §4 · p. 4

Abstract

from arXiv · show

Copious amounts of relevance judgments are necessary for the effective training and accurate evaluation of retrieval systems. Conventionally, these judgments are made by human assessors, rendering this process expensive and laborious. A recent study by Thomas et al. from Microsoft Bing suggested that large language models (LLMs) can accurately perform the relevance assessment task and provide human-quality judgments, but unfortunately their study did not yield any reusable software artifacts. Our work presents UMBRELA (a recursive acronym that stands for UMbrela is the Bing RELevance Assessor), an open-source toolkit that reproduces the results of Thomas et al. using OpenAI's GPT-4o model and adds more nuance to the original paper. Across Deep Learning Tracks from TREC 2019 to 2023, we find that LLM-derived relevance judgments correlate highly with rankings generated by effective multi-stage retrieval systems. Our toolkit is designed to be easily extensible and can be integrated into existing multi-stage retrieval and evaluation pipelines, offering researchers a valuable resource for studying retrieval evaluation methodologies. UMBRELA will be used in the TREC 2024 RAG Track to aid in relevance assessments, and we envision our toolkit becoming a foundation for further innovation in the field. UMBRELA is available at https://github.com/castorini/umbrela.

1 INTRODUCTION

Human relevance judgments are costly and can be inaccurate when assessors do not grasp search intent. UMBRELA reproduces Thomas et al.’s LLM assessment work as an open-source toolkit and reports strong alignment with retrieval-system rankings across TREC Deep Learning Tracks from 2019–2023.

1 INTRODUCTION

  • Human relevance assessments are time-consuming, labor-intensive, and costly, and may misalign with searchers’ intentions when assessors do not grasp the information need.LLMs may offer a cost-effective alternative and can help fill gaps in incomplete judgments.Source: §1 · p. 1
  • UMBRELA reproduces Thomas et al. [21] as an open-source toolkit, following their zero-shot DNA prompting technique.DNA means Descriptive, Narrative, and Aspects.Source: §1 · p. 1
  • Across TREC Deep Learning Tracks from 2019–2023, GPT-4o assessments verified Thomas et al.’s [21] findings and correlated highly with rankings from human judgments.The authors report that these correlations support LLMs as effective relevance assessors.Source: §1 · p. 1
  • UMBRELA takes a query and passages as input, then assigns relevance grades based on the LLM configuration.Source: §1 · p. 1
  • The toolkit is shared with the research community and is slated for use in the TREC 2024 RAG Track.The authors present this deployment as a demonstration of its practical utility for automatic relevance assessment.Source: §1 · p. 1

2 RELATED WORK

Prior work has pursued automated relevance assessment to reduce the effort of preparing retrieval test data. LLM annotation studies report improved alignment with human assessments when models receive detailed prompting and guidance.

2 RELATED WORK

  • Automated techniques have been developed to address the substantial effort required to prepare large-scale retrieval test data.Source: §2 · p. 1
  • LLM-based IR evaluation studies have used zero-, one-, and few-shot prompting, roles, detailed instructions, and multiple judges to improve alignment with human assessments.Source: §2 · p. 2
  • Both proprietary and open-source LLMs have shown effectiveness in text annotation, and prompting techniques have been presented as substitutes for manual annotation.Source: §2 · p. 2

3 METHODOLOGY

UMBRELA reproduces LLM relevance assessment using TREC Deep Learning Track judgments and zero-shot DNA prompting with GPT-4o. Its procedure uses a four-grade relevance scale and removes noncanonical near-duplicates from the 2022–2023 data.

3 METHODOLOGY

  • UMBRELA applies LLMs to relevance assessment, reproducing the work of Thomas et al. [21].Source: §3 · p. 2

3.1 DL TREC Judgments

  • Human qrels from TREC Deep Learning Tracks 2019–2023 were reassessed with UMBRELA using a four-grade scale from irrelevant to perfectly relevant.The original judgments were made by NIST assessors; the authors treat them as “gold” labels based on assessors’ understanding and general knowledge.Source: §3.1 · p. 2
  • For the 2022–2023 tracks, the study retained canonical passages and excluded other near-duplicates from both qrels and retrieved runs.The published qrel counts therefore exceed the number of judgments actually assigned by humans.Source: §3.1 · p. 2

3.2 Prompting Details

  • UMBRELA follows Thomas et al. [21] with zero-shot GPT-4o prompting using DNA’s descriptive, narrative, and aspects sections.Descriptive and narrative sections contextualize the query and passage; aspects guides the model through relevance-labeling steps and specifies the output structure.Source: §3.2 · p. 2
  • Table 2 reports Cohen’s κ agreement and Kendall’s τ and Spearman’s ρ correlations for nDCG@10 evaluations, with near-duplicate preprocessing marked by an asterisk.Source: Table 2 · p. 3

4 RESULTS

Across the TREC Deep Learning Tracks, UMBRELA’s GPT-4o judgments show fair agreement with human labels and high correlation in retrieval-system rankings, while label-level results and case studies reveal discrepancies and ambiguity.

4 RESULTS

  • The GPT-4o experiments used temperature 0, top p=1, frequency penalty 0.5, and presence penalty 0, following Thomas et al.’s parameter configuration.No stopwords were used.Source: §4 · p. 3
  • Cohen κ scores show fair agreement between human and LLM judgments in both four-label and binarized comparisons.The binarized form merges labels 0–1 as non-relevant and 2–3 as relevant.Source: §4 · p. 3
  • LLM label prediction accuracy is roughly 75% for non-relevant, 50% for related, 30% for highly relevant, and 45% for perfectly relevant labels.These accuracies are measured with respect to human labels across the datasets.Source: §4 · p. 3
  • High Kendall τ and Spearman ρ correlations for nDCG@10 evaluations indicate alignment between retrieval-system rankings based on human and LLM judgments.Figure 3 compares evaluations using original human and LLM assessments.Source: §4 · p. 3, Figure 3 · p. 4
  • For a query about Thai people’s daily life, the authors judge a human-assigned perfectly relevant label unwarranted for a passage about the Thai flag, and consider the LLM judgment more accurate.The authors note that ambiguity may reflect inaccurate assessments or incomplete information about user intent.Source: §4 · pp. 3–4
  • In a Medicare query case, the passage describes CPAP rather than defining mechanical ventilation, leading the authors to question the human-assigned label.They report similar classification failures for this query.Source: §4 · p. 4

5 CONCLUSION

The paper reports that GPT-4o can perform relevance assessment across TREC Deep Learning Tracks from 2019–2023, with significant correlations between human- and LLM-assessed judgments. It packages the methodology as the open-source UMBRELA toolkit.

5 CONCLUSION

  • Across TREC Deep Learning Tracks from 2019–2023, the study reports GPT-4o relevance-assessment prediction quality and significant correlations between human- and LLM-assessed judgments, and releases the methodology as open-source UMBRELA.Source: §5 · p. 4
Loading 2406.06519v1…