PaperlayerView exact revision on arXiv

Source-linked AI summary

Redefining Retrieval Evaluation in the Era of LLMs

Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, Fabrizio Silvestri

arXiv:2510.21440v1cs.CLcs.IR

TL;DR

Traditional IR metrics reflect human relevance and rank-based examination, which do not capture how LLMs use retrieved passages or how distractors affect RAG. The paper introduces utility-based annotations and UDCG, an LLM-oriented metric. Across five QA datasets and six LLMs, the metrics outperform traditional and existing RAG-oriented approaches by up to 36%.

  • Problem

    Traditional IR metrics do not capture LLM-oriented passage utility, positional behavior, or the distracting effects of irrelevant passages in RAG.

    Source: §1 · p. 1, §3 · pp. 3–4

  • Method

    The paper scores passages by positive or negative utility for an LLM and uses those scores with an LLM-oriented positional discount in UDCG.

    Source: §1 · p. 2, §4 · p. 4, §5 · p. 5

  • Results

    By up to 36%, both metrics outperform traditional IR metrics and existing RAG-oriented approaches in predicting end-to-end system performance.

    Source: §8 · p. 9

  • Takeaways & Limitations

    UDCG provides a more reliable indicator of retrieval component quality for RAG systems by correlating with end-to-end answer accuracy.

    Source: §1 · p. 2

    The annotation requires LLM output probabilities, limiting use to settings with logit access and excluding text-only black-box APIs.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Traditional Information Retrieval (IR) metrics, such as nDCG, MAP, and MRR, assume that human users sequentially examine documents with diminishing attention to lower ranks. This assumption breaks down in Retrieval Augmented Generation (RAG) systems, where search results are consumed by Large Language Models (LLMs), which, unlike humans, process all retrieved documents as a whole rather than sequentially. Additionally, traditional IR metrics do not account for related but irrelevant documents that actively degrade generation quality, rather than merely being ignored. Due to these two major misalignments, namely human vs. machine position discount and human relevance vs. machine utility, classical IR metrics do not accurately predict RAG performance. We introduce a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones. Building on this foundation, we propose UDCG (Utility and Distraction-aware Cumulative Gain), a metric using an LLM-oriented positional discount to directly optimize the correlation with the end-to-end answer accuracy. Experiments on five datasets and six LLMs demonstrate that UDCG improves correlation by up to 36% compared to traditional metrics. Our work provides a critical step toward aligning IR evaluation with LLM consumers and enables more reliable assessment of RAG components

1 Introduction

Traditional IR metrics misalign with RAG because they reflect human rank-by-rank examination and relevance judgments, while LLMs process retrieved content differently. The paper introduces passage utility annotations and UDCG to better evaluate retrieval for RAG.

1 Introduction

  • Traditional metrics assume humans inspect documents sequentially and treat irrelevant passages alike, although LLMs process context holistically and distractors can degrade generation.These mismatches may cause retrieval components optimized for traditional metrics to harm end-to-end RAG performance.Source: §1 · pp. 1–2
  • The annotation assigns continuous utility scores that capture both relevant passages’ beneficial contributions and irrelevant passages’ distracting effects on LLM generation.It reframes passage assessment around contribution to end-to-end system performance rather than human-oriented relevance alone.Source: §1 · p. 2
  • Ranking passages by utility and selecting the top-k yields significantly better LLM accuracy than selection based on traditional relevance annotations.The paper presents this as empirical evidence that utility-aware rankings improve RAG performance.Source: §1 · p. 2
  • Up to 36%, UDCG shows stronger correlation with end-to-end RAG accuracy than traditional IR metrics across five benchmarks and six LLMs.The metric combines continuous passage utility with an LLM-oriented positional discount.Source: §1 · p. 2

2 Related Work

Prior RAG research describes how distracting passages and prompt position affect LLM responses, while retrieval evaluation often relies on human-oriented metrics or costly end-to-end assessment. This paper combines LLM-oriented relevance with distraction in an IR metric for RAG.

2.1 Retrieval Augmented Generation

  • In RAG, semantically related passages that lack the answer can distract LLMs, while passage position can shape how they use context.Prior work defines a distracting-effect score using the probability that an LLM does not abstain when given an irrelevant passage and question.Source: §2.1 · p. 3

2.2 IR Evaluation for RAG

  • End-to-end RAG evaluation can obscure retrieval’s contribution and require repeated generation, while practitioners commonly assess retrieved content with traditional IR metrics.These metrics were designed around human relevance and lower-ranked documents receiving less weight.Source: §2.2 · p. 3
  • Prior RAG-oriented approaches include eRAG’s single-document response-quality scores and Semantic Perplexity’s answer-confidence variation; this paper combines LLM-oriented relevance with distraction.The proposed combination forms a holistic IR metric for the RAG setting.Source: §2.2 · p. 3

3 Flaws of Traditional IR Metrics in RAG

Traditional IR metrics can misrepresent RAG performance because LLMs use retrieved context differently from human searchers and can be harmed by distracting passages. Experiments illustrate mismatches in both distractor effects and positional behavior.

3 Flaws of Traditional IR Metrics in RAG

  • LLM utility does not necessarily match human-centric relevance, and LLMs lack humans’ intuitive filtering of irrelevant documents.The cited evidence notes that irrelevant passages from high-performing retrieval systems can be more distracting and harmful than those from weaker systems.Source: §3 · pp. 3–4
  • Up to 9 accuracy points separate scenarios with weak versus hard distractors, although traditional IR metrics score them equally.The comparison uses answer accuracy on NQ3 across several LLMs; all differences are statistically significant.Source: Table 1 · p. 4, §3 · p. 4
  • Because LLMs process the entire prompt holistically and show positional bias, human-style rank discounts may not reflect how they use retrieved passages.Traditional metrics reward relevant documents less as their rank falls, unlike the non-monotonic behavior illustrated for LLM accuracy.Source: §3 · p. 4
  • As a relevant passage shifts among irrelevant passages, LLM accuracy exhibits a U-shaped pattern while nDCG, MAP, and MRR decrease monotonically.The figure compares traditional metrics with LLM answer accuracy as the relevant passage changes position.Source: Figure 1 · p. 4, §3 · p. 4

4 An LLM-oriented Annotation Schema

The annotation schema defines passage utility for an LLM through its probability of not abstaining, distinguishing positive utility from distraction. It also relies on relevance labels and assumes relevant passages do not contain distracting information.

4 An LLM-oriented Annotation Schema

  • Passage utility u(q, p) is the probability that an LLM does not abstain when given question q and passage p.For a relevant passage, this is interpreted as the likelihood that the LLM infers the answer; for an irrelevant passage, it is negative utility reflecting distraction.Source: §4 · p. 4
  • Relevance is labeled 1 when a passage contains the answer and -1 otherwise, using human annotation or an LLM-as-judge approach.The relevance label determines whether non-abstention is treated as positive utility or distracting effect.Source: §4 · p. 4
  • The annotation assumes a lone relevant passage contains no distracting information, so the LLM answers correctly or abstains rather than answering incorrectly.The authors report never observing an incorrect answer in preliminary experiments under this assumption.Source: §4 · p. 4
  • Scoring each passage separately is described as cheaper than end-to-end evaluation of the full retrieved list.With average passage length w, list size k, and answer length g, the stated costs are O(gk^2w^2) for full-list generation and O(kw^2) for passage utilities.Source: §4 · p. 5

5 Defining a RAG-tailored Metric

UDCG scores a retrieved context using passage utilities, accounting for relevance, distraction, and, in its learnable version, LLM positional bias. A training-free variant trades off that positional modeling for simpler applicability.

5 Defining a RAG-tailored Metric

  • UDCGθ scores the full retrieved context using passage utilities and accounts for relevance, distraction, and LLM positional bias.Its sigmoid-bounded score uses separate positional weights for positive and negative utility.Source: §5 · p. 5
  • UDCGθ learns 2k positional parameters from utility features to predict end-to-end answer accuracy.The weights α_i and β_i apply to positive and negative utility, respectively.Source: §5 · p. 5
  • UDCGθ requires training for each retrieved-list size k, so each context size needs its own dedicated training process.The paper presents this dependence as a drawback motivating a training-free version.Source: §5 · p. 5
  • The approach assumes relevance annotations are available; otherwise, the paper says they can be computed with an LLM-as-judge at similar cost to estimating abstention probability.Source: §5 · p. 5
  • The training-free UDCG combines average passage relevance and average distraction, omitting LLM positional bias.A single hyperparameter γ balances the terms; γ = 1/3 provides good results across multiple LLMs and datasets in the experiments.Source: §5 · p. 5

6 Experimental Setting

The experiments span five question-answering datasets and six LLMs, with both general-knowledge and specialized-domain coverage. Evaluation uses passage-relevance judgments and answer scoring under an abstention setting.

6.1 Datasets and Models

  • The evaluation uses five question-answering datasets covering general knowledge and biomedical questions, with Wikipedia-based corpora or PubMed as the source.The datasets are PopQA, NoMIRACL, Natural Questions, TriviaQA, and BioASQ; NoMIRACL supplies pre-retrieved passages and relevance annotations.Source: §6.1 · p. 5
  • Answer generation uses six instruction-tuned LLMs from different model sizes and families, with greedy decoding for reproducibility.The models are Llama-3.2-3B, Llama-3.1-8B, Llama-3.3-70B, Mistral-7B, Gemma-3-4B, and Qwen-2.5-7B.Source: §6.1 · p. 5

6.2 Evaluation Strategy

  • Passage relevance is judged with Claude 3.7 Sonnet using the query and reference answer; a passage counts as relevant only if it contains information needed to answer.NoMIRACL uses relevance annotations provided by the dataset.Source: §6.2 · p. 6
  • LLMs are instructed to abstain when context is insufficient, and an answer is correct only when context includes a relevant passage and the answer matches the reference.Gemini 2.0 Flash evaluates semantic equivalence of the response.Source: §6.2 · p. 6

7 Experimental Results

Across four experiments, utility-based passage rankings improved RAG answer outcomes, while UDCG and UDCGθ correlated more strongly with end-to-end answer accuracy than baseline metrics. UDCG also maintained stable correlations as context size varied from one to ten passages.

7.1 Ideal Rankings of Different Annotations

  • 87% of queries had a relevant passage among the top 25; binary relevance and utility selection kept relevant passages ahead, whereas eRAG’s selected contexts contained a relevant passage in only 71%–77% of cases.The eRAG range varies by LLM.Source: §7.1 · p. 6, Table 2 · p. 6
  • Utility-based selection yielded up to 2% more correct answers than binary labels and halved wrong answers when no relevant passage was present.The reported answer outcomes are averaged across tested datasets, and the differences were statistically significant.Source: §7.1 · pp. 6–7, Table 2 · p. 6

7.2 Correlation between IR Metrics and Answer Accuracy

  • UDCGθ achieved 36% (+10pts) higher Spearman correlation than NDCG, averaged across considered LLMs and datasets; UDCGθ and UDCG led across all tested settings.UDCGθ, trained only on NQ, remained competitive across the other benchmarks.Source: §7.2 · p. 7, Table 3 · p. 7

7.3 Ablation Study for UDCG

  • Removing distraction features reduced UDCGθ correlation by 3.4%–8%; binary relevance caused drops up to 17%, while removing distraction effects reduced UDCG correlation by 4% on average.The ablation compares full models with relevance-only variants.Source: §7.3 · p. 8, Table 4 · p. 8

7.4 Sensitivity to Context Length

  • Across k=1–10, UDCG correlations stayed between 0.554 and 0.704 and beat all baselines; NDCG and Precision variability was 1.6 and 1.38 times UDCG’s, respectively.The context-length experiment used Qwen 7B on NQ.Source: §7.4 · p. 8, Figure 2 · p. 8

8 Conclusions

The paper presents UDCG as a RAG-oriented metric built on positive or negative passage utility, and reports stronger prediction of end-to-end performance than traditional and existing RAG-oriented metrics. It also notes that removing positional discounting yields nearly identical performance, while UDCGθ is less practical when context sizes vary.

8 Conclusions

  • UDCG replaces human-centric relevance with positive or negative passage utility and uses an LLM-oriented positional discount.The discount is designed to optimize correlation with end-to-end answer accuracy.Source: §8 · p. 8
  • UDCGθ requires training for each retrieved-passage count k, making it less practical for dynamic context sizes.Source: §8 · p. 8
  • By up to 36%, both metrics outperform traditional IR metrics and existing RAG-oriented approaches in predicting end-to-end system performance.The experiments cover five QA datasets and six LLMs.Source: §8 · p. 9

Limitations

The evaluation is limited to English-language question answering with specific, verifiable answers, and the annotation schema requires access to the LLM’s output probabilities.

Limitations

  • The experiments focus on question answering with specific, verifiable answers; applicability to multihop QA and fact verification remains unexplored.Source: Limitations · p. 9
  • Computing abstention likelihood requires LLM output probabilities, excluding blackbox commercial APIs that return only generated text.Source: Limitations · p. 9
  • The evaluation uses only English datasets, leaving multilingual and cross-lingual RAG effectiveness uninvestigated.Source: Limitations · p. 9

A Traditional IR Metrics

Traditional IR metrics center on document relevance and differ in how they measure the presence and rank positions of relevant results.

A Traditional IR Metrics

  • Precision measures the relevant fraction of retrieved documents, while recall measures the fraction of relevant documents retrieved.The passage notes recall is no longer meaningful for modern web-scale IR because queries may have thousands of relevant documents.Source: A Traditional IR Metrics · p. 12
  • MAP averages precision at relevant-document positions across queries, rewarding systems that rank relevant results higher.Source: A Traditional IR Metrics · p. 12
  • MRR averages the reciprocal rank of the first relevant document, making it suitable for navigational queries seeking one answer.Source: A Traditional IR Metrics · p. 12
  • nDCG uses graded relevance and logarithmically discounts lower-ranked documents, then normalizes DCG by ideal DCG.Source: A Traditional IR Metrics · p. 12
  • HITS checks whether at least one relevant passage appears among the top-k retrieved results.Source: A Traditional IR Metrics · p. 12

B Open-source Alternative for Relevance

A comparison of Claude 3.7 Sonnet and Llama-3.3-70B Instruct found substantial agreement on passage relevance judgments, supporting Llama as an open-source alternative.

B Open-source Alternative for Relevance

Loading 2510.21440v1…