Source-linked AI summary
Evaluating Retrieval Quality in Retrieval-Augmented Generation
Alireza Salemi, Hamed Zamani
TL;DR
RAG retrieval evaluation is computationally expensive, and conventional relevance labels correlate weakly with downstream system performance. eRAG individually applies the RAG system’s LLM to retrieved documents, converts downstream task performance into document relevance labels, and aggregates them with retrieval metrics. Across evaluated datasets, eRAG achieved higher downstream correlation and substantial computational savings than baseline and end-to-end approaches.
Problem
Traditional end-to-end RAG evaluation is computationally expensive, while relevance-label evaluation has limited alignment with downstream RAG performance.
Method
eRAG individually feeds each retrieved document to the RAG system’s LLM, evaluates its output against downstream ground truth, and aggregates document labels with set-based or ranking metrics.
Results
0.168–0.494 absolute Kendall’s tau improvements were observed across evaluated datasets, and eRAG used up to 50 times less memory than end-to-end evaluation.
Takeaways & Limitations
eRAG’s document-level downstream annotations more closely track RAG performance while requiring less memory and inference time than end-to-end evaluation.
Abstract
from arXiv · showhide
Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is then evaluated based on the downstream task ground truth labels. In this manner, the downstream performance for each document serves as its relevance label. We employ various downstream task metrics to obtain document-level annotations and aggregate them using set-based or ranking metrics. Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods, with improvements in Kendall's $τ$ correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.
1 INTRODUCTION
RAG retrieval evaluation is limited by costly, opaque end-to-end assessment and weakly aligned relevance labels. The paper proposes eRAG, which evaluates each retrieved document through the RAG system’s own LLM and aggregates downstream task performance into retrieval-list scores.
- Evaluation challenges: End-to-end evaluation is resource-intensive and obscures which retrieved document contributed to the generated output.Processing long sequences requires GPUs with substantial memory, while interleaving retrieval results can require repeated pipeline recomputation.
- Evaluation challenges: Human or LLM-generated relevance judgments may not reflect whether a document helps the RAG system’s own LLM complete its task.Human annotation is costly, and documents judged relevant by preferences may not be useful to the downstream LLM.
- eRAG approach: eRAG feeds each retrieved document individually to the RAG LLM and uses downstream metrics such as accuracy, exact match, or ROUGE for document-level annotations.Set-based or ranking metrics then aggregate these annotations into one score for each retrieval result list.
- Results: 0.168–0.494 absolute Kendall’s tau improvement was achieved across evaluated datasets, with eRAG attaining the highest correlation with downstream RAG performance.The evaluation covered question answering, fact-checking, and dialogue generation on the KILT benchmark.
- Results: 50 times less memory was consumed by eRAG than by end-to-end evaluation.The paper also reports investigations of retrieval augmentation methods, retrieved-document quantity, and LLM size.
2 EVALUATING RETRIEVERS IN RAG
Prior retrieval-evaluation methods use costly human judgments, weak document-ground-truth matching, or external LLM judges that may not match the downstream RAG model. eRAG instead uses the RAG system’s own LLM to score documents individually, then evaluates ranked retrieval lists with standard metrics.
- Existing relevance labels: Human relevance annotation is costly and may label documents useful to people but not useful to the RAG system’s LLM.This limits its practicality for evaluating all documents in a corpus.
- Existing relevance labels: Documents containing downstream ground-truth outputs provide weak labels that may fail for long-text generation or text classification.Downstream labels may not appear inside documents, and one document can be used in multiple ways.
- Existing relevance labels: LLM judges can remain mismatched with the RAG pipeline when the labeling model differs from the downstream LLM.Using the RAG system’s own LLM as the arbiter addresses this mismatch.
- eRAG approach: eRAG feeds each retrieved document individually to the RAG LLM and evaluates its generated answer against the expected downstream output.The resulting document-level labels can be used to evaluate retrieval rankings.
- Computational cost: O(lk d^2) computation replaces O(l k^2 d^2) end-to-end computation when k documents of average length d produce an output of length l.This follows from processing documents separately rather than as one combined ranked list.
- Ranking evaluation: Precision, Recall, MAP, MRR, NDCG, and Hit Rate aggregate document relevance scores into retrieval-list evaluation scores.For non-binary labels, precision averages relevance values while Hit Ratio uses the maximum value.
3 EXPERIMENTS
Experiments evaluate eRAG across knowledge-intensive tasks, retrieval settings, LLM sizes, augmentation methods, and computational costs. eRAG consistently correlates strongly with downstream RAG performance while reducing runtime and memory consumption relative to end-to-end evaluation.
- Experimental Setup: The experiments use NQ, TriviaQA, HotpotQA, FEVER, and Wizard of Wikipedia from the KILT benchmark, with validation data used because test labels are unavailable.The retrieval corpus is the KILT Wikipedia dump, preprocessed into passages.
- Main Findings: Using T5-small with FiD and 50 retrieved documents, eRAG achieves the highest correlation with downstream LLM performance across retrieval models and datasets.KILT Provenance and LLM-based document annotation show the lowest correlations among the compared approaches.
- Main Findings: As the number of retrieved documents increases, correlation declines for evaluation methods, but eRAG remains consistently more correlated with downstream LLM performance.The decline reflects that document-level metrics score relevance independently while the LLM uses information from the full document set.
- LLM Size: Across T5-small and T5-base, eRAG shows no significant correlation difference attributable to LLM size, supporting effectiveness across the tested model sizes.The comparison uses T5-small with 60M parameters and T5-base with 220M parameters.
- Retrieval Augmentation: eRAG is evaluated with both FiD and IPA using T5-small and 10 BM25-retrieved documents, with the document count constrained by IPA input-size limitations.FiD encodes documents individually before decoding jointly, whereas IPA appends retrieved results to the LLM input.
- Efficiency: 2.468 times faster is eRAG on average than end-to-end evaluation, with dataset-level speedups ranging from 1.232 to 3.252 times.The smallest speedup occurs for the long-text generation task on Wizard of Wikipedia.
- Efficiency: 30 to 48 times greater memory efficiency is achieved by eRAG in the document-level configuration, compared with 7 to 15 times in the query-level configuration.The experiments compare maximum memory requirements under separately defined batch-size configurations.
4 CONCLUSION
The paper introduces eRAG for evaluating retrieval models through per-document downstream performance, finding higher correlation and greater efficiency than end-to-end evaluation.
- eRAG uses each retrieved document individually with the LLM to generate document-level relevance labels from downstream-task performance.
- eRAG achieves significantly higher correlation with downstream LLM performance than the evaluated retrieval approaches.
- eRAG is more efficient than end-to-end evaluation in both memory consumption and inference time.