PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Retrieval-Augmented Generation: A Survey

Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, Zhaofeng Liu

arXiv:2405.07437v2cs.CLcs.AI

TL;DR

RAG evaluation lacks a simple, stable framework because hybrid retrieval-generation systems rely on dynamic knowledge and varied tasks. The paper proposes Auepora, analyzes benchmark targets, datasets, and metrics, and concludes that current benchmarks remain incomplete and require broader, more realistic coverage.

  • Problem

    RAG evaluation is challenging because retrieval, generation, their interaction, and dynamic knowledge sources must be assessed together.

  • Method

    The paper introduces Auepora, a unified process for analyzing RAG evaluation targets, datasets, and metrics across existing benchmarks.

  • Results

    The survey systematically identifies evaluation challenges and analyzes current RAG benchmarks across retrieval, generation, and whole-system dimensions.

  • Takeaways & Limitations

    Effective RAG benchmarks should reflect the dynamic interplay between retrieval accuracy and generative quality and practical real-world requirements.

  • Takeaways & Limitations

    Retrieval metrics such as traditional Recall and Precision cannot fully capture RAG retrieval nuances, while current benchmarks often lack holistic coverage.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct A Unified Evaluation Process of RAG (Auepora) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accuracy, and faithfulness, within the current RAG benchmarks, encompassing the possible output and ground truth pairs. We then analyze the various datasets and metrics, discuss the limitations of current benchmarks, and suggest potential directions to advance the field of RAG benchmarks.

1 Introduction

RAG combines external information retrieval with language-model generation, but its hybrid architecture and dynamic knowledge sources make evaluation increasingly complex. The paper introduces Auepora to structure analysis of RAG evaluation targets, datasets, and metrics.

  • RAG integrates information retrieval with generative language models to improve the factual grounding of generated responses.Retrieval supplies external information intended to reduce plausible but factually incorrect outputs.
  • The RAG system contains retrieval and generation components, with retrieval organized into indexing and searching phases.Indexing prepares knowledge sources for retrieval, while searching fetches relevant information.
  • Evaluating RAG is difficult because systems combine multiple components, depend on dynamic external databases, and serve diverse downstream tasks.These conditions require metrics that capture both retrieval accuracy and generative quality.
  • Auepora analyzes RAG benchmarks through evaluation targets, datasets, and metrics across three corresponding phases.The framework is presented as a unified process for examining current benchmark design.
  • The paper classifies evaluation challenges across retrieval, generation, and whole-system assessment, then analyzes existing benchmarks and proposes future directions.Its benchmark analysis addresses strengths, limitations, and recommendations for RAG evaluation.

2 Challenges in Evaluating RAG Systems

RAG evaluation must assess retrieval, generation, and their interaction rather than treating components independently. Dynamic knowledge, subjective outputs, practical constraints, and incomplete benchmark coverage complicate comprehensive evaluation.

  • Hybrid RAG evaluation requires assessing retrieval, generation, and the whole system, each with distinct difficulties.The complete system also depends on how the components interact.
  • Retrieval: Retrieval evaluation must account for document precision, recall, relevance, temporal changes, source diversity, and misleading information.Traditional Recall and Precision do not fully capture the nuances of RAG retrieval systems.
  • Generation: Generation evaluation must measure faithfulness, accuracy, query relevance, and coherence, while subjective tasks make correctness variable.Creative generation and open-ended question answering lack a single stable notion of high-quality output.
  • RAG System as a Whole: Whole-system evaluation must measure the added value of retrieval, along with response latency and handling of ambiguous or complex queries.Component-level evaluation alone cannot fully explain overall RAG performance.
  • Conclusion: Current benchmarks often cover only selected RAG assessment aspects and lack comprehensive, holistic analysis.Evaluation also faces misleading or contradictory results from searches over dynamic databases and requires diverse real-world datasets.

3 A Unified Evaluation Process of RAG (Auepora)

Auepora organizes RAG evaluation around targets, datasets, and metrics, covering possible evaluable-output and ground-truth pairings across retrieval, generation, and additional requirements.

  • Auepora overview: Auepora structures benchmark analysis around three questions: what to evaluate, how to evaluate, and how to measure.These correspond to the Target, Dataset, and Metrics modules.
  • Evaluation target: The framework derives evaluation targets from pairings between evaluable outputs and ground truths, including output-to-output and output-to-ground-truth relationships.This pairing strategy is intended to cover all possible evaluation targets at the outset.
  • Retrieval: Retrieval targets include relevance between relevant documents and queries, plus accuracy comparing relevant documents with candidate documents.Relevance measures query matching, while accuracy assesses whether relevant documents receive higher scores than less relevant ones.
  • Generation: Generation targets include response relevance to the query, faithfulness to relevant documents, and correctness against a sample response.These relations evaluate query alignment, source consistency, and factual or contextual correctness respectively.
  • Additional requirements: Additional requirements span latency, diversity, noise robustness, negative rejection, counterfactual robustness, readability, toxicity, and perplexity.Some benchmarks also address broader application abilities, including creation, reading, updating, deletion, hallucination modification, and multi-document summarization.
  • Benchmark analysis: The survey compares evaluation frameworks, datasets, and metrics while noting that human evaluation and LLM judges support quality assessment when traditional ground truths are difficult to obtain.Its benchmark analysis distinguishes tools that specify targets from benchmarks focused on particular retrieval or generation aspects.

4 Discussion

Current RAG benchmarks must evaluate more than basic question answering because retrieval, generation, data sources, and practical constraints all vary. The survey highlights dataset specificity, judge inconsistency, and resource costs as central challenges.

  • Benchmark scope: Traditional QA benchmarks verify basic RAG capability but make retrieval effects difficult to isolate with strong LLMs.The survey calls for diverse, RAG-specific benchmarks covering multiple question types and answer properties.
  • Datasets: RAG benchmarks require tailored datasets for different targets, increasing evaluation effort and resource requirements.Dataset diversity spans news articles and structured databases, reflecting varied RAG use cases.
  • Metrics: LLM-based judging offers scalable evaluation but challenges human alignment, grading consistency, and standardization across use cases.Correctness, clarity, richness, examples, scales, and prompts can produce differing assessments.
  • Resources: Using LLMs for benchmark data generation and validation is resource-intensive, motivating methods that preserve validity with less data.Benchmarks must balance thorough evaluation against limited computational resources.

5 Conclusion

The survey presents Auepora as a structured way to analyze RAG evaluations through targets, datasets, and measures. It identifies gaps and points toward targeted, practical, and user-aligned benchmarks.

  • 5 Conclusion: Auepora structures RAG evaluation analysis around targets, datasets, and measures.The framework is intended to clarify evaluation complexities and benchmark effectiveness.
  • 5 Conclusion: The survey emphasizes targeted benchmarks that reflect the dynamic interplay between retrieval accuracy and generative quality.It also considers practical requirements for real-world applications.
  • 5 Conclusion: The paper identifies gaps in current methodologies and suggests future directions for more effective, user-aligned RAG benchmarks.

A.1 Retrieval Component

RAG retrieval components include sparse retrieval, dense retrieval, and web search engines, with relevant documents evaluated through scores or rankings. Dense methods are commonly paired with sparse methods, while web search expands coverage at higher cost and with less transparency.

  • Retrieval types: RAG retrieval comprises sparse retrieval, dense retrieval, and web search engine approaches.Evaluation commonly uses retrieved relevant documents with numerical scores or rankings.
  • Sparse retrieval: Sparse retrieval uses keyword matching and word frequency but may miss semantically relevant documents without keyword overlap.TF-IDF and BM25 are examples of sparse retrieval methods.
  • Dense retrieval: Dense retrieval uses deep models to capture semantic meaning and find relevant documents despite minimal keyword overlap.This supports complex queries requiring contextual understanding.
  • Web search: Web search can provide diverse and semantically relevant documents from the broader web, but search engines may be opaque and expensive at scale.
  • Hybrid retrieval: Dense retrieval is frequently combined with sparse retrieval to balance precision and breadth, while web search is increasingly used in benchmark assessment.

Indexing

Indexing organizes document collections for efficient retrieval, often after chunking documents into smaller units. Sparse and dense retrieval use different index structures suited to their representations.

  • Indexing: Indexing processes document collections and creates structures designed for fast, efficient search.Examples include HuggingFace datasets and Wikipedia pages.
  • Chunking: Chunking can improve retrieval by restricting similarity scoring to individual chunks rather than long articles.Semantic embeddings are less accurate for long articles, while desired content is often brief.
  • Sparse indexing: Sparse retrieval stores inverse document frequency values for terms to support rapid query lookup and scoring.
  • Dense indexing: Dense retrieval encodes documents as vectors and indexes them with approximate nearest-neighbor methods such as HNSW or IVF.Retrieval then uses predefined distance metrics to identify nearby items.

Search

The search component submits queries to web or local retrieval systems and returns relevant documents with scores. An optional reranker can refine this ranking using query–document relationships and other document features.

  • Queries are formatted for sparse or dense local retrieval methods or submitted through APIs for web search.The retrieval system returns relevant documents together with their scores.
  • An optional reranker refines retrieved-document rankings using semantic relationships, document importance, popularity, or custom measures.Rerankers are typically larger and more complex models that consider additional query and document features.

A.2 Generation Component

The generation component produces the LLM response and any structured or formatted output derived from that response.

  • The evaluable generation output is the LLM response together with its structured or formatted form.

Prompting

Prompting combines the query, retrieved information, and instructions before inference, while post-processing adapts the generated response to the task’s required output structure. Prompting strategies such as CoT, ToT, Self-Note, and RaR shape generation, and recent benchmarks evaluate some of these skills.

  • Prompting: Prompting merges the query, retrieval outcomes, and instructions into one input for the language model.The prompting stage is the interface between retrieval results and generation.
  • Prompting: Chain of Thought, Tree of Thought, and Self-Note are prompting tactics that shape model outputs for intricate tasks.The passages specifically characterize step-by-step prompting as useful for augmenting LLMs on complex tasks.
  • Prompting: Rephrase and Respond refines queries within prompts to improve comprehension and response performance across diverse tasks.
  • Prompting: Recent domain-specific RAG benchmarks evaluate prompting-engineering skills including Chain of Thought and Rephrase and Respond.
  • Inference: The final prompted string is passed to the LLM, which generates an answer to the initial query for downstream tasks.Inference operates on input derived from both retrieval and prompting stages.
  • Inference: Post-processing may format generated outputs or extract information required by the task.Examples include parsing or named entity recognition for extraction and formatting outputs for multichoice classification.
Loading 2405.07437v2…