PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, Zheng Zhang

arXiv:2408.08067v2cs.CLcs.AI

TL;DR

RAG evaluation remains difficult because systems combine retrieval and generation, long responses require fine-grained assessment, and metric reliability matters. RAGChecker applies claim-level diagnostics to both modules; its evaluation found stronger human-judgment correlations than other metrics and surfaced trade-offs across RAG systems.

  • Problem

    RAG evaluation must address modular system behavior, fine distinctions in long responses, and whether metrics reliably align with human judgments.

    Source: §1 · p. 1

  • Method

    RAGChecker uses claim-level entailment checking to evaluate overall response quality and diagnose retriever and generator behavior.

    Source: §1 · p. 2

  • Results

    RAGChecker correlated significantly better with human judgments than other evaluation metrics and revealed design trade-offs across eight RAG systems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its diagnostic metrics provide signals that can guide improvements to RAG systems and understanding of their component behaviors.

    Source: §5 · p. 9

    Model predictions still show a clear gap from human annotators’ judgments.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Despite Retrieval-Augmented Generation (RAG) showing promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems. This work has been open sourced at https://github.com/amazon-science/RAGChecker.

1 Introduction

RAG evaluation must address modular error sources, long-form responses, and whether metrics align with human judgment. RAGChecker uses claim-level diagnostics for retrieval and generation, validates them against human judgments, and applies them to eight systems across ten domains.

1 Introduction

  • Existing evaluation is challenging because RAG combines retriever and generator modules, while common metrics can miss semantic detail in long responses and may not align with human judgments.Source: §1 · pp. 1–2
  • RAGChecker evaluates retrieval and generation through claim-level checking and provides diagnostic metrics intended to identify sources of errors.Its metric suite includes overall, retriever, and generator measures.Source: §1 · p. 2
  • Meta-evaluation found RAGChecker significantly better correlated with human judgments than other evaluation metrics.Source: §1 · p. 2
  • Experiments on eight RAG systems across ten domains revealed trade-offs between retrieval improvement and noise introduction, as well as faithful open-source models’ tendency to trust context blindly.Source: §1 · p. 2

2 Related Work

RAG systems use external knowledge to support responses across tasks and applications. Prior evaluation work studies generator capabilities or end-to-end quality, but many approaches rely on manually constructed tests, simple accuracy, or measures ill-suited to complex long answers.

2.1 Retrieval Augmented Generation

  • RAG uses external knowledge to address outdated information and hallucination concerns, with precision and reliability especially important in legal, medical, and finance settings.Source: §2.1 · p. 2
  • RAG has been applied to open-domain question answering, code generation, and dialogue, and appears in products such as Bing Search and Langchain.Source: §2.1 · p. 2

2.2 Evaluation of RAG

  • Generator-focused evaluations test capabilities such as noise robustness and factuality, but manually constructed datasets do not reveal how generation interacts with different retrieval behaviors.Source: §2.2 · p. 3
  • End-to-end frameworks such as TruLens, RAGAS, and ARES decompose quality into aspects including context relevance, groundedness, and answer relevance.Source: §2.2 · p. 3
  • Several RAG benchmarks use human evaluation or textual accuracy, which the paper says cannot handle complex scenarios requiring long-answer evaluation.Source: §2.2 · p. 3

3 RAGCHECKER Framework

RAGChecker treats a RAG system as a retriever followed by a generator and supplies fine-grained metrics for users comparing systems and developers diagnosing errors. It evaluates inputs and responses through claim extraction and entailment, then measures overall answer quality and component-specific retrieval and generation behaviors.

3 RAGCHECKER Framework

  • RAGChecker frames RAG as a retriever and generator operating over a query and documents, and designs its framework for users ranking systems and developers diagnosing errors.The framework combines a richly annotated benchmark with fine-grained metrics.Source: §3 · p. 3

3.1 Inputs to RAGCHECKER

  • Each benchmark sample contains a query, token-chunked documents, and a complete, correct ground-truth answer.Source: §3.1 · p. 4

3.2 Fine-grained Evaluation with Claim Entailment

  • The evaluation decomposes texts into claims and uses entailment checking to determine whether a claim is supported by a reference text.Source: §3.2 · p. 4

3.3 RAGCHECKER Metrics

  • The metric suite provides an overall score for comparing systems and separate retriever and generator measures for diagnosing error sources.Source: §3.3 · p. 4, Figure 1 · p. 4

3.3.1 Overall Metrics

  • Overall response quality uses claim-level precision and recall against the ground-truth answer, whose harmonic average is the F1 score.Source: §3.3.1 · p. 5

3.3.2 Retriever Metrics

  • Retriever claim recall measures ground-truth claims covered by retrieved chunks, while context precision counts relevant chunks; chunking can limit attainable claim-level precision below 100%.Source: §3.3.2 · p. 5

3.3.3 Generator Metrics

  • Generator metrics assess use of retrieved relevant information through context utilization and faithfulness to the provided context.Source: §3.3.3 · pp. 5–6
  • Generator diagnostics distinguish incorrect claims tied to relevant noise, irrelevant noise, or neither, and track correct claims based on self-knowledge rather than retrieved chunks.Source: §3.3.3 · p. 5

4 Experiments

The experiments assess RAGChecker’s human alignment and diagnostic metrics across a multi-domain benchmark, RAG systems, and tuning choices. E5-Mistral_GPT-4 performs best overall, while the analyses reveal trade-offs involving retrieval coverage, context use, and noise.

4.1 Experimental Setup

  • The evaluation uses 8 customized systems combining two retrievers with four generators and a 4,162-query benchmark spanning 10 domains.The benchmark repurposes public datasets and converts some short answers to long-form answers.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Meta Evaluation

  • RAGChecker has the strongest correlation with human preferences across correctness, completeness, and overall assessment among the evaluated metrics.The meta-evaluation uses 280 pairs of responses, labeled by two annotators for all three aspects.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 Main Results

  • E5-Mistral_GPT-4 is the best-performing system across the averaged results for 8 RAG systems and 10 datasets.The table reports overall precision, recall, and F1 alongside retriever and generator diagnostics.Source: Table 3 · p. 8, §4.3 · p. 8
  • E5-Mistral improves overall Precision, Recall, and F1 over BM25 with each generator; Llama3-70B also outperforms Llama3-8B on every generator metric.The generator comparison includes context utilization, noise sensitivity, and hallucination.Source: §4.3 · p. 8
  • Context utilization correlates more strongly with overall F1 than other generator metrics and stays relatively stable across retrievers; E5-Mistral also accompanies higher faithfulness and lower hallucination.The analyses report reduced self-knowledge alongside the higher faithfulness.Source: §4.3 · p. 8
  • Higher retriever claim recall comes with greater generator noise sensitivity; systems trust relevant chunks more than irrelevant ones, and open-source models distinguish accurate information from noise less well than GPT-4.The authors describe this as chunk-level faithfulness and note that open-source models tend to trust context blindly, especially as retrieval improves.Source: §4.3 · pp. 8–9

4.4 Diagnosis on RAG Settings for Improvements

  • F1 rose 51.7→53.4 with k 5→20 and 52.6→53.4 with size 150→300; prompts raised faithfulness but also noise sensitivity, while overlap minimally affected generation.The chunk increases also raised claim recall and faithfulness; prompt changes raised context utilization as well as noise sensitivity.Source: §4.4 · p. 9

Suggestions to RAG Builders

  • Retriever and chunk settings trade retrieval gains against cost and noise; prompt priorities should target the desired balance among context utilization, noise sensitivity, and faithfulness.The authors note that retrieval gains saturate as the total relevant information is fixed, so chunk counts and sizes need not be too large.Source: Suggestions to RAG Builders · p. 9

5 Conclusion

RAGCHECKER’s evaluation of eight RAG systems yielded insights into retriever and generator behavior and design trade-offs, supported by human assessments. The appendices describe a benchmark curated from existing datasets, its answer-generation process, and corpus downsampling.

5 Conclusion

  • Human assessments showed strong correlation with RAGCHECKER’s metrics, and evaluation of eight systems surfaced component behaviors and design trade-offs.The conclusion says these findings offer guidance for future RAG advances.Source: §5 · p. 9

A Details for Benchmark Curation

  • The benchmark repurposes open-domain QA datasets, removing overly simple questions and converting short answers into long-form answers for RAG evaluation.It draws on RobustQA, KIWI, ClapNQ, and NovelQA.Source: A Details for Benchmark Curation · p. 14

A.1 Data Sources

  • RobustQA domains use BioASQ expert QA pairs or GPT-4-expanded FiQA and LoTTE answers; the benchmark also uses ClapNQ, KIWI, and NovelQA data.NovelQA retrieval is restricted to the corresponding novel; KIWI contributes 71 QA pairs and 429 papers.Source: §A.1 · p. 14

A.2 Long-form Answer Generation

  • GPT-4 expands RobustQA and NovelQA short answers using their associated passages or evidence, and RefChecker filters out answers not entailed by those sources.Figure 2 shows the conversion prompt for Novel, Finance, Lifestyle, Recreation, Technology, Science, and Writing.Source: §A.2 · p. 14, Figure 2 · p. 15

A.3 Corpus Downsampling for Science and Biomedical Domains

  • Science and Biomedical corpora are downsampled to reduce dense-indexing costs, using BM25 candidates for Biomedical and BM25 plus dense-retriever candidates for Science.The candidate documents are combined with annotated relevant documents.Source: §A.3 · p. 15

A.4 License of The Datasets

The appendix lists dataset licenses and gives the complete metric formulations. The formulas use claim entailment against ground truth and retrieved chunks to classify retrieval chunks and define metrics.

A.4 License of The Datasets

  • Dataset licenses vary: RobustQA, ClapNQ, and NovelQA annotations use Apache-2.0; Finance and KIWI use CC-BY-SA-4.0; BioASQ uses CC BY 2.5; LoTTE licenses are unspecified.Source: §A.4 · p. 15

B The complete formula for all metrics

  • The metric formulations classify retrieved chunks as relevant when they entail at least one ground-truth claim, and otherwise as irrelevant.They use RefChecker to decompose text into claims and assess entailment against model responses, ground truth, or retrieved chunks.Source: B The complete formula for all metrics · pp. 15–16

C Details of Meta Evaluation

The meta-evaluation appendix reports a stronger correlation for RAGCHECKER than RAGAS Answer Similarity. It describes pairwise human response judgments, the annotation criteria, and GPT-4 critiques shown alongside responses.

C Details of Meta Evaluation

  • Ten annotators compare two responses per meta-evaluation instance; seven are in-house annotators and three are graduate students.The students were paid 15 USD per hour, for a total cost of 255 dollars.Source: C Details of Meta Evaluation · p. 16
  • Annotators choose among five preference levels and judge correctness, completeness, and overall assessment.Figure 3 presents the annotation interface and instructions; the guideline is designed to keep judgments agnostic to evaluation metrics.Source: C Details of Meta Evaluation · p. 16, Figure 3 · p. 17
  • Each response is displayed with GPT-4-generated critiques comparing it with the ground truth, and guideline examples include human-written explanations.Source: C Details of Meta Evaluation · p. 17
  • RAGCHECKER shows a stronger correlation than RAGAS Answer Similarity in the plotted prediction-score distributions.Figure 4 compares the metrics’ score distributions and mean lines.Source: C Details of Meta Evaluation · p. 17

D Details of the Experiment Setup

The experiment setup specifies the retriever and generator versions and the retrieval, chunking, and generation settings. Open-source model inference uses an AWS instance with eight NVIDIA A10G GPUs.

D Details of the Experiment Setup

  • The setup uses e5-mistral-7b-instruct for E5-Mistral retrieval and lists the generator versions for GPT-4, Llama3-8B, Llama3-70B, and Mixtral-8x7B.The GPT-4 version is gpt-4-turbo-2024-04-09.Source: D Details of the Experiment Setup · p. 17
  • Documents are split into 300-token chunks with 0.2 overlap, and each query uses the top 20 retrieved chunks as generation context.Inference for open-source models uses a g5.48xlarge instance with eight NVIDIA A10G GPUs; generation temperature is 0.0 and the maximum length for proprietary models is 2,048 tokens.Source: D Details of the Experiment Setup · p. 17

E Detailed Experiment Results

The detailed results are organized across benchmark datasets and meta-evaluation, with tables covering RAG systems and metric correlations and a figure comparing evaluator score distributions.

E Detailed Experiment Results

  • Evaluation results for different RAG systems are reported across the benchmark datasets in Tables 6–15.The supplied passages identify these tables but do not provide their result values.Source: E Detailed Experiment Results · p. 17
  • Table 5 reports correlations with human evaluations of correctness, completeness, and overall assessment.Table 4 summarizes the metrics included in the meta-evaluation.Source: Table 5 · p. 18, Table 4 · p. 18
  • Figure 4 compares RAGCHECKER and RAGAS Answer Similarity prediction-score distributions against human preference labels.Each point is an instance; colored areas show score distributions and dashed lines show means.Source: Figure 4 · p. 19
  • The main experiments use a default response-generation prompt for the eight RAG baseline systems.Source: Figure 5 · p. 20

F Diagnosis on RAG for Improvements

The diagnosis examines how retrieval settings and generation prompts affect RAG behavior, finding modest end-to-end gains from more context alongside trade-offs in noise and model-specific prompt responses.

F Diagnosis on RAG for Improvements

  • The tuning experiments vary k, chunk size, overlap ratio, and generation prompts across three baselines and the Writing, Finance, and KIWI domains.Source: F Diagnosis on RAG for Improvements · p. 20
  • Increasing k adds potentially less relevant context, while larger chunks add surrounding context to relevant facts; both improve retrieval claim recall, but their effects on context precision differ.Context precision decreases with larger k and increases with larger chunk sizes.Source: F Diagnosis on RAG for Improvements · p. 20
  • More context generally improves generator faithfulness but worsens context utilization as noise grows, raising relevant-noise sensitivity; the faithfulness trend is less pronounced for Llama3.Source: F Diagnosis on RAG for Improvements · p. 20
  • End-to-end RAG performance is slightly better with more context, mainly through improved recall, but gains saturate; with limited context, larger chunks and smaller k are preferred, especially for Finance and Writing.Source: F Diagnosis on RAG for Improvements · p. 20
  • The optimized prompt generally improves context utilization while worsening noise sensitivity, reflecting tension among prompt requirements.The prompt adds detailed requirements for faithfulness, context utilization, and lower noise sensitivity.Source: F Diagnosis on RAG for Improvements · p. 20
  • GPT-4 generally improves on faithfulness-related metrics with the optimized prompt, while Llama3 does not show the same behavior and has negligible overall F1 difference.The passage relates Llama3’s response to its already strong faithfulness.Source: F Diagnosis on RAG for Improvements · pp. 20–21
  • RAG builders can combine modular-metric performance with user preferences and generator capabilities when optimizing prompts.Source: F Diagnosis on RAG for Improvements · p. 21
  • Overlap adjustments generally improve context precision but do not consistently or clearly affect other performance metrics, so extensive overlap tuning may be unnecessary.The authors attribute the precision change to retrieving more chunks containing the same useful-information segments.Source: F Diagnosis on RAG for Improvements · p. 21

G Performance Validation of RefChecker with Llama3 Extractor and Checker

The authors test Llama3-70B-Instruct as both RefChecker’s extractor and checker, reporting stronger performance than the best purely open-source combinations in all three context settings.

G Performance Validation of RefChecker with Llama3 Extractor and Checker

H Limitations

The authors identify limits in retriever diagnostics, checking-result distinctions, and benchmark scope, and note possible societal costs and incentives associated with using RAGCHECKER.

H Limitations

  • Retriever diagnostics focus on ground-truth claim recall and retrieved-context precision, which may miss information density, diversity, and coherence.Source: H Limitations · p. 22
  • RAGCHECKER does not distinguish Neutral from Contradiction checking results, although they may affect final response quality differently.Source: H Limitations · p. 22
  • The benchmark repurposes existing text-only datasets and is limited to English queries and corpus, so it may not represent the full range of RAG tasks and languages.Source: H Limitations · p. 22
  • The authors suggest refining diagnostic metrics, accounting for checking-result differences, and expanding the benchmark for a more comprehensive assessment.Source: H Limitations · p. 23

I Potential Negative Societal Impacts

  • Metric-focused optimization could detract from broader utility and ethical considerations, while computational and financial requirements could disadvantage smaller organizations.The authors also caution that quantitative measures may neglect user experience and ethical implications.Source: I Potential Negative Societal Impacts · p. 23

Related papers

Loading 2408.08067v2…