PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems

Robert Friel, Masha Belyi, Atindriyo Sanyal

arXiv:2407.11005v2cs.CLcs.AI

TL;DR

Production RAG systems require application-specific tuning, but evaluation lacks unified benchmarks and criteria. RAGBench combines a multi-domain dataset with the TRACe metric framework, and its fine-tuned DeBERTa performs competitively with billion-parameter LLM judges and commercial RAG evaluation systems.

  • Problem

    Production RAG systems require application-specific tuning across design choices, while evaluation datasets and criteria remain fragmented and cross-domain generalization is an open question.

    Source: §1 · pp. 1–2

  • Method

    RAGBench standardizes examples from 12 component datasets across varied RAG settings and uses TRACe’s four metrics with LLM-generated labels validated against human judgments.

    Source: §3.1 · p. 3, §6 · p. 9

  • Results

    A 400M-parameter DeBERTa fine-tuned on RAGBench outperforms few-shot LLM judges across numerous domains and task types and performs competitively with billion-parameter judges and commercial systems.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    TRACe offers a consistent approach to measuring RAG performance across dimensions, with the potential to provide practitioners granular, actionable insights.

    Source: §1 · p. 2, §6 · p. 9

    The gap between the best-performing RAG evaluator and ground truth remains large.

    Source: §6 · p. 10

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has become a standard architectural pattern for incorporating domain-specific knowledge into user-facing chat applications powered by Large Language Models (LLMs). RAG systems are characterized by (1) a document retriever that queries a domain-specific corpus for context information relevant to an input query, and (2) an LLM that generates a response based on the provided query and context. However, comprehensive evaluation of RAG systems remains a challenge due to the lack of unified evaluation criteria and annotated datasets. In response, we introduce RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k examples. It covers five unique industry-specific domains and various RAG task types. RAGBench examples are sourced from industry corpora such as user manuals, making it particularly relevant for industry applications. Further, we formalize the TRACe evaluation framework: a set of explainable and actionable RAG evaluation metrics applicable across all RAG domains. We release the labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench. RAGBench explainable labels facilitate holistic evaluation of RAG systems, enabling actionable feedback for continuous improvement of production applications. Thorough extensive benchmarking, we find that LLM-based RAG evaluation methods struggle to compete with a finetuned RoBERTa model on the RAG evaluation task. We identify areas where existing approaches fall short and propose the adoption of RAGBench with TRACe towards advancing the state of RAG evaluation systems.

1 Introduction

Production RAG systems need application-specific tuning across consequential design choices, while existing evaluation datasets and criteria are fragmented. RAGBench addresses this with a multi-domain benchmark and expanded metric suite; its fine-tuned model outperforms few-shot LLM judges across numerous domains and task types.

1 Introduction

  • Production RAG systems require application-specific tuning because retriever and generator choices affect quality and costs, and many configurations must be evaluated.Design variables include context format and length, retriever model, number of retrieved documents, and generation model.Source: §1 · pp. 1–2
  • RAG evaluation lacks unified benchmarks and criteria; existing datasets are often small, use disjoint labels, or rely on LLM-generated data and labels.Most studies use small in-domain datasets, leaving cross-domain generalization an open question.Source: §1 · p. 2
  • RAGBench combines context relevance and answer faithfulness with new context utilization and answer completeness metrics for more granular, actionable evaluation.The authors argue this metric suite better describes overall RAG system performance.Source: §1 · p. 2
  • A 400M-parameter DeBERTa-large fine-tuned on RAGBench outperforms few-shot LLM judges across numerous domains and task types.The authors present this comparison as motivation for using RAGBench to advance RAG evaluation models.Source: §1 · p. 2

2 Related Work

Prior work includes response-level datasets, hallucination-focused benchmarks, and automated evaluators using varied methods and metrics. RAGBench differentiates itself by supporting multidimensional RAG evaluation, including evaluation with a fine-tuned NLI model.

2 Related Work

  • Unlike response-level benchmarks using exact match or ROUGE, RAGBench evaluates components such as retriever relevance, response adherence, and completeness.Those ground-truth datasets are designed for end-to-end RAG system evaluation.Source: §2 · p. 2
  • Hallucination-focused benchmarks provide labels for detecting hallucinations but do not offer RAGBench’s granularity for understanding the system as a whole.The cited benchmarks include both RAG-specific and open-domain factuality datasets.Source: §2 · p. 2
  • Automated RAG evaluators include prompt-based LLM judges, fine-tuned NLI models, and heuristic methods, but established benchmarks for comparing them are lacking.The approaches differ in the metrics and evaluation methods they use.Source: §2 · p. 2
  • A fine-tuned NLI model trained on RAGBench outperforms LLM judges in hallucination and attribution detection and excels on the proposed RAG evaluation metrics.The paper frames this as extending NLI-based RAG evaluation beyond hallucination detection.Source: §2 · p. 3

3 RAGBench Construction

RAGBench standardizes diverse real-world RAG datasets and introduces TRACe metrics for evaluating retriever and generator quality. Its construction includes GPT-4 annotation, human-alignment checks, and a case study showing that RAG configuration choices correspond to changes in TRACe metrics.

3.1 Component Datasets

  • RAGBench standardizes 12 component datasets spanning five domains and varied RAG tasks, including multi-document reasoning, hybrid text-and-table numerical reasoning, and long legal-contract contexts.Questions come from crowd-sourced, expert-written, real user, and automatically generated sources; responses are generated with GPT-3.5 and Claude 3 Haiku except where source data already includes LLM responses, and CUAD uses only Claude 3 Haiku.Source: §3.1 · pp. 3–4
  • RAGBench totals 100k samples split into train, validation, and test sets, with no query overlap across splits from the same source.Source: §3.1 · p. 4

3.2 TRACe Evaluation Framework

  • TRACe measures Relevance, Utilization, Adherence, and Completeness; the metrics distinguish context retrieved, context used, response grounding, and relevant information incorporated.Adherence is evaluated at example level as whether all response parts are grounded, with supported response tokens also annotated for granular evaluation.Source: §3.2 · pp. 5–6
  • High Relevance and Utilization can still coincide with low Completeness when the generator uses irrelevant context to produce a low-quality response.Source: §3.2 · pp. 5–6

3.3 RAGBench Statistics

  • Component datasets contain 1%–20% hallucinated responses; ExpertQA, CovidQA, and MS Marco have the highest reported shares, while CUAD, FinQA, and TAT-QA have about 1% each.Figure 3 presents normalized distributions of Relevance, Utilization, and Completeness labels.Source: §3.3 · p. 6, Figure 3 · p. 6

3.4 LLM annotator

  • GPT-4 is prompted to label Adherence, Relevance, and Utilization, while Completeness is derived from span-level Relevance and Utilization annotations.The annotation prompt asks for relevant and utilized document substrings and identifies response sentences supported by context; post-processing is used to improve annotation quality.Source: §3.4 · p. 6
  • GPT-4 annotations achieve 93% example-level and 95% span-level agreement with human judgments on DelucionQA, with high GPT-4–human correlation also reported for relevance and utilization.Relevance and utilization were checked on DelucionQA(40), a small annotated subset.Source: §3.4 · pp. 6–7, Table 2 · p. 7

3.5 RAG Case Study

  • GPT-4o with chain-of-thought prompting yields the least hallucination, while step-by-step prompts increase context utilization and response completeness; retriever choice affects context relevance.GPT-3.5 hallucinates more when prompted with chain-of-thought than with less detailed prompts.Source: Figure 4 · p. 7, §3.5 · p. 8

4 Experiments

The experiments compare prompted LLM evaluators with a fine-tuned DeBERTa judge and assess evaluation performance using task-specific metrics.

4.1 LLM Judge

  • The LLM-judge baselines are zero-shot GPT-3.5, RAGAS, and TruLens, with RAGAS using few-shot prompts.RAGAS and TruLens evaluate Adherence and Context Relevance; GPT-3.5 is queried with the annotation prompt.Source: §4.1 · p. 8

4.2 Fine-tuned Judge

  • The fine-tuned DeBERTa-v3-Large judge uses separate prediction heads to estimate TRACe metrics in one forward pass and predicts spans at a 0.5 threshold.Context-token heads estimate Relevance and Utilization, while a response-token head estimates Adherence.Source: §4.2 · p. 8

4.3 Evaluation

  • Evaluation reports AUROC for response-level hallucination detection and RMSE for example-level Context Relevance and Utilization predictions.These metrics support comparison with less-granular RAG evaluation approaches.Source: §4.3 · pp. 8–9

5 Results

On RAGBench test splits, the fine-tuned DeBERTa model achieves competitive performance with billion-parameter LLM judges, while Context Relevance is harder to predict than Utilization.

5 Results

  • The comparison uses test splits of each RAGBench component dataset, against baseline LLM methods and a DeBERTa model trained on the full RAGBench train split.Source: §5 · p. 9
  • 0.64–0.86 AUROC for hallucination detection and 0.04–0.26 RMSE for Relevance and Utilization are reported for the RAGBench-trained DeBERTa model across domains and tasks.The model achieves competitive performance with billion-parameter LLM judges across numerous domain subsets.Source: §5 · p. 9
  • Relevance RMSE is generally higher than Utilization RMSE because relevance requires identifying context information needed to answer, beyond semantic relatedness.Utilization can be assessed through semantic comparison between context and response.Source: §5 · p. 9

6 Conclusion

The paper introduces RAGBench and TRACe for standardized, multidimensional RAG evaluation, with LLM-generated labels that correlate highly with human judgments. A fine-tuned DeBERTa model performs competitively with large LLM judges and commercial systems, though the gap to ground truth remains large.

6 Conclusion

  • TRACe defines four RAG evaluation metrics—uTilization, Relevance, Adherence, and Completeness—and its automated LLM-generated labels show high correlation with human judgments.The framework aims to standardize assessment across multiple dimensions of RAG system performance.Source: §6 · p. 9
  • RAGBench provides real-world RAG examples for training and benchmarking evaluation models, while TRACe offers a consistent framework for measuring system performance.The dataset and framework are intended to support evaluation across varied RAG dimensions.Source: §6 · p. 9
  • A 400M-parameter DeBERTa model fine-tuned on RAGBench performs competitively with billion-parameter LLM judges and commercial RAG evaluation systems, but the best evaluator remains far from ground truth.The authors motivate future work on more powerful evaluation models to narrow that gap.Source: §6 · pp. 9–10
  • The authors argue that standardized benchmarks and methods can provide more precise, actionable insights into RAG systems and facilitate their iterative improvement.The stated application scope includes real-world RAG applications.Source: §6 · p. 10

7 Appendix

The appendix provides links to RAGBench data and code, then documents the public datasets and transformations used to assemble its RAG examples. Its sources span manuals, technical support, finance, science, general question answering, and legal contracts.

7.1 RAGBench Code and Data

  • RAGBench data and inference and evaluation code are released through Hugging Face and GitHub.The dataset listing points readers to its model card and documentation.Source: §7.1 · p. 14

7.2 RAGBench Dataset Details

  • PubMedQA’s PQA-U subset is reframed as a RAG task by retrieving four abstract chunks per question and generating new LLM responses.The original responses and labels are not used.Source: §7.2 · p. 14
  • CovidQA-RAG, HotpotQA, and MS Marco supply research, multi-hop Wikipedia, and search-query contexts, respectively, with RAGBench generating responses for the resulting examples.HotpotQA contexts are reduced to four documents while retaining the answer document; MS Marco’s human answers are not used.Source: §7.2 · p. 14
  • CUAD contributes expert-authored legal-contract questions as single-contract contexts, without additional retrieval, and samples five questions per document.Responses are generated with an LLM in the RAG setting.Source: §7.2 · pp. 14–15
  • DelucionQA and EManual supply manual-based questions, with RAG contexts formed by retrieving up to three documents and LLMs generating responses.DelucionQA uses questions based on a Jeep Gladiator manual; EManual covers Samsung Smart TV and remote manuals.Source: §7.2 · p. 15
  • TechQA contributes real-world IBM forum questions and linked technical documents, subsampled to ten documents per question while retaining the answer document when applicable.The resulting context-question examples receive LLM-generated responses.Source: §7.2 · p. 15
  • FinQA and TAT-QA contribute financial question-answer data requiring numerical reasoning over report text and tables, with LLM responses generated for RAGBench.FinQA retains its original splits and receives two responses per context-query example; TAT-QA uses its full dataset.Source: §7.2 · p. 15
  • HAGRID and ExpertQA contribute existing LLM-generated responses alongside questions and relevant contexts, rather than receiving newly generated responses.HAGRID is based on MIRACL and uses Wikipedia contexts; ExpertQA contains expert-curated questions and passages across fields.Source: §7.2 · pp. 15–16

7.3 Response Generation Prompt

The response-generation prompt combines a question with its context documents and instructs the LLM to answer using those context pieces.

7.3 Response Generation Prompt

  • The prompt places context documents, separated by line breaks, alongside the question for response generation.The context and question are inserted into the template for each sample.Source: §7.3 · p. 16
  • The prompt is used to generate LLM responses for each RAGBench sample.Source: §7.3 · p. 16
  • The prompt directs the model to answer the question using the supplied context pieces.Source: §7.3 · p. 16

7.4 GPT Labeling Prompt

The GPT-4 labeling prompt supplies a question, documents, and a response, then requests structured judgments about document relevance, overall support, sentence-level support, and document use. Its output schema records these judgments and their supporting sentence keys.

7.4 GPT Labeling Prompt

  • The labeling prompt frames GPT-4’s task as assessing which documents help answer a question and whether each response sentence is supported by them.The input contains documents split into keyed sentences, the question, and a response also split into keyed sentences.Source: §7.4 · p. 16
  • The requested JSON schema includes relevance, overall-support, sentence-support, and utilized-sentence fields.It also specifies key types and a valid JSON response format.Source: §7.4 · pp. 16–18
  • The prompt asks for all document sentences relevant to answering the question, including useful sentences not used in the response.It instructs the judge to ignore the response for this relevance judgment and omit sentences whose removal would not affect answerability.Source: §7.4 · p. 17
  • The overall-support fields require a step-by-step explanation of response claims and a boolean judgment of whether the response as a whole is supported.The boolean reflects the conclusion of the explanation.Source: §7.4 · p. 17
  • For each response sentence, the prompt requests an explanation, supporting document-sentence keys, and a fully-supported boolean.It specifies conditions for empty support lists and distinguishes full support from partial or incomplete support.Source: §7.4 · pp. 17–18
  • The prompt separately records which document sentences were used to construct the answer, including implicitly used sentences.It omits sentences that could be removed without affecting the answer.Source: §7.4 · p. 18

7.5 Annotation Post-Processing Steps

The annotation pipeline strengthens Adherence labels by cross-checking response- and sentence-level judgments, then performs consistency and schema cleanup. The authors also report a qualitative benefit to Relevance and Utilization labels.

7.5 Annotation Post-Processing Steps

  • Detailed GPT-4-turbo annotations with explanations use chain-of-thought and redundancy to encourage label quality.Source: §7.5 · p. 18
  • Cross-checking response- and sentence-level Adherence labels triggers re-annotation when GPT-4 judgments conflict, with examples re-annotated up to three times.The authors treat partially supported sentences as hallucinations and use a sentence-level fully_supported annotation to resolve residual conflicts.Source: §7.5 · p. 18
  • The authors qualitatively observe that extra Adherence checks also improve the quality and stability of Relevance and Utilization labels.Source: §7.5 · p. 18
  • A final cleanup step removes or replaces off-schema keys that GPT-4-turbo injects into annotation JSON.Source: §7.5 · p. 18

7.6 Annotation Alignment with Human Judgements

The authors assess annotation alignment against human labels and simulated-system rankings, while a RAG case study uses prompts that constrain answers to provided context. The ranking analysis reports high agreement for GPT-4 annotations, with lower agreement for granular relevance scores.

7.6.1 Adherence Alignment with DelucionQA

  • After excluding examples with any “Neither” sentence, the analysis covers 421 examples and compares LLM Adherence annotations with human judgments.DelucionQA uses human labels for responses to queries about the 2023 Jeep Gladiator; Table 4 reports F1 and Accuracy on human-annotated subsets.Source: §7.6.1 · pp. 18–19, Table 4 · p. 19

7.6.2 Relevance and Utilization Alignment with DelucionQA

  • A random 40-example DelucionQA test subset receives Relevance and Utilization labels for comparison with human annotations using sentence-level F1 and overall accuracy.Table 4 reports F1 and Accuracy for human-annotated subsets.Source: §7.6.2 · p. 19, Table 4 · p. 19

7.6.3 Rank-based Alignment for Adherence and Relevance

  • GPT-4 annotations achieve Kendall’s τ from 0.78 to 1 against simulated-system rankings, while granular example-level Relevance scores produce lower agreement.The authors attribute the difference to granular scores capturing differences in retrieved context length; Table 5 reports the ranking comparison.Source: §7.6.3 · p. 19, Table 5 · p. 19

7.7 RAG Case Study

  • The case-study prompts require context-based answers and an explicit missing-information response; one also requests step-by-step reasoning and document quotes.Table 6 reports comprehensive RAG case-study results, but the supplied passage gives no result values.Source: §7.7 · p. 20

7.8 DeBERTa model training

The model-training setup specifies the hardware, training duration, learning rates, warmup, and decay schedule.

7.8 DeBERTa model training

  • Training uses a Google Cloud Platform A-100 GPU instance.Source: §7.8 · p. 20
  • The model is trained for 3 epochs.Source: §7.8 · p. 20
  • The stated initial learning rates are 5−6 for base-model layers and 2−5 for the heads, with warmup and linear decay.Source: §7.8 · p. 20
Loading 2407.11005v2…