Source-linked AI summary
Ragas: Automated Evaluation of Retrieval Augmented Generation
Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert
TL;DR
RAG evaluation is difficult because systems must be assessed across retrieval and generation qualities, often without human annotations or reference answers. Ragas addresses this gap with reference-free LLM-based metrics for faithfulness, answer relevance, and context relevance, evaluated using the WikiEval dataset. On WikiEval, its predictions were closely aligned with human judgments, especially for faithfulness and answer relevance.
Problem
RAG systems require evaluation of grounded generation, question-focused answers, and focused retrieved context, but human annotations and reference answers are often unavailable.
Method
Ragas uses automated LLM-based metrics to measure faithfulness, answer relevance, and context relevance without relying on ground-truth answers.
Results
Ragas predictions were closely aligned with human judgments on WikiEval, especially for faithfulness and answer relevance.
Takeaways & Limitations
Ragas provides developers with reference-free evaluation insights for RAG systems through an easily integrated framework.
Takeaways & Limitations
The WikiEval pages were selected for events since the start of 2022, beyond the reported training cutoff of the experimental model.
Abstract
from arXiv · showhide
We introduce Ragas (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAG systems are composed of a retrieval and an LLM based generation module, and provide LLMs with knowledge from a reference textual database, which enables them to act as a natural language layer between a user and textual databases, reducing the risk of hallucinations. Evaluating RAG architectures is, however, challenging because there are several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages in a faithful way, or the quality of the generation itself. With Ragas, we put forward a suite of metrics which can be used to evaluate these different dimensions \textit{without having to rely on ground truth human annotations}. We posit that such a framework can crucially contribute to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.
1 Introduction
RAG addresses gaps in LMs’ memorization by retrieving relevant corpus passages and providing them to the model, but evaluating such systems remains difficult. Ragas is introduced as an automated framework for assessing retrieval-augmented generation without requiring reference answers.
- RAG supplements language models by retrieving relevant passages from a corpus and feeding them to the model with the original question.
- RAG performance depends on the retrieval model, corpus, language model, and prompt formulation, making automated evaluation important.
- Perplexity-based evaluation may not predict downstream performance and requires access to language-model probabilities unavailable for some closed models.
- Ragas is a framework for automated assessment of retrieval-augmented generation systems when reference answers may be unavailable.
- Ragas estimates proxies for correctness and retrieved-passage usefulness and integrates with llama-index and Langchain workflows.
2 Related Work
Prior work evaluates factuality and text quality using prompting, token probabilities, sampling stability, or reference-answer comparisons. These approaches can be constrained by weak hallucination detection, inaccessible probabilities, or the need for references.
- Few-shot prompting has been proposed for factuality prediction, but recent analyses find that standard prompting struggles to detect hallucinations.
- Token-probability methods estimate factuality from the model’s confidence in generated text or converted multiple-choice judgments.
- SelfCheckGPT evaluates factuality for models without token-probability access by sampling multiple answers and comparing their semantic stability.
- GPTScore uses prompts specifying an evaluation aspect and scores generated passages using average token probabilities from an autoregressive language model.
- BERTScore, MoverScore, and BARTScore commonly compare generated answers with one or more reference answers using contextual representations or model scores.
3 Evaluation Strategies
Ragas evaluates RAG systems without human annotations or reference answers through three automated quality dimensions: faithfulness, answer relevance, and context relevance. LLM prompts operationalize each dimension using statement verification, question reconstruction, or relevant-sentence extraction.
- Ragas focuses on faithfulness, answer relevance, and context relevance as reference-free quality dimensions for RAG evaluation.Faithfulness concerns grounding in context; answer relevance concerns addressing the question; context relevance concerns focused retrieval.
- Faithfulness: Faithfulness decomposes an answer into statements and uses an LLM to verify whether each statement can be inferred from the retrieved context.
- Faithfulness: The faithfulness score F is the fraction of answer statements judged supported by the context.F = |V| / |S|, where |V| counts supported statements and |S| counts total statements.
- Answer relevance: Answer relevance generates potential questions from the answer, embeds them, and compares each with the original question using cosine similarity.
- Context relevance: Context relevance extracts context sentences crucial for answering the question and divides their number by the total number of context sentences.The metric penalizes redundant information in retrieved context.
4 The WikiEval Dataset
WikiEval was created to test Ragas against human judgments of faithfulness, answer relevance, and context relevance. It uses recent Wikipedia pages, generated questions and answers, deliberately varied answer or context quality, and annotator comparisons.
- WikiEval contains question-context-answer examples built from 50 Wikipedia pages covering events since the start of 2022.Pages with recent edits were prioritized.
- Questions were generated from introductory Wikipedia sections and constrained to be answerable, moderately difficult, reasonable, and based on non-trivial information.
- ChatGPT generated answers using the corresponding introductory sections as context for the evaluation examples.
- Human annotators agreed around 95% of the time for faithfulness and context relevance and around 90% for answer relevance.Disagreements were resolved through discussion.
- The dataset varied faithfulness by comparing contextual and no-context answers, answer relevance by comparing complete and incomplete answers, and context relevance by adding related but less relevant sentences.
5 Experiments
The experiment compares Ragas metrics with GPT Score and GPT Ranking against human judgments on faithfulness, answer relevance, and context relevance. Ragas aligns more closely overall, especially for faithfulness, while context relevance remains difficult.
- Ragas predictions are much closer to human judgments than the GPT Score and GPT Ranking baselines.The comparison measures how often model preferences match annotator preferences.
- Ragas is generally highly accurate for faithfulness.Faithfulness evaluates whether answers are grounded in the provided context.
- Ragas shows lower agreement for answer relevance because candidate-answer differences are often subtle.The GPT Ranking baseline asks ChatGPT to rank answers according to answer relevance.
- Context relevance is the hardest quality dimension for Ragas to evaluate.ChatGPT often struggles to select crucial sentences, particularly in longer contexts.
6 Conclusions
The paper argues for reference-free RAG evaluation covering faithfulness, answer relevance, and context relevance, and introduces WikiEval and Ragas to support it. On WikiEval, Ragas predictions closely align with human predictions, especially for faithfulness and answer relevance.
- Reference-free RAG evaluation should assess faithfulness, answer relevance, and context relevance.These dimensions respectively concern grounding in retrieved context, addressing the question, and keeping retrieved context focused.
- WikiEval provides human judgments to support development of a framework evaluating these three aspects.
- Ragas implements the three quality aspects and is designed to provide developers with insights without ground truth.The framework is described as easy to use.
- Ragas predictions closely align with human predictions, especially for faithfulness and answer relevance.
A Examples from WikiEval
The WikiEval examples illustrate paired high- and low-quality cases for faithfulness, answer relevance, and context relevance. Each table focuses on one of these evaluation dimensions.
- Table 2 presents WikiEval answers with high and low faithfulness.
- Table 3 presents WikiEval answers with high and low answer relevance.
- Table 4 presents WikiEval examples with high and low context relevance.