PaperlayerView exact revision on arXiv

Source-linked AI summary

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems

Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia

arXiv:2311.09476v2cs.CLcs.AIcs.IR

TL;DR

RAG evaluation traditionally requires costly, domain-specific human annotations, while heuristic model-based evaluators may adapt poorly across evaluation contexts. ARES trains lightweight judges on synthetic data and calibrates their scores with a small human validation set through prediction-powered inference. Across eight knowledge-intensive datasets, it accurately scores and ranks RAG systems while using only a few hundred annotations.

  • Problem

    Traditional RAG evaluation requires substantial domain-specific annotations, while fixed heuristic prompts offer limited adaptability and no quality guarantees.

  • Method

    ARES generates synthetic query–answer data, fine-tunes separate lightweight judges for three RAG criteria, and uses PPI with a small human validation set to produce confidence intervals.

  • Results

    Across eight KILT, SuperGLUE, and AIS datasets, ARES accurately scores and ranks RAG systems, outperforming RAGAS and requiring 78% fewer annotations than the baseline approach.

  • Takeaways & Limitations

    ARES provides component-level RAG evaluation that supports accurate comparison of competitive systems and configurations with minimal human annotation.

  • Takeaways & Limitations

    ARES still relies on roughly 150–300 annotations, which may require specialized expertise in domains such as law, medicine, and finance.

Abstract

from arXiv · show

Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating RAG systems along the dimensions of context relevance, answer faithfulness, and answer relevance. By creating its own synthetic training data, ARES finetunes lightweight LM judges to assess the quality of individual RAG components. To mitigate potential prediction errors, ARES utilizes a small set of human-annotated datapoints for prediction-powered inference (PPI). Across eight different knowledge-intensive tasks in KILT, SuperGLUE, and AIS, ARES accurately evaluates RAG systems while using only a few hundred human annotations during evaluation. Furthermore, ARES judges remain effective across domain shifts, proving accurate even after changing the type of queries and/or documents used in the evaluated RAG systems. We make our code and datasets publicly available on Github.

1 Introduction

ARES addresses costly, expertise-intensive RAG evaluation by generating tailored lightweight judges for context relevance, answer faithfulness, and answer relevance. It combines synthetic training data with limited human validation to score, rank, and compare RAG systems.

  • Motivation and contribution: Traditional evaluation requires domain-specific annotations for questions, passages, and responses, imposing substantial expertise and cost.Production human preferences are another option but also demand considerable annotation effort.
  • Motivation and contribution: ARES targets RAG evaluation across context relevance, answer faithfulness, and answer relevance.It evaluates whether retrieved information is pertinent and whether generated answers are grounded and relevant.
  • Motivation and contribution: ARES generates tailored LLM judges from synthetic query–passage–answer data and fine-tunes them with contrastive learning.Separate judges assess the three RAG quality criteria.
  • Motivation and contribution: 150 annotated datapoints or more, plus five or more in-domain query-answer examples, provide ARES with its required human validation and prompting inputs.The validation set supports evaluation while few-shot examples guide synthetic data generation.
  • Results: ARES accurately scores RAG systems across six KILT and SuperGLUE datasets, outperforming existing automated evaluation approaches.It also supports comparison of competitive configurations and publicly releases its code and datasets.

2 Related Work

Related work includes RAG systems, general LLM-based evaluation, knowledge-intensive attribution resources, and specialized RAG evaluators. ARES is positioned against methods that require extra question structure or rely on inflexible heuristic prompts.

  • RAG and evaluation: RAG combines a retriever with a language model that uses retrieved passages to generate responses.It supports applications including question answering, fact-checking, and customer support.
  • RAG and evaluation: LLM-based evaluation enables rapid assessment in new settings but includes out-of-the-box judges and self-refinement approaches without statistical guarantees.The cited related methods include MT-Bench, Chatbot Arena, and AutoCalibrate.
  • RAG and evaluation: Knowledge-intensive evaluation research has examined attribution and factuality while producing guidelines and datasets for analyzing LLM pipelines.Examples include LongEval, Hagrid, and ALCE.
  • RAG-specific evaluators: EXAM requires queries with several associated sub-questions, whereas RAGAS relies on a handful of heuristic prompts with limited adaptability to new corpora.These design requirements distinguish both systems from ARES.

3 ARES

ARES builds evaluation judges from in-domain passages through synthetic data generation, filtering, and negative-example construction. The resulting query–passage–answer triples support judges tailored to the target corpus.

  • Inputs and synthetic data: ARES requires an in-domain passage set, roughly 150 annotated validation datapoints or more, and at least five in-domain query-answer examples.The examples prompt the language model during synthetic data generation.
  • Inputs and synthetic data: An LM generates synthetic queries and answers from corpus passages, producing positive and negative query–passage–answer examples.Few-shot mappings between passages, queries, and answers guide generation.
  • Inputs and synthetic data: ARES filters low-quality synthetic queries by testing whether each query retrieves its original passage as the top result.The system primarily uses FLAN-T5 XXL for generation but can use another high-quality model.
  • Negative construction: Negative examples use unrelated in-domain passages for context relevance and answers generated from other passages for faithfulness and relevance.These negatives are constructed to train judges against incorrect or mismatched triples.

1. Weak Negative Generation: For context rel-

ARES prepares separate judges for three RAG criteria, applies them to system outputs, and uses prediction-powered inference to estimate scores with confidence intervals. This combines large-scale unlabeled predictions with a small human validation set.

  • Preparing LLM judges: ARES trains separate binary-classifier judges for context relevance, answer faithfulness, and answer relevance.Each judge classifies concatenated query-document-answer triples as positive or negative.
  • Preparing LLM judges: ARES uses a human preference validation set to monitor judge improvement after each epoch and stops after three epochs without loss improvement.The validation set contains positive and negative examples for the three criteria.
  • Scoring systems: ARES averages judge labels on sampled in-domain outputs to calculate each RAG system’s context relevance, answer faithfulness, and answer relevance.The same scores support ranking competing systems and configurations.
  • Confidence intervals: Prediction-powered inference combines a small labeled set with predictions on many unlabeled datapoints to construct tighter confidence intervals.ARES uses a rectifier function to estimate judge errors and reports 95% confidence intervals.
  • Confidence intervals: ARES ranks systems using confidence-interval midpoints, enabling comparisons across RAG systems and configurations.This addresses the inaccuracy risk of relying only on synthetic-judge predictions or the costly alternative of extensively labeling each system’s outputs.

4 Experiments

The experiments test ARES across diverse retrieval, query, and answer settings using mock RAG systems with known performance, then evaluate scoring and ranking accuracy with Kendall’s τ.

  • Experimental setup: ARES uses deployable fine-tuned judges, while the in-context baseline uses OpenAI’s gpt-3.5-turbo-16k and RAGAS provides an automated comparison baseline.The fine-tuned judges rank systems without external APIs; similarity search uses FAISS and text-embedding-ada-002.
  • Datasets: The evaluation spans KILT datasets including Natural Questions, HotpotQA, FEVER, and Wizards of Wikipedia, covering single- and multi-passage reasoning, fact verification, and dialogue.These datasets use Wikipedia passages but vary in query and answer formats.
  • Datasets: SuperGLUE experiments use MultiRC and ReCoRD as open-domain retrieval tasks over domain-specific and news article passages.MultiRC covers seven passage domains, while ReCoRD uses CNN and Daily Mail news articles.
  • Evaluation design: Mock RAG systems are created from artificial query-passage-answer triples with empirically known positive and negative examples, enabling controlled ranking tests.Negative pairs and triples are sampled from the same or random Wikipedia documents.
  • Evaluation design: Nine dataset splits range from 70.0% to 90.0% success in 2.5-point increments, representing mock systems separated by small accuracy margins.The known success percentages establish the appropriate ranking for testing ARES scoring and ranking.
  • Metrics: Kendall’s τ measures correlation between correct and ARES rankings, with values greater than 0.9 considered successful.The metric is motivated by pairwise comparisons of model, retriever, and document-preprocessing choices.

5 Results & Analysis

ARES more accurately ranks and scores RAG systems than RAGAS, sampled annotations, and a GPT-3.5 judge while using substantially fewer human annotations. It also evaluates answer faithfulness on AIS and generalizes across moderate domain shifts, though performance degrades under more drastic shifts.

  • 5.1 ARES Ranking: ARES outperformed the GPT-3.5 judge by an average Kendall’s τ of 0.06 across context relevance and answer relevance.The fine-tuned judge more precisely distinguished configurations, while GPT-3.5 was more readily deployable but incurred querying costs.
  • 5.1 ARES Ranking: ARES ranked RAG systems more accurately than RAGAS across KILT and SuperGLUE, improving Kendall’s τ by 0.065 for context relevance and 0.132 for answer relevance on average.PPI further improved the ranking accuracy of the fine-tuned judge across all tested datasets.
  • 5.2 ARES Performance on AIS: ARES scored AIS datasets within 2.5 accuracy points of the correct scores using 200 annotated datapoints, distinguishing faithful from hallucinated answers.The AIS evaluation used query–passage–answer examples in which answers were either faithful or non-attributed to the retrieved passage.
  • 5.3 ARES Ranking of Existing RAG Systems: On existing RAG systems, ARES averaged Kendall’s τ of 0.91 for context relevance and 0.97 for answer relevance, exceeding RAGAS by 0.16 and 0.15, respectively.Its PPI confidence bounds captured ground-truth average outcomes more than 95% of the time.
  • 5.4 Strengths and Limits of Cross-Domain Applications: ARES judges generalized across KILT and SuperGLUE domain shifts, with PPI mitigating accuracy losses and additional PPI examples continuing to improve performance.However, drastic shifts involving languages, text-to-code, or extraction tasks remained outside the judges’ effective generalization range.

6 Conclusion

ARES is an automated RAG evaluation framework that fine-tunes lightweight LLM judges on synthetic data and evaluates pipeline components separately with minimal human annotation. Across eight datasets, it accurately scores and ranks systems on context relevance, answer faithfulness, and answer relevance, while identifying further avenues for stronger variants.

  • 6 Conclusion: ARES fine-tunes lightweight LLM judges on synthetically generated queries and answers to evaluate each RAG component separately.The framework targets context relevance, answer faithfulness, and answer relevance, supporting more targeted system understanding.
  • 6 Conclusion: ARES accurately scores and ranks RAG systems across eight datasets in KILT, SuperGLUE, and AIS, outperforming RAGAS.The evaluation covers context relevance, answer faithfulness, and answer relevance scores.
  • 6 Conclusion: ARES variants could improve by replacing human labeling with GPT-4, strengthening synthetic-data construction, using logits for PPI intervals, and testing more sophisticated judges.These are proposed avenues for future exploration rather than evaluated improvements in this work.

7 Limitations

ARES reduces annotation needs but remains bounded by annotation expertise, GPU requirements, and English-only evaluation. These constraints particularly affect specialized domains, access to computational resources, and transfer to other languages.

  • 7 Limitations: ARES relies on roughly 150–300 human preference validation annotations, and specialized domains may require annotators with domain expertise.The annotations can be easier to generate for general-domain applications than for law, medicine, and finance.
  • 7 Limitations: ARES uses substantial GPU resources: DeBERTa-v3-Large and FLAN-T5-XXL required about 32GB of memory and several hours for fine-tuning or generation.Commercial GPUs are available but may remain inaccessible to some researchers and practitioners because of cost.
  • 7 Limitations: All evaluated datasets are English, so the framework’s performance in other languages remains to be explored with different judges and synthetic-data generators.The paper identifies multilingual evaluation as future work for understanding ARES’s strengths and weaknesses.

A.1 Fine-tuning Configuration for LLM Judges

The LLM judges use a standard supervised fine-tuning configuration with cross-entropy loss, Adam optimization, a linear classification head, dropout, and a warmup-decay learning schedule.

  • A.1 Fine-tuning Configuration for LLM Judges: LLM judge training uses cross-entropy loss with Adam optimization.These choices define the loss and optimizer used for the judge models.
  • A.1 Fine-tuning Configuration for LLM Judges: The classification head is a single linear layer with 0.1 dropout applied to the [CLS] token’s final hidden state.The [CLS] representation supplies the input to the classification head.
  • A.1 Fine-tuning Configuration for LLM Judges: All experimental configurations use linear warmup and decay with a 5e-6 learning rate and batch size 32.The schedule and hyperparameters are shared across the reported configurations.

A.2 GPT Prompting for Context Relevance Scoring

ARES uses few-shot prompts to classify whether retrieved documents provide relevant or sufficient context, with prompt wording adapted to the task and dataset.

  • A.2 GPT Prompting for Context Relevance Scoring: For NQ, HotpotQA, MultiRC, and ReCoRD, context relevance prompts ask whether a document is relevant to a dialogue or sufficient for answering a question.The prompts require a strict [[Yes]] or [[No]] verdict without additional explanation.
  • A.2 GPT Prompting for Context Relevance Scoring: The question-based prompt evaluates whether a document is sufficient for answering the question and uses few-shot question-document examples.Its required output is also a binary [[Yes]] or [[No]] decision without explanation.
  • A.2 GPT Prompting for Context Relevance Scoring: For FEVER, the prompt asks whether a document is sufficient to determine a statement’s factuality.The fact-checking formulation adapts the relevance judgment to statement-document factuality assessment.
  • A.2 GPT Prompting for Context Relevance Scoring: WoW has a separate context-relevance prompt, indicating that prompt design is adapted across evaluation datasets.The supplied passage identifies the WoW prompt but does not provide its full wording.
  • A.2 GPT Prompting for Context Relevance Scoring: The dialogue-based prompt evaluates document relevance by considering the document’s content and relation to the dialogue.It uses few-shot dialogue and document examples before requesting the binary verdict.

A.3 GPT Prompting for Answer Faithfulness Scoring

The appendix specifies prompting procedures for judging answer faithfulness and relevance, generating synthetic queries and answers, and evaluating ARES with PPI across RAG systems and domains. It also reports annotation-count requirements, GPT-4 labeling trade-offs, real-world ranking results, and cross-domain generalizability.

  • A.3 GPT Prompting for Answer Faithfulness Scoring: Faithfulness scoring uses eight few-shot examples and asks whether an answer adds information beyond, or contradicts, the document.The verdict format is strictly "[[Yes]]" for faithful answers and "[[No]]" otherwise.
  • A.3 GPT Prompting for Answer Faithfulness Scoring: For FEVER and WoW, the faithfulness prompt replaces “question” with “statement” and “dialogue,” respectively.
  • A.4 GPT Prompting for Answer Relevance Scoring: Answer-relevance scoring uses eight few-shot examples and checks whether the answer addresses all question aspects using correct information from the document.The judge outputs only "[[Yes]]" or "[[No]]" according to the specified relevance criterion.
  • A.4 GPT Prompting for Answer Relevance Scoring: For FEVER and WoW, the answer-relevance prompt similarly substitutes “statement” and “dialogue” for “question.”
  • A.5 Prompting for Generation of Synthetic Queries and Answers: Synthetic query-and-answer generation uses few-shot prompting with Question, Document, and Answer examples, while incorrect or contradictory answers use correspondingly incorrect examples.The generation procedure is described for FLAN-T5 and FLAN-T5 XXL prompts.
  • A.6 Synthetic Query and Answer Generation: Below about 100–150 human preference datapoints, ARES cannot meaningfully distinguish alternate RAG systems by context- or answer-relevance accuracy.
Loading 2311.09476v2…