ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems

1 Introduction

• Domain-specific RAG evaluation demands costly expert annotations or human preference comparisons, while fixed-prompt model evaluators offer limited adaptability and no quality guarantees.Suitable RAG designs vary with domain, corpus size, and cost/latency budgets. RAGAS exemplifies evaluation using fixed, heuristically written prompts.Source: §1 · p. 1
• Approximately 150 or more annotated validation datapoints accompany an in-domain passage set and few-shot query–answer examples as ARES’s required inputs.The query–answer examples support synthetic data generation, with five or more given as an example. Prediction-powered inference (PPI; Angelopoulos et al. 2023) uses validation annotations to improve scoring accuracy and provide confidence intervals.Source: §1 · p. 2
• ARES generates synthetic question–answer pairs from in-domain passages, then contrastively fine-tunes three lightweight judges for context relevance, answer faithfulness, and answer relevance.These dimensions assess whether retrieved information addresses the question, answers are grounded in retrieved context, and answers address the question. ARES’s code and datasets are publicly available on Github.Source: §1 · pp. 1–2
• 59.3 and 14.4 percentage-point average accuracy gains over RAGAS were reported for context relevance and answer relevance, respectively, across six KILT and SuperGLUE datasets.RAGAS is the existing automated evaluation framework used for this comparison.Source: §1 · pp. 1–2
• 78% fewer annotations were required by ARES than the annotation-based baseline, alongside substantially greater evaluation accuracy.ARES also consistently distinguished competitive RAG systems separated by only a few points in ground-truth metrics, supporting comparison of approaches and configurations.Source: §1 · p. 2
• Within 2.5 percentage points of the ground-truth average, ARES predicted answer hallucinations on the AIS attribution dataset (Rashkin et al., 2022).Source: §1 · p. 2

• Retrieval supports domain knowledge, factual grounding (Shuster et al., 2021; Huo et al., 2023), and source-based transparency (Mialon et al., 2023).RAG combines LLMs with retrieval systems (Lewis et al., 2020; Khattab et al., 2021; Izacard et al., 2022).Source: §2 · p. 2
• AutoCalibrate (Liu et al., 2023b) uses a self-refinement prompt to iteratively improve the LLM judge toward human preferences, but offers no statistical guarantees for the accuracy of its predictions.Source: §2 · p. 2
• LongEval (Krishna et al., 2023), Hagrid and ALCE (Gao et al., 2023) support analysis of knowledge-intensive LLM pipelines.LongEval provides guidelines, while Hagrid and ALCE provide datasets; related work uses LLMs to assess attribution and factuality (Min et al., 2023; Gekhman et al., 2023; Yue et al., 2023).Source: §2 · p. 2
• RAGAS uses heuristic hand-written prompts that offer limited adaptability to new evaluation settings and substantially underperform ARES.New corpora are one example of settings where adaptability is limited.Source: §2 · pp. 2–3
• EXAM scores generated responses through exam-question answering, requiring several sub-questions per query—a burden ARES avoids.A simulated reader, implemented as a QA system, answers questions using the generated response.Source: §2 · p. 2

3 ARES

• ARES trains judges on synthetic query–passage–answer triples, then combines their RAG-system scores with human annotations through prediction-powered inference to estimate quality confidence intervals.The judges score a sample of each evaluated system’s in-domain query–document–answer triples.Source: §3 · p. 3
• Approximately 150 annotated datapoints or more, five or more in-domain query–answer examples, and an in-domain passage set are ARES’s required inputs.The annotations form the human preference validation set; the query–answer examples support few-shot prompting during synthetic-data generation.Source: §3 · p. 3

3.1 LLM Generation of Synthetic Dataset

• ARES primarily uses FLAN-T5 XXL with few-shot prompts to generate positive and negative query–passage–answer triples, while allowing alternative high-quality generators.Training examples cover relevant and irrelevant passages, as well as correct and incorrect answers.Source: §3.1 · p. 3
• ARES filters synthetic queries by requiring their original passage to rank first under retrieval and generates equal numbers of negatives using two strategies.The retrieval check removes low-quality queries before judge training.Source: §3.1 · p. 3

3.2 Preparing LLM Judges

• ARES fine-tunes DeBERTa-v3-Large judges to assess context relevance, answer faithfulness, and answer relevance.Context relevance checks whether retrieved passages answer the query; faithfulness checks whether answers remain supported by those passages. Answer relevance checks whether answers are relevant given the query and retrieved passage.Source: §3.2 · p. 3

1. Weak Negative Generation: For context rel-

• ARES trains DeBERTa-v3-Large judges using metric-specific weak and strong synthetic negatives, balancing positive and negative counts for context relevance and answer relevance.Weak negatives use unrelated in-domain passages for context relevance and FLAN-T5 XXL answers from other passages for answer faithfulness and relevance. Strong negatives use same-document passages, falling back to sampling BM25’s top-10 similar passages when unavailable, and few-shot-prompted contradictory FLAN-T5 XXL answers for answer faithfulness and relevance.Source: §3.2 · p. 3, §1 · p. 3

3.2 Preparing LLM Judges

• Each metric receives a separate judge with a binary classifier head that labels concatenated query-document-answer triples as positive or negative.Training stops after three epochs without validation-loss improvement. The pipeline requires in-domain passages, at least 150 human-annotated validation datapoints, and at least five few-shot examples of in-domain queries and answers.Source: §3.2 · p. 3, Figure 1 · p. 4

3.3 Ranking RAG Systems with Confidence Intervals

• ARES averages predicted labels across sampled query-document-answer triples for each metric, then ranks systems using confidence-interval midpoints.The rankings support comparisons between different RAG systems and between configurations of the same system within a domain.Source: §3.3 · p. 4
• Prediction-powered inference (PPI; Angelopoulos et al. 2023) combines limited human annotations with abundant judge predictions to construct tighter confidence intervals.Judge-only scores can be inaccurate, while human-only evaluation requires costly additional annotations for each system. PPI learns a rectifier from validation-set predictions to estimate judge errors; the experiments use 95% confidence intervals.Source: §3.3 · p. 4

4.1 Models

• ARES trains DeBERTa-v3-Large (He et al., 2021) judges on synthetic queries and answers generated by FLAN-T5 XXL (Chung et al., 2022).The judges rank RAG systems without external APIs and can run on commercial GPUs.Source: §4.1 · p. 5
• Baselines use zero/few-shot GPT-3.5 and RAGAS, while similarity search filters synthetic queries that cannot retrieve their source passages.The experiments use gpt-3.5-turbo-16k, version 10/23, (Brown et al., 2020) and RAGAS version 0.0.18. Filtering uses FAISS IndexFlatL2 with text-embedding-ada-002 embeddings.Source: §4.1 · p. 5

4.2 Datasets

• Experiments span KILT (Petroni et al., 2021) and SuperGLUE (Wang et al., 2019), covering question answering, fact verification, dialogue, and entity completion.KILT tasks include NQ for single-passage questions, HotpotQA for multipassage reasoning, FEVER for SUPPORTS/REFUTES judgments, and WoW for passage-grounded dialogue responses. MultiRC and ReCoRD are adapted to open-domain retrieval over seven domain collections and news articles, respectively; ReCoRD asks for placeholder entities.Source: §4.2 · p. 5
• Answer faithfulness is not evaluated on KILT or SuperGLUE because human-annotated hallucinated answers are unavailable.The framework is instead tested on real attribution datasets in Section 5.2.Source: §4.2 · p. 5
• Nine mock RAG systems per dataset span 70%–90% success rates in 2.5-percentage-point increments, providing known rankings for evaluating ARES.Original examples supply positives; passages and answers sampled from the same or entirely random Wikipedia documents supply negatives. These artificial query-passage-answer combinations test context relevance and answer relevance across small performance gaps.Source: §4.2 · p. 5

4.3 Metrics

• Kendall’s τ and prediction accuracy evaluate ARES, emphasizing accurate pairwise comparisons between RAG configurations across different performance gaps.Kendall’s τ compares experimental rankings with perfectly accurate rankings; values greater than 0.9 are considered successful.Source: §4.3 · p. 6

5.1 ARES Ranking

• 0.08 higher average Kendall’s τ accompanied 78% fewer annotations for ARES versus sampled annotations across context and answer relevance.The sampled baseline used 150 annotations per mock RAG system, totaling 1,350 annotations. PPI improved the fine-tuned judge’s ranking accuracy on every tested dataset.Source: §5.1 · p. 6
• 0.06 higher average Kendall’s tau distinguished ARES from the GPT-3.5 judge across context and answer relevance.GPT-3.5 was more readily deployable because it required no additional fine-tuning, but incurred querying costs dependent on timing and token usage.Source: §5.1 · p. 6
• 0.065 higher Kendall’s τ for context relevance and 0.132 higher for answer relevance marked ARES’s average ranking gains over RAGAS.ARES used DeBERTa-v3-Large judges and PPI with 300 human validation datapoints. Triple-level prediction accuracy exceeded RAGAS by 59.9 percentage points for context relevance and 14.4 percentage points for answer relevance.Source: §5.1 · p. 6, Table 1 · p. 7
• 150 human annotations were the minimum required in tests spanning 25–400 annotations; GPT-4 generations performed worse than human annotations as replacements.The authors nevertheless considered replacing human annotations with GPT-4 generations promising.Source: §5.1 · p. 7

5.2 ARES Performance on AIS

• 2.5 accuracy points bounded ARES’s deviation from correct AIS scores, using only 200 human validation annotations per system to evaluate answer faithfulness.Evaluation covered WoW and CNN/DM, excluding ToTTo’s table reasoning and QRECC’s passage summarization.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 ARES Ranking of Existing RAG Systems

• 0.91 Kendall’s tau for context relevance and 0.97 for answer relevance were ARES’s average ranking results on existing RAG systems.ARES exceeded RAGAS by 0.16 for context relevance and 0.15 for answer relevance; confidence intervals captured ground-truth averages more than 95% of the time, averaging 7.4 and 6.1 points wide, respectively. ColBERTv2 was the best-performing retriever, and GPT-4 was the best-performing generative LLM.Source: §5.3 · p. 8

5.4 Strengths and Limits of Cross-Domain Applications

• 300 human validation datapoints supported strong cross-domain generalization of ARES judges through PPI.Tests changed query types, document types, or both between training and evaluation. PPI mitigated losses in judge accuracy, and additional PPI examples further improved cross-domain performance.Source: §5.4 · p. 8
• ARES judges failed to generalize reliably across more drastic shifts involving languages, text-to-code tasks, or extraction tasks.An NQ-fine-tuned judge achieved Kendall’s tau over both context relevance and answer relevance of 0.33 on XGLUE (Liang et al., 2020), 0.28 on CodeSearchNet (Husain et al., 2019), and 0.38 on T-Rex (Petroni et al., 2021). Each cross-domain shift requires in-domain passages and few-shot query examples to reconfigure ARES judges.Source: §5.4 · p. 8

• Eight datasets across KILT, SuperGLUE, and AIS show ARES outperforming RAGAS in scoring and ranking RAG systems for context relevance, answer faithfulness, and answer relevance.Source: §6 · p. 8
• ARES fine-tunes lightweight LLM judges on synthetically generated queries and answers, evaluating RAG components separately with minimal human annotation to support targeted improvements.Source: §6 · p. 8
• Future directions include replacing human labeling with GPT-4, improving synthetic training data, using judge logits to improve PPI confidence intervals, and fine-tuning more sophisticated judges.Source: §6 · pp. 8–9

7 Limitations

• Roughly 150-300 human validation datapoints are required, potentially needing expert annotators for specialized domains.More annotations are better; applications in law, medicine, and finance may require specialized expertise.Source: §7 · p. 9
• About 32GB of GPU memory was required for ARES’s models, with hardware costs limiting accessibility for some researchers and practitioners.DeBERTa-v3-Large (304M) fine-tuning and FLAN-T5-XXL (11.3B) generation each took several hours.Source: §7 · p. 9
• ARES was evaluated only on English datasets, leaving its applicability to other languages unexplored.Future work should investigate other languages using different LLMs for judging and synthetic data generation.Source: §7 · p. 9

A.1 Fine-tuning Configuration for LLM Judges

• LLM judge training uses cross-entropy loss with Adam.Source: §A.1 · p. 11
• The classification head consists of a single linear classification layer.Source: §A.1 · p. 11
• The learning schedule uses linear warmup and linear decay (Howard and Ruder, 2018).Source: §A.1 · p. 11
• 5e-6 is the learning rate across all experimental configurations.Source: §A.1 · p. 11
• 32 is the training batch size across all experimental configurations.Source: §A.1 · p. 11
• 0.1 dropout is applied to the classification head’s input, the final hidden state of the [CLS] token.Source: §A.1 · p. 11

• 8 few-shot examples guide context-relevance scoring for NQ, HotpotQA, MultiRC, and ReCoRD, judging whether a document is sufficient to answer the question.Source: §A.2 · p. 11
• FEVER’s expert fact-checking prompt judges whether a document is sufficient to determine a statement’s factuality.Source: §A.2 · p. 11
• WoW’s expert dialogue prompt judges whether a document is relevant for responding to the provided dialogue.Source: §A.2 · pp. 11–12

A.3 GPT Prompting for Answer Faithfulness Scoring

• 8 few-shot examples guide answer-faithfulness scoring on NQ, HotpotQA, MultiRC, and ReCoRD, rejecting answers that add information beyond or contradict the document.The prompt requires only [[Yes]] or [[No]], without explanation. For FEVER and WoW, “question” becomes “statement” and “dialogue,” respectively.Source: §A.3 · p. 12

A.4 GPT Prompting for Answer Relevance Scoring

• 8 few-shot examples guide answer-relevance scoring on NQ, HotpotQA, MultiRC, and ReCoRD, requiring coverage of all question aspects using only correct document information.The prompt requires only [[Yes]] or [[No]], without explanation. For FEVER and WoW, “question” becomes “statement” and “dialogue,” respectively.Source: §A.4 · p. 12

A.5 Prompting for Generation of Synthetic Queries and Answers

• 5 few-shot examples prompt FLAN-T5 to generate synthetic queries and answers; replacing demonstrations with incorrect or contradictory answers generates those answer types.The prompting structure remains unchanged when generating incorrect or contradictory answers.Source: §A.5 · p. 12

A.6 Synthetic Query and Answer Generation

• FLAN-T5 XXL generates synthetic queries from in-domain passages, then generates answers conditioned on a synthetic query and its passage.Separate query-generation and answer-generation prompts end with an unfinished Query: or Answer: field for completion.Source: §A.6 · pp. 12–13
• About 100-150 human validation datapoints were needed for ARES to meaningfully distinguish RAG systems by context relevance and answer relevance accuracies.Kendall’s tau measured correlation between the correct ranking and ARES’s ranking of pseudo RAG systems.Source: Table 3 · p. 15
• 0.05 to 0.30 decreases in Kendall’s tau occurred in most settings when 500 GPT-4 labels replaced human validation annotations.Human annotation requirements fell from hundreds to fewer than ten examples for few-shot prompts, and PPI efficacy improved with more GPT-4 labels. Evaluation used the fine-tuned DeBERTa-v3-Large judge.Source: Table 4 · p. 15
• ARES ranked RAG systems more accurately than RAGAS and a few-shot GPT-3.5 judge across all explored datasets for context relevance and answer relevance.PPI improved ranking accuracy over the fine-tuned judge without PPI. ARES and the GPT-3.5 judge used 300 human validation annotations; the sampled-annotation benchmark used 150 annotated datapoints per mock RAG system.Source: Table 5 · p. 16
• Fine-tuned judges generalized across changes in query type, document type, or both for context relevance and answer relevance.PPI used 300 labeled validation examples, with additional examples improving performance. It mitigated out-of-domain judge accuracy losses, including NQ-to-FEVER answer relevance.Source: Table 6 · p. 16