Source-linked AI summary
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
TL;DR
Existing RAG benchmarks often isolate retrieval, factuality, and reasoning, leaving end-to-end performance on realistic multi-hop queries insufficiently measured. The paper introduces FRAMES, a unified evaluation dataset and multi-step retrieval framework for these capabilities. The benchmark exposes gaps in current LLMs while multi-step retrieval improves performance, with potential pretraining contamination remaining a limitation.
Problem
Existing benchmarks typically evaluate retrieval, factuality, and reasoning in isolation rather than jointly measuring end-to-end RAG performance.
Method
The paper introduces FRAMES, an 824-question benchmark, and evaluates baseline and retrieval-based prompting for factuality, retrieval, reasoning, and synthesis.
Results
State-of-the-art LLMs struggle with FRAMES’ complex multi-hop tasks, while multi-step retrieval and reasoning improve performance over baseline prompting.
Takeaways & Limitations
FRAMES provides an integrated benchmark for assessing current RAG capabilities and identifying gaps in retrieval and reasoning.
Takeaways & Limitations
Potential pretraining-data contamination may artificially inflate performance and reduce FRAMES’ ability to measure true generalization.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated significant performance improvements across various cognitive tasks. An emerging application is using LLMs to enhance retrieval-augmented generation (RAG) capabilities. These systems require LLMs to understand user queries, retrieve relevant information, and synthesize coherent and accurate responses. Given the increasing real-world deployment of such systems, comprehensive evaluation becomes crucial. To this end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set), a high-quality evaluation dataset designed to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. While previous work has provided datasets and benchmarks to evaluate these abilities in isolation, FRAMES offers a unified framework that provides a clearer picture of LLM performance in end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions that require the integration of information from multiple sources. We present baseline results demonstrating that even state-of-the-art LLMs struggle with this task, achieving 0.40 accuracy with no retrieval. The accuracy is significantly improved with our proposed multi-step retrieval pipeline, achieving an accuracy of 0.66 (>50% improvement). We hope our work will help bridge evaluation gaps and assist in developing more robust and capable RAG systems.
1 Introduction
FRAMES addresses fragmented RAG evaluation by jointly testing factual retrieval, multi-hop reasoning, and coherent synthesis in realistic end-to-end settings. The paper introduces an 824-sample benchmark and a multi-step retrieval approach that improves performance on complex queries.
- Existing benchmarks often assess retrieval, factuality, and reasoning separately, limiting holistic evaluation of real-world RAG systems.
- FRAMES integrates fact retrieval, reasoning across multiple constraints, and accurate information synthesis in one end-to-end evaluation framework.
- 824 test samples evaluate LLMs’ ability to retrieve and reason across multiple documents within a unified framework.
- The authors provide empirical evaluation of state-of-the-art LLMs across factuality, retrieval, and reasoning tasks spanning diverse domains.
- The benchmark includes complex multi-hop and temporal reasoning tasks intended to reveal limitations in existing LLM capabilities.
- 0.408 accuracy with single-step inference increased to 0.66 with multi-step retrievals.
2 FRAMES
FRAMES is an 824-question benchmark for end-to-end RAG evaluation, built from multi-document questions and quality-controlled human annotation. Its questions span diverse topics, article counts, and reasoning types while addressing ambiguity, guesswork, reliability, and contamination concerns.
- FRAMES contains 824 questions designed to evaluate factuality, retrieval, and reasoning together in end-to-end RAG scenarios.
- Data collection: Human annotators created multi-article questions after synthetic generation showed hallucination-related problems, improving reliability and accuracy.
- Dataset Statistics: Questions span diverse Wikipedia topics and require between 2 and 15 articles to answer.
- Dataset Statistics: Approximately 36% of questions require two articles, 35% require three, and 16% require four articles.
- Dataset Statistics: Approximately 36% of questions require reasoning over multiple constraints, while approximately 20% require numerical reasoning.
- Quality Checks: Quality controls included temporal disambiguation, removal of binary-answer questions, Wikipedia-only sourcing, and reasoning beyond simple fact retrieval.
3 Empirical Analysis
The experiments compare single-step baselines with iterative BM25 retrieval and search planning. Multi-step retrieval improves accuracy, reaching 0.66 under the best reported configuration, while single-step errors remain concentrated in several reasoning types.
- Single-Step Evaluations: Gemini-Pro-1.5-0514 achieves 0.408 accuracy with naive prompting, 0.452 with two BM25-retrieved documents, 0.474 with four, and 0.729 with Oracle Prompt.The Oracle Prompt supplies all gold Wikipedia articles, providing an upper-bound comparison for the single-step setting.
- Single-Step Evaluations: BM25 retrieval improves multiple-constraint accuracy by approximately 8% and post-processing accuracy by approximately 10%.The reported improvements are attributed to adding relevant articles containing facts for each constraint.
- Multi-Step Evaluations: The multi-step pipeline generates search queries, retrieves top documents with BM25, adds new documents to context, and produces a final response after n iterations.Experiments compare vanilla retrieval with retrieval guided by explicit search-planning instructions.
- Multi-Step Evaluations: Vanilla multi-step retrieval raises accuracy from approximately 0.45 to approximately 0.52 for (k,n,n_docs)=(5,5,2).Increasing steps and queries adds relevant documents and improves recall, but the process requires six non-parallelizable inference calls in this setting.
- Multi-Step Evaluations: 0.66 accuracy is reached after five retrieval iterations with search-planning instructions, approaching the 0.73 Oracle Prompt result.The best configuration reported in Figure 4 is (k,n,n_docs)=(5,5,10); the instructions include example query sequences, non-repetition, and step-by-step reasoning.
4 Related Works
Prior RAG benchmarks typically isolate retrieval, factual correctness, or explainability rather than evaluating these capabilities together. FRAMES unifies factual retrieval, multi-hop reasoning, temporal disambiguation, and coherent synthesis in realistic multi-document queries.
- Existing Benchmarks: NaturalQuestions emphasizes retrieval precision, TriviaQA factual correctness, and ELI5 explainability without rigorous multi-hop reasoning evaluation.These benchmarks therefore cover distinct aspects of RAG evaluation rather than the full combination of capabilities.
- FRAMES: FRAMES evaluates factual retrieval, reasoning, and synthesis through complex multi-hop queries requiring information integration from multiple sources.It also includes temporal disambiguation and assesses whether retrieved information is synthesized into coherent responses.
5 Conclusion
The paper introduces FRAMES as an integrated benchmark for factuality, retrieval accuracy, and reasoning in RAG systems. Its experiments expose gaps in complex multi-hop reasoning, while realistic multi-document queries provide a more holistic evaluation context.
- Contributions: FRAMES tests RAG systems across factuality, retrieval accuracy, and reasoning using a comprehensive evaluation dataset.The experiments with state-of-the-art LLMs reveal gaps in handling complex, multi-hop reasoning tasks.
- Implications: FRAMES offers an integrated framework that evaluates RAG systems more holistically than existing benchmarks through realistic multi-document queries.The authors state that this provides a clearer picture of current capabilities and limitations in real-world applications.
- Implications: The findings underscore the importance of improving both retrieval mechanisms and reasoning capabilities to improve overall performance.
Limitations
FRAMES may be affected by pretraining-data contamination and limited coverage of real-world queries and scenarios.
- Pretraining-data contamination could artificially inflate performance metrics and reduce FRAMES’ ability to measure true generalization.The authors suggest newer or synthetic data, or methods to quantify and account for contamination.
- Limited diversity may constrain FRAMES’ applicability to some domains or use cases.The authors identify broader coverage as important for more robust and reliable RAG evaluation.
Ethical Considerations
The paper highlights ethical considerations for RAG research, including source bias, misuse risks, computational impacts, and the need for transparency.
- Researchers should consider biases in information sources, safeguards against misuse, and the environmental impact of computationally intensive RAG approaches.The paper encourages efficient and accessible implementations.
- The authors encourage transparent methodologies and continued community dialogue about ethical aspects of RAG research.
A Future Work
Future work targets stronger retrieval and reasoning capabilities for complex multi-hop RAG tasks, while the appendices document prompts used for synthetic generation and evaluation.
- Future research: Future retrieval research could develop dense retrievers trained directly on multi-hop retrieval tasks, including ColBERT- or SimCSE-based approaches.These approaches are described as potentially adapting to diverse and complex queries iteratively.
- Future research: Improving LLM reasoning remains a significant challenge, with process supervision identified as one avenue for future research.
- Synthetic data generation: The synthetic-generation prompt uses k_context and n_questions as placeholders for the number of input articles and questions generated per inference.
- Evaluation: The response-rating prompt compares model responses with ground-truth answers and checks whether the gold answer is present in the response.
- Human annotation: Human annotation instructions describe creating factoid questions requiring multi-hop reasoning across multiple Wikipedia articles.