PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

Yixuan Tang, Yi Yang

arXiv:2401.15391v1cs.CL

TL;DR

Existing RAG benchmarks and systems are not designed to evaluate queries requiring retrieval and reasoning over multiple supporting documents. MultiHop-RAG constructs a news-based benchmark and tests both retrieval and LLM answering, finding current methods inadequate on these tasks. The dataset is intended as a resource for developing and benchmarking multi-hop RAG systems, with scope limitations in answer format, evidence count, and framework coverage.

  • Problem

    Existing RAG benchmarks mainly evaluate single-evidence queries and do not assess retrieval and reasoning for complex multi-hop queries.

  • Method

    MultiHop-RAG combines a news-article knowledge base with GPT-4-assisted claim and query construction, ground-truth answers, supporting evidence, and two benchmark experiments.

  • Results

    Existing RAG implementations are inadequate for retrieving and answering multi-hop queries across both benchmark experiments.

  • Takeaways & Limitations

    MultiHop-RAG provides a publicly released resource for developing and benchmarking RAG systems on multi-hop queries.

  • Takeaways & Limitations

    The dataset restricts answers to simple forms, limits supporting evidence to four pieces, and evaluates a basic LlamaIndex RAG framework.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, utilizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAG in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. The MultiHop-RAG and implemented RAG system is publicly available at https://github.com/yixuantt/MultiHop-RAG/.

1 Introduction

Multi-hop RAG queries require retrieving and reasoning over evidence from multiple documents, but existing benchmarks mainly assess single-evidence cases. MultiHop-RAG addresses this gap with a dataset and benchmark experiments showing current systems remain inadequate.

  • Multi-hop queries require evidence from multiple documents to formulate an answer.Examples include comparing companies’ profit margins or analyzing a company’s sales trend over time.
  • Existing RAG benchmarks mainly evaluate queries solvable from one piece of evidence and do not assess multi-hop retrieval and reasoning.MultiHop-RAG is introduced to make benchmarking more closely resemble real-world scenarios.
  • MultiHop-RAG categorizes multi-hop queries into inference, comparison, temporal, and null types.The first three require evidence retrieval and analysis from multiple sources, while null queries test whether models hallucinate when answers are unavailable.
  • The dataset is constructed from news articles using GPT-4 to generate diverse multi-hop queries requiring retrieval and reasoning over multiple documents.The process extracts factual sentences and rephrases them into claims with disambiguated topics and entities.
  • Two benchmark experiments compare embedding models for evidence retrieval and evaluate state-of-the-art LLMs answering queries given retrieved text.The experiments include GPT-4, GPT-3.5, PaLM, Claude-2, Llama2-70B, and Mixtral-8x7B.
  • Both experiments indicate that current RAG implementations are inadequate for effectively retrieving and answering multi-hop queries.The MultiHop-RAG dataset is publicly released as a resource for developing and benchmarking RAG systems.

2 RAG with multi-Hop queries

Multi-hop queries require a retrieval set whose pieces of evidence collectively support an answer, followed by reasoning across those pieces. The paper organizes these queries into four types and evaluates RAG through separate retrieval and generation tasks.

  • 2.2 Multi-Hop Query: A multi-hop query requires retrieving and reasoning over multiple supporting evidence pieces whose retrieval set collectively provides the answer.A comparison query may require retrieving relevant evidence from several company reports and comparing the resulting values.
  • 2.2 Multi-Hop Query: The four query types are inference, comparison, temporal, and null queries.Inference deduces an answer, comparison evaluates similarities or differences, temporal queries analyze event order, and null queries lack an answer in the retrieval set.
  • 2.2 Multi-Hop Query: Null queries test whether an LLM produces a null response rather than hallucinating when the answer cannot be derived from the retrieved set.The expected response is “insufficient information” or similar.
  • 2.3 Evaluation Metrics: Multi-hop RAG systems are assessed through retrieval evaluation and generation evaluation.Retrieval compares retrieved chunks with ground-truth evidence, while generation compares the LLM response with the ground-truth answer.
  • 2.3 Evaluation Metrics: Retrieval evaluation uses MAP@K, MRR@K, and Hit@K for the top-K retrieved chunks, excluding null queries from evidence comparison.These metrics assess how well retrieved chunks match the query’s associated ground-truth evidence.

3 A Benchmarking Dataset: MultiHop-RAG

MultiHop-RAG is built from English news articles by extracting evidence, generating and checking claims, and assembling multi-hop queries linked by shared entities or topics. The resulting dataset covers diverse news and query types, with most queries requiring multiple evidence pieces.

  • 3.1 MultiHop-RAG Construction: The construction process creates multi-hop queries, ground-truth evidence sets, and answers from a collection of news articles.The dataset collection covers English-language news across entertainment, business, sports, technology, health, and science.
  • 3.1 MultiHop-RAG Construction: Factual or opinion sentences are extracted as evidence, retaining articles whose evidence overlaps in keywords with other articles.This supports later query construction using evidence drawn from multiple sources.
  • 3.1 MultiHop-RAG Construction: GPT-4 paraphrases evidence into claims, while UniEval fact-checks alignment between each claim and its original evidence.Shared entities or topics across evidence pieces serve as bridge-entities or bridge-topics linking claims.
  • 3.1 MultiHop-RAG Construction: Claim sets contain at least two and no more than four claims, and GPT-4 generates queries using information from each claim.The construction includes the associated news source to mimic real-world RAG scenarios.
  • 3.1 MultiHop-RAG Construction: The dataset includes inference, comparison, temporal, and null queries with type-specific answer requirements.Inference identifies an entity, comparison typically yields “yes” or “no,” temporal queries yield an ordering judgment, and null queries require insufficient information.
  • 3.2 Descriptive Statistics: MultiHop-RAG covers 609 distinct news articles averaging 2,046 tokens, and about 88% of queries are non-null.Approximately 42% of queries require two evidence pieces, 30% require three, and 15% require four.

4 Benchmarking RAG system using MultiHop-RAG

MultiHop-RAG benchmarks retrieval and generation for queries requiring multiple pieces of evidence. Experiments show substantial difficulty in both retrieving relevant chunks and reasoning over them, while suggesting several directions for improvement.

  • Benchmark scope: MultiHop-RAG supports retrieval-related and generation-related RAG tasks using a knowledge base of multi-hop queries and evidence.The benchmark distinguishes retrieving relevant text from generating responses given retrieved text.
  • Retrieval-related Task: 256-token document chunks are embedded, stored in a vector database, and retrieved by cosine similarity before reranking.The system retrieves 20 related chunks and then selects top-K chunks with a reranker.
  • Retrieval-related Task: 0.7467 is the highest Hits@10 with reranking, while the highest Hits@4 falls to 0.6625.The reported retrieval metrics highlight difficulty finding relevant evidence for multi-hop queries under limited context windows.
  • Generation-related Task: GPT-4 reaches only 0.56 accuracy with retrieved chunks, indicating that retrieval shortcomings constrain response generation.The generation experiment uses the best-performing retrieval model before feeding retrieved texts to the LLM.
  • Generation-related Task: With ground-truth evidence, GPT-4 reaches 0.89 accuracy, compared with 0.74 for Google-PaLM and 0.32 for Llama2-70B.Mixtral-8x7B achieves 0.36 in the same ground-truth-evidence setting.
  • Generation-related Task: GPT-4 and Mixtral-8x7B are relatively robust on null queries, but Mixtral performs worse on comparison and temporal queries.The reported errors involve logical negation and identifying chronological order; query decomposition, agents, and hybrid retrieval are proposed exploration areas.

5 Related Work

Existing RAG benchmarks and multi-document QA datasets cover parts of the problem but do not fully evaluate retrieval and reasoning over multi-hop evidence in a RAG knowledge base. MultiHop-RAG is positioned to address this distinction while reducing reliance on sources overlapping with LLM training data.

  • RAG Evaluation: RGB and RECALL mainly evaluate response generation and do not specifically address retrieval accuracy.The related-work discussion contrasts their focus with extracting relevant evidence from a large knowledge base.
  • Positioning MultiHop-RAG: MultiHop-RAG differs by evaluating retrieval and reasoning together with evidence drawn from a large news-based knowledge base.This setting is distinguished from claim verification and embedding-API evaluations that are not contextualized within RAG systems.
  • Multi-document QA datasets: HotpotQA, MultiRC, and 2WikiMultiHopQA assess reasoning across multiple documents but do not emphasize retrieving evidence from a knowledge base.Their primary data sources are Wikipedia, which may overlap substantially with existing LLM training data.

6 Conclusion

MultiHop-RAG is introduced as a dataset for queries requiring retrieval and reasoning over multiple supporting evidence pieces, with resources for benchmarking RAG systems.

  • MultiHop-RAG targets queries that require retrieval and reasoning from multiple pieces of supporting evidence.
  • The dataset includes a knowledge base, multi-hop queries, ground-truth answers, and associated supporting evidence.
  • The paper describes a hybrid human–GPT-4 construction process and explores two benchmarking use cases.
  • The dataset is publicly released as a resource for advancing and benchmarking RAG systems.

Limitations

The dataset restricts answer formats and evidence-set size, while its experiments use a basic RAG framework, leaving several directions for future expansion.

  • Ground-truth answers are restricted to simple responses such as yes, no, entity names, or temporal indicators.
  • The answer format supports straightforward accuracy evaluation, but future work could allow free-text answers and use more sophisticated generation metrics.
  • Each query currently has a maximum of four supporting evidence pieces, limiting coverage of queries requiring more evidence.
  • The experiments use a basic LlamaIndex RAG framework, while future evaluations could examine more advanced RAG or LLM-agent frameworks.

A Appendix A: GPT-4 Prompts Used for Data Generation

The appendix presents the prompts used to guide GPT-4 in generating claims and multi-hop queries for the dataset.

  • The appendix presents prompts used to guide GPT-4 for data generation.
  • Table 7 documents claim-generation prompting together with the topics and entities associated with generated claims.
  • Tables 8–10 document prompts for inference, comparison, and temporal multi-hop query generation.

B Appendix B: Dataset Examples

The appendix provides examples and prompting materials for the four MultiHop-RAG query types, pairing queries with answers and supporting evidence for evaluation.

  • Examples cover inference, comparison, temporal, and null query types in Tables 12–15.
  • Each example query is paired with a ground-truth answer for evaluating generation accuracy.
  • Multiple supporting evidence pieces accompany the examples for assessing retrieval performance.
  • Prompting materials include claim-generation, inference-query, comparison-query, temporal-query, and null-query prompts in Tables 7–11.
Loading 2401.15391v1…