PaperlayerView exact revision on arXiv

Source-linked AI summary

CRAG -- Comprehensive RAG Benchmark

Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen-tau Yih, Xin Luna Dong

arXiv:2406.04744v2cs.CL

TL;DR

Existing RAG datasets do not adequately represent the diversity and dynamism of real-world QA, despite RAG’s promise for addressing LLM knowledge limitations. CRAG combines a varied factual QA benchmark with mock retrieval APIs and evaluation tasks; its results show that trustworthy QA remains a substantial challenge.

  • Problem

    Existing RAG datasets do not adequately represent the diverse and dynamic nature of real-world QA, while hallucination and factual accuracy remain challenges for LLM question answering.

    Source: §1 · pp. 1–2

  • Method

    CRAG provides 4,409 QA pairs across five domains and eight question categories, mock web and KG retrieval APIs, and evaluation tasks for RAG systems.

    Source: §1 · p. 2

  • Results

    44% accuracy with straightforward RAG, while industry RAG solutions answer only 63% of questions without any hallucination; accuracy is lower for more dynamic, less popular, or more complex facts.

    Source: §1 · p. 3

  • Takeaways & Limitations

    CRAG’s evaluations highlight gaps and research directions toward fully trustworthy QA, and the authors plan to maintain and expand the benchmark.

    Source: §1 · p. 3, §6 · p. 10

    The three tasks do not cover search retrieval, though they could be extended to use all 220K benchmark webpages as a search corpus.

    Source: §2 · p. 4

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.

1 Introduction

CRAG addresses the lack of a comprehensive RAG benchmark for diverse, dynamic QA with varied factual questions, retrieval mock APIs, and an evaluation mechanism. Its evaluation exposes substantial gaps in trustworthy question answering.

1 Introduction

  • CRAG contains 4,409 QA pairs across five domains and eight question categories, spanning popular to long-tail entities and temporal ranges from seconds to years.Questions draw on smart-assistant use cases, are paraphrased for expression diversity, and have manually verified ground truths.Source: §1 · p. 2
  • CRAG supplements its QA pairs with mock APIs for web search and knowledge graphs, including up to 50 real-world search pages per question and mock KGs with 2.6 million entities.The KG retrieval candidates deliberately include realistic noise.Source: §1 · p. 2
  • Its evaluation mechanism covers retrieval summarization, KG and web retrieval, and end-to-end RAG, penalizing hallucinated answers more than missing answers.An automatic evaluation mechanism supports faster evaluations and iterations.Source: §1 · pp. 2–3
  • Straightforward RAG reaches 44% accuracy, while industry RAG solutions answer only 63% of questions without any hallucination.The evaluation also finds lower accuracy for facts with higher dynamism, lower popularity, or higher complexity.Source: §1 · p. 3
  • Existing QA benchmarks and newer LLM or RAG benchmarks cover limited retrieval settings or particular capabilities, while CRAG is designed for comprehensive, realistic, dynamic, and diverse testing.The authors also commit to maintaining the benchmark for research on RAG and general QA.Source: §1 · p. 3

2 Problem Description

The three CRAG tasks use the same question-answer pairs but vary the external information available to a RAG system. Together they test generation, structured-data querying and synthesis, and ranking larger retrieval sets, but not search retrieval itself.

2 Problem Description

  • A RAG QA system takes a question as input and generates an answer using retrieved external information or knowledge internalized in the model; the answer should avoid hallucination.Source: §2 · p. 3
  • The three tasks share question-answer pairs but differ in the external data available for retrieval, enabling comparisons across different RAG capabilities.Source: §2 · p. 4
  • Task 1 supplies up to five potentially relevant, but not guaranteed relevant, web pages to test answer generation.Source: §2 · p. 4
  • Task 2 adds mock KG APIs that return structured data, testing structured-source querying and synthesis across sources.Answers may or may not exist in the mock KGs.Source: §2 · p. 4
  • Task 3 provides web results and mock APIs with 50 candidate pages, testing how systems rank larger, potentially noisier retrieval sets.The tasks do not cover search retrieval, though the authors note they could be extended to use all 220K benchmark webpages as a search corpus.Source: §2 · p. 4

3 Dataset Description

CRAG combines diverse question-answer pairs with simulated web and Knowledge Graph retrieval. Its estimated web-search coverage is broad but incomplete, with recall gains flattening as more results are included.

3.1 Question answering pairs

  • CRAG covers five domains and eight question types in English, using questions built from both Knowledge Graphs and web content.It includes 2,425 Web Questions and 1,984 KG Questions, with KG questions divided nearly evenly across head, torso, and tail entities.Source: §3.1 · p. 5, Table 4 · p. 5
  • KG questions are generated from entity-relation templates, while web questions are written by annotators from corresponding search results.The KG construction uses more than 600 templates and samples entities across popularity levels.Source: §3.1 · p. 5

3.2 Contents for retrieval

  • Retrieval content includes up to 50 Brave Search HTML pages per question and mock KGs with 2.6 million entities and 38 mock APIs.The mock KGs include randomly selected entities and hard negatives with similar names.Source: §3.2 · pp. 5–6
  • 85% overall estimated web recall requires all 50 pages, while the top five reach about 69%; full pages add about 20% recall over snippets.Estimated recall is 93% for Web Questions versus 74% for KG Questions.Source: §3.2 · p. 6, Figure 2 · p. 6
  • Web and KG data snapshots were collected concurrently with real-time and fast-changing questions to capture information as it existed when questions were asked.The question slices are sized to yield metrics with at most a 5% margin of error at 95% confidence for most cases.Source: §3.1 · p. 5, §3.2 · p. 6

4 Metrics and Evaluation

CRAG scores answers to distinguish useful correctness from missing and incorrect responses, then evaluates systems through human grading and model-based evaluation.

4.1 Metrics

  • Perfect, acceptable, missing, and incorrect answers receive scores of 1, 0.5, 0, and -1, respectively, and truthfulness is their average.The scoring method penalizes hallucinated answers and prefers missing answers to incorrect ones.Source: §4.1 · pp. 6–7

4.2 Evaluation

  • Human evaluation assigns the four answer labels directly, while automatic evaluation combines perfect and acceptable answers as accurate and uses a three-way score.The automatic labels are accurate, incorrect, and missing.Source: §4.2 · p. 7
  • Automatic evaluation first checks exact ground-truth matches, then uses two LLM evaluators; its average F1 scores against human evaluation are 94.7% and 98.9%.The evaluators are ChatGPT and Llama 3, used to avoid self-preference.Source: §4.2 · p. 7
  • The data is split randomly into 30% validation, 30% public test, and 40% private test sets.The challenge uses the private set for submitted solutions, while offline users can report results on the public test set.Source: §4.2 · p. 7

5 Benchmarking

CRAG evaluations show that industry systems remain imperfect and that difficult, dynamic, less popular, and complex questions continue to challenge RAG. Straightforward retrieval improves accuracy, but retrieval noise and context limits constrain performance.

5.1 Straightforward RAG solutions

  • The straightforward-solution experiment uses the 1,335-question public test set, brief-answer prompts, and fixed-length web and KG context windows.The SOTA evaluation queried five systems built on leading LLMs and search engines and used manual grading; its scores also apply traffic weights by question type and macro-average across domains.Source: §5.1 · p. 7, §5.2 · p. 9
  • GPT-4 Turbo achieves 34% accuracy alone, while straightforward RAG reaches up to 44%; no RAG solution exceeds 20% truthfulness in the auto-evaluation.Task 2 truthfulness is higher than Task 1, consistent with KG knowledge being brief and precise, while Task 3 improves over Task 2 with better ranking and recall.Source: §5.1 · p. 8, Table 5 · p. 8
  • For GPT-4 Turbo, truthfulness falls from 21% for head entities to 11% for torso and 8% for tail, while straightforward RAG improves torso and tail quality but lowers head quality.Llama 3 70B is similar or slightly better on simple and comparison questions, but weaker on set and post-processing questions than GPT-4 Turbo.Source: §5.1 · pp. 8–9

5.2 State-of-the-art industry solutions

  • On the CRAG benchmark, SOTA solutions achieved highest truthfulness of 51%, while hallucination rates ranged from 16% to 25%; real-time, fast-changing, torso/tail-entity, and reasoning-intensive questions remained difficult.Their truthfulness scores are not completely comparable to straightforward solutions because retrieval access and evaluation methods differed; latency ranged from 3.4s to 11.6s.Source: §5.2 · p. 9
  • Industry systems improve truthfulness on set and false-premise questions relative to straightforward solutions, while high dynamism, lower popularity, and complex reasoning remain challenges.Lower truthfulness persists for aggregation, multi-hop, and post-processing queries.Source: §5.2 · pp. 9–10

6 Conclusion

The authors propose CRAG as a comprehensive RAG benchmark and plan to expand it with multilingual, multimodal, and multiturn questions.

6 Conclusion

  • The authors plan to expand CRAG with multilingual questions, multimodal questions, and multiturn conversations.They intend to maintain the benchmark as research needs and challenges evolve.Source: §6 · p. 10

A.1.1 Constructing QA pairs from KGs

CRAG’s KG-derived questions are built from entity-relation templates, sampled entities, and their associated attributes. More complex questions use deterministic templates or two-hop paths.

A.1.1 Constructing QA pairs from KGs

  • KG questions pair templates for an entity type and relation with sampled entities and their associated attribute values as answers.Entities are sampled from top, middle, and bottom popularity buckets, with equal numbers of questions in each bucket.Source: §A.1.1 · p. 13
  • Comparison, aggregation, set, post-processing, and false-premise questions use crisp, deterministic templates and entities selected to fit each question.The entity types for these categories are selected using heuristics.Source: §A.1.1 · p. 13
  • Multi-hop questions follow a two-relation KG path, with a template describing the path and the answer taken from its second triplet.For example, the template can ask for the CEO of a company’s parent company.Source: §A.1.1 · p. 13

A.1.2 Definition of dynamism categories

Table 7 is titled as a definition of dynamism categories.

A.1.2 Definition of dynamism categories

  • Table 7 is identified as defining dynamism categories.The supplied passage gives the table title but not its category definitions.Source: Table 7 · p. 13

A.1.3 Constructing QA pairs from web contents

Web questions begin with annotator-written questions that could be answered by search. Annotators then review search results to establish answers, especially for time-sensitive questions.

A.1.3 Constructing QA pairs from web contents

  • Annotators write questions that could be answered through web search, guided by general instructions and examples.One example asks for the most popular action movie in 2023.Source: §A.1.3 · p. 13
  • Annotators review generated search results to determine ground-truth answers, using further searches when results lack the necessary information.For fast-changing and real-time questions, the answer is determined by the query time when the search occurred.Source: §A.1.3 · p. 13
  • Web questions also receive domain, dynamism, question-type, and answer-URL labels.The URL is one that contains the answer to the question.Source: §A.1.3 · p. 14

A.1.4 Validation for the QA pairs

The appendices describe validation, mock retrieval data, evaluation procedures, challenge design, system testing, latency measurement, and a boundary of the benchmark’s evaluation scope.

A.1.4 Validation for the QA pairs

  • Question and meta-label audits used two human reviews, with a third auditor resolving conflicts; answer audits corrected 5% of answers, and targeted checks added 2% updates.The answer audits covered web and KG questions; the latter were manually checked against mock APIs.Source: §A.1.4 · p. 14

A.1.6 The mock data and mock APIs

  • Mock retrieval data pairs full HTML pages from Brave Search with noisy mock KGs containing about 2.6M entities and a signal-to-noise ratio below 1/30.The pages and KGs provide unstructured and structured information, respectively.Source: §A.1.6 · p. 14

A.2.1 Human evaluation

  • Human evaluation grades answers as missing, incorrect, acceptable, or accurate, while automatic evaluation merges acceptable and perfect answers and penalizes incorrect ones.The human-evaluation protocol uses two graders and a third to resolve conflicts; the automatic truthfulness score is accuracy minus hallucination.Source: §A.2.1 · pp. 15–16, §A.2.2 · p. 16

A.2.3 KDD Cup 2024 Meta CRAG challenge

  • The KDD Cup split data into validation, public-test, and private-test sets of 30%, 30%, and 40%, using automatic evaluation in Stage 1 and automatic screening followed by manual evaluation for finalists.The shared validation and public-test sets contained 2,706 examples.Source: §A.2.3 · p. 17

A.3.2 Performance of straightforward RAG solutions

  • Straightforward RAG prompts include query time and retrieved references, and Table 11 summarizes the solutions’ results.For Tasks 2 and 3, references include web results and KG responses.Source: §A.3.1 · p. 18, §A.3.2 · p. 19, Table 11 · p. 19

A.4.2 Latency

  • The industry-system quality comparison omitted Meta SG’s on-device speech components; latency was measured from query submission to completed response.Perplexity.ai’s API latency was 2,455 ms, but the paper notes API and web-interface latency are not directly comparable.Source: §A.4.1 · p. 19, §A.4.2 · p. 20

A.5 Limitations

  • The benchmark does not directly evaluate first-stage retrieval-candidate construction, though users can build a retriever over its 220K web pages.The paper says this corpus does not match the entire web but supports fair comparisons at manageable cost.Source: §A.5 · p. 20

Related papers

Loading 2406.04744v2…