PaperlayerView exact revision on arXiv

Source-linked AI summary

CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models

Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, Enhong Chen

arXiv:2401.17043v3cs.CL

TL;DR

Existing RAG benchmarks focus largely on question answering and do not fully evaluate retrieval and knowledge-base construction. CRUD-RAG introduces a Chinese benchmark across Create, Read, Update, and Delete tasks. The paper reports that optimizing retrieval, context length, knowledge-base construction, and the language model matters for best results.

  • Problem

    Existing RAG benchmarks mainly assess question answering and often omit retrieval methods beyond knowledge-intensive tasks and external knowledge-base construction.

    Source: §1 · p. 2

  • Method

    CRUD-RAG builds Chinese datasets for four CRUD application types and evaluates RAG systems across tasks and system factors.

    Source: §1 · pp. 2–4

  • Results

    The authors highlight optimizing the retrieval model, context length, knowledge-base construction, and underlying LLM to achieve the best results.

    Source: §5 · p. 28

  • Takeaways & Limitations

    The authors present their findings as guidance for researchers and practitioners developing and refining RAG systems.

    Source: §5 · p. 28

    Because manual annotation is costly and the dataset is large, the authors initially polished one-fifth of it manually.

    Source: §3.4 · p. 13

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate "hallucinated" content. However, the evaluation of RAG systems is challenging, as existing benchmarks are limited in scope and diversity. Most of the current benchmarks predominantly assess question-answering applications, overlooking the broader spectrum of situations where RAG could prove advantageous. Moreover, they only evaluate the performance of the LLM component of the RAG pipeline in the experiments, and neglect the influence of the retrieval component and the external knowledge database. To address these issues, this paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios. Specifically, we have categorized the range of RAG applications into four distinct types-Create, Read, Update, and Delete (CRUD), each representing a unique use case. "Create" refers to scenarios requiring the generation of original, varied content. "Read" involves responding to intricate questions in knowledge-intensive situations. "Update" focuses on revising and rectifying inaccuracies or inconsistencies in pre-existing texts. "Delete" pertains to the task of summarizing extensive texts into more concise forms. For each of these CRUD categories, we have developed comprehensive datasets to evaluate the performance of RAG systems. We also analyze the effects of various components of the RAG system, such as the retriever, the context length, the knowledge base construction, and the LLM. Finally, we provide useful insights for optimizing the RAG technology for different scenarios.

1 INTRODUCTION

Existing RAG benchmarks mainly assess question answering and often omit retrieval and knowledge-base construction. CRUD-RAG evaluates four application types with Chinese datasets and experiments on factors affecting system performance.

Motivation

  • Existing reference-required benchmarks center on question answering and often omit retrieval methods outside knowledge-intensive tasks and external knowledge-base construction.The authors argue that these limitations leave the full performance of RAG systems unevaluated.Source: §1 · p. 2

Benchmark scope

  • CRUD-RAG covers text continuation, single- and multi-document question answering, hallucination modification, and open-domain multi-document summarization across four CRUD application types.The four types are Create, Read, Update, and Delete.Source: §1 · pp. 2–3

Dataset construction

  • The benchmark uses recent Chinese news and automatically created datasets, with task-specific construction strategies for summarization, continuation, and question answering.For multi-document summarization, GPT-4-generated events and summaries guide web searches for 10 related, non-duplicate reports; continuation retrieval uses 10 reports per continuation sentence.Source: §1 · pp. 3–4

Experiments

  • The paper systematically evaluates RAG performance and investigates context length, chunk size, embedding model, retrieval algorithms, and the LLM.The authors state that their experiments inform suggestions for building effective RAG systems.Source: §1 · p. 4

2 RELATED WORK

Related work reviews RAG pipeline and component optimization, benchmark approaches, and citation-focused evaluation. The paper positions CRUD-RAG as a broader framework spanning RAG components and application scenarios.

2.1 Retrieval-Augmented Generation

  • Conventional RAG uses indexing, retrieval, and generation; retriever optimization spans pre-retrieval processing, model optimization, and post-retrieval processing.Generator methods include Chain-of-Note for assessing retrieved-document relevance and domain-specific fine-tuning.Source: §2.1 · pp. 4–5

2.2 RAG Benchmarks

  • RAG benchmarks use reference-required measures such as accuracy and reference-free measures of context relevance, faithfulness, and answer relevance.Examples include RGB [8] and RAG-Instruct-Benchmark-Tester for reference-required evaluation, and TruLens-Eval, RAGAS [13], and ARES [44] for reference-free evaluation.Source: §2.2 · p. 6

2.3 Citation-Enhanced RAG

  • Citation-enhanced RAG work evaluates whether generated claims are supported, with methods including AIS, SCIFI [7], and ALiiCE [54].The paper distinguishes its benchmark as covering multiple RAG components and application scenarios rather than focusing specifically on citation generation.Source: §2.3 · p. 7

3 CRUD-RAG: A COMPREHENSIVE CHINESE BENCHMARK FOR RAG

CRUD-RAG addresses QA-centered, component-limited RAG evaluation with a Chinese benchmark organized around Create, Read, Update, and Delete. It supplies tailored tasks and datasets for those scenarios and evaluates system components across them.

3 CRUD-RAG: A COMPREHENSIVE CHINESE BENCHMARK FOR RAG

  • CRUD-RAG broadens RAG evaluation beyond question answering with tailored tasks for Create, Read, Update, and Delete, while examining component effects across tasks.The benchmark includes text continuation, question answering, hallucination modification, and multi-document summarization.Source: §3 · pp. 7–8, Figure 2 · p. 8

3.1 News Collection

  • News data was chosen to reduce overlap with LLM pretraining, support timely dataset rebuilding, and provide diverse topics; the collected corpus contains more than 80,000 articles.The authors describe news as readily collectable and suitable for testing performance across domains and situations.Source: §3.1 · pp. 8–9

3.2 Open-domain Multi-document Summarization: RAG Application in "Delete"

  • To avoid costly manual selection and summarization, the multi-document summarization dataset is built in reverse: GPT-4 creates event summaries, then Baidu search finds 10 related reports per event.The expanded reports form the retrieval corpus used to generate summaries for the original articles.Source: §3.2 · pp. 9–11

3.3 Text continuation: RAG Application in "Create"

  • For text continuation, articles are split into input and target halves, while Baidu search adds documents relevant to each continuation paragraph to the retrieval database.The task tests whether retrieval can support coherent, informative continuation consistent with the article.Source: §3.3 · p. 11

3.4 Question Answering: RAG Application in "Read"

  • The QA datasets pair single-document factual questions with multi-document questions requiring inference across two or three documents; GPT-4 CoT is used to construct the latter.Multi-document questions are refined to require logically connected reasoning rather than simply juxtaposing topics.Source: §3.4 · pp. 12–13
  • 5.8% of generated queries required refinement, which the authors say indicates that GPT-4-generated queries are generally high quality.The dataset generation process includes manual review and refinement of queries judged to need it.Source: §3.4 · p. 13

3.5 Hallucination Modification: RAG Application in "Update"

  • For hallucination modification, GPT-4 corrects hallucinated continuations from UHGEval, while real news continuations are placed in the retrieval database for retrieval and correction tests.The input combines the beginning text with a hallucinated continuation.Source: §3.5 · pp. 13–14

3.6 Evaluation Method

  • RAGQuestEval generates questions from ground-truth references, answers them using references and generations, and scores answerable-question coverage as recall and answer similarity as precision.Precision uses token-level F1 similarity and excludes unanswerable questions.Source: §3.6 · p. 15

4 EXPERIMENT

The experiments examine how retrieval, external knowledge-base construction, and the language model affect RAG system performance.

4 EXPERIMENT

  • The experiments examine how the retrieval model, external knowledge-base construction, and language model affect RAG performance.The authors frame component-level evaluation as addressing a gap in existing RAG benchmark evaluations.Source: §4 · p. 15

4.1 Experimental Settings

The experiments vary RAG system components to measure their impact across tasks, using a defined baseline and two families of evaluation metrics.

4.1 Experimental Settings

  • The system retrieves top-k similar vectors from a vector database, then supplies the retrieved chunks and query to an LLM for answer generation.Chunk size, overlap, and embedding model affect context quality; the retriever affects context richness and diversity, while top-k determines context size.Source: §4.1 · pp. 15–16
  • The baseline uses chunk size 128, 0% overlap, bge-base embeddings, dense retrieval, top-k 8, and GPT-3.5; experiments vary one component at a time.The tested alternatives include chunk sizes 64–512, overlap rates 0%–70%, four embedding models, four retriever configurations, top-k values 2–10, and six LLMs.Source: §4.1 · p. 16
  • Evaluation uses semantic-similarity metrics (BLEU, ROUGE-L, and bertScore) and RAGQuestEval for key-information coverage.To reduce costs with GPT-3.5 as the baseline, experiments use one-fifth of the dataset.Source: §4.1 · p. 16

4.2 Analyzing the Impact of Chunk Size on RAG Performance in Different Tasks

Chunk-size effects differ across RAG tasks: larger chunks suit some generation and multi-document tasks, while smaller chunks suit more localized extraction or correction.

4.2 Analyzing the Impact of Chunk Size on RAG Performance in Different Tasks

  • Different RAG tasks have different optimal chunk sizes, with results varying across text continuation, summarization, question answering, and hallucination modification.The experiments compare chunk sizes using semantic-similarity and key-information metrics.Source: §4.2 · p. 16, Table 3 · p. 17
  • For text continuation, larger chunks improve BLEU and ROUGE-L, while RAGQuestEval follows the same pattern.The authors attribute this to larger blocks preserving document structure, whereas small chunks can fragment content and impede coherent generation.Source: §4.2 · p. 16
  • For multi-document summarization, larger chunks increase generated length and key-information recall but lower BLEU and key-information precision; ROUGE-L and bertScore remain nearly unchanged.The authors hypothesize that larger chunks retrieve more relevant content but require finer selection, reducing precision.Source: §4.2 · pp. 16–17
  • For single-document question answering, overly large chunks reduce key-information recall and precision, making smaller chunks more suitable for extracting a specific sentence or subparagraph.Source: §4.2 · p. 17
  • For multi-document question answering, larger chunks improve key-information recall and precision as well as semantic similarity, with retained article structure supporting reasoning.Source: §4.2 · p. 17
  • For hallucination modification, smaller chunks improve semantic-similarity metrics such as BLEU; the errors often concern individual words or entities rather than whole documents.The authors note that this task can use relevant portions without requiring comprehension of the full document.Source: §4.2 · p. 18

4.3 Analyzing the Impact of Chunk Overlap on RAG Performance in Different Tasks

Chunk overlap has task-dependent effects: it generally benefits continuation, summarization, and question answering, but has little impact on hallucination correction.

4.3 Analyzing the Impact of Chunk Overlap on RAG Performance in Different Tasks

  • For text continuation, increasing chunk overlap slightly improves BLEU, ROUGE-L, and bertScore, while RAGQuestEval improves more noticeably.The authors interpret this pattern as consistent with overlap preserving idea flow in creative generation.Source: §4.3 · p. 19
  • For multi-document summarization, all metrics improve slightly with greater overlap, and key-information recall is highest at 70% overlap.Despite concern that overlap could reduce context variety, the reported recall does not decrease.Source: §4.3 · p. 19
  • For question answering, overlap minimally affects semantic-similarity metrics but improves key-information accuracy and recall for single- and two-document tasks, less so for three-document tasks.The authors suggest that larger overlap may reduce available context in three-document QA, which requires richer context.Source: §4.3 · p. 19
  • For hallucination modification, chunk-overlap changes have minimal effect, consistent with errors typically being localized to individual words or entities.Source: §4.3 · p. 19

4.4 Analyzing the Impact of Retriever on RAG Performance in Different Tasks

Hybrid retrieval with reranking performs best in most tasks, but the relative strengths of dense and BM25 retrieval vary by task. Multi-document question answering particularly favors dense retrieval, while BM25 is competitive in continuation and performs well when precise retrieval matters.

Text continuation and summarization

  • In text continuation and summarization, dense retrieval and BM25 are similar overall, but BM25 favors key-information precision while dense retrieval can better capture recall.For continuation, BM25 can retrieve content relevant to user intent while overlooking some details; in summarization, it can miss important information.Source: §4.4 · pp. 20–21

Question answering

  • Dense retrieval has a clearer advantage over BM25 in multi-document reasoning QA, and Hybrid+Rerank improves all evaluation metrics in question answering.The reported advantage is especially evident when questions require synthesizing three documents.Source: §4.4 · pp. 21–22

Hallucination modification

  • For hallucination modification, BM25 performs slightly better than or equal to dense retrieval and uses fewer computational resources.The findings indicate that retrieval-algorithm performance depends on the RAG task.Source: §4.4 · p. 22

Retrieval accuracy evaluation

  • MRR retrieval-accuracy evaluation covers question answering and hallucination modification, but excludes continuation and open-domain summarization because their criteria lack clear ground truth.The pure hybrid algorithm is not evaluated separately because it could alter retrieved-content order and affect later processing.Source: §4.4 · p. 22

Overall results

  • Hybrid+Rerank outperforms other retrieval methods in most tasks, while BM25 and dense retrieval perform comparably in many cases.In question answering, performance declines for all methods as the number of documents increases.Source: §4.4 · p. 22

4.5 Analyzing the Impact of Embedding Model on RAG Performance in Different Tasks

Embedding-model performance depends on the task: the tested models are close overall, but individual tasks show notable exceptions. MRR also declines as question-answering tasks require more documents.

Task-specific results

  • Embedding-model performance differences are small on creative continuation, where the relevance of retrieved content is often ambiguous.Source: §4.5 · p. 23
  • For single-document QA requiring precise document localization, m3e-base performs much worse than the other models, matching the finding of [38].The paper reports the opposite pattern for hallucination modification, where m3e-base outperforms other models on all metrics despite ranking lowest on the retrieval benchmark.Source: §4.5 · p. 23

Retrieval accuracy

  • BGE, GTE, M3E, and STELLA have relatively close MRR performance across tasks, with no single method leading in all tasks.The paper therefore emphasizes task requirements when selecting an embedding method.Source: §4.5 · p. 23
  • As question-answering document count rises from one to three, MRR declines for all tested embedding methods.This trend aligns with the paper’s end-to-end experiments on multi-document understanding.Source: §4.5 · p. 23

4.6 Analyzing the Impact of Top-k on RAG Performance in Different Tasks

The effect of increasing top-k varies by task: it can improve information coverage, but may also reduce summary quality or precision. Hallucination modification is relatively insensitive to top-k except when it becomes too large.

Text continuation

  • Increasing top-k benefits continuation across semantic-similarity and RAGQuestEval metrics, with higher key-information recall and precision.The paper attributes the gains to greater diversity and accuracy from more documents.Source: §4.6 · p. 23

Open-domain multi-document summarization

  • For multi-document summarization, higher top-k increases key-information recall but decreases precision, while BLEU drops and summaries become longer and lower-quality.ROUGE-L and BERTScore remain almost unchanged.Source: §4.6 · p. 23

Question answering

  • For QA, increasing top-k improves key-information recall and precision in multi-document tasks, while single-document QA sees little semantic-similarity change.For single-document QA, the RAGQuestEval gains are especially notable when the initial top-k is too small.Source: §4.6 · pp. 23–24

Hallucination modification

  • In hallucination modification, top-k has little effect on semantic-similarity and key-information metrics unless it becomes too large.The paper notes that this task needs little context for correction and that models show some resistance to interference.Source: §4.6 · p. 25

4.7 Analyzing the Impact of LLM on RAG Performance in Different Tasks

LLM performance varies across tasks: GPT-4 excels in continuation and summarization and in multi-document QA, while some smaller models outperform it on single-document QA. Newer models show mixed changes relative to predecessors.

Text continuation and summarization

  • GPT-4 performs better than other tested large models in text continuation and excels in summarization on semantic accuracy, key-information recall and precision, and concision.Source: §4.7 · p. 25

Question answering

  • For multi-document QA, GPT-4 is far ahead of other models, while Qwen and Baichuan2 can outperform GPT-series models on single-document QA.Baichuan2-13B also performs better than GPT-3.5 on multi-document QA.Source: §4.7 · p. 25

Hallucination modification

  • ChatGLM2, Qwen-7B, and Baichuan2 do not complete hallucination modification well because some models generate overly long text with redundant or irrelevant content.The task requires correcting hallucinations while retaining other information.Source: §4.7 · p. 25

Overall results

  • Across most tasks and metrics, GPT-4 performs excellently; Qwen-7B and Qwen-14B also perform well, especially in continuation and summarization.Baichuan2-13B is described as competitive with GPT-4 in QA.Source: §4.7 · p. 25

Latest LLM evaluation

  • GPT-4o performs similarly to GPT-4 or improves slightly, while Qwen2-7b improves significantly over Qwen-7b in multiple tasks.The authors report that the benchmark remains challenging for these newer models.Source: §4.7 · p. 25

4.8 Suggestions for Optimizing Your RAG System

The section offers task-specific guidance for tuning retrieval, context construction, and model choices in RAG systems. Recommendations vary by task and by whether precision, recall, or semantic understanding is needed.

4.8 Suggestions for Optimizing Your RAG System

  • Top-k should vary by scenario: larger values suit creative generation, single-document QA, and multi-document reasoning, while smaller values favor high precision and moderate values balance summary precision and recall.Larger top-k can increase computational cost in creative tasks.Source: §4.8 · pp. 26–27
  • Chunk size should preserve article structure for creative generation and reasoning, be smaller for factual QA, and vary in summarization according to whether recall or precision matters more.Smaller chunks are suggested for summary scenarios when more information should be recalled; larger chunks when precision matters.Source: §4.8 · p. 27
  • Large chunk overlap is recommended for creative generation, summarization, and QA to maintain semantic consistency, while smaller overlap can be considered for error correction.Source: §4.8 · p. 27
  • Embedding-model leaderboard performance may differ from performance in the RAG system, so selection should be evaluated and adjusted for the specific scenario.The section points to the mteb leaderboard [38] as a reference for retrieval-task performance.Source: §4.8 · pp. 27–28
  • BM25 saves computational resources and often retrieves relevant documents, but its keyword matching may miss semantic content; the authors recommend it for precision-focused generation such as hallucination modification and summarization.For question answering and creative generation, they recommend dense algorithms based on deep-learning embeddings instead.Source: §4.8 · p. 28
  • A hybrid dense-and-BM25 retriever adds little overall quality, while reranking retrieval results improves almost all tasks, especially reasoning.The authors recommend hybrid retrieval plus reranking when conditions permit.Source: §4.8 · p. 28
  • Qwen-14B performs similarly to GPT-4 on text continuation and summary generation, while Baichuan2-13B is close to GPT-4 on QA, supporting task- and cost-dependent model choice.The authors note GPT-4's high invocation cost.Source: §4.8 · p. 28

5 CONCLUSION

The paper introduces CRUD-RAG, a comprehensive, scenario-specific framework with large-scale datasets for evaluating RAG systems across varied text-generation contexts. Its experiments report that incorporating external knowledge can significantly enhance generated-content quality, and its findings address multiple factors involved in RAG tuning.

5 CONCLUSION

  • CRUD-RAG organizes evaluation into Create, Read, Update, and Delete task types and provides large-scale datasets tailored to each category.The framework is described as assessing RAG capabilities and limitations across varied textual contexts and operational conditions.Source: §5 · p. 28
  • Experiments demonstrate that RAG systems can significantly enhance generated-content quality by incorporating information from external knowledge sources.Source: §5 · p. 28
  • The study identifies retrieval models, context length, knowledge-base construction, and the underlying LLM as factors to optimize when fine-tuning RAG systems.The authors present their findings as guidance for researchers and practitioners refining RAG systems.Source: §5 · p. 28

Related papers

Loading 2401.17043v3…