PaperlayerView exact revision on arXiv

Source-linked AI summary

RAG vs. GraphRAG: A Systematic Evaluation and Key Insights

Haoyu Han, Li Ma, Yu Wang, Harry Shomer, Yongjia Lei, Zhisheng Qi, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Charu C. Aggarwal, Jiliang Tang

arXiv:2502.11371v3cs.IR

TL;DR

Text-based GraphRAG has shown promise, but heterogeneous task, dataset, and system settings leave its comparison with RAG insufficiently understood. This paper conducts a unified benchmark across QA and query-based summarization, then studies hybrid strategies and practical evaluation issues. It finds complementary task-dependent strengths: RAG favors precise single-hop and detail-oriented queries, while GraphRAG favors multi-hop reasoning and corpus-level diverse summaries, with hybrid approaches improving QA performance.

  • Problem

    Existing text-based GraphRAG studies use heterogeneous protocols, leaving the relative strengths, limitations, and trade-offs between RAG and GraphRAG insufficiently understood on general benchmarks.

  • Method

    The paper benchmarks RAG and four GraphRAG categories across QA and query-based summarization using standardized preprocessing, retrieval, and generation settings, then evaluates Selection and Integration strategies.

  • Results

    RAG is stronger for single-hop and detail-oriented QA, while GraphRAG benefits multi-hop reasoning and produces more corpus-level, diverse summaries; hybrid strategies improve QA performance.

  • Takeaways & Limitations

    Effective retrieval-augmented generation should adapt retrieval and aggregation to query needs while combining graph-based reasoning benefits with realistic efficiency constraints.

  • Takeaways & Limitations

    Current GraphRAG systems remain limited by incomplete or noisy graph construction, additional computation and storage overhead, and position effects in LLM-as-a-Judge summarization evaluation.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) improves large language models (LLMs) by retrieving relevant information from external sources and has been widely adopted for text-based tasks. For structured data, such as knowledge graphs, Graph Retrieval-Augmented Generation (GraphRAG) retrieves and aggregates information along graph structures. More recently, GraphRAG has been extended to general text settings by organizing unstructured text into graph representations, showing promise for reasoning and grounding. Despite these advances, existing GraphRAG systems for text data are often tailored to specific tasks, datasets, and system designs, resulting in heterogeneous evaluation protocols. Consequently, a systematic understanding of the relative strengths, limitations, and trade-offs between RAG and GraphRAG on widely used text benchmarks remains limited. In this paper, we present a comprehensive benchmark study comparing RAG and GraphRAG on established text-based tasks, including question answering and query-based summarization. We introduce a unified evaluation protocol that standardizes data preprocessing, retrieval configurations, and generation settings, enabling fair and reproducible comparisons. Our results highlight the distinct strengths of RAG and GraphRAG across different tasks and evaluation perspectives. Building on these findings, we explore selection and integration strategies that combine the strengths of both paradigms, leading to consistent performance improvements. We further analyze failure modes, efficiency trade-offs, and evaluation biases, and highlight key considerations for designing and evaluating retrieval-augmented generation systems.

1 Introduction

The paper benchmarks RAG and GraphRAG under a unified protocol to clarify their task-dependent strengths, costs, and evaluation challenges. It finds complementary behavior and motivates hybrid strategies that combine both paradigms.

  • Benchmark scope: The benchmark compares RAG with four GraphRAG categories across question answering and query-based summarization under standardized preprocessing, retrieval, and generation settings.The GraphRAG categories are KG-based, community-based, text-centric graph-guided, and hierarchical summary-based systems.
  • Task-level findings: RAG performs better on single-hop and detail-oriented factual queries, whereas GraphRAG is more effective for multi-hop, reasoning-intensive questions.
  • Design trade-offs: Community-based global search can sacrifice query-specific details while providing corpus-level aggregation that benefits broad or diverse summarization outputs.
  • Evaluation challenges: LLM-as-a-Judge summarization evaluation is sensitive to candidate presentation order, introducing position effects that can confound comparisons.
  • Efficiency considerations: GraphRAG often incurs higher construction cost, retrieval latency, and storage footprint, with performance sensitive to graph-construction quality.
  • Hybrid strategies: Selection and Integration strategies combine RAG and GraphRAG strengths and achieve consistent improvements while exposing effectiveness–efficiency trade-offs.

2 Related Works

Prior work has established RAG and GraphRAG as useful retrieval paradigms, but text-based GraphRAG lacks a consistent comparison with standard RAG. Heterogeneous protocols and graph-construction costs motivate systematic evaluation under unified settings.

  • RAG: RAG retrieves relevant information from external sources and is widely applied across text-based tasks and domains.
  • GraphRAG: GraphRAG incorporates relational signals from graph-structured data and has been extended to text-centric retrieval through graph representations.
  • Research gap: Text-based GraphRAG studies use heterogeneous graph-construction methods, retrieval configurations, and evaluation criteria, limiting direct comparison with standard RAG.
  • Practical trade-offs: Graph construction adds indexing time, retrieval latency, and storage costs and can be sensitive to construction quality.
  • Motivation: These gaps motivate a systematic benchmark under unified experimental settings.

3 Evaluation Framework

The evaluation framework separates retrieval from generation and standardizes core settings where possible. It compares dense RAG with four GraphRAG classes across QA and query-based summarization in single- and multi-document settings.

  • Evaluation protocol: The framework saves retrieved evidence for each method and uses a unified generation script to produce outputs under identical settings whenever applicable.
  • RAG pipeline: Dense RAG segments documents into chunks, embeds them in a shared vector space, and retrieves top-ranked chunks by query similarity.
  • GraphRAG classes: KG-based GraphRAG constructs a knowledge graph from text, aligns query-relevant entities to graph nodes, and traverses multi-hop neighborhoods for relational retrieval.
  • GraphRAG classes: The evaluated GraphRAG classes include KG-based, community-based, text-centric graph-guided, and hierarchical summary-based approaches.Community methods organize KGs into hierarchical communities; text-centric methods retain chunks as retrieval units; hierarchical methods support coarse-to-fine retrieval.
  • Tasks: Experiments cover single-hop and multi-hop QA plus single-document and multi-document summarization, with retrieval scope matched to the task setting.
  • Standardization: Core settings standardize graph construction, approximately 256-token chunking, and embedding and retrieval budgets where applicable.Graphs required by several GraphRAG methods are constructed using GPT-4o-mini.

4 Question Answering

The QA evaluation compares RAG and GraphRAG across single-hop, multi-hop, and fine-grained query categories under standard metrics and inference settings. Results show complementary strengths: RAG favors detailed factual questions, while GraphRAG favors multi-hop reasoning, with performance shaped by graph design, construction quality, and inference enhancements.

  • Datasets and Evaluation Metrics: The benchmark covers NQ, HotPotQA, MultiHop-RAG, and NovelQA, using F1, precision, recall, or accuracy according to each dataset's established evaluation protocol.NQ represents single-hop QA; HotPotQA and MultiHop-RAG represent multi-hop QA, while NovelQA provides finer-grained query categories.
  • QA Main Results: RAG performs strongly on detailed single-hop questions, including NQ and the single-hop and detail-oriented NovelQA subsets.The result is reported for the vanilla comparison across the main QA benchmarks.
  • QA Main Results: GraphRAG methods perform best on multi-hop QA benchmarks and remain competitive on the multi-hop NovelQA subset.The reported examples include HippoRAG2 and Community-GraphRAG (Local).
  • QA Main Results: Community-GraphRAG (Global) can lose fine-grained evidence in high-level community summaries, hurting detail-oriented and Null-query QA while benefiting Comparison and Temporal queries.The latter query types require global information, whereas Null queries ideally require insufficient-information answers.
  • QA Main Results: KG-based GraphRAG generally underperforms on QA when extracted graphs omit answer-critical entities and relations.Only about 65.8% of answer entities appeared in the constructed KG for HotPotQA and 65.5% for NQ.
  • QA with Reranking and Iterative Retrieval: Reranking and iterative retrieval improve all methods over vanilla inference, with larger absolute gains on MultiHop-RAG and GraphRAG typically outperforming RAG under enhanced strategies.The single-hop advantage of RAG on detail-oriented questions remains, while Community-GraphRAG (Local) with IRCoT performs notably poorly on Null queries.
  • Comparative QA Analysis: RAG and GraphRAG answer distinct query sets, with 13.6% GraphRAG-only and 11.6% RAG-only queries on MultiHop-RAG.These mutually exclusive correct-answer groups motivate selection and integration strategies that exploit their complementary behavior.
  • Improving QA Performance: Selection and Integration both improve QA performance, while Integration generally performs better and Selection is more computationally efficient.On MultiHop-RAG with Llama 3.1-70B, improvements over the best baseline are 1.1% for Selection and 6.4% for Integration; Integration runs both methods for every query.

5 Query-Based Summarization

The section evaluates RAG and GraphRAG on query-based summarization, contrasting reference-based results with LLM-as-a-Judge assessments and examining position bias. Results depend on retrieval design and query scope: detailed, role- or event-focused benchmarks favor methods retaining specific evidence, while global search supports broader summaries.

  • Motivation: A systematic comparison remains needed because prior GraphRAG summarization work emphasized global summarization and heterogeneous evaluation settings.The section uses standard query-based summarization benchmarks and reference-based metrics to address this gap.
  • Reference-based results: RAG, RaptorRAG, and HippoRAG2 generally perform well because they retrieve original text chunks aligned with ground-truth summaries.
  • Reference-based results: KG-based GraphRAG improves when triplets are combined with corresponding text, adding details that bring outputs closer to human-written references.
  • Reference-based results: Community-based GraphRAG performs better with local search because it retrieves entities, relations, and low-level communities, whereas global search retrieves only high-level summaries.The selected datasets emphasize specific roles or events, making detailed information important.
  • Integration: The Integration strategy often performs comparably to RAG alone, so concatenating RAG and GraphRAG evidence does not reliably improve alignment with detailed references.
  • LLM-as-a-Judge: LLM-as-a-Judge evaluations compare comprehensiveness and diversity under two presentation orders, with Figure 4 covering local and global GraphRAG comparisons.The evaluation reports each method's preference proportion separately for the two orders to examine position effects.

6 Conclusion

The conclusion synthesizes a unified evaluation of RAG and GraphRAG across question answering and query-based summarization, using controlled comparisons to identify where graph structures help. It reports task-dependent strengths, hybrid QA strategies, and practical challenges involving graph quality, efficiency, and evaluation reliability.

  • Conclusion: The study benchmarks RAG and GraphRAG across question answering and query-based summarization under controlled settings to clarify when explicit graph structures help.
  • Conclusion: RAG is consistently effective for single-hop, detail-oriented queries, whereas GraphRAG is more advantageous for multi-hop, reasoning-intensive QA and more corpus-level, diverse summaries.
  • Conclusion: Selection and Integration combine RAG and GraphRAG strengths and improve QA performance.
  • Practical challenges: Practical challenges include incomplete or noisy graph construction, additional computation and storage overhead, and position effects in LLM-as-a-Judge summarization evaluation.
  • Evaluation scope: The evaluation covers four QA datasets and four query-based summarization datasets spanning single-hop, multi-hop, single-document, and multi-document settings.

B.1 Results with LLaMA 3.1-70B on NQ and Hotpot datasets

This appendix section reports additional LLaMA 3.1-70B and LLaMA 3.1-8B results for RAG and GraphRAG on QA datasets. It organizes results by dataset, method, model size, and table, but the supplied passages do not state numerical outcomes.

  • Reported results: Table 8 reports RAG and GraphRAG performance on NQ and Hotpot using LLaMA 3.1-70B.
  • Reported results: Table 9 reports RAG and GraphRAG performance on MultiHop-RAG using LLaMA 3.1-70B.
  • Reported results: Table 10 reports KG-GraphRAG (Triplets) performance using LLaMA 3.1-8B.
  • Reported results: Tables 11–15 report LLaMA 3.1-70B results for RAG, KG-GraphRAG variants, and Community-GraphRAG local and global search.

C Retrieval accuracy of different methods

Retrieval accuracy depends strongly on graph construction and inference strategy. Community-based graph retrieval improves coverage, while iterative retrieval and reranking generally improve QA but can reduce abstention accuracy on insufficient-information queries.

  • Retrieval accuracy is measured by whether the ground-truth answer string appears in the retrieved context, using HotpotQA and NQ.
  • Only 65.8% of answer entities appear in the HotpotQA KG and 65.5% in the NQ KG, limiting KG-GraphRAG retrieval coverage.
  • Community-GraphRAG improves retrieval performance by leveraging community-level summarization, whereas triplet-only KG-GraphRAG suffers from incomplete knowledge graphs.
  • Iterative retrieval improves both RAG and GraphRAG overall, but Community-GraphRAG (Local) NULL-query accuracy falls from 80.07 to 50.50 on MultiHop-RAG.
  • RAG remains stronger for single-hop and detail-oriented questions, while GraphRAG performs better on multi-hop and reasoning-intensive queries.
  • Reranking consistently improves QA, but can reduce abstention accuracy for NULL queries, suggesting stronger evidence selection may encourage over-generation.

H RAG and GraphRAG Integration

Combining RAG and GraphRAG retrieval often improves question answering, but gains depend on model capacity and query type. For query-based summarization, integration is generally comparable to RAG because detailed references favor directly retrieved text.

  • Concatenating RAG and GraphRAG results before generation improves performance in most integration settings.
  • Integration can hurt the Llama 3.1–8B model on MultiHop-RAG by increasing hallucination and incorrect answers, whereas the 70B model is more robust to longer contexts.
  • Integration yields notable gains for Comparison and Temporal queries across both model sizes.
  • For query-based summarization, integration is generally comparable to RAG rather than significantly better because human-written references emphasize detailed, faithful text representations.
  • GraphRAG’s added structured content does not consistently improve alignment with detailed ground-truth summaries and can produce slightly lower scores.

I Query-based Summarization Results with Llama 3.1-70B model

The Llama 3.1-70B experiments report query-based summarization results across single-document and multi-document settings. The supplied passages identify the evaluated datasets and corresponding result tables but do not state the numerical outcomes.

  • The experiments use LLaMA 3.1-70B for query-based summarization.
  • Single-document summarization results are reported in Table 25.
  • Multi-document summarization results are reported in Table 26.

K The LLM-as-a-Judge Results on more datasets

Additional LLM-as-a-Judge results on SQuALITY and ODSum-meeting show trends consistent with the main-paper evaluations. The evaluation compares RAG with local and global GraphRAG under different presentation orders, while graph-extraction quality is also examined.

  • Additional LLM-as-a-Judge results on SQuALITY and ODSum-meeting show trends consistent with the main reported evaluations.
  • Using GPT-4o instead of GPT-4o-mini for graph extraction generally improves GraphRAG on question answering and summarization, while the relative RAG-versus-GraphRAG conclusion remains consistent across graph-construction backbones.
  • The evaluation uses an expert prompt to select which generated summary performs better according to specified criteria.
  • Figure 9 compares RAG with GraphRAG-Local and GraphRAG-Global under two answer presentation orders.
  • Order 1 presents the RAG result before GraphRAG, whereas Order 2 reverses that presentation order.

M Computation and Storage Analysis

The analysis compares the token budgets of RAG and Community-GraphRAG and tests whether matching their retrieved input sizes changes the reported performance conclusions.

  • The study analyzes retrieved-token counts alongside runtime and storage, with the token results reported in Table 31.
  • RAG retrieves the top-10 text chunks, whereas Community-GraphRAG (Local) retrieves the top-10 entities and their associated relations.
  • Community-GraphRAG produces significantly more input tokens because it includes entities, descriptions, relations, relation descriptions, and community summaries.
  • A token-matched experiment increases RAG’s retrieved text chunks to equal Community-GraphRAG’s total input-token count, with results reported across Tables 32–35.
  • Increasing RAG’s input size yields slight performance gains, while the study states that its main conclusions remain unchanged.
Loading 2502.11371v3…